I looked into docker and then realized the problem I'm actually trying to solve was solved in like 1970 with users and permissions.
I just made a agent user limited to its own home folder, and added my user to its group. Then I run Claude code etc as the agent user.
So it can only read write /home/agent, and it cannot read or write my files.
I add myself to agent group so I can read/write the agent files.
I run into permission issues sometimes but, it's pretty smooth for the most part.
Oh also I gave it root to a $3 VPS. It's so nice having a sysadmin! :) That part definitely feels a bit deviant though!