I'm convinced that VMs are the right primitive here, for now. Being able to give an agent full root and passing it in just the stuff you want it to have is super easy and it's extremely foolproof. I have my assistants free to install software, run docker, build their own nested VMs, etc. knowing that the boundary is sound and that no capabilities will ever be sacrificed.
I might switch to LXC to reduce the weight somewhat (easy with incus) but this requires providing a more limited set of tools (i.e. podman instead of docker).
bwrap is great, but you're stuck with the limitations of the environment, which depending on what you're doing may neuter the agent.