Skip to content

About

Benchmark of Python sandboxes for AI-agent code (Monty, Ouros, smolagents, RestrictedPython, Pyodide), toward a Python runtime with declarative capabilities

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 

Repository files navigation

python-with-capabilities

This project studies Python runtimes that guardrail AI-agent code with capabilities. The goal (see AGENTS.md) is a runtime with three properties:

  • Python syntax and semantics, so that off-the-shelf LLMs use it correctly;
  • a declarative capability policy restricting what an agent can do;
  • a policy one can reason about and, ideally, prove things about.

The repository holds a survey of prior art and a reproducible benchmark of the existing in-process Python sandboxes for agent code. Sister project: bash-with-capabilities.

Main take-away: Monty is the best

Monty (Pydantic, Rust) is the best in-process Python sandbox for agents in our benchmark of 2026-11-17. The others are Ouros, smolagents' LocalPythonExecutor, RestrictedPython and Pyodide.

  • 🔐 No escape, and the richest policy.
    • Monty blocks all 47 escape probes. Ouros lets linecache read a host file. Pyodide in Node runs host commands through os.system and pickle, and opens host sockets.
    • Monty is the only runtime that enforces read-only and read-write path grants (with MountDir), an environment allowlist, and host-function grants, all with no violation.
    • It has no import allowlist, but it needs none: os and pathlib are virtual and reach only what is mounted.
  • 🧯 Contains runaway code. Its time limit cannot be caught by the code it stops: it stops infinite loops, exception-swallowing loops, recursion, regex backtracking and huge powers. Memory caps exist but are off by default.
  • ⚡ Fast. Monty runs at 1.1–2.6× CPython's time, with a separate worker process and 26 MiB peak RSS. smolagents is 5–80× slower than CPython, and Pyodide needs 4.3 s just to start.

Caveats:

  • Monty covers a subset of Python: 74 of 102 conformance scripts match CPython. The gaps are mostly in classes (4/11: decorators on methods, @property), generators and some stdlib modules. Most failures are loud.
  • It has 2 silent errors: round(2.675, 2) returns 2.68, and a caught assert prints an extra line.
  • It cannot run third-party packages.
  • Grants other than mounts are host callbacks (code), and its only static check is a type checker, not a policy checker.

Pyodide has the best fidelity (100/102, since it is real CPython), but it is not a sandbox in Node. smolagents and RestrictedPython filter live CPython objects; they block every escape probe here, but have the lowest fidelity: 64 and 51 of 102, with 6 silent errors for smolagents.

Contents

  • PRIOR-ART.md: a survey covering:
    • CPython's failed sandboxes (rexec, Bastion) and object-capability proposals;
    • CaMeL and agent policy layers;
    • in-process runtimes;
    • isolation in containers, microVMs and the kernel.
  • bench/: the benchmark.

Reproduce

cd bench
./setup.sh             # one virtualenv per runtime (uv) and Pyodide (npm), pinned versions
./run.py selfcheck     # validate the instruments: CPython is the known positive, a no-op the known negative
./run.py all --json results/$(date +%F).json --report results/$(date +%F).md

The harness needs Linux and Python 3 (standard library only). The resource suite also needs systemd-run --user, prlimit and GNU time.

About

Benchmark of Python sandboxes for AI-agent code (Monty, Ouros, smolagents, RestrictedPython, Pyodide), toward a Python runtime with declarative capabilities

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages