Add host-backed OS and filesystem access to CodeMode sandbox (#262)

* feat(code_mode): expose host-backed OS access to the sandbox

Sandboxed `run_code` had no way to reach the filesystem, environment, or
wall clock: Monty supports it through an OS callback / `AbstractOS` and
directory mounts, but `CodeMode` never threaded `os`/`mount` into
`feed_start` or the snapshot resume loop, so callers couldn't enable it.

Add `os` and `mount` options on `CodeMode`/`CodeModeToolset`, thread them
through `feed_start` and every `resume` site (OS auto-dispatch stops the
moment a resume omits them), and make the `run_code` description reflect
whether host-backed access is configured.

* test(code_mode): harden OS-access tests around the threading invariants

Add edge cases that pin the behaviours most likely to regress: OS access
surviving across REPL-persisted `run_code` calls, a raising `os` callback
degrading to `ModelRetry` instead of crashing the loop, and `mount`
accepting a `list[MountDir]`. Hoist the never-invoked callback used by the
description/forwarding assertions into one shared helper.

* docs(code_mode): tighten and verify the filesystem/OS access section

Trim the host-access docs to the essentials and make the example
self-contained (drop the undefined helper). The snippet and the documented
`mount`/callback constructions are run end-to-end to confirm they work.

* docs(code_mode): correct per-request scoping wording

`os`/`mount` are static capability fields (no per-run resolver), so the
"stateful AbstractOS rooted at a per-user directory" guidance over-claimed.
Reword to: build CodeMode per request to scope access. Every other doc line
was re-checked empirically against pydantic-monty 0.0.17.

* fix(code_mode): don't advertise env/clock for mount-only sandboxes

A `mount` only exposes filesystem paths; `os.getenv`/`os.environ` and
`datetime.now()`/`date.today()` still require an `os` handler. The
description used one host-access note for both, so mount-only agents were
told env/clock were routed to the host and would emit calls that fail and
burn run_code retries (verified against pydantic-monty 0.0.17).

Split the description into three states (none / mount-only filesystem / os),
and correct the README and docstrings that conflated the two.

* docs(code_mode): correct two run_code description claims verified against monty

Audited every statement in the run_code description, docstrings, and README
against pydantic-monty 0.0.17. Two were imprecise:
- "imported at the top of your snippet" -- mid-snippet imports work, so the
  rule is just "before use".
- OS-enabled note said calls route "to the host environment", but an
  in-memory AbstractOS (e.g. OSAccess) handles them too -- it's the
  configured OS handler, not necessarily the host.

* docs(code_mode): clarify overlay-mode write persistence and fix wording

The mount docs implied writes reach the host, but MountDir defaults to
copy-on-write overlay mode, so writes stay in the sandbox unless mode is
'read-write'. Also tighten two awkward/redundant doc lines.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(code_mode): rename public OS/mount surface to be backend-neutral

The public type aliases leaked the Monty backend name into a surface we
can't rename later. Rename them to match the existing CodeMode/CodeModeToolset
convention, and rename the os= parameter to os_access= so it stops shadowing
the stdlib os module that sandboxed code itself uses.

- MontyOS -> CodeModeOS, MontyOSCallback -> CodeModeOSCallback, MontyMount -> CodeModeMount
- CodeMode/CodeModeToolset param os= -> os_access= (mount unchanged)
- internal resume()/feed_start() forwarding keeps Monty's literal os= kwarg

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(code_mode): stop shadowing the os module in the execution loop

The OS/mount threading named its parameter `os`, shadowing the stdlib
module inside the execution-loop helpers. Rename the variable to
`os_access` (matching the public field) while keeping Monty's required
`os=` keyword only at the resume/feed_start call sites. Also inline the
single-use restriction-line helper into `_base_description`.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(code_mode): make CodeMode config fields keyword-only

The option list keeps growing; pin tools/max_retries as the only
positional args and force os_access/mount (and future config) to be
passed by name via a KW_ONLY sentinel, so adding options can't silently
shift positional meaning.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(code_mode): make os_access/mount docs clear on first read

Public docs should let a reader grasp the host-access surface without
reverse-engineering it. Reframe the docstrings and README around when to
reach for each primitive instead of what is switched off, drop the
type-restating prose the annotations already carry, and lead with concrete
tasks (share a dataset; inject just the secrets the agent needs).

Tighten the os-access test sweep so each test asserts exactly its invariant:
drop redundant negative description asserts (one note is interpolated, so the
positive phrase alone proves selection), drop an assertion already owned by
another test, and type the tmp_path fixtures.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(code_mode): clarify os_access callback return semantics

The raw-callback example claimed non-allow-listed keys "stay hidden" by
returning NOT_HANDLED. Verified against Monty: NOT_HANDLED *refuses* the
call (raises in the sandbox -> model retry), it does not return None. A
model probing for an optional secret would crash and burn retries.

Distinguish the two return modes explicitly so users don't pick the
wrong one: return a value (incl. None) to answer/hide, NOT_HANDLED to
refuse a capability outright.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(code_mode): lock in os_access value-vs-NOT_HANDLED semantics

Returning a value (including None) from an os_access callback answers
the call -- a None reads back like an unset env var, so the sandbox
keeps running. Returning NOT_HANDLED refuses the call, raising in the
sandbox and surfacing as ModelRetry. These two paths are easy to
confuse and silently regress, so pin both.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* code_mode: warn default run_code prompt about unavailable fs/env; drop Any from CodeModeOSCallback

The PR's dynamic restriction note covered the mount and os-enabled states but not
the default state, which still listed os/pathlib as importable without saying
their I/O fails -- the exact wasted-retry case this change reduces. Extend the
default note to state filesystem, env, and clock calls are unavailable without a
mount or OS handler.

Also type the public CodeModeOSCallback alias with object instead of Any: the
harness only forwards the callback to Monty and never calls it, so object
typechecks cleanly (pyright strict, 0 errors) and keeps Any out of the public API.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: David SF <david.sanchez@pydantic.dev>
This commit is contained in:
Aditya Vardhan
2026-06-15 11:32:21 -05:00
committed by GitHub
co-authored by Claude David SF
parent 05ae737182
commit c56a4b42f8
5 changed files with 396 additions and 24 deletions
+73 -3
View File
@@ -140,22 +140,92 @@ for msg in result.all_messages():
tool_returns = part.metadata['tool_returns'] # dict[str, ToolReturnPart]
```
## Filesystem and OS access
Sandboxed code runs with no access to the host's files, environment, or clock. Two parameters grant
it -- reach for them when the agent's task genuinely needs the host.
**`mount` -- share host directories.** Reach for this when the agent works with real files: analyzing
a dataset you've dropped in a folder and writing a report back, editing a checkout, or processing a
batch of documents. Sandboxed `pathlib` code reads and writes under the mounted path. (For
environment variables or the clock, use `os_access` instead.)
```python
from pydantic_monty import MountDir
from pydantic_ai_harness import CodeMode
# The agent can read /work/data.csv and write /work/summary.md back to the host:
CodeMode(mount=MountDir('/work', '/tmp/agent-workspace', mode='read-write'))
```
**`os_access` -- answer the sandbox's OS calls yourself.** Reach for this when the agent needs
environment variables, the current date and time, or filesystem behavior you control. Hand it a
ready-made OS implementation, or a callback that decides each call -- so you can inject just the
secrets it needs, pin "now" for reproducible runs, or route file access to your own store.
```python
from pydantic_monty import NOT_HANDLED, OSAccess
from pydantic_ai_harness import CodeMode
# Give the agent a fixed set of environment values:
CodeMode(os_access=OSAccess(environ={'API_BASE': 'https://api.example.com'}))
# ...or intercept each call to decide what the agent may see:
allowed_env = {'API_KEY': 'sk-...'}
def my_os(fn, args, kwargs):
if fn == 'os.getenv':
# Answer the call: allow-listed keys resolve, every other key reads back
# as None -- absent, exactly like a real unset variable.
return allowed_env.get(args[0])
# Refuse everything else: NOT_HANDLED makes the call fail in the sandbox.
return NOT_HANDLED
CodeMode(os_access=my_os)
```
Your callback's return value decides the call's fate, and the two outcomes are easy to confuse:
- **Return any value** -- including `None`, `''`, or `0` -- and that becomes the result the sandbox
sees. `os.getenv` returning `None` looks exactly like a normal unset variable, so the agent's code
keeps running. This is how you *hide* something: answer with an empty value.
- **Return `NOT_HANDLED`** and the call is treated as unsupported: it raises inside the sandbox and
the model gets a retry. This *refuses* a capability outright -- use it to block, not to say "no
value". Returning `NOT_HANDLED` for a key the agent reasonably expects will burn retries.
Both expose the real host to model-written code, so grant only what the task needs. Access is fixed
when the capability is built, so construct `CodeMode` per request to scope it.
A `MountDir` defaults to copy-on-write `mode='overlay'`: the sandbox reads host files and sees its
own writes, but those writes do **not** reach the host. Pass `mode='read-write'` to persist them, or
`mode='read-only'` to forbid writes.
> Monty-specific: these hooks use Monty's `AbstractOS`/`MountDir` types.
## Sandbox restrictions
Code runs inside [Monty](https://github.com/pydantic/monty), a sandboxed Python subset. Key restrictions:
- No class definitions
- No third-party imports (allowed stdlib: `sys`, `typing`, `asyncio`, `math`, `json`, `re`, `datetime`, `os`, `pathlib`)
- No wall-clock or timing primitives: `asyncio.sleep`, `datetime.datetime.now()`/`datetime.date.today()`, and the `time` module are unavailable
- No wall-clock or timing primitives by default (`asyncio.sleep`, `datetime.now()`, `date.today()`, `time`) -- `datetime.now()`/`date.today()` become available with an `os_access` handler (above); `asyncio.sleep`/`time` never do
- No `import *`
- Filesystem I/O needs an `os_access` handler or a `mount`; `os.getenv`/`os.environ` need an `os_access` handler
- Tools requiring approval or with deferred execution are excluded from the sandbox
## API
```python
CodeMode(
tools: ToolSelector = 'all', # 'all', list[str], callable, or dict
max_retries: int = 3, # retries on sandbox execution errors
tools: ToolSelector = 'all', # 'all', list[str], callable, or dict
max_retries: int = 3, # retries on sandbox execution errors
os_access: CodeModeOS | None = None, # host handler for env vars, clock, and file I/O
mount: CodeModeMount | None = None, # host directories to share with the sandbox
)
```
+2 -2
View File
@@ -1,6 +1,6 @@
"""Code mode capability: route tool calls through a sandboxed Python environment."""
from pydantic_ai_harness.code_mode._capability import CodeMode
from pydantic_ai_harness.code_mode._toolset import CodeModeToolset
from pydantic_ai_harness.code_mode._toolset import CodeModeMount, CodeModeOS, CodeModeOSCallback, CodeModeToolset
__all__ = ['CodeMode', 'CodeModeToolset']
__all__ = ['CodeMode', 'CodeModeMount', 'CodeModeOS', 'CodeModeOSCallback', 'CodeModeToolset']
+34 -3
View File
@@ -2,14 +2,14 @@
from __future__ import annotations
from dataclasses import dataclass, field
from dataclasses import KW_ONLY, dataclass, field
from pydantic_ai import AbstractToolset
from pydantic_ai.capabilities import AbstractCapability, CapabilityOrdering
from pydantic_ai.capabilities._tool_search import ToolSearch as _ToolSearch
from pydantic_ai.tools import AgentDepsT, ToolSelector
from pydantic_ai_harness.code_mode._toolset import CodeModeToolset
from pydantic_ai_harness.code_mode._toolset import CodeModeMount, CodeModeOS, CodeModeToolset
@dataclass
@@ -34,6 +34,23 @@ class CodeMode(AbstractCapability[AgentDepsT]):
# Sandbox only specific tools
agent = Agent('openai:gpt-5', capabilities=[CodeMode(tools=['search', 'fetch'])])
```
By default, sandboxed code cannot touch the host -- no filesystem, environment
variables, or clock. Two parameters open it up:
- `mount` shares specific host directories: reach for it when the agent reads or
writes real files.
- `os_access` routes the sandbox's OS calls to a handler you provide: reach for it
when the agent needs environment variables, the clock, or filesystem behavior you
control.
Both expose the real host to model-written code, so grant only what the task needs.
```python
from pydantic_monty import MountDir
agent = Agent('openai:gpt-5', capabilities=[CodeMode(mount=MountDir('/work', '/tmp/agent-work'))])
```
"""
tools: ToolSelector[AgentDepsT] = field(default='all')
@@ -48,10 +65,24 @@ class CodeMode(AbstractCapability[AgentDepsT]):
max_retries: int = 3
"""Maximum number of retries for the `run_code` tool (syntax errors count as retries)."""
_: KW_ONLY
os_access: CodeModeOS | None = None
"""Give sandboxed code environment variables, the clock, and file I/O through a handler you provide; unset, they are unavailable."""
mount: CodeModeMount | None = None
"""Host directories to expose to sandboxed `pathlib` code; each mount's `mode` controls whether writes reach the host."""
def get_ordering(self) -> CapabilityOrdering:
"""CodeMode wraps around ToolSearch so that search_tools stays native."""
return CapabilityOrdering(position='outermost', wraps=[_ToolSearch])
def get_wrapper_toolset(self, toolset: AbstractToolset[AgentDepsT]) -> AbstractToolset[AgentDepsT] | None:
"""Wrap the agent's assembled toolset, splitting it into native + sandboxed subsets if needed."""
return CodeModeToolset(wrapped=toolset, tool_selector=self.tools, max_retries=self.max_retries)
return CodeModeToolset(
wrapped=toolset,
tool_selector=self.tools,
max_retries=self.max_retries,
os_access=self.os_access,
mount=self.mount,
)
+93 -16
View File
@@ -26,6 +26,7 @@ except ImportError: # pragma: no cover
try:
from pydantic_monty import (
AbstractOS,
ExternalException,
ExternalResult,
ExternalReturnValue,
@@ -37,7 +38,9 @@ try:
MontyRuntimeError,
MontySyntaxError,
MontyTypingError,
MountDir,
NameLookupSnapshot,
OsFunction,
)
except ImportError as _import_error: # pragma: no cover
raise ImportError(
@@ -48,6 +51,14 @@ from typing_extensions import NotRequired, TypedDict
# Type alias for the dispatch callback passed to _execution_loop.
_DispatchFn = Callable[[str, dict[str, Any]], Coroutine[Any, Any, Any]]
# A raw OS callback. Return `pydantic_monty.NOT_HANDLED` to defer the call to the
# sandbox's default, which leaves it unavailable.
CodeModeOSCallback = Callable[[OsFunction, tuple[object, ...], dict[str, object]], object]
# Accepted by `CodeMode.os_access`: a ready-made OS implementation or a raw callback.
CodeModeOS = AbstractOS | CodeModeOSCallback
# Accepted by `CodeMode.mount`: one or more host-directory mounts.
CodeModeMount = MountDir | list[MountDir]
class _RunCodeArguments(TypedDict):
code: Annotated[str, Field(description='The Python code to execute in the sandbox.')]
@@ -69,14 +80,38 @@ _RUN_CODE_ARGS_VALIDATOR: SchemaValidatorProt = _RUN_CODE_ADAPTER.validator # p
# and to reconstruct multimodal types (e.g. BinaryContent) from Monty results (validate_python).
_TOOL_RETURN_CONTENT_TA: TypeAdapter[Any] = TypeAdapter(ToolReturnContent)
_RUN_CODE_BASE_DESCRIPTION = """\
_RUN_CODE_DESCRIPTION_HEAD = """\
Write and run Python code in a sandboxed environment.
The sandbox uses Monty, a subset of Python. Key restrictions:
- **No classes**: class definitions are not supported
- **No third-party libraries**: only the standard library modules listed below can be used
- **Importable standard library modules**: `sys`, `typing`, `asyncio`, `math`, `json`, `re`, `datetime`, `os`, `pathlib`. These must be imported at the top of your snippet before use, just like in regular Python. For example: `import asyncio` then `results = await asyncio.gather(tool_one(...), tool_two(...))`.
- **No wall-clock or timing primitives**: `asyncio.sleep`, `datetime.datetime.now()`, `datetime.date.today()`, and the `time` module are unavailable.
- **Importable standard library modules**: `sys`, `typing`, `asyncio`, `math`, `json`, `re`, `datetime`, `os`, `pathlib`. These must be imported before use, just like in regular Python. For example: `import asyncio` then `results = await asyncio.gather(tool_one(...), tool_two(...))`."""
# Timing/OS restriction line, swapped depending on what host access the agent
# configured. Three states, because `mount` and `os` enable different things:
# a `mount` only exposes filesystem paths, while environment and clock calls
# require an `os` handler.
_NO_OS_RESTRICTION = (
'- **No filesystem, environment, or timing primitives**: `pathlib.Path` I/O, '
'`os.getenv`/`os.environ`, `datetime.datetime.now()`, `datetime.date.today()`, `asyncio.sleep`, '
'and the `time` module are unavailable here (no filesystem mount or OS handler is configured). '
'`os` and `pathlib` import successfully, but their I/O operations are not supported in this '
'configuration.'
)
_MOUNT_ONLY_NOTE = (
'- **Mounted filesystem access**: `pathlib.Path` operations under the configured mount '
'point(s) are routed to the host. `os.getenv`/`os.environ`, `datetime.datetime.now()`, '
'`datetime.date.today()`, `asyncio.sleep`, and the `time` module remain unavailable.'
)
_OS_ENABLED_NOTE = (
'- **Host-backed OS access**: `pathlib.Path` operations, `os.getenv`/`os.environ`, '
'`datetime.datetime.now()`, and `datetime.date.today()` are routed to the OS handler '
'configured for this agent (availability depends on that configuration). `asyncio.sleep` and '
'the `time` module remain unavailable.'
)
_RUN_CODE_DESCRIPTION_TAIL = """\
- **No `import *`**: wildcard imports are not supported
State is preserved between calls (REPL-style). Set `restart: true` to reset state.
@@ -90,6 +125,22 @@ Returns the last expression's value directly. If `print()` was also called, retu
"""
def _base_description(*, has_os: bool, has_mount: bool) -> str:
"""Assemble the `run_code` base description with the right OS-access restriction line.
`os` routes environment, clock, and filesystem calls; a `mount` alone only
exposes filesystem paths, so a mount-only sandbox must not advertise env or
clock access (the model would generate calls that fail and burn retries).
"""
if has_os:
restriction = _OS_ENABLED_NOTE
elif has_mount:
restriction = _MOUNT_ONLY_NOTE
else:
restriction = _NO_OS_RESTRICTION
return f'{_RUN_CODE_DESCRIPTION_HEAD}\n{restriction}\n{_RUN_CODE_DESCRIPTION_TAIL}'
def _functions_header(*, has_sync: bool, has_async: bool) -> str:
"""Build the functions-header paragraph for the `run_code` tool description."""
base = (
@@ -184,6 +235,12 @@ class CodeModeToolset(WrapperToolset[AgentDepsT]):
max_retries: int = 3
"""Maximum number of retries for the `run_code` tool (syntax errors count as retries)."""
os_access: CodeModeOS | None = None
"""Give sandboxed code environment variables, the clock, and file I/O through a handler you provide; unset, they are unavailable."""
mount: CodeModeMount | None = None
"""Host directories to expose to sandboxed `pathlib` code; each mount's `mode` controls whether writes reach the host."""
# init=False so `replace()` in `for_run` produces a fresh instance with _repl=None,
# giving each agent run isolated REPL state. Lazy-initialized on first call_tool.
_repl: MontyRepl | None = field(default=None, init=False, repr=False)
@@ -236,7 +293,9 @@ class CodeModeToolset(WrapperToolset[AgentDepsT]):
callable_defs, sanitized_to_original = self._partition_callable_tools(sandboxed_tools)
description = self._build_description(callable_defs)
description = self._build_description(
callable_defs, has_os=self.os_access is not None, has_mount=self.mount is not None
)
if _RUN_CODE_TOOL_NAME in native_tools:
raise UserError(
@@ -399,7 +458,7 @@ class CodeModeToolset(WrapperToolset[AgentDepsT]):
capture = _PrintCapture()
try:
monty_state = self._repl.feed_start(code, print_callback=capture)
monty_state = self._repl.feed_start(code, print_callback=capture, os=self.os_access, mount=self.mount)
completed = await _execution_loop(
monty_state,
dispatch=dispatch_tool_call,
@@ -407,6 +466,8 @@ class CodeModeToolset(WrapperToolset[AgentDepsT]):
sanitized_to_original=sanitized_to_original,
sequential_tools=sequential_tools,
global_sequential=global_sequential,
os_access=self.os_access,
mount=self.mount,
)
except MontySyntaxError as e:
raise ModelRetry(f'Syntax error in code:\n{_prepend_prints(e.display(), capture)}') from e
@@ -504,10 +565,11 @@ class CodeModeToolset(WrapperToolset[AgentDepsT]):
return callable_defs, sanitized_to_original
@staticmethod
def _build_description(callable_defs: dict[str, ToolDefinition]) -> str:
def _build_description(callable_defs: dict[str, ToolDefinition], *, has_os: bool, has_mount: bool) -> str:
"""Render the `run_code` description: base prose + TypedDicts + function signatures."""
base = _base_description(has_os=has_os, has_mount=has_mount)
if not callable_defs:
return _RUN_CODE_BASE_DESCRIPTION
return base
sigs, conflicting = _get_sigs_and_conflicting(callable_defs)
type_blocks = FunctionSignature.render_type_definitions(sigs, conflicting)
@@ -520,7 +582,7 @@ class CodeModeToolset(WrapperToolset[AgentDepsT]):
has_async = any(not td.sequential for td in callable_defs.values())
header = _functions_header(has_sync=has_sync, has_async=has_async)
sections = [_RUN_CODE_BASE_DESCRIPTION, header]
sections = [base, header]
if type_blocks:
sections.append('```python\n' + '\n\n'.join(type_blocks) + '\n```')
sections.append('```python\n' + '\n\n'.join(function_blocks) + '\n```')
@@ -579,6 +641,8 @@ async def _execution_loop(
sanitized_to_original: dict[str, str],
sequential_tools: set[str],
global_sequential: bool,
os_access: CodeModeOS | None,
mount: CodeModeMount | None,
) -> MontyComplete:
"""Drive the Monty REPL via the synchronous snapshot API until completion.
@@ -597,6 +661,9 @@ async def _execution_loop(
- **Global sequential mode** (DBOS/Temporal): all tools are deferred via
`resume({'future': ...})` but stored as bare coroutines and awaited
one-at-a-time at `FutureSnapshot` to prevent interleaving.
`os`/`mount` must be passed to every `resume` call (not just `feed_start`):
Monty's auto-dispatch of OS calls stops the moment a resume omits them.
"""
pending: dict[int, asyncio.Task[Any] | Coroutine[Any, Any, Any]] = {}
# Results from parallel tasks that were awaited early (at a sequential-tool
@@ -605,7 +672,7 @@ async def _execution_loop(
try:
while not isinstance(monty_state, MontyComplete):
if isinstance(monty_state, NameLookupSnapshot):
monty_state = monty_state.resume()
monty_state = monty_state.resume(os=os_access, mount=mount)
elif isinstance(monty_state, FunctionSnapshot):
monty_state = await _handle_function_snapshot(
monty_state,
@@ -616,6 +683,8 @@ async def _execution_loop(
global_sequential=global_sequential,
pending=pending,
pre_resolved=pre_resolved,
os_access=os_access,
mount=mount,
)
else:
monty_state = await _resolve_future_snapshot(
@@ -623,6 +692,8 @@ async def _execution_loop(
pending=pending,
pre_resolved=pre_resolved,
global_sequential=global_sequential,
os_access=os_access,
mount=mount,
)
finally:
for item in pending.values(): # pragma: no cover
@@ -644,16 +715,20 @@ async def _handle_function_snapshot(
global_sequential: bool,
pending: dict[int, asyncio.Task[Any] | Coroutine[Any, Any, Any]],
pre_resolved: dict[int, ExternalResult],
os_access: CodeModeOS | None,
mount: CodeModeMount | None,
) -> FunctionSnapshot | FutureSnapshot | NameLookupSnapshot | MontyComplete:
"""Handle a single FunctionSnapshot from the Monty execution loop."""
fn_name = snapshot.function_name
if fn_name not in callable_defs:
return snapshot.resume({'exception': NameError(f'Unknown function: {fn_name}')})
return snapshot.resume({'exception': NameError(f'Unknown function: {fn_name}')}, os=os_access, mount=mount)
if snapshot.args:
return snapshot.resume(
{'exception': TypeError(f'{fn_name}() does not accept positional arguments; use keyword arguments')}
{'exception': TypeError(f'{fn_name}() does not accept positional arguments; use keyword arguments')},
os=os_access,
mount=mount,
)
original_name = sanitized_to_original.get(fn_name, fn_name)
@@ -666,8 +741,8 @@ async def _handle_function_snapshot(
pre_resolved[cid] = await _resolve_coro(pending.pop(cid))
outcome = await _resolve_coro(dispatch(original_name, snapshot.kwargs))
if 'return_value' in outcome:
return snapshot.resume({'return_value': outcome['return_value']})
return snapshot.resume({'exception': outcome['exception']})
return snapshot.resume({'return_value': outcome['return_value']}, os=os_access, mount=mount)
return snapshot.resume({'exception': outcome['exception']}, os=os_access, mount=mount)
# Deferred execution -- store for later resolution at FutureSnapshot.
if global_sequential:
@@ -676,7 +751,7 @@ async def _handle_function_snapshot(
else:
# Eagerly schedule as a Task for concurrent execution.
pending[snapshot.call_id] = asyncio.ensure_future(dispatch(original_name, snapshot.kwargs))
return snapshot.resume({'future': ...})
return snapshot.resume({'future': ...}, os=os_access, mount=mount)
async def _resolve_future_snapshot(
@@ -685,11 +760,13 @@ async def _resolve_future_snapshot(
pending: dict[int, asyncio.Task[Any] | Coroutine[Any, Any, Any]],
pre_resolved: dict[int, ExternalResult],
global_sequential: bool,
os_access: CodeModeOS | None,
mount: CodeModeMount | None,
) -> FunctionSnapshot | FutureSnapshot | NameLookupSnapshot | MontyComplete:
"""Resolve pending tool calls at a FutureSnapshot."""
pending_ids = snapshot.pending_call_ids
if not pending_ids: # pragma: no cover
return snapshot.resume(results={})
return snapshot.resume(results={}, os=os_access, mount=mount)
results: dict[int, ExternalResult] = {}
for cid in pending_ids:
@@ -708,7 +785,7 @@ async def _resolve_future_snapshot(
for cid, outcome in zip(gather_ids, settled):
results[cid] = _settle_outcome(outcome)
return snapshot.resume(results=results)
return snapshot.resume(results=results, os=os_access, mount=mount)
async def _resolve_coro(
+194
View File
@@ -8,6 +8,7 @@ loaded by the project (no extra dev dependency needed).
from __future__ import annotations
from pathlib import Path
from typing import Any, TypeVar
import pytest
@@ -24,6 +25,7 @@ from pydantic_ai.toolsets.abstract import ToolsetTool
from pydantic_ai.toolsets.function import FunctionToolset
from pydantic_ai.usage import RunUsage
from pydantic_core import SchemaValidator, core_schema
from pydantic_monty import NOT_HANDLED, MountDir, OSAccess, OsFunction
from typing_extensions import TypedDict
from pydantic_ai_harness import CodeMode
@@ -1856,6 +1858,198 @@ class TestToolSearchIntegration:
assert ToolSearch in ordering.wraps
def _unused_os_callback(fn: OsFunction, args: tuple[Any, ...], kwargs: dict[str, Any]) -> Any:
"""An `os` callback for tests that only assert description/forwarding, never run code."""
return NOT_HANDLED # pragma: no cover - never invoked by these tests
class TestCodeModeOSAccess:
"""`CodeMode(os_access=...)` / `mount=...` give sandboxed code host-backed OS access."""
async def test_description_default_notes_no_fs_env_or_clock(self) -> None:
"""Without `os`/`mount`, the description states filesystem, env, and clock calls are
unavailable, so the model does not waste retries calling `pathlib`/`os` I/O."""
wrapper = CodeMode[None]().get_wrapper_toolset(_build_function_toolset(add))
assert isinstance(wrapper, CodeModeToolset)
description = (await wrapper.get_tools(build_run_context(None)))['run_code'].tool_def.description
assert description is not None
assert 'No filesystem, environment, or timing primitives' in description
assert 'their I/O operations are not supported in this configuration' in description
async def test_description_with_os_callback_notes_host_access(self) -> None:
"""An `os` callback swaps the restriction line for the host-access note."""
wrapper = CodeMode[None](os_access=_unused_os_callback).get_wrapper_toolset(_build_function_toolset(add))
assert isinstance(wrapper, CodeModeToolset)
description = (await wrapper.get_tools(build_run_context(None)))['run_code'].tool_def.description
assert description is not None
assert 'Host-backed OS access' in description
async def test_description_mount_only_advertises_filesystem_not_env_or_clock(self, tmp_path: Path) -> None:
"""A `mount` without `os` advertises filesystem access only -- it must not tell the model
that env/clock are host-backed, since a mount cannot route `os.getenv`/`datetime.now()`."""
wrapper = CodeMode[None](mount=MountDir('/work', str(tmp_path))).get_wrapper_toolset(
_build_function_toolset(add)
)
assert isinstance(wrapper, CodeModeToolset)
description = (await wrapper.get_tools(build_run_context(None)))['run_code'].tool_def.description
assert description is not None
# The regression guard: a mount must select the filesystem note, not the OS note that would
# (wrongly) advertise env/clock as host-routed -- this assert fails if the OS note is picked.
assert 'Mounted filesystem access' in description
async def test_description_host_access_note_shows_with_no_sandboxed_tools(self) -> None:
"""The host-access note appears even when no tools are sandboxed (base description)."""
# `tools=[]` sandboxes nothing, so `run_code` renders the base description path.
wrapper = CodeMode[None](os_access=_unused_os_callback, tools=[]).get_wrapper_toolset(
_build_function_toolset(add)
)
assert isinstance(wrapper, CodeModeToolset)
description = (await wrapper.get_tools(build_run_context(None)))['run_code'].tool_def.description
assert description is not None
assert 'Host-backed OS access' in description
async def test_os_callback_dispatches_inside_run_code(self) -> None:
"""An `os` callback is threaded through `feed_start` and every `resume`, so OS calls
keep dispatching even after a tool call suspends and resumes the sandbox."""
def os_cb(fn: OsFunction, args: tuple[Any, ...], kwargs: dict[str, Any]) -> Any:
if fn == 'os.getenv':
return 'envval'
return NOT_HANDLED # pragma: no cover - sandbox only calls os.getenv here
wrapper = CodeMode[None](os_access=os_cb).get_wrapper_toolset(_build_function_toolset(add))
assert isinstance(wrapper, CodeModeToolset)
ctx = await build_ctx(None, wrapper)
tools = await wrapper.get_tools(ctx)
# The tool call forces a FunctionSnapshot -> FutureSnapshot round-trip; the os.getenv
# afterwards only resolves if `os` survived those resumes.
code = "import os\nx = await add(a=2, b=3)\nhome = os.getenv('THING')\n{'sum': x, 'home': home}"
result = await wrapper.call_tool('run_code', {'code': code}, ctx, tools['run_code'])
assert result.return_value == {'sum': 5, 'home': 'envval'}
async def test_os_access_persists_across_run_code_calls(self) -> None:
"""`os` is supplied on every `feed_start`, so OS access still works on a later
`run_code` call that reuses the persisted (non-fresh) REPL."""
def os_cb(fn: OsFunction, args: tuple[Any, ...], kwargs: dict[str, Any]) -> Any:
if fn == 'os.getenv':
return 'persisted'
return NOT_HANDLED # pragma: no cover - sandbox only calls os.getenv here
wrapper = CodeMode[None](os_access=os_cb).get_wrapper_toolset(_build_function_toolset(add))
assert isinstance(wrapper, CodeModeToolset)
ctx = await build_ctx(None, wrapper)
tools = await wrapper.get_tools(ctx)
first = await wrapper.call_tool('run_code', {'code': "import os\nos.getenv('A')"}, ctx, tools['run_code'])
assert first.return_value == 'persisted'
# Second call reuses the REPL (so `import os` carries over) and must still dispatch.
second = await wrapper.call_tool('run_code', {'code': "os.getenv('B')"}, ctx, tools['run_code'])
assert second.return_value == 'persisted'
async def test_abstract_os_instance_dispatches_inside_run_code(self) -> None:
"""An `AbstractOS` instance is accepted as the `os` value and dispatches OS calls."""
wrapper = CodeMode[None](os_access=OSAccess(environ={'THING': 'fromabs'})).get_wrapper_toolset(
_build_function_toolset(add)
)
assert isinstance(wrapper, CodeModeToolset)
ctx = await build_ctx(None, wrapper)
tools = await wrapper.get_tools(ctx)
result = await wrapper.call_tool('run_code', {'code': "import os\nos.getenv('THING')"}, ctx, tools['run_code'])
assert result.return_value == 'fromabs'
async def test_os_callback_exception_becomes_model_retry(self) -> None:
"""A raising `os` callback surfaces as a `ModelRetry`, like any other sandbox runtime
error -- it must not crash the agent loop."""
def os_cb(fn: OsFunction, args: tuple[Any, ...], kwargs: dict[str, Any]) -> Any:
raise ValueError('boom from os')
wrapper = CodeMode[None](os_access=os_cb).get_wrapper_toolset(_build_function_toolset(add))
assert isinstance(wrapper, CodeModeToolset)
ctx = await build_ctx(None, wrapper)
tools = await wrapper.get_tools(ctx)
with pytest.raises(ModelRetry, match='boom from os'):
await wrapper.call_tool('run_code', {'code': "import os\nos.getenv('X')"}, ctx, tools['run_code'])
async def test_os_callback_returning_value_answers_call_including_none(self) -> None:
"""Returning a value from the `os` callback -- even `None` -- *answers* the call.
Allow-listed keys resolve; every other key reads back as `None`, exactly like a real
unset env var, so the sandbox keeps running with no retry. This is how a callback hides
a secret: by answering with an empty value, not by refusing the call.
"""
allowed = {'API_KEY': 'sk-xxx'}
def os_cb(fn: OsFunction, args: tuple[Any, ...], kwargs: dict[str, Any]) -> Any:
if fn == 'os.getenv':
return allowed.get(args[0])
return NOT_HANDLED # pragma: no cover - sandbox only calls os.getenv here
wrapper = CodeMode[None](os_access=os_cb).get_wrapper_toolset(_build_function_toolset(add))
assert isinstance(wrapper, CodeModeToolset)
ctx = await build_ctx(None, wrapper)
tools = await wrapper.get_tools(ctx)
code = "import os\n{'allowed': os.getenv('API_KEY'), 'hidden': os.getenv('SECRET')}"
result = await wrapper.call_tool('run_code', {'code': code}, ctx, tools['run_code'])
assert result.return_value == {'allowed': 'sk-xxx', 'hidden': None}
async def test_os_callback_not_handled_refuses_call_as_model_retry(self) -> None:
"""Returning `NOT_HANDLED` *refuses* the call rather than answering it.
The OS function is treated as unsupported, so it raises in the sandbox and surfaces as
`ModelRetry`. This is the counterpart to returning a value: refusing is not the same as
answering `None`, and using it for a key the model expects will burn retries.
"""
def os_cb(fn: OsFunction, args: tuple[Any, ...], kwargs: dict[str, Any]) -> Any:
return NOT_HANDLED
wrapper = CodeMode[None](os_access=os_cb).get_wrapper_toolset(_build_function_toolset(add))
assert isinstance(wrapper, CodeModeToolset)
ctx = await build_ctx(None, wrapper)
tools = await wrapper.get_tools(ctx)
with pytest.raises(ModelRetry, match='not supported in this environment'):
await wrapper.call_tool('run_code', {'code': "import os\nos.getenv('X')"}, ctx, tools['run_code'])
async def test_mount_exposes_host_directory(self, tmp_path: Path) -> None:
"""A `mount` exposes a host directory inside the sandbox, threaded through resumes."""
(tmp_path / 'data.txt').write_text('hello-from-host')
wrapper = CodeMode[None](mount=MountDir('/work', str(tmp_path))).get_wrapper_toolset(
_build_function_toolset(add)
)
assert isinstance(wrapper, CodeModeToolset)
ctx = await build_ctx(None, wrapper)
tools = await wrapper.get_tools(ctx)
code = "from pathlib import Path\nawait add(a=1, b=1)\nPath('/work/data.txt').read_text()"
result = await wrapper.call_tool('run_code', {'code': code}, ctx, tools['run_code'])
assert result.return_value == 'hello-from-host'
async def test_mount_accepts_list_of_directories(self, tmp_path: Path) -> None:
"""`mount` accepts a `list[MountDir]`; each directory is exposed at its virtual path."""
(tmp_path / 'a').mkdir()
(tmp_path / 'b').mkdir()
(tmp_path / 'a' / 'f.txt').write_text('AA')
(tmp_path / 'b' / 'f.txt').write_text('BB')
mounts = [MountDir('/a', str(tmp_path / 'a')), MountDir('/b', str(tmp_path / 'b'))]
wrapper = CodeMode[None](mount=mounts).get_wrapper_toolset(_build_function_toolset(add))
assert isinstance(wrapper, CodeModeToolset)
ctx = await build_ctx(None, wrapper)
tools = await wrapper.get_tools(ctx)
code = "from pathlib import Path\nPath('/a/f.txt').read_text() + Path('/b/f.txt').read_text()"
result = await wrapper.call_tool('run_code', {'code': code}, ctx, tools['run_code'])
assert result.return_value == 'AABB'
def test_capability_forwards_os_and_mount_to_toolset(self, tmp_path: Path) -> None:
"""`CodeMode` forwards `os_access`/`mount` onto the `CodeModeToolset` it builds."""
mount = MountDir('/work', str(tmp_path))
wrapper = CodeMode[None](os_access=_unused_os_callback, mount=mount).get_wrapper_toolset(
_build_function_toolset(add)
)
assert isinstance(wrapper, CodeModeToolset)
assert wrapper.os_access is _unused_os_callback
assert wrapper.mount is mount
def _search_tool_def(description: str = 'Search for tools.') -> ToolDefinition:
"""Create a ToolDefinition mimicking the search_tools tool from ToolSearchToolset."""
from pydantic_ai.toolsets._tool_search import _SEARCH_TOOLS_NAME