mirror of
https://github.com/pydantic/pydantic-ai-harness.git
synced 2026-07-21 02:45:34 +00:00
Remove third-party MCP endpoint from pydantic-ai-harness skill (#368)
* Use local tools in harness skill quick start * Align CodeMode skill security guidance
This commit is contained in:
@@ -83,36 +83,35 @@ Requires Python 3.10+ and `pydantic-ai-slim>=2.1.0`.
|
||||
|
||||
## Quick Start
|
||||
|
||||
A harness capability is added to the agent like any other. Here `CodeMode` wraps an MCP server's tools into
|
||||
a single `run_code` tool that the model drives with Python.
|
||||
A harness capability is added to the agent like any other. Here `CodeMode` wraps locally registered tools
|
||||
into a single `run_code` tool that the model drives with Python.
|
||||
|
||||
```python {test="skip"}
|
||||
from pydantic_ai import Agent
|
||||
from pydantic_ai.capabilities import MCP # MCP ships in core pydantic-ai
|
||||
|
||||
from pydantic_ai_harness import CodeMode
|
||||
|
||||
agent = Agent(
|
||||
'anthropic:claude-sonnet-4-6',
|
||||
capabilities=[
|
||||
# native=False routes the MCP tools through a local toolset so CodeMode can wrap them.
|
||||
# Without it, providers with native MCP run the tools server-side and bypass the sandbox.
|
||||
MCP('https://hn.caseyjhand.com/mcp', native=False),
|
||||
CodeMode(),
|
||||
],
|
||||
)
|
||||
agent = Agent('anthropic:claude-sonnet-4-6', capabilities=[CodeMode()])
|
||||
|
||||
|
||||
@agent.tool_plain
|
||||
def get_temperature_f(city: str) -> float:
|
||||
return {'Paris': 68.0, 'Tokyo': 77.0}[city]
|
||||
|
||||
|
||||
@agent.tool_plain
|
||||
def convert_temp(fahrenheit: float) -> float:
|
||||
return round((fahrenheit - 32) * 5 / 9, 1)
|
||||
|
||||
result = agent.run_sync(
|
||||
'Across the top and best Hacker News feeds, find the most-discussed story with at '
|
||||
'least 100 points and summarize its comment thread in one paragraph.'
|
||||
'Compare the weather in Paris and Tokyo, and report both temperatures in Celsius.'
|
||||
)
|
||||
print(result.output)
|
||||
#> The most-discussed story clearing 100 points is ...
|
||||
#> Paris is 20.0 C and Tokyo is 25.0 C.
|
||||
```
|
||||
|
||||
Instead of one model round-trip per tool call, the model writes a single Python script that fetches both
|
||||
feeds with `asyncio.gather`, dedupes and ranks them in plain Python, and pulls the winning thread --
|
||||
collapsing many calls into one `run_code`.
|
||||
The model writes a single Python script that fetches both temperatures with `asyncio.gather` and then
|
||||
converts them -- performing four tool calls across two dependent stages in one `run_code` invocation.
|
||||
|
||||
## Key Practices
|
||||
|
||||
@@ -122,6 +121,6 @@ collapsing many calls into one `run_code`.
|
||||
|
||||
## Common Gotchas
|
||||
|
||||
- **`native=True` tools bypass `CodeMode`.** Provider-native MCP servers and web search execute server-side, so `run_code` never sees them. Construct them with `native=False` to keep them local and wrappable.
|
||||
- **`native=True` tools bypass `CodeMode`.** Provider-native MCP servers and web search execute server-side, so `run_code` never sees them. Use `native=False` for client-side dispatch that `CodeMode` can wrap, but do not treat a remote server as trusted or sandboxed; see the [Code Mode trust boundary](./references/CODE-MODE.md#sandbox-restrictions).
|
||||
- **The Monty sandbox is a Python subset.** No class definitions, no third-party imports, and only a small stdlib allowlist -- read [Code Mode](./references/CODE-MODE.md#sandbox-restrictions) before debugging generated code that fails to run.
|
||||
- **`CodeMode` needs its extra.** Install `pydantic-ai-harness[codemode]`, not the bare package.
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Code Mode
|
||||
|
||||
Standard tool calling takes one model round-trip per tool call. `CodeMode` wraps eligible tools into a
|
||||
single `run_code` tool so the model can write Python that loops, branches, aggregates, and parallelizes tool work inside a sandbox.
|
||||
`CodeMode` wraps eligible tools into a single `run_code` tool so the model can write Python that loops,
|
||||
branches, aggregates, and parallelizes multiple tool calls inside a sandbox.
|
||||
|
||||
Use it when the agent needs to call several tools, transform intermediate results, run concurrent
|
||||
tool work with `asyncio.gather`, or anywhere a simple Python script is more reliable than the model alone, such as for mathematics.
|
||||
@@ -37,6 +37,8 @@ def convert_temp(fahrenheit: float) -> float:
|
||||
The model could generate code like:
|
||||
|
||||
```python {test="skip" lint="skip"}
|
||||
import asyncio
|
||||
|
||||
paris, tokyo = await asyncio.gather(
|
||||
get_weather(city='Paris'),
|
||||
get_weather(city='Tokyo'),
|
||||
@@ -46,7 +48,7 @@ tokyo_c = await convert_temp(fahrenheit=tokyo['temp_f'])
|
||||
{'paris': paris_c, 'tokyo': tokyo_c}
|
||||
```
|
||||
|
||||
This reduces four model round-trips to one.
|
||||
This performs four tool calls, across two dependent stages, inside one `run_code` invocation.
|
||||
|
||||
## Choose Which Tools Are Sandboxed
|
||||
|
||||
@@ -57,7 +59,7 @@ from pydantic_ai_harness import CodeMode
|
||||
|
||||
CodeMode(tools='all')
|
||||
CodeMode(tools=['search', 'fetch'])
|
||||
CodeMode(tools=lambda ctx, td: td.name != 'dangerous_tool')
|
||||
CodeMode(tools=lambda ctx, td: td.name in {'search', 'fetch'})
|
||||
CodeMode(tools={'code_mode': True})
|
||||
```
|
||||
|
||||
@@ -121,8 +123,17 @@ Key restrictions:
|
||||
- No third-party imports
|
||||
- No `import *`
|
||||
- Only a small stdlib subset is allowed: `sys`, `typing`, `asyncio`, `math`, `json`, `re`, `datetime`, `os`, `pathlib`
|
||||
- No wall-clock or timing primitives: `asyncio.sleep`, `datetime.datetime.now()`, `datetime.date.today()`, and the `time` module are unavailable
|
||||
- Tools that need approval or deferred execution are excluded from the sandbox
|
||||
- No wall-clock or timing primitives by default: `datetime.datetime.now()` and `datetime.date.today()` require an `os_access` handler; `asyncio.sleep` and the `time` module are unavailable
|
||||
- Filesystem I/O requires an `os_access` handler or a `mount`; `os.getenv` and `os.environ` require an `os_access` handler
|
||||
- Tools requiring approval or with deferred (`CallDeferred`) execution are sandboxed like any other tool; without a `HandleDeferredToolCalls` (or equivalent) capability to resolve them inline, calling one from `run_code` raises an error that surfaces to the model as a retry
|
||||
|
||||
Leave `os_access` and `mount` unset unless the task requires host access, and grant only the resources
|
||||
the task needs. A mount defaults to copy-on-write `mode='overlay'`; use `mode='read-only'` to forbid
|
||||
writes or `mode='read-write'` only when sandbox writes must persist on the host.
|
||||
|
||||
The sandbox constrains the model-generated Python, not the implementation of the tools it calls.
|
||||
Wrapped tools retain their normal host and network access, so expose only tools with the authority and
|
||||
input validation appropriate for model-generated calls.
|
||||
|
||||
When a generated example keeps failing, check these restrictions before changing the rest of the agent.
|
||||
|
||||
@@ -130,8 +141,12 @@ When a generated example keeps failing, check these restrictions before changing
|
||||
|
||||
```python {test="skip" lint="skip"}
|
||||
CodeMode(
|
||||
tools: ToolSelector = 'all', # 'all', list[str], callable, or dict
|
||||
max_retries: int = 3, # retries on sandbox execution errors
|
||||
tools: ToolSelector = 'all', # 'all', list[str], callable, or dict
|
||||
max_retries: int = 3, # retries on sandbox execution errors
|
||||
*,
|
||||
os_access: CodeModeOS | None = None, # host handler for env vars, clock, and file I/O
|
||||
mount: CodeModeMount | None = None, # host directories to share with the sandbox
|
||||
dynamic_catalog: bool = False, # move the tool catalog into dynamic instructions
|
||||
)
|
||||
```
|
||||
|
||||
|
||||
Reference in New Issue
Block a user