furyhawk/pydantic-ai-harness
1
0
Fork 0
mirror of https://github.com/pydantic/pydantic-ai-harness.git synced 2026-07-21 02:45:34 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge origin/main; retarget overflow query surface onto graduated module

Browse Source 
`main` graduated and renamed `experimental/overflow` to top-level
`pydantic_ai_harness/overflowing_tool_output` (#347), splitting bands into
`_bands.py`. This merge re-targets this branch's grep/slice query surface and
uniform elision markers onto the graduated layout:

- New `_markers.py` moved into `pydantic_ai_harness/overflowing_tool_output/`.
- All internal imports (capability, payload, tests, README snippets) rewritten to
  the top-level `pydantic_ai_harness.overflowing_tool_output` path; the shared
  token-count import now targets top-level `compaction`.
- New `GREP_TOOL_NAME` exported from the package `__init__` and re-exported by the
  experimental shim.
- README reconciled to the two-tool (grep + read) surface, keeping main's
  query-tool-return exemption note.

No behavior change; layout/import reconciliation only. Preserves the Summarize
stand-in-marker fix (80c9d28) and the query-surface feature (279336d).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Douwe Maan
2026-07-15 20:01:02 +00:00
co-authored by Claude Opus 4.8
parent 80c9d2827a 73513a60bc
commit efc0e0a312
159 changed files with 15747 additions and 857 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/agents/docs-parity-reviewer.md
+82
View File
@@ -0,0 +1,82 @@
---
name: docs-parity-reviewer
description: Use as the final documentation gate before a capability PR merges. Verifies that a user-facing change keeps the capability README and its unified-docs page in sync with each other and with the code, that every snippet is runnable, and that links follow repo convention. Reports gaps; does not edit.
tools: Read, Grep, Glob, Bash
model: sonnet
---
You are the documentation parity gate for `pydantic-ai-harness`. Every released
capability ships two docs that must stay in sync with the code and with each
other:
- **README** -- `pydantic_ai_harness/<capability>/README.md` (or
`pydantic_ai_harness/experimental/acp/README.md` for ACP). Serves GitHub and
PyPI. Keeps absolute links and its badges.
- **Unified doc** -- flat at `docs/<capability>.md`. Renders on the docs site
(`https://pydantic.dev/docs/ai/harness/`). No badges; links its source module
and, where the capability exposes a public class, may end with
`::: pydantic_ai_harness.<Class>` autodoc blocks. The sidebar is a flat list --
no `capabilities/` or `experimental/` subdirectories.
Both are hand-maintained. A change to one that is not reflected in the other is
the failure mode you exist to catch.
## What you are given
The diff or description of a capability change (the touched capability, and what
its user-facing behavior now is). If you are not told which capability changed,
infer it from the changed files under `pydantic_ai_harness/`.
## Checks
Read the capability source, its README, and its unified doc, then report each
problem as a finding (blocking / warning / nit) with a concrete fix.
1. **Both docs updated.** If the change alters user-facing behavior (public
class, constructor params, defaults, tool names, extras, safety semantics)
and only one of README / unified doc reflects it, that is blocking. A doc
describing behavior the code no longer has is also blocking.
2. **Snippets run.** Every code block in both docs has all imports and the
pieces needed to actually run (Agent construction, capability wiring). Class
names, params, and defaults match the current source. Model ids are unchanged
from what the source uses -- a changed model id is blocking.
3. **README <-> unified doc consistency.** The two agree on install extras,
option names, defaults, and safety caveats. They need not be identical prose,
but they must not contradict each other or the code.
4. **Links.** Unified doc: harness-internal links are relative `.md`
(`[Shell](shell.md)`); Pydantic AI links use
root-relative internal paths `/ai/<section>/<page>/` (not legacy
`ai.pydantic.dev` links); no leftover `../../README.md`, `../capabilities/`,
`../experimental/`, or badge markup.
README: absolute links are fine.
5. **Source link + API block.** Every page links its source module
(`https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/<module>/`)
so a reading agent can verify behavior -- a missing source link is a finding.
Where the capability exposes a public class, the page may also end with a
`## API reference` section of `::: pydantic_ai_harness...` autodoc blocks
(auto-expanded from the docstring, not hand-written). If a class docstring is
too thin to render a useful API section, flag it -- the fix is a richer
docstring, not a hand-written table.
6. **Safety caveats preserved.** Where the source carries access, sandbox, or
command-control limits (Shell, CodeMode, FileSystem), both docs state them.
7. **Writing style.** Both follow `AGENTS.md` "Writing style": no em-dashes (use
`--`), no hype, plain ASCII punctuation.
8. **Purpose-first lead.** The opening paragraph of both docs states what the
capability is for and when to use it. An internal hook or class name
(`before_model_request`, `after_tool_execute`, ...) in the first paragraph,
ahead of the purpose, is a finding -- move the mechanism lower.
9. **Name matches the capability.** The doc filename, its `# H1`, and the
README `# H1` all use the capability's descriptive name (e.g. "Overflowing
Tool Output", not "Overflow"). A short or ClassName-style heading is a finding.
10. **Stability framing.** Graduated capabilities carry the soft "The API may
change between releases..." note mirrored from the README, not a
`HarnessExperimentalWarning` block or "removed in any release" wording. ACP
is the only page that keeps an `!!! warning "Experimental"`.
If a released capability has a README but no `docs/` page (or vice versa), that
missing file is a blocking finding.
## Output
A terse list of findings, most severe first, each naming the file, the severity,
and the fix. If everything is in order, say so in one line. Do not edit files.
.github/workflows/main.yml
+41 -63
View File
@@ -148,69 +148,6 @@ jobs:
- run: uv sync --locked --group dev --all-extras
- run: uv run --no-sync pytest
# Early-warning signal for the pydantic-ai v2 beta line. The harness targets
# the v1 line today (the `>=1.105.0` floor and the lock both resolve to v1),
# but that `>=` constraint also admits the v2 prereleases once prerelease
# resolution is enabled. This job pins the latest v2 beta so we see v2
# breakage as it lands -- e.g. v2 dropped the `calls` argument from
# `ToolManager.get_parallel_execution_mode`, which CodeMode still calls with
# the v1 signature -- rather than discovering it the day v2 is released.
#
# Deliberately absent from the `check` gate below: v2 is unreleased and the
# harness does not yet claim to support it, so a red result here is expected
# and must not block merges. It is a visible signal, not a required check.
test-v2-beta:
runs-on: ubuntu-latest
name: test on pydantic-ai v2 beta
timeout-minutes: 20
# The whole point of the job is to resolve v2 prereleases; allow them for
# every uv invocation in this job rather than threading flags per step.
env:
UV_PRERELEASE: allow
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: astral-sh/setup-uv@cec208311dfd045dd5311c1add060b2062131d57 # v8.0.0
with:
python-version: '3.14'
enable-cache: true # zizmor: ignore[cache-poisoning] -- Job does not produce release artifacts and does not have sensitive permissions
cache-suffix: v2-beta
# `env -u UV_LOCKED`: same as `test-latest` -- under the workflow-level
# `UV_LOCKED=1`, `uv lock` would assert the lockfile is current instead of
# applying the upgrade, so this step would fail the moment a newer v2 beta
# exists rather than testing it.
- run: |
env -u UV_LOCKED uv lock \
--prerelease allow \
--upgrade-package pydantic-ai-slim \
--upgrade-package pydantic-graph
- run: uv sync --locked --group dev --all-extras
# A v2 break can hang instead of failing cleanly: e.g. an unhandled
# exception raised inside a DBOS workflow leaves DBOS's background
# (non-daemon) recovery thread alive, so the Python process never exits
# and the step would otherwise ride to the 20-minute job timeout. The
# `timeout` wrapper caps the step so any such hang fails fast (exit 124)
# rather than burning a full runner slot. `-k 30` escalates to SIGKILL if
# pytest ignores the initial SIGTERM. A per-test plugin (pytest-timeout)
# would not catch this -- the hang happens after the test body finishes,
# during interpreter shutdown -- so the cap has to be on the process.
#
# Two instrumentation assertions depend on pydantic-ai v1's OpenTelemetry
# attribute and span names (`gen_ai.usage.*`, span `agent run`). v2 deliberately
# renamed these (aggregated-usage attributes, and GenAI-semconv span names
# such as `invoke_agent {name}`), so they are expected-red on v2 and carry
# no signal here. Deselect them so this job stays a meaningful early-warning
# for capability breakage (e.g. code mode) rather than documented
# instrumentation drift. If either test is renamed the deselect stops
# matching and the job goes red, which is the right prompt to revisit.
- run: |
timeout -k 30 300 uv run --no-sync pytest \
--deselect "tests/logfire_variables/test_managed_prompt.py::test_baggage_propagates_to_run_and_child_spans" \
--deselect "tests/logfire_variables/test_managed_prompt.py::test_provider_backed_resolution_tags_v1_instrumentation_spans"
coverage:
needs: [test]
runs-on: ubuntu-latest
@@ -263,3 +200,44 @@ jobs:
- run: uv build
- uses: pypa/gh-action-pypi-publish@ed0c53931b1dc9bd32cbe73a98c7f6766f8a527e # v1.13.0
send-tweet:
name: Send tweet
needs: [release]
if: needs.release.result == 'success'
runs-on: ubuntu-latest
environment: release
steps:
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: '3.12'
- name: Install dependencies
run: pip install tweepy==4.14.0
- name: Send tweet
shell: python
run: |
import os
import tweepy
client = tweepy.Client(
access_token=os.getenv("TWITTER_ACCESS_TOKEN"),
access_token_secret=os.getenv("TWITTER_ACCESS_TOKEN_SECRET"),
consumer_key=os.getenv("TWITTER_CONSUMER_KEY"),
consumer_secret=os.getenv("TWITTER_CONSUMER_SECRET"),
)
version = os.getenv("VERSION")
tweet = os.getenv("TWEET").format(version=version)
client.create_tweet(text=tweet)
env:
# The tag ref (e.g. "v1.2.3") already carries the leading v, so it is
# used verbatim in both the message and the release URL.
VERSION: ${{ github.ref_name }}
TWEET: |
Pydantic AI Harness {version} is out! 🎉
https://github.com/pydantic/pydantic-ai-harness/releases/tag/{version}
TWITTER_CONSUMER_KEY: ${{ secrets.TWITTER_CONSUMER_KEY }}
TWITTER_CONSUMER_SECRET: ${{ secrets.TWITTER_CONSUMER_SECRET }}
TWITTER_ACCESS_TOKEN: ${{ secrets.TWITTER_ACCESS_TOKEN }}
TWITTER_ACCESS_TOKEN_SECRET: ${{ secrets.TWITTER_ACCESS_TOKEN_SECRET }}
.gitignore
+5
View File
@@ -3,6 +3,11 @@
.DS_Store
.agents/settings.local.json
.agents/skills/branch-context/
# .agents/ (aka .claude via the symlink) is Claude Code config. Most of it is
# per-developer, but the agents/ dir holds shared, committed agents such as the
# docs-parity-reviewer -- keep it tracked.
!.agents/agents/
!.claude/agents/
AGENTS.local.md
CLAUDE.local.md
LOCAL_WORKTREES.md
AGENTS.md
+19 -18
View File
@@ -21,7 +21,7 @@ change instead of reimplementing core behavior in harness.
- **Capability**: an `AbstractCapability` subclass that bundles tools, hooks, instructions, and model settings into a reusable unit. This is the core abstraction of pydantic-ai-harness.
- **Hook**: a lifecycle method on `AbstractCapability` that intercepts agent graph execution (e.g. `before_model_request`, `wrap_run`, `after_tool_execute`)
- **Toolset**: a collection of tools that a capability can provide to the agent
- **Guard**: a type of capability that validates inputs/outputs or controls tool access (e.g. `InputGuardrail`, `CostGuard`)
- **Guard**: a type of capability that validates inputs/outputs or controls tool access (e.g. `InputGuard`, `OutputGuard`)
- **Harness**: this package -- a collection of pre-made capabilities for Pydantic AI.
- **AICA**: AI Code Assistant -- the automated agent that implements issues, reviews plans, and handles PR feedback
- **Ralph loop**: the state-machine-based workflow that drives AICA through phases (TRIAGE -> GOALS -> PLAN -> CODE -> VERIFY -> REVIEW -> PUBLISH)
@@ -44,9 +44,12 @@ Before implementing or reviewing a capability change:
signatures when needed. Do not assume a contributor's local checkout layout.
5. Use `pydantic_ai_harness.code_mode` as the exemplar for capability shape,
docs, tests, and public exports until another capability becomes a better
example. `code_mode` is a released top-level capability; new capabilities
start under `pydantic_ai_harness.experimental` (see
`agent_docs/capability-authoring.md`, "Experimental Vs Released Exports").
example. Capabilities live in their own top-level submodule
`pydantic_ai_harness/<name>/` (module name = capability name; one module per
capability or strategy) and are not re-exported from the root `__init__.py`,
so each keeps its own optional dependencies. The `experimental` tier is
retired; ACP is the sole remaining experimental capability (see
`agent_docs/capability-authoring.md`, "Capability Submodules And Exports").
## Capabilities API reference
@@ -111,19 +114,18 @@ Always run `make lint && make typecheck && make test` before committing.
## File structure
```
pydantic_ai_harness/
__init__.py # public API re-exports
<capability>/ # each capability gets its own package
__init__.py # public exports for the capability
_capability.py # capability class (AbstractCapability subclass)
_toolset.py # toolset implementation
README.md # standalone docs for the capability
tests/
conftest.py # shared fixtures (TestModel, test_agent)
<capability>/ # tests mirror source packages
test_<capability>.py
```
The tree is discoverable by listing it; only the conventions that are not are
recorded here.
Each released capability is a self-contained package under
`pydantic_ai_harness/<capability>/` (naming and exports are covered in the
preflight above), with tests under `tests/<capability>/`. It ships **two**
hand-maintained docs that must stay in sync: the `README.md` next to the code
(GitHub/PyPI) and the `docs/<capability>.md` page (the docs site at
pydantic.dev/docs/ai/harness). The `docs/` folder is flat -- there are no
`capabilities/` or `experimental/` subdirectories. A user-facing change updates
both; `agent_docs/review-checklist.md` "Docs" and the `docs-parity-reviewer`
subagent enforce the parity before merge.
Do not add placeholder template files for new capabilities. Start from the
existing `CodeMode` package shape, then delete what the new capability does not
@@ -153,4 +155,3 @@ need.
- Always link sources for any claims made during research
- Run `make lint && make typecheck && make test` before every commit
- Commit messages should summarize the "why", not the "what"
- All GitHub comments must start with "Claude here: "
README.md
+67 -20
View File
@@ -15,7 +15,7 @@ Pydantic AI's [capabilities](https://ai.pydantic.dev/capabilities/) and [hooks](
The [capability matrix](#capability-matrix) tracks where we are. [Tell us what to prioritize.](#help-us-prioritize)
**Contents:** [Installation](#installation) · [Quick start](#quick-start) · [Capability matrix](#capability-matrix) · [An ecosystem agent](#an-ecosystem-agent) · [Help us prioritize](#help-us-prioritize) · [Build your own](#build-your-own) · [Contributing](#contributing) · [Version policy](#version-policy) · [Pydantic AI references](#pydantic-ai-references) · [License](#license)
**Contents:** [Installation](#installation) · [Quick start](#quick-start) · [DynamicWorkflow](#orchestrating-sub-agents-dynamicworkflow) · [Capability matrix](#capability-matrix) · [An ecosystem agent](#an-ecosystem-agent) · [Help us prioritize](#help-us-prioritize) · [Build your own](#build-your-own) · [Contributing](#contributing) · [Version policy](#version-policy) · [Pydantic AI references](#pydantic-ai-references) · [License](#license)
## Installation
@@ -26,14 +26,15 @@ uv add pydantic-ai-harness
Extras for specific capabilities:
```bash
uv add "pydantic-ai-harness[codemode]" # CodeMode (adds the Monty sandbox)
uv add "pydantic-ai-harness[logfire]" # ManagedPrompt (Logfire-managed prompts)
uv add "pydantic-ai-harness[acp]" # ACP (serve an agent to editors over the Agent Client Protocol)
uv add "pydantic-ai-harness[codemode]" # CodeMode (adds the Monty sandbox)
uv add "pydantic-ai-harness[dynamic-workflow]" # DynamicWorkflow (adds the Monty sandbox)
uv add "pydantic-ai-harness[logfire]" # ManagedPrompt (Logfire-managed prompts)
uv add "pydantic-ai-harness[acp]" # ACP (serve an agent to editors over the Agent Client Protocol)
```
The `code-mode` extra is also supported as an alias.
Requires Python 3.10+ and `pydantic-ai-slim>=1.95.1`.
Requires Python 3.10+ and `pydantic-ai-slim>=2.1.0`.
## Quick start
@@ -93,7 +94,47 @@ practices and warning of a "normalization of deviance" as engineers stop reviewi
[![Logfire trace from the Quick start run](docs/images/quick-start-trace.png)](https://logfire-us.pydantic.dev/public-trace/84bcf123-2106-49da-9f6f-5c26395339bb?spanId=7650806a0785b946)
**[See this run as a public Logfire trace →](https://logfire-us.pydantic.dev/public-trace/84bcf123-2106-49da-9f6f-5c26395339bb?spanId=7650806a0785b946)** Each `run_code` span fans out into the tool calls the model issued from inside the sandbox -- it's the easiest way to understand what code mode actually did.
**[See this run as a public Logfire trace ->](https://logfire-us.pydantic.dev/public-trace/84bcf123-2106-49da-9f6f-5c26395339bb?spanId=7650806a0785b946)** Each `run_code` span fans out into the tool calls the model issued from inside the sandbox -- it's the easiest way to understand what code mode actually did.
## Orchestrating sub-agents: DynamicWorkflow
`CodeMode` gives the model one script for its *tools*. `DynamicWorkflow` does the same for *sub-agents*. Without it, an orchestrator delegates one tool call at a time: call a sub-agent, wait, read the result into context, think, call the next one. Ten delegations cost ten model round-trips, and every intermediate result flows through the orchestrator's context whether it needed to see it or not.
With it, the model writes one Python script in which each sub-agent is an async function, and the whole tree runs in a single tool call:
```python
from pydantic_ai import Agent
from pydantic_ai_harness.dynamic_workflow import DynamicWorkflow
reviewer = Agent('anthropic:claude-sonnet-4-6', name='reviewer', description='Reviews code for bugs.')
summarizer = Agent('anthropic:claude-sonnet-4-6', name='summarizer', description='Summarizes findings.')
orchestrator = Agent(
'anthropic:claude-opus-4-7',
capabilities=[DynamicWorkflow(agents=[reviewer, summarizer])],
)
```
The script the model writes looks like this -- fan out, chain, and only the last line's value returns to its context:
```python
import asyncio
reports = await asyncio.gather(
reviewer(task="Review auth.py for bugs:\n<file contents>"),
reviewer(task="Review parser.py for bugs:\n<file contents>"),
)
await summarizer(task="Summarize these findings:\n" + "\n\n".join(reports))
```
It composes with the rest of the harness:
- **Budgets**: `max_agent_calls` is an exact, host-enforced ceiling on sub-agent runs (it holds even under concurrent fan-out), and by default the whole tree's token spend lands on the parent run's `usage`.
- **On-demand**: `defer_loading=True` keeps the catalog out of the prompt until the model loads the capability, and `reveal()` adds a sub-agent mid-run without disturbing the prompt cache.
`DynamicWorkflow`'s API is subject to change while planned extensions (structured sub-agent inputs, durable workflows) settle the call contract. Breaking changes ship deprecation warnings where practical.
[Full tutorial →](pydantic_ai_harness/dynamic_workflow/)
## Capability matrix
@@ -107,24 +148,29 @@ We studied leading coding agents, agent frameworks, and Claw-style assistants to
| | **Tool search** | Progressive tool discovery for large tool sets | :white_check_mark: [Pydantic&nbsp;AI](https://pydantic.dev/docs/ai/tools-toolsets/toolsets/#deferred-loading) | |
| | **File system** | Read, write, edit, search files with path traversal prevention | :white_check_mark: [Docs](pydantic_ai_harness/filesystem/) | [pydantic-ai-backend](https://github.com/vstorm-co/pydantic-ai-backend) (vstorm&#8209;co) |
| | **Shell** | Execute commands with allowlists, denylists, and timeouts | :white_check_mark: [Docs](pydantic_ai_harness/shell/) | [pydantic-ai-backend](https://github.com/vstorm-co/pydantic-ai-backend) (vstorm&#8209;co) |
| | **Repo context injection** | Auto-load CLAUDE.md/AGENTS.md and repo structure | :construction: [PR&nbsp;#175](https://github.com/pydantic/pydantic-ai-harness/pull/175) | [pydantic-deep](https://github.com/vstorm-co/pydantic-deepagents) (vstorm&#8209;co) |
| | **Repo context injection** | Auto-load CLAUDE.md/AGENTS.md and repo structure | :white_check_mark: [Docs](pydantic_ai_harness/context/) | [pydantic-deep](https://github.com/vstorm-co/pydantic-deepagents) (vstorm&#8209;co) |
| | **Docs lookup** | On-demand `read_pyai_docs` tool for Pydantic AI docs | :white_check_mark: [Docs](pydantic_ai_harness/docs/) | |
| | **Verification loop** | Run tests after edits, auto-fix failures | :construction: [PR&nbsp;#169](https://github.com/pydantic/pydantic-ai-harness/pull/169) | |
| **Editor integration** | **ACP** | Serve an agent to editors (Zed, etc.) over the [Agent Client Protocol](https://agentclientprotocol.com) -- streamed text, diff-rendered edits, tool approval | :white_check_mark: [Docs](pydantic_ai_harness/experimental/acp/) (experimental) | |
| **Context management** | **Sliding window** | Trim conversation history to stay within token limits | :construction: [PR&nbsp;#191](https://github.com/pydantic/pydantic-ai-harness/pull/191) | [summarization-pydantic-ai](https://github.com/vstorm-co/summarization-pydantic-ai) (vstorm&#8209;co) |
| | **Context compaction** | LLM-powered summarization of older messages | :construction: [PR&nbsp;#191](https://github.com/pydantic/pydantic-ai-harness/pull/191) | [summarization-pydantic-ai](https://github.com/vstorm-co/summarization-pydantic-ai) (vstorm&#8209;co) |
| | **Limit warnings** | Warn agent before hitting context/iteration limits | :construction: [PR&nbsp;#191](https://github.com/pydantic/pydantic-ai-harness/pull/191) | [summarization-pydantic-ai](https://github.com/vstorm-co/summarization-pydantic-ai) (vstorm&#8209;co) |
| | **Tool output management** | Truncate, summarize, or spill large tool outputs | :construction: [PR&nbsp;#185](https://github.com/pydantic/pydantic-ai-harness/pull/185) | |
| **Prompt management** | **Managed prompt** | Back an agent's instructions with a [Logfire](https://pydantic.dev/logfire)-managed prompt, editable without shipping code | :white_check_mark: [Docs](pydantic_ai_harness/logfire/) | |
| **Context management** | **Sliding window** | Trim conversation history to stay within token limits | :white_check_mark: [Docs](pydantic_ai_harness/compaction/) | [summarization-pydantic-ai](https://github.com/vstorm-co/summarization-pydantic-ai) (vstorm&#8209;co) |
| | **Context compaction** | LLM-powered summarization of older messages | :white_check_mark: [Docs](pydantic_ai_harness/compaction/) | [summarization-pydantic-ai](https://github.com/vstorm-co/summarization-pydantic-ai) (vstorm&#8209;co) |
| | **Limit warnings** | Warn agent before hitting context/iteration limits | :white_check_mark: [Docs](pydantic_ai_harness/compaction/) | [summarization-pydantic-ai](https://github.com/vstorm-co/summarization-pydantic-ai) (vstorm&#8209;co) |
| | **Tool output management** | Truncate, summarize, or spill large tool outputs | :white_check_mark: [Docs](pydantic_ai_harness/overflowing_tool_output/) | |
| | **System reminders** | Inject periodic reminders to counteract instruction drift | :construction: [PR&nbsp;#181](https://github.com/pydantic/pydantic-ai-harness/pull/181) | |
| **Memory &&nbsp;persistence** | **Memory** | Persistent key-value memory across sessions | :construction: [PR&nbsp;#179](https://github.com/pydantic/pydantic-ai-harness/pull/179) | [pydantic-deep](https://github.com/vstorm-co/pydantic-deepagents) (vstorm&#8209;co) |
| | **Session persistence** | Save and restore full conversation state | :construction: [PR&nbsp;#176](https://github.com/pydantic/pydantic-ai-harness/pull/176) | |
| | **Checkpointing** | Save, rewind, and fork conversation state | :memo: [#196](https://github.com/pydantic/pydantic-ai-harness/issues/196) | [pydantic-deep](https://github.com/vstorm-co/pydantic-deepagents) (vstorm&#8209;co) |
| **Agent orchestration** | **Sub-agents** | Delegate subtasks to specialized child agents | :construction: [PR&nbsp;#178](https://github.com/pydantic/pydantic-ai-harness/pull/178) | [subagents-pydantic-ai](https://github.com/vstorm-co/subagents-pydantic-ai) (vstorm&#8209;co) |
| **Memory &&nbsp;persistence** | **Memory** | Persistent, namespaced notebook with bounded prompt injection, on-demand search, and concurrency-safe stores | :white_check_mark: [Docs](pydantic_ai_harness/memory/) | [pydantic-deep](https://github.com/vstorm-co/pydantic-deepagents) (vstorm&#8209;co) |
| | **Session persistence** | Save and restore full conversation state | :white_check_mark: [Docs](pydantic_ai_harness/step_persistence/) | |
| | **Checkpointing** | Snapshot, resume (`continue_run`), and fork (`fork_run`) a run | :white_check_mark: [Docs](pydantic_ai_harness/step_persistence/) | [pydantic-deep](https://github.com/vstorm-co/pydantic-deepagents) (vstorm&#8209;co) |
| | **Media externalization** | Offload large `BinaryContent` to content-addressed stores (building blocks) | :white_check_mark: [Docs](pydantic_ai_harness/media/) | |
| **Agent orchestration** | **Sub-agents** | Delegate subtasks to specialized child agents | :white_check_mark: [Docs](pydantic_ai_harness/subagents/) | [subagents-pydantic-ai](https://github.com/vstorm-co/subagents-pydantic-ai) (vstorm&#8209;co) |
| | **Dynamic workflow** | Orchestrate sub-agents from a model-written Python script -- fan-out, chaining, voting in one tool call | :white_check_mark: [Docs](pydantic_ai_harness/dynamic_workflow/) | |
| | **Skills** | Progressive tool loading -- search, activate, deactivate | :construction: [PR&nbsp;#183](https://github.com/pydantic/pydantic-ai-harness/pull/183) | [pydantic-ai-skills](https://github.com/DougTrajano/pydantic-ai-skills) (DougTrajano), [pydantic-deep](https://github.com/vstorm-co/pydantic-deepagents) (vstorm&#8209;co) |
| | **Planning** | Break complex tasks into structured plans before execution | :construction: [PR&nbsp;#180](https://github.com/pydantic/pydantic-ai-harness/pull/180) | |
| | **Planning** | Break complex tasks into structured plans before execution | :white_check_mark: [Docs](pydantic_ai_harness/planning/) | |
| | **Runtime authoring** | Let an agent author, validate, and load real capabilities at runtime | :white_check_mark: [Docs](pydantic_ai_harness/runtime_authoring/) | |
| | **Task tracking** | Track tasks, subtasks, and dependencies | :memo: [#65](https://github.com/pydantic/pydantic-ai-harness/issues/65) | [pydantic-ai-todo](https://github.com/vstorm-co/pydantic-ai-todo) (vstorm&#8209;co) |
| | **Teams** | Multi-agent teams with shared state and message bus | :memo: [#195](https://github.com/pydantic/pydantic-ai-harness/issues/195) | [pydantic-deep](https://github.com/vstorm-co/pydantic-deepagents) (vstorm&#8209;co) |
| **Safety &&nbsp;guardrails** | **Input guardrails** | Validate user input before the agent run starts | :construction: [PR&nbsp;#182](https://github.com/pydantic/pydantic-ai-harness/pull/182) | [pydantic-ai-shields](https://github.com/vstorm-co/pydantic-ai-shields) (vstorm&#8209;co) |
| | **Output guardrails** | Validate model output after the run completes | :construction: [PR&nbsp;#182](https://github.com/pydantic/pydantic-ai-harness/pull/182) | [pydantic-ai-shields](https://github.com/vstorm-co/pydantic-ai-shields) (vstorm&#8209;co) |
| **Safety &&nbsp;guardrails** | **Input guardrails** | Validate user input before the agent run starts | :white_check_mark: [Docs](pydantic_ai_harness/guardrails/) | [pydantic-ai-shields](https://github.com/vstorm-co/pydantic-ai-shields) (vstorm&#8209;co) |
| | **Output guardrails** | Validate model output after the run completes | :white_check_mark: [Docs](pydantic_ai_harness/guardrails/) | [pydantic-ai-shields](https://github.com/vstorm-co/pydantic-ai-shields) (vstorm&#8209;co) |
| | **Cost/token budgets** | Enforce token and cost limits per run | :construction: [PR&nbsp;#182](https://github.com/pydantic/pydantic-ai-harness/pull/182) | [pydantic-ai-shields](https://github.com/vstorm-co/pydantic-ai-shields) (vstorm&#8209;co) |
| | **Tool access control** | Block tools or require approval before execution | :construction: [PR&nbsp;#182](https://github.com/pydantic/pydantic-ai-harness/pull/182) | [pydantic-ai-shields](https://github.com/vstorm-co/pydantic-ai-shields) (vstorm&#8209;co) |
| | **Async guardrails** | Run validation concurrently with model requests | :construction: [PR&nbsp;#182](https://github.com/pydantic/pydantic-ai-harness/pull/182) | [pydantic-ai-shields](https://github.com/vstorm-co/pydantic-ai-shields) (vstorm&#8209;co) |
@@ -136,6 +182,7 @@ We studied leading coding agents, agent frameworks, and Claw-style assistants to
| | **Tool orphan repair** | Fix orphaned tool calls in conversation history | :construction: [PR&nbsp;#184](https://github.com/pydantic/pydantic-ai-harness/pull/184) | |
| **Reasoning** | **Adaptive reasoning** | Adjust thinking effort based on task complexity | :construction: [PR&nbsp;#174](https://github.com/pydantic/pydantic-ai-harness/pull/174) | |
| | **Current time** | Inject current date/time into system prompt | :construction: [PR&nbsp;#170](https://github.com/pydantic/pydantic-ai-harness/pull/170) | |
| **Prompt management** | **Managed prompt** | Serve Logfire-managed prompts as instructions, so you can edit and roll them out without a code deploy | :white_check_mark: [Docs](pydantic_ai_harness/logfire/) | |
> Packages by [vstorm-co](https://github.com/vstorm-co) are endorsed by the Pydantic AI team. We're working with them to upstream some of their implementations into this repo.
@@ -280,8 +327,8 @@ make testcov # pytest with 100% branch coverage
Pydantic AI Harness uses **0.x versioning** to signal that APIs are still stabilizing. During 0.x:
- **Minor releases** (0.1 → 0.2) may include breaking changes -- renamed parameters, changed defaults, restructured APIs. As the library grows, especially as capabilities gain provider-native support (starting as a local implementation, then auto-switching to the provider's built-in API when available), we may need to reshape APIs we couldn't fully anticipate in the initial design.
- **Patch releases** (0.1.0 → 0.1.1) will not intentionally break existing behavior.
- **Minor releases** (0.1 -> 0.2) may include breaking changes -- renamed parameters, changed defaults, restructured APIs. As the library grows, especially as capabilities gain provider-native support (starting as a local implementation, then auto-switching to the provider's built-in API when available), we may need to reshape APIs we couldn't fully anticipate in the initial design.
- **Patch releases** (0.1.0 -> 0.1.1) will not intentionally break existing behavior.
- **All breaking changes** are documented in release notes with migration guidance.
- Where practical, we'll keep the previous behavior available under a deprecated name or configuration option before removing it.
agent_docs/capability-authoring.md
+34 -13
View File
@@ -25,27 +25,48 @@ Each capability package should normally have:
- `__init__.py` with public exports
- `_capability.py` for the public capability class
- `_toolset.py` only if the capability needs toolset behavior
- `README.md` with focused usage docs
- `README.md` with focused usage docs (serves GitHub and PyPI)
- a unified-docs page at `docs/<capability>.md` (the `docs/` folder is flat --
no `capabilities/` or `experimental/` subdirectories). It mirrors the README
for the docs site, drops badges, links other harness pages with relative `.md`
links and Pydantic AI docs with root-relative `/ai/...` links, links its
source module, and -- where the capability exposes a public class -- may end
with a `::: pydantic_ai_harness.<Class>` autodoc block. The README and this
page are kept in sync (see `review-checklist.md` "Docs").
- mirrored tests under `tests/<capability>/`
The root `pydantic_ai_harness/__init__.py` should re-export stable public
capabilities. Keep implementation helpers private unless users need them.
### Experimental Vs Released Exports
### Capability Submodules And Exports
New capabilities land under `pydantic_ai_harness/experimental/`. Each
experimental package calls `warn_experimental('<name>')` in its `__init__.py` so
importing it emits `HarnessExperimentalWarning`. Anything under `experimental`
may change or be removed in any release, without a deprecation period.
The `experimental` tier is retired. ACP is the sole remaining experimental
capability (`pydantic_ai_harness/experimental/acp/`); do not add new capabilities
there.
Promote a capability to a top-level package and a top-level re-export in
`pydantic_ai_harness/__init__.py` only when its API is stable.
New capabilities land as a top-level submodule `pydantic_ai_harness/<name>/`.
They are not re-exported from the root `pydantic_ai_harness/__init__.py`: each
capability keeps its own optional dependencies, so importing the root package
must not pull in a capability's extras. Users import a capability from its
submodule (`from pydantic_ai_harness.<name> import ...`).
Top-level exports in `pydantic_ai_harness/__init__.py` are the intended public
surface. Once an export has shipped in a published release it is a
backward-compatibility commitment: do not move, rename, or break it. (`CodeMode`
is a shipped, released example.) Do not relocate an already-top-level capability
into `experimental`.
Naming: the module name is the capability name, one module per capability or
strategy. Prefer a longer descriptive name over a terse one (e.g.
`overflowing_tool_output`, not `overflow`). A known term is fine as-is (e.g.
`compaction`). If you are unsure what to name a capability, ask the user (via the
ask-user tool) rather than guessing -- a name is a public commitment once shipped.
When a capability's module path changes, keep the old path working as a
`DeprecationWarning` shim so existing imports do not break.
Top-level re-exports in `pydantic_ai_harness/__init__.py` (`CodeMode`,
`FileSystem`, `Shell`, `ManagedPrompt`) are the exception, not the rule. Once an
export has shipped in a published release it is a backward-compatibility
commitment: do not move, rename, or break it. Do not add new top-level
re-exports.
APIs are subject to change between releases; breaking changes ship deprecation
warnings where practical.
## API Design
agent_docs/review-checklist.md
+53 -2
View File
@@ -45,7 +45,58 @@ well before now, or that was built against unreleased Pydantic AI changes.
## Docs
- Capability README or root README is updated for user-facing behavior.
- Examples match declared extras.
Every released capability ships two hand-maintained docs that must stay in sync
with the code and with each other:
- the **README** next to the implementation (`pydantic_ai_harness/<capability>/README.md`,
or `pydantic_ai_harness/experimental/<capability>/README.md` for ACP), which
serves GitHub and PyPI, and
- the **unified doc** on the docs site, flat under `docs/<capability>.md`. The
sidebar is a flat list under "Pydantic AI Harness" -- no `capabilities/` or
`experimental/` subdirectories.
Checks:
- Both the README and the unified doc are updated for any user-facing change
(public class, params, defaults, tool names, extras, safety semantics). A
change reflected in only one of them is a defect, not a follow-up.
- The two do not contradict each other or the source on extras, option names,
defaults, or safety caveats.
- Every snippet in both docs is runnable: all imports present, class/param names
match the source, model ids unchanged from what the source uses. Imports use
the canonical module path (never `pydantic_ai_harness.experimental.*` for a
graduated capability).
- **Purpose-first lead.** The opening paragraph of each page and README states
what the capability is for and when to reach for it -- no internal hook or
class name (`before_model_request`, `after_tool_execute`, ...) before the
purpose. Mechanism belongs lower down.
- **Name matches the capability.** The doc filename, its `# H1`, and the
README's `# H1` all use the capability's descriptive name (e.g.
"Overflowing Tool Output", not "Overflow"; "Runtime Authoring", not
"Authoring").
- **Source link.** Each page links its source module
(`https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/<module>/`)
so a reading agent can verify behavior. Where the capability exposes a public
class, the page may also end with a `## API reference` section of
`::: pydantic_ai_harness...` autodoc blocks (auto-expanded from the docstring,
not hand-written).
- **Stability framing.** Graduated capabilities carry the soft note "The API may
change between releases..." mirrored from their README -- NOT a
`HarnessExperimentalWarning` block or "removed in any release" wording. ACP is
the only page that keeps an `!!! warning "Experimental"` (it may still be
removed).
- Links: harness-internal links are relative `.md`; Pydantic AI docs use
root-relative internal links `/ai/<section>/<page>/` (verify the route resolves
on the live `pydantic.dev/docs` site before using it).
- Docs explain composition constraints and safety implications.
- The PR links an issue.
The mechanical half of these checks (README present + linked, flat page present,
source link present, name matches, no experimental strings on non-ACP pages, no
hook name in the lead) is enforced by `tests/test_docs_parity.py`. The semantic
half (does the prose match the code, are snippets truly runnable) is what the
reviewer below is for.
This is the last documentation gate before merge. Run the `docs-parity-reviewer`
subagent (`.agents/agents/docs-parity-reviewer.md`) on the change as the final
review step; treat its blocking findings as merge blockers.
docs/acp.md
+246
View File
@@ -0,0 +1,246 @@
---
title: ACP (Agent Client Protocol)
description: Serve a Pydantic AI agent to editors and terminal UIs over the Agent Client Protocol -- streamed text, diff-rendered file edits, human-in-the-loop tool approval, and per-workspace sessions.
---
# ACP (Agent Client Protocol)
Editors like [Zed](https://zed.dev/docs/ai/external-agents) speak ACP: a stdio JSON-RPC protocol that lets a TUI or editor drive an external coding agent -- streaming its text, rendering its file edits as diffs, and prompting the user to approve sensitive tool calls. Reach for this capability when you want a Pydantic AI `Agent` to appear as a first-class agent inside one of those editors, without implementing the ACP server side yourself.
!!! warning "Experimental"
Unlike the graduated capabilities in these docs, ACP itself may still be **removed**, not just changed -- it lives under `pydantic_ai_harness.experimental` and may change or be removed in any release, without a deprecation period. Import it from the experimental path -- there is no top-level export:
```python
from pydantic_ai_harness.experimental.acp import run_acp_stdio_sync
```
Importing any experimental capability emits a `HarnessExperimentalWarning`. Silence **all** harness experimental warnings with a single filter (no per-capability lines needed):
```python
import warnings
from pydantic_ai_harness.experimental import HarnessExperimentalWarning
warnings.filterwarnings('ignore', category=HarnessExperimentalWarning)
```
## The problem
To plug a Pydantic AI agent into an ACP editor you would otherwise have to implement the ACP server side by hand -- chunking streamed text under the wire limit, rendering tool calls as diffs, mapping the protocol's permission requests onto the agent's tools, and managing per-workspace sessions.
## The solution
`run_acp_stdio` serves any Pydantic AI `Agent` as an ACP agent over stdin/stdout. The editor launches your script as a subprocess and talks to it; the adapter translates between ACP and the agent's run loop:
| ACP needs | The adapter provides |
|---|---|
| Streamed assistant text and reasoning | Agent text/thinking deltas, chunked under the wire limit |
| Rich tool calls (`kind`, file `locations`, diffs) | A presenter that recognizes `FileSystem`/`Shell` tool calls |
| Human-in-the-loop tool approval | Maps ACP permission requests to Pydantic AI's deferred-approval tools |
| Per-workspace sessions | A `session_config` hook to root tools at the client's working directory |
| Cancellation, multi-turn history, session close | Handled per session |
## Installation
```bash
uv add "pydantic-ai-harness[acp]"
```
This pulls in the [`agent-client-protocol`](https://pypi.org/project/agent-client-protocol/) SDK. The rest of the harness does not depend on it -- only `pydantic_ai_harness.experimental.acp` does.
## Quick start
Write a script that builds your agent and serves it:
```python
# my_acp_agent.py
from pydantic_ai import Agent
from pydantic_ai_harness.experimental.acp import run_acp_stdio_sync
def build_agent() -> Agent[None, str]:
return Agent('anthropic:claude-sonnet-4-6', instructions='You are a coding assistant.')
if __name__ == '__main__':
run_acp_stdio_sync(build_agent())
```
`run_acp_stdio_sync` blocks for the lifetime of the connection -- it is the `main()` of an agent the editor launches. Inside an existing event loop, use the async `run_acp_stdio` instead.
## Connecting from an editor
ACP clients launch the agent as a subprocess. In Zed, register it as an [external agent](https://zed.dev/docs/ai/external-agents) in `settings.json`:
```json
{
"agent_servers": {
"My Pydantic AI Agent": {
"type": "custom",
"command": "python",
"args": ["/absolute/path/to/my_acp_agent.py"],
"env": { "ANTHROPIC_API_KEY": "..." }
}
}
}
```
Any ACP-compatible client works the same way -- point it at `python my_acp_agent.py`.
The provider environment must be available to the launched subprocess. GUI editors and SDK-based test wrappers may not source your interactive shell startup files. If a real-model agent exits before initialize or fails provider auth, first verify that the command's process can see variables such as `ANTHROPIC_API_KEY`.
## Rooting tools at the workspace
A coding agent should read and write files in the workspace the editor opened, not wherever the subprocess started. ACP gives each session a working directory (`cwd`); a `session_config` factory turns that into per-session tools:
```python
from pydantic_ai import Agent
from pydantic_ai_harness.experimental.acp import AcpSession, AcpSessionConfig, run_acp_stdio_sync
from pydantic_ai_harness.filesystem import FileSystem
from pydantic_ai_harness.shell import Shell
agent = Agent('anthropic:claude-sonnet-4-6')
def session_config(session: AcpSession) -> AcpSessionConfig[None]:
# Root file and shell tools at the workspace the client opened.
return AcpSessionConfig(
deps=None,
toolsets=[
FileSystem[None](root_dir=session.cwd).get_toolset(),
Shell[None](cwd=session.cwd).get_toolset(),
],
)
if __name__ == '__main__':
run_acp_stdio_sync(agent, session_config=session_config)
```
The factory runs once per session with the client's `AcpSession` setup (its `cwd`, `mcp_servers`, and capabilities) and returns an `AcpSessionConfig` whose `deps` and `toolsets` apply to every run in that session. This is correct across multiple concurrent sessions in one process, where a single static `FileSystem` could not be.
## Editor-native filesystem and shell (optional)
The local [`FileSystem`](filesystem.md) and [`Shell`](shell.md) above operate on the agent process's own disk and subprocesses. An editor's source of truth is different: unsaved buffers, its own idea of the workspace layout, and -- for a remote or containerized editor -- the machine the code actually lives on. When the client advertises support, `acp_filesystem` and `acp_terminal` give the agent `read_file`/`write_file`/`run_command` tools that route through the client, so it acts where the user is:
```python
from pydantic_ai_harness.experimental.acp import AcpSession, AcpSessionConfig, acp_filesystem, acp_terminal
from pydantic_ai_harness.filesystem import FileSystem
from pydantic_ai_harness.shell import Shell
def session_config(session: AcpSession) -> AcpSessionConfig[None]:
# Use the editor's filesystem/terminal when offered; otherwise fall back to local.
fs = acp_filesystem(session) or FileSystem[None](root_dir=session.cwd).get_toolset()
shell = acp_terminal(session) or Shell[None](cwd=session.cwd).get_toolset()
return AcpSessionConfig(deps=None, toolsets=[fs, shell])
```
Each helper returns `None` when the client did not advertise the capability, so the `or` falls back to local and the agent works either way. The tool names match the local `FileSystem`/`Shell`, so rich rendering stays identical.
## Tool approval
Mark a tool to require approval and ACP relays the decision to the client, which shows the user an approve/reject prompt:
```python
@agent.tool_plain(requires_approval=True)
def delete_file(path: str) -> str:
...
```
The lifecycle the client sees is `pending` (awaiting approval) -> `in_progress` (granted, running) -> `completed`/`failed`, so an unapproved action is never shown as already running. "Always allow"/"always reject" decisions are remembered for the session, scoped by default to the exact call (tool name plus arguments) so approving one call never silently approves a different one. Pass `permission_policy` to widen or narrow that scope.
## Rich tool rendering
By default the adapter recognizes the harness `FileSystem` and `Shell` tool calls by name and annotates them with an ACP `kind` (`read`/`edit`/`search`/`execute`), the file `locations` they touch, and an inline diff for edits -- so the editor renders click-to-file links and diff views instead of opaque JSON. Pass `tool_presenter` to add rendering for your own tools (optionally with `chain_presenters` ahead of the default `default_coding_presenter`), or `lambda _call: None` to disable it.
## MCP servers
An ACP client may offer MCP servers during session setup. This adapter does not connect them itself; a `session_config` is the place to turn `session.mcp_servers` into Pydantic AI toolsets (for example with `pydantic_ai.mcp.MCPServerStdio`). If a client sends MCP servers and no `session_config` is installed to consume them, the session request is rejected rather than silently ignoring them. A spec-following client only sends HTTP/SSE MCP servers when the agent advertises support during `initialize`; when your `session_config` connects them, say so:
```python
from acp import schema
from pydantic_ai_harness.experimental.acp import PydanticAIACPAgent
PydanticAIACPAgent(
agent,
session_config=connect_mcp_servers,
mcp_capabilities=schema.McpCapabilities(http=True, sse=True),
)
```
## Prompt content types
The agent advertises which prompt content it accepts. The default is **text only**, so a client is not invited to send blocks a text model cannot handle. Enable the kinds your model supports:
```python
from acp import schema
run_acp_stdio_sync(agent, prompt_capabilities=schema.PromptCapabilities(image=True, embedded_context=True))
```
## Session persistence
Pass a `session_store` to let a client reopen a past conversation with `session/load`. Each committed turn is persisted as two parts -- the model's message history and the client-visible transcript -- and reopening restores the history into the agent and replays the transcript to the client, so its UI is rebuilt as the user last saw it. Without a store, `session/load` is advertised as unsupported.
```python
from pydantic_ai_harness.experimental.acp import InMemorySessionStore
run_acp_stdio_sync(agent, session_store=InMemorySessionStore())
```
`InMemorySessionStore` keeps sessions for the lifetime of the process. Implement the `SessionStore` protocol (`save`/`load` a `StoredSession`) over a file or database to make them survive a restart -- the stored values are Pydantic models, so they serialize with Pydantic. Session persistence is for *reopening a conversation*; it is orthogonal to per-run durability. To also make individual turns crash-resilient, add a [step-durability capability](step-persistence.md) -- each ACP turn is one agent run, so the two layers compose with no glue.
## Model selection
Pass `models` to advertise a stable ACP session config option named `model` (using Pydantic AI [model names](/ai/models/)). The first is each session's default. A selection is applied as a per-run override -- the shared agent is never mutated -- and is persisted with the session when a `session_store` is set.
```python
run_acp_stdio_sync(agent, models=['anthropic:claude-sonnet-4-6', 'anthropic:claude-opus-4-8', 'openai:gpt-4o'])
```
A model id is any string a Pydantic AI model accepts, so newer models not yet in `KnownModelName` work too. Pass `models='all'` to offer every model Pydantic AI knows. To advertise ids `infer_model` does not understand (OAuth or subscription models), pass `model_resolver` to map the selected id to a prebuilt `Model`.
## Cancellation and limitations
- **Cancellation.** `session/cancel` and `session/close` cancel the in-flight turn; close waits for it to unwind before returning. Cooperative async tools stop promptly. A synchronous tool already running in a worker thread cannot be force-stopped, so prefer async tools for cancellation-sensitive work.
- **Approval detection.** Tools that require approval are recognized when they live in a `FunctionToolset` (which the harness `FileSystem`/`Shell` and `@agent.tool` all use). A tool whose approval requirement is decided dynamically per call (by raising `ApprovalRequired` from its body) starts as `in_progress`, and any side effects it ran before raising have already happened -- use an `ApprovalRequiredToolset` for actions that must not partially execute before approval.
- **Overwrite diffs.** `write_file` renders an overwrite as if creating a new file, so the diff understates what it replaced.
- **Live terminal panes.** `acp_terminal` returns a command's captured output; it does not embed a live terminal pane in the tool call.
- **Images.** Prompt image blocks are off by default and must be enabled via `prompt_capabilities` with a model that accepts them.
- **Slash commands.** The adapter does not yet advertise any commands, so no slash commands appear in the client. Planned.
## API
```python {test="skip"}
run_acp_stdio( # async; serve until the client disconnects
agent,
*,
deps=None,
name=None, # advertised name; defaults to the agent's name
version='0.1.0',
session_config=None, # per-session deps/toolsets from the client's setup
permission_policy=None, # scope of remembered "always" approval decisions
prompt_capabilities=None, # defaults to text-only
mcp_capabilities=None, # MCP transports to advertise; needs a session_config to connect them
tool_presenter=None, # defaults to the FileSystem/Shell presenter
session_store=None, # enables session/load by persisting each session
models=None, # models offered as the `model` config option ('all' for every known model)
model_resolver=None, # maps an advertised model id to the Model used for the run
usage_limits=None, # per-run request/token ceilings
)
run_acp_stdio_sync(...) # synchronous wrapper, same arguments
PydanticAIACPAgent(agent, *, ...) # the ACP agent object, to embed in a custom server
```
The module also exports the session types (`AcpSession`, `AcpSessionConfig`, `McpServer`), the store types (`SessionStore`, `StoredSession`, `InMemorySessionStore`), the client toolsets (`AcpFileSystemToolset`, `AcpTerminalToolset`, `acp_filesystem`, `acp_terminal`), the permission types (`ToolCallPermission`, `default_permission_scope`), and the presentation helpers (`ToolCallPresentation`, `chain_presenters`, `default_coding_presenter`).
Source: [`pydantic_ai_harness/experimental/acp/`](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/experimental/acp/).
## Further reading
- [Agent Client Protocol](https://agentclientprotocol.com) -- protocol specification
- [Zed external agents](https://zed.dev/docs/ai/external-agents) -- editor-side configuration
- [Human-in-the-loop tool approval](/ai/tools-toolsets/deferred-tools/#human-in-the-loop-tool-approval) (Pydantic AI)
- [Pydantic AI capabilities](/ai/core-concepts/capabilities/)
docs/code-mode.md
+327
View File
@@ -0,0 +1,327 @@
---
title: Code Mode
description: Wrap an agent's tools into a single sandboxed run_code tool so the model orchestrates many calls in one Python program instead of many round-trips.
---
# Code Mode
`CodeMode` replaces individual tool calls with a single sandboxed Python execution environment. Instead of the model issuing one tool call per action, it writes a Python program that calls your tools as functions -- with loops, conditionals, variables, and `asyncio.gather` -- all inside a sandboxed [Monty](https://github.com/pydantic/monty) runtime.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/code_mode/)
## The problem
Standard tool calling costs one model round-trip per tool call. An agent that needs to fetch 10 items and process each one makes 11+ model calls -- slow, expensive, and heavy on context. The conversation history grows with every intermediate result, and everything runs sequentially unless the model deliberately batches parallel tool calls.
## The solution
`CodeMode` wraps the agent's tools into a single `run_code` tool. The model writes one Python snippet that orchestrates many tool calls locally: fan them out with `asyncio.gather`, filter and transform results in plain Python, and return only what matters.
| Standard tool calling | Code mode |
|---|---|
| 1 model call per tool | 1 model call for N tools |
| Sequential by default | Parallel via `asyncio.gather` |
| No local computation | Filter, transform, aggregate in code |
| Large conversation history | Compact -- fewer messages |
## Installation
Code mode requires the Monty sandbox, available via the `codemode` extra (the `code-mode` extra is an equivalent alias):
```bash
uv add "pydantic-ai-harness[codemode]"
```
## Usage
Construct an `Agent` with `CodeMode()` in its `capabilities`, then register tools as usual. Every tool becomes callable from inside `run_code`:
```python
from pydantic_ai import Agent
from pydantic_ai_harness import CodeMode
agent = Agent('anthropic:claude-sonnet-4-6', capabilities=[CodeMode()])
@agent.tool_plain
def get_weather(city: str) -> dict:
"""Get current weather for a city."""
return {'city': city, 'temp_f': 72, 'condition': 'sunny'}
@agent.tool_plain
def convert_temp(fahrenheit: float) -> float:
"""Convert Fahrenheit to Celsius."""
return round((fahrenheit - 32) * 5 / 9, 1)
result = agent.run_sync("What's the weather in Paris and Tokyo, in Celsius?")
print(result.output)
```
Inside a single `run_code` call, the model writes code like the following (illustrative -- the exact code the model emits will vary):
```python
import asyncio
paris, tokyo = await asyncio.gather(
get_weather(city='Paris'),
get_weather(city='Tokyo'),
)
paris_c = await convert_temp(fahrenheit=paris['temp_f'])
tokyo_c = await convert_temp(fahrenheit=tokyo['temp_f'])
{'paris': paris_c, 'tokyo': tokyo_c}
```
Both weather lookups run in parallel, the conversions run locally, and the whole thing collapses into one model round-trip instead of five.
## Selective tool sandboxing
By default, `CodeMode(tools='all')` sandboxes every tool. The `tools` field is a Pydantic AI `ToolSelector`, so you can control precisely which tools go through the sandbox. Tools that match the selector become callables inside `run_code`; non-matching tools stay visible to the model as regular tool calls.
```python
from pydantic_ai_harness import CodeMode
# By name -- only these tools are available inside run_code
CodeMode(tools=['search', 'fetch'])
# By predicate -- (ctx, tool_def) -> bool | Awaitable[bool]
CodeMode(tools=lambda ctx, td: td.name != 'dangerous_tool')
# By metadata -- combine with SetToolMetadata or a toolset's .with_metadata()
CodeMode(tools={'code_mode': True})
```
### Metadata-based selection
Use metadata when the decision should travel with a tool or toolset, rather than with one `CodeMode` instance. This suits shared toolsets: the toolset author tags the tools that are safe and useful to call from generated code, and each agent opts into that tag with `CodeMode(tools={...})`.
`CodeMode(tools={'code_mode': True})` uses the standard Pydantic AI [`ToolSelector`](/ai/api/pydantic-ai/tools/) metadata form. A tool is sandboxed when its `ToolDefinition.metadata` contains all of the selector's key-value pairs. Extra metadata on the tool is fine, and nested dictionaries are matched by deep inclusion.
The common pattern is to tag an entire toolset with `.with_metadata(...)`:
```python
from pydantic_ai import Agent
from pydantic_ai.toolsets import FunctionToolset
from pydantic_ai_harness import CodeMode
def search(query: str) -> str:
"""Search the web."""
return f'results for {query}'
def fetch(url: str) -> str:
"""Fetch a URL."""
return f'contents of {url}'
search_tools = FunctionToolset(tools=[search, fetch]).with_metadata(code_mode=True)
agent = Agent(
'anthropic:claude-sonnet-4-6',
toolsets=[search_tools],
capabilities=[CodeMode(tools={'code_mode': True})],
)
```
Here `search` and `fetch` are removed from the model-facing tool list and become callable functions inside `run_code`. Tools without `metadata['code_mode'] == True` stay visible as regular tool calls.
## Tool Search interaction
When you mark tools or whole toolsets `defer_loading=True` ([Tool Search](/ai/tools-toolsets/tools-advanced/#tool-search)), `CodeMode` keeps them out of `run_code` while they're undiscovered -- they pass straight through, so Tool Search drives them as usual (sent on the wire with `defer_loading` on providers with native tool search; otherwise dropped until discovered, with a `search_tools` tool alongside `run_code`). Once the model discovers a tool it comes back with `defer_loading=False`, and from then on `CodeMode` folds it into `run_code` like any other tool, so it's callable from generated code.
That fold-in grows `run_code`'s description, which invalidates the prompt-cache prefix once at the moment of discovery (turns with no discovery stay cache-warm). Two ways to avoid the bust:
- Pass `dynamic_catalog=True` to keep `run_code`'s description static across discoveries. The catalog of sandboxed-tool signatures moves into the agent instructions (as a dynamic [`InstructionPart`](/ai/api/pydantic-ai/messages/#pydantic_ai.messages.InstructionPart)) and newly-discovered tools are announced via [`ctx.enqueue`](/ai/api/pydantic-ai/tools/#pydantic_ai.tools.RunContext.enqueue) instead of by rebuilding the description:
```python
from pydantic_ai_harness import CodeMode
CodeMode(dynamic_catalog=True)
```
This pays off when paired with Tool Search: the tool-definitions block stays byte-stable so the prefix cache survives discoveries, at the cost of a larger (but cache-friendly) system prompt. With a fixed toolset and no Tool Search, the default keeps the system prompt shorter and is the better choice.
- To instead keep a Tool Search corpus fully native -- never folded into `run_code`, but not callable from inside it -- exclude it with a `tools` selector; corpus members carry `with_native` set to the managing native tool:
```python
from pydantic_ai_harness import CodeMode
CodeMode(tools=lambda ctx, td: td.with_native is None)
```
## Return values
The last expression in the snippet is automatically captured as the return value -- the model does not need to `print()`. Reserve `print()` for supplementary logging: printed text is surfaced separately, wrapped alongside the last-expression result.
| Scenario | Return |
|---|---|
| No print output | Last expression value |
| With print output | `{'output': '<printed text>', 'result': <last expression>}` |
| Multimodal content (e.g. images) | Returned natively for model processing |
## REPL state
State persists between `run_code` calls within the same agent run -- variables, imports, and function definitions carry over. Pass `restart: true` in the tool call to reset state.
## Observability
Nested tool calls inside `run_code` produce their own spans when instrumented with [Logfire](https://pydantic.dev/logfire) or any OpenTelemetry backend -- the easiest way to understand what code mode actually did, since each `run_code` span fans out into the tool calls the model issued from inside the sandbox. See the [Pydantic AI Logfire docs](/ai/integrations/logfire/) for setup.
The `run_code` tool return also carries metadata with every nested call, keyed by call id:
```python
from pydantic_ai import Agent
from pydantic_ai.messages import ToolReturnPart
from pydantic_ai_harness import CodeMode
agent = Agent('anthropic:claude-sonnet-4-6', capabilities=[CodeMode()])
@agent.tool_plain
def get_weather(city: str) -> dict:
"""Get current weather for a city."""
return {'city': city, 'temp_f': 72}
result = agent.run_sync("What's the weather in Paris?")
for msg in result.all_messages():
for part in msg.parts:
if isinstance(part, ToolReturnPart) and part.tool_name == 'run_code':
metadata = part.metadata or {}
tool_calls = metadata['tool_calls'] # dict[str, ToolCallPart]
tool_returns = metadata['tool_returns'] # dict[str, ToolReturnPart]
```
## In practice
A representative run wires `CodeMode` up against an MCP server and a web search and asks it to find the most-discussed Hacker News story across three feeds, pull the comment thread and the submitter's profile, and search the web for follow-up coverage. `CodeMode` collapses that into two `run_code` calls: the first fetches all three feeds in parallel via `asyncio.gather`, dedupes by id, filters by score, and ranks by comment count -- in plain Python; the second batches the three follow-up calls (`hn_get_thread`, `hn_get_user`, `duckduckgo_search`) together.
[![CodeMode's first run_code: parallel asyncio.gather over three HN feeds, then a dedupe and a score filter](images/code-mode-trace.png)](https://logfire-us.pydantic.dev/public-trace/84bcf123-2106-49da-9f6f-5c26395339bb?spanId=7650806a0785b946)
**[See the full Logfire trace ->](https://logfire-us.pydantic.dev/public-trace/84bcf123-2106-49da-9f6f-5c26395339bb?spanId=7650806a0785b946)** Each `run_code` span fans out into the tool calls the model issued from inside the sandbox.
## Filesystem and OS access
Sandboxed code runs with no access to the host's files, environment, or clock. Two parameters grant it access -- reach for them only when the agent's task genuinely needs the host.
Both parameters are fixed when the capability is built, so construct `CodeMode` per request to scope host access to that request.
### `mount` -- share host directories
Reach for `mount` when the agent works with real files: analyzing a dataset you've dropped in a folder and writing a report back, editing a checkout, or processing a batch of documents. Sandboxed `pathlib` code reads and writes under the mounted path. (For environment variables or the clock, use `os_access` instead.)
```python
from pydantic_ai import Agent
from pydantic_monty import MountDir
from pydantic_ai_harness import CodeMode
# The agent can read /work/data.csv and write /work/summary.md back to the host:
agent = Agent(
'anthropic:claude-sonnet-4-6',
capabilities=[CodeMode(mount=MountDir('/work', '/tmp/agent-workspace', mode='read-write'))],
)
```
A `MountDir` defaults to copy-on-write `mode='overlay'`: the sandbox reads host files and sees its own writes, but those writes do **not** reach the host. Pass `mode='read-write'` to persist them, or `mode='read-only'` to forbid writes. `mount` also accepts a list of `MountDir` for multiple mount points.
### `os_access` -- answer the sandbox's OS calls yourself
Reach for `os_access` when the agent needs environment variables, the current date and time, or filesystem behavior you control. Hand it a ready-made OS implementation (`AbstractOS`), or a callback that decides each call -- so you can inject just the secrets it needs, pin "now" for reproducible runs, or route file access to your own store.
```python
from pydantic_ai import Agent
from pydantic_monty import OSAccess
from pydantic_ai_harness import CodeMode
# Give the agent a fixed set of environment values:
agent = Agent(
'anthropic:claude-sonnet-4-6',
capabilities=[CodeMode(os_access=OSAccess(environ={'API_BASE': 'https://api.example.com'}))],
)
```
A callback receives each OS call and decides its fate:
```python
from pydantic_ai import Agent
from pydantic_monty import NOT_HANDLED
from pydantic_ai_harness import CodeMode
allowed_env = {'API_KEY': 'sk-...'}
def my_os(fn, args, kwargs):
if fn == 'os.getenv':
# Answer the call: allow-listed keys resolve, every other key reads back
# as None -- absent, exactly like a real unset variable.
return allowed_env.get(args[0])
# Refuse everything else: NOT_HANDLED makes the call fail in the sandbox.
return NOT_HANDLED
agent = Agent('anthropic:claude-sonnet-4-6', capabilities=[CodeMode(os_access=my_os)])
```
Your callback's return value decides the call's fate, and the two outcomes are easy to confuse:
- **Return any value** -- including `None`, `''`, or `0` -- and that becomes the result the sandbox sees. `os.getenv` returning `None` looks exactly like a normal unset variable, so the agent's code keeps running. This is how you *hide* something: answer with an empty value.
- **Return `NOT_HANDLED`** and the call is treated as unsupported: it raises inside the sandbox and the model gets a retry. This *refuses* a capability outright -- use it to block, not to say "no value". Returning `NOT_HANDLED` for a key the agent reasonably expects will burn retries.
!!! warning "Both expose the real host to model-written code"
`mount` and `os_access` hand model-generated Python real access to your filesystem and environment. Grant only what the task needs, and prefer constructing `CodeMode` per request so the granted access is scoped to that request.
!!! note "Monty-specific types"
These hooks use Monty's `AbstractOS`/`MountDir` types from `pydantic_monty`.
## Sandbox restrictions
Code runs inside [Monty](https://github.com/pydantic/monty), a sandboxed Python subset. Key restrictions:
- No class definitions.
- No third-party imports. Allowed stdlib modules: `sys`, `typing`, `asyncio`, `math`, `json`, `re`, `datetime`, `os`, `pathlib` (each must be imported before use).
- No wall-clock or timing primitives by default: `asyncio.sleep`, `datetime.datetime.now()`, `datetime.date.today()`, and the `time` module. `datetime.datetime.now()` / `datetime.date.today()` become available with an `os_access` handler (above); `asyncio.sleep` and `time` never do.
- No `import *`.
- Filesystem I/O needs an `os_access` handler or a `mount`; `os.getenv` / `os.environ` need an `os_access` handler.
- Tools requiring approval or with deferred (`CallDeferred`) execution are sandboxed like any other tool; without a `HandleDeferredToolCalls` (or equivalent) capability on the agent to resolve them inline, calling one from `run_code` raises an error that surfaces to the model as a retry.
## Agent spec (YAML/JSON)
`CodeMode` works with Pydantic AI's [agent spec](/ai/core-concepts/agent-spec/) feature for defining agents in YAML or JSON:
```yaml
# agent.yaml
model: anthropic:claude-sonnet-4-6
capabilities:
- CodeMode: {}
```
```python
from pydantic_ai import Agent
from pydantic_ai_harness import CodeMode
agent = Agent.from_file('agent.yaml', custom_capability_types=[CodeMode])
result = agent.run_sync('...')
print(result.output)
```
Pass `custom_capability_types` so the spec loader knows how to instantiate `CodeMode`. Arguments can be passed in the YAML too:
```yaml
capabilities:
- CodeMode:
tools: ['search', 'fetch']
max_retries: 5
```
## Further reading
- [Tool use via code](https://www.anthropic.com/engineering/code-execution-with-mcp) (Anthropic)
- [Code mode in production](https://blog.cloudflare.com/code-mode/) (Cloudflare)
- [Pydantic AI capabilities](/ai/core-concepts/capabilities/)
## API reference
::: pydantic_ai_harness.CodeMode
docs/compaction.md
+264
View File
@@ -0,0 +1,264 @@
---
title: Compaction
description: A menu of strategies -- clear, dedupe, trim, or summarize -- for keeping an agent's conversation history within the model's context window.
---
# Compaction
Compaction is a menu of strategies for keeping an agent's conversation history within a model's context window. Each strategy is a Pydantic AI `Capability` that edits the message history just before each request goes out. The edits **persist** into the run's message history, so a trim, clear, or summary carries forward to later steps -- it is not recomputed from the full history every turn.
All strategies preserve tool-call / tool-return **pairing**. Core does not validate this, and a provider rejects an orphaned pair, so the pairing guarantee is what makes these safe to drop into an agent. The zero-LLM strategies never call a model; only `SummarizingCompaction` (and `TieredCompaction` when it escalates that far) spends tokens.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/compaction/)
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
## The problem
An agent that runs for many turns accumulates history: tool outputs, file reads, model reasoning, repeated content. Left unchecked, that history outgrows the model's context window and the next request fails. Compaction keeps the history bounded, and the right strategy depends on where the bloat lives and how much you can afford to spend reclaiming it.
## The menu
| Capability | Cost | What it does | Reach for it when |
|---|---|---|---|
| `ClampOversizedMessages` | zero-LLM | Head/tail-truncates a single oversized part (response text, tool-call args) | One runaway generation blew past the context cap and no other strategy can reach it |
| `SlidingWindow` | zero-LLM | Drops the oldest whole messages down to a tail | You only need the recent turns and can discard old context entirely |
| `ClearToolResults` | zero-LLM | Blanks the content of old tool *results* in place, keeping the last `keep_pairs` | Tool outputs dominate context and can be re-fetched on demand (the cheap first tier) |
| `DeduplicateFileReads` | zero-LLM | Blanks every file read superseded by a newer read of the same file | The agent re-reads files and only the latest version matters |
| `SummarizingCompaction` | one LLM call | Summarizes older messages into a structured summary, keeping the recent tail | Old context still matters but must be compressed; use behind the cheap tiers |
| `TieredCompaction` | escalates | Runs cheap passes first, summarizes only if still over `target_tokens` | You want a sensible default: spend the expensive summary only when needed |
| `LimitWarner` | zero-LLM | Injects an URGENT/CRITICAL warning as limits approach | You want the agent to wrap up rather than have its history rewritten |
## Triggers
Every size-based strategy triggers on `max_messages` and/or `max_tokens` (estimated). Token counts use a ~4-chars-per-token heuristic by default; pass a `tokenizer` callable (for example `tiktoken`) for accuracy. `DeduplicateFileReads` runs on every request when no trigger is set (it is cheap and near-lossless). `TieredCompaction` triggers and stops on a single `target_tokens` budget. `ClampOversizedMessages` triggers per *part* (`max_part_tokens` / `max_part_chars`), not on the whole history -- the failure it targets is one oversized part, not a large total.
## The recommended default: `TieredCompaction`
The field consensus (Anthropic, OpenCode, Letta) is to clear and dedupe first, and summarize only when that is not enough. Summarization turns input tokens into output tokens, which are billed at a premium and generated serially, so it is genuinely expensive. The zero-LLM strategies touch only the cheaper input side.
`TieredCompaction` encodes that escalation: it runs each tier in order, re-measures the token count after each, and stops as soon as the conversation fits `target_tokens`. Order the tiers cheap-to-expensive so the expensive summarization tier is only reached when the cheap passes cannot reclaim enough.
```python
from pydantic_ai import Agent
from pydantic_ai_harness.compaction import (
ClearToolResults,
DeduplicateFileReads,
SummarizingCompaction,
TieredCompaction,
)
from pydantic_ai.messages import ToolCallPart
def my_file_key(call: ToolCallPart) -> str | None:
if call.tool_name != 'read_file':
return None
return call.args_as_dict().get('path')
agent = Agent(
'openai:gpt-4o',
capabilities=[
TieredCompaction(
tiers=[
DeduplicateFileReads(file_key=my_file_key),
ClearToolResults(max_tokens=1, keep_pairs=3),
SummarizingCompaction(max_messages=1, keep_messages=20), # model inherits the run's
],
target_tokens=120_000,
)
],
)
```
A tier inside `TieredCompaction` is driven directly by the orchestrator, which re-measures after each tier and stops once under `target_tokens`. A tier's own `max_*` trigger is therefore irrelevant when it runs inside `TieredCompaction` -- set it to anything valid (for example `ClearToolResults(max_tokens=1)`). Any object with `async def compact(messages, ctx) -> list[ModelMessage]` (the `CompactionStrategy` protocol) can be a tier, so you can plug in your own.
## `ClampOversizedMessages`: surviving a runaway generation
A single model response of repeated whitespace, or a single tool call with a giant payload, can produce one part so large the *next* request exceeds the provider's context cap. None of the other strategies can reach it: `SlidingWindow` drops the oldest messages but the offender is the newest; `ClearToolResults` only touches tool *results*; `LimitWarner` never edits history; and feeding the history to `SummarizingCompaction` hits the same cap.
`ClampOversizedMessages` truncates the offending part in place, keeping a head slice and a tail slice with a `[clamped: removed N of M characters]` marker between them. Degenerate generations are low-entropy repetition, so a head/tail slice loses little.
```python
from pydantic_ai import Agent
from pydantic_ai_harness.compaction import ClampOversizedMessages
agent = Agent(
'openai:gpt-4o',
capabilities=[
ClampOversizedMessages(max_part_tokens=50_000, keep_head_chars=2_000, keep_tail_chars=2_000)
],
)
```
A part is clamped only when it is oversized *and* the clamp actually shrinks it, so keep `keep_head_chars + keep_tail_chars` well below your per-part threshold.
It clamps two kinds of part inside each `ModelResponse`:
- **Response text** (`TextPart`) -- the critical case, a runaway model-response text part.
- **Tool-call args** (`ToolCallPart`), when `clamp_tool_call_args=True` (the default) -- the same failure shape for a giant payload (for example a runaway `write_plan`). The args are replaced with a small JSON object `{"_clamped": "<head>...<tail>"}` so they stay valid function arguments; the original call already executed, so this only shrinks the history copy. Set `clamp_tool_call_args=False` to clamp response text only.
Request-side parts (user prompts, tool *returns*, system prompts) are deliberately out of scope: user input should not be silently rewritten, and oversized tool returns are the job of `ClearToolResults`.
Use it as the first tier of `TieredCompaction`, before `ClearToolResults`:
```python
from pydantic_ai_harness.compaction import (
ClampOversizedMessages,
ClearToolResults,
TieredCompaction,
)
TieredCompaction(
tiers=[
ClampOversizedMessages(max_part_tokens=50_000),
ClearToolResults(max_tokens=1, keep_pairs=3),
],
target_tokens=120_000,
)
```
## `ClearToolResults`: the cheap first tier
Tool outputs typically dominate an agent's context, and the agent can usually re-run a tool if it needs the data again. `ClearToolResults` replaces the content of the oldest tool *results* with a short placeholder while keeping the most recent `keep_pairs` tool-call / tool-return pairs intact. The tool calls stay paired with their now-blanked results, so the history stays valid.
```python
from pydantic_ai import Agent
from pydantic_ai_harness.compaction import ClearToolResults
agent = Agent(
'openai:gpt-4o',
capabilities=[ClearToolResults(max_tokens=100_000, keep_pairs=3)],
)
```
Set `clear_tool_inputs=True` to also blank the arguments of the cleared calls, and `exclude_tools` to a set of tool names whose results are never cleared.
## `DeduplicateFileReads`: drop superseded reads
When the same file is read more than once, only the latest read keeps its content; earlier reads are blanked with a placeholder, with pairing preserved.
There is no default `file_key`: identifying a file read is agent-specific, and a wrong guess would drop live data. Supply a callable mapping a `ToolCallPart` to a stable file key, or `None` when the call is not a file read:
```python
from pydantic_ai import Agent
from pydantic_ai.messages import ToolCallPart
from pydantic_ai_harness.compaction import DeduplicateFileReads
def file_key(call: ToolCallPart) -> str | None:
if call.tool_name != 'read_file':
return None
return call.args_as_dict().get('path')
agent = Agent('openai:gpt-4o', capabilities=[DeduplicateFileReads(file_key=file_key)])
```
With no `max_messages` or `max_tokens` trigger set, `DeduplicateFileReads` runs on every request. It is cheap and near-lossless, so that default is usually what you want.
## `SlidingWindow`: keep only the recent tail
When the conversation exceeds the configured threshold, `SlidingWindow` discards the oldest whole messages down to a tail, preserving tool-call / tool-return pairs. Reach for it when you only need the recent turns and can discard old context entirely.
```python
from pydantic_ai import Agent
from pydantic_ai_harness.compaction import SlidingWindow
agent = Agent(
'openai:gpt-4o',
capabilities=[SlidingWindow(max_messages=80, keep_messages=40)],
)
```
By default `preserve_first_user_message=True` keeps the first user turn (in addition to system prompts) even when it falls outside the window, so the agent does not lose the original task. Pass `keep_tokens` instead of `keep_messages` to trim to a token budget rather than a message count.
## `SummarizingCompaction`: compress, do not discard
When old context still matters but must be compressed, `SummarizingCompaction` summarizes the older messages with a dedicated model call and replaces them with a single structured summary, preserving the recent tail and tool-call integrity. It is the expensive tier, so it is best used behind the cheaper passes (see `TieredCompaction`).
```python
from pydantic_ai import Agent
from pydantic_ai_harness.compaction import SummarizingCompaction
agent = Agent(
'openai:gpt-4o',
capabilities=[
SummarizingCompaction(
model='openai:gpt-4o-mini',
max_messages=60,
keep_messages=20,
)
],
)
```
`model` accepts a model name or a `Model`; when left `None` it inherits the running agent's model. No token caps are imposed on the summary call. By default `incremental=True` extends any existing summary from a prior compaction rather than regenerating it from scratch.
### Usage accounting
The summary call is a real request to the model, so its full usage -- tokens **and** the request itself -- is folded into the run's `ctx.usage`. This is deliberate: it keeps cost honest, keeps the request count consistent (a model request that did not count as one would be the surprise), and lets a `UsageLimits` request limit catch a runaway compaction. A run-request or iteration limiter will therefore see compaction calls among its requests.
## `LimitWarner`: warn instead of rewrite
`LimitWarner` never edits history. As the run approaches a configured limit, it injects an URGENT (then CRITICAL) warning as a trailing user turn, so the model wraps up rather than having its context rewritten under it. Models tend to pay more attention to user messages than system messages, which is why the warning is a user turn. Previous warnings from this capability are stripped before deciding whether to inject a new one.
```python
from pydantic_ai import Agent
from pydantic_ai_harness.compaction import LimitWarner
agent = Agent(
'openai:gpt-4o',
capabilities=[
LimitWarner(
max_iterations=40,
max_context_tokens=100_000,
)
],
)
```
Warnings begin at `warning_threshold` (default `0.7`, a fraction of the limit) and become CRITICAL for iterations once the remaining request count drops to `critical_remaining_iterations` (default `3`). It watches three kinds of limit -- `max_iterations`, `max_context_tokens`, and `max_total_tokens` -- and by default warns on whichever are configured; narrow that with `warn_on`.
## Cache tradeoff
Clearing, deduplicating, clamping, and summarizing all rewrite message content, which invalidates the provider's prompt cache from the edit point onward -- the next request pays a cache-write. For `ClearToolResults`, use `min_clear_tokens` to skip clearing that reclaims too little to be worth busting the cache. For `ClampOversizedMessages` the cache bust is unavoidable, because the alternative is a failed request.
## Tracing
When core instrumentation is active (the `Instrumentation` capability, `agent.instrument`, or `Agent.instrument_all()`), each strategy emits a `compact_messages` span on the run's tracer the moment it actually compacts -- that is, in `before_model_request`, once the strategy's threshold is exceeded (`ClampOversizedMessages` emits only when a part is actually clamped). `TieredCompaction` emits a single span for the whole escalation rather than one per tier, because it drives each tier's `compact` directly. Without instrumentation the tracer is a no-op, so the span adds no overhead.
The span name is the static `compact_messages`; the strategy is an attribute, not part of the name, to keep span cardinality low. Attributes:
| Attribute | Type | Meaning |
|---|---|---|
| `gen_ai.conversation.compacted` | bool | Always `true`; the OpenTelemetry GenAI convention's flag for a compacted context |
| `compaction.strategy` | str | Strategy class name (for example `SlidingWindow`, `SummarizingCompaction`) |
| `compaction.messages_before` | int | Message count before compaction |
| `compaction.messages_after` | int | Message count after compaction |
| `compaction.tokens_before` | int | Estimated token count before compaction |
| `compaction.tokens_after` | int | Estimated token count after compaction |
`gen_ai.conversation.compacted` is the GenAI semantic convention's flag; the rest is harness-specific. Token counts use the strategy's `tokenizer` when set, otherwise the ~4-chars-per-token heuristic. Raw message content is not recorded.
## Out of scope
These strategies compress or drop context *inside* the window. Moving large tool outputs *out* of the window -- overflowing them to a file the agent (or a subagent) can query on demand -- is a separate capability ([overflowing tool output](overflowing-tool-output.md)), not lossy truncation. Prefer it over capping individual tool outputs.
## API reference
The recommended default is `TieredCompaction`; the other strategies below can be used standalone or plugged in as its tiers.
::: pydantic_ai_harness.compaction.TieredCompaction
::: pydantic_ai_harness.compaction.ClampOversizedMessages
::: pydantic_ai_harness.compaction.ClearToolResults
::: pydantic_ai_harness.compaction.DeduplicateFileReads
::: pydantic_ai_harness.compaction.SlidingWindow
::: pydantic_ai_harness.compaction.SummarizingCompaction
::: pydantic_ai_harness.compaction.LimitWarner
docs/context.md
+117
View File
@@ -0,0 +1,117 @@
---
title: Context
description: Discover and load a repo's accumulated coding-assistant context engineering -- instruction files, skills, sub-agents, and hooks.
---
# Context
`RepoContext` discovers and loads a repo's accumulated coding-assistant context engineering (CE): the instruction files (`CLAUDE.md`/`AGENTS.md`) scattered across the tree and the assets under `.claude`/`.agents`/`.codex`/`.grok` (skills, sub-agents, hooks).
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/context/)
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
## The problem
A repo accumulates CE for whatever coding assistant worked in it: instruction files (`CLAUDE.md`/`AGENTS.md`) scattered across the tree, and assets under `.claude`/`.agents`/`.codex`/`.grok` (skills, sub-agents, hooks). An agent that loads only the top-level instruction file misses the ancestor context and has no idea the rest of the setup exists, so it can neither honor it nor translate it.
## The solution
`RepoContext` bundles three strategies, each independently toggleable. Construct it with `RepoContext(...)` in an `Agent`'s `capabilities`, anchored at the deepest directory the agent works in:
```python
from pathlib import Path
from pydantic_ai import Agent
from pydantic_ai_harness.context import RepoContext
agent = Agent(
'anthropic:claude-sonnet-4-6',
capabilities=[RepoContext(workspace_dir=Path('.'), home_dir=Path.home())],
)
result = agent.run_sync('Summarize the coding-assistant setup in this repo.')
print(result.output)
```
### 1. Walk-up instruction autoload (on by default)
Loads `CLAUDE.md`/`AGENTS.md` from `workspace_dir` and every ancestor up to `home_dir` (inclusive). Precedence is ancestor-first, workspace-last: broadest context first, most specific last. Files are deduped by resolved real path and by content hash, so a symlinked `AGENTS.md -> CLAUDE.md` or two ancestors sharing identical content load once.
When `home_dir` is `None` (the default), only `workspace_dir` is scanned -- no walk-up. Pass `home_dir=Path.home()` to walk up to your home directory.
### 2. Asset inventory (on by default)
Exposes one tool, `inventory_agent_context()`, that reports where the repo's CE assets live -- the `.claude`/`.agents`/`.codex`/`.grok` roots and, within each, the `skills/` (`SKILL.md`), `agents/` (`.md`), and `settings.json` (hooks) it contains. It returns a structured `AgentContextInventory`; it locates assets and does not parse them, leaving translation to the orchestrator.
Rename the tool with `inventory_tool_name`, or scope which roots it scans with `asset_roots`.
### 3. Nested-on-traversal (off by default)
When the model lists or reads a directory, surface that directory's `CLAUDE.md`/`AGENTS.md`. This couples to the host's list/read tools, so it is opt-in and configurable:
```python
from pathlib import Path
from pydantic_ai import Agent
from pydantic_ai_harness import FileSystem
from pydantic_ai_harness.context import RepoContext
agent = Agent(
'anthropic:claude-sonnet-4-6',
capabilities=[
FileSystem(root_dir='.'),
RepoContext(
workspace_dir=Path('.'),
nested_traversal=True,
traversal_tool_names=frozenset({'list_directory', 'read_file'}), # the FileSystem tool names to hook
traversal_path_arg='path', # the path arg key
nested_inject='pointer', # or 'contents'
)
],
)
```
`nested_inject='pointer'` (default) appends a one-line note pointing at the file; `'contents'` inlines the file body. Each directory is surfaced at most once per run.
## Cache cost
Injecting file contents into the system prompt costs prompt-cache stability: a changed prefix re-bills the whole cached region. `RepoContext` keeps the two cache-relevant paths separate:
- Strategy 1 reads its files once at run start and injects them as static system instructions, so the cached prefix stays byte-identical across turns.
- Strategy 3 is volatile (it depends on which directory was just touched), so its note is appended to the tool result in the message tail -- never to the system prompt -- and cannot invalidate the cached prefix.
## Configuration
```python
RepoContext(
workspace_dir, # Path -- the deepest dir the agent works in (required)
home_dir=None, # Path | None -- shallowest dir to stop walk-up at, inclusive
filenames=('CLAUDE.md', 'AGENTS.md'),
autoload_instructions=True, # Strategy 1
expose_inventory_tool=True, # Strategy 2
inventory_tool_name='inventory_agent_context',
nested_traversal=False, # Strategy 3
nested_inject='pointer', # 'pointer' | 'contents'
traversal_tool_names=frozenset({'list_directory', 'read_file'}),
traversal_path_arg='path',
asset_roots=('.claude', '.agents', '.codex', '.grok'),
)
```
## Scope
`RepoContext` locates and loads CE; it does not parse skill/sub-agent frontmatter or hook bodies, and it does not rewrite or translate assets. Strategy 1 reads its files once per run, so mid-run edits to those files are not reloaded.
## Further reading
- [Pydantic AI capabilities](/ai/core-concepts/capabilities/)
- [Pydantic AI hooks](/ai/core-concepts/hooks/)
## API reference
::: pydantic_ai_harness.context.RepoContext
::: pydantic_ai_harness.context.AgentContextInventory
::: pydantic_ai_harness.context.AssetRoot
docs/dynamic-workflow.md
+265
View File
@@ -0,0 +1,265 @@
---
title: Dynamic Workflow
description: Let an orchestrator agent coordinate a catalog of sub-agents by writing one sandboxed Python script -- fan-out, chaining, voting, and retry loops in a single tool call.
---
# Dynamic Workflow
`DynamicWorkflow` is for the case where the coordination *between* sub-agents is the actual work. Say you have a few specialists -- one reviews code, one summarizes findings, one writes the final note. Each is easy to call on its own; the hard part is the choreography: review three files at once, keep only the reports that found something, summarize those, and hand the summary to the writer. Reach for this capability when that orchestration involves fan-out, chaining, voting, or retry loops that you do not want to run one model turn at a time, with every intermediate result flowing back through the orchestrator's context.
!!! note "Import path"
Import this capability from its submodule -- there is no top-level `pydantic_ai_harness` re-export:
```python
from pydantic_ai_harness.dynamic_workflow import DynamicWorkflow
```
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
## The idea
The usual way to coordinate sub-agents is one tool call per step. The agent calls the reviewer and waits, reads the result, calls the reviewer again, waits again, and so on. Every intermediate result travels back into the agent's context, and every step that depends on the previous one is a separate model turn.
`DynamicWorkflow` takes a different route. You hand it a catalog of named sub-agents, and it gives the model a single tool, `run_workflow`. Inside that tool the model writes ordinary Python: each of your sub-agents is an `async` function it can call, loop over, and combine. The script runs to completion in one tool call, and only its final value comes back to the model. The choreography moves out of the conversation and into code.
If you have met [Code Mode](code-mode.md), this will feel familiar -- the same [Monty](https://github.com/pydantic/monty) sandbox and the same idea: write a script instead of many tool calls. The difference is what the script gets to call. In Code Mode it calls the agent's own tools; here it calls whole sub-agents.
## How this relates to Subagents
The harness has two delegation capabilities. They trade in the same currency -- named, isolated sub-agent runs -- but at different altitudes:
- [`SubAgents`](subagents.md) exposes one `delegate_task(agent_name, task)` tool. Each delegation is its own tool call and its own model turn. It is the right fit when delegations are occasional, or when each result needs the parent's judgment before the next one.
- `DynamicWorkflow` moves the choreography into a script. Fan-out, chaining, voting, and retry loops all run inside one tool call, and intermediate results never enter the parent's context.
Start with `SubAgents` if you are not sure. A `delegate_task` orchestrator converts to a workflow catalog without changing the sub-agents themselves.
## Installation
The script runs inside the Monty sandbox, so install the extra:
```bash
uv add "pydantic-ai-harness[dynamic-workflow]"
```
## Your first workflow
Two sub-agents, one orchestrator:
```python
from pydantic_ai import Agent
from pydantic_ai_harness.dynamic_workflow import DynamicWorkflow
reviewer = Agent('openai:gpt-5', name='reviewer', description='Reviews code for bugs.')
summarizer = Agent('openai:gpt-5', name='summarizer', description='Summarizes findings.')
orchestrator = Agent(
'openai:gpt-5',
capabilities=[DynamicWorkflow(agents=[reviewer, summarizer])],
)
```
`reviewer` and `summarizer` are plain agents -- the same `Agent` you already know. Their `name` becomes the function name the model calls in the script, so pick names that are valid Python identifiers. Their `description` tells the model what each one is for; write it the way you would document a function. `DynamicWorkflow(agents=[...])` bundles them into one capability and hands the orchestrator a single `run_workflow` tool.
## What the model does with it
When the orchestrator decides to use the tool, it does not call your sub-agents one at a time. It writes a script:
```python
import asyncio
reports = await asyncio.gather(
reviewer(task="Review auth.py for bugs:\n<file contents>"),
reviewer(task="Review parser.py for bugs:\n<file contents>"),
)
await summarizer(task="Summarize these review findings:\n" + "\n\n".join(reports))
```
The parts that matter:
- Each sub-agent is an `async` function. You call it with `await`.
- You pass the work as a single keyword argument, `task`. Always by keyword -- `reviewer(task="...")`, not `reviewer("...")`.
- `asyncio.gather(...)` runs the two reviews concurrently instead of one after the other.
- The last expression's value becomes the result the model sees. The intermediate `reports` list never leaves the sandbox.
Each call is a full `Agent.run`, with its own model loop, message history, tools, and typed output. Two things follow: calls are **isolated** (a sub-agent remembers nothing from an earlier call, so put everything it needs into `task`), and calls **cost tokens and take time** (which is why this capability gives you budgets, below).
## Sub-agents can return structured data
A sub-agent returns whatever its `output_type` produces. The default is a string, but give a sub-agent a Pydantic model and the script receives a `dict`:
```python
from pydantic import BaseModel
class Score(BaseModel):
value: int
reason: str
critic = Agent('openai:gpt-5', name='critic', description='Scores an answer 0-10.', output_type=Score)
```
Inside the script the model reads the fields by subscript, the way it would read a JSON object:
```python
result = await critic(task="Score this answer: ...")
result["value"] # not result.value
```
The catalog the model sees renders each output type as a `TypedDict`, so it knows the fields and reads them by subscript on its own.
## How results come back
The value of the script's last expression becomes the tool result -- the model does not `print()` it.
| The script... | The model receives |
| --- | --- |
| ends in a value, no print | that value directly (or `{}` if it is `None`) |
| prints and ends in a value | `{"output": "<printed text>", "result": <value>}` |
| prints and ends in `None` | `{"output": "<printed text>"}` |
`print()` is for debug logging; it stringifies, so let the last expression carry the real result.
## Choosing sub-agent models
By default each sub-agent uses the model it was constructed with. Set `inherit_model=True` when the host passes a per-run model override to the parent agent (for example from a `/model` command) and every sub-agent dispatch should follow that resolved parent model. Leave it `False` when a sub-agent is deliberately pinned to a different model.
## Keeping it safe: budgets
A sub-agent is non-deterministic, costs tokens, and can fan out into more sub-agents. `DynamicWorkflow` gives you a hard count ceiling, token budgets, and a guard against runaway sandbox scripts.
### `max_agent_calls` -- an exact count
```python
DynamicWorkflow(agents=[...], max_agent_calls=50) # 50 is the default
```
A hard, host-enforced ceiling on the number of sub-agent runs in one parent run. It is one budget shared across every `run_workflow` call in that run, and it holds exactly even when the script fans out with `asyncio.gather`. When the budget runs out, the workflow stops calling sub-agents and returns a terminal result that includes the sub-agent results that did complete, so nothing you already paid for is wasted. This is the only knob that bounds the number of runs exactly.
### `sub_agent_usage_limits` and `forward_usage` -- bounding cost
`sub_agent_usage_limits` is a `UsageLimits` applied to each sub-agent run. `forward_usage` controls whether the whole tree shares one usage counter:
| `forward_usage` | Counter | What the limit means |
| --- | --- | --- |
| `True` (default) | the parent's `usage` is shared across the tree | a tree-wide cap. Under concurrent fan-out it is best-effort: several sub-agents can pass the check before any of them adds to the count. |
| `False` | each sub-agent run counts on its own | per-run limits. A per-run `total_tokens_limit` of `T` with `max_agent_calls` of `N` bounds the tree to roughly `N * T` tokens. |
!!! warning "The parent `run()` usage limit is not forwarded"
The `usage_limits` you pass to the parent `run()` is not forwarded into sub-agents -- it is re-checked only at the parent's own request boundaries. To bound sub-agents, set `sub_agent_usage_limits`; for an exact ceiling on the number of runs, use `max_agent_calls`.
### `resource_limits` -- guarding the script itself
These limits guard the orchestration script's own memory and allocations, not the sub-agents it calls. The default backstop is 256 MB and 50 million allocations, with no time limit.
```python
DynamicWorkflow(agents=[...], resource_limits={'max_duration_secs': 30})
```
`max_duration_secs` measures the time your script spends running sandbox code, not wall-clock time. While the script waits on a sub-agent it is suspended and that time does not count, so the cap will not fire on a normal workflow no matter how long the sub-agents take. Its one job is catching a pure-CPU runaway -- a `while True:` loop that never awaits, which none of the sub-agent budgets can stop because it never calls a sub-agent. Pass `'unlimited'` to remove every limit, or a partial dict that merges onto the backstop so you override only the caps you name.
### Workflows do not nest
A sub-agent cannot start its own workflow; a nested `run_workflow` call returns a terminal error instead of running. The practical rule: do not give the sub-agents in your catalog the `DynamicWorkflow` capability. They are the leaves of the orchestration, not orchestrators.
## Renaming a sub-agent: `WorkflowAgent`
By default a sub-agent shows up under its own `name` and `description`. To give it a different name or description for one workflow without editing the agent itself, wrap it in a `WorkflowAgent`:
```python
from pydantic_ai_harness.dynamic_workflow import WorkflowAgent
DynamicWorkflow(
agents=[
WorkflowAgent(
reviewer,
name='check',
description='Checks one code change and returns actionable review findings.',
),
],
)
```
Now the model calls `check(task=...)`. Passing a bare agent is shorthand for wrapping it in a `WorkflowAgent` with no overrides.
## Adding sub-agents mid-run: `reveal()`
The catalog is fixed when a run starts, which keeps it in the prompt-cache prefix across turns. To make a new sub-agent available during a run (say once a fixer agent has been provisioned), keep a reference to the `DynamicWorkflow` instance and call `reveal()`:
```python
workflow = DynamicWorkflow(agents=[reviewer])
orchestrator = Agent('openai:gpt-5', deps_type=MyDeps, capabilities=[workflow])
# later, from the host or from another tool:
workflow.reveal(fixer)
```
The revealed sub-agent becomes callable on the next step; the model learns about it through a short announcement message that carries the new function's signature. The `run_workflow` description itself stays frozen at the agents present when the run started, so a runtime reveal never moves the prompt-cache prefix. `reveal()` is append-only and validates immediately -- a missing name, an invalid identifier, a reserved keyword, or a name collision raises `UserError` at the call site.
## Loading it only when needed: `defer_loading`
`DynamicWorkflow` carries a fair amount of instruction text, and most turns do not need it. Keep it collapsed to a one-line entry until the model actually loads it:
```python
DynamicWorkflow(
agents=[reviewer, summarizer],
id='workflow',
defer_loading=True,
)
```
`defer_loading=True` needs a stable `id`. See [on-demand capabilities](/ai/core-concepts/capabilities/#on-demand-capabilities) for the full picture.
## What runs in the sandbox
The script runs in Monty, a subset of Python. Knowing the edges matters:
- No class definitions, and no third-party libraries.
- Useful standard-library modules: `asyncio`, `math`, `json`, `re`, `typing`. Import what you use; other modules are unavailable or stubbed.
- No wall-clock or timing primitives -- no `asyncio.sleep`, no `datetime.now()`, no `time`.
- `asyncio.gather(...)` runs sub-agents concurrently but does not support `return_exceptions=True`.
Before a script runs it is statically type-checked against the sub-agent signatures. A misspelled function, a positional `task`, or a wrong-typed argument costs one retry, but no sub-agent budget and no sandbox execution.
!!! warning "Errors abort the whole script"
A sub-agent that raises cannot be caught inside the script -- one failure aborts the whole script and the model retries it. Write scripts where sub-agents do not depend on catching each other's errors. If a script fails after some sub-agents already finished, the retry prompt lists those completed results, so the model can reuse them as plain values instead of paying for the same calls again.
## Observability
The [Logfire](https://pydantic.dev/logfire) trace is the best way to see what a workflow did. Each sub-agent run appears nested under the `run_workflow` span, and the span carries the exact `code` argument the model wrote, so you can read the script it actually ran. Until first-class progress streaming ships, set `event_stream_handler` on each sub-agent `Agent` to watch sub-agent runs inside the one tool call.
## API
```python
DynamicWorkflow( # all parameters are keyword-only
agents=[...], # Sequence[AbstractAgent | WorkflowAgent], required
tool_name='run_workflow',
max_agent_calls=50,
max_retries=3,
forward_usage=True,
inherit_model=False, # True -> sub-agents run with the parent run's resolved model
sub_agent_usage_limits=None, # UsageLimits per sub-agent run; None -> pydantic-ai default
resource_limits=None, # None -> backstop (256 MB, 50M allocs, no time cap);
# 'unlimited' -> off; a dict is merged onto the backstop
id=None, # required when defer_loading=True
description=None, # one-line catalog entry shown while deferred
defer_loading=False,
)
workflow.reveal(agent) # AbstractAgent | WorkflowAgent; validates before appending
WorkflowAgent(
agent, # Agent, required, positional
name=None, # sandbox function name; falls back to agent.name
description=None, # function docstring; falls back to agent.description
)
```
`DynamicWorkflowToolset` and `WorkflowResourceLimits` are also exported from the module for advanced use.
Source: [`pydantic_ai_harness/dynamic_workflow/`](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/dynamic_workflow/).
## Further reading
- [Code Mode](code-mode.md) -- the same sandbox, calling the agent's own tools instead of sub-agents.
- [Subagents](subagents.md) -- one-delegation-per-tool-call sub-agents, without the scripted choreography.
- [Rewriting Bun in Rust](https://bun.com/blog/bun-in-rust) (Bun) -- the same pattern at scale, via Claude Code's dynamic workflows.
- [Capabilities](/ai/core-concepts/capabilities/) and [on-demand capabilities](/ai/core-concepts/capabilities/#on-demand-capabilities).
docs/filesystem.md
+188
View File
@@ -0,0 +1,188 @@
---
title: FileSystem
description: Give a Pydantic AI agent sandboxed, glob-filtered file access scoped to a single directory tree, with symlink-safe containment checks.
---
# FileSystem
`FileSystem` gives an agent a fixed set of file tools -- read, write, edit, list,
search, find, create, and inspect -- all scoped to a single `root_dir`. Every path is
resolved and containment-checked (symlinks included) before any I/O, and access
is filtered through allow / deny / protected glob patterns.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/filesystem/)
## The problem
Letting an agent touch the filesystem directly is risky: path traversal
(`../../etc/passwd`), symlinks that escape the project, clobbering `.git`, or
leaking `.env` secrets. Hand-rolling the guards around every tool call is
repetitive and easy to get subtly wrong.
`FileSystem` centralizes those guards. It exposes one bounded, sandboxed
toolset so you configure the boundary once and reuse it across agents.
## Usage
Add `FileSystem` to your agent's `capabilities` with a `root_dir`. Everything
the agent reads or writes is confined to that directory.
```python
from pydantic_ai import Agent
from pydantic_ai_harness import FileSystem
agent = Agent(
'anthropic:claude-sonnet-4-6',
capabilities=[FileSystem(root_dir='./workspace')],
)
result = agent.run_sync('Read config.toml and tell me the package name.')
print(result.output)
```
`root_dir` defaults to the current directory (`.`), but passing an explicit
workspace path is the recommended practice -- the sandbox is only as tight as
the root you give it.
## Tools
`FileSystem` contributes eight tools, all path-scoped to `root_dir`:
| Tool | Purpose |
|---|---|
| `read_file` | Read a text file with line numbers and a content hash. Binary files are detected and not dumped. Supports `offset`/`limit` paging. |
| `write_file` | Create or overwrite a file. Optional `expected_hash` rejects stale writes (optimistic concurrency). |
| `edit_file` | Exact-string replacement; `old_text` must match exactly once. Optional `expected_hash`. |
| `list_directory` | List a directory's entries with type indicators and sizes. |
| `search_files` | Regex search over file contents, optionally narrowed by an `include_glob`. |
| `find_files` | Glob search over file names (e.g. `*.py`, `**/*.json`). |
| `create_directory` | Create a directory and any missing parents. |
| `file_info` | Metadata for a file or directory (size, type, line count, hash, symlink target). |
Tool errors the model can correct -- a missing file, a denied path, a stale
edit -- are surfaced as
[`ModelRetry`](/ai/core-concepts/agent/#reflection-and-self-correction),
so the agent gets the error message back and can adjust rather than aborting
the run.
## Security model
- **Containment.** Paths resolve relative to `root_dir`; anything resolving
outside -- via `..`, an absolute path, or a symlink -- is rejected. Symlinks
are resolved with `os.path.realpath` *before* the containment check, closing
the TOCTTOU window.
- **Binary detection.** `read_file` returns a placeholder instead of dumping
binary bytes into the model context.
- **Optimistic concurrency.** `write_file`/`edit_file` accept an
`expected_hash` so an agent operating on a stale read is told to re-read
rather than silently overwriting newer content.
## Pattern filtering
Three independent glob lists control access. Patterns are matched with
`fnmatch`, whose `*` spans `/`, so `*.py` matches `src/main.py` and you rarely
need `**`.
| Field | Effect |
|---|---|
| `allowed_patterns` | If non-empty, only matching paths are accessible (allowlist). |
| `denied_patterns` | Matching paths are always rejected (denylist). |
| `protected_patterns` | Matching paths are read-only -- reads succeed, writes are rejected. |
`protected_patterns` defaults to `.git/*`, `.env`, `.env.*`, `*.pem`, `*.key`,
and `**/secrets*`. Pass an empty list to disable protection.
```python
from pydantic_ai import Agent
from pydantic_ai_harness import FileSystem
agent = Agent(
'anthropic:claude-sonnet-4-6',
capabilities=[
FileSystem(
root_dir='./workspace',
allowed_patterns=['*.py', '*.toml'],
denied_patterns=['**/node_modules/*'],
),
],
)
```
### Direct access vs. walkers
The three rules apply at two different granularities:
- **Direct access** (`read_file`, `write_file`, `edit_file`, `file_info`,
`create_directory`) gates the operation's target path. You must name a path
that the patterns permit.
- **Walkers** (`list_directory`, `search_files`, `find_files`) gate their root
by deny/protected patterns, but **not** by `allowed_patterns` -- a directory
root like `.` never matches a file pattern such as `src/*.py`, so requiring
it to would make every listing fail. Instead, the root is always walked and
each **entry** is filtered against all three lists. A directory listing can
never surface a path the agent couldn't otherwise read or write.
So with `allowed_patterns=['*.py']`, `list_directory('.')` succeeds and shows
only the `.py` entries; `read_file('notes.md')` is rejected.
Note that the walkers filter entries with write-level access, so
`protected_patterns` matches are omitted from `list_directory`, `search_files`,
and `find_files` output even though those exact paths remain directly readable
via `read_file`/`file_info`.
!!! note
Dotfiles and dot-directories (`.git`, `.env`, `.github`, ...) are skipped by
all three walkers -- `list_directory`, `search_files`, and `find_files` --
regardless of patterns.
## Configuration
```python
from pydantic_ai_harness import FileSystem
FileSystem(
root_dir='.', # str | Path -- sandbox root
allowed_patterns=[], # allowlist globs (empty = allow all)
denied_patterns=[], # denylist globs
protected_patterns=[...], # read-only globs (defaults to secrets/.git)
max_read_lines=2000, # cap for a single read_file
max_search_results=1000, # cap for search_files
max_find_results=1000, # cap for find_files
)
```
The three integer limits must be positive; they are validated at construction
and raise `ValueError` otherwise.
## Agent spec (YAML/JSON)
`FileSystem` works with Pydantic AI's
[agent spec](/ai/core-concepts/agent-spec/):
```yaml
model: anthropic:claude-sonnet-4-6
capabilities:
- FileSystem:
root_dir: ./workspace
allowed_patterns: ['*.py', '*.toml']
```
```python
from pydantic_ai import Agent
from pydantic_ai_harness import FileSystem
agent = Agent.from_file('agent.yaml', custom_capability_types=[FileSystem])
```
Pass `custom_capability_types` so the spec loader knows how to instantiate
`FileSystem`.
## Further reading
- [Pydantic AI capabilities](/ai/core-concepts/capabilities/)
- [Toolsets](/ai/tools-toolsets/toolsets/)
- [the capabilities overview](index.md)
## API reference
::: pydantic_ai_harness.FileSystem
docs/guardrails.md
+180
View File
@@ -0,0 +1,180 @@
---
title: Input & Output Guardrails
description: Validate the user prompt before it reaches the model and the model output before it reaches the caller, with allow/block/replace/retry verdicts and optional parallel execution.
---
# Input & Output Guardrails
Guardrails put a validation layer on the two edges of an agent run: the prompt on its way *in* to the model, and the output on its way *out* to the caller. Reach for them when unstructured input or output must be screened before it is acted on -- a prompt-injection attempt you never want to send, PII you must redact, an off-topic request you want to refuse cheaply, or an answer that must cite its sources before you show it. Without a guardrail the framework sends whatever the user typed and returns whatever the model produced, verbatim; a guardrail interposes a callable you control that gets the final say.
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
## The problem
Agents take unstructured input from users and return unstructured output to callers. On its own the framework does not reason about "this is unsafe to send" or "this is unsafe to show" -- a prompt-injection attempt reaches the model as-is, and any output the model produces is returned untouched. You need a place to inspect the value and decide what happens next.
## The solution
Two capabilities -- `InputGuard` and `OutputGuard` -- each wrap a `guard` callable you supply. The guard inspects a value (the prompt, or the output) and returns one of four outcomes:
| Outcome | `InputGuard` | `OutputGuard` |
|---|---|---|
| **allow** | send the prompt to the model | return the output to the caller |
| **block** | skip the model call; a refusal message becomes the response | raise `OutputBlocked` |
| **replace** | rewrite the prompt sent to the model (redaction) | substitute a sanitized output |
| **retry** | -- (not valid for input) | send the output back to the model to try again |
The asymmetry between input `block` and output `block` is deliberate. Blocking the input spends no tokens, so a graceful refusal is almost always right. Blocking the output means the model already produced something you do not want exposed, so raising forces the caller to decide what to do next.
Both `InputGuard`, `OutputGuard`, and their supporting types are top-level exports:
```python
from pydantic_ai import Agent
from pydantic_ai_harness import GuardResult, InputGuard, OutputGuard
def no_secrets(prompt: str) -> bool:
return 'api_key' not in prompt.lower()
def no_pii(output: object) -> GuardResult:
if 'SSN' in str(output):
return GuardResult.block('The response contained personal data.')
return GuardResult.allow()
agent = Agent(
'openai:gpt-5.4',
capabilities=[
InputGuard(guard=no_secrets),
OutputGuard(guard=no_pii),
],
)
```
A guard returns a bare `bool` (`True` = allow, `False` = block) for the simple case, or a `GuardResult` for the richer outcomes. Guards may also be async -- return an awaitable `bool`/`GuardResult`, for example to call a moderation API.
`OutputGuard` receives the output unchanged -- no automatic stringification. For a string output the guard reads it directly; for a typed (Pydantic model) output the guard gets the model instance, so pick the serialization that fits the check (read a field, or call `output.model_dump_json()` for JSON text). This avoids the trap of `str(MyModel(...))` producing a `MyModel(field=...)` repr that hides field contents from regex-based checks.
## `GuardResult`
Construct a `GuardResult` with its classmethods, not the raw fields:
```python
from pydantic_ai_harness import GuardResult
GuardResult.allow() # let the value through
GuardResult.block('reason') # refuse; `reason` is optional (a default is used otherwise)
GuardResult.replace(cleaned_value) # substitute a sanitized value and continue
GuardResult.retry('instruction') # OutputGuard only: ask the model to redo the output
```
The block/retry message is produced at the moment the guard decides, so it can carry the guard's own reasoning rather than a string frozen at construction time.
## Redaction (`replace`)
Return `GuardResult.replace(value)` to sanitize rather than refuse. `InputGuard` rewrites the prompt sent to the model; `OutputGuard` substitutes the output returned to the caller.
```python
def scrub_emails(text: str) -> GuardResult:
cleaned = EMAIL_RE.sub('[email]', text)
return GuardResult.replace(cleaned) if cleaned != text else GuardResult.allow()
agent = Agent(
'openai:gpt-5.4',
capabilities=[
InputGuard(guard=scrub_emails), # strip PII before it reaches the model
OutputGuard(guard=scrub_emails), # strip PII before it reaches the caller
],
)
```
Input redaction requires sequential mode -- it is incompatible with `parallel=True`, since a parallel guard runs alongside a model call that has already started with the original prompt.
## Retry (`retry`)
`OutputGuard` can send a bad output back to the model instead of blocking it. Return `GuardResult.retry(instruction)` -- the instruction is the retry prompt the model sees. This reuses pydantic-ai's normal retry machinery and counts against the run's output-retry budget.
```python
def must_cite_sources(output: object) -> GuardResult:
if not has_citations(output):
return GuardResult.retry('Include at least one source citation.')
return GuardResult.allow()
OutputGuard(guard=must_cite_sources)
```
## Accessing run context
A guard may take a `RunContext` as its first parameter when it needs run state -- `deps` for tenant- or role-aware policy, message history for conversation-aware checks. The parameter is detected from the signature, so prompt-only guards need not declare it:
```python
from pydantic_ai import RunContext
from pydantic_ai_harness import InputGuard
def tenant_policy(ctx: RunContext[MyDeps], prompt: str) -> bool:
return ctx.deps.tier == 'pro' or 'advanced-feature' not in prompt
InputGuard(guard=tenant_policy)
```
## Parallel input guards
A slow guard (an LLM classifier, a network call) run sequentially adds its latency to every turn. Set `parallel=True` to run the guard concurrently with the model call instead, overlapping the two so the guard adds no latency on the pass path. The model call is cancelled the moment the guard reports a violation.
```python
InputGuard(guard=slow_async_classifier, parallel=True)
```
Parallel mode trades tokens for latency: sequential mode never calls the model when the guard blocks, but parallel mode has already started the model call -- if the guard trips only after the model has responded, those tokens were spent. For fast local checks (regex, keyword lookup) sequential is the better default. `replace` is not available under `parallel=True`.
## Hard-fail path
`block` is the graceful path. To make the caller see an exception instead, raise from the guard:
```python
from pydantic_ai_harness import InputBlocked
def strict_guard(prompt: str) -> bool:
if contains_credentials(prompt):
raise InputBlocked('credentials detected')
return True
```
Any exception raised by the guard propagates as-is -- use `InputBlocked` / `OutputBlocked` from this module, or your own exception types.
## Streaming
`OutputGuard` inspects the **final** output only -- during `run_stream()` partial chunks reach the caller before the guard runs, so a `block` or `replace` verdict cannot un-send content already streamed. Use `run()` / `run_sync()` when the output must be screened before any of it is exposed. `GuardResult.retry()` is **not** supported under `run_stream()` and surfaces there as `UnexpectedModelBehavior`. `InputGuard` (including `parallel=True`) works the same in streamed and non-streamed runs.
## Tracing
`replace` and `block` are recorded as spans on the active OpenTelemetry tracer, so a redaction or refusal shows up in [Logfire](https://pydantic.dev/logfire) traces (`guardrail redacted input`, `guardrail blocked output`, and so on) with `guardrail.*` attributes. Content attributes -- the original/replacement values for a redaction and the refusal `message` for a block -- are attached **only** when `RunContext.trace_include_content` is enabled, since these can quote the very content the guard exists to keep out of traces.
`OutputGuard` positions its block/redact spans so they are always captured by an enclosing `Instrumentation` span regardless of capability order, while `InputGuard` runs innermost so any capability that morphs messages (a prompt rewriter, a context manager) runs first and the guard sees the final prompt the model will receive.
## Relationship to `pydantic-ai-shields`
[`pydantic-ai-shields`](https://github.com/vstorm-co/pydantic-ai-shields) provides opinionated implementations on top of these primitives (prompt-injection detectors, PII scrubbers, keyword blocklists). Use the guardrails here when you want to plug in your own validation logic; reach for shields when you need a batteries-included detector.
## API
```python
InputGuard(
guard, # Callable[..., bool | GuardResult | Awaitable[bool | GuardResult]]
parallel=False, # run concurrently with the model call
)
OutputGuard(
guard, # Callable[..., bool | GuardResult | Awaitable[bool | GuardResult]]
)
```
The guard callable takes the inspected value -- the prompt for `InputGuard`, the output for `OutputGuard` -- optionally preceded by a `RunContext`. `InputGuardFunc` and `OutputGuardFunc` are the exported signature aliases; `GuardrailError` is the base for `InputBlocked` and `OutputBlocked`.
Source: [`pydantic_ai_harness/guardrails/`](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/guardrails/).
docs/index.md
+150
View File
@@ -0,0 +1,150 @@
---
title: Pydantic AI Harness
description: The official capability library for Pydantic AI -- pick-and-choose batteries that turn your agent into a coding agent, research assistant, or anything else.
---
# Pydantic AI Harness
**The batteries for your [Pydantic AI](/ai/) agent.**
Pydantic AI's [capabilities](/ai/core-concepts/capabilities/) and [hooks](/ai/core-concepts/hooks/) API is how you give an agent its harness -- bundles of tools, lifecycle hooks, instructions, and model settings that extend what the agent can do without any framework changes.
**Pydantic AI Harness** is the official capability library for Pydantic AI, maintained by the [Pydantic AI](https://github.com/pydantic/pydantic-ai) team. Pydantic AI core ships the capabilities that require model or framework support, plus the ones fundamental to every agent -- [web search](/ai/core-concepts/capabilities/#provider-adaptive-tools), [tool search](/ai/tools-toolsets/deferred-tools/), [thinking](/ai/core-concepts/capabilities/#thinking). Everything else lives here: standalone building blocks you pick and choose to turn your agent into a coding agent, a research assistant, or anything else. This is also where new capabilities start -- as they stabilize and prove themselves broadly essential, they can graduate into core.
## What goes where?
Pydantic AI core ships the agent loop, model providers, the capabilities/hooks abstraction, and two kinds of capabilities:
- **Capabilities that require model or framework support** -- anything backed by provider native tools (like [image generation](/ai/core-concepts/capabilities/#provider-adaptive-tools)), provider-specific APIs (like compaction via the OpenAI or Anthropic APIs), or deep agent graph integration. These go hand-in-hand with model class code and need to ship together.
- **Capabilities that are fundamental to the agent experience** -- things nearly every agent benefits from, like [web search](/ai/core-concepts/capabilities/#provider-adaptive-tools), [tool search](/ai/tools-toolsets/deferred-tools/), and [thinking](/ai/core-concepts/capabilities/#thinking). These feel like qualities of the agent itself, not accessories.
**Pydantic AI Harness** is where everything else lives: standalone capabilities that make specific categories of agents powerful, or that are still finding their final shape. Context management, memory, guardrails, file system access, code execution, multi-agent orchestration -- these are the building blocks you pick and choose based on what your agent needs to do.
The harness is also where new capabilities *start*. It ships as a separate package so capabilities can iterate faster without the strict backward-compatibility requirements of core. As a capability stabilizes and proves itself broadly essential, it can graduate into core -- [code mode](code-mode.md) is an early candidate.
Many capabilities benefit from a "fall up" pattern: they typically start as a local implementation that works with every model, then gain provider-native support that uses the provider's built-in API when available -- auto-switching between the two. This is how [web search](/ai/core-concepts/capabilities/#provider-adaptive-tools), [web fetch](/ai/core-concepts/capabilities/#provider-adaptive-tools), and [image generation](/ai/core-concepts/capabilities/#provider-adaptive-tools) already work in core, and the same approach is coming for skills, code mode, and context compaction.
## Installation
```bash
uv add pydantic-ai-harness
```
Some capabilities need an extra to pull in their optional dependencies:
```bash
uv add "pydantic-ai-harness[codemode]" # Code Mode (adds the Monty sandbox)
uv add "pydantic-ai-harness[dynamic-workflow]" # Dynamic Workflow (adds the Monty sandbox)
uv add "pydantic-ai-harness[logfire]" # Managed Prompt (Logfire-managed prompts)
uv add "pydantic-ai-harness[acp]" # ACP (Agent Client Protocol SDK)
```
The `code-mode` extra is also supported as an alias for `codemode`.
Requires Python 3.10+ and `pydantic-ai-slim>=2.1.0`.
## Quick start
Install the harness alongside the Pydantic AI extras this example uses:
```bash
uv add "pydantic-ai-slim[anthropic,mcp,duckduckgo,logfire]" "pydantic-ai-harness[code-mode]"
```
```python
import logfire
from pydantic_ai import Agent
from pydantic_ai.capabilities import MCP, WebSearch
from pydantic_ai_harness import CodeMode
# See https://pydantic.dev/docs/ai/integrations/logfire/ for setup details.
logfire.configure()
logfire.instrument_pydantic_ai()
agent = Agent(
'anthropic:claude-opus-4-7',
capabilities=[
# Wraps every tool into a single run_code tool, sandboxed by Monty
# (https://github.com/pydantic/monty -- pulled in by the [code-mode] extra).
# The model writes Python that calls multiple tools with loops, conditionals,
# asyncio.gather, and local filtering -- one model round-trip for N tool calls.
CodeMode(),
# Connect to any MCP server -- here, the open-source Hacker News server
# (https://github.com/cyanheads/hn-mcp-server). native=False forces the
# local MCP toolset so CodeMode can wrap the tools; without it,
# providers that natively support MCP server connectors execute the tools
# server-side and bypass the sandbox.
MCP('https://hn.caseyjhand.com/mcp', native=False),
# Provider-adaptive web search; native=False routes through the local
# DuckDuckGo fallback (the [duckduckgo] extra above) so CodeMode can batch
# web searches alongside the HN calls in a single run_code.
WebSearch(native=False),
],
)
result = agent.run_sync(
"Across the top, best, and 'show HN' Hacker News feeds, find the most-discussed "
"story with at least 100 points. Pull its comment thread, its submitter's profile, "
"and any web coverage. Summarize what you find in one paragraph."
)
print(result.output)
"""
The most-discussed HN story across top/best/show clearing 100 points is "Vibe coding
and agentic engineering are getting closer than I'd like" by Simon Willison (748 points,
853 comments, on the Best feed), submitted by long-time HNer e12e. The piece argues
that the two modes Willison once kept mentally separate -- throwaway "vibe coding" and
disciplined "agentic engineering" -- are blurring, since agents like Claude Code now
reliably handle non-trivial tasks like "build a JSON API endpoint that runs a SQL query"
with tests and docs on the first pass. The HN thread is unusually substantive, with
commenters debating whether LLMs created or merely *exposed* sloppy engineering
practices and warning of a "normalization of deviance" as engineers stop reviewing diffs.
"""
```
[![Logfire trace from the Quick start run](images/quick-start-trace.png)](https://logfire-us.pydantic.dev/public-trace/84bcf123-2106-49da-9f6f-5c26395339bb?spanId=7650806a0785b946)
**[See this run as a public Logfire trace ->](https://logfire-us.pydantic.dev/public-trace/84bcf123-2106-49da-9f6f-5c26395339bb?spanId=7650806a0785b946)** Each `run_code` span fans out into the tool calls the model issued from inside the sandbox -- it's the easiest way to understand what code mode actually did.
## Capabilities
Each capability is a self-contained battery you drop into an agent's `capabilities=[...]` list. They compose with each other and with Pydantic AI's [built-in capabilities](/ai/core-concepts/capabilities/).
| Capability | What it does | Extra |
|---|---|---|
| [Code Mode](code-mode.md) | Wraps the agent's tools into a single `run_code` tool, sandboxed by [Monty](https://github.com/pydantic/monty). The model writes Python that calls the tools as functions -- with loops, conditionals, `asyncio.gather`, and local filtering -- collapsing N tool calls into one model round-trip. | `codemode` |
| [FileSystem](filesystem.md) | Sandboxed file access scoped to a root directory: read, write, edit, search, and find files. Rejects path traversal above the root, resolves symlinks before authorizing, and keeps `.git/`, `.env`, key files, and secrets read-only by default. | -- |
| [Shell](shell.md) | Command execution in a subprocess rooted at a working directory, gated by allowlists, denylists, timeouts, and optional environment-variable stripping (including a preset for common LLM provider credentials). | -- |
| [Context](context.md) | Auto-loads repo context -- `CLAUDE.md`/`AGENTS.md` and repository structure -- so the agent starts a run already oriented in the project. | -- |
| [Pydantic AI Docs](pydantic-ai-docs.md) | An on-demand `read_pyai_docs` tool that pulls Pydantic AI documentation into the run when the agent needs it, instead of preloading it. | -- |
| [Compaction](compaction.md) | Keeps a run within token limits: sliding-window trimming, LLM-powered summarization of older messages, and warnings before the context or iteration ceiling is hit. | -- |
| [Overflowing Tool Output](overflowing-tool-output.md) | Reduces an oversized tool return when it is produced -- truncate, spill to a queryable file, or summarize -- so a large payload does not persist in history and get re-sent every request. | -- |
| [Step Persistence](step-persistence.md) | Saves and restores full conversation state; snapshot, resume (`continue_run`), and fork (`fork_run`) a run. | -- |
| [Media](media.md) | Offloads large `BinaryContent` to content-addressed stores (local or S3) so big media does not bloat message history. | -- |
| [Subagents](subagents.md) | Delegates subtasks to specialized child agents through a delegate tool. | -- |
| [Dynamic Workflow](dynamic-workflow.md) | Orchestrates sub-agents from a model-written Python script -- fan-out, chaining, and voting in a single tool call. | `dynamic-workflow` |
| [Planning](planning.md) | Breaks a complex task into a structured plan before execution and tracks progress against it. | -- |
| [Memory](memory.md) | Gives an agent a persistent, namespaced notebook with bounded prompt injection, on-demand search, and concurrency-safe stores. | -- |
| [Runtime Authoring](runtime-authoring.md) | Lets an agent author, validate, and load real capabilities at runtime. | -- |
| [Guardrails](guardrails.md) | Validates user input before a run starts and model output after it completes -- block or redact, with structured results. | -- |
| [Managed Prompt](managed-prompt.md) | Backs an agent's instructions with a [Logfire-managed prompt](https://logfire.pydantic.dev/docs/reference/advanced/prompt-management/), so you can version, label, and roll out prompt changes from the Logfire UI without redeploying -- with a code default that keeps the agent working when no remote value is available. | `logfire` |
| [ACP](acp.md) *(experimental)* | Serves an agent to editors (Zed, etc.) over the [Agent Client Protocol](https://agentclientprotocol.com) -- streamed text, diff-rendered edits, and tool approval. | `acp` |
Most capabilities are stable within the [version policy](#version-policy) below. [ACP](acp.md) is the exception -- it is still experimental, imported from `pydantic_ai_harness.experimental.acp`, and may change or be removed in a future release.
## Build your own
[Capabilities](/ai/core-concepts/capabilities/#building-custom-capabilities) are the primary extension point for Pydantic AI. Any of the capabilities in this library can serve as a reference for building your own.
Publishing as a standalone package? Use the `pydantic-ai-<name>` naming convention -- see [Publishing capability packages](/ai/guides/extensibility/#publishing-capability-packages).
## Version policy
Pydantic AI Harness uses **0.x versioning** to signal that APIs are still stabilizing. During 0.x, minor releases (0.1 -> 0.2) may include breaking changes -- renamed parameters, changed defaults, restructured APIs -- while patch releases (0.1.0 -> 0.1.1) will not intentionally break existing behavior. All breaking changes are documented in release notes with migration guidance. This is why the harness is a separate package from [Pydantic AI](https://github.com/pydantic/pydantic-ai), which has a [stricter version policy](/ai/project/version-policy/). As the core capabilities stabilize, the library will move toward 1.0 with matching stability guarantees.
## Pydantic AI references
- [Capabilities](/ai/core-concepts/capabilities/) -- what capabilities are, built-in capabilities, building your own
- [Hooks](/ai/core-concepts/hooks/) -- lifecycle hooks reference, ordering, error handling
- [Extensibility](/ai/guides/extensibility/) -- publishing packages, third-party ecosystem
- [Toolsets](/ai/tools-toolsets/toolsets/) -- building tools for capabilities
- [API reference](/ai/api/pydantic-ai/capabilities/) -- full API docs
docs/managed-prompt.md
+227
View File
@@ -0,0 +1,227 @@
---
title: Managed Prompt
description: Back a Pydantic AI agent's instructions with a Logfire-managed prompt so you can version, label, and roll it out without redeploying.
---
# Managed Prompt
`ManagedPrompt` backs an agent's instructions with a
[Logfire-managed prompt](https://logfire.pydantic.dev/docs/reference/advanced/prompt-management/),
so you can iterate on your system prompt from the Logfire UI -- versioned, labelled, and rolled
out -- without touching code or redeploying. It's a Pydantic AI [capability](index.md), so you
wire it in through the `capabilities=` parameter on `Agent`.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/logfire/)
Install the `logfire` extra:
```bash
uv add "pydantic-ai-harness[logfire]"
```
!!! note "A first-party `Managed` capability is in flight"
A broader, first-party `Managed` capability is being built in
[pydantic-ai#5107](https://github.com/pydantic/pydantic-ai/pull/5107) and will eventually be
importable as `pydantic_ai.managed.logfire.Managed` -- covering instructions, model settings,
and whole-spec variables. Until then, `ManagedPrompt` is the supported path for backing an
agent's instructions with a Logfire-managed prompt.
## The problem it solves
Prompts are critical to agent behavior, but iterating on them through the normal
edit -> review -> deploy loop is slow. You can't easily A/B test a change, and you can't roll it
back the moment it misbehaves in production without shipping a new build.
`ManagedPrompt` moves the prompt out of your codebase and into Logfire's managed-variable store.
It declares the backing managed variable for you and resolves it **once per run**, feeding the
resolved value into the agent's instructions. Resolution happens inside the run's
[`wrap_run`](/ai/api/pydantic-ai/capabilities/#pydantic_ai.capabilities.AbstractCapability.wrap_run)
hook, using the
[`ResolvedVariable`](https://logfire.pydantic.dev/docs/reference/advanced/managed-variables/) as a
context manager that stays open for the whole run -- so the selected label and version are attached
as baggage to every child span of the agent run. You get a direct correlation between a run's
behavior and the exact prompt version that produced it, plus instant iteration and rollback from
the Logfire UI.
## Usage
Pass the prompt name and a default value. The name `support_agent` is declared as the managed
variable `prompt__support_agent` -- the naming Logfire's Prompt management uses (hyphens in a name
become underscores). The `default` keeps the agent working until a remote value is published, so
your code always runs even before you create the prompt in Logfire.
```python
import logfire
from pydantic_ai import Agent
from pydantic_ai_harness.logfire import ManagedPrompt
logfire.configure()
agent = Agent(
'openai:gpt-5',
capabilities=[
ManagedPrompt(
'support_agent',
default='You are a helpful customer support agent. Be friendly and concise.',
label='production',
)
],
)
result = agent.run_sync('My order never arrived.')
print(result.output)
```
Pinning `label='production'` is the recommended default: the resolved value only changes on a
deliberate prompt rollout, which keeps the provider prompt cache hot (see
[Prompt-cache trade-off](#prompt-cache-trade-off) below).
## Targeting
For deterministic A/B assignment (the same user always sees the same label), pass a
`targeting_key`. It can be a static string or a callable that derives the key from the
[`RunContext`](/ai/api/pydantic-ai/tools/#pydantic_ai.tools.RunContext) -- handy when the
key lives in your agent's `deps`:
```python
from dataclasses import dataclass
from pydantic_ai import Agent
from pydantic_ai_harness.logfire import ManagedPrompt
@dataclass
class Deps:
user_id: str
agent = Agent(
'openai:gpt-5',
deps_type=Deps,
capabilities=[
ManagedPrompt(
'support_agent',
default='You are a helpful customer support agent.',
targeting_key=lambda ctx: ctx.deps.user_id,
),
],
)
```
Pass `attributes` (a mapping, or a callable returning one) for condition-based targeting rules.
When `label` is omitted, the variable's rollout and targeting rules pick the label. When both
`targeting_key` and `attributes` are omitted, Logfire falls back to its own targeting context and
then to the active trace id.
For Logfire-side targeting that lives outside the agent (e.g. set once per request handler), use
Logfire's
[`targeting_context`](https://logfire.pydantic.dev/docs/reference/advanced/managed-variables/) in
an outer scope; `ManagedPrompt` only needs `targeting_key` / `attributes` when the key comes from
the agent's `RunContext`.
## Templating with deps
By default the resolved prompt is used verbatim. Pass `render_template=True` to render it as a
Handlebars template against the agent's `deps` -- the same mechanism as
[`TemplateStr`](/ai/api/pydantic-ai/agent/) -- so `{{field}}` is filled
from `deps`:
```python
from dataclasses import dataclass
from pydantic_ai import Agent
from pydantic_ai_harness.logfire import ManagedPrompt
@dataclass
class Deps:
customer_name: str
agent = Agent(
'openai:gpt-5',
deps_type=Deps,
capabilities=[
ManagedPrompt(
'support_agent',
default='You are helping {{customer_name}}. Be friendly and concise.',
render_template=True,
),
],
)
```
Rendering requires `pydantic-handlebars` (install `pydantic-ai-slim[spec]`). It is off by default.
## Prompt-cache trade-off
The resolved value lands in the agent's **system instructions**. Provider prompt caches (Anthropic,
OpenAI, etc.) key strictly by prefix -- `tools -> system -> messages` -- so any change to the system
block invalidates the cached prefix for the affected runs.
| Mode | Cache impact |
| --- | --- |
| Pinned `label='production'`, no rollout split | **Cache-stable.** The value only changes on a deliberate prompt rollout, which is the same cost as a redeploy. |
| Percentage rollout across labels (no `label=`) | Different runs land on different labels -> splits the cache into one lane per label. |
| `targeting_key` per user/tenant with multiple labels in play | Cache lanes per assigned label; deterministic per key but still N lanes overall. |
| Mid-traffic label flip in the Logfire UI | One-shot cold-invalidation for everyone on that label. |
In short: pinning a `label` keeps the cache hot; using `ManagedPrompt` as an A/B platform is opt-in
cache cost. If you don't need rollouts, `label='production'` is the recommended default.
## Bringing your own variable
Declaring the same name more than once is fine -- each `ManagedPrompt` builds its own backing
variable, so sharing a prompt across several agents just works. Pass an existing
[`logfire.variables.Variable`](https://logfire.pydantic.dev/docs/reference/advanced/managed-variables/)
as the first argument instead of a name when you want to declare the variable yourself -- for
example a template variable, or one registered for `variables_push`:
```python
import logfire
from pydantic_ai import Agent
from pydantic_ai_harness.logfire import ManagedPrompt
logfire.configure()
support_prompt = logfire.var(
name='prompt__support_agent',
type=str,
default='You are a helpful customer support agent. Be friendly and concise.',
)
agent = Agent('openai:gpt-5', capabilities=[ManagedPrompt(support_prompt, label='production')])
```
When `name` is a prompt name (not a `Variable`), pass `logfire_instance=` to declare the variable
on a specific Logfire instance instead of the module-level default. `default` is required when
`name` is a prompt name and is ignored when you pass a `Variable` (which already carries its own
default and instance).
## How it composes
- **Resolves once per run.** A label flip or rollout change that lands in Logfire mid-run is not
picked up until the next run starts -- the trade-off for run-stable instructions and a single
baggage scope across all child spans.
- **Runs outermost.** The capability wraps
[`Instrumentation`](/ai/api/pydantic-ai/capabilities/#pydantic_ai.capabilities.Instrumentation)
so the resolved variable's baggage covers the agent run span as well as its children. On recent
Logfire versions both the selected label and the version are propagated as separate baggage
attributes.
- **Concurrency-safe.** Resolution is isolated per run via a context variable, so a single
capability instance is safe to share across concurrent runs.
- **Inspectable mid-run.** `ManagedPrompt.resolved` exposes the active run's `ResolvedVariable`
(`value`, `label`, `version`, `reason`) for inspection -- e.g. from inside a tool. It is `None`
outside a run.
## API reference
The resolved prompt is a `str`. Pass the bare prompt name (the `prompt__` prefix and
hyphen-to-underscore normalization are applied for you) and a `default`, then use `label`,
`targeting_key`, `attributes`, `render_template`, and `logfire_instance` to control resolution.
::: pydantic_ai_harness.ManagedPrompt
docs/media.md
+151
View File
@@ -0,0 +1,151 @@
---
title: Media Externalization
description: Content-addressed stores and walker helpers that move large BinaryContent payloads out of message history into deduplicated storage and put them back on demand.
---
# Media Externalization
A conversation that carries images, audio, or other `BinaryContent` inlines those bytes into every message. Persist that history and each snapshot re-serializes the payloads; the same image referenced by ten messages is ten copies of the bytes. Media externalization solves that: content-addressed stores write each payload once, keyed by its own hash, and leave a short `media+sha256://` URI in its place. Reach for it whenever binary payloads would otherwise balloon what you store or send.
!!! note "Import path"
Import these helpers from their submodule -- there is no top-level `pydantic_ai_harness` re-export:
```python
from pydantic_ai_harness.media import (
DiskMediaStore,
S3MediaStore,
SqliteMediaStore,
externalize_media,
restore_media,
)
```
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
## Building blocks, not a capability
These are building blocks. There is no class you add to `Agent(capabilities=[...])` yet. [`StepPersistence`](step-persistence.md) already uses them to keep snapshots small when messages carry `BinaryContent`, and a forthcoming `MediaExternalizer` capability will reuse the same stores to rewrite `BinaryContent` into URL parts before the model sees them.
## Why content-addressing
The URI is derived from the payload hash, so identical bytes deduplicate automatically. The same bytes are stored once no matter how many messages or snapshots reference them, and moving the underlying storage is a one-line swap because the URI does not change.
## Stores
Every store implements the `MediaStore` protocol -- `put`, `get`, `exists`, `public_url`, and `get_metadata`, all async and content-addressed.
| Store | Backed by | Use when |
|---|---|---|
| `DiskMediaStore(directory=...)` | A directory on disk | Local runs and tests |
| `SqliteMediaStore(database=...)` | A SQLite database | A single-file store that travels with the data |
| `S3MediaStore(bucket=, endpoint=, region=, ...)` | S3 or an S3-compatible bucket | Shared or production storage |
`S3MediaStore` uses path-style URLs plus handrolled SigV4, so it is compatible with AWS S3, Cloudflare R2 (`region='auto'`), MinIO, and other S3-compatible providers. `SqliteMediaStore` also accepts `connection=` instead of `database=` to share a `sqlite3.Connection`.
## Walker helpers
`externalize_media` and `restore_media` walk a message node and swap payloads for URIs and back:
```python
from pydantic_ai_harness.media import DiskMediaStore, externalize_media, restore_media
store = DiskMediaStore(directory='./media')
# Replace BinaryContent larger than the threshold with media+sha256:// URIs.
lean = await externalize_media(message, media_store=store, threshold_bytes=32_000)
# Later, rehydrate the URIs back into BinaryContent.
full = await restore_media(lean, media_store=store)
```
`externalize_media` only externalizes payloads over `threshold_bytes`; smaller ones stay inline. Round-trip is transparent -- `restore_media` returns `BinaryContent` with the original bytes. If you need to key media yourself, `media_uri_for` and `parse_media_uri` give you the raw URI round-trip.
## Public URLs
When a store is fronted by a CDN, a local HTTP server, or a signed-URL service, pass a `public_url=` resolver (or use `make_static_public_url`) to turn a stored `media+sha256://` URI into a URL the model can fetch directly. Without a resolver, `public_url(...)` returns `None`.
A static base URL, for a public bucket or CDN:
```python
from pydantic_ai_harness.media import S3MediaStore, make_static_public_url
store = S3MediaStore(
bucket='my-bucket',
endpoint='https://<acc>.r2.cloudflarestorage.com',
region='auto',
access_key_id=..., secret_access_key=...,
key_prefix='media/',
public_url=make_static_public_url('https://pub-abc.r2.dev', key_prefix='media/'),
)
```
A presigned or rotating-signature URL -- pass any async callable that takes `(uri, MediaContext)`:
```python
from pydantic_ai_harness.media import MediaContext, S3MediaStore
async def presign(uri: str, ctx: MediaContext) -> str:
key = 'media/' + uri.removeprefix('media+sha256://') + '.bin'
return await my_signer.generate(key, ttl=3600, content_type=ctx.media_type)
store = S3MediaStore(..., public_url=presign)
```
This is what the forthcoming `MediaExternalizer` will use to swap `BinaryContent` parts for `ImageUrl` / `AudioUrl` / other URL parts before the model sees the message, letting providers fetch big media over the wire without re-encoding bytes into the request body. Emitting a URL is always safe: pydantic-ai providers transparently download the bytes when the target model does not natively accept that URL type, so you only ever lose wire savings, never correctness.
## `MediaContext`
Every store method and both user-supplied callables (`PublicUrlResolver`, `KeyStrategy`) accept a `MediaContext` -- an extensible per-operation bag:
```python
from collections.abc import Mapping
from dataclasses import dataclass, field
@dataclass(frozen=True, kw_only=True)
class MediaContext:
media_type: str | None = None # e.g. 'image/png'
filename: str | None = None # original filename, when known
metadata: Mapping[str, str] = field(default_factory=dict) # user-supplied tags
```
All fields default, so you pass what you have and ignore the rest; new fields are added non-breakingly as use cases emerge. `get_metadata(uri)` round-trips the user-supplied `metadata` mapping on all three stores; `media_type` is persisted separately (as the byte payload's `Content-Type`).
## `KeyStrategy`
The default on-store key layout is `<sha256>.bin`. `DiskMediaStore` and `S3MediaStore` accept a `key_strategy=` override to fit an existing layout; `SqliteMediaStore` does not, since its primary key is the digest:
```python
from pydantic_ai_harness.media import DiskMediaStore, MediaContext
def by_media_type(uri: str, ctx: MediaContext) -> str:
digest = uri.removeprefix('media+sha256://')
ext = {'image/png': '.png', 'image/jpeg': '.jpg'}.get(ctx.media_type or '', '.bin')
return f'images/{digest}{ext}'
store = DiskMediaStore('runs', key_strategy=by_media_type)
```
If your strategy depends on `ctx.media_type`, the same context must be supplied at read time for `get`/`exists` to find the blob. `DiskMediaStore` rejects strategies that produce absolute paths or `..` segments, to keep writes inside the store directory. `default_key_strategy` is exported if you want to build on it.
## API
| Symbol | Purpose |
|---|---|
| `MediaStore` | Async content-addressed store protocol (`put` / `get` / `exists` / `public_url` / `get_metadata`) |
| `DiskMediaStore`, `SqliteMediaStore`, `S3MediaStore` | Concrete stores |
| `MediaContext` | Per-operation context (media type, filename, tags) threaded through store operations |
| `KeyStrategy`, `default_key_strategy` | On-store key layout |
| `PublicUrlResolver`, `make_static_public_url` | Resolve a stored URI to a public URL |
| `externalize_media`, `restore_media` | Walk a message node to externalize / rehydrate payloads |
| `media_uri_for`, `parse_media_uri` | Compute and parse a `media+sha256://` URI |
Source: [`pydantic_ai_harness/media/`](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/media/).
## Related
- [Step Persistence](step-persistence.md) -- the first consumer of these stores, externalizing `BinaryContent` in run snapshots.
docs/memory.md
+227
View File
@@ -0,0 +1,227 @@
---
title: Memory
description: Persistent, namespaced agent notebooks with bounded prompt injection, on-demand search, and concurrency-safe stores.
---
# Memory
Give an agent a persistent notebook that it can update, search, and reuse across runs without loading every stored file into every prompt.
> [!NOTE]
> Import this capability from its submodule. It is not re-exported from `pydantic_ai_harness`:
>
> ```python
> from pydantic_ai_harness.memory import Memory
> ```
Memory is a released, non-experimental capability. Pydantic AI Harness is still on 0.x releases, so the API may change between minor releases. See the [version policy](index.md#version-policy).
## Notebook model
Memory gives each agent a notebook made of Markdown files:
- `MEMORY.md` is the main notebook. By default, a bounded excerpt and the names of other files are added to the current request as delimited user-role context.
- Other files hold longer or focused notes. The model reads them on demand or finds them with bounded text search.
The model gets four tools:
| Tool | Purpose |
| --- | --- |
| `write_memory` | Append to a file or replace one unique text fragment. Writes use optimistic concurrency and an idempotency identifier derived from the run and tool call. |
| `read_memory` | Read a bounded prefix of one memory file. |
| `delete_memory` | Delete a file. The main notebook is protected. |
| `search_memory` | Search across notebook files, subject to configured result, character, and file-scan limits. |
```python
from pydantic_ai import Agent
from pydantic_ai_harness.memory import FileStore, Memory
agent = Agent(
'anthropic:claude-sonnet-4-6',
capabilities=[Memory(FileStore('.agent-memory'))],
defer_model_check=True,
)
```
The namespace is resolved by application code, not supplied to the tools. The model therefore cannot select another user's namespace in a tool call.
## Injection modes and limits
Automatic injection is enabled by default. Trusted usage guidance remains in model instructions, while model-written memory is enclosed in `<memory>` delimiters in a user-role part on the current request. Together, the guidance, main notebook, and file listing share a finite `max_tokens` budget, estimated at four characters per token. The default is 2,000 approximate tokens. `max_lines` is an additional limit on the main notebook. Backend reads are limited by `max_memory_size`, and the number of requested paths is derived from the prompt budget, so the capability never requests an unbounded file or listing. Content that does not fit is omitted with a prompt directing the model to use `read_memory` or `search_memory`.
```python
from pydantic_ai_harness.memory import FileStore, Memory
memory = Memory(
FileStore('.agent-memory'),
max_tokens=2_000,
max_lines=200,
)
```
Only the current request retains the injected user-role part, so copies do not accumulate in message history. Each model request receives the latest bounded snapshot, including after `write_memory` or an external update changes `MEMORY.md`.
Set `inject_memory=False` for cache-stable prompts or durable workflows. The tools remain available, and the model can fetch memory only when it needs it:
```python
from pydantic_ai_harness.memory import FileStore, Memory
memory = Memory(FileStore('.agent-memory'), inject_memory=False)
```
With `injection_errors='ignore'` (the default), a store failure skips automatic injection and emits content-safe telemetry. Spans record the backend type and a hash of the resolved scope; successful injection records counts, and failures record the exception type. They do not record memory content. Set `injection_errors='raise'` when a run must fail rather than proceed without injected memory. Namespace and store resolver failures always propagate. Tool failures are still returned as tool errors; this setting controls automatic injection only.
## Persistence and concurrency
The store contract includes optimistic compare-and-swap mutations and idempotency. A write based on a stale revision fails with a conflict instead of overwriting a concurrent edit. Replaying the same run and tool call does not apply its mutation twice; reusing that operation identifier with different arguments raises `MemoryOperationConflictError`. These guarantees belong to the mutation operation, so custom stores must implement them atomically rather than composing separate read and write calls.
Every `MemoryStore.read` call includes a finite `max_chars`, and every `list_paths` call includes a finite `limit`. A store returns the bounded prefix plus `MemoryFile.truncated=True` when more content exists, while its version still represents the complete file. `read_memory` marks that bounded result as truncated. `write_memory` refuses to append or edit an oversized externally supplied file because doing so would derive new content from a partial read; remediate or replace it through the backing store first. A custom `SearchableMemoryStore.search` must likewise honor `max_file_chars` as well as the result limits.
| Store | Persistence and concurrency boundary |
| --- | --- |
| `InMemoryStore()` | Process lifetime; atomic across tasks using that store instance. |
| `FileStore(directory)` | Local filesystem; atomic Markdown replacement plus a hidden SQLite journal provide recovery, cross-process compare-and-swap, and durable idempotency receipts. |
| `SqliteMemoryStore(database=...)` | Durable single-host storage; compare-and-swap and idempotency are enforced in database transactions. |
| `PostgresMemoryStore(pool)` | Durable shared storage; compare-and-swap and idempotency are enforced in database transactions. The caller owns the pool lifecycle. |
```python
from pydantic_ai_harness.memory import FileStore, Memory, SqliteMemoryStore
local_memory = Memory(FileStore('.agent-memory'))
sqlite_memory = Memory(SqliteMemoryStore(database='.agent-memory.db'))
```
`SqliteMemoryStore` can instead use a caller-owned `sqlite3.Connection`. Because operations run off the event loop, create that connection with `check_same_thread=False` and manage its lifecycle in the application. The connection must be dedicated to the store and idle at the start of every operation; a call fails rather than commit or roll back an active caller transaction.
`FileStore` keeps the journal at `.memory-store.sqlite3` inside its root. Keep it with the Markdown files when copying or backing up the store. Editing a Markdown file outside the capability changes its content version and can produce a conflict with a prepared operation; the journal recovers operations interrupted between transaction preparation and filesystem replacement.
`PostgresMemoryStore` accepts the driver-neutral `PostgresPool` protocol, so the harness does not require a particular PostgreSQL driver. Install and manage the driver in your application (for example, `uv add asyncpg`):
```python
import asyncpg
from pydantic_ai_harness.memory import Memory, PostgresMemoryStore
async def build_memory() -> tuple[Memory[None], asyncpg.Pool]:
pool = await asyncpg.create_pool('postgres://localhost/app')
memory = Memory(PostgresMemoryStore(pool))
return memory, pool
```
Call `build_memory` during application startup and close the returned pool during shutdown. The store does not manage it.
## Namespaces
Use a namespace resolver when one `Agent` serves multiple users. It runs once per run from your typed dependencies, and its result is hidden from the model-facing tool schema.
```python
from dataclasses import dataclass
from pydantic_ai import Agent
from pydantic_ai_harness.memory import FileStore, Memory
@dataclass
class AppDeps:
user_id: str
agent = Agent(
'anthropic:claude-sonnet-4-6',
deps_type=AppDeps,
capabilities=[
Memory(
FileStore('/var/lib/myapp/memory'),
namespace=lambda ctx: ctx.deps.user_id,
)
],
defer_model_check=True,
)
```
Namespace isolation controls which records the capability addresses. It is not an authorization system for a custom or shared backing store. Validate the identity in application dependencies, restrict backend credentials, and ensure custom stores cannot escape the resolved namespace.
## Search
`search_memory` performs literal text search and always applies three bounds:
- `max_search_results` limits returned matches, default 10.
- `max_search_result_chars` limits the combined scope-relative filename and snippet text, default 4,000 characters.
- `max_search_files` limits how many files a fallback scan may inspect, default 1,000.
The bundled stores implement `SearchableMemoryStore`. For a custom store that implements only `MemoryStore`, `search_memory` requests at most `max_search_files + 1` paths, scans at most `max_search_files`, and performs bounded reads. Lexical scoring uses only each scope-relative filename and its bounded content; tenant namespaces and agent names never affect relevance. Implement the optional search protocol for an indexed or semantic backend while preserving the same tenant boundary and result limits. Semantic ranking is not built in.
Before backend dispatch, queries are limited to 1,000 characters and 32 unique whitespace-separated terms. Repeated case-insensitive terms are collapsed so they cannot inflate scoring or scan work.
## Configuration
```python
from pydantic_ai_harness.memory import FileStore, Memory
Memory(
FileStore('.agent-memory'),
store_resolver=None, # optional per-run store resolver
agent_name='main', # agent segment inside the namespace
namespace='', # string or per-run resolver
inject_memory=True, # False keeps prompts cache-stable
max_tokens=2_000, # finite approximate total injection budget
max_lines=200, # additional main-notebook line limit
max_memory_size=65_536, # per-file read, search, and write boundary
max_search_results=10,
max_search_result_chars=4_000,
max_search_files=1_000,
injection_errors='ignore', # or 'raise'
guidance=None, # None uses the default notebook guidance
)
```
## Agent specs
Register `Memory` as a custom capability type when constructing an agent from a Python spec:
```python
from pydantic_ai import Agent
from pydantic_ai_harness.memory import Memory
agent = Agent.from_spec(
{
'model': 'anthropic:claude-sonnet-4-6',
'capabilities': [
{'Memory': {'backend': 'file', 'directory': '.agent-memory'}},
],
},
custom_capability_types=[Memory],
defer_model_check=True,
)
```
The serializable backends are `memory`, `file`, and `sqlite`. A namespace callable and a live PostgreSQL pool must be configured in Python.
## Durable execution compatibility
| Execution mode | Support |
| --- | --- |
| Normal `Agent.run` calls | Supported with automatic injection or on-demand tools. |
| Temporal and Prefect | Use `inject_memory=False` with a statically configured store and on-demand tools. Automatic injection performs backend I/O in a model-request hook and is not workflow-safe. |
| DBOS | Normal execution works, but ordinary `FunctionToolset` calls are not DBOS-durable. Wrap memory operations in application-provided DBOS steps when durability is required. |
The memory backend and the workflow state backend are independent. Durable execution does not make an in-memory notebook persistent.
## Security and provenance
Memory is model-written, untrusted content that can re-enter future prompts. Keeping it in a delimited user-role part lowers its authority relative to model instructions, but this is not a hard prompt-injection boundary. Use `inject_memory=False` when less-trusted actors can write to the store, and expose memory only through application-controlled retrieval when stronger isolation is required. Do not store secrets unless the backend, retention policy, and access controls are appropriate. Sanitize content before rendering it into another trust domain.
Memory records do not carry source citations or verified provenance. If an application needs auditable facts, store provenance in the note itself or implement a custom store and schema. Optimistic concurrency prevents lost updates; it does not establish that a remembered claim is true.
## API reference
- [`pydantic_ai_harness.memory` source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/memory/)
- [Pydantic AI capabilities](/ai/core-concepts/capabilities/)
- [Pydantic AI hooks](/ai/core-concepts/hooks/)
The public module exports `Memory`, `MemoryToolset`, the bundled stores, the store protocols, mutation and search result models, and conflict exceptions. Import them from `pydantic_ai_harness.memory`.
::: pydantic_ai_harness.memory.Memory
::: pydantic_ai_harness.memory.MemoryToolset
docs/nav.json
+25
View File
@@ -0,0 +1,25 @@
[
{
"label": "Pydantic AI Harness",
"items": [
{ "label": "Overview", "slug": "index" },
{ "label": "Code Mode", "slug": "code-mode" },
{ "label": "FileSystem", "slug": "filesystem" },
{ "label": "Shell", "slug": "shell" },
{ "label": "Context", "slug": "context" },
{ "label": "Pydantic AI Docs", "slug": "pydantic-ai-docs" },
{ "label": "Compaction", "slug": "compaction" },
{ "label": "Overflowing Tool Output", "slug": "overflowing-tool-output" },
{ "label": "Step Persistence", "slug": "step-persistence" },
{ "label": "Media", "slug": "media" },
{ "label": "Subagents", "slug": "subagents" },
{ "label": "Dynamic Workflow", "slug": "dynamic-workflow" },
{ "label": "Planning", "slug": "planning" },
{ "label": "Memory", "slug": "memory" },
{ "label": "Runtime Authoring", "slug": "runtime-authoring" },
{ "label": "Guardrails", "slug": "guardrails" },
{ "label": "Managed Prompt", "slug": "managed-prompt" },
{ "label": "ACP", "slug": "acp" }
]
}
]
docs/overflowing-tool-output.md
+250
View File
@@ -0,0 +1,250 @@
---
title: Overflowing Tool Output
description: Reduce oversized tool returns when they are produced -- truncate, spill to a queryable file, or summarize -- so a large payload does not persist in history.
---
# Overflowing Tool Output
`OverflowingToolOutput` reduces a tool return that is large enough to dominate the context
window. Tool returns persist in history as `ToolReturnPart`s, so an oversized one is re-sent
on every later model request, paying its token cost for the rest of the run. This capability
intercepts a return when it is produced, reduces it once, and lets the reduced form persist --
the reduction is not recomputed per request.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/overflowing_tool_output/)
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
## The problem
A tool can return a payload large enough to dominate the context window: a big file read, a
verbose log, a large JSON document. Because tool returns persist in history, an oversized one
is re-sent on every later model request, paying its token cost for the rest of the run.
This is the overflow-to-file follow-up the [compaction](compaction.md) capability names as out
of scope: it moves large tool outputs *out* of the window at production time, rather than
compressing or dropping context already inside it.
## The three modes
| Mode | Cost | Lossy? | What the model gets |
|---|---|---|---|
| `Truncate` | zero-LLM | yes | A head / tail / head+tail clamp of the text |
| `Spill` | zero-LLM | no | A handle + preview + shape sketch; full payload read back on demand |
| `Summarize` | one LLM call | yes | A size-gated summary (inherits the run's model by default) |
`Spill` is lossless: the full payload is persisted and the model reads slices of it through
the registered `read_tool_result(handle, offset, limit, from_end, pattern)` tool (the Claude
Code pattern, the core [#4352](https://github.com/pydantic/pydantic-ai/issues/4352) design).
That tool is bounded: `offset >= 0`, `limit` clamped to a built-in line cap, the joined output
capped, and `pattern` is a literal substring (not a regex), so a model-supplied value cannot
hang the host with catastrophic backtracking.
## Usage
Construct an `Agent` with `OverflowingToolOutput()` in its `capabilities`. With no arguments it
uses the default band: spill returns of 10,000 characters or more, with a bounded truncation
fallback if the store cannot accept the write.
```python
from pydantic_ai import Agent
from pydantic_ai_harness.overflowing_tool_output import OverflowingToolOutput
agent = Agent('openai:gpt-4o', capabilities=[OverflowingToolOutput()])
```
The capability registers a single `read_tool_result` tool so the model can page back into any
spilled payload. Its own returns are exempt from reduction.
## Bands: combine the modes
Configure an ordered list of size `bands`. Each band is a `(over, action)` pair: when a
return's measured size reaches `over`, its action runs. The band with the largest threshold
that fits wins; anything below the smallest threshold passes through.
```python
from pydantic_ai import Agent
from pydantic_ai_harness.overflowing_tool_output import (
Band,
OverflowingToolOutput,
Spill,
Summarize,
Truncate,
)
agent = Agent(
'openai:gpt-4o',
capabilities=[
OverflowingToolOutput(
bands=[
Band(over=100_000, action=Spill()), # huge: keep losslessly, read back on demand
Band(over=20_000, action=Summarize()), # large: compress with the run's model
Band(over=5_000, action=Truncate()), # medium: cheap clamp
],
# below 5,000: passthrough
)
],
)
```
The default band, when you pass no `bands`, is `Spill(then=Truncate())` at a 10,000-character
threshold: lossless when a store accepts the write, a bounded truncation otherwise -- zero LLM
cost and no silent drop.
`Passthrough()` is an explicit no-op action for `bands` or `per_tool` lists, leaving matching
returns untouched.
### Fallbacks with `then`
Every action takes an optional `then`, applied when the action cannot run: a `Spill` whose
store errors, a `Truncate` / `Summarize` on a binary payload, a `Summarize` whose model call
raises. `then` chains, so `Summarize(then=Spill(then=Truncate()))` degrades summarize ->
spill -> truncate.
### Per-tool overrides and filtering
`per_tool` replaces the global band list for named tools (file reads to `head`, logs to
`tail`); `tool_filter` (a `ToolSelector`) scopes which tools the capability touches at all.
```python
from pydantic_ai import Agent
from pydantic_ai_harness.overflowing_tool_output import (
Band,
OverflowingToolOutput,
Truncate,
TruncationStrategy,
)
agent = Agent(
'openai:gpt-4o',
capabilities=[
OverflowingToolOutput(
per_tool={
'read_file': [Band(over=8_000, action=Truncate(strategy=TruncationStrategy.head))],
'run_shell': [Band(over=8_000, action=Truncate(strategy=TruncationStrategy.tail))],
},
tool_filter=['read_file', 'run_shell', 'search'],
)
],
)
```
`TruncationStrategy` has three members: `head` (keep the first characters, good for headers and
schemas), `tail` (keep the last characters, good for build and test output where errors land
last), and `head_tail` (keep both ends, elide the middle -- the default).
## Both `return_value` and `content` are reduced
A `ToolReturn` carries a `return_value` and an optional `content` that core renders as a
separate, model-visible part which also persists in history. This capability measures and
reduces both with the same band logic (they spill to distinct handles). Text `content` is
reduced in place; non-text `content` (multimodal parts) that overflows is left unreduced with
a `warnings.warn`, since it cannot be safely truncated.
## Size unit
Thresholds are measured in characters by default. Set `over_tokens=True` to measure in
estimated tokens (the same ~4-chars-per-token heuristic as [compaction](compaction.md)); pass a
`tokenizer` callable for accuracy. `Truncate.max_chars` is always characters -- truncation is a
character operation regardless of the threshold unit. Set `strip_ansi=True` to strip ANSI
escape sequences from text returns before measuring and reducing.
## Spill store
Spilled payloads go through the narrow `OverflowStore` protocol. The default `LocalFileStore`
writes one file per `(run_id, tool_call_id, retry)` under a stable root directory and keeps it
after the run, so a later `read_tool_result` -- in this run or a subsequent agent/run -- can
still reach it. The handle is backend-addressable (a relative key), not an absolute local
path, so a durable backend (Temporal, a blob store, or the core `ExecutionEnvironment`
workspace once #4352 lands) can resolve the same handle in another process. Supply your own
backend with `store=...`.
```python
from typing import Protocol
class OverflowStore(Protocol):
async def write(self, key: str, data: bytes) -> str: ... # returns a handle
async def read(self, handle: str) -> bytes: ...
```
### Security model (shared root, not isolation)
The store root is stable and shareable on purpose -- spilled files must be readable by a later
agent or run -- so security does not come from per-instance isolation. It comes from two
mechanisms: the root is created with `0700` (owner-only) permissions, and `read` resolves the
target (following symlinks) and rejects anything that escapes the root via symlink, `..`, or
an absolute path. Handle segments are also sanitized so a crafted handle cannot traverse out.
### Cleanup: keep-forever by default, opt-in TTL pruning
By default the store keeps spilled files forever -- deleting on run end would break a later
agent that still wants to read a spill. To bound disk use, opt into age-based pruning:
```python
from datetime import timedelta
from pydantic_ai import Agent
from pydantic_ai_harness.overflowing_tool_output import LocalFileStore, OverflowingToolOutput
store = LocalFileStore(cleanup_after=timedelta(hours=6)) # default: None = keep forever
agent = Agent('openai:gpt-4o', capabilities=[OverflowingToolOutput(store=store)])
```
When set, a `write` schedules a background prune (a daemon thread, off the hot path) that
deletes files whose modification time (`st_mtime`) is older than `cleanup_after`. Pruning is
non-blocking and non-erroring: any failure is caught and surfaced via `warnings.warn`, never
propagated into the agent run, so cleanup can never fail a run or block the hot path.
Last-read time (`st_atime`) is unreliable on `noatime`/`relatime` mounts and is not used.
Prefer external cleanup (cron, a sweeper) over the in-process TTL? Point it at the store root
and delete by mtime:
```python
import time
from pathlib import Path
root = Path('/tmp/pyai_harness_overflow') # or your configured base_dir
cutoff = time.time() - 6 * 3600
for path in root.rglob('*'):
if path.is_file() and path.stat().st_mtime < cutoff:
path.unlink(missing_ok=True)
```
## Usage accounting
A `Summarize` call is a real request to the model, so its full usage -- tokens and the
request itself -- folds into the run's `ctx.usage`, exactly like `SummarizingCompaction`. No
token caps are imposed on the summary call. A `UsageLimits` request limit will see it.
By default `Summarize` inherits the running agent's model (`ctx.model`). Pass a model id or
instance to `Summarize(model=...)` to override, or a `summarize` callable to bypass the
built-in prompt entirely. The `summary_prompt` template on the capability must contain both
`{tool_name}` and `{output}` placeholders.
## Edge cases
- Binary returns spill verbatim and are never stringify-truncated; `Truncate` / `Summarize`
on binary fall through to `then`.
- Structured / nested returns spill (or summarize) by preference -- truncating JSON produces
invalid JSON. `Spill` includes a one-line shape sketch of the top level.
- `ModelRetry` and tool errors never reach this hook (they are raised, not returned), so the
model always gets the full error it needs to recover.
- A large `ToolReturn.content` is reduced with the same bands as `return_value`; non-text
content that overflows is left unreduced with a warning.
- Multiple oversized returns in one step get distinct handles (keyed per `tool_call_id`);
retries get distinct handles too (keyed per `retry`), so a retried call never clobbers the
earlier attempt's spill.
## Relationship to other capabilities
- Distinct from [compaction](compaction.md), which compresses or drops context already inside
the window; this capability moves large tool outputs out of the window at production time.
- Consumes core [#4352](https://github.com/pydantic/pydantic-ai/issues/4352) (the canonical
queryable-file primitive) through the `OverflowStore` seam once it lands.
- Distinct from `ClampOversizedMessages`, which clamps runaway model responses, not tool
returns.
## API reference
::: pydantic_ai_harness.overflowing_tool_output.OverflowingToolOutput
docs/planning.md
+143
View File
@@ -0,0 +1,143 @@
---
title: Planning
description: Give an agent a structured, self-updating task plan through a single write_plan tool, without ever invalidating the prompt cache.
---
# Planning
`Planning` gives the model a structured, self-updating task plan through a single `write_plan` tool -- and surfaces the current plan back to the model every turn without ever invalidating the prompt cache.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/planning/)
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
## The problem
Long agentic runs drift: the model loses track of what it set out to do and what's left. The usual fix -- keep a running plan and re-inject it into the system prompt each turn -- invalidates the prompt cache. The system prompt sits at the front of the request, so every plan edit changes the cached prefix and forces the whole conversation to be re-processed at full token price.
## The solution
`Planning` gives the model one tool, `write_plan`, that owns the plan (whole-plan replacement -- pass the full list every call, no indices). The current plan is surfaced back to the model as an ephemeral reminder appended to the tail of each request, behind a cache breakpoint:
- The reminder is added in `wrap_model_request`, which runs *after* the durable history is persisted, so it reaches the model but is never written to `message_history`. No reminders accumulate across turns.
- A `CachePoint` is placed immediately *before* the reminder, so the cached prefix (tools + system + real conversation) stays byte-identical turn over turn. Only the reminder falls outside the cache.
So the plan stays current in the model's view while the cached prefix is never invalidated; the only added cost is re-reading the reminder each turn.
## Usage
Construct an `Agent` with `Planning()` in its `capabilities`. The `write_plan` tool is registered automatically, and the static usage guidance is added to the system prompt:
```python
from pydantic_ai import Agent
from pydantic_ai_harness.planning import Planning
agent = Agent('anthropic:claude-sonnet-4-6', capabilities=[Planning()])
result = agent.run_sync('Refactor the auth module and add tests.')
print(result.output)
```
## The tool
| Tool | Purpose |
|---|---|
| `write_plan(items)` | Create or replace the full plan. The model passes the entire ordered list every time, including unchanged, completed, and cancelled steps. |
Each item is a `content` string plus a `status` (`pending`, `in_progress`, `completed`, `cancelled`). The convention -- stated in the guidance and noted in the tool's reply -- is to keep exactly one step `in_progress`.
There is no `get_plan` tool: the current plan is already in the model's context via the tail reminder every turn.
## Why whole-plan replacement
Addressing steps by mutable integer index (insert/remove/reorder) is error-prone for both the code (index bookkeeping) and the model (indices it just saw can go stale within a turn). Restating the whole plan each call removes that: there are no indices to track, and a later call can't corrupt partial state. For short plans the token cost is negligible.
## Caching guarantee
The plan is never injected into the system prompt or instructions. Static usage guidance goes there (cache-stable); only the mutable plan rides the ephemeral tail reminder. Across turns:
- the durable history grows append-only and is replayed byte-identically, so the whole prefix is a cache hit;
- the reminder and its `CachePoint` live only in the per-request copy, so they can't invalidate anything and aren't persisted.
`CachePoint` is supported on Anthropic and Amazon Bedrock; on providers without prompt caching it's simply ignored (nothing to bust).
## Configuration
```python
from pydantic_ai_harness.planning import Planning
Planning(
guidance=None, # static system-prompt guidance; None = default, '' = omit
cache_ttl='5m', # TTL for the cache breakpoint before the reminder ('5m' | '1h')
)
```
- `guidance` -- static planning guidance added to the system prompt. It is identical on every request, so it stays cache-stable. Leave it as `None` for the built-in default, or set `''` to omit guidance entirely.
- `cache_ttl` -- TTL for the cache breakpoint placed before the plan reminder. One of `'5m'` or `'1h'`.
## Observing the plan
Plan state is per-run (a fresh, isolated plan each run via `for_run`), so it doesn't live on the `Planning()` instance you construct. To see the final plan, read the most recent `write_plan` tool return from the run's messages -- its content is the rendered plan:
```python
from pydantic_ai import Agent
from pydantic_ai.messages import ToolReturnPart
from pydantic_ai_harness.planning import Planning
agent = Agent('anthropic:claude-sonnet-4-6', capabilities=[Planning()])
result = agent.run_sync('Refactor the auth module and add tests.')
plans = [
part.content
for message in result.all_messages()
for part in message.parts
if isinstance(part, ToolReturnPart) and part.tool_name == 'write_plan'
]
latest_plan = plans[-1] if plans else None
print(latest_plan)
```
## Composition
`Planning` contributes a single leaf toolset (`write_plan`), some static instructions, and a `wrap_model_request` hook. It does not wrap or intercept other toolsets, so it composes cleanly alongside other capabilities and your own tools in the same `Agent(..., capabilities=[...])`.
The tail reminder is only appended when the last message in the request is a `ModelRequest` and the plan is non-empty, so an empty plan adds nothing to the request. Because the reminder and its `CachePoint` live only in the per-request copy and never enter the durable history, `Planning` is safe with message-history replay and does not accumulate stale reminders across turns.
## Agent spec (YAML/JSON)
`Planning` works with Pydantic AI's [agent spec](/ai/core-concepts/agent-spec/) feature for defining agents in YAML or JSON:
```yaml
# agent.yaml
model: anthropic:claude-sonnet-4-6
capabilities:
- Planning: {}
```
```python
from pydantic_ai import Agent
from pydantic_ai_harness.planning import Planning
agent = Agent.from_file('agent.yaml', custom_capability_types=[Planning])
result = agent.run_sync('...')
print(result.output)
```
Pass `custom_capability_types` so the spec loader knows how to instantiate `Planning`. Arguments can be passed in the YAML too:
```yaml
capabilities:
- Planning:
cache_ttl: '1h'
```
## Further reading
- [Pydantic AI capabilities](/ai/core-concepts/capabilities/)
- [Hooks](/ai/core-concepts/hooks/) -- `wrap_model_request` is the ephemeral injection point used here
- [Anthropic prompt caching](https://docs.claude.com/en/docs/build-with-claude/prompt-caching)
- [Code Mode](code-mode.md) -- another prompt-cache-aware harness capability
## API reference
::: pydantic_ai_harness.planning.Planning
docs/pydantic-ai-docs.md
+90
View File
@@ -0,0 +1,90 @@
---
title: Pydantic AI Docs
description: Give an agent a tool that locates and returns Pydantic AI documentation on demand instead of preloading it into the system prompt.
---
# Pydantic AI Docs
`PyaiDocs` gives an agent a single tool, `read_pyai_docs(topic)`, that locates a Pydantic AI documentation page and returns it verbatim. Nothing is bundled into context up front. Each call resolves the topic from a configured local checkout first, then falls back to fetching the page from `pydantic/pydantic-ai:main`, so it works whether or not you have a local checkout (the remote fallback needs network access).
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/docs/)
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
## The problem
An agent that authors Pydantic AI capabilities, hooks, tools, or toolsets needs the current docs for those APIs. Preloading the docs into the system prompt spends context the agent rarely needs in full, and pins a snapshot that drifts from `main`.
## The solution
`PyaiDocs` exposes one tool, `read_pyai_docs(topic)`, that locates the requested page and returns it verbatim. Each call resolves the topic from a configured local checkout first, then falls back to fetching the page from `pydantic/pydantic-ai:main`, so it works whether or not you have a local checkout (the remote fallback needs network access).
The available topics are `capabilities`, `hooks`, `tools`, `tools-advanced`, `toolsets`, and `agent`.
## Usage
Construct an `Agent` with `PyaiDocs()` in its `capabilities`. Point `local_docs_path` at a local Pydantic AI docs checkout to read from disk first, or omit it to always fetch from the remote source:
```python
from pathlib import Path
from pydantic_ai import Agent
from pydantic_ai_harness.docs import PyaiDocs
agent = Agent(
'anthropic:claude-sonnet-4-6',
capabilities=[PyaiDocs(local_docs_path=Path('~/pydantic/ai/base/docs').expanduser())],
)
result = agent.run_sync('Read the toolsets docs, then explain how to build a FunctionToolset.')
print(result.output)
```
The capability also adds a short static instruction telling the model that the `read_pyai_docs` tool exists and to read the relevant topic before authoring or modifying a Pydantic AI capability, hook, tool, or toolset, rather than relying on memory. The instruction is cache-stable, so it does not invalidate the prompt-cache prefix between turns.
## Resolution order
Each call resolves in this order:
1. **Local checkout** -- when `local_docs_path` (or the `PYDANTIC_AI_HARNESS_DOCS_PATH` env var) is set and `{path}/{topic}.md` exists, that file is read and returned.
2. **Remote fetch** -- otherwise the page is fetched from `https://raw.githubusercontent.com/pydantic/pydantic-ai/main/docs/{topic}.md`.
3. **Neither resolves** -- a descriptive error naming the local path tried and the URL.
The capability never runs git. Keep the local checkout current yourself; the remote path always reads `main`, so it is the fresh fallback.
`local_docs_path` takes precedence over the `PYDANTIC_AI_HARNESS_DOCS_PATH` env var. Both have `~` expanded, so a raw `~/...` path resolves to the local checkout instead of silently falling through to the remote source. With neither set, every call goes straight to the remote source.
## Configuration
| Option | Default | Purpose |
| --- | --- | --- |
| `local_docs_path` | `None` | Local pyai docs checkout to read first. Falls back to the `PYDANTIC_AI_HARNESS_DOCS_PATH` env var, then to the remote source. |
| `cache` | `True` | Memoize each returned doc in-process for the capability's lifetime, so a topic is read or fetched at most once. |
Caching lives on the capability instance and is shared across the toolsets it builds, so a memoized topic survives multiple agent runs that reuse the same `PyaiDocs`. Set `cache=False` to re-read or re-fetch on every call -- useful when the local checkout changes underneath a long-lived capability.
## Agent spec (YAML/JSON)
`PyaiDocs` works with Pydantic AI's [agent spec](/ai/core-concepts/agent-spec/) feature for defining agents in YAML or JSON. Its serialization name is `PyaiDocs`:
```yaml
# agent.yaml
model: anthropic:claude-sonnet-4-6
capabilities:
- PyaiDocs: {}
```
```python
from pydantic_ai import Agent
from pydantic_ai_harness.docs import PyaiDocs
agent = Agent.from_file('agent.yaml', custom_capability_types=[PyaiDocs])
result = agent.run_sync('...')
print(result.output)
```
Pass `custom_capability_types` so the spec loader knows how to instantiate `PyaiDocs`.
## API reference
::: pydantic_ai_harness.docs.PyaiDocs
docs/runtime-authoring.md
+95
View File
@@ -0,0 +1,95 @@
---
title: Runtime Authoring
description: Let an agent write, validate, and persist real pydantic-ai capabilities at runtime, live on the next run.
---
# Runtime Authoring
`RuntimeAuthoring` lets an agent author, validate, and persist real pydantic-ai capabilities while it runs. It exposes three tools that let the model write a capability class to disk as Python source, validate it immediately, and manage the set of authored capabilities. Each authored capability is a real `pydantic_ai.capabilities.AbstractCapability` subclass, so it can contribute instructions, model settings, a toolset, native tools, or a lifecycle hook.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/runtime_authoring/)
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
## The problem
A coding agent often discovers, mid-task, that it wants a behavior its host does not yet have: a guardrail, an extra instruction, a tool, a request hook. The capability surface to express that already exists -- but normally only a developer can write a capability class, wire it into the agent, and restart. The agent itself cannot extend its own host while it runs.
## The solution
`RuntimeAuthoring` exposes three tools:
- `author_capability(name, code)` -- write `code` to `<directory>/<name>.py`, import it, and validate it. Validation requires exactly one `pydantic_ai.capabilities.AbstractCapability` subclass that constructs with no arguments; the side-effect-free static getters (`get_instructions`, `get_toolset`, `get_native_tools`, `get_model_settings`, `get_serialization_name`) are exercised. The async lifecycle hooks are not run -- they need a live `RunContext`.
- `list_authored_capabilities()` -- list authored capabilities with their status and any validation error.
- `disable_authored_capability(name)` -- stop a capability from being injected on the next run.
A "hook" is not a standalone object in pydantic-ai -- it is a method on a capability. So authoring a hook means authoring a capability that overrides one lifecycle method. A single overridden hook is a valid capability.
## Usage
Construct `RuntimeAuthoring` with a `directory` for the authored files, then add it to the agent's `capabilities`:
```python
from pathlib import Path
from pydantic_ai import Agent
from pydantic_ai_harness.runtime_authoring import RuntimeAuthoring
authoring = RuntimeAuthoring(directory=Path('.authored'))
agent = Agent('anthropic:claude-sonnet-4-6', capabilities=[authoring])
```
The agent can now call `author_capability`, `list_authored_capabilities`, and `disable_authored_capability`. `RuntimeAuthoring` also contributes static, cache-stable system-prompt guidance explaining these tools. Leave `guidance=None` for the default text, or pass your own string; set `guidance=''` to omit it entirely.
## Activation boundary
A capability **cannot** be added to a live, already-executing run. pydantic-ai resolves the effective capability set once at the start of each run (the run's root capability is fixed; there is no setter). So an authored capability is live on the **next** `agent.run(...)`, not the run that authored it. Authoring writes and validates the capability immediately, but its tools and hooks only exist once the next run's toolset and capability chain are assembled at run start.
### Integration contract
The orchestrator drives the loop, so it owns the one-line contract: thread the store's active capabilities into each run via `agent.run(..., capabilities=...)`. With that in place, the authored capability is live on the very next loop iteration -- no process restart:
```python
from pathlib import Path
from pydantic_ai import Agent
from pydantic_ai_harness.runtime_authoring import RuntimeAuthoring
authoring = RuntimeAuthoring(directory=Path('.authored'))
agent = Agent('anthropic:claude-sonnet-4-6', capabilities=[authoring])
history = None
done = False
next_prompt = 'Start the task.'
while not done:
extra = authoring.store.load_active()
result = await agent.run(next_prompt, message_history=history, capabilities=extra)
history = result.all_messages()
# ... decide `next_prompt` and `done` from `result` ...
```
`authoring.store` is the disk-backed `CapabilityStore` over the same `directory`. `store.load_active()` re-imports and re-constructs every active authored capability for injection into the next run. Entries that fail to load (corrupt source, construction error) are skipped, not raised, so one bad capability never blocks the rest.
## Persistence
Authored capabilities persist to disk: each is one `<directory>/<name>.py` file, indexed by a sibling `manifest.json`. A fresh process picks them up by constructing a new `RuntimeAuthoring` over the same `directory` and calling `store.load_active()`.
`manifest.json` records each capability's name, module file, class name, status (`active` or `disabled`), and last validation error. That is the surface a UI can read to show what the agent has authored. The manifest is written atomically (temp file plus `os.replace`), so a crash mid-write never leaves a partial file that reads back as "no capabilities".
Capability names must be lowercase letters, digits, and underscores, starting with a letter. Reusing a name replaces the previous capability of that name. A code that imports but fails validation is still written to disk (so it can be inspected) and recorded with its `last_error` set; `load_active()` skips it.
## Trust boundary
`RuntimeAuthoring` executes arbitrary Python in-process at import, construction, and run time. That is the same trust boundary an agent that already runs shell commands and edits files operates under, which is the deliberate choice here. Do not point it at a directory whose contents you would not run yourself, and treat authored capabilities as code the agent is executing on your host.
Because authored capabilities hold live code, they are not spec-serializable (`get_serialization_name()` returns `None`) and are persisted as source rather than as an [agent spec](/ai/core-concepts/agent-spec/).
## Typing
Imported authored code is dynamic, but nothing typed `Any` crosses back into the harness: every value pulled from an authored module is narrowed with `isinstance`/`issubclass` before use, and loaded instances are typed `AbstractCapability[object]`. Because `AgentDepsT` is contravariant, an `AbstractCapability[object]` is accepted by any agent's `capabilities=` parameter.
## API reference
::: pydantic_ai_harness.runtime_authoring.RuntimeAuthoring
::: pydantic_ai_harness.runtime_authoring.CapabilityStore
docs/shell.md
+255
View File
@@ -0,0 +1,255 @@
---
title: Shell
description: Give a Pydantic AI agent shell command execution with allow/deny controls, environment scrubbing, and managed background processes.
---
# Shell
`Shell` gives an agent the ability to run shell commands, with allow/deny
controls, environment scrubbing, and managed background processes. It exposes
command-execution tools rooted at a working directory and cleans up any
background processes automatically when the agent run ends.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/shell/)
## The problem
Agents frequently need to run a build, a test suite, a linter, or a quick
`grep`. Wiring up subprocess handling -- streaming output, timeouts, truncation,
killing runaway processes, and cleaning up background jobs at the end of a run --
is fiddly boilerplate that every agent reinvents.
`Shell` bundles that plumbing into a single [capability](/ai/core-concepts/capabilities/):
configurable allow/deny lists, output truncation tuned to keep the useful tail,
optional sticky working directory, environment control that can keep host
secrets out of spawned commands, and automatic cleanup of background processes
when the run finishes.
## Usage
Construct `Shell` with a working directory and pass it to an `Agent` via the
`capabilities` parameter:
```python
from pydantic_ai import Agent
from pydantic_ai_harness import Shell
agent = Agent(
'anthropic:claude-sonnet-4-6',
capabilities=[Shell(cwd='./workspace', allowed_commands=['ls', 'cat', 'rg'])],
)
result = agent.run_sync('List the Python files and summarize the largest one.')
print(result.output)
```
By default `Shell` runs in the current directory with the built-in destructive-command
denylist active -- `Shell()` alone is a working (if permissive) configuration.
## Tools
`Shell` contributes four tools to the agent:
| Tool | Purpose |
|---|---|
| `run_command` | Run a command synchronously and return labelled stdout/stderr plus exit code. Honors a per-call or default timeout. |
| `start_command` | Launch a long-running command (server, watcher) in the background; returns an ID. |
| `check_command` | Report the status and accumulated output of a background command. |
| `stop_command` | Terminate a background command and return its final output. |
`run_command` accepts an optional `timeout_seconds` argument that overrides
`default_timeout` for a single call. `check_command` and `stop_command` take the
`command_id` string returned by `start_command`.
Output is labelled with `[stdout]` / `[stderr]` markers and an `[exit code: N]`
line on non-zero exit. When it exceeds `max_output_chars` the **tail** is kept
(the head is dropped), so errors, stack traces, and the `[stderr]` section --
which all land at the end -- survive truncation.
## Command controls
Two mutually exclusive lists decide which executables may run, plus filters for
shell operators and interactive commands:
| Field | Effect |
|---|---|
| `allowed_commands` | If non-empty, only these executables may run (allowlist). |
| `denied_commands` | These executables are always rejected (denylist). |
| `denied_operators` | Shell operators (e.g. `>`, `>>`, `\|`) that are rejected when present. |
| `allow_interactive` | If `False` (default), commands that expect a TTY (`vi`, `sudo`, `ssh`, ...) are blocked. |
`allowed_commands` and `denied_commands` are mutually exclusive -- set one, not
both. Setting both raises a `ValueError` at construction. `denied_commands`
defaults to a list of destructive commands (`rm`, `rmdir`, `mkfs`, `dd`,
`format`, `shutdown`, `reboot`, `halt`, `poweroff`, `init`); pass an empty list
to disable it. The executable name is extracted with `shlex`, so arguments don't
bypass the check.
A denied command surfaces to the model as a
[`ModelRetry`](/ai/tools-toolsets/tools-advanced/#tool-retries), not a hard error:
the run continues and the model can pick an allowed command instead.
!!! warning "Best-effort, not a security boundary"
These command checks are best-effort. A sufficiently motivated agent can
defeat them (e.g. `bash -c '...'`, env-var indirection). For hard
guarantees, run the agent inside OS-level isolation -- a container or
sandbox.
## Environment control
By default a spawned command inherits the agent process's full environment. In a
sandbox that holds LLM API keys, tokens, or other secrets, a command the model
writes can read them. Two fields control what the subprocess sees:
| Field | Effect |
|---|---|
| `env` | Explicit environment that replaces inheritance entirely. The subprocess sees exactly these variables and nothing else. |
| `denied_env_patterns` | Glob patterns (`fnmatch`) for variable names stripped from the base environment. Mirrors `denied_commands`. |
`env` is a hard boundary for inherited environment variables: set it and inherited secrets cannot reach the
subprocess at all (you supply `PATH` and anything else the command needs).
`denied_env_patterns` is a denylist over the inherited environment -- lighter to
configure when you only need to drop a few known-sensitive names. The two
compose: when both are set, patterns also filter the explicit `env`. Leaving
both unset preserves the inherit-everything default.
```python
import os
from pydantic_ai_harness import Shell
from pydantic_ai_harness.shell import LLM_API_KEY_ENV_PATTERNS
# Strip provider credentials from the inherited environment.
Shell(cwd='./repo', denied_env_patterns=LLM_API_KEY_ENV_PATTERNS)
# Or hand the subprocess a fixed environment, inheriting nothing.
Shell(cwd='./repo', env={'PATH': os.environ['PATH'], 'HOME': os.environ['HOME']})
```
`LLM_API_KEY_ENV_PATTERNS` covers common provider prefixes (`ANTHROPIC_*`,
`GATEWAY_*`, `GEMINI_*`, `GOOGLE_*`, `OPENAI_*`, `OPENROUTER_*`) plus
`PYDANTIC_AI_GATEWAY_API_KEY`. It targets LLM credentials only -- it does not
cover other host secrets (a `LOGFIRE_TOKEN`, a GitHub token, cloud
credentials), and its prefixes are coarse, so `GOOGLE_*` also strips
non-credential vars like `GOOGLE_APPLICATION_CREDENTIALS`. Treat it as a
starting point and add your own patterns. It is not the default: stripping
environment variables silently would break agents that rely on inherited
credentials, so it is opt-in.
`env` is enforced at spawn, not applied as a post-hoc filter on a running
process: the subprocess starts with exactly the resolved environment (your
`env`, minus anything `denied_env_patterns` removes from it). That makes it a
real boundary for inherited environment variables, unlike the best-effort command denylist. It is not a full
security boundary: a command running under the same OS identity can still read
host files -- use OS-level isolation for that. The flip side is that a
pattern broad enough to strip `PATH` or `HOME`, or an `env` that omits them, can
break command resolution. External commands may still run via the shell's
built-in default `PATH` on some systems, but don't rely on it -- set `PATH`
explicitly when you replace the environment.
## Background processes
`start_command` writes stdout/stderr to temp files and returns a short ID. Use
`check_command(command_id)` to poll and `stop_command(command_id)` to terminate
and collect final output. Processes are launched in their own session
(`start_new_session`) so the whole process group can be signalled -- `SIGTERM`,
escalating to `SIGKILL` after a grace period.
On run end, the toolset's cleanup terminates every still-running background
process and deletes its temp files. The agent runtime enters toolsets via an
`AsyncExitStack`, so this cleanup runs whether the run succeeds or raises -- an
agent that forgets to call `stop_command` won't leak processes.
```python
from pydantic_ai import Agent
from pydantic_ai_harness import Shell
agent = Agent(
'anthropic:claude-sonnet-4-6',
capabilities=[Shell(cwd='./app', allowed_commands=['npm', 'curl'])],
)
result = agent.run_sync(
'Start the dev server with `npm run dev`, wait for it to boot, '
'then curl http://localhost:3000/health and report the status.'
)
print(result.output)
```
## Working directory
By default each command runs in `cwd` and `cd` has no lasting effect. Set
`persist_cwd=True` to make `cd` sticky across calls: each command is wrapped so
that after it runs, its final working directory is recorded to a private temp
file, and that directory is carried into subsequent calls. The path is only
updated when the command exits `0`, and the record is written out-of-band (not
to stdout) so command output can never spoof the tracked directory.
```python
from pydantic_ai import Agent
from pydantic_ai_harness import Shell
agent = Agent(
'anthropic:claude-sonnet-4-6',
capabilities=[Shell(cwd='.', persist_cwd=True, allowed_commands=['cd', 'ls', 'pwd'])],
)
```
Each run gets a fresh toolset instance, so the tracked directory and any
background processes are isolated between concurrent runs and always start back
at the configured `cwd`.
## Configuration
Every field of `Shell` with its default:
```python
from pydantic_ai_harness import Shell
Shell(
cwd='.', # str | Path -- working directory
allowed_commands=[], # allowlist (mutually exclusive with denied)
denied_commands=[...], # denylist (defaults to destructive commands)
denied_operators=[], # blocked shell operators
default_timeout=30.0, # seconds, per run_command
max_output_chars=50_000, # output cap returned to the model
persist_cwd=False, # make cd sticky across calls
allow_interactive=False, # allow TTY-style commands
env=None, # explicit env, replacing inheritance (None = inherit)
denied_env_patterns=[], # glob patterns stripped from the env
)
```
## Agent spec (YAML/JSON)
`Shell` works with Pydantic AI's
[agent spec](/ai/core-concepts/agent-spec/), so you can declare it in a
config file instead of Python:
```yaml
# agent.yaml
model: anthropic:claude-sonnet-4-6
capabilities:
- Shell:
cwd: ./workspace
allowed_commands: ['ls', 'cat', 'rg', 'pytest']
```
```python
from pydantic_ai import Agent
from pydantic_ai_harness import Shell
agent = Agent.from_file('agent.yaml', custom_capability_types=[Shell])
```
Pass `custom_capability_types` so the spec loader knows how to instantiate
`Shell`.
## Further reading
- [Pydantic AI capabilities](/ai/core-concepts/capabilities/)
- [Toolsets](/ai/tools-toolsets/toolsets/)
## API reference
::: pydantic_ai_harness.Shell
docs/step-persistence.md
+447
View File
@@ -0,0 +1,447 @@
---
title: Step Persistence
description: Record what an agent did at each boundary, save provider-valid snapshots to resume or fork from, and track tool side effects across crashes.
---
# Step Persistence
`StepPersistence` records what an agent did at each boundary, separate from whether the run can be safely resumed. It is the persistence substrate for orchestrators that delegate to sub-agents -- for example, an AICA orchestrator that spawns a `code_librarian` to investigate one symbol, then continues that delegate's investigation with a follow-up question.
It is not a full graph-state checkpoint. Capability-state restore, workspace snapshots, and graph-node resume are out of scope and tracked separately (see `pydantic-ai-harness` issues #149 and #196).
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/step_persistence/)
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
## What it gives you
1. **Append-only step events.** Every interesting boundary (run start/end, model request, tool call, failure) appends a `StepEvent`. A run killed mid-tool-call still leaves a usable event trail.
2. **Continuable snapshots.** A `ContinuableSnapshot` is saved only at boundaries where the message history is provider-valid: every `ToolCallPart` has a matching `ToolReturnPart` or `RetryPromptPart`, with no orphan, duplicate, or out-of-order returns. Pass the snapshot's `messages` back to `Agent.run(message_history=...)` to continue or fork.
3. **Tool-effect ledger.** Every tool call's lifecycle (`started`, `completed`, `failed`) is recorded against `(run_id, tool_call_id)`. After a crash, a tool with a `started` record and no terminal update should be treated as `unknown_after_crash`: the side effect may or may not have happened.
4. **Lineage metadata.** `conversation_id` (sequence) and `parent_run_id` (hierarchy) are independent axes. See [Three-level identity](#three-level-identity).
## Quick start
```python
import asyncio
from pydantic_ai import Agent
from pydantic_ai_harness.step_persistence import StepPersistence, InMemoryStepStore
store = InMemoryStepStore()
librarian = Agent(
'openai:gpt-5',
capabilities=[StepPersistence(store=store, agent_name='code_librarian')],
)
async def main():
await librarian.run('Find ThinkingPartDelta and confirm the callable allowance')
asyncio.run(main())
```
That is the whole setup. `run_id` is always per-`Agent.run` call, matching pydantic_ai's `RunContext.run_id`. For multi-turn logical grouping use `conversation_id=` -- that is the pydantic_ai-native primitive for it (see [Three-level identity](#three-level-identity)).
`run_id` resolution per call:
- **Explicit `run_id='libr-1'`** becomes the id for this one call. This suits single-shot use cases (a deterministic id for testing, replay, debugging, or a one-off scripted run). Reusing one capability instance with the same explicit `run_id` across multiple `.run()` calls raises `ValueError` in `before_run`. The tool-effect ledger is keyed by `(run_id, tool_call_id)` and providers reuse deterministic tool-call ids, so a silent collision would erase the `unknown_after_crash` signal. Use `conversation_id=` for multi-turn grouping instead.
- **`agent_name` set, `run_id` unset** derives `'{agent_name}-{8-char-hex}'`, freshly materialised in `for_run` per `.run()` call. Reusing one capability instance across runs yields distinct ids (`code_librarian-a3b2`, `code_librarian-c9d1`, and so on). This is the recommended default for delegate capabilities.
- **Neither set** falls back to `ctx.run_id` (pydantic_ai's auto-generated id) per `.run()` call, and to a UUID4 if that is absent.
The orchestrator pattern -- one logical agent serving many turns -- uses `conversation_id`, not a shared `run_id`:
```python
import asyncio
from pydantic_ai import Agent
from pydantic_ai_harness.step_persistence import StepPersistence, InMemoryStepStore
store = InMemoryStepStore()
orchestrator = Agent(
'openai:gpt-5',
capabilities=[StepPersistence(store=store, agent_name='orchestrator')],
)
async def main():
for turn in turns:
await orchestrator.run(turn, conversation_id='orch-conv')
# All turns of this orchestrator, chronological:
records = await store.list_runs(conversation_id='orch-conv')
asyncio.run(main())
```
## Three-level identity
The capability mirrors pydantic_ai's identity stack:
| Concept | Definition | Granularity |
| --- | --- | --- |
| `conversation_id` | The dialogue. Resolved by pydantic_ai from the `conversation_id=` argument to `Agent.run`, or the most recent `conversation_id` on `message_history`, or a fresh UUID7. | sequence of runs |
| `run_id` | One `Agent.run` invocation. | one step in the sequence |
| `step_index` | Graph-node count within a run (`ctx.run_step`). | one node within one run |
`StepEvent.conversation_id` and `RunRecord.conversation_id` are populated from `ctx.conversation_id`. So three `.run()` calls sharing one `conversation_id` produce three distinct `run_id`s, all queryable as a group:
```python
import asyncio
async def main():
runs = await store.list_runs(conversation_id='conv-abc') # 3 records, chronological
asyncio.run(main())
```
## Continuing a delegate's investigation
pydantic_ai already has `message_history=` for "carry on with this prior context". `StepPersistence` does not introduce a parallel mechanism. It exposes one helper that loads the most recent provider-valid snapshot:
```python
import asyncio
from pydantic_ai import Agent
from pydantic_ai_harness.step_persistence import (
StepPersistence,
InMemoryStepStore,
continue_run,
)
store = InMemoryStepStore()
librarian = Agent(
'openai:gpt-5',
capabilities=[StepPersistence(store=store, agent_name='code_librarian')],
)
async def main():
# Earlier: tag the first turn with a conversation id so the follow-up can find it.
await librarian.run(
'Find ThinkingPartDelta and confirm the callable allowance',
conversation_id='libr-conv',
)
# Later (possibly a different process):
prior_run = (await store.list_runs(conversation_id='libr-conv'))[-1].run_id
history = await continue_run(store, run_id=prior_run)
await librarian.run(
'Read _apply_provider_details_delta and check the path',
message_history=history,
conversation_id='libr-conv', # keep the conversation grouping
)
asyncio.run(main())
```
`fork_run(store, run_id=...)` returns the same shape but is intended when the caller wants a branched logical run from that snapshot point (the new run gets a fresh `run_id` and probably a fresh `conversation_id`).
### What "safe to continue from" means
`continue_run` only returns the messages of the latest provider-valid snapshot for that `run_id`. Snapshots are written at two boundaries:
- after every `CallToolsNode` completes (all tool calls returned), and
- at `after_run`, as a fallback if the run reached no such boundary.
A run that crashed mid-tool-call has events (`tool_call_started`) but no snapshot for that point. `continue_run` returns the snapshot from the previous safe boundary, not the failed step. If no continuable snapshot exists at all, `continue_run` raises `LookupError`.
## Run lineage: `parent_run_id`
`parent_run_id` is a lineage label, not a functional dependency. It does two things:
- Every `StepEvent` and `RunRecord` carries it, so you can filter and group.
- `store.list_runs(parent_run_id='orch-1')` returns every delegate run pointing at that orchestrator.
It is auto-inferred for in-process delegation: when an orchestrator's tool synchronously calls a delegate's `Agent.run(...)`, the delegate's `StepPersistence` picks up the orchestrator's `run_id` via a `ContextVar` that the orchestrator's `wrap_run` set. No threading required:
```python
import asyncio
from pydantic_ai import Agent
from pydantic_ai_harness.step_persistence import StepPersistence, InMemoryStepStore
store = InMemoryStepStore()
orchestrator = Agent(
'openai:gpt-5',
capabilities=[StepPersistence(store=store, agent_name='orchestrator')],
)
librarian = Agent(
'openai:gpt-5',
capabilities=[StepPersistence(store=store, agent_name='code_librarian')],
)
@orchestrator.tool_plain
async def ask_librarian(question: str) -> str:
result = await librarian.run(question) # parent_run_id auto-filled
return result.output
async def main():
# Tag the orchestrator turn so the lookup below can find its run_id.
await orchestrator.run(
'Where is ThinkingPartDelta defined?',
conversation_id='orch-conv',
)
# All librarian runs now point at the orchestrator's run_id:
orch_run_id = (await store.list_runs(conversation_id='orch-conv'))[-1].run_id
delegates = await store.list_runs(parent_run_id=orch_run_id)
asyncio.run(main())
```
Set `parent_run_id=` explicitly to override (for example, cross-process delegation where `ContextVar`s do not propagate).
`parent_run_id` is distinct from `conversation_id`. The orchestrator and delegate usually live in different conversations (the orchestrator talks to a user; the delegate talks to itself). But they share a parent-child link.
## Inspecting a run tree
`list_runs` returns matches sorted by `started_at` ascending across all backends -- pick the most recent with `[-1]`.
```python
import asyncio
async def main():
# Every delegate of one orchestrator run (chronological)
delegates = await store.list_runs(parent_run_id='orch-3f2a')
# Every run in one dialogue (multi-turn conversation across many .run() calls)
turns = await store.list_runs(conversation_id='conv-abc')
latest_turn = turns[-1]
# Filters combine (AND):
focused = await store.list_runs(
parent_run_id='orch-3f2a',
conversation_id='libr-conv',
)
# Detail per run:
events = await store.list_events(run_id=delegates[0].run_id)
snapshot = await store.latest_snapshot(run_id=delegates[0].run_id)
unresolved = await store.list_unresolved_tool_effects(run_id=delegates[0].run_id)
asyncio.run(main())
```
## Failure recovery
```python
import asyncio
async def main():
# An earlier delegate run died mid-investigation.
events = await store.list_events(run_id='libr-3f2a')
unresolved = await store.list_unresolved_tool_effects(run_id='libr-3f2a')
for record in unresolved:
# status == 'started' with no terminal update -- unknown_after_crash.
print(f'tool {record.tool_name} ({record.tool_call_id}) may or may not have run')
print(f' idempotency_key={record.idempotency_key} '
f'effect_summary={record.effect_summary}')
# Decide whether to resume or branch:
history = await continue_run(store, run_id='libr-3f2a')
# If the unresolved tools were read-only and safe to redo:
await librarian.run('continue investigating', message_history=history,
conversation_id='libr-conv')
# If side effects might have happened and the orchestrator wants a fresh attempt:
history = await fork_run(store, run_id='libr-3f2a')
# ... pass to a new delegate run with a different agent_name / conversation_id.
asyncio.run(main())
```
Side-effect deduplication is the orchestrator's responsibility. Tools that write external state should annotate their in-flight `ToolEffectRecord` via `annotate_tool_effect`:
```python
from pydantic_ai import RunContext
from pydantic_ai_harness.step_persistence import annotate_tool_effect
@orchestrator.tool
async def set_label(ctx: RunContext[Deps], issue: int, label: str) -> str:
await annotate_tool_effect(
store,
ctx,
idempotency_key=f'issue-{issue}::label::{label}',
effect_summary=f'set label {label!r} on issue #{issue}',
)
await github.set_label(issue, label) # the actual side effect
return 'ok'
```
The helper reads the active `run_id` from the `StepPersistence` `ContextVar` and `tool_call_id` / `tool_name` from `ctx`, then merges the metadata into the prior record. It is a no-op when called outside a step-persistence-wrapped tool call. `after_tool_execute` preserves both fields when it writes the terminal `completed` / `failed` entry.
## Backends
- `InMemoryStepStore` -- process-local; great for tests.
- `FileStepStore(directory)` -- directory layout under `<directory>/<run_id>/`:
- `run.json` -- `RunRecord` (lineage)
- `events.jsonl` -- append-only `StepEvent`s
- `tool_effects.jsonl` -- append-only `ToolEffectRecord`s, scoped to this run
- `snapshots/{seq}.json` -- `ContinuableSnapshot`s, named by a per-run monotonic counter (not `step_index`, which would collide when the same `run_id` is reused across `Agent.run` calls, since `ctx.run_step` resets to 0 each call).
- `SqliteStepStore(database='runs.db')` -- single SQLite file with tables `runs`, `events`, `snapshots`, `tool_effects`, and a sibling `media` table for externalized blobs (see [Persisting media](#persisting-media) below). WAL mode is enabled; `tool_effects` upserts per `(run_id, tool_call_id)` so the latest state wins; snapshots use `AUTOINCREMENT seq` to mirror `FileStepStore._next_snapshot_seq`. Pass `connection=` instead of `database=` to share a `sqlite3.Connection` with the rest of your application; the connection must be opened with `check_same_thread=False` because hook calls are dispatched onto a worker thread.
All three implement the same async `StepStore` protocol, so capability hooks never block the event loop on the file/sqlite backends (I/O is dispatched via `anyio.to_thread`).
`FileStepStore` validates `run_id` against `[A-Za-z0-9_.-]{1,200}` (and rejects `..`) to prevent path traversal. Callers passing user-controlled IDs should still sanitise first.
## Persisting media
`BinaryContent` payloads (images, audio, documents, video) inlined as base64 inside a snapshot would balloon every file or row containing the message. Both `FileStepStore` and `SqliteStepStore` externalize any `BinaryContent.data` at or above **64 KiB** through a configured `MediaStore`, leaving a URI reference in the snapshot. Round-trip is transparent: `latest_snapshot(...).messages[*]` returns `BinaryContent` with the original bytes.
| StepStore | Default `media_store` | Where blobs live |
| ------------------- | --------------------------------------- | ------------------------------------- |
| `InMemoryStepStore` | not applicable | bytes stay in the in-memory snapshot |
| `FileStepStore` | `DiskMediaStore(<root>/media/)` | `<root>/media/<sha256>.bin` |
| `SqliteStepStore` | `SqliteMediaStore(database=<same db>)` | sibling `media` table in the same DB |
Override the destination by passing your own `MediaStore`:
```python
from pydantic_ai_harness.step_persistence import FileStepStore
from pydantic_ai_harness.media import S3MediaStore
store = FileStepStore(
'runs',
media_store=S3MediaStore(
bucket='my-bucket',
endpoint='https://<account>.r2.cloudflarestorage.com',
region='auto',
access_key_id=...,
secret_access_key=...,
),
media_threshold_bytes=64 * 1024, # raise or lower if you want
)
```
Opt out entirely (keep bytes inline in the snapshot JSON/row):
```python
from pydantic_ai_harness.step_persistence import FileStepStore, SqliteStepStore
FileStepStore('runs', media_store=None)
SqliteStepStore(database='runs.db', media_store=None)
```
URIs are `media+sha256://<hex>`, content-addressed. The same blob written through any `MediaStore` resolves the same way, so dedup is automatic and moving the underlying storage is a one-line swap. The shipped implementations are:
- `DiskMediaStore(directory)` -- one file per blob at `<directory>/<sha256>.bin`.
- `SqliteMediaStore(database=...)` or `SqliteMediaStore(connection=...)` -- one row per blob (`INSERT OR IGNORE` for content-addressed dedup).
- `S3MediaStore(bucket=, endpoint=, region=, access_key_id=, secret_access_key=)` -- path-style URLs plus handrolled SigV4. Compatible with AWS S3, Cloudflare R2 (`region='auto'`), MinIO, and other S3-compatible providers. PUT/GET/HEAD only -- no multipart, lifecycle, or listing in v1.
### Exposing externalized bytes as URLs
Each store accepts a `public_url=` callable that turns the canonical `media+sha256://<hex>` URI into a URL the model can fetch directly. The forthcoming `MediaExternalizer` capability will use this to swap `BinaryContent` parts for `ImageUrl` / `AudioUrl` / other URL parts before the model sees the message, letting providers fetch big media over the wire without re-encoding bytes into the request body.
Static base URL (public R2 bucket, CDN):
```python
from pydantic_ai_harness.media import S3MediaStore, make_static_public_url
store = S3MediaStore(
bucket='my-bucket',
endpoint='https://<acc>.r2.cloudflarestorage.com',
region='auto',
access_key_id=..., secret_access_key=...,
key_prefix='media/',
public_url=make_static_public_url('https://pub-abc.r2.dev', key_prefix='media/'),
)
```
Presigned or rotating-signature URL -- pass any async callable that takes `(uri, MediaContext)`:
```python
from pydantic_ai_harness.media import MediaContext, S3MediaStore
async def presign(uri: str, ctx: MediaContext) -> str:
key = 'media/' + uri.removeprefix('media+sha256://') + '.bin'
return await my_signer.generate(key, ttl=3600, content_type=ctx.media_type)
store = S3MediaStore(..., public_url=presign)
```
### `MediaContext`, an extensible per-operation bag
Every `MediaStore` method (`put`, `get`, `exists`, `public_url`, `get_metadata`) and both user-supplied callables (`PublicUrlResolver`, `KeyStrategy`) accept a `MediaContext`:
```python
from collections.abc import Mapping
from dataclasses import dataclass, field
@dataclass(frozen=True, kw_only=True)
class MediaContext:
media_type: str | None = None # e.g. 'image/png'
filename: str | None = None # original filename, when known
metadata: Mapping[str, str] = field(default_factory=dict) # user-supplied tags
```
All fields default; new fields are added non-breakingly as use cases emerge. Pass what you have, ignore the rest.
**Persistence by store.** `get_metadata(uri)` round-trips the user-supplied `metadata` mapping on all three stores. `media_type` is also persisted but is not part of what `get_metadata` returns (it is stored for the byte payload itself, for example as the `Content-Type`).
- `SqliteMediaStore` writes `metadata` to a JSON column and `media_type` to a dedicated column.
- `S3MediaStore` sends `metadata` as signed `x-amz-meta-*` headers (ASCII alphanumeric plus dash key names) and `media_type` as `Content-Type`; `get_metadata` reads the `x-amz-meta-*` values back from the HEAD response.
- `DiskMediaStore` writes a sidecar JSON file (`<resolved>.meta.json`) alongside each blob, atomic via tmp plus rename. Sidecars are absent only when the put carried no metadata.
### `key_strategy`: controlling the backend storage path
Default is `<sha256>.bin`. `DiskMediaStore` and `S3MediaStore` accept overrides to fit existing layouts; `SqliteMediaStore` does not (its primary key is the digest, so a user-chosen key would either break dedup or be a no-op):
```python
from pydantic_ai_harness.media import DiskMediaStore, MediaContext
def by_media_type(uri: str, ctx: MediaContext) -> str:
digest = uri.removeprefix('media+sha256://')
ext = {'image/png': '.png', 'image/jpeg': '.jpg'}.get(ctx.media_type or '', '.bin')
return f'images/{digest}{ext}'
store = DiskMediaStore('runs', key_strategy=by_media_type)
```
**Caveat**: if your strategy depends on `context.media_type` (for example, to pick an extension), `get(uri)` and `exists(uri)` will not find the blob unless the same context is supplied at read time. For pure path-organisation strategies (no context dependency) the constraint does not apply.
`DiskMediaStore` rejects strategies that produce absolute paths or paths containing `..` segments, to prevent escaping the store directory.
Separately, all three stores accept a `public_url=` resolver, useful when a CDN, local HTTP server, or signed-URL service fronts the bytes. Without it `public_url(...)` returns `None` (the model never sees a URL unless a resolver is configured and it returns a string).
pydantic_ai providers transparently download bytes from a URL when the target model does not natively accept that URL type, so emitting a URL is always safe: you only ever lose wire savings, never correctness.
!!! note "The future `MediaExternalizer` capability"
When it lands, the composition will be `Agent(capabilities=[MediaExternalizer(store), StepPersistence(...)])` and `StepPersistence` will see already-URL-ified messages, so the externalize walk becomes a no-op. The existing API does not change.
### Persisting to unsupported backends
DynamoDB, Postgres, Redis, GCS, and other backends are out of scope for this release. Write your own `StepStore` (about ten methods on a Protocol) or your own `MediaStore` (three methods) and pass it via `store=` / `media_store=`. Please open an issue if you ship one -- we want to feed the eventual shared adapter layer with N >= 3 real implementations before abstracting.
## What this capability does not do
- It does not restore capability per-run state, graph-node state, retry counters, or in-flight streaming responses.
- It does not deduplicate replayed side effects automatically. Tools that write artifacts, labels, PRs, or external state should call `annotate_tool_effect(store, ctx, ...)` (see [Failure recovery](#failure-recovery)) so the orchestrator can decide whether replay is safe.
- It does not clean up old snapshots or events. Retention is the caller's responsibility.
- It does not emit OpenTelemetry spans. pydantic_ai's `Instrumentation` capability already spans `agent run` / `chat` / `running tool` and populates `gen_ai.agent.name`, `gen_ai.agent.call.id`, `gen_ai.conversation.id` via baggage. A future change may add step-persistence attributes to the active span; that is tracked as a follow-up issue.
## Related
- [Capabilities overview](index.md)
- [Code Mode](code-mode.md)
## API reference
::: pydantic_ai_harness.step_persistence.StepPersistence
docs/subagents.md
+227
View File
@@ -0,0 +1,227 @@
---
title: Subagents
description: Let an agent delegate self-contained tasks to named child agents via a single delegate_task tool, with per-delegate budgets and failure handling.
---
# Subagents
`SubAgents` lets an agent delegate self-contained tasks to named child agents. It takes a sequence of `SubAgent` entries and exposes a single `delegate_task(agent_name, task)` tool. Each delegation runs the chosen sub-agent in its own run -- with its own message history, so it never sees the parent conversation -- and returns its output to the parent.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/subagents/)
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
## The problem
A single agent that does everything accumulates a large tool set and a long context. Splitting the work across specialized sub-agents keeps each context focused, but wiring up delegation by hand means writing a tool per agent, forwarding deps, threading usage limits, and telling the model what it can delegate to.
## The solution
`SubAgents` takes a sequence of `SubAgent` entries and exposes a single `delegate_task(agent_name, task)` tool. Each delegation runs the chosen sub-agent in its own run -- with its own message history, so it never sees the parent conversation -- and returns its output to the parent. The available sub-agents are listed in the system prompt as a static instruction, so the listing stays in the cached prefix.
```python
from pydantic_ai import Agent
from pydantic_ai_harness.subagents import SubAgent, SubAgents
researcher = Agent('anthropic:claude-sonnet-4-6', name='researcher', description='Researches a topic and reports findings')
writer = Agent('anthropic:claude-sonnet-4-6', name='writer', description='Turns notes into polished prose')
orchestrator = Agent(
'anthropic:claude-opus-4-7',
capabilities=[SubAgents(agents=[SubAgent(researcher), SubAgent(writer)])],
)
result = orchestrator.run_sync('Research the history of TLS and write a one-paragraph summary.')
print(result.output)
```
A delegate's name -- how the parent model refers to it, and how it is listed in the prompt -- is the agent's own `name`, or a `SubAgent(name=...)` override. Two delegates resolving to the same name is an error, and an agent with no name and no override is rejected.
## The tool
| Tool | Purpose |
|---|---|
| `delegate_task(agent_name, task)` | Run the named sub-agent on a self-contained task and return its output. |
- The sub-agent runs with its own message history, so `task` must be self-contained.
- An unknown `agent_name` raises `ModelRetry`, so the model can correct itself.
- The result returned to the parent is `str(result.output)`.
## Deps, usage, tools, and capabilities
- **Deps are forwarded.** The parent run's `deps` are passed to each sub-agent, so sub-agents share the parent's `AgentDepsT` (enforced by the type signature -- every sub-agent is an `AbstractAgent[AgentDepsT, Any]`).
- **Usage is shared by default.** The parent's `usage` is passed to each sub-agent run, so token usage aggregates and a parent `usage_limits` applies across the whole agent tree. Set `forward_usage=False` to give each sub-agent run its own accounting.
- **Tools can be inherited.** With `inherit_tools=True`, the parent agent's own tools (registered directly or via `toolsets`) are added to each sub-agent run, on top of the sub-agent's own. Tools contributed by the parent's capabilities are not inherited: they are bound to capability instances registered in the parent run, and would arrive without the hooks and instructions they depend on. Use `shared_capabilities` to give sub-agents a capability. This also excludes the delegate tool itself, so a sub-agent can't recurse into further delegation. Off by default.
- **Capabilities can be shared.** `shared_capabilities` are applied to every sub-agent run -- e.g. give all sub-agents a common guardrail, memory, or planning capability without rebuilding each `Agent`.
- **Sub-agent events can be streamed.** Pass an `event_stream_handler` and it's forwarded to each sub-agent run, so the sub-agent's model-streaming and tool events surface to the caller (the handler receives the sub-agent's own `RunContext`).
## Per-delegate run controls
Each `SubAgent` carries its own budgets, so one delegate's controls do not touch the others. A `SubAgent` with no controls set runs with the `SubAgents` defaults.
```python
from pydantic_ai import Agent
from pydantic_ai.usage import UsageLimits
from pydantic_ai_harness.subagents import SubAgent, SubAgents
reproducer = Agent('anthropic:claude-sonnet-4-6', instructions='Reproduce the reported bug from a minimal script.')
librarian = Agent('anthropic:claude-sonnet-4-6', instructions='Find relevant docs, issues, and prior art.')
orchestrator = Agent(
'anthropic:claude-opus-4-7',
capabilities=[
SubAgents(
agents=[
SubAgent(reproducer, usage_limits=UsageLimits(request_limit=35), timeout_seconds=600, max_calls=1),
SubAgent(librarian, usage_limits=UsageLimits(request_limit=18), timeout_seconds=300, max_calls=2),
]
)
],
)
```
| Field | Effect |
|---|---|
| `usage_limits` | A request/token budget for one delegation. The child runs with its own usage accounting, so the budget counts only that child's requests and tokens (not the parent's or siblings'), even when `forward_usage=True`. The tradeoff: that child's tokens no longer aggregate into the parent's `usage`. Reaching the budget is a soft outcome (see below), not a run-stopping `UsageLimitExceeded`. |
| `timeout_seconds` | A wall-clock budget for one delegation. When the child exceeds it, its run is cancelled and the parent gets a soft steering message instead of hanging on the child. The cancelled child's `event_stream_handler` (if any) stops receiving events without a terminal event. |
| `max_calls` | The maximum number of delegations to this sub-agent per parent run. Once reached, further delegations return a soft budget-exhausted message without running the child. Counts are scoped to one `Agent.run` (a `run_id`) and cleared when it ends, so each parent run and each level of a nested tree budgets independently. |
| `on_failure` | A steering message returned to the parent for any soft degradation of this delegate, in place of the built-in default. Setting it also makes child failures soft (see below). |
| `contain_errors` | Whether an unexpected crash in this delegate is caught and returned to the parent as a bounded `ModelRetry` instead of aborting the parent run (see below). Unset inherits the `SubAgents(contain_errors=...)` default (off). |
## Failure handling
A *soft outcome* returns a steering message to the parent as a normal tool result, so its model reads the message and decides what to do next (rather than immediately re-delegating, which a `ModelRetry` invites). A timeout, a reached `usage_limits` budget, and an exhausted `max_calls` budget are always soft. When `on_failure` is set, the message it carries replaces the built-in default for these outcomes.
A sub-agent run that fails with a *soft model error* (`ModelRetry`, `UnexpectedModelBehavior`, e.g. it exhausted its own retries) is, by default, converted into a `ModelRetry` for the parent -- so the parent's model sees `Sub-agent '<name>' failed: ...` and can react by re-delegating. The delegate tool defaults to `tool_retries=2`, so the parent aborts only after that many consecutive delegate failures; the counter resets after any successful delegation. Raise `tool_retries` to tolerate a flakier sub-agent, or set `None` to inherit the parent agent's default tool retries. Set `on_failure` for a delegate to make its failures soft instead: the child error returns the `on_failure` message as a normal tool result.
Hard errors propagate to stop the whole run. A `UsageLimitExceeded` from a child that has *no* per-delegate `usage_limits` (so it shares the parent's accounting) means the whole tree is out of budget and propagates; a child reaching its *own* `usage_limits` is soft, as above.
An *unexpected crash* -- any other exception the child raises, such as a provider `ModelAPIError`/`FallbackExceptionGroup` or a plain `ValueError` from a bad tool argument -- propagates by default and aborts the parent run. Set `contain_errors=True` (per delegate, or as the `SubAgents` default) to catch it and return it to the parent as a bounded `ModelRetry` instead, so one delegate crash cannot kill the whole run. Containment stays loud: the exception rides the retry message (`Sub-agent '<name>' crashed: ...`), it is logged via the standard `logging` module, and `tool_retries` still bounds consecutive crashes into an abort. This is orthogonal to `on_failure` -- a contained crash always raises the loud retry, never the soft `on_failure` return, so a genuine bug is never masked as success. Cancellation, a shared `UsageLimitExceeded`, pydantic-ai control-flow signals (`CallDeferred`, `ApprovalRequired`, the `Skip*` signals), and `UserError` always propagate regardless of `contain_errors`.
## Discovery
The sub-agents are listed in the system prompt via `get_instructions`, using each agent's `description` (or a `SubAgent(description=...)` override). A sub-agent with no description is listed by name alone.
## Loading sub-agents from disk
A repo's markdown agent definitions become delegates without writing any `Agent` code. By default every `*.md` file under the conventional folders is loaded as a sub-agent, alongside the explicitly-passed `agents`.
```python
from pydantic_ai import Agent
from pydantic_ai_harness.subagents import SubAgents
orchestrator = Agent(
'anthropic:claude-opus-4-7',
capabilities=[SubAgents(inherit_tools=True)], # auto-loads ./.agents/agents/ and ~/.agents/agents/
)
```
`agent_folders` controls where definitions come from. It defaults to `'agents'`, the conventional layout:
- A folder-name `str` (the default `'agents'`): for the project root (cwd) then the home root, load from `<root>/.agents/<name>/`, falling back to `<root>/.claude/<name>/` when `<root>/.agents/` is absent.
- A sequence of paths loads from exactly those folders, in order.
- `None` disables disk loading, exposing only the explicitly-passed `agents`.
### Definition format
A definition is a markdown file with optional frontmatter:
```markdown
---
name: researcher
description: Researches a topic and reports findings
tools: Read, Grep
---
You research topics. Report your findings, each with a source.
```
- `name` is the delegate name (how the parent refers to it and how it is listed). It falls back to the filename stem when absent.
- `description` drives the prompt listing.
- The markdown body becomes the agent's instructions.
- `tools` (or `allowed-tools`) is a comma-separated string or a YAML block list. See "Tools" below.
- `model` and `color` are ignored: the model is inherited from the parent (see below), and `color` has no pyai equivalent.
Frontmatter is read by a small, dependency-free parser limited to those keys (`pyyaml` is not a harness dependency). Full YAML frontmatter is not supported.
### Models and effort
Disk agents inherit the parent run's model by default. Per agent, the caller can override the model and set a thinking/effort level via `agent_overrides`, keyed by the agent's name:
```python
from pydantic_ai_harness.subagents import AgentOverride, SubAgents
SubAgents(
agent_folders='agents',
agent_overrides={'researcher': AgentOverride(model='anthropic:claude-sonnet-4-6', effort='high')},
)
```
Every agent the capability builds runs at a minimum thinking-effort floor. `MINIMUM_EFFORT_FLOOR` and the `clamp_effort(level, floor=...)` helper are exported so an orchestrator can apply the same floor to its own agents (that orchestrator-side application is the caller's responsibility). `clamp_effort` maps `None`/`False` to the floor, leaves `True` (provider-default effort) unchanged, and raises a concrete level below the floor up to it. Effort is applied through pyai's `ModelSettings.thinking`.
### Tools
A disk agent gets no tools by default (`inherit_tools` is `False`); set `inherit_tools=True` to expose the parent's tools to it through the `inherit_tools` mechanism, in which case its `tools` frontmatter is ignored. To map the frontmatter tool names to specific toolsets instead, pass a `tool_resolver`: it receives each tool name (so it can honor entries like `Bash(git:*)`) and returns the toolsets that provide it, or `None` for an unknown name, which is skipped with a warning.
```python
from pydantic_ai_harness.subagents import SubAgents
def resolve(tool_name: str):
return TOOLSETS.get(tool_name) # -> Sequence[AgentToolset[object]] | None
SubAgents(agent_folders='agents', tool_resolver=resolve)
```
### Precedence
When the same name appears in more than one source, the higher-precedence one wins and the others are skipped with a warning: explicitly-passed `agents` first, then the project folder, then the home folder (and, for an explicit path sequence, earlier paths before later ones). A duplicate name within the explicitly-passed `agents` list is still an error.
## Configuration
```python
SubAgents(
agents=(), # Sequence[SubAgent[AgentDepsT]] -- each pairs an agent with its run controls
agent_folders='agents',# folder-name str (convention) | Sequence[Path] | None (disable)
agent_overrides={}, # Mapping[str, AgentOverride] -- per-disk-agent model/effort override
tool_resolver=None, # Callable[[str], Sequence[AgentToolset[object]] | None] -- disk-agent tool mapping
forward_usage=True, # share the parent's usage with sub-agent runs
inherit_tools=False, # expose the parent's own tools to sub-agents (capability tools excluded)
shared_capabilities=(),# capabilities applied to every sub-agent run
event_stream_handler=None, # forwarded to each sub-agent run to stream its events
tool_name='delegate_task',
tool_retries=2, # extra delegate-tool attempts after a sub-agent error before aborting (None inherits the agent default)
contain_errors=False, # default for SubAgent.contain_errors: contain an unexpected crash as a bounded retry
)
```
```python
SubAgent(
agent, # AbstractAgent[AgentDepsT, Any] -- the child agent to run
name=None, # delegate name; defaults to the agent's own `name`
description=None, # prompt-listing description; defaults to the agent's own `description`
usage_limits=None, # per-delegation request/token budget (isolated accounting)
timeout_seconds=None, # per-delegation wall-clock budget
max_calls=None, # max delegations to this sub-agent per parent run
on_failure=None, # steering message for soft degradations of this delegate
contain_errors=None, # contain an unexpected crash as a bounded retry; None inherits the SubAgents default
)
```
`SubAgents` is not serializable via the [agent spec](/ai/core-concepts/agent-spec/) (it holds live `Agent` instances), so `get_serialization_name()` returns `None`.
## Notes
- Sub-agents can themselves have `SubAgents`, forming a tree. Share `usage` (the default) and set a `usage_limits` on the top-level run to bound the whole tree.
- Delegations the model issues in parallel run as independent sub-agent runs.
## Further reading
- [Pydantic AI capabilities](/ai/core-concepts/capabilities/)
- [Multi-agent applications](/ai/guides/multi-agent-applications/)
## API reference
::: pydantic_ai_harness.subagents.SubAgents
::: pydantic_ai_harness.subagents.SubAgent
::: pydantic_ai_harness.subagents.AgentOverride
pydantic_ai_harness/.agents/skills/pydantic-ai-harness/SKILL.md
+53 -28
View File
@@ -1,8 +1,8 @@
---
name: pydantic-ai-harness
description: Extend Pydantic AI agents with batteries-included capabilities from pydantic-ai-harness -- currently Code Mode, which collapses many tool calls into one sandboxed Python execution. Use when the user mentions pydantic-ai-harness, CodeMode, Monty, code mode, or tool sandboxing, when they want an agent to run agent-written Python, or when a Pydantic AI agent would benefit from orchestrating multiple tool calls in a single sandboxed script.
description: Extend Pydantic AI agents with batteries-included capabilities from pydantic-ai-harness -- Code Mode (collapse many tool calls into one sandboxed Python execution), a filesystem and shell, sub-agents, planning, context compaction, and more. Use when the user mentions pydantic-ai-harness, CodeMode, Monty, code mode, or tool sandboxing, when they want first-party filesystem/shell/sub-agent/planning/compaction capabilities for a Pydantic AI agent, when they want an agent to run agent-written Python, or when a Pydantic AI agent would benefit from orchestrating multiple tool calls in a single sandboxed script.
license: MIT
compatibility: Requires Python 3.10+ and pydantic-ai-slim>=1.95.1
compatibility: Requires Python 3.10+ and pydantic-ai-slim>=2.1.0
metadata:
version: "0.1.0"
author: pydantic
@@ -33,13 +33,39 @@ Do **not** use this skill for:
## Supported Capabilities
| Capability | Description | Reference |
|---|---|---|
| `CodeMode` | Wraps eligible tools into a single sandboxed `run_code` tool so the model orchestrates them in Python | [Code Mode](./references/CODE-MODE.md) |
`CodeMode` has a full reference below; it is the flagship capability and the one this skill goes deep on.
The rest ship today and each has its own README with API and examples.
More capability areas are tracked in the
[capability matrix](https://github.com/pydantic/pydantic-ai-harness#capability-matrix); as they stabilize,
this skill grows to cover them.
Each capability lives in its own submodule and is imported from there
(`from pydantic_ai_harness.<module> import ...`). Capabilities are not importable from the top-level
`pydantic_ai_harness` package by design, so each one keeps its own optional dependencies isolated.
`CodeMode`, `FileSystem`, `Shell`, and `ManagedPrompt` also have top-level re-exports (importable directly
from `pydantic_ai_harness`).
APIs are subject to change between releases; breaking changes ship deprecation warnings where practical.
| Capability | Module | Description |
|---|---|---|
| `CodeMode` | `pydantic_ai_harness.code_mode` (also top-level) | Wraps eligible tools into a single sandboxed `run_code` tool so the model orchestrates them in Python -- see [Code Mode](./references/CODE-MODE.md) |
| `FileSystem` | `pydantic_ai_harness.filesystem` (also top-level) | Read, write, edit, and search files under a root directory, with traversal prevention |
| `Shell` | `pydantic_ai_harness.shell` (also top-level) | Run commands in a subprocess with allowlists, a default denylist, timeouts, and env masking |
| `ManagedPrompt` | `pydantic_ai_harness.logfire` (also top-level) | Back an agent's instructions with a Logfire-managed prompt |
| `SubAgents` | `pydantic_ai_harness.subagents` | Delegate subtasks to specialized child agents |
| `DynamicWorkflow` | `pydantic_ai_harness.dynamic_workflow` | Orchestrate sub-agents from a model-written Python script |
| `Planning` | `pydantic_ai_harness.planning` | Break complex tasks into structured plans before execution |
| compaction family (`SlidingWindow`, `SummarizingCompaction`, ...) | `pydantic_ai_harness.compaction` | Trim or summarize conversation history to stay within token limits |
| `OverflowingToolOutput` | `pydantic_ai_harness.overflowing_tool_output` | Truncate, summarize, or spill large tool outputs |
| `RepoContext` | `pydantic_ai_harness.context` | Auto-load CLAUDE.md/AGENTS.md and repo structure |
| `StepPersistence` | `pydantic_ai_harness.step_persistence` | Save, restore, resume, and fork run state |
| `PyaiDocs` | `pydantic_ai_harness.docs` | On-demand `read_pyai_docs` tool for Pydantic AI docs |
| `RuntimeAuthoring` | `pydantic_ai_harness.runtime_authoring` | Let an agent author, validate, and load real capabilities at runtime |
| media externalization | `pydantic_ai_harness.media` | Offload large `BinaryContent` to content-addressed stores |
Still experimental: an ACP server adapter, imported from `pydantic_ai_harness.experimental.acp`. Importing it
emits a `HarnessExperimentalWarning`.
The full, current list with links and status is in the
[capability matrix](https://github.com/pydantic/pydantic-ai-harness#capability-matrix).
## Install
@@ -53,40 +79,39 @@ Each capability declares its own extra. Code Mode needs the Monty sandbox:
uv add "pydantic-ai-harness[codemode]" # `code-mode` is also accepted as an alias
```
Requires Python 3.10+ and `pydantic-ai-slim>=1.95.1`.
Requires Python 3.10+ and `pydantic-ai-slim>=2.1.0`.
## Quick Start
A harness capability is added to the agent like any other. Here `CodeMode` wraps an MCP server's tools into
a single `run_code` tool that the model drives with Python.
A harness capability is added to the agent like any other. Here `CodeMode` wraps locally registered tools
into a single `run_code` tool that the model drives with Python.
```python {test="skip"}
from pydantic_ai import Agent
from pydantic_ai.capabilities import MCP # MCP ships in core pydantic-ai
from pydantic_ai_harness import CodeMode
agent = Agent(
'anthropic:claude-sonnet-4-6',
capabilities=[
# native=False routes the MCP tools through a local toolset so CodeMode can wrap them.
# Without it, providers with native MCP run the tools server-side and bypass the sandbox.
MCP('https://hn.caseyjhand.com/mcp', native=False),
CodeMode(),
],
)
agent = Agent('anthropic:claude-sonnet-4-6', capabilities=[CodeMode()])
@agent.tool_plain
def get_temperature_f(city: str) -> float:
return {'Paris': 68.0, 'Tokyo': 77.0}[city]
@agent.tool_plain
def convert_temp(fahrenheit: float) -> float:
return round((fahrenheit - 32) * 5 / 9, 1)
result = agent.run_sync(
'Across the top and best Hacker News feeds, find the most-discussed story with at '
'least 100 points and summarize its comment thread in one paragraph.'
'Compare the weather in Paris and Tokyo, and report both temperatures in Celsius.'
)
print(result.output)
#> The most-discussed story clearing 100 points is ...
#> Paris is 20.0 C and Tokyo is 25.0 C.
```
Instead of one model round-trip per tool call, the model writes a single Python script that fetches both
feeds with `asyncio.gather`, dedupes and ranks them in plain Python, and pulls the winning thread --
collapsing many calls into one `run_code`.
The model writes a single Python script that fetches both temperatures with `asyncio.gather` and then
converts them -- performing four tool calls across two dependent stages in one `run_code` invocation.
## Key Practices
@@ -96,6 +121,6 @@ collapsing many calls into one `run_code`.
## Common Gotchas
- **`native=True` tools bypass `CodeMode`.** Provider-native MCP servers and web search execute server-side, so `run_code` never sees them. Construct them with `native=False` to keep them local and wrappable.
- **`native=True` tools bypass `CodeMode`.** Provider-native MCP servers and web search execute server-side, so `run_code` never sees them. Use `native=False` for client-side dispatch that `CodeMode` can wrap, but do not treat a remote server as trusted or sandboxed; see the [Code Mode trust boundary](./references/CODE-MODE.md#sandbox-restrictions).
- **The Monty sandbox is a Python subset.** No class definitions, no third-party imports, and only a small stdlib allowlist -- read [Code Mode](./references/CODE-MODE.md#sandbox-restrictions) before debugging generated code that fails to run.
- **`CodeMode` needs its extra.** Install `pydantic-ai-harness[codemode]`, not the bare package.
pydantic_ai_harness/.agents/skills/pydantic-ai-harness/references/CODE-MODE.md
+23 -8
View File
@@ -1,7 +1,7 @@
# Code Mode
Standard tool calling takes one model round-trip per tool call. `CodeMode` wraps eligible tools into a
single `run_code` tool so the model can write Python that loops, branches, aggregates, and parallelizes tool work inside a sandbox.
`CodeMode` wraps eligible tools into a single `run_code` tool so the model can write Python that loops,
branches, aggregates, and parallelizes multiple tool calls inside a sandbox.
Use it when the agent needs to call several tools, transform intermediate results, run concurrent
tool work with `asyncio.gather`, or anywhere a simple Python script is more reliable than the model alone, such as for mathematics.
@@ -37,6 +37,8 @@ def convert_temp(fahrenheit: float) -> float:
The model could generate code like:
```python {test="skip" lint="skip"}
import asyncio
paris, tokyo = await asyncio.gather(
get_weather(city='Paris'),
get_weather(city='Tokyo'),
@@ -46,7 +48,7 @@ tokyo_c = await convert_temp(fahrenheit=tokyo['temp_f'])
{'paris': paris_c, 'tokyo': tokyo_c}
```
This reduces four model round-trips to one.
This performs four tool calls, across two dependent stages, inside one `run_code` invocation.
## Choose Which Tools Are Sandboxed
@@ -57,7 +59,7 @@ from pydantic_ai_harness import CodeMode
CodeMode(tools='all')
CodeMode(tools=['search', 'fetch'])
CodeMode(tools=lambda ctx, td: td.name != 'dangerous_tool')
CodeMode(tools=lambda ctx, td: td.name in {'search', 'fetch'})
CodeMode(tools={'code_mode': True})
```
@@ -121,8 +123,17 @@ Key restrictions:
- No third-party imports
- No `import *`
- Only a small stdlib subset is allowed: `sys`, `typing`, `asyncio`, `math`, `json`, `re`, `datetime`, `os`, `pathlib`
- No wall-clock or timing primitives: `asyncio.sleep`, `datetime.datetime.now()`, `datetime.date.today()`, and the `time` module are unavailable
- Tools that need approval or deferred execution are excluded from the sandbox
- No wall-clock or timing primitives by default: `datetime.datetime.now()` and `datetime.date.today()` require an `os_access` handler; `asyncio.sleep` and the `time` module are unavailable
- Filesystem I/O requires an `os_access` handler or a `mount`; `os.getenv` and `os.environ` require an `os_access` handler
- Tools requiring approval or with deferred (`CallDeferred`) execution are sandboxed like any other tool; without a `HandleDeferredToolCalls` (or equivalent) capability to resolve them inline, calling one from `run_code` raises an error that surfaces to the model as a retry
Leave `os_access` and `mount` unset unless the task requires host access, and grant only the resources
the task needs. A mount defaults to copy-on-write `mode='overlay'`; use `mode='read-only'` to forbid
writes or `mode='read-write'` only when sandbox writes must persist on the host.
The sandbox constrains the model-generated Python, not the implementation of the tools it calls.
Wrapped tools retain their normal host and network access, so expose only tools with the authority and
input validation appropriate for model-generated calls.
When a generated example keeps failing, check these restrictions before changing the rest of the agent.
@@ -130,8 +141,12 @@ When a generated example keeps failing, check these restrictions before changing
```python {test="skip" lint="skip"}
CodeMode(
tools: ToolSelector = 'all', # 'all', list[str], callable, or dict
max_retries: int = 3, # retries on sandbox execution errors
tools: ToolSelector = 'all', # 'all', list[str], callable, or dict
max_retries: int = 3, # retries on sandbox execution errors
*,
os_access: CodeModeOS | None = None, # host handler for env vars, clock, and file I/O
mount: CodeModeMount | None = None, # host directories to share with the sandbox
dynamic_catalog: bool = False, # move the tool catalog into dynamic instructions
)
```
pydantic_ai_harness/__init__.py
+33
View File
@@ -5,23 +5,56 @@ from typing import TYPE_CHECKING
if TYPE_CHECKING:
from .code_mode import CodeMode
from .filesystem import FileSystem
from .guardrails import (
GuardrailError,
GuardResult,
InputBlocked,
InputGuard,
InputGuardFunc,
OutputBlocked,
OutputGuard,
OutputGuardFunc,
)
from .logfire import ManagedPrompt
from .shell import LLM_API_KEY_ENV_PATTERNS, Shell
__all__ = [
'CodeMode',
'FileSystem',
'GuardResult',
'GuardrailError',
'InputBlocked',
'InputGuard',
'InputGuardFunc',
'LLM_API_KEY_ENV_PATTERNS',
'ManagedPrompt',
'OutputBlocked',
'OutputGuard',
'OutputGuardFunc',
'Shell',
]
_GUARDRAIL_EXPORTS = {
'GuardResult',
'GuardrailError',
'InputBlocked',
'InputGuard',
'InputGuardFunc',
'OutputBlocked',
'OutputGuard',
'OutputGuardFunc',
}
def __getattr__(name: str) -> object:
if name == 'CodeMode':
from .code_mode import CodeMode
return CodeMode
if name in _GUARDRAIL_EXPORTS:
from . import guardrails
return getattr(guardrails, name)
if name == 'FileSystem':
from .filesystem import FileSystem
pydantic_ai_harness/_monty_exec.py
+224
View File
@@ -0,0 +1,224 @@
"""Shared Monty execution loop for code-execution capabilities.
Drives a Monty REPL via the synchronous snapshot API (`feed_start`/`resume`),
dispatching external function calls back to a host-supplied async callback.
Two capabilities build on this:
- `code_mode`: the dispatch callback runs the agent's own tools.
- `dynamic_workflow`: the dispatch callback runs sub-agents.
The synchronous snapshot API (rather than `feed_run_async`) is used deliberately:
it avoids background threads and `call_soon_threadsafe`, so the loop is safe inside
restricted event loops such as Temporal's workflow sandbox.
"""
from __future__ import annotations
import asyncio
from collections.abc import Callable, Container, Coroutine
from dataclasses import dataclass, field
from typing import Any
try:
from pydantic_monty import (
AbstractOS,
ExternalException,
ExternalResult,
ExternalReturnValue,
FunctionSnapshot,
FutureSnapshot,
MontyComplete,
MountDir,
NameLookupSnapshot,
OsFunction,
)
except ImportError as _import_error: # pragma: no cover
raise ImportError(
'pydantic-monty is required for code-execution capabilities. Install it with: '
'pip install "pydantic-ai-harness[code-mode]" or "pydantic-ai-harness[dynamic-workflow]"'
) from _import_error
# Dispatch callback: given the sandbox function name and keyword arguments,
# perform the host-side work (tool call or sub-agent run) and return the result.
DispatchFn = Callable[[str, dict[str, Any]], Coroutine[Any, Any, Any]]
MontyState = FunctionSnapshot | FutureSnapshot | NameLookupSnapshot | MontyComplete
# OS handler and host-directory mounts that route environment, clock, and filesystem
# calls from inside the sandbox. A raw callback or a ready-made `AbstractOS` is accepted.
MontyOSCallback = Callable[[OsFunction, tuple[object, ...], dict[str, object]], object]
MontyOS = AbstractOS | MontyOSCallback
MontyMount = MountDir | list[MountDir]
# A coroutine not yet scheduled on the event loop, or its running Task.
PendingCall = asyncio.Task[Any] | Coroutine[Any, Any, Any]
def is_sandbox_panic(exc: BaseException) -> bool:
"""Whether `exc` is a Rust-side sandbox panic surfacing through pyo3.
pyo3 raises `pyo3_runtime.PanicException`, a `BaseException` (not `Exception`) subclass
from a module that cannot be imported, so it is matched by name. The model can provoke a
panic from inside the sandbox (e.g. awaiting the same external call twice in one
`asyncio.gather`), so callers should convert it to a retry rather than let it tear down
the whole agent run.
"""
return type(exc).__name__ == 'PanicException'
class PrintCapture:
"""Accumulates print-callback chunks from a Monty REPL."""
def __init__(self) -> None:
self._chunks: list[str] = []
def __call__(self, _stream: str, text: str) -> None:
self._chunks.append(text)
@property
def joined(self) -> str:
return ''.join(self._chunks)
def prepend_to(self, error_message: str) -> str:
"""Prefix captured stdout to an error message, so the model sees what printed before the error."""
printed = self.joined.rstrip('\n')
if not printed:
return error_message
return f'[stdout before error]\n{printed}\n[/stdout before error]\n{error_message}'
@dataclass
class MontyExecutor:
"""Drives a Monty REPL to completion, dispatching external calls to a host callback.
Single-use: it accumulates per-run state in `_pending`/`_pre_resolved`, so construct a
fresh executor for each `run` rather than reusing or sharing one across concurrent runs.
External calls are handled by execution mode:
- **Parallel** (`async def`): deferred via `resume({'future': ...})` and eagerly
scheduled as `asyncio.Task`s. Resolved at `FutureSnapshot` via `asyncio.gather`.
- **Per-call sequential** (`def`, name in `sequential_names`): resolved inline at
`FunctionSnapshot`. Any pending parallel tasks are awaited first (barrier).
- **Global sequential** (DBOS/Temporal): all calls deferred but stored as bare
coroutines and awaited one-at-a-time to prevent interleaving.
"""
dispatch: DispatchFn
valid_names: Container[str]
sequential_names: set[str] = field(default_factory=set[str])
global_sequential: bool = False
# OS handler and mounts. Monty auto-dispatches OS calls only while every `resume`
# carries them, so they are threaded through each resume below, not just `feed_start`.
os_access: MontyOS | None = None
mount: MontyMount | None = None
# Parallel calls deferred but not yet resolved, keyed by Monty call id.
_pending: dict[int, PendingCall] = field(default_factory=dict[int, PendingCall], init=False)
# Parallel results awaited early at a sequential barrier, before their FutureSnapshot is reached.
_pre_resolved: dict[int, ExternalResult] = field(default_factory=dict[int, ExternalResult], init=False)
async def run(self, state: MontyState) -> MontyComplete:
"""Drive the REPL from `state` until it completes."""
try:
while not isinstance(state, MontyComplete):
if isinstance(state, NameLookupSnapshot):
state = state.resume(os=self.os_access, mount=self.mount)
elif isinstance(state, FunctionSnapshot):
state = await self._handle_function(state)
else:
state = await self._resolve_futures(state)
finally:
cancelled: list[asyncio.Task[Any]] = []
for call in self._pending.values():
if isinstance(call, asyncio.Task):
call.cancel()
cancelled.append(call)
else:
call.close()
if cancelled:
# `cancel()` only schedules a `CancelledError` at each task's next suspension
# point; await them so dispatched work (e.g. sub-agent runs mutating shared
# usage) has fully unwound before this returns. `return_exceptions=True` keeps
# one task's teardown error from masking the original exception, and the
# results are deliberately discarded.
await asyncio.gather(*cancelled, return_exceptions=True)
return state
async def _handle_function(self, snapshot: FunctionSnapshot) -> MontyState:
"""Dispatch (or defer) a single external function call."""
name = snapshot.function_name
if name not in self.valid_names:
return snapshot.resume(
{'exception': NameError(f'Unknown function: {name}')}, os=self.os_access, mount=self.mount
)
if snapshot.args:
return snapshot.resume(
{'exception': TypeError(f'{name}() does not accept positional arguments; use keyword arguments')},
os=self.os_access,
mount=self.mount,
)
if name in self.sequential_names:
# Rendered as `def` (sync), so the sandbox code doesn't `await` the result --
# resolve inline. Await pending parallel tasks first (barrier) for ordering.
# The dispatch coroutine is created only after the barrier: it is not in
# `_pending`, so if it existed while the barrier awaits and we were cancelled
# there, `run`'s cleanup would never close it.
for cid in list(self._pending):
self._pre_resolved[cid] = await _await_external(self._pending.pop(cid))
# The wrapped outcome (`{'return_value': ...}` / `{'exception': ...}`) is already
# exactly the payload `resume` expects.
return snapshot.resume(
await _await_external(self.dispatch(name, snapshot.kwargs)), os=self.os_access, mount=self.mount
)
# Deferred execution -- resolved later at FutureSnapshot.
call = self.dispatch(name, snapshot.kwargs)
if self.global_sequential:
# Keep the bare coroutine unscheduled; it's awaited one-at-a-time to avoid interleaving.
self._pending[snapshot.call_id] = call
else:
# Schedule now as a Task so concurrently-deferred calls actually run in parallel.
self._pending[snapshot.call_id] = asyncio.ensure_future(call)
return snapshot.resume({'future': ...}, os=self.os_access, mount=self.mount)
async def _resolve_futures(self, snapshot: FutureSnapshot) -> MontyState:
"""Resolve the deferred calls a `FutureSnapshot` is waiting on."""
pending_ids = snapshot.pending_call_ids
results: dict[int, ExternalResult] = {}
for cid in pending_ids:
if cid in self._pre_resolved:
results[cid] = self._pre_resolved.pop(cid)
elif self.global_sequential:
results[cid] = await _await_external(self._pending.pop(cid))
# Gather any remaining parallel tasks concurrently. They stay in `_pending` until
# gather returns, so the cleanup in `run` can still cancel them if this is cancelled.
gather_ids = [cid for cid in pending_ids if cid not in results]
if gather_ids:
settled = await asyncio.gather(*(self._pending[cid] for cid in gather_ids), return_exceptions=True)
for cid, outcome in zip(gather_ids, settled):
del self._pending[cid]
results[cid] = _wrap_gathered(outcome)
return snapshot.resume(results=results, os=self.os_access, mount=self.mount)
async def _await_external(call: PendingCall) -> ExternalReturnValue | ExternalException:
"""Await a single deferred call and wrap its outcome for Monty."""
try:
result = await call
except Exception as exc:
return ExternalException(exception=exc)
return ExternalReturnValue(return_value=result)
def _wrap_gathered(outcome: Any) -> ExternalReturnValue | ExternalException:
"""Wrap an `asyncio.gather(return_exceptions=True)` outcome for Monty."""
if isinstance(outcome, Exception):
return ExternalException(exception=outcome)
if isinstance(outcome, BaseException): # pragma: no cover
raise outcome
return ExternalReturnValue(return_value=outcome)
pydantic_ai_harness/code_mode/README.md
+7 -4
View File
@@ -2,6 +2,8 @@
Replace individual tool calls with a single sandboxed Python execution environment.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/code_mode/)
## The problem
Standard tool calling requires one model round-trip per tool call. An agent that needs to fetch 10 items and process each one makes 11+ model calls -- slow, expensive, and context-heavy.
@@ -57,7 +59,7 @@ The [harness Quick start](../../README.md#quick-start) wires `CodeMode` up again
[![CodeMode's first run_code: parallel asyncio.gather over three HN feeds, then a dedupe and a score filter](../../docs/images/code-mode-trace.png)](https://logfire-us.pydantic.dev/public-trace/84bcf123-2106-49da-9f6f-5c26395339bb?spanId=7650806a0785b946)
**[See the full Logfire trace →](https://logfire-us.pydantic.dev/public-trace/84bcf123-2106-49da-9f6f-5c26395339bb?spanId=7650806a0785b946)** Each `run_code` span fans out into the tool calls the model issued from inside the sandbox -- the easiest way to understand what code mode actually did. See the [Pydantic AI Logfire docs](https://ai.pydantic.dev/logfire/) for setup details.
**[See the full Logfire trace ->](https://logfire-us.pydantic.dev/public-trace/84bcf123-2106-49da-9f6f-5c26395339bb?spanId=7650806a0785b946)** Each `run_code` span fans out into the tool calls the model issued from inside the sandbox -- the easiest way to understand what code mode actually did. See the [Pydantic AI Logfire docs](https://ai.pydantic.dev/logfire/) for setup details.
## Installation
@@ -239,19 +241,20 @@ Code runs inside [Monty](https://github.com/pydantic/monty), a sandboxed Python
- No class definitions
- No third-party imports (allowed stdlib: `sys`, `typing`, `asyncio`, `math`, `json`, `re`, `datetime`, `os`, `pathlib`)
- No wall-clock or timing primitives by default (`asyncio.sleep`, `datetime.now()`, `date.today()`, `time`) -- `datetime.now()`/`date.today()` become available with an `os_access` handler (above); `asyncio.sleep`/`time` never do
- No wall-clock or timing primitives by default (`asyncio.sleep`, `datetime.datetime.now()`, `datetime.date.today()`, `time`) -- `datetime.datetime.now()`/`datetime.date.today()` become available with an `os_access` handler (above); `asyncio.sleep`/`time` never do
- No `import *`
- Filesystem I/O needs an `os_access` handler or a `mount`; `os.getenv`/`os.environ` need an `os_access` handler
- Tools requiring approval or with deferred execution are excluded from the sandbox
- Tools requiring approval or with deferred (`CallDeferred`) execution are sandboxed like any other tool; without a `HandleDeferredToolCalls` (or equivalent) capability on the agent to resolve them inline, calling one from `run_code` raises an error that surfaces to the model as a retry
## API
```python
```python {test="skip"}
CodeMode(
tools: ToolSelector = 'all', # 'all', list[str], callable, or dict
max_retries: int = 3, # retries on sandbox execution errors
os_access: CodeModeOS | None = None, # host handler for env vars, clock, and file I/O
mount: CodeModeMount | None = None, # host directories to share with the sandbox
dynamic_catalog: bool = False, # keep run_code's description cache-stable; catalog moves into instructions
)
```
pydantic_ai_harness/code_mode/_toolset.py
+41 -228
View File
@@ -2,12 +2,11 @@
from __future__ import annotations
import asyncio
import inspect
import keyword
import re
import warnings
from collections.abc import Callable, Coroutine, Sequence
from collections.abc import Callable, Sequence
from dataclasses import dataclass, field, replace
from typing import Annotated, Any
@@ -35,19 +34,12 @@ except ImportError: # pragma: no cover
try:
from pydantic_monty import (
AbstractOS,
ExternalException,
ExternalResult,
ExternalReturnValue,
FunctionSnapshot,
FutureSnapshot,
Monty,
MontyComplete,
MontyRepl,
MontyRuntimeError,
MontySyntaxError,
MontyTypingError,
MountDir,
NameLookupSnapshot,
OsFunction,
)
except ImportError as _import_error: # pragma: no cover
@@ -56,8 +48,7 @@ except ImportError as _import_error: # pragma: no cover
) from _import_error
from typing_extensions import NotRequired, TypedDict
# Type alias for the dispatch callback passed to _execution_loop.
_DispatchFn = Callable[[str, dict[str, Any]], Coroutine[Any, Any, Any]]
from pydantic_ai_harness._monty_exec import MontyExecutor, PrintCapture, is_sandbox_panic
# A raw OS callback. Return `pydantic_monty.NOT_HANDLED` to defer the call to the
# sandbox's default, which leaves it unavailable.
@@ -183,6 +174,18 @@ _TOOL_SEARCH_ADDENDUM = (
_INVALID_IDENT_CHARS = re.compile(r'[^a-zA-Z0-9_]')
def _is_code_execution_tool(tool_def: ToolDefinition) -> bool:
"""Whether a tool is itself a code-execution sandbox that takes a code string.
Such tools (this `run_code`, or DynamicWorkflow's `run_workflow`) carry `code_arg_name`
metadata -- the same marker instrumentation reads to render the argument as code. They must
not be folded into `run_code`: nesting one code sandbox inside another would make the model
write a script that passes a second script as a string literal. They stay native so the two
code surfaces sit side by side.
"""
return bool(tool_def.metadata and 'code_arg_name' in tool_def.metadata)
def _sanitize_tool_name(name: str) -> str:
"""Turn a tool name into a valid Python identifier.
@@ -352,6 +355,10 @@ class CodeModeToolset(WrapperToolset[AgentDepsT]):
# Keep the local fallback native so `Model.prepare_request` can drop it
# when the provider supports the native tool.
native_tools[name] = tool
elif _is_code_execution_tool(tool.tool_def):
# A tool that is itself a code-execution sandbox (e.g. DynamicWorkflow's
# `run_workflow`) is a peer of `run_code`, not something to fold inside it.
native_tools[name] = tool
elif await matches_tool_selector(self.tool_selector, ctx, tool.tool_def):
sandboxed_tools[name] = tool
else:
@@ -459,7 +466,7 @@ class CodeModeToolset(WrapperToolset[AgentDepsT]):
nested_returns: dict[str, ToolReturnPart] = {}
call_counter = 0
async def dispatch_tool_call(original_name: str, kwargs: dict[str, Any]) -> Any:
async def dispatch_tool_call(sandbox_name: str, kwargs: dict[str, Any]) -> Any:
"""Dispatch a single tool call from inside the sandbox.
Returns the serialized tool result on success. On failure, the
@@ -468,6 +475,7 @@ class CodeModeToolset(WrapperToolset[AgentDepsT]):
`await` site.
"""
nonlocal call_counter
original_name = sanitized_to_original.get(sandbox_name, sandbox_name)
call_counter += 1
parent_id = ctx.tool_call_id or 'pyd_ai_code_mode'
tool_call_id = f'{parent_id}__{call_counter}'
@@ -531,24 +539,22 @@ class CodeModeToolset(WrapperToolset[AgentDepsT]):
self._repl = MontyRepl()
assert self._repl is not None
capture = _PrintCapture()
capture = PrintCapture()
try:
monty_state = self._repl.feed_start(code, print_callback=capture, os=self.os_access, mount=self.mount)
completed = await _execution_loop(
monty_state,
completed = await MontyExecutor(
dispatch=dispatch_tool_call,
callable_defs=callable_defs,
sanitized_to_original=sanitized_to_original,
sequential_tools=sequential_tools,
valid_names=callable_defs,
sequential_names=sequential_tools,
global_sequential=global_sequential,
os_access=self.os_access,
mount=self.mount,
)
).run(monty_state)
except MontySyntaxError as e:
raise ModelRetry(f'Syntax error in code:\n{_prepend_prints(e.display(), capture)}') from e
raise ModelRetry(f'Syntax error in code:\n{capture.prepend_to(e.display())}') from e
except MontyTypingError as e: # pragma: no cover -- MontyRepl.feed_start doesn't raise this
raise ModelRetry(f'Type error in code:\n{_prepend_prints(e.display(), capture)}') from e
raise ModelRetry(f'Type error in code:\n{capture.prepend_to(e.display())}') from e
except MontyRuntimeError as e:
# Exceptions raised inside dispatch_tool_call (e.g. UserError from
# ApprovalRequired, or ModelRetry from a wrapped tool) are passed
@@ -559,7 +565,20 @@ class CodeModeToolset(WrapperToolset[AgentDepsT]):
# ModelRetry from a wrapped tool gets double-wrapped
# (ModelRetry → MontyRuntimeError → ModelRetry), but the retry
# semantics are the same -- the model gets another chance.
raise ModelRetry(f'Runtime error:\n{_prepend_prints(e.display(), capture)}') from e
raise ModelRetry(f'Runtime error:\n{capture.prepend_to(e.display())}') from e
except BaseException as e:
# Convert a model-provokable sandbox panic to a retry (see `is_sandbox_panic`);
# anything else (CancelledError, ...) re-raises unchanged.
if not is_sandbox_panic(e):
raise
# The panic aborts the VM mid-execution, so the REPL's accumulated state cannot
# be trusted; drop it so the retry starts from a fresh, type-checked session.
self._repl = None
raise ModelRetry(
'The code aborted inside the sandbox and the session was reset. This can happen '
'when the same tool call is awaited more than once in one asyncio.gather -- give '
'each gathered call its own invocation. Revise the code and try again.'
) from e
result = completed.output
printed = capture.joined
@@ -721,194 +740,6 @@ def _get_sigs_and_conflicting(
return sigs, FunctionSignature.get_conflicting_type_names(sigs)
async def _execution_loop(
monty_state: FunctionSnapshot | FutureSnapshot | NameLookupSnapshot | MontyComplete,
*,
dispatch: _DispatchFn,
callable_defs: dict[str, ToolDefinition],
sanitized_to_original: dict[str, str],
sequential_tools: set[str],
global_sequential: bool,
os_access: CodeModeOS | None,
mount: CodeModeMount | None,
) -> MontyComplete:
"""Drive the Monty REPL via the synchronous snapshot API until completion.
Uses Monty's `feed_start`/`resume` snapshot API instead of `feed_run_async`
to avoid background threads and `call_soon_threadsafe`. This makes it safe
to run inside restricted event loops like Temporal's workflow sandbox.
Tool calls are handled based on their execution mode:
- **Parallel tools** (`async def`): deferred via `resume({'future': ...})`
and eagerly scheduled as `asyncio.Task`s for concurrent execution.
Resolved at `FutureSnapshot` via `asyncio.gather`.
- **Sequential tools** (`def`): resolved inline at `FunctionSnapshot`
via `resume({'return_value': ...})` or `resume({'exception': ...})`. Before
dispatching, any pending parallel tasks are awaited to maintain ordering.
- **Global sequential mode** (DBOS/Temporal): all tools are deferred via
`resume({'future': ...})` but stored as bare coroutines and awaited
one-at-a-time at `FutureSnapshot` to prevent interleaving.
`os`/`mount` must be passed to every `resume` call (not just `feed_start`):
Monty's auto-dispatch of OS calls stops the moment a resume omits them.
"""
pending: dict[int, asyncio.Task[Any] | Coroutine[Any, Any, Any]] = {}
# Results from parallel tasks that were awaited early (at a sequential-tool
# barrier) but whose FutureSnapshot hasn't been reached yet.
pre_resolved: dict[int, ExternalResult] = {}
try:
while not isinstance(monty_state, MontyComplete):
if isinstance(monty_state, NameLookupSnapshot):
monty_state = monty_state.resume(os=os_access, mount=mount)
elif isinstance(monty_state, FunctionSnapshot):
monty_state = await _handle_function_snapshot(
monty_state,
dispatch,
callable_defs,
sanitized_to_original,
sequential_tools=sequential_tools,
global_sequential=global_sequential,
pending=pending,
pre_resolved=pre_resolved,
os_access=os_access,
mount=mount,
)
else:
monty_state = await _resolve_future_snapshot(
monty_state,
pending=pending,
pre_resolved=pre_resolved,
global_sequential=global_sequential,
os_access=os_access,
mount=mount,
)
finally:
for item in pending.values(): # pragma: no cover
if isinstance(item, asyncio.Task):
item.cancel()
else:
item.close()
return monty_state
async def _handle_function_snapshot(
snapshot: FunctionSnapshot,
dispatch: _DispatchFn,
callable_defs: dict[str, ToolDefinition],
sanitized_to_original: dict[str, str],
*,
sequential_tools: set[str],
global_sequential: bool,
pending: dict[int, asyncio.Task[Any] | Coroutine[Any, Any, Any]],
pre_resolved: dict[int, ExternalResult],
os_access: CodeModeOS | None,
mount: CodeModeMount | None,
) -> FunctionSnapshot | FutureSnapshot | NameLookupSnapshot | MontyComplete:
"""Handle a single FunctionSnapshot from the Monty execution loop."""
fn_name = snapshot.function_name
if fn_name not in callable_defs:
return snapshot.resume({'exception': NameError(f'Unknown function: {fn_name}')}, os=os_access, mount=mount)
if snapshot.args:
return snapshot.resume(
{'exception': TypeError(f'{fn_name}() does not accept positional arguments; use keyword arguments')},
os=os_access,
mount=mount,
)
original_name = sanitized_to_original.get(fn_name, fn_name)
if fn_name in sequential_tools:
# Per-tool sequential: rendered as `def` (sync), so must resolve inline --
# the sandbox code doesn't `await` the result. Await pending parallel
# tasks first (barrier) to maintain ordering.
for cid in list(pending):
pre_resolved[cid] = await _resolve_coro(pending.pop(cid))
outcome = await _resolve_coro(dispatch(original_name, snapshot.kwargs))
if 'return_value' in outcome:
return snapshot.resume({'return_value': outcome['return_value']}, os=os_access, mount=mount)
return snapshot.resume({'exception': outcome['exception']}, os=os_access, mount=mount)
# Deferred execution -- store for later resolution at FutureSnapshot.
if global_sequential:
# Bare coroutine -- don't schedule on the event loop yet.
pending[snapshot.call_id] = dispatch(original_name, snapshot.kwargs)
else:
# Eagerly schedule as a Task for concurrent execution.
pending[snapshot.call_id] = asyncio.ensure_future(dispatch(original_name, snapshot.kwargs))
return snapshot.resume({'future': ...}, os=os_access, mount=mount)
async def _resolve_future_snapshot(
snapshot: FutureSnapshot,
*,
pending: dict[int, asyncio.Task[Any] | Coroutine[Any, Any, Any]],
pre_resolved: dict[int, ExternalResult],
global_sequential: bool,
os_access: CodeModeOS | None,
mount: CodeModeMount | None,
) -> FunctionSnapshot | FutureSnapshot | NameLookupSnapshot | MontyComplete:
"""Resolve pending tool calls at a FutureSnapshot."""
pending_ids = snapshot.pending_call_ids
if not pending_ids: # pragma: no cover
return snapshot.resume(results={}, os=os_access, mount=mount)
results: dict[int, ExternalResult] = {}
for cid in pending_ids:
if cid in pre_resolved:
results[cid] = pre_resolved.pop(cid)
elif global_sequential:
results[cid] = await _resolve_coro(pending.pop(cid))
# Gather remaining parallel tasks.
gather_ids = [cid for cid in pending_ids if cid not in results]
if gather_ids:
tasks = [pending[cid] for cid in gather_ids]
settled = await asyncio.gather(*tasks, return_exceptions=True)
for cid in gather_ids:
del pending[cid]
for cid, outcome in zip(gather_ids, settled):
results[cid] = _settle_outcome(outcome)
return snapshot.resume(results=results, os=os_access, mount=mount)
async def _resolve_coro(
coro: Coroutine[Any, Any, Any] | asyncio.Task[Any],
) -> ExternalReturnValue | ExternalException:
"""Await a single coroutine/task and wrap the result for Monty."""
try:
result = await coro
except Exception as exc:
return ExternalException(exception=exc)
else:
return ExternalReturnValue(return_value=result)
def _settle_outcome(outcome: Any) -> ExternalReturnValue | ExternalException:
"""Wrap an `asyncio.gather(return_exceptions=True)` outcome for Monty."""
if isinstance(outcome, Exception):
return ExternalException(exception=outcome)
if isinstance(outcome, BaseException): # pragma: no cover
raise outcome
return ExternalReturnValue(return_value=outcome)
def _prepend_prints(error_message: str, capture: _PrintCapture) -> str:
"""Prepend any captured print output to an error message.
When sandbox code prints debug output before crashing, this preserves
that output in the error so the model can use it for debugging.
"""
printed = capture.joined.rstrip('\n')
if not printed:
return error_message
return f'[stdout before error]\n{printed}\n[/stdout before error]\n{error_message}'
def _contains_multimodal(value: Any) -> bool:
"""Check if a value is or directly contains multimodal content (images, audio, etc.)."""
if is_multi_modal_content(value):
@@ -916,21 +747,3 @@ def _contains_multimodal(value: Any) -> bool:
if isinstance(value, list):
return any(is_multi_modal_content(item) for item in value) # pyright: ignore[reportUnknownVariableType]
return False
class _PrintCapture:
"""Accumulates print-callback chunks from the Monty REPL.
Pulled out to module scope (rather than a closure inside `call_tool`) so
the callback path is testable in isolation and visible to coverage.py.
"""
def __init__(self) -> None:
self._chunks: list[str] = []
def __call__(self, _stream: str, text: str) -> None:
self._chunks.append(text)
@property
def joined(self) -> str:
return ''.join(self._chunks)
pydantic_ai_harness/experimental/compaction/README.md → pydantic_ai_harness/compaction/README.md
+35 -22
View File
@@ -1,32 +1,24 @@
# Compaction capabilities
# Compaction
> [!WARNING]
> **Experimental.** These capabilities live under `pydantic_ai_harness.experimental` and may
> change or be removed in any release, without a deprecation period. Import them from the
> experimental path -- there is no top-level export:
> [!NOTE]
> Import these capabilities from their submodule -- there is no top-level `pydantic_ai_harness` re-export:
>
> ```python
> from pydantic_ai_harness.experimental.compaction import TieredCompaction
> from pydantic_ai_harness.compaction import TieredCompaction
> ```
>
> Importing any experimental capability emits a `HarnessExperimentalWarning`. Silence **all**
> harness experimental warnings with a single filter (no per-capability lines needed):
>
> ```python
> import warnings
> from pydantic_ai_harness.experimental import HarnessExperimentalWarning
>
> warnings.filterwarnings('ignore', category=HarnessExperimentalWarning)
> ```
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
A menu of strategies for keeping an agent's conversation history within a model's context
window. Each is a Pydantic AI `Capability` that runs in the `before_model_request` hook; edits
**persist** into the run's message history, so a trim/clear/summary carries forward to later
window. Each is a Pydantic AI `Capability` that edits the message history just before each
request goes out; edits **persist** into the run's message history, so a trim/clear/summary carries forward to later
steps (it is not recomputed from the full history every turn).
All strategies preserve tool-call / tool-return **pairing** -- core does not validate this, and a
provider rejects an orphaned pair. The zero-LLM strategies never call a model.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/compaction/)
## The menu
| Capability | Cost | What it does | Reach for it when |
@@ -36,7 +28,7 @@ provider rejects an orphaned pair. The zero-LLM strategies never call a model.
| `ClearToolResults` | zero-LLM | Blanks the content of old tool *results* in place, keeping the last `keep_pairs` | Tool outputs dominate context and can be re-fetched on demand (the cheap first tier) |
| `DeduplicateFileReads` | zero-LLM | Blanks every file read superseded by a newer read of the same file | The agent re-reads files and only the latest version matters |
| `SummarizingCompaction` | one LLM call | Summarizes older messages into a structured summary, keeping the recent tail | Old context still matters but must be compressed; use behind the cheap tiers |
| `TieredCompaction` | escalates | Runs cheap passes first, summarizes only if still over `target_tokens` | You want the SOTA default: spend the expensive summary only when needed |
| `TieredCompaction` | escalates | Runs cheap passes first, summarizes only if still over `target_tokens` | You want a sensible default: spend the expensive summary only when needed |
| `LimitWarner` | zero-LLM | Injects an URGENT/CRITICAL warning as limits approach | You want the agent to wrap up rather than have its history rewritten |
## Triggers
@@ -62,7 +54,7 @@ low-entropy repetition, so a head/tail slice loses little.
```python
from pydantic_ai import Agent
from pydantic_ai_harness.experimental.compaction import ClampOversizedMessages
from pydantic_ai_harness.compaction import ClampOversizedMessages
agent = Agent(
'openai:gpt-4o',
@@ -89,7 +81,7 @@ user input should not be silently rewritten, and oversized tool returns are the
Use it as the first tier of `TieredCompaction`, before `ClearToolResults`:
```python
from pydantic_ai_harness.experimental.compaction import (
from pydantic_ai_harness.compaction import (
ClampOversizedMessages,
ClearToolResults,
TieredCompaction,
@@ -104,6 +96,23 @@ TieredCompaction(
)
```
## `SlidingWindow` and `ClearToolResults` options
`SlidingWindow` keeps the last `keep_messages` down to a tail; pass `keep_tokens` instead for a token
budget rather than a message count. By default `preserve_first_user_message=True` keeps the first user
turn even when it falls outside the window, so the agent does not lose the original task.
`ClearToolResults` keeps the last `keep_pairs` intact. Set `clear_tool_inputs=True` to also blank the
arguments of the cleared calls, and `exclude_tools` to a set of tool names whose results are never
cleared.
## `LimitWarner` thresholds
Warnings begin at `warning_threshold` (default `0.7`, a fraction of the limit) and escalate to CRITICAL
for iterations once the remaining request count drops to `critical_remaining_iterations` (default `3`).
It watches `max_iterations`, `max_context_tokens`, and `max_total_tokens`, warning on whichever are
configured; narrow that with `warn_on`.
## Cost: why summarization is the last resort
Summarization turns input tokens into output tokens, which are billed at a premium and generated
@@ -113,7 +122,7 @@ that is not enough -- which is exactly what `TieredCompaction` encodes:
```python
from pydantic_ai import Agent
from pydantic_ai_harness.experimental.compaction import (
from pydantic_ai_harness.compaction import (
ClearToolResults,
DeduplicateFileReads,
SummarizingCompaction,
@@ -151,6 +160,11 @@ from the edit point onward -- the next request pays a cache-write. Use `ClearToo
`SummarizingCompaction(model=...)` accepts a model name or `Model`; when left `None` it inherits the
running agent's model. No token caps are imposed on the summary call.
By default `incremental=True` extends an existing summary from a prior compaction rather than
regenerating it from scratch, and `preserve_first_user_message=True` keeps the original task turn even
when it falls outside the window. Pass `keep_tokens` to trim the retained tail to a token budget instead
of `keep_messages`.
## Usage accounting
The summary call is a real request to the model, so its full usage -- tokens **and** the request
@@ -168,7 +182,6 @@ call is not a file read:
```python
from pydantic_ai.messages import ToolCallPart
def my_file_key(call: ToolCallPart) -> str | None:
if call.tool_name != 'read_file':
return None
pydantic_ai_harness/compaction/__init__.py
+27
View File
@@ -0,0 +1,27 @@
"""Compaction capabilities: keep an agent's conversation history within the context window.
Each capability lives in its own module; shared utilities (token estimation, the
`CompactionStrategy` protocol, tool-pair-safe cutoffs, in-place clearing) live in `_shared`.
"""
from pydantic_ai_harness.compaction._clamp_oversized_messages import ClampOversizedMessages
from pydantic_ai_harness.compaction._clear_tool_results import ClearToolResults
from pydantic_ai_harness.compaction._deduplicate_file_reads import DeduplicateFileReads
from pydantic_ai_harness.compaction._limit_warner import LimitWarner, WarningKind
from pydantic_ai_harness.compaction._shared import CompactionStrategy, estimate_token_count
from pydantic_ai_harness.compaction._sliding_window import SlidingWindow
from pydantic_ai_harness.compaction._summarizing_compaction import SummarizingCompaction
from pydantic_ai_harness.compaction._tiered_compaction import TieredCompaction
__all__ = [
'ClampOversizedMessages',
'ClearToolResults',
'CompactionStrategy',
'DeduplicateFileReads',
'LimitWarner',
'SlidingWindow',
'SummarizingCompaction',
'TieredCompaction',
'WarningKind',
'estimate_token_count',
]
pydantic_ai_harness/experimental/compaction/_clamp_oversized_messages.py → pydantic_ai_harness/compaction/_clamp_oversized_messages.py
+2 -3
View File
@@ -17,12 +17,11 @@ from pydantic_ai.messages import (
)
from pydantic_ai.tools import RunContext
from pydantic_ai_harness.experimental.compaction._shared import compact_with_span, estimate_text_tokens
from pydantic_ai_harness.compaction._shared import compact_with_span, estimate_text_tokens
if TYPE_CHECKING:
from pydantic_ai.models import ModelRequestContext
_CLAMP_MARKER = '\n[clamped: removed {removed} of {original} characters]\n'
"""Inserted between the head and tail slices of a clamped part. ``{removed}`` and ``{original}``
are filled with character counts."""
@@ -71,7 +70,7 @@ class ClampOversizedMessages(AbstractCapability[AgentDepsT]):
Example:
```python
from pydantic_ai import Agent
from pydantic_ai_harness.experimental.compaction import ClampOversizedMessages
from pydantic_ai_harness.compaction import ClampOversizedMessages
agent = Agent(
'openai:gpt-4o',
pydantic_ai_harness/experimental/compaction/_clear_tool_results.py → pydantic_ai_harness/compaction/_clear_tool_results.py
+2 -2
View File
@@ -11,7 +11,7 @@ from pydantic_ai.capabilities import AbstractCapability
from pydantic_ai.messages import ModelMessage
from pydantic_ai.tools import RunContext
from pydantic_ai_harness.experimental.compaction._shared import (
from pydantic_ai_harness.compaction._shared import (
compact_with_span,
estimate_token_count,
exceeds,
@@ -43,7 +43,7 @@ class ClearToolResults(AbstractCapability[AgentDepsT]):
Example:
```python
from pydantic_ai import Agent
from pydantic_ai_harness.experimental.compaction import ClearToolResults
from pydantic_ai_harness.compaction import ClearToolResults
agent = Agent(
'openai:gpt-4o',
pydantic_ai_harness/experimental/compaction/_deduplicate_file_reads.py → pydantic_ai_harness/compaction/_deduplicate_file_reads.py
+2 -4
View File
@@ -11,7 +11,7 @@ from pydantic_ai.capabilities import AbstractCapability
from pydantic_ai.messages import ModelMessage, ToolCallPart
from pydantic_ai.tools import RunContext
from pydantic_ai_harness.experimental.compaction._shared import (
from pydantic_ai_harness.compaction._shared import (
compact_with_span,
exceeds,
iter_tool_pairs,
@@ -39,8 +39,7 @@ class DeduplicateFileReads(AbstractCapability[AgentDepsT]):
```python
from pydantic_ai import Agent
from pydantic_ai.messages import ToolCallPart
from pydantic_ai_harness.experimental.compaction import DeduplicateFileReads
from pydantic_ai_harness.compaction import DeduplicateFileReads
def file_key(call: ToolCallPart) -> str | None:
if call.tool_name != 'read_file':
@@ -48,7 +47,6 @@ class DeduplicateFileReads(AbstractCapability[AgentDepsT]):
args = call.args_as_dict()
return args.get('path')
agent = Agent('openai:gpt-4o', capabilities=[DeduplicateFileReads(file_key=file_key)])
```
"""
pydantic_ai_harness/experimental/compaction/_limit_warner.py → pydantic_ai_harness/compaction/_limit_warner.py
+2 -2
View File
@@ -10,7 +10,7 @@ from pydantic_ai.capabilities import AbstractCapability
from pydantic_ai.messages import ModelMessage, ModelRequest, SystemPromptPart, UserPromptPart
from pydantic_ai.tools import RunContext
from pydantic_ai_harness.experimental.compaction._shared import estimate_token_count
from pydantic_ai_harness.compaction._shared import estimate_token_count
if TYPE_CHECKING:
from pydantic_ai.models import ModelRequestContext
@@ -43,7 +43,7 @@ class LimitWarner(AbstractCapability[AgentDepsT]):
Example:
```python
from pydantic_ai import Agent
from pydantic_ai_harness.experimental.compaction import LimitWarner
from pydantic_ai_harness.compaction import LimitWarner
agent = Agent(
'openai:gpt-4o',
pydantic_ai_harness/experimental/compaction/_shared.py → pydantic_ai_harness/compaction/_shared.py
-1
View File
@@ -331,7 +331,6 @@ def prepend_first_user_message(
# Tool-pair inspection and in-place clearing
# ---------------------------------------------------------------------------
_CLEARED_TOOL_ARGS = '{}'
"""Replacement for cleared tool-call arguments.
pydantic_ai_harness/experimental/compaction/_sliding_window.py → pydantic_ai_harness/compaction/_sliding_window.py
+2 -2
View File
@@ -11,7 +11,7 @@ from pydantic_ai.capabilities import AbstractCapability
from pydantic_ai.messages import ModelMessage
from pydantic_ai.tools import RunContext
from pydantic_ai_harness.experimental.compaction._shared import (
from pydantic_ai_harness.compaction._shared import (
compact_with_span,
exceeds,
find_safe_cutoff,
@@ -37,7 +37,7 @@ class SlidingWindow(AbstractCapability[AgentDepsT]):
Example:
```python
from pydantic_ai import Agent
from pydantic_ai_harness.experimental.compaction import SlidingWindow
from pydantic_ai_harness.compaction import SlidingWindow
agent = Agent(
'openai:gpt-4o',
pydantic_ai_harness/experimental/compaction/_summarizing_compaction.py → pydantic_ai_harness/compaction/_summarizing_compaction.py
+2 -2
View File
@@ -20,7 +20,7 @@ from pydantic_ai.messages import (
)
from pydantic_ai.tools import RunContext
from pydantic_ai_harness.experimental.compaction._shared import (
from pydantic_ai_harness.compaction._shared import (
compact_with_span,
exceeds,
find_first_user_message,
@@ -151,7 +151,7 @@ class SummarizingCompaction(AbstractCapability[AgentDepsT]):
Example:
```python
from pydantic_ai import Agent
from pydantic_ai_harness.experimental.compaction import SummarizingCompaction
from pydantic_ai_harness.compaction import SummarizingCompaction
agent = Agent(
'openai:gpt-4o',
pydantic_ai_harness/experimental/compaction/_tiered_compaction.py → pydantic_ai_harness/compaction/_tiered_compaction.py
+2 -2
View File
@@ -11,7 +11,7 @@ from pydantic_ai.capabilities import AbstractCapability
from pydantic_ai.messages import ModelMessage
from pydantic_ai.tools import RunContext
from pydantic_ai_harness.experimental.compaction._shared import (
from pydantic_ai_harness.compaction._shared import (
CompactionStrategy,
compact_with_span,
estimate_token_count,
@@ -36,7 +36,7 @@ class TieredCompaction(AbstractCapability[AgentDepsT]):
Example:
```python
from pydantic_ai import Agent
from pydantic_ai_harness.experimental.compaction import (
from pydantic_ai_harness.compaction import (
ClearToolResults,
SummarizingCompaction,
TieredCompaction,
pydantic_ai_harness/experimental/context/README.md → pydantic_ai_harness/context/README.md
+33 -23
View File
@@ -1,26 +1,18 @@
# RepoContext
# Context
> [!WARNING]
> **Experimental.** This capability lives under `pydantic_ai_harness.experimental` and may
> change or be removed in any release, without a deprecation period. Import it from the
> experimental path -- there is no top-level export:
> [!NOTE]
> Import this capability from its submodule -- there is no top-level `pydantic_ai_harness` re-export:
>
> ```python
> from pydantic_ai_harness.experimental.context import RepoContext
> from pydantic_ai_harness.context import RepoContext
> ```
>
> Importing any experimental capability emits a `HarnessExperimentalWarning`. Silence **all**
> harness experimental warnings with a single filter (no per-capability lines needed):
>
> ```python
> import warnings
> from pydantic_ai_harness.experimental import HarnessExperimentalWarning
>
> warnings.filterwarnings('ignore', category=HarnessExperimentalWarning)
> ```
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
Discover and load a repo's accumulated coding-assistant context engineering (CE).
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/context/)
## The problem
A repo accumulates CE for whatever coding assistant worked in it: instruction
@@ -37,7 +29,7 @@ idea the rest of the setup exists, so it can neither honor it nor translate it.
from pathlib import Path
from pydantic_ai import Agent
from pydantic_ai_harness.experimental.context import RepoContext
from pydantic_ai_harness.context import RepoContext
agent = Agent(
'anthropic:claude-sonnet-4-6',
@@ -53,6 +45,9 @@ context first, most specific last. Files are deduped by resolved real path and b
content hash, so a symlinked `AGENTS.md -> CLAUDE.md` or two ancestors sharing
identical content load once.
When `home_dir` is `None` (the default), only `workspace_dir` is scanned -- no
walk-up. Pass `home_dir=Path.home()` to walk up to your home directory.
### 2. Asset inventory (on by default)
Exposes one tool, `inventory_agent_context()`, that reports where the repo's CE
@@ -61,6 +56,9 @@ the `skills/` (SKILL.md), `agents/` (`.md`), and `settings.json` (hooks) it
contains. It returns a structured `AgentContextInventory`; it locates assets and
does not parse them, leaving translation to the orchestrator.
Rename the tool with `inventory_tool_name`, or scope which roots it scans with
`asset_roots`.
### 3. Nested-on-traversal (off by default)
When the model lists or reads a directory, surface that directory's
@@ -68,12 +66,24 @@ When the model lists or reads a directory, surface that directory's
opt-in and configurable:
```python
RepoContext(
workspace_dir=Path('.'),
nested_traversal=True,
traversal_tool_names=frozenset({'list_dir', 'read_file'}), # match your tools
traversal_path_arg='path', # the path arg key
nested_inject='pointer', # or 'contents'
from pathlib import Path
from pydantic_ai import Agent
from pydantic_ai_harness import FileSystem
from pydantic_ai_harness.context import RepoContext
agent = Agent(
'anthropic:claude-sonnet-4-6',
capabilities=[
FileSystem(root_dir='.'),
RepoContext(
workspace_dir=Path('.'),
nested_traversal=True,
traversal_tool_names=frozenset({'list_directory', 'read_file'}), # the FileSystem tool names to hook
traversal_path_arg='path', # the path arg key
nested_inject='pointer', # or 'contents'
)
],
)
```
@@ -120,4 +130,4 @@ files once per run, so mid-run edits to those files are not reloaded.
## Further reading
- [Pydantic AI capabilities](https://ai.pydantic.dev/capabilities/)
- [Pydantic AI hooks](https://ai.pydantic.dev/capabilities/#lifecycle-hooks)
- [Pydantic AI hooks](https://ai.pydantic.dev/hooks/)
pydantic_ai_harness/context/__init__.py
+8
View File
@@ -0,0 +1,8 @@
"""Context capability: discover and load a repo's accumulated context engineering."""
from pydantic_ai_harness.context._capability import RepoContext
from pydantic_ai_harness.context._inventory import AgentContextInventory, AssetRoot
from pydantic_ai_harness.context._loader import ContextFile
from pydantic_ai_harness.context._toolset import RepoContextToolset
__all__ = ['AgentContextInventory', 'AssetRoot', 'ContextFile', 'RepoContext', 'RepoContextToolset']
pydantic_ai_harness/experimental/context/_capability.py → pydantic_ai_harness/context/_capability.py
+3 -4
View File
@@ -12,19 +12,18 @@ from pydantic_ai.messages import ToolCallPart
from pydantic_ai.tools import AgentDepsT, RunContext, ToolDefinition
from pydantic_ai.toolsets import AgentToolset
from pydantic_ai_harness.experimental.context._loader import (
from pydantic_ai_harness.context._loader import (
ContextFile,
discover_instruction_files,
find_dir_context_file,
render_context_file,
render_context_files,
)
from pydantic_ai_harness.experimental.context._toolset import RepoContextToolset
from pydantic_ai_harness.context._toolset import RepoContextToolset
if TYPE_CHECKING:
from pydantic_ai._instructions import AgentInstructions
_INVENTORY_HINT = (
'Call `{tool_name}` to map where this repo keeps its coding-assistant setup '
'(instruction dirs, skills, sub-agents, and hooks) so you can read and translate it.'
@@ -63,7 +62,7 @@ class RepoContext(AbstractCapability[AgentDepsT]):
from pathlib import Path
from pydantic_ai import Agent
from pydantic_ai_harness.experimental.context import RepoContext
from pydantic_ai_harness.context import RepoContext
agent = Agent(
'anthropic:claude-sonnet-4-6',
pydantic_ai_harness/experimental/context/_inventory.py → pydantic_ai_harness/context/_inventory.py
View File
pydantic_ai_harness/experimental/context/_loader.py → pydantic_ai_harness/context/_loader.py
View File
pydantic_ai_harness/experimental/context/_toolset.py → pydantic_ai_harness/context/_toolset.py
+1 -1
View File
@@ -8,7 +8,7 @@ from pathlib import Path
from pydantic_ai.tools import AgentDepsT
from pydantic_ai.toolsets import FunctionToolset
from pydantic_ai_harness.experimental.context._inventory import AgentContextInventory, scan_assets
from pydantic_ai_harness.context._inventory import AgentContextInventory, scan_assets
class RepoContextToolset(FunctionToolset[AgentDepsT]):
pydantic_ai_harness/experimental/docs/README.md → pydantic_ai_harness/docs/README.md
+10 -17
View File
@@ -1,26 +1,18 @@
# PyaiDocs
# Pydantic AI Docs
> [!WARNING]
> **Experimental.** This capability lives under `pydantic_ai_harness.experimental` and may
> change or be removed in any release, without a deprecation period. Import it from the
> experimental path -- there is no top-level export:
> [!NOTE]
> Import this capability from its submodule -- there is no top-level `pydantic_ai_harness` re-export:
>
> ```python
> from pydantic_ai_harness.experimental.docs import PyaiDocs
> from pydantic_ai_harness.docs import PyaiDocs
> ```
>
> Importing any experimental capability emits a `HarnessExperimentalWarning`. Silence **all**
> harness experimental warnings with a single filter (no per-capability lines needed):
>
> ```python
> import warnings
> from pydantic_ai_harness.experimental import HarnessExperimentalWarning
>
> warnings.filterwarnings('ignore', category=HarnessExperimentalWarning)
> ```
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
Give an agent a tool that locates and returns Pydantic AI documentation on demand.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/docs/)
## The problem
An agent that authors Pydantic AI capabilities, hooks, tools, or toolsets needs the
@@ -32,7 +24,8 @@ the agent rarely needs in full and pins a snapshot that drifts from `main`.
`PyaiDocs` exposes one tool, `read_pyai_docs(topic)`, that locates the requested page and
returns it verbatim -- nothing is bundled into context up front. Each call resolves the
topic from a configured local checkout first, then falls back to fetching the page from
`pydantic/pydantic-ai:main`, so it works in any environment.
`pydantic/pydantic-ai:main`, so it works whether or not you have a local checkout (the
remote fallback needs network access).
Topics: `capabilities`, `hooks`, `tools`, `tools-advanced`, `toolsets`, `agent`.
@@ -40,7 +33,7 @@ Topics: `capabilities`, `hooks`, `tools`, `tools-advanced`, `toolsets`, `agent`.
from pathlib import Path
from pydantic_ai import Agent
from pydantic_ai_harness.experimental.docs import PyaiDocs
from pydantic_ai_harness.docs import PyaiDocs
agent = Agent(
'anthropic:claude-sonnet-4-6',
pydantic_ai_harness/docs/__init__.py
+6
View File
@@ -0,0 +1,6 @@
"""Docs capability: an on-demand tool that locates Pydantic AI documentation."""
from pydantic_ai_harness.docs._capability import PyaiDocs
from pydantic_ai_harness.docs._toolset import PyaiDocsToolset, PyaiDocsTopic
__all__ = ['PyaiDocs', 'PyaiDocsToolset', 'PyaiDocsTopic']
pydantic_ai_harness/experimental/docs/_capability.py → pydantic_ai_harness/docs/_capability.py
+2 -2
View File
@@ -11,7 +11,7 @@ from pydantic_ai.capabilities import AbstractCapability
from pydantic_ai.tools import AgentDepsT
from pydantic_ai.toolsets import AgentToolset
from pydantic_ai_harness.experimental.docs._toolset import PyaiDocsToolset, PyaiDocsTopic
from pydantic_ai_harness.docs._toolset import PyaiDocsToolset, PyaiDocsTopic
if TYPE_CHECKING:
from pydantic_ai._instructions import AgentInstructions
@@ -46,7 +46,7 @@ class PyaiDocs(AbstractCapability[AgentDepsT]):
from pathlib import Path
from pydantic_ai import Agent
from pydantic_ai_harness.experimental.docs import PyaiDocs
from pydantic_ai_harness.docs import PyaiDocs
agent = Agent(
'anthropic:claude-sonnet-4-6',
pydantic_ai_harness/experimental/docs/_toolset.py → pydantic_ai_harness/docs/_toolset.py
View File
pydantic_ai_harness/dynamic_workflow/README.md
+537
View File
@@ -0,0 +1,537 @@
# Dynamic Workflow
Let one agent coordinate a whole team of sub-agents by writing a small Python script.
> [!NOTE]
> Import this capability from its submodule -- there is no top-level `pydantic_ai_harness` re-export:
>
> ```python
> from pydantic_ai_harness.dynamic_workflow import DynamicWorkflow
> ```
>
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
> The extensions planned in [What is coming](#what-is-coming), structured sub-agent inputs and durable
> workflows, touch the sub-agent call contract itself.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/dynamic_workflow/)
## The idea
Say you have a few specialist agents. One reviews code. One summarizes findings. One writes the
final note. Each one is easy to call on its own. The hard part is the choreography between them. You
want to review three files at once, keep only the reports that found something, summarize those, and
hand the summary to the writer.
The usual way to do this is one tool call per step. The agent calls the reviewer and waits. It reads
the result, calls the reviewer again, and waits again. And so on. Every intermediate result travels
back into the agent's context, and every step that depends on the previous one is a separate model
turn.
`DynamicWorkflow` takes a different route. You hand it a catalog of named sub-agents, and it gives
the model a single tool, `run_workflow`. Inside that tool the model writes ordinary Python. Each of
your sub-agents is an `async` function it can call, loop over, and combine. The script runs to
completion in one tool call, and only its final value comes back to the model.
The choreography moves out of the conversation and into code.
Claude Code ships this same idea as a feature, also called dynamic workflows. Jarred Sumner used it
to port Bun from Zig to Rust: around 750,000 lines of Rust, 99.8% of the existing test suite passing,
and eleven days from first commit to merge. One workflow mapped the right Rust lifetime for every
struct field. The next wrote every file as a behavior-identical port, with hundreds of agents in
parallel and two reviewers on each file. A fix loop then drove the build and tests until both ran
clean. Claude Code runs that at the scale of a whole session. This capability brings the same idea
into your own Pydantic AI agents, inside a single `run_workflow` tool call. You can
[read the Bun story here](https://bun.com/blog/bun-in-rust).
> **Tip**
>
> If you have met [Code Mode](../../code_mode/README.md), this will feel familiar. It is the same
> sandbox and the same idea: write a script instead of many tool calls. The difference is what the
> script gets to call. In Code Mode it calls the agent's own tools. Here it calls whole sub-agents.
## How this relates to SubAgents
The harness has two delegation capabilities. They trade in the same currency, named and isolated
sub-agent runs, but they work at different altitudes:
- [`SubAgents`](../subagents/README.md) exposes one `delegate_task(agent_name, task)` tool. Each
delegation is its own tool call and its own model turn. The parent calls, waits, reads the result
into context, then decides the next step. It is simple to reason about. It is the right fit when
delegations are occasional, or when each result needs the parent's judgment before the next one.
- `DynamicWorkflow` moves the choreography into a script. Fan-out, chaining, voting, and retry loops
all run inside one tool call, and intermediate results never enter the parent's context. It is the
right fit when the coordination between sub-agents is the actual work.
Start with `SubAgents` if you are not sure. A `delegate_task` orchestrator converts to a workflow
catalog without changing the sub-agents themselves.
## Install
The script runs inside the [Monty](https://github.com/pydantic/monty) sandbox, so install the extra:
```bash
uv add "pydantic-ai-harness[dynamic-workflow]"
```
## Your first workflow
Let's build the smallest thing that works. Two sub-agents, one orchestrator.
```python
from pydantic_ai import Agent
from pydantic_ai_harness.dynamic_workflow import DynamicWorkflow
reviewer = Agent('openai:gpt-5', name='reviewer', description='Reviews code for bugs.')
summarizer = Agent('openai:gpt-5', name='summarizer', description='Summarizes findings.')
orchestrator = Agent(
'openai:gpt-5',
capabilities=[DynamicWorkflow(agents=[reviewer, summarizer])],
)
```
That is the whole setup. Here is what each piece does:
1. `reviewer` and `summarizer` are plain agents. There is nothing special about them. They are the
same `Agent` you already know.
2. Their `name` becomes the function name the model calls in the script, so pick names that are
valid Python identifiers.
3. Their `description` tells the model what each one is for. The model reads these to decide how to
wire them together, so write them the way you would document a function.
4. `DynamicWorkflow(agents=[...])` bundles them into one capability and hands the orchestrator a
single `run_workflow` tool.
## What the model does with it
When the orchestrator decides to use the tool, it does not call your sub-agents one at a time. It
writes a script. For a "review these two files and summarize" task, the script it writes looks like
this:
```python
import asyncio
reports = await asyncio.gather(
reviewer(task="Review auth.py for bugs:\n<file contents>"),
reviewer(task="Review parser.py for bugs:\n<file contents>"),
)
await summarizer(task="Summarize these review findings:\n" + "\n\n".join(reports))
```
Here are the parts that matter most, because they are the core of how you use this capability:
- Each sub-agent is an `async` function. You call it with `await`.
- You pass the work as a single keyword argument, `task`. Always by keyword. Write
`reviewer(task="...")`, not `reviewer("...")`.
- `asyncio.gather(...)` runs the two reviews at the same time instead of one after the other.
- The last line's value becomes the result the model sees. The intermediate `reports` list never
leaves the sandbox.
> **Info: what "call a sub-agent" actually means**
>
> Each call is a full `Agent.run`. It has its own model loop, its own message history, its own
> tools, and its own typed output. It is not a lightweight function. It is a real agent doing real
> work. Two things follow from that, and both matter when you write or debug workflows:
>
> - **Calls are isolated.** A sub-agent remembers nothing from an earlier call. Put everything it
> needs into `task`.
> - **Calls cost tokens and take time.** That is why this capability gives you budgets, which we
> get to below.
Because the coordination is ordinary Python, it scales past one-shot fan-out. Take a rule like
"re-dispatch only the files that failed review, with the reviewer's issues attached, for up to two
more rounds." That is a `for` loop over `asyncio.gather`, with the retry task text rebuilt from each
failed review. Without this, the model would run that control flow turn by turn, one round-trip per
step, with every intermediate draft flowing through its context.
## Sub-agents can return structured data
A sub-agent returns whatever its `output_type` produces. By default that is a string. But give a
sub-agent a Pydantic model, and the script receives a `dict`:
```python
from pydantic import BaseModel
class Score(BaseModel):
value: int
reason: str
critic = Agent('openai:gpt-5', name='critic', description='Scores an answer 0-10.', output_type=Score)
```
Inside the script, the model reads the fields by subscript, the way you read a JSON object:
```python
result = await critic(task="Score this answer: ...")
result["value"] # not result.value
```
The catalog the model sees renders each output type as a `TypedDict`, so it knows the fields and
reads them by subscript on its own.
## A complete, runnable example
Now let's put it together into something you can actually run. This orchestrator runs a small
tournament. It drafts three candidate answers in parallel, scores each one, picks the winner, and
refines it. All of that happens in a single tool call.
You need an Anthropic key and the `anthropic` package:
```bash
export ANTHROPIC_API_KEY=sk-...
uv run --with 'pydantic-ai-harness[dynamic-workflow]' --with anthropic --with logfire python wf.py
```
```python
# wf.py
import asyncio
import logfire
from pydantic import BaseModel
from pydantic_ai import Agent
from pydantic_ai.usage import UsageLimits
from pydantic_ai_harness.dynamic_workflow import DynamicWorkflow
# With Logfire configured, the trace shows the orchestrator turn, the run_workflow call (including
# the exact script the model wrote), and every sub-agent run nested underneath it.
logfire.configure(send_to_logfire='if-token-present', service_name='dynamic-workflow')
logfire.instrument_pydantic_ai()
MODEL = 'anthropic:claude-sonnet-4-6' # or 'anthropic:claude-opus-4-8'
class Score(BaseModel):
value: int # 0-10
reason: str
drafter = Agent(
MODEL,
name='drafter',
description='Writes one candidate answer to a task.',
instructions='Write one concise candidate answer to the task.',
)
critic = Agent(
MODEL,
name='critic',
description='Scores a candidate answer 0-10, returns {value, reason}.',
output_type=Score,
instructions='Score the candidate 0-10 with a one-line reason.',
)
editor = Agent(
MODEL,
name='editor',
description='Improves an answer given a critique.',
instructions='Improve the given answer using the critique. Return only the answer.',
)
orchestrator = Agent(
MODEL,
instructions=(
'Use run_workflow to: draft 3 candidate answers in parallel, score each with the critic, '
'pick the highest-scoring one, then have the editor refine it using that critique. '
'Return the refined answer.'
),
capabilities=[DynamicWorkflow(agents=[drafter, critic, editor])],
)
async def main() -> None:
result = await orchestrator.run(
'Explain, for a new hire, why our service uses idempotency keys on payment requests.',
usage_limits=UsageLimits(request_limit=20),
)
logfire.info('done', answer=result.output, requests=result.usage.requests)
asyncio.run(main())
```
Given just those three sub-agents and the instructions, the model writes and runs a script along
these lines:
```python
import asyncio
# 1. Draft three candidates at the same time.
drafts = await asyncio.gather(
drafter(task="explain idempotency keys on payments"),
drafter(task="explain idempotency keys on payments"),
drafter(task="explain idempotency keys on payments"),
)
# 2. Score each one. Structured output arrives as {"value": int, "reason": str}.
scores = await asyncio.gather(*[critic(task="Score this answer:\n" + d) for d in drafts])
# 3. Pick the winner and refine it, all in plain Python, no extra model turns.
best = max(range(len(drafts)), key=lambda i: scores[i]["value"])
await editor(task="Answer:\n" + drafts[best] + "\n\nCritique:\n" + scores[best]["reason"])
```
Read that script and notice what did not happen. The three drafts, the three scores, and the
selection logic never traveled back through the orchestrator's context. The model issued one tool
call and got back one answer. The comparison, the `max(...)`, and the string assembly are ordinary
Python running in the sandbox.
> **Tip**
>
> The [Logfire](https://pydantic.dev/logfire) trace is the best way to see what a workflow did.
> Each sub-agent run appears nested under the `run_workflow` span, and the span carries the exact
> `code` argument the model wrote, so you can read the script it actually ran.
## How results come back
The value of the script's last expression becomes the tool result. The model does not `print()` it.
For the common cases, that is all you need to know. If you want the exact rules, including what
happens when the script also prints for debugging, here they are:
> **Info: the precise return shape**
>
> | Sub-agent `output_type` | Value inside the script |
> | --- | --- |
> | `str` (the default) | the string |
> | a Pydantic model | a `dict`, read as `r['field']` |
> | list or scalar | the list or scalar |
>
> And here is how the final tool result is shaped:
>
> | The script... | The model receives |
> | --- | --- |
> | ends in a value, no print | that value directly (or `{}` if it is `None`) |
> | prints and ends in a value | `{"output": "<printed text>", "result": <value>}` |
> | prints and ends in `None` | `{"output": "<printed text>"}` |
>
> `print()` is for debug logging. It stringifies, so it is the wrong tool for returning structured
> data. Let the last expression carry the real result.
## Choosing sub-agent models
By default, each sub-agent uses the model it was constructed with. Set `inherit_model=True` when the
host passes a per-run model override to the parent agent, for example from a `/model` command, and
every sub-agent dispatch should follow that resolved parent model. Leave it `False` when a sub-agent
is deliberately pinned to a different model.
## Keeping it safe: budgets
A sub-agent is non-deterministic and costs tokens, and it can fan out into more sub-agents. So a
workflow needs two kinds of ceiling. It needs a cap on *how many* sub-agent runs happen, and a cap
on *how much* they spend. `DynamicWorkflow` gives you both, plus a guard against runaway sandbox
scripts.
### `max_agent_calls`: an exact count
```python
DynamicWorkflow(agents=[...], max_agent_calls=50) # 50 is the default
```
This is a hard, host-enforced ceiling on the number of sub-agent runs in one parent run. It is one
budget shared across every `run_workflow` call in that run, not a per-script allowance. It holds
exactly, even when the script fans out with `asyncio.gather`. When the budget runs out, the workflow
stops calling sub-agents and returns a terminal result that tells the model to conclude with what it
has. That result includes the sub-agent results that did complete, so nothing you already paid for
is wasted.
> **Note**
>
> `max_agent_calls` is the only knob that bounds the number of runs exactly. Reach for it when you
> need a guarantee. The token-based limits below are budgets, not guarantees.
### `sub_agent_usage_limits` and `forward_usage`: bounding cost
`sub_agent_usage_limits` is a `UsageLimits` applied to each sub-agent run. How tight a ceiling it
gives depends on `forward_usage`, which controls whether the whole tree shares one usage counter:
| `forward_usage` | Counter | What the limit means |
| --- | --- | --- |
| `True` (default) | the parent's `usage` is shared across the tree | a tree-wide cap, checked against the shared counter. Under concurrent fan-out it is best-effort: several sub-agents can pass the check before any of them adds to the count. |
| `False` | each sub-agent run counts on its own | per-run limits. A per-run `total_tokens_limit` of `T` with `max_agent_calls` of `N` bounds the tree to roughly `N * T` tokens. |
> **Warning**
>
> The `usage_limits` you pass to the parent `run()` is not forwarded into sub-agents. Core does not
> expose that limit value to the capability, so it is re-checked only at the parent's own request
> boundaries. If you want to bound sub-agents, set `sub_agent_usage_limits`. If you want an exact
> ceiling on the number of runs, use `max_agent_calls`.
### `resource_limits`: guarding the script itself
These limits guard the orchestration script's own memory and allocations, not the sub-agents it
calls. The default backstop is 256 MB and 50 million allocations, with no time limit.
```python
DynamicWorkflow(agents=[...], resource_limits={'max_duration_secs': 30})
```
There is no default duration cap. To see why, it helps to know what the timer actually measures.
> **Info: what `max_duration_secs` measures**
>
> Monty checks this limit once per bytecode step. So it measures the time your script spends
> running sandbox code. It does not measure wall-clock time.
>
> This is the part that matters for sub-agents. While your script waits on a sub-agent, it is
> suspended on the host. It is not running. That time does not count. The same is true whether you
> await one sub-agent at a time or fan several out with `asyncio.gather`. A normal workflow spends
> most of its time waiting, so the cap will not fire on it, no matter how long the sub-agents take.
>
> So what is the cap for? One thing: a pure-CPU runaway. Picture a `while True:` loop that never
> awaits. It burns a whole core and blocks the event loop, and none of the sub-agent budgets
> (`max_agent_calls`, `sub_agent_usage_limits`) can stop it, because it never calls a sub-agent. If
> you want that guard, set `max_duration_secs` yourself.
>
> Two more knobs. Pass `'unlimited'` to remove every limit. Pass a partial dict like
> `{'max_memory': ...}` and it merges onto the backstop, so you override only the caps you name and
> the rest keep their defaults.
### Workflows do not nest
A sub-agent cannot start its own workflow. If one tries, the nested `run_workflow` call returns a
terminal error instead of running.
> **Tip**
>
> Here is the practical rule: do not give the sub-agents in your catalog the `DynamicWorkflow`
> capability. They are the leaves of the orchestration, not orchestrators themselves.
## Renaming a sub-agent: `WorkflowAgent`
By default, a sub-agent shows up in the script under its own `name` and `description`. Sometimes you
want a different name or a different description for one particular workflow, without editing the
agent itself. Wrap it in a `WorkflowAgent`:
```python
from pydantic_ai_harness.dynamic_workflow import WorkflowAgent
DynamicWorkflow(
agents=[
WorkflowAgent(
reviewer,
name='check',
description='Checks one code change and returns actionable review findings.',
),
],
)
```
Now the model calls `check(task=...)` instead of `reviewer(task=...)`. Passing a bare agent is just
shorthand for wrapping it in a `WorkflowAgent` with no overrides.
## Adding sub-agents while a run is going: `reveal()`
The catalog is fixed when a run starts, which keeps it in the prompt-cache prefix across turns. But
sometimes you learn during a run that a new sub-agent should be available, say once a fixer agent has
been provisioned. Keep a reference to the `DynamicWorkflow` instance and call `reveal()`:
```python
workflow = DynamicWorkflow(agents=[reviewer])
orchestrator = Agent('openai:gpt-5', deps_type=MyDeps, capabilities=[workflow])
# later, from the host or from another tool:
workflow.reveal(fixer)
```
The revealed sub-agent becomes callable on the next step. The model learns about it through a short
announcement message that carries the new function's signature. The `run_workflow` description itself
stays frozen at the agents present when the run started, so even a runtime reveal never moves the
prompt-cache prefix.
> **Note**
>
> `reveal()` is append-only. Once a sub-agent appears it stays for the rest of the run, and there
> is no way to remove or hide it again. Plan the catalog as something that only grows.
>
> It validates right away. A missing name, an invalid identifier, a reserved keyword, or a name
> collision raises `UserError` at the call site. And if you share one `DynamicWorkflow` instance
> across concurrent runs, `reveal()` reaches all in-flight runs and joins the baseline for runs that
> start afterward.
## Loading it only when needed: `defer_loading`
`DynamicWorkflow` carries a fair amount of instruction text, and most turns do not need it. You can
keep it collapsed to a one-line entry until the model actually loads it. That pays close to zero
tokens on turns that never orchestrate:
```python
DynamicWorkflow(
agents=[reviewer, summarizer],
id='workflow',
defer_loading=True,
)
```
`defer_loading=True` needs a stable `id`. See
[on-demand capabilities](https://pydantic.dev/docs/ai/core-concepts/capabilities/#on-demand-capabilities)
for the full picture.
## What runs in the sandbox
The script runs in Monty, a subset of Python. The subset is what makes the sandbox safe, so it is
worth knowing where the edges are:
- No class definitions, and no third-party libraries.
- Useful standard-library modules: `asyncio`, `math`, `json`, `re`, `typing`. Import what you use.
Other modules are unavailable or stubbed.
- No wall-clock or timing primitives. There is no `asyncio.sleep`, no `datetime.now()`, and no
`time` module.
- `asyncio.gather(...)` runs sub-agents concurrently, but it does not support
`return_exceptions=True`.
Before a script runs, it is statically type-checked against the sub-agent signatures. A misspelled
function, a positional `task`, or a wrong-typed argument costs one retry, but no sub-agent budget and
no sandbox execution.
> **Warning: errors abort the whole script**
>
> A sub-agent that raises cannot be caught inside the script. One failure aborts the whole script,
> and the model retries it. So write scripts where sub-agents do not depend on catching each other's
> errors. If a script does fail after some sub-agents already finished, the retry prompt lists those
> completed results, so the model can reuse them as plain values instead of paying for the same
> calls again.
## What is coming
A suspended Monty program is a small serializable value you can dump, reload, and fork. That points
at two patterns that do not ship yet. The first is forking one expensive shared prefix into N
best-of-N branches. The second is durable workflows that resume from a persisted snapshot after a
crash or a redeploy. Two smaller extensions are also planned: structured sub-agent inputs (a
`parameters` schema per `WorkflowAgent`, instead of only `task: str`) and first-class progress
streaming. Until then, set `event_stream_handler` on each sub-agent `Agent`, or use Logfire, to
watch sub-agent runs inside the one tool call.
## API
```python
DynamicWorkflow( # all parameters are keyword-only
agents=[...], # Sequence[AbstractAgent | WorkflowAgent], required
tool_name='run_workflow',
max_agent_calls=50,
max_retries=3,
forward_usage=True,
inherit_model=False, # True -> sub-agents run with the parent run's resolved model
sub_agent_usage_limits=None, # UsageLimits per sub-agent run; None -> pydantic-ai default
resource_limits=None, # None -> backstop (256 MB, 50M allocs, no time cap);
# 'unlimited' -> off; a dict is merged onto the backstop
id=None, # required when defer_loading=True
description=None, # one-line catalog entry shown while deferred
defer_loading=False,
)
workflow.reveal(agent) # AbstractAgent | WorkflowAgent; validates before appending
WorkflowAgent(
agent, # Agent, required, positional
name=None, # sandbox function name; falls back to agent.name
description=None, # function docstring; falls back to agent.description
)
```
## Further reading
- [Code Mode](../../code_mode/README.md), the same sandbox, calling the agent's own tools instead of
sub-agents.
- [SubAgents](../subagents/README.md), one-delegation-per-tool-call sub-agents, without the
scripted choreography.
- [Tool use via code](https://www.anthropic.com/engineering/code-execution-with-mcp) (Anthropic),
the mechanism this applies to sub-agents.
- [Building Effective Agents](https://www.anthropic.com/engineering/building-effective-agents)
(Anthropic), the orchestration patterns a script can express.
- [Rewriting Bun in Rust](https://bun.com/blog/bun-in-rust) (Bun), the same pattern at scale: Jarred
Sumner's port of Bun from Zig to Rust with Claude Code dynamic workflows.
- [Capabilities](https://pydantic.dev/docs/ai/core-concepts/capabilities/) and
[on-demand capabilities](https://pydantic.dev/docs/ai/core-concepts/capabilities/#on-demand-capabilities).
- [Monty](https://github.com/pydantic/monty), the sandbox.
pydantic_ai_harness/dynamic_workflow/__init__.py
+10
View File
@@ -0,0 +1,10 @@
"""Dynamic workflow capability: orchestrate sub-agents from a sandboxed Python script."""
from pydantic_ai_harness.dynamic_workflow._capability import DynamicWorkflow
from pydantic_ai_harness.dynamic_workflow._toolset import (
DynamicWorkflowToolset,
WorkflowAgent,
WorkflowResourceLimits,
)
__all__ = ['DynamicWorkflow', 'DynamicWorkflowToolset', 'WorkflowAgent', 'WorkflowResourceLimits']
pydantic_ai_harness/dynamic_workflow/_capability.py
+162
View File
@@ -0,0 +1,162 @@
"""Dynamic workflow capability: orchestrate sub-agents from a sandboxed Python script."""
from __future__ import annotations
from collections.abc import Sequence
from dataclasses import dataclass, field
from typing import Literal
from pydantic_ai.agent.abstract import AbstractAgent
from pydantic_ai.capabilities import AbstractCapability
from pydantic_ai.tools import AgentDepsT
from pydantic_ai.usage import UsageLimits
from pydantic_ai_harness.dynamic_workflow._toolset import (
DynamicWorkflowToolset,
WorkflowAgent,
WorkflowResourceLimits,
index_workflow_agents,
validate_workflow_agent,
)
@dataclass(kw_only=True)
class DynamicWorkflow(AbstractCapability[AgentDepsT]):
"""Capability that lets the model orchestrate named sub-agents from a Python script.
Instead of one sub-agent per tool call, the model writes a single Python script (run in a
Monty sandbox) that calls each sub-agent as an async function and composes the results: fan
out with `asyncio.gather`, chain one agent's output into the next, vote, or loop until done.
```python
from pydantic_ai import Agent
from pydantic_ai_harness.dynamic_workflow import DynamicWorkflow
reviewer = Agent('openai:gpt-5', name='reviewer', description='Review code for bugs.')
summarizer = Agent('openai:gpt-5', name='summarizer', description='Summarize findings.')
orchestrator = Agent(
'openai:gpt-5',
capabilities=[DynamicWorkflow(agents=[reviewer, summarizer])],
)
```
Each sub-agent runs isolated (its own message history) with the parent's `deps` forwarded;
by default the parent's `usage` accumulator is shared so the whole tree's spend is tallied in
one place. Use `max_agent_calls` for a hard, host-enforced ceiling on sub-agent runs. Workflows
do not nest. Set `defer_loading=True` (with a stable `id`) to keep the tool out of the prompt
until the model loads the capability.
"""
agents: Sequence[AbstractAgent[AgentDepsT, object] | WorkflowAgent[AgentDepsT]]
"""Sub-agents the orchestration script can call as async functions.
Read at construction only; later mutation of the passed sequence is ignored. A raw agent is
shorthand for `WorkflowAgent(agent)`, using the agent's own `name` and `description`; use a
`WorkflowAgent` entry for a per-use-site override. Use `reveal()` to add a sub-agent after
construction.
"""
_catalog: list[WorkflowAgent[AgentDepsT]] = field(init=False, repr=False)
"""Normalized catalog passed by reference to toolsets."""
tool_name: str = 'run_workflow'
"""Name of the orchestration tool exposed to the model."""
max_agent_calls: int = 50
"""Maximum total sub-agent runs per agent run: an exact, host-enforced ceiling that holds even
under concurrent fan-out (unlike a parent `usage_limits`)."""
max_retries: int = 3
"""Maximum retries for the orchestration tool (syntax/runtime errors count as retries)."""
forward_usage: bool = True
"""Share the parent run's `usage` accumulator with sub-agents, tallying the whole tree's
token and request spend in one place.
This does **not** forward the parent's `usage_limits` into sub-agent runs (`RunContext` does
not expose the limit value): set `sub_agent_usage_limits` to bound sub-agents, or
`max_agent_calls` for an exact ceiling on the number of runs.
"""
inherit_model: bool = False
"""Run every sub-agent with the parent run's resolved model instead of its constructed model.
Use this when the host can switch models per run, such as a `/model` command that passes a
run-level model override to the parent agent. Without it, that per-run choice silently leaves
catalog sub-agents on the model they were bound to when constructed; `inherit_model=True` makes
the workflow crew follow the parent run's resolved model. Keep `False` to pin sub-agents to
their own configured models.
"""
sub_agent_usage_limits: UsageLimits | None = None
"""`UsageLimits` applied to every sub-agent run, replacing pydantic-ai's default.
With `forward_usage=False`, a per-run `total_tokens_limit` of `T` plus `max_agent_calls` of
`N` bounds the tree to roughly `N * T` tokens (each run can overshoot by its final response,
since core checks token limits after a response arrives). With `forward_usage=True` the limit
is checked against the shared counter -- a tree-wide cap, best-effort under concurrent fan-out.
`None` keeps the default (`request_limit=50`, no token limit).
"""
resource_limits: WorkflowResourceLimits | Literal['unlimited'] | None = None
"""Sandbox limits guarding the orchestration script's own memory/allocations.
`None` applies a safe backstop (256 MB, 50M allocations, no execution-time cap); `'unlimited'`
removes all limits; a `WorkflowResourceLimits` mapping is merged onto the backstop, overriding
only the caps it names. There is no default `max_duration_secs`: Monty's timer bounds in-sandbox
execution time and time awaiting sub-agents does not count against it, so set one only to guard
a pure-CPU `while True` loop.
"""
@classmethod
def get_serialization_name(cls) -> str | None:
# Not spec-serializable: `agents` holds live Agent objects, not YAML-expressible config.
return None
def __post_init__(self) -> None:
catalog = [self._normalize_workflow_agent(entry) for entry in self.agents]
index_workflow_agents(catalog)
self._catalog = catalog
def _normalize_workflow_agent(
self, entry: AbstractAgent[AgentDepsT, object] | WorkflowAgent[AgentDepsT]
) -> WorkflowAgent[AgentDepsT]:
"""Normalize a public catalog entry to the internal wrapper form."""
if isinstance(entry, WorkflowAgent):
return entry
return WorkflowAgent(agent=entry)
def reveal(self, agent: AbstractAgent[AgentDepsT, object] | WorkflowAgent[AgentDepsT]) -> None:
"""Reveal a sub-agent on the next model step (the supported runtime API for doing so).
The sub-agent is announced to the model on the next step and becomes callable then; the
`run_workflow` description stays frozen at the agents present when the run started. Reveal
is append-only: a revealed sub-agent cannot be removed for the rest of the run. Its resolved
name must be a valid, unique sandbox function name, or this raises `UserError` at the call
site. If one `DynamicWorkflow` instance is shared across concurrent runs, `reveal()` reaches
all in-flight runs and joins the baseline catalog for runs that start afterwards.
"""
entry = self._normalize_workflow_agent(agent)
existing_names: set[str] = set()
for catalog_entry in self._catalog:
existing_names.add(validate_workflow_agent(catalog_entry, existing_names))
validate_workflow_agent(entry, existing_names)
self._catalog.append(entry)
def get_toolset(self) -> DynamicWorkflowToolset[AgentDepsT]:
"""Provide the orchestration toolset to the agent."""
return DynamicWorkflowToolset(
# Toolsets keep this same list object; `reveal()` appends to it so in-flight
# toolsets can fold in the new sub-agent on the next step.
agents=self._catalog,
tool_name=self.tool_name,
max_agent_calls=self.max_agent_calls,
max_retries=self.max_retries,
forward_usage=self.forward_usage,
inherit_model=self.inherit_model,
sub_agent_usage_limits=self.sub_agent_usage_limits,
resource_limits=self.resource_limits,
toolset_id=self.id,
owning_capability=self,
)
pydantic_ai_harness/dynamic_workflow/_toolset.py
+723
View File
@@ -0,0 +1,723 @@
"""Toolset for the `DynamicWorkflow` capability.
Exposes a single `run_workflow` tool: the model writes a Python orchestration
script (run in a Monty sandbox) that calls named sub-agents as async functions
and composes their results -- fan-out, chaining, voting, loops -- in one step.
"""
from __future__ import annotations
import contextvars
import copy
import json
import keyword
import warnings
from collections.abc import Mapping, Sequence
from dataclasses import dataclass, field
from typing import Annotated, Any, Generic, Literal
from pydantic import Field, TypeAdapter
from pydantic_ai import AbstractToolset, RunContext, ToolDefinition
from pydantic_ai.agent.abstract import AbstractAgent
from pydantic_ai.capabilities import AbstractCapability, WrapperCapability
from pydantic_ai.exceptions import ModelRetry, UsageLimitExceeded, UserError
from pydantic_ai.function_signature import FunctionSignature
from pydantic_ai.tools import AgentDepsT
from pydantic_ai.toolsets.abstract import SchemaValidatorProt, ToolsetTool
from pydantic_ai.usage import UsageLimits
from pydantic_core import to_jsonable_python
from typing_extensions import Self, TypedDict
try:
from pydantic_monty import Monty, MontyRepl, MontyRuntimeError, MontySyntaxError, MontyTypingError, ResourceLimits
except ImportError as _import_error: # pragma: no cover
raise ImportError(
'pydantic-monty is required for DynamicWorkflow. '
'Install it with: uv add "pydantic-ai-harness[dynamic-workflow]"'
) from _import_error
from pydantic_ai_harness._monty_exec import MontyExecutor, PrintCapture, is_sandbox_panic
# Set while a workflow script is executing, so a sub-agent that itself tries to run a workflow can
# be refused -- workflows do not nest. asyncio copies the context into each task `asyncio.gather`
# schedules, so concurrently-dispatched sub-agents inherit this flag (the capability is asyncio-only).
_in_workflow: contextvars.ContextVar[bool] = contextvars.ContextVar('pydantic_ai_harness_in_workflow', default=False)
class WorkflowResourceLimits(TypedDict, total=False):
"""Caps on the orchestration script's own sandbox resources (not sub-agent latency).
A harness-owned view of the sandbox limits the capability supports, so the public API does
not depend on the underlying sandbox's own types. Every field is optional; an omitted field
keeps its backstop value.
"""
max_duration_secs: float
"""Ceiling on the time the script spends executing sandbox code. Monty checks it per bytecode
step, so time spent awaiting sub-agents does not count against it -- neither a single `await`
nor a concurrent `asyncio.gather` batch, because during that wait the script is suspended on
the host, not running sandbox code. There is no default cap. Set one to bound a pure-CPU
`while True` loop, which would otherwise burn a core and block the event loop -- the one
runaway the sub-agent budgets do not catch."""
max_memory: int
"""Maximum sandbox memory, in bytes."""
max_allocations: int
"""Maximum number of sandbox allocations."""
def _default_resource_limits() -> ResourceLimits:
"""Backstop sandbox limits; no duration cap -- see `WorkflowResourceLimits.max_duration_secs`."""
return {
'max_memory': 256 * 1024 * 1024,
'max_allocations': 50_000_000,
}
# The keys `WorkflowResourceLimits` accepts. A `total=False` TypedDict does not validate keys at
# runtime, so a typo (e.g. `max_durations_secs`) would otherwise merge through and be silently
# dropped -- quietly disabling the only guard against a pure-CPU `while True`. We reject unknowns.
_RESOURCE_LIMIT_KEYS = frozenset(WorkflowResourceLimits.__annotations__)
_MODEL_SAFE_EXCEPTION_MESSAGE_TYPES = (UsageLimitExceeded,)
_MAX_COMPLETED_DISPATCHES = 20
_MAX_TASK_PREVIEW_CHARS = 120
_MAX_RESULT_PREVIEW_CHARS = 300
_TRUNCATED_MARKER = ' ... [truncated]'
def _resolve_resource_limits(limits: WorkflowResourceLimits | Literal['unlimited'] | None) -> ResourceLimits:
"""Resolve the public `resource_limits` value to the limits handed to the sandbox.
A partial mapping merges *onto* the backstop rather than replacing it, so `{'max_memory': ...}`
never silently drops the allocations backstop. Full semantics: `DynamicWorkflow.resource_limits`.
"""
if limits is None:
return _default_resource_limits()
if limits == 'unlimited':
return {}
unknown = set(limits) - _RESOURCE_LIMIT_KEYS
if unknown:
raise UserError(
f'Unknown `resource_limits` key(s): {sorted(unknown)}. Valid keys are {sorted(_RESOURCE_LIMIT_KEYS)}.'
)
return {**_default_resource_limits(), **limits}
class _WorkflowArguments(TypedDict):
code: Annotated[str, Field(description='The Python orchestration script to execute in the sandbox.')]
_WORKFLOW_ARGS_ADAPTER = TypeAdapter(_WorkflowArguments)
_WORKFLOW_ARGS_JSON_SCHEMA = _WORKFLOW_ARGS_ADAPTER.json_schema()
_WORKFLOW_ARGS_VALIDATOR: SchemaValidatorProt = _WORKFLOW_ARGS_ADAPTER.validator # pyright: ignore[reportAssignmentType]
class _BudgetExhausted(RuntimeError):
"""The run's `max_agent_calls` budget is spent; no further sub-agent runs are allowed."""
def __init__(self, max_agent_calls: int) -> None:
super().__init__(f'sub-agent call budget ({max_agent_calls}) exhausted')
_WORKFLOW_BASE_DESCRIPTION = """\
Write and run a Python orchestration script in a sandbox to coordinate multiple sub-agents.
Use this to break a task across specialized sub-agents and combine their results in a single step --
fan work out in parallel, chain one agent's output into the next, vote across several, or loop until
done -- instead of delegating to one sub-agent at a time.
The sandbox uses Monty, a subset of Python. Key restrictions:
- **No classes** and **no third-party libraries**.
- **Useful standard-library modules**: `asyncio`, `math`, `json`, `re`, `typing`. Import what you use
at the top of the script. Other modules are unavailable or stubbed -- don't rely on them.
- **No wall-clock or timing primitives** (`asyncio.sleep`, `datetime.now()`, the `time` module).
Each sub-agent below is an async function. Call it with the `task` keyword argument -- write
`reviewer(task="...")`, not `reviewer("...")`; all parameters are keyword-only. A sub-agent returns
that agent's output: a string by default, or -- if it has a structured `output_type` -- a dict, whose
fields you read by subscript (`r["field"]`), not attribute (`r.field`). Each sub-agent call is an
independent run with no memory of earlier calls; include all needed context in `task`. Run several
at once with `asyncio.gather` rather than awaiting each sequentially:
```python
import asyncio
reviews = await asyncio.gather(reviewer(task="check auth"), reviewer(task="check parsing"))
```
`asyncio.gather` does **not** support `return_exceptions=True`, and a sub-agent that raises cannot be
caught inside the script: one failure aborts the whole script and you retry it. Design the script so
sub-agents don't depend on catching each other's errors.
The last expression's value is captured as the result -- you do **not** need to `print()` it, and
printing produces a string representation, not structured data. Use `print()` only for debug logging.
Return shapes: no print returns the last expression value (or `{}` if it is `None`); print plus a
non-`None` value returns `{"output": "<printed text>", "result": <last expression>}`; print plus
`None` returns `{"output": "<printed text>"}`. If a script fails after some sub-agent calls complete,
those completed results are reported back so a retry can reuse them.\
"""
def _is_valid_sandbox_name(name: str) -> bool:
"""Whether `name` can be exposed as a sandbox function: a non-keyword Python identifier.
`str.isidentifier()` alone is not enough -- Python keywords (`for`, `class`, `async`, ...) are
valid identifiers but cannot be used as function names, so the model could never call them.
Callers guard the empty/`None` case before this is reached.
"""
return name.isidentifier() and not keyword.iskeyword(name)
# Every sub-agent is exposed with the same fixed parameters -- `(*, task: str)` -- and a per-agent
# return schema. Render each catalog entry through core's `FunctionSignature` (the renderer
# code_mode and Pydantic AI already use). This keeps the catalog format consistent across
# capabilities, forces keyword-only `task` to match `dispatch` (which reads `kwargs['task']`), and
# renders docstrings safely -- a hand-rolled f-string breaks on a newline or a quote inside a
# description.
_SUB_AGENT_PARAMS_SCHEMA: dict[str, Any] = {
'type': 'object',
'properties': {'task': {'type': 'string'}},
'required': ['task'],
}
_NO_CONFLICTING_TYPE_NAMES: frozenset[str] = frozenset()
def _agent_return_schema(agent: AbstractAgent[AgentDepsT, object]) -> dict[str, Any] | None:
"""The sub-agent's output JSON schema, or `None` when it cannot be derived."""
try:
return agent.output_json_schema()
except Exception:
return None
def _agent_signature(name: str, agent: AbstractAgent[AgentDepsT, object]) -> FunctionSignature:
"""Build the sandbox function signature for one sub-agent."""
return FunctionSignature.from_schema(
name=name,
parameters_schema=_SUB_AGENT_PARAMS_SCHEMA,
return_schema=_agent_return_schema(agent),
)
def _render_agent_block(
signature: FunctionSignature,
description: str | None,
*,
conflicting_type_names: frozenset[str] = _NO_CONFLICTING_TYPE_NAMES,
) -> str:
"""Render one sub-agent as the async function signature shown to the model."""
return signature.render(
'...',
description=description,
is_async=True,
conflicting_type_names=conflicting_type_names,
)
def _render_catalog(catalog: Mapping[str, WorkflowAgent[AgentDepsT]], *, max_agent_calls: int) -> str:
"""Render the available sub-agents as async function signatures for the tool description."""
signatures = {name: _agent_signature(name, entry.agent) for name, entry in catalog.items()}
signature_list = list(signatures.values())
conflicting = FunctionSignature.get_conflicting_type_names(signature_list)
type_blocks = FunctionSignature.render_type_definitions(signature_list, conflicting)
function_blocks = [
_render_agent_block(signatures[name], entry.resolved_description, conflicting_type_names=conflicting)
for name, entry in catalog.items()
]
listing = '```python\n' + '\n\n'.join([*type_blocks, *function_blocks]) + '\n```'
budget = (
f'This run can make at most {max_agent_calls} sub-agent calls in total -- one budget shared across '
'every `run_workflow` call in the run, not per script; plan fan-out width accordingly.'
)
return f'{_WORKFLOW_BASE_DESCRIPTION}\n\n{budget}\n\nAvailable sub-agents:\n\n{listing}'
def _render_reveal(
name: str,
catalog: Mapping[str, WorkflowAgent[AgentDepsT]],
tool_name: str,
) -> str:
"""Announcement enqueued when a sub-agent is revealed mid-run.
Delivered as a conversation message (not folded into the cached tool description), so the
prompt-cache prefix stays stable while the model still learns the agent is now callable.
"""
signatures = {agent_name: _agent_signature(agent_name, entry.agent) for agent_name, entry in catalog.items()}
signature = signatures[name]
conflicting = FunctionSignature.get_conflicting_type_names(list(signatures.values()))
type_blocks = FunctionSignature.render_type_definitions([signature], conflicting)
function_block = _render_agent_block(
signature, catalog[name].resolved_description, conflicting_type_names=conflicting
)
block = '\n\n'.join([*type_blocks, function_block])
return f'A new sub-agent is now available to call from inside the `{tool_name}` script:\n\n```python\n{block}\n```'
def _workflow_result(result: object, printed: str) -> object:
"""Shape the tool return: the script's result, its captured `print()` output, or both."""
if not printed:
return result if result is not None else {}
if result is None:
return {'output': printed}
return {'output': printed, 'result': result}
@dataclass(frozen=True)
class _CompletedDispatch:
"""One sub-agent result completed by the failed workflow script."""
agent_name: str
task: str
result: object
def _truncate_preview(value: str, max_chars: int) -> str:
"""Trim a preview with an explicit marker so the model can see it is incomplete."""
if len(value) <= max_chars:
return value
return value[: max_chars - len(_TRUNCATED_MARKER)] + _TRUNCATED_MARKER
def _json_preview(value: object, max_chars: int) -> str:
"""Render a compact JSON preview of a completed sub-agent result."""
rendered = json.dumps(value, ensure_ascii=True, separators=(',', ':'))
return _truncate_preview(rendered, max_chars)
def _completed_dispatch_lines(completed: list[_CompletedDispatch]) -> list[str]:
"""Format the most recent completed sub-agent results for model-facing salvage."""
shown = completed[-_MAX_COMPLETED_DISPATCHES:]
lines = [
f'{entry.agent_name}(task={json.dumps(_truncate_preview(entry.task, _MAX_TASK_PREVIEW_CHARS), ensure_ascii=True)})'
f' -> {_json_preview(entry.result, _MAX_RESULT_PREVIEW_CHARS)}'
for entry in shown
]
omitted = len(completed) - len(shown)
if omitted:
lines.insert(0, f'... {omitted} earlier completed result(s) omitted ...')
return lines
def _completed_retry_section(completed: list[_CompletedDispatch]) -> str:
"""Build the optional retry-message section listing salvageable completed results."""
lines = _completed_dispatch_lines(completed)
if not lines:
return ''
listing = '\n'.join(f'- {line}' for line in lines)
return (
'\n\nCompleted sub-agent results from the failed script '
'(reuse these values instead of re-calling them; their budget was already spent):\n'
f'{listing}'
)
def _budget_terminal_result(
*,
max_agent_calls: int,
last_error: str,
completed_dispatches: list[_CompletedDispatch],
) -> dict[str, object]:
"""Build the terminal result returned after the exact sub-agent-call budget is exhausted."""
return {
'error': (
f'This run exhausted its sub-agent call budget ({max_agent_calls}). '
'Conclude using the results already gathered; further sub-agent calls in '
'this run will be refused.'
),
'last_error': last_error,
'completed': _completed_dispatch_lines(completed_dispatches),
}
@dataclass(frozen=True)
class WorkflowAgent(Generic[AgentDepsT]):
"""One sub-agent exposed to the orchestration script as an async function.
`WorkflowAgent` is the per-use-site override for when the agent's own `name`
or `description` is not what this workflow should show, such as renaming the
sandbox function or re-describing the agent for this catalog. Passing a bare
agent to `DynamicWorkflow(agents=[...])` is equivalent to `WorkflowAgent(agent)`.
"""
agent: AbstractAgent[AgentDepsT, object]
"""The sub-agent to run when the script calls this function."""
name: str | None = None
"""Sandbox function name; must be a valid Python identifier and unique across the
workflow. Falls back to the agent's `name`."""
description: str | None = None
"""Description shown to the model in the sub-agent catalog, rendered as the sandbox
function's docstring. An explicit value overrides the agent's own `description`.
When neither is set, the model sees only the bare signature."""
@property
def resolved_name(self) -> str | None:
"""The sandbox function name: the explicit `name`, else the agent's `name`."""
return self.name or self.agent.name
@property
def resolved_description(self) -> str | None:
"""The catalog description: the explicit `description`, else the agent's own `description`."""
return self.description or self.agent.description
def validate_workflow_agent(entry: WorkflowAgent[AgentDepsT], existing_names: set[str]) -> str:
"""Validate one sub-agent entry against the names already taken."""
name = entry.resolved_name
if not name:
raise UserError(
'DynamicWorkflow sub-agent has no `name` and its agent has no `name`; '
'set `WorkflowAgent(name=...)` so it can be exposed as a sandbox function.'
)
if not _is_valid_sandbox_name(name):
raise UserError(
f'DynamicWorkflow sub-agent name {name!r} cannot be exposed as a sandbox function: '
'it must be a Python identifier that is not a reserved keyword. Rename it.'
)
if name in existing_names:
raise UserError(f'DynamicWorkflow has two sub-agents named {name!r}; names must be unique.')
return name
def index_workflow_agents(
agents: Sequence[WorkflowAgent[AgentDepsT]],
) -> dict[str, WorkflowAgent[AgentDepsT]]:
"""Index validated sub-agent entries by resolved sandbox name."""
if not agents:
raise UserError('DynamicWorkflow requires at least one sub-agent in `agents`.')
by_name: dict[str, WorkflowAgent[AgentDepsT]] = {}
existing_names: set[str] = set()
for entry in agents:
name = validate_workflow_agent(entry, existing_names)
existing_names.add(name)
by_name[name] = entry
return by_name
@dataclass(kw_only=True)
class DynamicWorkflowToolset(AbstractToolset[AgentDepsT]):
"""Single-tool toolset that runs sub-agent orchestration scripts in a Monty sandbox."""
agents: list[WorkflowAgent[AgentDepsT]]
"""Sub-agents callable from the orchestration script, each as an async function.
`DynamicWorkflow.reveal()` is the only supported way to add a sub-agent mid-run.
The toolset observes appends to this list as the reveal channel. Any entry that
arrives invalid is a contract violation that raises."""
tool_name: str = 'run_workflow'
"""Name of the tool exposed to the model."""
max_agent_calls: int = 50
"""Maximum total sub-agent runs per agent run (an exact, host-enforced ceiling)."""
max_retries: int = 3
"""Maximum retries for the `run_workflow` tool (syntax/runtime errors count as retries)."""
forward_usage: bool = True
"""Share the parent run's `usage` accumulator with sub-agents. See
`DynamicWorkflow.forward_usage` for what is and is not forwarded."""
inherit_model: bool = False
"""Run every sub-agent with the parent run's resolved model instead of its constructed model.
See `DynamicWorkflow.inherit_model` for when to use this."""
sub_agent_usage_limits: UsageLimits | None = None
"""`UsageLimits` applied to every sub-agent run, replacing pydantic-ai's default.
See `DynamicWorkflow.sub_agent_usage_limits` for the budgeting semantics."""
resource_limits: WorkflowResourceLimits | Literal['unlimited'] | None = None
"""Sandbox limits guarding the orchestration script's own memory/allocations (not sub-agents).
See `DynamicWorkflow.resource_limits` for the `None`/`'unlimited'`/partial-dict semantics."""
toolset_id: str | None = None
"""Stable toolset id; defaults to the tool name."""
owning_capability: AbstractCapability[AgentDepsT] | None = None
"""The `DynamicWorkflow` capability this toolset belongs to, when capability-provided.
Used to resolve whether the capability is visible to the model on the current step (deferred
and unloaded means hidden), by identity against the run's capability registry -- ids cannot be
used for this, because an id-less capability is registered under a run-generated key and a
wrapper capability is registered in place of what it wraps. `None` (a toolset used directly,
outside any capability) is always visible."""
# Per-run count of sub-agent calls; reset on `for_run`.
_call_count: int = field(default=0, init=False, repr=False)
# Sub-agents indexed by resolved sandbox name; seeded from `agents` in `__post_init__` and
# extended in place as runtime appends to `agents` are revealed (`_reveal_pending`).
_by_name: dict[str, WorkflowAgent[AgentDepsT]] = field(init=False, repr=False)
# Tool description, frozen at run start. Rendered from the agents present when the run began
# and never re-rendered after a reveal, so the description -- and thus the prompt-cache
# prefix -- never changes mid-run.
_description: str = field(init=False, repr=False)
def __post_init__(self) -> None:
if self.max_agent_calls < 1:
raise UserError('DynamicWorkflow `max_agent_calls` must be at least 1.')
_resolve_resource_limits(self.resource_limits) # validate keys now, not at the first tool call
self._rebuild()
def _rebuild(self) -> None:
"""Rebuild the name index and the frozen tool description from the current `agents`.
Unusable entries raise `UserError`. `DynamicWorkflow.__post_init__` and
`DynamicWorkflow.reveal()` validate entries eagerly, so this fails only when the
lower-level toolset list was mutated outside those APIs.
"""
by_name = index_workflow_agents(self.agents)
self._by_name = by_name
self._description = _render_catalog(by_name, max_agent_calls=self.max_agent_calls)
@property
def id(self) -> str | None:
return self.toolset_id or self.tool_name
async def for_run(self, ctx: RunContext[AgentDepsT]) -> Self:
"""Fresh instance per run so the sub-agent-call budget is per-run.
Clone shallowly, keeping `agents` shared so reveals stay visible to this run,
reset the per-run call budget, and rebuild the per-run index. Entries are
validated eagerly by `DynamicWorkflow.__post_init__` and `DynamicWorkflow.reveal()`,
so a strict rebuild here cannot fail unless this list was mutated outside those
APIs. That is unsupported and fails fast.
"""
clone = copy.copy(self)
clone._call_count = 0
clone._rebuild()
return clone
def _fold_reveals(self, ctx: RunContext[AgentDepsT]) -> None:
"""Fold sub-agents appended to `agents` since the run started into the name index.
Diffs the live `agents` list against the names already known (`_by_name`, which holds the
baseline plus anything revealed so far) and folds each newcomer in -- so `dispatch` resolves
it -- enqueuing an announcement for the model. The frozen `_description` is untouched, so the
cached prompt prefix is unaffected. Re-seeing an already-known entry (a baseline agent, or
one revealed on an earlier step) is a no-op, identified by object identity so it is never
mistaken for a name collision.
A newcomer whose name is missing, invalid, or already taken by a different
sub-agent is a contract violation and raises `UserError`.
Synchronous and await-free between snapshotting `agents` and mutating `_by_name`, so a
concurrently-running `dispatch` never observes a half-revealed agent -- the same
await-free-critical-section reasoning that keeps `max_agent_calls` exact under fan-out.
"""
for entry in tuple(self.agents):
name = entry.resolved_name
existing = self._by_name.get(name) if name else None
if existing is entry:
continue
name = validate_workflow_agent(entry, set(self._by_name))
self._by_name[name] = entry
try:
ctx.enqueue(_render_reveal(name, self._by_name, self.tool_name))
except UserError as exc:
warnings.warn(
f'DynamicWorkflow revealed sub-agent {name!r}, but could not enqueue its announcement: '
f'{exc}. It is callable in this run, but the model will not see the reveal message.',
stacklevel=2,
)
def _visible_to_model(self, ctx: RunContext[AgentDepsT]) -> bool:
"""Whether this toolset's capability is visible to the model on this step.
A deferred capability's toolset still gets `get_tools` calls while unloaded (its tool
is indexed for `load_capability`/`search_tools`), but announcing a reveal then would
leak the sub-agent signature into the conversation while the catalog is still hidden.
The owning capability is resolved against the run's registry by identity, unwrapping
`WrapperCapability` chains -- a wrapper over a leaf capability is registered in place of
the leaf, and its `defer_loading`/registry id are what core keys loading on. Matching by
id instead would misfire both ways: an id-less capability is registered under a
run-generated key (so a lookup by tool name can hit an unrelated capability), and a
deferred wrapper hides a non-deferred inner capability. No registry match (a toolset used
directly, outside any capability) means always visible.
"""
owner = self.owning_capability
if owner is None:
return True
for capability_id, registered in ctx.capabilities.items():
candidate: AbstractCapability[AgentDepsT] | None = registered
while candidate is not None:
if candidate is owner:
if registered.defer_loading is not True:
return True
return capability_id in ctx.loaded_capability_ids
candidate = candidate.wrapped if isinstance(candidate, WrapperCapability) else None
return True
async def get_tools(self, ctx: RunContext[AgentDepsT]) -> dict[str, ToolsetTool[AgentDepsT]]:
# Skip reveal processing while the capability is deferred and unloaded: the newcomers
# stay pending in `agents` and are folded in and announced on the first step after the
# model loads the capability.
if self._visible_to_model(ctx):
self._fold_reveals(ctx)
return {
self.tool_name: ToolsetTool(
toolset=self,
tool_def=ToolDefinition(
name=self.tool_name,
description=self._description,
parameters_json_schema=_WORKFLOW_ARGS_JSON_SCHEMA,
metadata={'code_arg_name': 'code', 'code_arg_language': 'python'},
sequential=True,
),
max_retries=self.max_retries,
args_validator=_WORKFLOW_ARGS_VALIDATOR,
)
}
async def _run_one(self, agent_name: str, task: str, ctx: RunContext[AgentDepsT]) -> Any:
"""Run one sub-agent against the shared per-run budget.
The budget check + increment must stay suspension-free: there must be no `await`
between them. asyncio only switches tasks at suspension points, so an await-free
check-then-increment is atomic across the concurrently-gathered dispatches, which
is what makes `max_agent_calls` an exact ceiling under fan-out. Insert an `await`
here (e.g. an async permission check) and the count can race past the limit; you
would then need an explicit reservation instead.
This exists precisely because `usage_limits` cannot give an exact ceiling here:
core's own limit check is split from its increment by the model-request `await`
(a TOCTOU race -- N gathered sub-agents all pass the check before any increments;
measured ~20x overshoot), and `RunContext` exposes `usage` but not `usage_limits`,
so the parent's configured limit can't be forwarded to sub-agents at all.
TODO: file upstream on pydantic-ai -- (a) expose `usage_limits` on `RunContext`,
(b) atomic reserve-then-request in the run loop. Until then, tree-wide token caps
stay best-effort (see `forward_usage` / `sub_agent_usage_limits` docstrings).
"""
if self._call_count >= self.max_agent_calls:
raise _BudgetExhausted(self.max_agent_calls)
self._call_count += 1
try:
result = await self._by_name[agent_name].agent.run(
task,
deps=ctx.deps,
model=ctx.model if self.inherit_model else None,
usage=ctx.usage if self.forward_usage else None,
usage_limits=self.sub_agent_usage_limits,
)
return to_jsonable_python(result.output)
except Exception as exc:
# Don't leak host internals (file paths, deps/agent reprs) to the model;
# surface the failing agent and error type by default.
message = f'sub-agent {agent_name!r} raised {type(exc).__name__}'
if isinstance(exc, _MODEL_SAFE_EXCEPTION_MESSAGE_TYPES):
message = f'{message}: {exc}'
raise RuntimeError(message) from exc
def _type_check(self, code: str, capture: PrintCapture) -> None:
"""Statically check the script against the sub-agent signatures before running it.
Each `run_workflow` call is a fresh sandbox with no accumulated state, so the check is
always sound. Catching a positional `task`, a misspelled function, or a wrong-typed
argument here costs a retry but no sub-agent budget.
"""
signatures = [_agent_signature(name, entry.agent) for name, entry in self._by_name.items()]
conflicting = FunctionSignature.get_conflicting_type_names(signatures)
parts = ['import asyncio\nfrom typing import Any, TypedDict, NotRequired, Literal']
parts.extend(FunctionSignature.render_type_definitions(signatures, conflicting))
parts.extend(
signature.render('raise NotImplementedError()', is_async=True, conflicting_type_names=conflicting)
for signature in signatures
)
try:
Monty(code, type_check=True, type_check_stubs='\n\n'.join(parts))
except MontyTypingError as e:
raise ModelRetry(f'Type error in workflow:\n{capture.prepend_to(e.display())}') from e
except MontySyntaxError as e:
raise ModelRetry(f'Syntax error in workflow:\n{capture.prepend_to(e.display())}') from e
async def call_tool(
self, name: str, tool_args: dict[str, Any], ctx: RunContext[AgentDepsT], tool: ToolsetTool[AgentDepsT]
) -> Any:
if _in_workflow.get():
return {
'error': (
'Workflows do not nest: this sub-agent was invoked from a workflow and cannot start '
'its own. Return your result to the orchestrating workflow instead.'
)
}
code = tool_args['code']
budget_exhausted = False
completed_dispatches: list[_CompletedDispatch] = []
async def dispatch(agent_name: str, kwargs: dict[str, Any]) -> Any:
nonlocal budget_exhausted
# The sandbox signature is `(*, task: str)`, but Monty does not validate kwargs against
# it at runtime, and the static check can be evaded through `Any` (e.g. `json.loads`
# results) -- so check here: a dropped extra kwarg or a non-string `task` would
# otherwise run the sub-agent on silently-wrong input. Each raises before the budget
# is touched.
if 'task' not in kwargs:
raise TypeError(f'{agent_name}() missing required keyword argument: task')
extra = sorted(set(kwargs) - {'task'})
if extra:
raise TypeError(f'{agent_name}() got unexpected keyword argument(s): {", ".join(extra)}; only task')
task = kwargs['task']
if not isinstance(task, str):
raise TypeError(f'{agent_name}() task must be a string, got {type(task).__name__}')
try:
output = await self._run_one(agent_name, task, ctx)
except _BudgetExhausted:
budget_exhausted = True
raise
completed_dispatches.append(_CompletedDispatch(agent_name=agent_name, task=task, result=output))
return output
limits = _resolve_resource_limits(self.resource_limits)
capture = PrintCapture()
self._type_check(code, capture)
in_workflow_token = _in_workflow.set(True)
try:
repl = MontyRepl(limits=limits)
monty_state = repl.feed_start(code, print_callback=capture)
# `_by_name` is not mutated while a script executes (reveals land in `get_tools`,
# which does not interleave with `call_tool`), so it is a stable name registry for
# the whole script. Sub-agents always run concurrently (the executor's defaults);
# durable ordering (global_sequential) lands with durability.
completed = await MontyExecutor(dispatch=dispatch, valid_names=self._by_name).run(monty_state)
except MontySyntaxError as e: # pragma: no cover -- backstop; `_type_check` rejects syntax errors first
raise ModelRetry(f'Syntax error in workflow:\n{capture.prepend_to(e.display())}') from e
except MontyRuntimeError as e:
if budget_exhausted:
# On this capability's deferred-future path, host-raised exceptions cannot be
# caught inside the sandbox (Monty's inline resume path can catch them). The
# flag proves this script hit the budget; under gather, the displayed error may
# be another independently surfaced failure from the same batch.
return _budget_terminal_result(
max_agent_calls=self.max_agent_calls,
last_error=capture.prepend_to(e.display()),
completed_dispatches=completed_dispatches,
)
raise ModelRetry(
f'Runtime error in workflow:\n{capture.prepend_to(e.display())}'
f'{_completed_retry_section(completed_dispatches)}'
) from e
except BaseException as e:
# Convert a model-provokable sandbox panic to a retry (see `is_sandbox_panic`);
# anything else (CancelledError, ...) re-raises unchanged.
if not is_sandbox_panic(e):
raise
if budget_exhausted:
return _budget_terminal_result(
max_agent_calls=self.max_agent_calls,
last_error='The workflow script aborted inside the sandbox after exhausting the sub-agent budget.',
completed_dispatches=completed_dispatches,
)
raise ModelRetry(
'The workflow script aborted inside the sandbox. This can happen when the same '
'sub-agent call is awaited more than once in one asyncio.gather -- give each gathered '
'call its own invocation. Revise the script and try again.'
f'{_completed_retry_section(completed_dispatches)}'
) from e
finally:
_in_workflow.reset(in_workflow_token)
return _workflow_result(completed.output, capture.joined)
pydantic_ai_harness/experimental/_warn.py
+15
View File
@@ -38,3 +38,18 @@ def warn_experimental(feature: str) -> None:
category=HarnessExperimentalWarning,
stacklevel=2,
)
def warn_moved(old: str, new: str) -> None:
"""Emit a `DeprecationWarning` that `experimental.old` now lives at top-level `new`.
Left behind at each old `pydantic_ai_harness.experimental.<name>` path when a capability
graduates out of `experimental`, so existing imports keep working with a clear pointer to
the new location.
"""
warnings.warn(
f'`pydantic_ai_harness.experimental.{old}` has moved to `pydantic_ai_harness.{new}`. '
f'Update your imports; this compatibility shim will be removed in a future release.',
category=DeprecationWarning,
stacklevel=2,
)
pydantic_ai_harness/experimental/acp/README.md
+3 -1
View File
@@ -21,6 +21,8 @@
Expose a Pydantic AI agent to editors and terminal UIs over the [Agent Client Protocol](https://agentclientprotocol.com).
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/experimental/acp/)
## The problem
Editors like [Zed](https://zed.dev/docs/ai/external-agents) speak ACP: a stdio JSON-RPC protocol that lets a TUI or editor drive an external coding agent -- streaming its text, rendering its file edits as diffs, and prompting the user to approve sensitive tool calls. To plug a Pydantic AI agent into one of these editors you would otherwise have to implement the ACP server side yourself.
@@ -223,7 +225,7 @@ Each completed turn reports its token counts (input/output/total, plus cached to
## API
```python
```python {test="skip"}
run_acp_stdio( # async; serve until the client disconnects
agent,
*,
pydantic_ai_harness/experimental/authoring/__init__.py
+12 -7
View File
@@ -1,16 +1,21 @@
"""Runtime capability authoring: let an agent write, validate, and register real capabilities."""
"""Deprecated import location for `pydantic_ai_harness.runtime_authoring`.
from pydantic_ai_harness.experimental._warn import warn_experimental
from pydantic_ai_harness.experimental.authoring._capability import RuntimeAuthoring
from pydantic_ai_harness.experimental.authoring._store import AuthoredCapability, CapabilityStore
from pydantic_ai_harness.experimental.authoring._toolset import AuthoringToolset
from pydantic_ai_harness.experimental.authoring._validate import (
This capability graduated out of `experimental`; importing from here still works but
emits a `DeprecationWarning`. Import from `pydantic_ai_harness.runtime_authoring` instead.
"""
from pydantic_ai_harness.experimental._warn import warn_moved
from pydantic_ai_harness.runtime_authoring import (
AuthoredCapability,
AuthoringToolset,
CapabilityStore,
CapabilityValidationError,
RuntimeAuthoring,
load_capability_instance,
validate_capability_file,
)
warn_experimental('authoring')
warn_moved('authoring', 'runtime_authoring')
__all__ = [
'AuthoredCapability',
pydantic_ai_harness/experimental/compaction/__init__.py
+17 -13
View File
@@ -1,20 +1,24 @@
"""Compaction capabilities: keep an agent's conversation history within the context window.
"""Deprecated import location for `pydantic_ai_harness.compaction`.
Each capability lives in its own module; shared utilities (token estimation, the
`CompactionStrategy` protocol, tool-pair-safe cutoffs, in-place clearing) live in `_shared`.
This capability graduated out of `experimental`; importing from here still works but
emits a `DeprecationWarning`. Import from `pydantic_ai_harness.compaction` instead.
"""
from pydantic_ai_harness.experimental._warn import warn_experimental
from pydantic_ai_harness.experimental.compaction._clamp_oversized_messages import ClampOversizedMessages
from pydantic_ai_harness.experimental.compaction._clear_tool_results import ClearToolResults
from pydantic_ai_harness.experimental.compaction._deduplicate_file_reads import DeduplicateFileReads
from pydantic_ai_harness.experimental.compaction._limit_warner import LimitWarner, WarningKind
from pydantic_ai_harness.experimental.compaction._shared import CompactionStrategy, estimate_token_count
from pydantic_ai_harness.experimental.compaction._sliding_window import SlidingWindow
from pydantic_ai_harness.experimental.compaction._summarizing_compaction import SummarizingCompaction
from pydantic_ai_harness.experimental.compaction._tiered_compaction import TieredCompaction
from pydantic_ai_harness.compaction import (
ClampOversizedMessages,
ClearToolResults,
CompactionStrategy,
DeduplicateFileReads,
LimitWarner,
SlidingWindow,
SummarizingCompaction,
TieredCompaction,
WarningKind,
estimate_token_count,
)
from pydantic_ai_harness.experimental._warn import warn_moved
warn_experimental('compaction')
warn_moved('compaction', 'compaction')
__all__ = [
'ClampOversizedMessages',
pydantic_ai_harness/experimental/context/__init__.py
+21 -8
View File
@@ -1,11 +1,24 @@
"""Context capability: discover and load a repo's accumulated context engineering."""
"""Deprecated import location for `pydantic_ai_harness.context`.
from pydantic_ai_harness.experimental._warn import warn_experimental
from pydantic_ai_harness.experimental.context._capability import RepoContext
from pydantic_ai_harness.experimental.context._inventory import AgentContextInventory, AssetRoot
from pydantic_ai_harness.experimental.context._loader import ContextFile
from pydantic_ai_harness.experimental.context._toolset import RepoContextToolset
This capability graduated out of `experimental`; importing from here still works but
emits a `DeprecationWarning`. Import from `pydantic_ai_harness.context` instead.
"""
warn_experimental('context')
from pydantic_ai_harness.context import (
AgentContextInventory,
AssetRoot,
ContextFile,
RepoContext,
RepoContextToolset,
)
from pydantic_ai_harness.experimental._warn import warn_moved
__all__ = ['AgentContextInventory', 'AssetRoot', 'ContextFile', 'RepoContext', 'RepoContextToolset']
warn_moved('context', 'context')
__all__ = [
'AgentContextInventory',
'AssetRoot',
'ContextFile',
'RepoContext',
'RepoContextToolset',
]
pydantic_ai_harness/experimental/docs/__init__.py
+17 -6
View File
@@ -1,9 +1,20 @@
"""Docs capability: an on-demand tool that locates Pydantic AI documentation."""
"""Deprecated import location for `pydantic_ai_harness.docs`.
from pydantic_ai_harness.experimental._warn import warn_experimental
from pydantic_ai_harness.experimental.docs._capability import PyaiDocs
from pydantic_ai_harness.experimental.docs._toolset import PyaiDocsToolset, PyaiDocsTopic
This capability graduated out of `experimental`; importing from here still works but
emits a `DeprecationWarning`. Import from `pydantic_ai_harness.docs` instead.
"""
warn_experimental('docs')
from pydantic_ai_harness.docs import (
PyaiDocs,
PyaiDocsToolset,
PyaiDocsTopic,
)
from pydantic_ai_harness.experimental._warn import warn_moved
__all__ = ['PyaiDocs', 'PyaiDocsToolset', 'PyaiDocsTopic']
warn_moved('docs', 'docs')
__all__ = [
'PyaiDocs',
'PyaiDocsToolset',
'PyaiDocsTopic',
]
pydantic_ai_harness/experimental/dynamic_workflow/__init__.py
+22
View File
@@ -0,0 +1,22 @@
"""Deprecated import location for `pydantic_ai_harness.dynamic_workflow`.
This capability graduated out of `experimental`; importing from here still works but
emits a `DeprecationWarning`. Import from `pydantic_ai_harness.dynamic_workflow` instead.
"""
from pydantic_ai_harness.dynamic_workflow import (
DynamicWorkflow,
DynamicWorkflowToolset,
WorkflowAgent,
WorkflowResourceLimits,
)
from pydantic_ai_harness.experimental._warn import warn_moved
warn_moved('dynamic_workflow', 'dynamic_workflow')
__all__ = [
'DynamicWorkflow',
'DynamicWorkflowToolset',
'WorkflowAgent',
'WorkflowResourceLimits',
]
pydantic_ai_harness/experimental/media/__init__.py
+9 -10
View File
@@ -1,28 +1,27 @@
"""Content-addressed media stores for offloading large binary parts.
"""Deprecated import location for `pydantic_ai_harness.media`.
Used by `pydantic_ai_harness.experimental.step_persistence` to keep snapshots small when
messages carry `BinaryContent` payloads. A forthcoming `MediaExternalizer`
capability will reuse these stores for in-flight wire-payload reduction
(rewriting `BinaryContent` to URL parts before the model sees them).
This capability graduated out of `experimental`; importing from here still works but
emits a `DeprecationWarning`. Import from `pydantic_ai_harness.media` instead.
"""
from pydantic_ai_harness.experimental._warn import warn_experimental
from pydantic_ai_harness.experimental.media._s3 import S3MediaStore
from pydantic_ai_harness.experimental.media._store import (
from pydantic_ai_harness.experimental._warn import warn_moved
from pydantic_ai_harness.media import (
DiskMediaStore,
KeyStrategy,
MediaContext,
MediaStore,
PublicUrlResolver,
S3MediaStore,
SqliteMediaStore,
default_key_strategy,
externalize_media,
make_static_public_url,
media_uri_for,
parse_media_uri,
restore_media,
)
from pydantic_ai_harness.experimental.media._walker import externalize_media, restore_media
warn_experimental('media')
warn_moved('media', 'media')
__all__ = [
'DiskMediaStore',
pydantic_ai_harness/experimental/overflow/__init__.py
+12 -17
View File
@@ -1,32 +1,27 @@
"""Overflow capability: reduce oversized tool returns at production time.
"""Deprecated import location for `pydantic_ai_harness.overflowing_tool_output`.
`OverflowingToolOutput` intercepts a tool return when it is produced and reduces it --
truncating, spilling to a queryable file, or summarizing -- so an oversized payload does
not persist in history and get re-sent on every later model request. Combine the three
modes through an ordered list of size `bands`.
Spilled payloads are queried on demand through two registered tools: `grep_tool_result` to
search a payload and `read_tool_result` to read a line or byte range of it. The
`OverflowStore` protocol is the seam for a durable backend (the local-file default ships
for single-process runs).
This capability graduated out of `experimental`; importing from here still works but
emits a `DeprecationWarning`. Import from `pydantic_ai_harness.overflowing_tool_output` instead.
"""
from pydantic_ai_harness.experimental._warn import warn_experimental
from pydantic_ai_harness.experimental.overflow._bands import (
from pydantic_ai_harness.experimental._warn import warn_moved
from pydantic_ai_harness.overflowing_tool_output import (
GREP_TOOL_NAME,
READ_TOOL_NAME,
Action,
Band,
LocalFileStore,
OverflowingToolOutput,
OverflowStore,
Passthrough,
Spill,
Summarize,
SummarizeFunc,
Truncate,
TruncationStrategy,
)
from pydantic_ai_harness.experimental.overflow._capability import OverflowingToolOutput
from pydantic_ai_harness.experimental.overflow._markers import GREP_TOOL_NAME, READ_TOOL_NAME
from pydantic_ai_harness.experimental.overflow._payload import TruncationStrategy
from pydantic_ai_harness.experimental.overflow._store import LocalFileStore, OverflowStore
warn_experimental('overflow')
warn_moved('overflow', 'overflowing_tool_output')
__all__ = [
'GREP_TOOL_NAME',
pydantic_ai_harness/experimental/planning/__init__.py
+19 -6
View File
@@ -1,9 +1,22 @@
"""Planning capability: model-owned, cache-friendly task planning for agents."""
"""Deprecated import location for `pydantic_ai_harness.planning`.
from pydantic_ai_harness.experimental._warn import warn_experimental
from pydantic_ai_harness.experimental.planning._capability import Planning
from pydantic_ai_harness.experimental.planning._toolset import PlanItem, PlanningToolset, TaskStatus
This capability graduated out of `experimental`; importing from here still works but
emits a `DeprecationWarning`. Import from `pydantic_ai_harness.planning` instead.
"""
warn_experimental('planning')
from pydantic_ai_harness.experimental._warn import warn_moved
from pydantic_ai_harness.planning import (
PlanItem,
Planning,
PlanningToolset,
TaskStatus,
)
__all__ = ['PlanItem', 'Planning', 'PlanningToolset', 'TaskStatus']
warn_moved('planning', 'planning')
__all__ = [
'PlanItem',
'Planning',
'PlanningToolset',
'TaskStatus',
]
pydantic_ai_harness/experimental/step_persistence/__init__.py
+19 -19
View File
@@ -1,29 +1,29 @@
"""Step-event persistence: append-only event log, continuable snapshots, tool-effect ledger."""
"""Deprecated import location for `pydantic_ai_harness.step_persistence`.
from pydantic_ai_harness.experimental._warn import warn_experimental
from pydantic_ai_harness.experimental.step_persistence._capability import StepPersistence
from pydantic_ai_harness.experimental.step_persistence._helpers import (
This capability graduated out of `experimental`; importing from here still works but
emits a `DeprecationWarning`. Import from `pydantic_ai_harness.step_persistence` instead.
"""
from pydantic_ai_harness.experimental._warn import warn_moved
from pydantic_ai_harness.step_persistence import (
ContinuableSnapshot,
EventKind,
FileStepStore,
InMemoryStepStore,
RunRecord,
SqliteStepStore,
StepEvent,
StepPersistence,
StepStore,
ToolEffectRecord,
ToolEffectStatus,
annotate_tool_effect,
continue_run,
fork_run,
is_provider_valid,
)
from pydantic_ai_harness.experimental.step_persistence._store import (
FileStepStore,
InMemoryStepStore,
SqliteStepStore,
StepStore,
)
from pydantic_ai_harness.experimental.step_persistence._types import (
ContinuableSnapshot,
EventKind,
RunRecord,
StepEvent,
ToolEffectRecord,
ToolEffectStatus,
)
warn_experimental('step_persistence')
warn_moved('step_persistence', 'step_persistence')
__all__ = [
'ContinuableSnapshot',
pydantic_ai_harness/experimental/subagents/__init__.py
+16 -7
View File
@@ -1,12 +1,21 @@
"""Sub-agent capability: delegate self-contained tasks to named child agents."""
"""Deprecated import location for `pydantic_ai_harness.subagents`.
from pydantic_ai_harness.experimental._warn import warn_experimental
from pydantic_ai_harness.experimental.subagents._capability import SubAgents, ToolResolver
from pydantic_ai_harness.experimental.subagents._disk import AgentOverride
from pydantic_ai_harness.experimental.subagents._effort import MINIMUM_EFFORT_FLOOR, clamp_effort
from pydantic_ai_harness.experimental.subagents._toolset import SubAgent, SubAgentToolset
This capability graduated out of `experimental`; importing from here still works but
emits a `DeprecationWarning`. Import from `pydantic_ai_harness.subagents` instead.
"""
warn_experimental('subagents')
from pydantic_ai_harness.experimental._warn import warn_moved
from pydantic_ai_harness.subagents import (
MINIMUM_EFFORT_FLOOR,
AgentOverride,
SubAgent,
SubAgents,
SubAgentToolset,
ToolResolver,
clamp_effort,
)
warn_moved('subagents', 'subagents')
__all__ = [
'MINIMUM_EFFORT_FLOOR',
pydantic_ai_harness/filesystem/README.md
+12 -3
View File
@@ -2,6 +2,8 @@
Give an agent sandboxed, pattern-filtered access to a directory tree.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/filesystem/)
## The problem
Letting an agent touch the filesystem directly is risky: path traversal
@@ -33,7 +35,7 @@ print(result.output)
| Tool | Purpose |
|---|---|
| `read_file` | Read a text file with line numbers and a content hash. Binary files are detected and not dumped. |
| `read_file` | Read a text file with line numbers and a content hash. Binary files are detected and not dumped. Supports `offset`/`limit` paging. |
| `write_file` | Create or overwrite a file. Optional `expected_hash` rejects stale writes (optimistic concurrency). |
| `edit_file` | Exact-string replacement; `old_text` must match exactly once. Optional `expected_hash`. |
| `list_directory` | List a directory's entries with type indicators and sizes. |
@@ -66,7 +68,7 @@ need `**`.
| `denied_patterns` | Matching paths are always rejected (denylist). |
| `protected_patterns` | Matching paths are read-only -- reads succeed, writes are rejected. |
`protected_patterns` defaults to `.git/`, `.env`/`.env.*`, `*.pem`, `*.key`,
`protected_patterns` defaults to `.git/*`, `.env`/`.env.*`, `*.pem`, `*.key`,
and `**/secrets*`. Pass an empty list to disable protection.
### Direct access vs. walkers
@@ -86,13 +88,20 @@ The three rules apply at two different granularities:
So with `allowed_patterns=['*.py']`, `list_directory('.')` succeeds and shows
only the `.py` entries; `read_file('notes.md')` is rejected.
> Dotfiles and dot-directories (`.git`, `.env`, `.github`, …) are skipped by
Note that the walkers filter entries with write-level access, so
`protected_patterns` matches are omitted from `list_directory`, `search_files`,
and `find_files` output even though those exact paths remain directly readable
via `read_file`/`file_info`.
> Dotfiles and dot-directories (`.git`, `.env`, `.github`, ...) are skipped by
> all three walkers -- `list_directory`, `search_files`, and `find_files` --
> regardless of patterns.
## Configuration
```python
from pydantic_ai_harness import FileSystem
FileSystem(
root_dir='.', # str | Path -- sandbox root
allowed_patterns=[], # allowlist globs (empty = allow all)
pydantic_ai_harness/guardrails/README.md
+186
View File
@@ -0,0 +1,186 @@
# Input & Output Guardrails
Validate the user prompt before it reaches the model, and the model output before it reaches the caller.
> [!NOTE]
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/guardrails/)
## The problem
Agents take unstructured input from users and return unstructured output to callers. Without a validation layer, a prompt injection attempt, PII-laden message, or off-topic question goes to the model as-is, and any output the model produces is returned verbatim. The framework does not reason about "this is unsafe to send" or "this is unsafe to show".
## The solution
Two capabilities -- `InputGuard` and `OutputGuard` -- each backed by a `guard` callable you supply. The guard inspects a value (the prompt, or the output) and returns one of four outcomes:
| Outcome | `InputGuard` | `OutputGuard` |
|---|---|---|
| **allow** | send the prompt to the model | return the output to the caller |
| **block** | skip the model call; a refusal message becomes the response (`SkipModelRequest`) | raise `OutputBlocked` |
| **replace** | rewrite the prompt sent to the model (redaction) | substitute a sanitized output |
| **retry** | -- (not valid for input) | send the output back to the model to try again (`ModelRetry`) |
A guard that raises an exception instead propagates it as a hard failure. The asymmetry between input `block` and output `block` is intentional: blocking the input spends no tokens, so a graceful refusal is almost always right; blocking the output means the model already produced something you do not want exposed, so raising forces the caller to decide what to do next.
## Usage
A guard returns a bare `bool` (`True` = allow, `False` = block) for the simple case, or a `GuardResult` for the richer outcomes.
```python
from pydantic_ai import Agent
from pydantic_ai_harness import GuardResult, InputGuard, OutputGuard
def no_secrets(prompt: str) -> bool:
return 'api_key' not in prompt.lower()
def no_pii(output: object) -> GuardResult:
if 'SSN' in str(output):
return GuardResult.block('The response contained personal data.')
return GuardResult.allow()
agent = Agent(
'openai:gpt-5.4',
capabilities=[
InputGuard(guard=no_secrets),
OutputGuard(guard=no_pii),
],
)
```
`OutputGuard` receives the output unchanged -- no automatic stringification. For a string output the guard reads it directly; for a typed (Pydantic model) output the guard gets the model instance, so pick the serialization that fits the check (read a field, or call `output.model_dump_json()` for JSON text). This avoids the trap of `str(MyModel(...))` producing a `MyModel(field=...)` repr that hides field contents from regex-based checks.
Guards may also be async -- return an awaitable `bool`/`GuardResult`, e.g. to call a moderation API.
## `GuardResult`
Construct a `GuardResult` with its classmethods, not the raw fields:
```python
from pydantic_ai_harness import GuardResult
GuardResult.allow() # let the value through
GuardResult.block('reason') # refuse; `reason` is optional (a default is used otherwise)
GuardResult.replace(cleaned_value) # substitute a sanitized value and continue
GuardResult.retry('instruction') # OutputGuard only: ask the model to redo the output
```
The block/retry message is produced at the moment the guard decides, so it can carry the guard's own reasoning rather than a string frozen at construction time.
## Redaction (`replace`)
Return `GuardResult.replace(value)` to sanitize rather than refuse. `InputGuard` rewrites the prompt sent to the model; `OutputGuard` substitutes the output returned to the caller.
```python
def scrub_emails(text: str) -> GuardResult:
cleaned = EMAIL_RE.sub('[email]', text)
return GuardResult.replace(cleaned) if cleaned != text else GuardResult.allow()
agent = Agent(
'openai:gpt-5.4',
capabilities=[
InputGuard(guard=scrub_emails), # strip PII before it reaches the model
OutputGuard(guard=scrub_emails), # strip PII before it reaches the caller
],
)
```
Input redaction requires sequential mode -- it is incompatible with `parallel=True`, since a parallel guard runs alongside a model call that has already started with the original prompt.
## Retry (`retry`)
`OutputGuard` can send a bad output back to the model instead of blocking it. Return `GuardResult.retry(instruction)` -- the instruction is the retry prompt the model sees. This reuses pydantic-ai's normal retry machinery and counts against the run's output-retry budget.
```python
def must_cite_sources(output: object) -> GuardResult:
if not has_citations(output):
return GuardResult.retry('Include at least one source citation.')
return GuardResult.allow()
OutputGuard(guard=must_cite_sources)
```
## Streaming
`OutputGuard` inspects the **final** output only -- during `run_stream()` partial chunks reach the caller before the guard runs, so a `block` or `replace` verdict cannot un-send content already streamed. Use `run()` / `run_sync()` when the output must be screened before any of it is exposed. `GuardResult.retry()` is **not** supported under `run_stream()` -- pydantic-ai does not retry output during streaming, and a `retry` verdict there surfaces as `UnexpectedModelBehavior`. `InputGuard` (including `parallel=True`) works the same in streamed and non-streamed runs.
## Tracing
`replace` and `block` are recorded as spans on the active OpenTelemetry tracer, so a redaction or refusal shows up in Logfire traces (`guardrail redacted input`, `guardrail blocked output`, etc.) with `guardrail.*` attributes. Content attributes -- the original/replacement values for a redaction and the refusal `message` for a block -- are attached **only** when `RunContext.trace_include_content` is enabled, since these can quote the very content the guard exists to keep out of traces. `retry` needs no special tracing: the retried model request appears in the trace on its own.
`OutputGuard` declares `position='outermost', wrapped_by=[Instrumentation]` so its block/redact spans are always captured by an enclosing `Instrumentation` span regardless of how the user orders capabilities. `InputGuard` declares `position='innermost'` so any capability that morphs messages (a prompt rewriter, a context manager) runs first and the guard sees the final prompt the model will receive.
## Parallel input guards
A slow guard (an LLM classifier, a network call) run sequentially adds its latency to every turn. Set `parallel=True` to run the guard concurrently with the model call instead, overlapping the two so the guard adds no latency on the pass path. The model call is cancelled the moment the guard reports a violation.
```python
InputGuard(guard=slow_async_classifier, parallel=True)
```
Parallel mode trades tokens for latency: sequential mode never calls the model when the guard blocks, but parallel mode has already started the model call -- if the guard trips only after the model has responded, those tokens were spent. For fast local checks (regex, keyword lookup) sequential is the better default. `replace` is not available under `parallel=True` (see [Redaction](#redaction-replace)).
## Accessing run context
A guard may take a `RunContext` as its first parameter when it needs run state -- `deps` for tenant- or role-aware policy, message history for conversation-aware checks. The parameter is detected from the signature, so prompt-only guards need not declare it:
```python
from pydantic_ai import RunContext
from pydantic_ai_harness import InputGuard
def tenant_policy(ctx: RunContext[MyDeps], prompt: str) -> bool:
return ctx.deps.tier == 'pro' or 'advanced-feature' not in prompt
InputGuard(guard=tenant_policy)
```
## Hard-fail path
`block` is the graceful path. To make the caller see an exception instead, raise from the guard:
```python
from pydantic_ai_harness import InputBlocked
def strict_guard(prompt: str) -> bool:
if contains_credentials(prompt):
raise InputBlocked('credentials detected')
return True
```
Any exception raised by the guard propagates as-is -- use `InputBlocked` / `OutputBlocked` from this module, or your own exception types.
## API
```python {test="skip"}
@dataclass
class GuardResult:
action: Literal['allow', 'block', 'replace', 'retry']
message: str | None = None
replacement: object | None = None
# classmethods: allow(), block(message=None), replace(value), retry(message)
InputGuard(
guard: Callable[..., bool | GuardResult | Awaitable[bool | GuardResult]],
parallel: bool = False,
)
OutputGuard(
guard: Callable[..., bool | GuardResult | Awaitable[bool | GuardResult]],
)
```
The guard callable takes the inspected value -- the prompt for `InputGuard`, the output for `OutputGuard` -- optionally preceded by a `RunContext`.
## Relationship to `pydantic-ai-shields`
`pydantic-ai-shields` provides opinionated implementations on top of these primitives (prompt-injection detectors, PII scrubbers, keyword blocklists, etc.). Use the guardrails here when you want to plug in your own validation logic; reach for shields when you need a batteries-included detector.
pydantic_ai_harness/guardrails/__init__.py
+25
View File
@@ -0,0 +1,25 @@
"""Input and output guardrails for Pydantic AI agents."""
from pydantic_ai_harness.guardrails._capability import (
GuardResult,
InputGuard,
InputGuardFunc,
OutputGuard,
OutputGuardFunc,
)
from pydantic_ai_harness.guardrails._exceptions import (
GuardrailError,
InputBlocked,
OutputBlocked,
)
__all__ = [
'GuardResult',
'GuardrailError',
'InputBlocked',
'InputGuard',
'InputGuardFunc',
'OutputBlocked',
'OutputGuard',
'OutputGuardFunc',
]
pydantic_ai_harness/guardrails/_capability.py
+456
View File
@@ -0,0 +1,456 @@
"""Input and output guardrail capabilities.
`InputGuard` intercepts the first model request and lets a user-supplied
callable decide what to do with the user prompt. `OutputGuard` runs as the
model output is processed and decides what to do with the agent output.
A guard returns a bare `bool` (`True` = allow) or a
[`GuardResult`][pydantic_ai_harness.guardrails.GuardResult] — one of four
outcomes:
- `allow` — let the value through unchanged.
- `block` — refuse: `InputGuard` short-circuits the model call with a refusal
message via [`SkipModelRequest`][pydantic_ai.exceptions.SkipModelRequest];
`OutputGuard` raises [`OutputBlocked`][pydantic_ai_harness.guardrails.OutputBlocked].
- `replace` — substitute a sanitized value (redaction) and continue.
- `retry` — send the output back to the model to try again (`OutputGuard` only).
A guard that raises propagates the exception so the caller sees a hard
failure. Guards may be sync or async and may optionally take a
[`RunContext`][pydantic_ai.tools.RunContext] as their first argument.
`replace` and `block` are recorded as spans on the active OpenTelemetry
tracer, so a redaction or refusal is visible in Logfire traces. Content
attributes (the original/replacement values and the refusal message) are
attached only when `RunContext.trace_include_content` is set.
"""
from __future__ import annotations
import asyncio
import inspect
from collections.abc import Awaitable, Callable, Sequence
from dataclasses import dataclass
from typing import TYPE_CHECKING, Any, Literal
from pydantic_ai.capabilities import AbstractCapability, CapabilityOrdering, Instrumentation, WrapModelRequestHandler
from pydantic_ai.exceptions import ModelRetry, SkipModelRequest, UserError
from pydantic_ai.messages import ModelMessage, ModelResponse, TextPart, UserPromptPart
from pydantic_ai.tools import AgentDepsT, RunContext
from typing_extensions import assert_never
from pydantic_ai_harness.guardrails._exceptions import OutputBlocked
if TYPE_CHECKING:
from pydantic_ai.models import ModelRequestContext
from pydantic_ai.output import OutputContext
_DEFAULT_INPUT_BLOCK_MESSAGE = 'Request blocked by input guardrail.'
_DEFAULT_OUTPUT_BLOCK_MESSAGE = 'Output blocked by output guardrail.'
_DEFAULT_OUTPUT_RETRY_MESSAGE = 'Output rejected by output guardrail.'
@dataclass(frozen=True, kw_only=True)
class GuardResult:
"""The outcome a guard reports for the value it inspected.
Construct one with the classmethods — `GuardResult.allow()`,
`GuardResult.block()`, `GuardResult.replace()`, `GuardResult.retry()` —
rather than the raw fields. A guard may also return a bare `bool`: `True`
is `allow()`, `False` is `block()`.
"""
action: Literal['allow', 'block', 'replace', 'retry']
"""What the capability should do with the inspected value."""
message: str | None = None
"""For `block`, the refusal text. For `retry`, the instruction sent back to the model."""
replacement: object | None = None
"""For `replace`, the value substituted for the inspected one."""
def __post_init__(self) -> None:
"""Reject field combinations the four-outcome contract does not allow."""
match self.action:
case 'allow':
if self.message is not None or self.replacement is not None:
raise UserError("GuardResult(action='allow') must not set `message` or `replacement`.")
case 'replace':
if self.replacement is None:
raise UserError("GuardResult(action='replace') requires a `replacement` value.")
case 'retry':
if self.message is None:
raise UserError("GuardResult(action='retry') requires a `message`.")
case 'block':
# `message=None` is valid: a default is supplied at the use site.
pass
case _: # pragma: no cover - assert_never exhaustiveness guard
assert_never(self.action)
@classmethod
def allow(cls) -> GuardResult:
"""Let the value through unchanged."""
return cls(action='allow')
@classmethod
def block(cls, message: str | None = None) -> GuardResult:
"""Refuse the value. `message` is the refusal text; `None` uses a default."""
return cls(action='block', message=message)
@classmethod
def replace(cls, value: object) -> GuardResult:
"""Substitute `value` for the inspected one and continue.
For `InputGuard`, `value` is the replacement prompt text sent to the
model. For `OutputGuard`, it is the agent output returned to the caller.
"""
return cls(action='replace', replacement=value)
@classmethod
def retry(cls, message: str) -> GuardResult:
"""Send the output back to the model to try again — `OutputGuard` only.
`message` is the instruction the model sees on the retry.
"""
return cls(action='retry', message=message)
GuardOutcome = bool | GuardResult
"""What a guard callable returns: a bare `bool` (`True` = allow), or a `GuardResult`."""
InputGuardFunc = (
Callable[[str], GuardOutcome | Awaitable[GuardOutcome]]
| Callable[[RunContext[AgentDepsT], str], GuardOutcome | Awaitable[GuardOutcome]]
)
"""Signature of the callable passed to `InputGuard`.
The callable receives the user prompt and returns `True` / `GuardResult`. It
may optionally take a [`RunContext`][pydantic_ai.tools.RunContext] as a first
argument — for `deps`, message history, or other run state — and may be sync
or async. Raising an exception is treated as a hard failure and propagates up
to the caller.
"""
OutputGuardFunc = (
Callable[[object], GuardOutcome | Awaitable[GuardOutcome]]
| Callable[[RunContext[AgentDepsT], object], GuardOutcome | Awaitable[GuardOutcome]]
)
"""Signature of the callable passed to `OutputGuard`.
The callable receives the agent output unchanged — for typed outputs this is
the Pydantic model — and returns `True` / `GuardResult`. It may optionally take
a [`RunContext`][pydantic_ai.tools.RunContext] first, and may be sync or async.
"""
def _takes_ctx(func: Callable[..., object]) -> bool:
"""Return `True` when `func` declares a leading `RunContext` parameter.
Detected by parameter count, not annotation: a guard always takes the
guarded value, so a second parameter means it also wants the run context.
This matches pydantic-ai's own optional-`ctx` convention for output
validators. A callable whose signature cannot be introspected is treated
as taking the value only.
"""
try:
parameters = inspect.signature(func).parameters
except ValueError: # pragma: no cover - callable without an introspectable signature
return False
return len(parameters) > 1
async def _evaluate(
guard: Callable[..., GuardOutcome | Awaitable[GuardOutcome]],
ctx: RunContext[AgentDepsT],
value: object,
) -> GuardResult:
"""Call `guard` (passing `ctx` when declared), await it, and normalize to `GuardResult`."""
outcome = guard(ctx, value) if _takes_ctx(guard) else guard(value)
if inspect.isawaitable(outcome):
outcome = await outcome
if isinstance(outcome, GuardResult):
return outcome
return GuardResult.allow() if outcome else GuardResult.block()
def _extract_prompt(ctx: RunContext[AgentDepsT], messages: Sequence[ModelMessage]) -> str | None:
"""Return the text of the most recent user prompt, or `None` if absent.
Prefers `ctx.prompt` (set at run start) and falls back to scanning the
message history for the last [`UserPromptPart`][pydantic_ai.messages.UserPromptPart]
so that sub-agent calls or resumed runs without a fresh prompt still work.
"""
if ctx.prompt is not None:
return ctx.prompt if isinstance(ctx.prompt, str) else str(ctx.prompt)
for message in reversed(messages):
for part in reversed(message.parts):
if isinstance(part, UserPromptPart):
return part.content if isinstance(part.content, str) else str(part.content)
return None
def _replace_prompt(messages: Sequence[ModelMessage], new_content: str) -> bool:
"""Rewrite the most recent user prompt to `new_content`. Returns whether one was found."""
for message in reversed(messages):
for part in reversed(message.parts):
if isinstance(part, UserPromptPart):
part.content = new_content
return True
return False
def _trace_block(ctx: RunContext[AgentDepsT], *, direction: str, message: str) -> None:
"""Record a zero-duration span marking a guardrail refusal.
The refusal message is attached only when `ctx.trace_include_content` is
set — it can quote sensitive content from the guarded value, and ops
audiences are broader than the user who sees the refusal text.
"""
attributes: dict[str, str] = {'guardrail.direction': direction, 'guardrail.action': 'block'}
if ctx.trace_include_content:
attributes['guardrail.message'] = message
ctx.tracer.start_span(f'guardrail blocked {direction}', attributes=attributes).end()
def _trace_redaction(ctx: RunContext[AgentDepsT], *, direction: str, original: object, replacement: object) -> None:
"""Record a zero-duration span marking a guardrail redaction.
The original and replacement values are attached only when
`ctx.trace_include_content` is set, since a redacted value is often the
sensitive content the guard exists to keep out of traces.
"""
attributes: dict[str, str] = {'guardrail.direction': direction, 'guardrail.action': 'replace'}
if ctx.trace_include_content:
attributes['guardrail.original'] = str(original)
attributes['guardrail.replacement'] = str(replacement)
ctx.tracer.start_span(f'guardrail redacted {direction}', attributes=attributes).end()
@dataclass
class InputGuard(AbstractCapability[AgentDepsT]):
"""Validate the user prompt before it reaches the model.
The `guard` callable receives the prompt text and returns one of the four
outcomes (see the module docstring). `replace` rewrites the prompt sent to
the model and also overwrites the original in the run's message history,
so a redacted secret is not retained; a `str` replacement overwrites a
multimodal prompt's other parts. `retry` is not valid for an input guard.
```python
from pydantic_ai import Agent
from pydantic_ai_harness import GuardResult, InputGuard
def no_secrets(prompt: str) -> GuardResult:
if 'api_key' in prompt.lower():
return GuardResult.block('Your message looks like it contains an API key.')
return GuardResult.allow()
agent = Agent('openai:gpt-5.4', capabilities=[InputGuard(guard=no_secrets)])
```
The guard may take a [`RunContext`][pydantic_ai.tools.RunContext] as a
first parameter when it needs run state — `deps` for tenant/role-aware
policy, `messages` for conversation-aware checks. The parameter is detected
from the signature, so prompt-only guards stay as-is.
Set `parallel=True` to run the guard concurrently with the model call
rather than before it, overlapping a slow guard (an LLM classifier, a
network call) with the model round-trip so it adds no latency on the pass
path. The model call is cancelled the moment the guard reports a
violation. Trade-off: sequential mode never calls the model on a blocked
prompt, whereas parallel mode has already started it — if the guard trips
only after the model has responded, those tokens were still spent. Prefer
sequential for fast local guards. `replace` (redaction) is incompatible
with `parallel=True`, since the model call has already started with the
original prompt.
Scope: the guard runs exactly once per run — on the first model request —
and evaluates the original user prompt. Subsequent model requests in the
same run (e.g. after tool calls) are not re-checked, since the user input
has not changed.
Ordering: declares `position='innermost'` so any capability that morphs
the messages (a prompt rewriter, a context manager) runs first and the
guard sees the final prompt that will reach the model.
"""
guard: InputGuardFunc[AgentDepsT]
"""Callable that decides what to do with the prompt before it reaches the model."""
parallel: bool = False
"""Run the guard concurrently with the model request and cancel the model call on failure."""
def get_ordering(self) -> CapabilityOrdering:
"""Sit innermost so message-morphing capabilities run first and the guard sees the final prompt."""
return CapabilityOrdering(position='innermost')
async def _run_guard(
self,
ctx: RunContext[AgentDepsT],
request_context: ModelRequestContext,
prompt: str,
) -> None:
"""Evaluate the guard and act on its verdict.
`allow` returns; `block` raises `SkipModelRequest`; `replace` rewrites
the prompt in `request_context`; `retry` and `replace` under
`parallel=True` raise `UserError`.
"""
verdict = await _evaluate(self.guard, ctx, prompt)
match verdict.action:
case 'allow':
return
case 'retry':
raise UserError(
'An InputGuard guard cannot return GuardResult.retry() — retry applies to model output only.'
)
case 'block':
message = verdict.message or _DEFAULT_INPUT_BLOCK_MESSAGE
_trace_block(ctx, direction='input', message=message)
raise SkipModelRequest(ModelResponse(parts=[TextPart(content=message)]))
case 'replace':
if self.parallel:
raise UserError(
'InputGuard(parallel=True) is incompatible with GuardResult.replace(): the model call has '
'already started with the original prompt. Use sequential mode for prompt redaction.'
)
replacement = verdict.replacement
if not isinstance(replacement, str):
raise UserError(
'GuardResult.replace() for an input guard must provide replacement prompt text (str).'
)
if not _replace_prompt(request_context.messages, replacement):
raise UserError('InputGuard could not find a user prompt to redact in the request.')
_trace_redaction(ctx, direction='input', original=prompt, replacement=replacement)
case _: # pragma: no cover - assert_never exhaustiveness guard
assert_never(verdict.action)
async def wrap_model_request(
self,
ctx: RunContext[AgentDepsT],
*,
request_context: ModelRequestContext,
handler: WrapModelRequestHandler,
) -> ModelResponse:
"""Check the prompt before the first model call.
Sequential mode runs the guard then the model. `parallel=True` races
the guard against the model call and cancels it on a violation.
"""
if ctx.run_step > 1:
return await handler(request_context)
prompt = _extract_prompt(ctx, request_context.messages)
if prompt is None:
return await handler(request_context)
if not self.parallel:
await self._run_guard(ctx, request_context, prompt)
return await handler(request_context)
async def run_handler() -> ModelResponse:
return await handler(request_context)
guard_task: asyncio.Task[None] = asyncio.create_task(self._run_guard(ctx, request_context, prompt))
handler_task: asyncio.Task[ModelResponse] = asyncio.create_task(run_handler())
try:
done, _ = await asyncio.wait(
{guard_task, handler_task},
return_when=asyncio.FIRST_COMPLETED,
)
if guard_task in done:
await guard_task
return await handler_task
response = await handler_task
await guard_task
return response
finally:
for task in (guard_task, handler_task):
if not task.done():
task.cancel()
await asyncio.gather(guard_task, handler_task, return_exceptions=True)
@dataclass
class OutputGuard(AbstractCapability[AgentDepsT]):
"""Validate the agent output as it is produced.
The `guard` callable receives the output — no automatic stringification, so
a typed output arrives as the Pydantic model instance — and returns one of
the four outcomes (see the module docstring): `allow` exposes the output,
`block` raises [`OutputBlocked`][pydantic_ai_harness.guardrails.OutputBlocked],
`replace` substitutes a sanitized output (redaction), and `retry` sends the
output back to the model to try again.
```python
from pydantic_ai import Agent
from pydantic_ai_harness import GuardResult, OutputGuard
def no_pii(output: object) -> GuardResult:
if 'SSN' in str(output):
return GuardResult.retry('Do not include personal identifiers.')
return GuardResult.allow()
agent = Agent('openai:gpt-5.4', capabilities=[OutputGuard(guard=no_pii)])
```
The guard runs as the output is processed, so `retry` reuses pydantic-ai's
normal retry machinery and counts against the run's output-retry budget.
Like `InputGuard`, the guard may take a
[`RunContext`][pydantic_ai.tools.RunContext] as a first parameter; it is
detected from the signature.
Ordering: declares `position='outermost'` so the guard sees the final
output after every inner capability has processed it, and `wrapped_by=
[Instrumentation]` so an enclosing `Instrumentation` span always captures
the guard's block/redact spans regardless of user list order.
Streaming caveats. `retry` is supported with
[`run()`][pydantic_ai.Agent.run] / `run_sync()` only — pydantic-ai does not
support output retries during [`run_stream()`][pydantic_ai.Agent.run_stream],
where a `retry` verdict surfaces as
[`UnexpectedModelBehavior`][pydantic_ai.exceptions.UnexpectedModelBehavior].
The guard inspects only the final output, not partial chunks, so during a
streamed run the caller may already have received partial output before a
`block` or `replace` verdict is reached — use `run()` when the output must
be screened before any of it is exposed.
"""
guard: OutputGuardFunc[AgentDepsT]
"""Callable that decides what to do with the agent output."""
def get_ordering(self) -> CapabilityOrdering:
"""Sit outermost (inside `Instrumentation`) so the guard sees the final processed output."""
return CapabilityOrdering(position='outermost', wrapped_by=[Instrumentation])
async def after_output_process(
self,
ctx: RunContext[AgentDepsT],
*,
output_context: OutputContext,
output: Any,
) -> Any:
"""Evaluate the guard against the processed output and act on its verdict."""
if ctx.partial_output:
return output
verdict = await _evaluate(self.guard, ctx, output)
match verdict.action:
case 'allow':
return output
case 'block':
message = verdict.message or _DEFAULT_OUTPUT_BLOCK_MESSAGE
_trace_block(ctx, direction='output', message=message)
raise OutputBlocked(message)
case 'retry':
raise ModelRetry(verdict.message or _DEFAULT_OUTPUT_RETRY_MESSAGE)
case 'replace':
_trace_redaction(ctx, direction='output', original=output, replacement=verdict.replacement)
return verdict.replacement
case _: # pragma: no cover - assert_never exhaustiveness guard
assert_never(verdict.action)
pydantic_ai_harness/guardrails/_exceptions.py
+20
View File
@@ -0,0 +1,20 @@
"""Exceptions raised by the guardrail capabilities."""
from __future__ import annotations
class GuardrailError(Exception):
"""Base exception for guardrail violations."""
class InputBlocked(GuardrailError):
"""Raised by a user-supplied input guard to hard-fail a run.
Prefer returning `False` from the guard callable to trigger a graceful
refusal via [`SkipModelRequest`][pydantic_ai.exceptions.SkipModelRequest].
Raise this explicitly when the caller should have to handle the failure.
"""
class OutputBlocked(GuardrailError):
"""Raised by [`OutputGuard`][pydantic_ai_harness.OutputGuard] when the final output fails validation."""
pydantic_ai_harness/logfire/README.md
+5 -3
View File
@@ -9,6 +9,8 @@ Install the extra:
pip install 'pydantic-ai-harness[logfire]'
```
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/logfire/)
## `ManagedPrompt`
Back an agent's instructions with a Logfire-managed
@@ -23,7 +25,7 @@ Back an agent's instructions with a Logfire-managed
### The problem
Prompts are critical to agent behavior, but iterating on them through the normal
edit → review → deploy loop is slow, and you can't easily A/B test a change or roll it
edit -> review -> deploy loop is slow, and you can't easily A/B test a change or roll it
back the moment it misbehaves in production.
### The solution
@@ -142,13 +144,13 @@ Rendering requires `pydantic-handlebars` (install `pydantic-ai-slim[spec]`). It
### Prompt-cache trade-off
The resolved value lands in the agent's **system instructions**. Provider prompt caches (Anthropic,
OpenAI, etc.) key strictly by prefix -- `tools → system → messages` -- so any change to the system
OpenAI, etc.) key strictly by prefix -- `tools -> system -> messages` -- so any change to the system
block invalidates the cached prefix for the affected runs.
| Mode | Cache impact |
| --- | --- |
| Pinned `label='production'`, no rollout split | **Cache-stable.** The value only changes on a deliberate prompt rollout, which is the same cost as a redeploy. |
| Percentage rollout across labels (no `label=`) | Different runs land on different labels → splits the cache into one lane per label. |
| Percentage rollout across labels (no `label=`) | Different runs land on different labels -> splits the cache into one lane per label. |
| `targeting_key` per user/tenant with multiple labels in play | Cache lanes per assigned label; deterministic per key but still N lanes overall. |
| Mid-traffic label flip in the Logfire UI | One-shot cold-invalidation for everyone on that label. |
pydantic_ai_harness/media/README.md
+66
View File
@@ -0,0 +1,66 @@
# Media Externalization
> [!NOTE]
> Import these helpers from their submodule -- there is no top-level `pydantic_ai_harness` re-export:
>
> ```python
> from pydantic_ai_harness.media import (
> DiskMediaStore,
> S3MediaStore,
> SqliteMediaStore,
> externalize_media,
> restore_media,
> )
> ```
>
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
Content-addressed stores and walker helpers that move large binary payloads out of message history and put them back on demand.
These are building blocks, not a capability. There is no class you add to `Agent(capabilities=[...])` yet. [`StepPersistence`](../step_persistence/) uses them to keep snapshots small when messages carry `BinaryContent`. A forthcoming `MediaExternalizer` capability will reuse the same stores to rewrite `BinaryContent` into URL parts before the model sees them.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/media/)
## Why
A conversation that carries images, audio, or other `BinaryContent` inlines those bytes into every message. Persist that history and each snapshot re-serializes the payloads. Content-addressed storage writes each payload once, keyed by its own hash, and leaves a short `media://` URI in its place. The same bytes are stored once no matter how many messages or snapshots reference them.
## Stores
Every store implements the `MediaStore` protocol: `put`, `get`, `exists`, `public_url`, and `get_metadata`, all async and content-addressed (the URI is derived from the payload hash, so identical bytes deduplicate).
| Store | Backed by | Use when |
|---|---|---|
| `DiskMediaStore(directory=...)` | A directory on disk | Local runs and tests |
| `SqliteMediaStore(...)` | A SQLite database | A single-file store that travels with the data |
| `S3MediaStore(...)` | S3 or an S3-compatible bucket | Shared or production storage |
A `KeyStrategy` controls the on-store layout, and a `PublicUrlResolver` (or `make_static_public_url`) turns a stored URI into a public URL when the store is served over HTTP.
## Walker helpers
`externalize_media` and `restore_media` walk a message node and swap payloads for URIs and back.
```python
store = DiskMediaStore(directory='./media')
# Replace BinaryContent larger than the threshold with media:// URIs.
lean = await externalize_media(message, media_store=store, threshold_bytes=32_000)
# Later, rehydrate the URIs back into BinaryContent.
full = await restore_media(lean, media_store=store)
```
`externalize_media` only externalizes payloads over `threshold_bytes`; smaller ones stay inline. `media_uri_for` and `parse_media_uri` give you the raw URI round-trip if you need to key media yourself.
## API
| Symbol | Purpose |
|---|---|
| `MediaStore` | Async content-addressed store protocol (`put` / `get` / `exists` / `public_url` / `get_metadata`) |
| `DiskMediaStore`, `SqliteMediaStore`, `S3MediaStore` | Concrete stores |
| `MediaContext` | Per-call context (e.g. tenant) threaded through store operations |
| `KeyStrategy`, `default_key_strategy` | On-store key layout |
| `PublicUrlResolver`, `make_static_public_url` | Resolve a stored URI to a public URL |
| `externalize_media`, `restore_media` | Walk a message node to externalize / rehydrate payloads |
| `media_uri_for`, `parse_media_uri` | Compute and parse a `media://` URI |
pydantic_ai_harness/media/__init__.py
+38
View File
@@ -0,0 +1,38 @@
"""Content-addressed media stores for offloading large binary parts.
Used by `pydantic_ai_harness.step_persistence` to keep snapshots small when
messages carry `BinaryContent` payloads. A forthcoming `MediaExternalizer`
capability will reuse these stores for in-flight wire-payload reduction
(rewriting `BinaryContent` to URL parts before the model sees them).
"""
from pydantic_ai_harness.media._s3 import S3MediaStore
from pydantic_ai_harness.media._store import (
DiskMediaStore,
KeyStrategy,
MediaContext,
MediaStore,
PublicUrlResolver,
SqliteMediaStore,
default_key_strategy,
make_static_public_url,
media_uri_for,
parse_media_uri,
)
from pydantic_ai_harness.media._walker import externalize_media, restore_media
__all__ = [
'DiskMediaStore',
'KeyStrategy',
'MediaContext',
'MediaStore',
'PublicUrlResolver',
'S3MediaStore',
'SqliteMediaStore',
'default_key_strategy',
'externalize_media',
'make_static_public_url',
'media_uri_for',
'parse_media_uri',
'restore_media',
]
pydantic_ai_harness/experimental/media/_s3.py → pydantic_ai_harness/media/_s3.py
+1 -2
View File
@@ -22,7 +22,7 @@ from datetime import datetime, timezone
from typing import TYPE_CHECKING
from urllib.parse import quote
from pydantic_ai_harness.experimental.media._store import (
from pydantic_ai_harness.media._store import (
_EMPTY_CONTEXT, # pyright: ignore[reportPrivateUsage]
KeyStrategy,
MediaContext,
@@ -36,7 +36,6 @@ from pydantic_ai_harness.experimental.media._store import (
if TYPE_CHECKING:
import httpx
_ALGORITHM = 'AWS4-HMAC-SHA256'
_SERVICE = 's3'
# Conservative subset for x-amz-meta-* keys: ASCII letters/digits/dash; lowercase.
pydantic_ai_harness/experimental/media/_store.py → pydantic_ai_harness/media/_store.py
-1
View File
@@ -61,7 +61,6 @@ class MediaContext:
_EMPTY_CONTEXT = MediaContext()
PublicUrlResolver = Callable[[str, MediaContext], 'str | None | Awaitable[str | None]']
"""User-supplied callable that turns a `media+sha256://<hex>` URI into a fetchable URL.
pydantic_ai_harness/experimental/media/_walker.py → pydantic_ai_harness/media/_walker.py
+1 -1
View File
@@ -16,7 +16,7 @@ from __future__ import annotations
import base64
from typing import TypeGuard
from pydantic_ai_harness.experimental.media._store import MediaContext, MediaStore
from pydantic_ai_harness.media._store import MediaContext, MediaStore
_EXTERNAL_MARKER = '__harness_external_media__'
pydantic_ai_harness/memory/README.md
+218
View File
@@ -0,0 +1,218 @@
# Memory
Give an agent a persistent notebook that it can update, search, and reuse across runs without loading every stored file into every prompt.
> [!NOTE]
> Import this capability from its submodule. It is not re-exported from `pydantic_ai_harness`:
>
> ```python
> from pydantic_ai_harness.memory import Memory
> ```
Memory is a released, non-experimental capability. Pydantic AI Harness is still on 0.x releases, so the API may change between minor releases. See the repository [version policy](https://github.com/pydantic/pydantic-ai-harness#version-policy).
## Notebook model
Memory gives each agent a notebook made of Markdown files:
- `MEMORY.md` is the main notebook. By default, a bounded excerpt and the names of other files are added to the current request as delimited user-role context.
- Other files hold longer or focused notes. The model reads them on demand or finds them with bounded text search.
The model gets four tools:
| Tool | Purpose |
| --- | --- |
| `write_memory` | Append to a file or replace one unique text fragment. Writes use optimistic concurrency and an idempotency identifier derived from the run and tool call. |
| `read_memory` | Read a bounded prefix of one memory file. |
| `delete_memory` | Delete a file. The main notebook is protected. |
| `search_memory` | Search across notebook files, subject to configured result, character, and file-scan limits. |
```python
from pydantic_ai import Agent
from pydantic_ai_harness.memory import FileStore, Memory
agent = Agent(
'anthropic:claude-sonnet-4-6',
capabilities=[Memory(FileStore('.agent-memory'))],
defer_model_check=True,
)
```
The namespace is resolved by application code, not supplied to the tools. The model therefore cannot select another user's namespace in a tool call.
## Injection modes and limits
Automatic injection is enabled by default. Trusted usage guidance remains in model instructions, while model-written memory is enclosed in `<memory>` delimiters in a user-role part on the current request. Together, the guidance, main notebook, and file listing share a finite `max_tokens` budget, estimated at four characters per token. The default is 2,000 approximate tokens. `max_lines` is an additional limit on the main notebook. Backend reads are limited by `max_memory_size`, and the number of requested paths is derived from the prompt budget, so the capability never requests an unbounded file or listing. Content that does not fit is omitted with a prompt directing the model to use `read_memory` or `search_memory`.
```python
from pydantic_ai_harness.memory import FileStore, Memory
memory = Memory(
FileStore('.agent-memory'),
max_tokens=2_000,
max_lines=200,
)
```
Only the current request retains the injected user-role part, so copies do not accumulate in message history. Each model request receives the latest bounded snapshot, including after `write_memory` or an external update changes `MEMORY.md`.
Set `inject_memory=False` for cache-stable prompts or durable workflows. The tools remain available, and the model can fetch memory only when it needs it:
```python
from pydantic_ai_harness.memory import FileStore, Memory
memory = Memory(FileStore('.agent-memory'), inject_memory=False)
```
With `injection_errors='ignore'` (the default), a store failure skips automatic injection and emits content-safe telemetry. Spans record the backend type and a hash of the resolved scope; successful injection records counts, and failures record the exception type. They do not record memory content. Set `injection_errors='raise'` when a run must fail rather than proceed without injected memory. Namespace and store resolver failures always propagate. Tool failures are still returned as tool errors; this setting controls automatic injection only.
## Persistence and concurrency
The store contract includes optimistic compare-and-swap mutations and idempotency. A write based on a stale revision fails with a conflict instead of overwriting a concurrent edit. Replaying the same run and tool call does not apply its mutation twice; reusing that operation identifier with different arguments raises `MemoryOperationConflictError`. These guarantees belong to the mutation operation, so custom stores must implement them atomically rather than composing separate read and write calls.
Every `MemoryStore.read` call includes a finite `max_chars`, and every `list_paths` call includes a finite `limit`. A store returns the bounded prefix plus `MemoryFile.truncated=True` when more content exists, while its version still represents the complete file. `read_memory` marks that bounded result as truncated. `write_memory` refuses to append or edit an oversized externally supplied file because doing so would derive new content from a partial read; remediate or replace it through the backing store first. A custom `SearchableMemoryStore.search` must likewise honor `max_file_chars` as well as the result limits.
| Store | Persistence and concurrency boundary |
| --- | --- |
| `InMemoryStore()` | Process lifetime; atomic across tasks using that store instance. |
| `FileStore(directory)` | Local filesystem; atomic Markdown replacement plus a hidden SQLite journal provide recovery, cross-process compare-and-swap, and durable idempotency receipts. |
| `SqliteMemoryStore(database=...)` | Durable single-host storage; compare-and-swap and idempotency are enforced in database transactions. |
| `PostgresMemoryStore(pool)` | Durable shared storage; compare-and-swap and idempotency are enforced in database transactions. The caller owns the pool lifecycle. |
```python
from pydantic_ai_harness.memory import FileStore, Memory, SqliteMemoryStore
local_memory = Memory(FileStore('.agent-memory'))
sqlite_memory = Memory(SqliteMemoryStore(database='.agent-memory.db'))
```
`SqliteMemoryStore` can instead use a caller-owned `sqlite3.Connection`. Because operations run off the event loop, create that connection with `check_same_thread=False` and manage its lifecycle in the application. The connection must be dedicated to the store and idle at the start of every operation; a call fails rather than commit or roll back an active caller transaction.
`FileStore` keeps the journal at `.memory-store.sqlite3` inside its root. Keep it with the Markdown files when copying or backing up the store. Editing a Markdown file outside the capability changes its content version and can produce a conflict with a prepared operation; the journal recovers operations interrupted between transaction preparation and filesystem replacement.
`PostgresMemoryStore` accepts the driver-neutral `PostgresPool` protocol, so the harness does not require a particular PostgreSQL driver. Install and manage the driver in your application (for example, `uv add asyncpg`):
```python
import asyncpg
from pydantic_ai_harness.memory import Memory, PostgresMemoryStore
async def build_memory() -> tuple[Memory[None], asyncpg.Pool]:
pool = await asyncpg.create_pool('postgres://localhost/app')
memory = Memory(PostgresMemoryStore(pool))
return memory, pool
```
Call `build_memory` during application startup and close the returned pool during shutdown. The store does not manage it.
## Namespaces
Use a namespace resolver when one `Agent` serves multiple users. It runs once per run from your typed dependencies, and its result is hidden from the model-facing tool schema.
```python
from dataclasses import dataclass
from pydantic_ai import Agent
from pydantic_ai_harness.memory import FileStore, Memory
@dataclass
class AppDeps:
user_id: str
agent = Agent(
'anthropic:claude-sonnet-4-6',
deps_type=AppDeps,
capabilities=[
Memory(
FileStore('/var/lib/myapp/memory'),
namespace=lambda ctx: ctx.deps.user_id,
)
],
defer_model_check=True,
)
```
Namespace isolation controls which records the capability addresses. It is not an authorization system for a custom or shared backing store. Validate the identity in application dependencies, restrict backend credentials, and ensure custom stores cannot escape the resolved namespace.
## Search
`search_memory` performs literal text search and always applies three bounds:
- `max_search_results` limits returned matches, default 10.
- `max_search_result_chars` limits the combined scope-relative filename and snippet text, default 4,000 characters.
- `max_search_files` limits how many files a fallback scan may inspect, default 1,000.
The bundled stores implement `SearchableMemoryStore`. For a custom store that implements only `MemoryStore`, `search_memory` requests at most `max_search_files + 1` paths, scans at most `max_search_files`, and performs bounded reads. Lexical scoring uses only each scope-relative filename and its bounded content; tenant namespaces and agent names never affect relevance. Implement the optional search protocol for an indexed or semantic backend while preserving the same tenant boundary and result limits. Semantic ranking is not built in.
Before backend dispatch, queries are limited to 1,000 characters and 32 unique whitespace-separated terms. Repeated case-insensitive terms are collapsed so they cannot inflate scoring or scan work.
## Configuration
```python
from pydantic_ai_harness.memory import FileStore, Memory
Memory(
FileStore('.agent-memory'),
store_resolver=None, # optional per-run store resolver
agent_name='main', # agent segment inside the namespace
namespace='', # string or per-run resolver
inject_memory=True, # False keeps prompts cache-stable
max_tokens=2_000, # finite approximate total injection budget
max_lines=200, # additional main-notebook line limit
max_memory_size=65_536, # per-file read, search, and write boundary
max_search_results=10,
max_search_result_chars=4_000,
max_search_files=1_000,
injection_errors='ignore', # or 'raise'
guidance=None, # None uses the default notebook guidance
)
```
## Agent specs
Register `Memory` as a custom capability type when constructing an agent from a Python spec:
```python
from pydantic_ai import Agent
from pydantic_ai_harness.memory import Memory
agent = Agent.from_spec(
{
'model': 'anthropic:claude-sonnet-4-6',
'capabilities': [
{'Memory': {'backend': 'file', 'directory': '.agent-memory'}},
],
},
custom_capability_types=[Memory],
defer_model_check=True,
)
```
The serializable backends are `memory`, `file`, and `sqlite`. A namespace callable and a live PostgreSQL pool must be configured in Python.
## Durable execution compatibility
| Execution mode | Support |
| --- | --- |
| Normal `Agent.run` calls | Supported with automatic injection or on-demand tools. |
| Temporal and Prefect | Use `inject_memory=False` with a statically configured store and on-demand tools. Automatic injection performs backend I/O in a model-request hook and is not workflow-safe. |
| DBOS | Normal execution works, but ordinary `FunctionToolset` calls are not DBOS-durable. Wrap memory operations in application-provided DBOS steps when durability is required. |
The memory backend and the workflow state backend are independent. Durable execution does not make an in-memory notebook persistent.
## Security and provenance
Memory is model-written, untrusted content that can re-enter future prompts. Keeping it in a delimited user-role part lowers its authority relative to model instructions, but this is not a hard prompt-injection boundary. Use `inject_memory=False` when less-trusted actors can write to the store, and expose memory only through application-controlled retrieval when stronger isolation is required. Do not store secrets unless the backend, retention policy, and access controls are appropriate. Sanitize content before rendering it into another trust domain.
Memory records do not carry source citations or verified provenance. If an application needs auditable facts, store provenance in the note itself or implement a custom store and schema. Optimistic concurrency prevents lost updates; it does not establish that a remembered claim is true.
## API reference
- [`pydantic_ai_harness.memory` source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/memory/)
- [Pydantic AI capabilities](https://pydantic.dev/docs/ai/core-concepts/capabilities/)
- [Pydantic AI hooks](https://pydantic.dev/docs/ai/core-concepts/hooks/)
The public module exports `Memory`, `MemoryToolset`, the bundled stores, the store protocols, mutation and search result models, and conflict exceptions. Import them from `pydantic_ai_harness.memory`.
pydantic_ai_harness/memory/__init__.py
+49
View File
@@ -0,0 +1,49 @@
"""Memory capability: a persistent, injected notebook plus on-demand memory files."""
from pydantic_ai_harness.memory._capability import Memory
from pydantic_ai_harness.memory._postgres import PostgresConnection, PostgresMemoryStore, PostgresPool
from pydantic_ai_harness.memory._store import (
FileStore,
InMemoryStore,
MemoryConflictError,
MemoryFile,
MemoryMutation,
MemoryOperation,
MemoryOperationConflictError,
MemorySearchMatch,
MemorySearchResult,
MemoryStore,
SearchableMemoryStore,
SqliteMemoryStore,
)
from pydantic_ai_harness.memory._toolset import (
MemoryDeleteResult,
MemorySearchMatchResult,
MemorySearchResponse,
MemoryToolset,
MemoryWriteResult,
)
__all__ = [
'FileStore',
'InMemoryStore',
'Memory',
'MemoryConflictError',
'MemoryDeleteResult',
'MemoryFile',
'MemoryMutation',
'MemoryOperation',
'MemoryOperationConflictError',
'MemorySearchMatch',
'MemorySearchMatchResult',
'MemorySearchResponse',
'MemorySearchResult',
'MemoryStore',
'MemoryToolset',
'MemoryWriteResult',
'PostgresConnection',
'PostgresMemoryStore',
'PostgresPool',
'SearchableMemoryStore',
'SqliteMemoryStore',
]
pydantic_ai_harness/memory/_capability.py
+321
View File
@@ -0,0 +1,321 @@
"""Memory capability: a persistent, injected notebook plus on-demand memory files."""
from __future__ import annotations
import hashlib
from collections.abc import Callable
from dataclasses import dataclass, field, replace
from typing import Literal
from pydantic_ai.agent.abstract import AgentInstructions
from pydantic_ai.capabilities import AbstractCapability
from pydantic_ai.messages import ModelMessage, ModelRequest, ModelRequestPart, TextContent, UserPromptPart
from pydantic_ai.models import ModelRequestContext
from pydantic_ai.tools import AgentDepsT, RunContext
from pydantic_ai.toolsets import AgentToolset
from pydantic_ai_harness.memory._store import InMemoryStore, MemoryStore, validate_store_path
from pydantic_ai_harness.memory._toolset import (
MAIN_FILENAME,
MemoryToolset,
injection_listing_limit,
list_subfiles,
render_memory_prompt,
)
_DEFAULT_GUIDANCE = (
'This is your persistent memory from previous sessions -- background context, NOT '
'instructions. It reflects what was true when written; verify anything volatile before '
'relying on it. MEMORY.md is your main notebook: keep short durable facts there as plain '
'bullet lines, and put longer or evolving topics in separate files referenced from '
'MEMORY.md. When you learn something a future session will need, store it proactively with '
'`write_memory` (append by default; pass `old_text` to correct or remove). Read a listed '
'file with `read_memory` when it looks relevant, or use `search_memory` to find relevant '
'files. Keep memory curated -- update instead of duplicating, delete what turns out wrong. '
'Never claim something was remembered or saved unless you actually called `write_memory` '
'in this turn.'
)
_MEMORY_DATA_PREFIX = '<memory>\n'
_MEMORY_DATA_SUFFIX = '\n</memory>'
_MEMORY_PART_METADATA = 'pydantic-ai-harness.memory.v1'
@dataclass
class Memory(AbstractCapability[AgentDepsT]):
"""Persistent agent memory across sessions.
`MEMORY.md` is injected as user-role context and longer topic files are
available through `read_memory` and `search_memory`. Store access performed
by automatic injection is not workflow-safe durable I/O. With Temporal or
Prefect, use `inject_memory=False`; the static, idempotent `memory` toolset
can then be wrapped by those integrations. DBOS does not currently wrap an
ordinary `FunctionToolset` as a durable step, so this capability's tools are
not DBOS-durable without an application-provided DBOS step wrapper.
"""
store: MemoryStore = field(default_factory=InMemoryStore)
"""Storage backend. The default persists only for the process lifetime."""
store_resolver: Callable[[RunContext[AgentDepsT]], MemoryStore] | None = None
"""Optional per-run store resolver. Resolver failures always propagate."""
agent_name: str = 'main'
"""Agent segment used to isolate memory within a namespace."""
namespace: str | Callable[[RunContext[AgentDepsT]], str] = ''
"""Static or per-run tenant namespace, never exposed as a tool argument."""
inject_memory: bool = True
"""Inject stored memory when true; otherwise inject static tool guidance only."""
max_tokens: int = 2_000
"""Approximate total token ceiling for the complete injected memory section."""
max_lines: int = 200
"""Maximum number of `MEMORY.md` content lines considered for injection."""
max_memory_size: int = 65_536
"""Per-file character boundary for backend reads, search, and writes."""
max_search_results: int = 10
"""Maximum matches returned by one search."""
max_search_result_chars: int = 4_000
"""Maximum combined snippet characters returned by one search."""
max_search_files: int = 1_000
"""Maximum files scanned by one search."""
guidance: str | None = None
"""Override injected usage guidance; `''` disables guidance."""
injection_errors: Literal['ignore', 'raise'] = 'ignore'
"""Whether store failures during automatic injection are ignored or raised."""
_resolved_scope: tuple[MemoryStore, str] | None = field(default=None, init=False, repr=False, compare=False)
def __post_init__(self) -> None:
_validate_positive('max_tokens', self.max_tokens)
_validate_non_negative('max_lines', self.max_lines)
_validate_positive('max_memory_size', self.max_memory_size)
_validate_positive('max_search_results', self.max_search_results)
_validate_positive('max_search_result_chars', self.max_search_result_chars)
_validate_positive('max_search_files', self.max_search_files)
if self.injection_errors not in ('ignore', 'raise'):
raise ValueError("injection_errors must be 'ignore' or 'raise'")
async def for_run(self, ctx: RunContext[AgentDepsT]) -> Memory[AgentDepsT]:
"""Return a clone with scope resolution isolated to this run."""
clone = replace(self)
clone._resolved_scope = None
clone._resolved_scope = clone._resolve_scope(ctx)
return clone
def resolve_scope(self, ctx: RunContext[AgentDepsT]) -> tuple[MemoryStore, str]:
"""Return the cached run scope, or resolve one for direct toolset use."""
if self._resolved_scope is not None:
return self._resolved_scope
return self._resolve_scope(ctx)
def _resolve_scope(self, ctx: RunContext[AgentDepsT]) -> tuple[MemoryStore, str]:
store = self.store_resolver(ctx) if self.store_resolver is not None else self.store
namespace = self.namespace(ctx) if callable(self.namespace) else self.namespace
scope = f'{namespace}/{self.agent_name}' if namespace else self.agent_name
validate_store_path(scope)
return store, scope
def get_toolset(self) -> AgentToolset[AgentDepsT] | None:
"""Provide the stable `memory` toolset."""
return MemoryToolset(self)
def get_instructions(self) -> AgentInstructions[AgentDepsT] | None:
"""Provide trusted static guidance about using memory.
Stored memory is added separately as user-role context by
`before_model_request` so model-written content is not placed in the
instruction channel.
"""
return self._render_guidance()
def _render_guidance(self) -> str | None:
guidance = _DEFAULT_GUIDANCE if self.guidance is None else self.guidance
if not guidance:
return None
return render_memory_prompt(
'',
[],
agent_name=self.agent_name,
guidance=guidance,
max_lines=self.max_lines,
max_tokens=self.max_tokens,
)
async def before_model_request(
self,
ctx: RunContext[AgentDepsT],
request_context: ModelRequestContext,
) -> ModelRequestContext:
"""Add a bounded memory snapshot to only the current user request."""
self._remove_previous_injection(request_context.messages)
if not self.inject_memory:
return request_context
store, scope = self.resolve_scope(ctx)
scope_hash = hashlib.sha256(scope.encode()).hexdigest()[:16]
with ctx.tracer.start_as_current_span(
'memory.inject', record_exception=False, set_status_on_exception=False
) as span:
if span.is_recording():
span.set_attributes(
{
'memory.backend': type(store).__name__,
'memory.scope_hash': scope_hash,
}
)
try:
main_path = f'{scope}/{MAIN_FILENAME}'
main = await store.read(main_path, max_chars=self.max_memory_size)
subfiles, files_truncated = await list_subfiles(
store,
scope,
limit=injection_listing_limit(self.max_tokens),
)
except Exception as exc:
if span.is_recording():
span.set_attributes(
{
'memory.outcome': 'error',
'memory.exception_type': type(exc).__name__,
}
)
if self.injection_errors == 'raise':
raise
return request_context
main_content = '' if main is None else main.content
rendered = ''
guidance = self._render_guidance()
content_budget = (
self.max_tokens * 4 - len(guidance or '') - len(_MEMORY_DATA_PREFIX) - len(_MEMORY_DATA_SUFFIX)
)
if content_budget > 0 and (main_content or subfiles or files_truncated):
rendered = render_memory_prompt(
main_content,
subfiles,
agent_name=self.agent_name,
guidance='',
max_lines=self.max_lines,
max_tokens=max(1, content_budget // 4),
main_truncated=main is not None and main.truncated,
files_truncated=files_truncated,
)[:content_budget]
rendered = f'{_MEMORY_DATA_PREFIX}{rendered}{_MEMORY_DATA_SUFFIX}'
part = UserPromptPart([TextContent(rendered, metadata=_MEMORY_PART_METADATA)])
latest = request_context.messages[-1]
if not isinstance(latest, ModelRequest): # pragma: no cover - guaranteed by the agent graph
raise RuntimeError('model request history must end with a ModelRequest')
request_context.messages[-1] = replace(latest, parts=[*latest.parts, part])
if span.is_recording():
span.set_attributes(
{
'memory.outcome': 'ok',
'memory.main_chars': len(main_content),
'memory.files': len(subfiles),
'memory.files_truncated': files_truncated,
'memory.main_truncated': main is not None and main.truncated,
'memory.injected_chars': len(rendered),
}
)
return request_context
def _remove_previous_injection(self, messages: list[ModelMessage]) -> None:
for index, message in enumerate(messages):
if not isinstance(message, ModelRequest):
continue
parts: list[ModelRequestPart] = []
changed = False
for part in message.parts:
if not isinstance(part, UserPromptPart) or isinstance(part.content, str):
parts.append(part)
continue
content = [
item
for item in part.content
if not (isinstance(item, TextContent) and item.metadata == _MEMORY_PART_METADATA)
]
if len(content) == len(part.content):
parts.append(part)
else:
changed = True
if content:
parts.append(replace(part, content=content))
if changed:
messages[index] = replace(message, parts=parts)
@classmethod
def from_spec(
cls,
*,
backend: Literal['memory', 'file', 'sqlite'] = 'memory',
directory: str = '.agent-memory',
database: str = '.agent-memory.db',
agent_name: str = 'main',
namespace: str = '',
inject_memory: bool = True,
max_tokens: int = 2_000,
max_lines: int = 200,
max_memory_size: int = 65_536,
max_search_results: int = 10,
max_search_result_chars: int = 4_000,
max_search_files: int = 1_000,
guidance: str | None = None,
injection_errors: Literal['ignore', 'raise'] = 'ignore',
) -> Memory[AgentDepsT]:
"""Construct a memory capability from serializable options."""
if backend != 'file' and directory != '.agent-memory':
raise ValueError('directory is only valid with backend="file"')
if backend != 'sqlite' and database != '.agent-memory.db':
raise ValueError('database is only valid with backend="sqlite"')
if backend == 'memory':
store: MemoryStore = InMemoryStore()
elif backend == 'file':
from pydantic_ai_harness.memory._store import FileStore
store = FileStore(directory)
elif backend == 'sqlite':
from pydantic_ai_harness.memory._store import SqliteMemoryStore
store = SqliteMemoryStore(database=database)
else:
raise ValueError(f'unknown backend {backend!r}; expected `memory`, `file`, or `sqlite`')
return cls(
store=store,
agent_name=agent_name,
namespace=namespace,
inject_memory=inject_memory,
max_tokens=max_tokens,
max_lines=max_lines,
max_memory_size=max_memory_size,
max_search_results=max_search_results,
max_search_result_chars=max_search_result_chars,
max_search_files=max_search_files,
guidance=guidance,
injection_errors=injection_errors,
)
@classmethod
def get_serialization_name(cls) -> str | None:
"""Return the name used by custom capability specs."""
return 'Memory'
def _validate_positive(name: str, value: int) -> None:
if value <= 0:
raise ValueError(f'{name} must be a positive integer')
def _validate_non_negative(name: str, value: int) -> None:
if value < 0:
raise ValueError(f'{name} must be a non-negative integer')
pydantic_ai_harness/memory/_postgres.py
+297
View File
@@ -0,0 +1,297 @@
"""PostgreSQL memory store over an asyncpg-compatible caller-owned pool."""
from __future__ import annotations
import re
from collections.abc import Sequence
from contextlib import AbstractAsyncContextManager
from typing import Protocol, runtime_checkable
import anyio
from pydantic_ai_harness.memory._store import (
MemoryConflictError,
MemoryFile,
MemoryMutation,
MemoryOperation,
MemoryOperationConflictError,
MemorySearchResult,
lexical_search,
validate_store_path,
validate_store_prefix,
)
_TABLE_RE = re.compile(r'[A-Za-z_][A-Za-z0-9_]{0,51}')
@runtime_checkable
class PostgresConnection(Protocol):
"""The acquired asyncpg-compatible connection surface used by the store."""
def transaction(self) -> AbstractAsyncContextManager[object]:
"""Return an async transaction context manager."""
... # pragma: no cover
async def execute(self, query: str, *args: object) -> object:
"""Execute a statement."""
... # pragma: no cover
async def fetchval(self, query: str, *args: object) -> object:
"""Return the first column of the first row, or `None`."""
... # pragma: no cover
async def fetchrow(self, query: str, *args: object) -> Sequence[object] | None:
"""Return the first row, or `None`."""
... # pragma: no cover
async def fetch(self, query: str, *args: object) -> Sequence[Sequence[object]]:
"""Return all rows."""
... # pragma: no cover
@runtime_checkable
class PostgresPool(Protocol):
"""The asyncpg-compatible pool surface used by `PostgresMemoryStore`."""
def acquire(self) -> AbstractAsyncContextManager[PostgresConnection]:
"""Acquire one connection for an operation or transaction."""
... # pragma: no cover
class PostgresMemoryStore:
"""Transactional PostgreSQL memory store with CAS and operation receipts."""
def __init__(self, pool: PostgresPool, *, table: str = 'agent_memory') -> None:
if not _TABLE_RE.fullmatch(table):
raise ValueError(f'invalid table name: {table!r}')
self._pool = pool
self._table = table
self._operations_table = f'{table}_operations'
self._version_sequence = f'{table}_versions'
self._metadata_table = f'{table}_metadata'
self._schema_ready = False
self._schema_lock = anyio.Lock()
async def _ensure_schema(self) -> None:
if self._schema_ready:
return
async with self._schema_lock:
if self._schema_ready:
return
async with self._pool.acquire() as connection, connection.transaction():
await connection.fetchval('SELECT pg_advisory_xact_lock(hashtext($1))', self._metadata_table)
await connection.execute(
f'CREATE TABLE IF NOT EXISTS {self._table} ('
'path TEXT PRIMARY KEY, content TEXT NOT NULL, '
'version BIGINT NOT NULL DEFAULT 1, last_operation_id TEXT)'
)
await connection.execute(
f'ALTER TABLE {self._table} ADD COLUMN IF NOT EXISTS version BIGINT NOT NULL DEFAULT 1'
)
await connection.execute(f'ALTER TABLE {self._table} ADD COLUMN IF NOT EXISTS last_operation_id TEXT')
await connection.execute(
f'CREATE TABLE IF NOT EXISTS {self._operations_table} ('
'id TEXT PRIMARY KEY, fingerprint TEXT NOT NULL, version TEXT, '
'existed BOOLEAN NOT NULL, completed BOOLEAN NOT NULL)'
)
await connection.execute(f'CREATE SEQUENCE IF NOT EXISTS {self._version_sequence} MINVALUE 0 START 0')
await connection.execute(
f'CREATE TABLE IF NOT EXISTS {self._metadata_table} ('
'id BOOLEAN PRIMARY KEY DEFAULT TRUE CHECK (id), versions_initialized BOOLEAN NOT NULL)'
)
initialized = await connection.fetchval(
f'INSERT INTO {self._metadata_table} (id, versions_initialized) VALUES (TRUE, TRUE) '
'ON CONFLICT (id) DO NOTHING RETURNING id'
)
if initialized is not None:
await connection.execute(f"UPDATE {self._table} SET version = nextval('{self._version_sequence}')")
self._schema_ready = True
async def _get_operation(self, connection: PostgresConnection, operation: MemoryOperation) -> MemoryMutation | None:
row = await connection.fetchrow(
f'SELECT fingerprint, version, existed, completed FROM {self._operations_table} WHERE id = $1',
operation.id,
)
if row is None:
return None
if str(row[0]) != operation.fingerprint:
raise MemoryOperationConflictError(f'operation id {operation.id!r} was reused with different arguments')
if not bool(row[3]): # pragma: no cover - uncommitted reservations are invisible
return None
return MemoryMutation(
version=str(row[1]) if row[1] is not None else None,
replayed=True,
existed=bool(row[2]),
)
async def _reserve_operation(
self, connection: PostgresConnection, operation: MemoryOperation
) -> MemoryMutation | None:
inserted = await connection.fetchval(
f'INSERT INTO {self._operations_table} (id, fingerprint, version, existed, completed) '
'VALUES ($1, $2, NULL, FALSE, FALSE) ON CONFLICT (id) DO NOTHING RETURNING id',
operation.id,
operation.fingerprint,
)
if inserted is not None:
return None
receipt = await self._get_operation(connection, operation)
if receipt is None: # pragma: no cover - the conflicting transaction completes before this statement resumes
raise RuntimeError(f'operation {operation.id!r} did not produce a committed receipt')
return receipt
async def _complete_operation(
self, connection: PostgresConnection, operation: MemoryOperation, mutation: MemoryMutation
) -> None:
await connection.execute(
f'UPDATE {self._operations_table} SET version = $2, existed = $3, completed = TRUE WHERE id = $1',
operation.id,
mutation.version,
mutation.existed,
)
async def read(self, path: str, *, max_chars: int) -> MemoryFile | None:
validate_store_path(path)
if max_chars <= 0:
raise ValueError('max_chars must be positive')
await self._ensure_schema()
async with self._pool.acquire() as connection:
row = await connection.fetchrow(
f'SELECT left(content, $2), version, last_operation_id, length(content) '
f'FROM {self._table} WHERE path = $1',
path,
max_chars,
)
if row is None:
return None
return MemoryFile(
content=str(row[0]),
version=str(row[1]),
operation_id=str(row[2]) if row[2] is not None else None,
truncated=int(str(row[3])) > max_chars,
)
async def get_operation(self, operation: MemoryOperation) -> MemoryMutation | None:
await self._ensure_schema()
async with self._pool.acquire() as connection:
return await self._get_operation(connection, operation)
async def write(
self,
path: str,
content: str,
*,
expected_version: str | None,
operation: MemoryOperation | None = None,
) -> MemoryMutation:
validate_store_path(path)
await self._ensure_schema()
async with self._pool.acquire() as connection, connection.transaction():
if operation is not None and (receipt := await self._reserve_operation(connection, operation)) is not None:
return receipt
if expected_version is None:
row = await connection.fetchrow(
f'INSERT INTO {self._table} (path, content, version, last_operation_id) '
f"VALUES ($1, $2, nextval('{self._version_sequence}'), $3) "
'ON CONFLICT (path) DO NOTHING RETURNING version',
path,
content,
operation.id if operation else None,
)
existed = False
else:
row = await connection.fetchrow(
f"UPDATE {self._table} SET content = $2, version = nextval('{self._version_sequence}'), "
'last_operation_id = $3 '
'WHERE path = $1 AND version::TEXT = $4 RETURNING version',
path,
content,
operation.id if operation else None,
expected_version,
)
existed = True
if row is None:
raise MemoryConflictError(f'memory path {path!r} changed before it could be written')
mutation = MemoryMutation(version=str(row[0]), replayed=False, existed=existed)
if operation is not None:
await self._complete_operation(connection, operation, mutation)
return mutation
async def delete(
self,
path: str,
*,
expected_version: str | None,
operation: MemoryOperation | None = None,
) -> MemoryMutation:
validate_store_path(path)
await self._ensure_schema()
async with self._pool.acquire() as connection, connection.transaction():
if operation is not None and (receipt := await self._reserve_operation(connection, operation)) is not None:
return receipt
await connection.fetchval(f"SELECT nextval('{self._version_sequence}')")
if expected_version is None:
exists = await connection.fetchval(f'SELECT EXISTS(SELECT 1 FROM {self._table} WHERE path = $1)', path)
if bool(exists):
raise MemoryConflictError(f'memory path {path!r} changed before it could be deleted')
existed = False
else:
row = await connection.fetchrow(
f'DELETE FROM {self._table} WHERE path = $1 AND version::TEXT = $2 RETURNING version',
path,
expected_version,
)
if row is None:
raise MemoryConflictError(f'memory path {path!r} changed before it could be deleted')
existed = True
mutation = MemoryMutation(version=None, replayed=False, existed=existed)
if operation is not None:
await self._complete_operation(connection, operation, mutation)
return mutation
async def list_paths(self, prefix: str = '', *, limit: int) -> list[str]:
validate_store_prefix(prefix)
if limit <= 0:
raise ValueError('limit must be positive')
await self._ensure_schema()
async with self._pool.acquire() as connection:
rows = await connection.fetch(
f'SELECT path FROM {self._table} WHERE starts_with(path, $1) ORDER BY path LIMIT $2', prefix, limit
)
return [str(row[0]) for row in rows]
async def search(
self,
prefix: str,
query: str,
*,
limit: int,
max_files: int,
max_chars: int,
max_file_chars: int,
) -> MemorySearchResult:
validate_store_prefix(prefix)
if not query.split() or limit <= 0 or max_files <= 0 or max_chars <= 0 or max_file_chars <= 0:
return MemorySearchResult(matches=[], scanned=0, truncated=False)
await self._ensure_schema()
async with self._pool.acquire() as connection:
rows = await connection.fetch(
f'SELECT path, left(content, $2), length(content) FROM {self._table} '
'WHERE starts_with(path, $1) ORDER BY path LIMIT $3',
prefix,
max_file_chars,
max_files + 1,
)
result = lexical_search(
[(str(row[0]), str(row[1])) for row in rows],
query,
limit=limit,
max_files=max_files,
max_chars=max_chars,
score_prefix=prefix,
)
return MemorySearchResult(
matches=result.matches,
scanned=result.scanned,
truncated=result.truncated or any(int(str(row[2])) > max_file_chars for row in rows),
)
pydantic_ai_harness/memory/_store.py
+1117
View File
File diff suppressed because it is too large Load Diff
pydantic_ai_harness/memory/_toolset.py
+662
View File
@@ -0,0 +1,662 @@
"""Memory tools over a scoped, versioned `MemoryStore`."""
from __future__ import annotations
import hashlib
import json
import re
from typing import TYPE_CHECKING, Literal
from opentelemetry.trace import Span
from pydantic_ai import ModelRetry
from pydantic_ai.tools import AgentDepsT, RunContext
from pydantic_ai.toolsets import FunctionToolset
from typing_extensions import TypedDict
from pydantic_ai_harness.memory._store import (
MemoryConflictError,
MemoryMutation,
MemoryOperation,
MemorySearchMatch,
MemorySearchResult,
MemoryStore,
SearchableMemoryStore,
)
if TYPE_CHECKING:
from pydantic_ai_harness.memory._capability import Memory
MAIN_FILENAME = 'MEMORY.md'
"""The main notebook file injected as bounded user-role context."""
_FILENAME_RE = re.compile(r'[A-Za-z0-9][A-Za-z0-9._-]{0,79}')
_CHARS_PER_TOKEN = 4
_MAX_CAS_ATTEMPTS = 16
_MAX_SEARCH_QUERY_CHARS = 1_000
_MAX_SEARCH_TERMS = 32
_MIN_FILENAME_LIST_CHARS = 6
_READ_TRUNCATION_MARKER = (
'\n\n[Truncated: this file exceeds `max_memory_size`; edit it externally before using `write_memory`.]'
)
class MemoryWriteResult(TypedDict):
"""Content-free result from `write_memory`."""
file: str
version: str
replayed: bool
status: Literal['created', 'appended', 'updated']
class MemoryDeleteResult(TypedDict):
"""Content-free result from `delete_memory`."""
file: str
version: str | None
replayed: bool
status: Literal['deleted', 'not_found']
class MemorySearchMatchResult(TypedDict):
"""One model-facing, scope-relative search match."""
file: str
snippet: str
score: float
class MemorySearchResponse(TypedDict):
"""Bounded result from `search_memory`."""
matches: list[MemorySearchMatchResult]
scanned: int
truncated: bool
def normalize_filename(file: str) -> str:
"""Validate a model-supplied memory filename and normalize it to `<name>.md`."""
name = file.strip()
if name and not name.endswith('.md'):
name = f'{name}.md'
if not _FILENAME_RE.fullmatch(name) or '..' in name:
raise ModelRetry(
f'{file!r} is not a valid memory filename -- use a short name like "postgres-migration.md" '
'(letters, digits, dots, dashes; no slashes).'
)
return name
def _normalize_search_query(query: str) -> str:
if len(query) > _MAX_SEARCH_QUERY_CHARS:
raise ModelRetry(f'Search queries must be at most {_MAX_SEARCH_QUERY_CHARS} characters.')
normalized = query.strip()
if not normalized:
raise ModelRetry('Pass a non-empty search query.')
terms: list[str] = []
seen: set[str] = set()
for term in normalized.split():
key = term.lower()
if key in seen:
continue
if len(terms) == _MAX_SEARCH_TERMS:
raise ModelRetry(f'Search queries must contain at most {_MAX_SEARCH_TERMS} unique terms.')
seen.add(key)
terms.append(term)
return ' '.join(terms)
async def list_subfiles(store: MemoryStore, scope: str, *, limit: int) -> tuple[list[str], bool]:
"""Return a bounded filename list and whether additional paths may exist."""
prefix = f'{scope}/'
names: list[str] = []
request_limit = limit + 2 # one main notebook plus one truncation sentinel
returned_paths = await store.list_paths(prefix, limit=request_limit)
paths = returned_paths[:request_limit]
for path in paths:
if not path.startswith(prefix):
raise RuntimeError('memory backend returned a path outside the requested scope')
name = path.removeprefix(prefix)
if '/' not in name and name != MAIN_FILENAME and name.endswith('.md'):
names.append(name)
names.sort()
return names[:limit], len(names) > limit or len(returned_paths) >= request_limit
def injection_listing_limit(max_tokens: int) -> int:
"""Derive a finite backend listing bound from the complete prompt budget."""
return max(1, max_tokens * _CHARS_PER_TOKEN // _MIN_FILENAME_LIST_CHARS)
def render_memory_prompt(
main_content: str,
subfiles: list[str],
*,
agent_name: str,
guidance: str,
max_lines: int,
max_tokens: int,
main_truncated: bool = False,
files_truncated: bool = False,
) -> str:
"""Render a complete memory section within the strict approximate token budget."""
budget = max_tokens * _CHARS_PER_TOKEN
rendered_guidance = guidance
source_lines = main_content.rstrip().splitlines()
kept_lines = source_lines[-max_lines:] if max_lines else []
dropped_lines = len(source_lines) - len(kept_lines)
max_filename_candidates = min(len(subfiles), budget // 4)
shown_files = subfiles[:max_filename_candidates]
omitted_files = len(subfiles) - len(shown_files)
rendered = _render_sections(
agent_name,
rendered_guidance,
kept_lines,
dropped_lines,
shown_files,
omitted_files,
main_truncated,
files_truncated,
)
while len(rendered) > budget and shown_files:
shown_files.pop()
omitted_files += 1
rendered = _render_sections(
agent_name,
rendered_guidance,
kept_lines,
dropped_lines,
shown_files,
omitted_files,
main_truncated,
files_truncated,
)
while len(rendered) > budget and kept_lines:
kept_lines.pop(0)
dropped_lines += 1
rendered = _render_sections(
agent_name,
rendered_guidance,
kept_lines,
dropped_lines,
shown_files,
omitted_files,
main_truncated,
files_truncated,
)
if len(rendered) > budget and dropped_lines:
dropped_lines = 0
rendered = _render_sections(
agent_name,
rendered_guidance,
kept_lines,
dropped_lines,
shown_files,
omitted_files,
main_truncated,
files_truncated,
)
if len(rendered) > budget and rendered_guidance:
rendered_guidance = rendered_guidance[: max(0, len(rendered_guidance) - (len(rendered) - budget))]
rendered = _render_sections(
agent_name,
rendered_guidance,
kept_lines,
dropped_lines,
shown_files,
omitted_files,
main_truncated,
files_truncated,
)
if len(rendered) <= budget:
return rendered
if omitted_files or files_truncated:
marker = '[Memory files omitted; use search_memory]'
return marker[:budget]
return rendered[:budget]
def _render_sections(
agent_name: str,
guidance: str,
main_lines: list[str],
dropped_lines: int,
subfiles: list[str],
omitted_files: int,
main_truncated: bool,
files_truncated: bool,
) -> str:
sections = [f'## Agent Memory ({agent_name})']
if guidance:
sections.append(guidance)
if main_lines or dropped_lines or main_truncated:
lines = list(main_lines)
if dropped_lines:
lines.insert(
0,
f'... [{dropped_lines} earlier lines; use read_memory("{MAIN_FILENAME}") for the full notebook] ...',
)
if main_truncated:
lines.append('... [notebook exceeds `max_memory_size`; bounded prefix shown] ...')
sections.append(f'### {MAIN_FILENAME}\n\n' + '\n'.join(lines))
if subfiles or omitted_files or files_truncated:
listing = [f'- {name}' for name in subfiles]
if files_truncated:
listing.append('- ... [more files omitted; use search_memory to find relevant memory]')
elif omitted_files:
listing.append(f'- ... [{omitted_files} more files; use search_memory to find relevant memory]')
sections.append('### Other memory files\n\n' + '\n'.join(listing))
return '\n\n'.join(sections)
def _apply_write(existing: str | None, content: str, old_text: str | None, name: str) -> tuple[str, str]:
if old_text is None:
if existing is not None and existing.strip():
return f'{existing.rstrip()}\n{content.rstrip()}\n', 'appended'
return f'{content.rstrip()}\n', 'created'
if existing is None:
raise ModelRetry(f'There is no memory file named {name!r} to edit -- omit `old_text` to create it.')
occurrences = existing.count(old_text) if old_text else 0
if occurrences == 0:
raise ModelRetry(
f'`old_text` was not found in {name!r} -- call `read_memory("{name}")` to see its current content.'
)
if occurrences > 1:
raise ModelRetry(
f'`old_text` appears {occurrences} times in {name!r}; it must match exactly once. '
'Add surrounding context to make it unique.'
)
return existing.replace(old_text, content, 1), 'updated'
class MemoryToolset(FunctionToolset[AgentDepsT]):
"""Scoped read/write/delete/search tools with CAS and durable idempotency.
The stable `memory` ID lets Temporal and Prefect wrap this static toolset.
DBOS does not currently turn an ordinary `FunctionToolset` into a durable
step, so applications requiring DBOS durability must provide that wrapper.
"""
def __init__(self, capability: Memory[AgentDepsT]) -> None:
super().__init__(id='memory')
self._capability = capability
self.add_function(self.write_memory, name='write_memory')
self.add_function(self.read_memory, name='read_memory')
self.add_function(self.delete_memory, name='delete_memory')
self.add_function(self.search_memory, name='search_memory')
async def write_memory(
self,
ctx: RunContext[AgentDepsT],
content: str,
file: str = MAIN_FILENAME,
old_text: str | None = None,
) -> MemoryWriteResult:
"""Write persistent memory by appending or uniquely replacing text.
Omit `old_text` to append, creating the file when necessary. Pass
`old_text` to replace exactly one matching passage; use an empty
`content` to remove that passage. Keep short durable facts in
`MEMORY.md`, and longer or evolving topics in separate files. Update
stale entries rather than adding contradictory duplicates.
Args:
ctx: Framework-provided run context.
content: Text to append, or replacement text for `old_text`.
file: Memory filename; defaults to `MEMORY.md`.
old_text: Exact passage to replace, which must occur once.
"""
capability = self._capability
name = normalize_filename(file)
if old_text is None and not content.strip():
raise ModelRetry('Nothing to write -- pass the text to append, or `old_text` to replace.')
store, scope = capability.resolve_scope(ctx)
path = f'{scope}/{name}'
operation = _operation(ctx, scope, 'write', path, {'content': content, 'file': file, 'old_text': old_text})
scope_hash = _scope_hash(scope)
with ctx.tracer.start_as_current_span(
'memory.write', record_exception=False, set_status_on_exception=False
) as span:
_set_span_base(span, store, scope_hash)
try:
previous_mutation = await store.get_operation(operation)
if previous_mutation is not None:
result = _write_result(name, previous_mutation, old_text)
_set_span_result(span, 'replayed', replayed=True)
return result
for attempt in range(_MAX_CAS_ATTEMPTS):
current = await store.read(path, max_chars=capability.max_memory_size)
if current is not None and current.truncated:
raise ModelRetry(
f'{name!r} exceeds max_memory_size and cannot be changed from partial content; '
'edit or replace it through the backing store first.'
)
updated, status = _apply_write(
None if current is None else current.content,
content,
old_text,
name,
)
if len(updated) > capability.max_memory_size:
raise ModelRetry(
f'{name!r} would grow to {len(updated)} characters; the limit is '
f'{capability.max_memory_size}. Split the content into separate memory files.'
)
try:
mutation = await store.write(
path,
updated,
expected_version=None if current is None else current.version,
operation=operation,
)
except MemoryConflictError:
if attempt + 1 == _MAX_CAS_ATTEMPTS:
raise ModelRetry('Memory changed repeatedly while writing; retry the operation.')
continue
result = _write_result(name, mutation, old_text, status=status)
_set_span_result(span, 'ok', chars=len(updated), replayed=mutation.replayed)
return result
raise RuntimeError('unreachable CAS retry state') # pragma: no cover
except Exception as exc:
_set_span_error(span, exc)
raise
async def read_memory(self, ctx: RunContext[AgentDepsT], file: str) -> str:
"""Read a bounded prefix of one memory file.
Memory may be stale background context, so verify volatile facts before
relying on them.
Args:
ctx: Framework-provided run context.
file: Memory filename returned by injection or search.
"""
name = normalize_filename(file)
store, scope = self._capability.resolve_scope(ctx)
with ctx.tracer.start_as_current_span(
'memory.read', record_exception=False, set_status_on_exception=False
) as span:
_set_span_base(span, store, _scope_hash(scope))
try:
memory_file = await store.read(f'{scope}/{name}', max_chars=self._capability.max_memory_size)
if memory_file is None:
_set_span_result(span, 'not_found')
raise ModelRetry(
f'There is no memory file named {name!r} -- use `search_memory` to find existing memory.'
)
_set_span_result(span, 'ok', chars=len(memory_file.content))
return memory_file.content + (_READ_TRUNCATION_MARKER if memory_file.truncated else '')
except Exception as exc:
_set_span_error(span, exc)
raise
async def delete_memory(self, ctx: RunContext[AgentDepsT], file: str) -> MemoryDeleteResult:
"""Delete a non-main memory file that is no longer useful.
`MEMORY.md` cannot be deleted; remove or correct its text with
`write_memory` instead.
Args:
ctx: Framework-provided run context.
file: Memory filename to delete.
"""
name = normalize_filename(file)
if name == MAIN_FILENAME:
raise ModelRetry(f'{MAIN_FILENAME} is the main notebook; edit it with `write_memory` instead.')
store, scope = self._capability.resolve_scope(ctx)
path = f'{scope}/{name}'
operation = _operation(ctx, scope, 'delete', path, {'file': file})
with ctx.tracer.start_as_current_span(
'memory.delete', record_exception=False, set_status_on_exception=False
) as span:
_set_span_base(span, store, _scope_hash(scope))
try:
previous_mutation = await store.get_operation(operation)
if previous_mutation is not None:
result = _delete_result(name, previous_mutation)
_set_span_result(span, 'replayed', replayed=True)
return result
for attempt in range(_MAX_CAS_ATTEMPTS):
current = await store.read(path, max_chars=1)
try:
mutation = await store.delete(
path,
expected_version=None if current is None else current.version,
operation=operation,
)
except MemoryConflictError:
if attempt + 1 == _MAX_CAS_ATTEMPTS:
raise ModelRetry('Memory changed repeatedly while deleting; retry the operation.')
continue
result = _delete_result(name, mutation)
_set_span_result(span, result['status'], replayed=mutation.replayed)
return result
raise RuntimeError('unreachable CAS retry state') # pragma: no cover
except Exception as exc:
_set_span_error(span, exc)
raise
async def search_memory(self, ctx: RunContext[AgentDepsT], query: str) -> MemorySearchResponse:
"""Search memory files in the current tenant and agent scope.
Results contain bounded snippets; call `read_memory` when a larger
bounded excerpt is relevant.
Args:
ctx: Framework-provided run context.
query: Terms to find in memory filenames and content.
"""
query = _normalize_search_query(query)
capability = self._capability
store, scope = capability.resolve_scope(ctx)
prefix = f'{scope}/'
with ctx.tracer.start_as_current_span(
'memory.search', record_exception=False, set_status_on_exception=False
) as span:
_set_span_base(span, store, _scope_hash(scope))
try:
if isinstance(store, SearchableMemoryStore):
result = await store.search(
prefix,
query,
limit=capability.max_search_results,
max_files=capability.max_search_files,
max_chars=capability.max_search_result_chars,
max_file_chars=capability.max_memory_size,
)
else:
result = await _fallback_search(
store,
prefix,
query,
limit=capability.max_search_results,
max_files=capability.max_search_files,
max_chars=capability.max_search_result_chars,
max_file_chars=capability.max_memory_size,
)
matches, bounded_truncated = _bounded_search_matches(
result.matches,
prefix,
limit=capability.max_search_results,
max_chars=capability.max_search_result_chars,
)
scanned = min(max(result.scanned, 0), capability.max_search_files)
truncated = result.truncated or bounded_truncated or result.scanned > capability.max_search_files
_set_span_result(
span,
'ok',
matches=len(matches),
scanned=scanned,
chars=sum(len(match['snippet']) for match in matches),
truncated=truncated,
)
return {'matches': matches, 'scanned': scanned, 'truncated': truncated}
except Exception as exc:
_set_span_error(span, exc)
raise
def _operation(
ctx: RunContext[AgentDepsT],
scope: str,
kind: Literal['write', 'delete'],
path: str,
model_args: dict[str, str | None],
) -> MemoryOperation:
tool_call_id = ctx.tool_call_id
if not tool_call_id:
raise RuntimeError('memory mutations require a stable tool_call_id')
run_id = ctx.run_id
if not run_id:
raise RuntimeError('memory mutations require a stable run_id')
operation_id = _digest({'scope': scope, 'run_id': run_id, 'tool_call_id': tool_call_id})
fingerprint = _digest({'kind': kind, 'path': path, 'model_args': model_args})
return MemoryOperation(id=operation_id, fingerprint=fingerprint)
def _digest(value: dict[str, object]) -> str:
encoded = json.dumps(value, sort_keys=True, separators=(',', ':'), ensure_ascii=True).encode()
return hashlib.sha256(encoded).hexdigest()
def _write_result(
name: str,
mutation: MemoryMutation,
old_text: str | None,
*,
status: str | None = None,
) -> MemoryWriteResult:
if mutation.version is None:
raise RuntimeError('memory write returned no version') # pragma: no cover
if status is None:
status = 'updated' if old_text is not None else ('appended' if mutation.existed else 'created')
if status == 'created':
typed_status: Literal['created', 'appended', 'updated'] = 'created'
elif status == 'appended':
typed_status = 'appended'
else:
typed_status = 'updated'
return {'file': name, 'version': mutation.version, 'replayed': mutation.replayed, 'status': typed_status}
def _delete_result(name: str, mutation: MemoryMutation) -> MemoryDeleteResult:
status: Literal['deleted', 'not_found'] = 'deleted' if mutation.existed else 'not_found'
return {'file': name, 'version': mutation.version, 'replayed': mutation.replayed, 'status': status}
def _search_match(match: MemorySearchMatch, prefix: str, snippet: str) -> MemorySearchMatchResult:
name = match.path.removeprefix(prefix)
return {'file': name, 'snippet': snippet, 'score': match.score}
async def _fallback_search(
store: MemoryStore,
prefix: str,
query: str,
*,
limit: int,
max_files: int,
max_chars: int,
max_file_chars: int,
) -> MemorySearchResult:
paths = await store.list_paths(prefix, limit=max_files + 1)
terms = query.lower().split()
scored: list[tuple[float, str, str]] = []
scanned = 0
content_truncated = False
for path in paths[:max_files]:
scanned += 1
if not path.startswith(prefix):
raise RuntimeError('memory backend returned a path outside the requested scope')
name = path.removeprefix(prefix)
if '/' in name or not _FILENAME_RE.fullmatch(name) or '..' in name:
continue
memory_file = await store.read(path, max_chars=max_file_chars)
if memory_file is None:
continue
content_truncated = content_truncated or memory_file.truncated
lower_path = name.lower()
lower_content = memory_file.content.lower()
score = float(sum(lower_content.count(term) + 2 * lower_path.count(term) for term in terms))
if score:
scored.append((score, path, memory_file.content))
scored.sort(key=lambda item: (-item[0], item[1]))
matches = [
MemorySearchMatch(path=path, snippet=_search_snippet(content, query, max_chars), score=score)
for score, path, content in scored[:limit]
]
return MemorySearchResult(
matches=matches,
scanned=scanned,
truncated=len(paths) > max_files or len(scored) > len(matches) or content_truncated,
)
def _search_snippet(content: str, query: str, max_chars: int) -> str:
lower = content.lower()
positions = [lower.find(term) for term in query.lower().split()]
found = [position for position in positions if position >= 0]
center = min(found) if found else 0
start = max(0, center - max_chars // 3)
end = min(len(content), start + max_chars)
start = max(0, end - max_chars)
snippet = content[start:end]
if start and len(snippet) >= 3:
snippet = f'...{snippet[3:]}'
if end < len(content) and len(snippet) >= 3:
snippet = f'{snippet[:-3]}...'
return snippet
def _bounded_search_matches(
matches: list[MemorySearchMatch],
prefix: str,
*,
limit: int,
max_chars: int,
) -> tuple[list[MemorySearchMatchResult], bool]:
bounded: list[MemorySearchMatchResult] = []
remaining = max_chars
truncated = len(matches) > limit
for match in matches[:limit]:
if not match.path.startswith(prefix):
raise RuntimeError('memory search backend returned a result outside the requested scope')
name = match.path.removeprefix(prefix)
if '/' in name or not _FILENAME_RE.fullmatch(name) or '..' in name:
raise RuntimeError('memory search backend returned a non-file or nested result')
remaining -= len(name)
if remaining < 0:
truncated = True
break
snippet = match.snippet[:remaining]
if len(snippet) < len(match.snippet):
truncated = True
bounded.append(_search_match(match, prefix, snippet))
remaining -= len(snippet)
return bounded, truncated
def _scope_hash(scope: str) -> str:
return hashlib.sha256(scope.encode()).hexdigest()[:16]
def _set_span_base(span: Span, store: MemoryStore, scope_hash: str) -> None:
if span.is_recording():
span.set_attributes({'memory.backend': type(store).__name__, 'memory.scope_hash': scope_hash})
def _set_span_result(span: Span, outcome: str, **attributes: str | int | bool) -> None:
if span.is_recording():
span.set_attributes(
{'memory.outcome': outcome, **{f'memory.{key}': value for key, value in attributes.items()}}
)
def _set_span_error(span: Span, exc: Exception) -> None:
if span.is_recording():
span.set_attributes({'memory.outcome': 'error', 'memory.exception_type': type(exc).__name__})
pydantic_ai_harness/experimental/overflow/README.md → pydantic_ai_harness/overflowing_tool_output/README.md
+48 -25
View File
@@ -1,34 +1,26 @@
# Overflow capability
# Overflowing Tool Output
> [!WARNING]
> **Experimental.** This capability lives under `pydantic_ai_harness.experimental` and may
> change or be removed in any release, without a deprecation period. Import it from the
> experimental path -- there is no top-level export:
> [!NOTE]
> Import this capability from its submodule -- there is no top-level `pydantic_ai_harness` re-export:
>
> ```python
> from pydantic_ai_harness.experimental.overflow import OverflowingToolOutput
> from pydantic_ai_harness.overflowing_tool_output import OverflowingToolOutput
> ```
>
> Importing any experimental capability emits a `HarnessExperimentalWarning`. Silence **all**
> harness experimental warnings with a single filter (no per-capability lines needed):
>
> ```python
> import warnings
> from pydantic_ai_harness.experimental import HarnessExperimentalWarning
>
> warnings.filterwarnings('ignore', category=HarnessExperimentalWarning)
> ```
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
A tool can return a payload large enough to dominate the context window. Tool returns
persist in history as `ToolReturnPart`s, so an oversized one is re-sent on every later
model request -- paying its token cost for the rest of the run. `OverflowingToolOutput`
intercepts a return in the `after_tool_execute` hook, reduces it once, and lets the reduced
intercepts a return when it is produced, reduces it once, and lets the reduced
form persist. The reduction is not recomputed per request.
This is the overflow-to-file follow-up the `compaction` README names as out of scope: it
moves large tool outputs *out* of the window at production time, rather than compressing or
dropping context already inside it.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/overflowing_tool_output/)
## The three modes
| Mode | Cost | Lossy? | What the model gets |
@@ -54,7 +46,9 @@ registered tools (the Claude Code pattern, the core
The intended loop is grep to locate, then read the range around a match. A wrong or expired
handle returns a guiding message rather than raising, so it never consumes a tool retry (PR
[#293](https://github.com/pydantic/pydantic-ai-harness/pull/293)).
[#293](https://github.com/pydantic/pydantic-ai-harness/pull/293)). The query tools' own
returns are exempt from reduction, so a `grep_tool_result` or `read_tool_result` result is
never itself spilled or truncated.
### Every elision leaves an explicit marker
@@ -81,7 +75,7 @@ that fits wins; anything below the smallest threshold passes through.
```python
from pydantic_ai import Agent
from pydantic_ai_harness.experimental.overflow import (
from pydantic_ai_harness.overflowing_tool_output import (
Band,
OverflowingToolOutput,
Spill,
@@ -107,6 +101,9 @@ agent = Agent(
The default band, when you pass no `bands`, is `Spill(then=Truncate())`: lossless when a
store accepts the write, a bounded truncation otherwise -- zero LLM cost and no silent drop.
`Passthrough()` is an explicit no-op action for `bands` or `per_tool` lists, leaving matching
returns untouched.
### Fallbacks with `then`
Every action takes an optional `then`, applied when the action cannot run: a `Spill` whose
@@ -120,12 +117,25 @@ spill -> truncate.
`tail`); `tool_filter` (a `ToolSelector`) scopes which tools the capability touches at all.
```python
OverflowingToolOutput(
per_tool={
'read_file': [Band(over=8_000, action=Truncate(strategy=TruncationStrategy.head))],
'run_shell': [Band(over=8_000, action=Truncate(strategy=TruncationStrategy.tail))],
},
tool_filter=['read_file', 'run_shell', 'search'],
from pydantic_ai import Agent
from pydantic_ai_harness.overflowing_tool_output import (
Band,
OverflowingToolOutput,
Truncate,
TruncationStrategy,
)
agent = Agent(
'openai:gpt-4o',
capabilities=[
OverflowingToolOutput(
per_tool={
'read_file': [Band(over=8_000, action=Truncate(strategy=TruncationStrategy.head))],
'run_shell': [Band(over=8_000, action=Truncate(strategy=TruncationStrategy.tail))],
},
tool_filter=['read_file', 'run_shell', 'search'],
)
],
)
```
@@ -134,7 +144,8 @@ OverflowingToolOutput(
Thresholds are measured in characters by default. Set `over_tokens=True` to measure in
estimated tokens (the same ~4-chars-per-token heuristic as `compaction`); pass a `tokenizer`
callable for accuracy. `Truncate.max_chars` is always characters -- truncation is a
character operation regardless of the threshold unit.
character operation regardless of the threshold unit. Set `strip_ansi=True` to strip ANSI
escape sequences from text returns before measuring and reducing.
## Spill store
@@ -147,6 +158,9 @@ workspace once #4352 lands) can resolve the same handle in another process. Supp
backend with `store=...`.
```python
from typing import Protocol
class OverflowStore(Protocol):
async def write(self, key: str, data: bytes) -> str: ... # returns a handle
async def read(self, handle: str) -> bytes: ...
@@ -168,7 +182,11 @@ agent that still wants to read a spill. To bound disk use, opt into age-based pr
```python
from datetime import timedelta
from pydantic_ai import Agent
from pydantic_ai_harness.overflowing_tool_output import LocalFileStore, OverflowingToolOutput
store = LocalFileStore(cleanup_after=timedelta(hours=6)) # default: None = keep forever
agent = Agent('openai:gpt-4o', capabilities=[OverflowingToolOutput(store=store)])
```
When set, a `write` schedules a background prune (a daemon thread, off the hot path) that
@@ -197,6 +215,11 @@ A `Summarize` call is a real request to the model, so its full usage -- tokens a
request itself -- folds into the run's `ctx.usage`, exactly like `SummarizingCompaction`. No
token caps are imposed on the summary call. A `UsageLimits` request limit will see it.
By default `Summarize` inherits the running agent's model (`ctx.model`). Pass a model id or
instance to `Summarize(model=...)` to override, or a `summarize` callable to bypass the
built-in prompt entirely. The `summary_prompt` template on the capability must contain both
`{tool_name}` and `{output}` placeholders.
## Edge cases
- Binary returns spill verbatim and are never stringify-truncated; `Truncate` / `Summarize`
pydantic_ai_harness/overflowing_tool_output/__init__.py
+41
View File
@@ -0,0 +1,41 @@
"""Overflow capability: reduce oversized tool returns at production time.
`OverflowingToolOutput` intercepts a tool return when it is produced and reduces it --
truncating, spilling to a queryable file, or summarizing -- so an oversized payload does
not persist in history and get re-sent on every later model request. Combine the three
modes through an ordered list of size `bands`.
Spilled payloads are read back on demand through the registered `read_tool_result` tool;
the `OverflowStore` protocol is the seam for a durable backend (the local-file default
ships for single-process runs).
"""
from pydantic_ai_harness.overflowing_tool_output._bands import (
Action,
Band,
Passthrough,
Spill,
Summarize,
SummarizeFunc,
Truncate,
)
from pydantic_ai_harness.overflowing_tool_output._capability import OverflowingToolOutput
from pydantic_ai_harness.overflowing_tool_output._markers import GREP_TOOL_NAME, READ_TOOL_NAME
from pydantic_ai_harness.overflowing_tool_output._payload import TruncationStrategy
from pydantic_ai_harness.overflowing_tool_output._store import LocalFileStore, OverflowStore
__all__ = [
'GREP_TOOL_NAME',
'READ_TOOL_NAME',
'Action',
'Band',
'LocalFileStore',
'OverflowStore',
'OverflowingToolOutput',
'Passthrough',
'Spill',
'Summarize',
'SummarizeFunc',
'Truncate',
'TruncationStrategy',
]
pydantic_ai_harness/experimental/overflow/_bands.py → pydantic_ai_harness/overflowing_tool_output/_bands.py
+1 -1
View File
@@ -16,7 +16,7 @@ from collections.abc import Awaitable, Callable
from dataclasses import dataclass
from typing import TYPE_CHECKING
from pydantic_ai_harness.experimental.overflow._payload import TruncationStrategy
from pydantic_ai_harness.overflowing_tool_output._payload import TruncationStrategy
if TYPE_CHECKING:
from pydantic_ai.models import Model
pydantic_ai_harness/experimental/overflow/_capability.py → pydantic_ai_harness/overflowing_tool_output/_capability.py
+5 -5
View File
@@ -15,7 +15,7 @@ from pydantic_ai.messages import ToolCallPart, ToolReturn, ToolReturnContent, Us
from pydantic_ai.tools import AgentDepsT, RunContext, ToolDefinition, ToolSelector, matches_tool_selector
from pydantic_ai.toolsets import AgentToolset
from pydantic_ai_harness.experimental.overflow._bands import (
from pydantic_ai_harness.overflowing_tool_output._bands import (
Action,
Band,
Passthrough,
@@ -23,7 +23,7 @@ from pydantic_ai_harness.experimental.overflow._bands import (
Summarize,
Truncate,
)
from pydantic_ai_harness.experimental.overflow._markers import (
from pydantic_ai_harness.overflowing_tool_output._markers import (
GREP_TOOL_NAME,
READ_TOOL_NAME,
elision_marker,
@@ -31,7 +31,7 @@ from pydantic_ai_harness.experimental.overflow._markers import (
spill_header,
summary_header,
)
from pydantic_ai_harness.experimental.overflow._payload import (
from pydantic_ai_harness.overflowing_tool_output._payload import (
is_binary,
json_sketch,
measure,
@@ -40,7 +40,7 @@ from pydantic_ai_harness.experimental.overflow._payload import (
to_text,
truncate_text,
)
from pydantic_ai_harness.experimental.overflow._store import LocalFileStore, OverflowStore
from pydantic_ai_harness.overflowing_tool_output._store import LocalFileStore, OverflowStore
_QUERY_TOOL_NAMES = (READ_TOOL_NAME, GREP_TOOL_NAME)
"""The spill query tools. Their own returns are exempt from reduction (never re-spilled)."""
@@ -105,7 +105,7 @@ class OverflowingToolOutput(AbstractCapability[AgentDepsT]):
Example:
```python
from pydantic_ai import Agent
from pydantic_ai_harness.experimental.overflow import (
from pydantic_ai_harness.overflowing_tool_output import (
Band,
OverflowingToolOutput,
Spill,
pydantic_ai_harness/experimental/overflow/_markers.py → pydantic_ai_harness/overflowing_tool_output/_markers.py
View File
pydantic_ai_harness/experimental/overflow/_payload.py → pydantic_ai_harness/overflowing_tool_output/_payload.py
+2 -2
View File
@@ -15,8 +15,8 @@ from typing import TypeGuard
from pydantic_ai.messages import ModelMessage, ModelRequest, SystemPromptPart
from pydantic_core import to_json
from pydantic_ai_harness.experimental.compaction._shared import estimate_token_count
from pydantic_ai_harness.experimental.overflow._markers import elision_marker
from pydantic_ai_harness.compaction._shared import estimate_token_count
from pydantic_ai_harness.overflowing_tool_output._markers import elision_marker
class TruncationStrategy(str, Enum):
pydantic_ai_harness/experimental/overflow/_store.py → pydantic_ai_harness/overflowing_tool_output/_store.py
View File
pydantic_ai_harness/experimental/planning/README.md → pydantic_ai_harness/planning/README.md
+10 -16
View File
@@ -1,26 +1,18 @@
# Planning
> [!WARNING]
> **Experimental.** This capability lives under `pydantic_ai_harness.experimental` and may
> change or be removed in any release, without a deprecation period. Import it from the
> experimental path -- there is no top-level export:
> [!NOTE]
> Import this capability from its submodule -- there is no top-level `pydantic_ai_harness` re-export:
>
> ```python
> from pydantic_ai_harness.experimental.planning import Planning
> from pydantic_ai_harness.planning import Planning
> ```
>
> Importing any experimental capability emits a `HarnessExperimentalWarning`. Silence **all**
> harness experimental warnings with a single filter (no per-capability lines needed):
>
> ```python
> import warnings
> from pydantic_ai_harness.experimental import HarnessExperimentalWarning
>
> warnings.filterwarnings('ignore', category=HarnessExperimentalWarning)
> ```
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
Give an agent a structured, self-updating task plan -- without ever invalidating the prompt cache.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/planning/)
## The problem
Long agentic runs drift: the model loses track of what it set out to do and what's left. The usual fix -- keep a running plan and re-inject it into the system prompt each turn -- invalidates the prompt cache. The system prompt sits at the front of the request, so every plan edit changes the cached prefix and forces the whole conversation to be re-processed at full token price.
@@ -36,7 +28,7 @@ So the plan stays current in the model's view while the cached prefix is never i
```python
from pydantic_ai import Agent
from pydantic_ai_harness.experimental.planning import Planning
from pydantic_ai_harness.planning import Planning
agent = Agent('anthropic:claude-sonnet-4-6', capabilities=[Planning()])
@@ -70,6 +62,8 @@ The plan is never injected into the system prompt or instructions. Static usage
## Configuration
```python
from pydantic_ai_harness.planning import Planning
Planning(
guidance=None, # static system-prompt guidance; None = default, '' = omit
cache_ttl='5m', # TTL for the cache breakpoint before the reminder ('5m' | '1h')
@@ -109,7 +103,7 @@ capabilities:
```python
from pydantic_ai import Agent
from pydantic_ai_harness.experimental.planning import Planning
from pydantic_ai_harness.planning import Planning
agent = Agent.from_file('agent.yaml', custom_capability_types=[Planning])
```
pydantic_ai_harness/planning/__init__.py
+6
View File
@@ -0,0 +1,6 @@
"""Planning capability: model-owned, cache-friendly task planning for agents."""
from pydantic_ai_harness.planning._capability import Planning
from pydantic_ai_harness.planning._toolset import PlanItem, PlanningToolset, TaskStatus
__all__ = ['PlanItem', 'Planning', 'PlanningToolset', 'TaskStatus']
pydantic_ai_harness/experimental/planning/_capability.py → pydantic_ai_harness/planning/_capability.py
+2 -3
View File
@@ -10,14 +10,13 @@ from pydantic_ai.messages import CachePoint, ModelRequest, ModelResponse, UserPr
from pydantic_ai.tools import AgentDepsT, RunContext
from pydantic_ai.toolsets import AgentToolset
from pydantic_ai_harness.experimental.planning._toolset import PlanningToolset, PlanState, render_plan
from pydantic_ai_harness.planning._toolset import PlanningToolset, PlanState, render_plan
if TYPE_CHECKING:
from pydantic_ai._instructions import AgentInstructions
from pydantic_ai.capabilities.abstract import WrapModelRequestHandler
from pydantic_ai.models import ModelRequestContext
_DEFAULT_GUIDANCE = (
'You have a planning tool, `write_plan`. For multi-step work, call it first to lay out the '
'steps, then call it again to update statuses as you start and finish each step. Pass the '
@@ -41,7 +40,7 @@ class Planning(AbstractCapability[AgentDepsT]):
```python
from pydantic_ai import Agent
from pydantic_ai_harness.experimental.planning import Planning
from pydantic_ai_harness.planning import Planning
agent = Agent('anthropic:claude-sonnet-4-6', capabilities=[Planning()])
```
pydantic_ai_harness/experimental/planning/_toolset.py → pydantic_ai_harness/planning/_toolset.py
View File
pydantic_ai_harness/experimental/authoring/README.md → pydantic_ai_harness/runtime_authoring/README.md
+33 -17
View File
@@ -1,26 +1,18 @@
# RuntimeAuthoring
# Runtime Authoring
> [!WARNING]
> **Experimental.** This capability lives under `pydantic_ai_harness.experimental` and may
> change or be removed in any release, without a deprecation period. Import it from the
> experimental path -- there is no top-level export:
> [!NOTE]
> Import this capability from its submodule -- there is no top-level `pydantic_ai_harness` re-export:
>
> ```python
> from pydantic_ai_harness.experimental.authoring import RuntimeAuthoring
> from pydantic_ai_harness.runtime_authoring import RuntimeAuthoring
> ```
>
> Importing any experimental capability emits a `HarnessExperimentalWarning`. Silence **all**
> harness experimental warnings with a single filter (no per-capability lines needed):
>
> ```python
> import warnings
> from pydantic_ai_harness.experimental import HarnessExperimentalWarning
>
> warnings.filterwarnings('ignore', category=HarnessExperimentalWarning)
> ```
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
Let an agent author, validate, and persist real pydantic-ai capabilities at runtime.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/runtime_authoring/)
## The problem
A coding agent often discovers, mid-task, that it wants a behavior its host does not
@@ -50,12 +42,16 @@ authoring a hook means authoring a capability that overrides one lifecycle metho
from pathlib import Path
from pydantic_ai import Agent
from pydantic_ai_harness.experimental.authoring import RuntimeAuthoring
from pydantic_ai_harness.runtime_authoring import RuntimeAuthoring
authoring = RuntimeAuthoring(directory=Path('.authored'))
agent = Agent('anthropic:claude-sonnet-4-6', capabilities=[authoring])
```
`RuntimeAuthoring` also contributes static, cache-stable system-prompt guidance
explaining these tools. Leave `guidance=None` for the default text, or pass your own
string; set `guidance=''` to omit it entirely.
## Activation boundary
A capability **cannot** be added to a live, already-executing run. pydantic-ai resolves
@@ -74,11 +70,22 @@ active capabilities into each run. With `agent.run(..., capabilities=...)`, the
capability is live on the very next loop iteration -- no process restart.
```python
from pathlib import Path
from pydantic_ai import Agent
from pydantic_ai_harness.experimental.authoring import RuntimeAuthoring
authoring = RuntimeAuthoring(directory=Path('.authored'))
agent = Agent('anthropic:claude-sonnet-4-6', capabilities=[authoring])
history = None
done = False
next_prompt = 'Start the task.'
while not done:
extra = authoring.store.load_active()
result = await agent.run(next_prompt, message_history=history, capabilities=extra)
history = result.all_messages()
# ... decide `next_prompt` and `done` from `result` ...
```
Because the capabilities also persist to disk (`<directory>/<name>.py` plus a
@@ -88,11 +95,20 @@ Because the capabilities also persist to disk (`<directory>/<name>.py` plus a
`manifest.json` records each capability's name, module file, class name, status, and last
validation error -- the surface a UI can read to show what the agent has authored.
Capability names must be lowercase letters, digits, and underscores, starting with a
letter; reusing a name replaces the previous capability of that name.
## Trust boundary and the sandboxed alternative
Authoring executes arbitrary Python in-process at import, construction, and run time. That
is the same trust boundary an agent that already runs shell commands and edits files
operates under, which is the deliberate choice here.
operates under, which is the deliberate choice here. Do not point it at a directory whose
contents you would not run yourself, and treat authored capabilities as code the agent is
executing on your host.
Because authored capabilities hold live code, they are not spec-serializable
(`get_serialization_name()` returns `None`) and are persisted as source rather than as an
agent spec.
The sandboxed alternative is the dormant `pa` Monty hook-slot registration system in the
Loopy tree (`pa/slots.py`, `pa/registration_tools.py`, `pa/capability.py`
pydantic_ai_harness/runtime_authoring/__init__.py
+20
View File
@@ -0,0 +1,20 @@
"""Runtime capability authoring: let an agent write, validate, and register real capabilities."""
from pydantic_ai_harness.runtime_authoring._capability import RuntimeAuthoring
from pydantic_ai_harness.runtime_authoring._store import AuthoredCapability, CapabilityStore
from pydantic_ai_harness.runtime_authoring._toolset import AuthoringToolset
from pydantic_ai_harness.runtime_authoring._validate import (
CapabilityValidationError,
load_capability_instance,
validate_capability_file,
)
__all__ = [
'AuthoredCapability',
'AuthoringToolset',
'CapabilityStore',
'CapabilityValidationError',
'RuntimeAuthoring',
'load_capability_instance',
'validate_capability_file',
]
pydantic_ai_harness/experimental/authoring/_capability.py → pydantic_ai_harness/runtime_authoring/_capability.py
+3 -4
View File
@@ -10,13 +10,12 @@ from pydantic_ai.capabilities import AbstractCapability
from pydantic_ai.tools import AgentDepsT
from pydantic_ai.toolsets import AgentToolset
from pydantic_ai_harness.experimental.authoring._store import CapabilityStore
from pydantic_ai_harness.experimental.authoring._toolset import AuthoringToolset
from pydantic_ai_harness.runtime_authoring._store import CapabilityStore
from pydantic_ai_harness.runtime_authoring._toolset import AuthoringToolset
if TYPE_CHECKING:
from pydantic_ai._instructions import AgentInstructions
_DEFAULT_GUIDANCE = (
'You can author new pydantic-ai capabilities at runtime with `author_capability(name, code)`. '
'A capability is a subclass of `pydantic_ai.capabilities.AbstractCapability` that constructs with '
@@ -48,7 +47,7 @@ class RuntimeAuthoring(AbstractCapability[AgentDepsT]):
from pathlib import Path
from pydantic_ai import Agent
from pydantic_ai_harness.experimental.authoring import RuntimeAuthoring
from pydantic_ai_harness.runtime_authoring import RuntimeAuthoring
authoring = RuntimeAuthoring(directory=Path('.authored'))
agent = Agent('anthropic:claude-sonnet-4-6', capabilities=[authoring])
pydantic_ai_harness/experimental/authoring/_store.py → pydantic_ai_harness/runtime_authoring/_store.py
+1 -1
View File
@@ -19,7 +19,7 @@ from typing import Literal
from pydantic import BaseModel, ValidationError
from pydantic_ai.capabilities import AbstractCapability
from pydantic_ai_harness.experimental.authoring._validate import (
from pydantic_ai_harness.runtime_authoring._validate import (
CapabilityValidationError,
load_capability_instance,
validate_capability_file,

Some files were not shown because too many files have changed in this diff Show More