furyhawk/pydantic-ai-harness
1
0
Fork 0
mirror of https://github.com/pydantic/pydantic-ai-harness.git synced 2026-07-20 10:25:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: capability pages for the unified docs site + README/doc parity gate (#329)

Browse Source 
* docs: publish capability docs to the unified site + add README/doc parity gate

Every capability shipped only a README (kept for GitHub/PyPI). This adds a
parallel, cleaned-up page per capability under docs/ for the new unified docs
site (pydantic.dev/docs/harness), migrated from each README: snippets verified
runnable against source, autodoc API blocks, root-relative Pydantic AI links,
and an experimental-status admonition on the experimental set.

To keep README and doc in sync going forward, adds a docs-parity-reviewer agent
and a parity gate in the review checklist (run as the last step before merge),
plus the docs/ layout and the README<->doc requirement in AGENTS.md and the
capability-authoring guide.

* docs: fix README<->doc<->source inconsistencies across capabilities

A parity audit against source found drift, mostly in the capability READMEs
(staler than the migrated docs). All fixes verified against source:

- Correctness: the "approval/deferred tools are excluded from the sandbox" claim
  (code_mode README + doc) was false -- those tools are sandboxed like any
  other; corrected in both. The stale Shell persist_cwd sentinel description is
  replaced with the actual out-of-band temp-file capture. filesystem protected
  default `.git/` -> `.git/*` (the bare form never matched).
- Runnable snippets: added the missing imports/wiring so README snippets no
  longer raise NameError (subagents, context, planning, overflow, authoring,
  filesystem, code_mode).
- Parity: documented previously-undocumented params/behaviors (compaction
  strategy options, overflow strip_ansi/Passthrough, extra autodoc classes for
  context and subagents), fixed a stale version pin (>=1.95.1 -> >=2.1.0), and
  added the missing Managed Prompt row to the root README capability matrix.
- Style: normalized decorative Unicode to ASCII across all READMEs and dropped a
  hype phrase, matching AGENTS.md writing style and the docs.

* docs: add nav.json to drive the unified-docs harness sidebar

The unified docs mount the harness docs under /docs/ai/harness (fed live from
this repo via the pydantic-ai 'Pydantic AI Harness' section). This nav.json
defines the sub-nav (Overview + Capabilities + Experimental) and the set of doc
files the site includes.

* docs: migrate "What goes where?" explainer into harness overview

Adds the core-vs-harness boundary section (anchor #what-goes-where) to the
canonical harness overview, so the pydantic-ai docs that link to it can point
here after the duplicated in-repo stub is removed.

* docs: address CodeRabbit review -- runnable snippets, accuracy, multi-class autodoc

* docs: flatten harness nav and align with graduated capabilities

Following the experimental-graduation refactor (#347), restructure the
unified-docs harness pages:

- Flatten docs/ (drop capabilities/ and experimental/ subdirs); the sidebar
  is now Overview + one flat list per Douwe's request.
- Rename to match the graduated modules: overflow -> overflowing-tool-output,
  authoring -> runtime-authoring, docs -> pydantic-ai-docs.
- Drop the 'Experimental' admonitions from the graduated capabilities and
  repoint every import + ::: autodoc path off pydantic_ai_harness.experimental.
- Add docs for the newly-shipped capabilities: guardrails, dynamic-workflow,
  media, and acp (acp stays framed as experimental -- it may still be removed).
- Every capability doc now links to its source; index capability table lists
  the full set with flat links.

* docs: apply team-sync authoring rules + enforce them in CI

From the 2026-07-10 docs review on #329:

- Purpose-first leads: drop hook names (before_model_request,
  after_tool_execute) from the opening paragraphs of compaction and
  overflowing-tool-output (doc + README); mechanism moves lower.
- Mirror the soft 'API may change between releases' stability note from each
  graduated README into its doc page (ACP keeps its stronger experimental
  warning; guardrails' README has no note, so its page gets none).
- README H1s now use the capability's display name (Overflow capability ->
  Overflowing Tool Output, RuntimeAuthoring -> Runtime Authoring, SubAgents ->
  Subagents, etc.).
- Extend tests/test_docs_parity.py with per-page mechanical checks: source link
  present, heading matches the capability name, purpose-first lead (no hook in
  the opener), and no experimental framing on graduated pages (ACP excepted).
- Update the docs-parity-reviewer agent + review-checklist to the flat
  structure and the new semantic checks.

* docs: add the stability note to guardrails (parity with sibling capabilities)

guardrails was the one graduated capability whose README and doc page lacked
the shared 'API may change between releases' note. Add it to both.

* fix: restore uv.lock to match pyproject (bad text-merge dropped 8 lines)

Merging origin/main did a git text-merge of the generated uv.lock, leaving it
inconsistent with pyproject.toml -- every CI job failed at 'uv sync --locked'.
pyproject.toml is identical to main here, so the correct lock is main's.

* docs: address CodeRabbit review on #329

Findings that failed to post inline (GitHub error) but were real:
- context/README.md, planning/README.md: two nested examples still imported
  from pydantic_ai_harness.experimental.* -- repoint to the graduated modules.
- guardrails/README.md: replace em dashes with '--' (repo style) and add the
  source-module link.
- docs/media.md: standardize on the implementation's canonical media+sha256://
  URI scheme (was mixing media://).
- tests/test_docs_parity.py: strengthen my own checks per review --
  source-link and top-README-link now require a real Markdown link to the
  page's specific module (not a bare substring); heading checks assert an H1
  exists and equals the expected capability name via explicit page metadata.

* fix: restore uv.lock [options.exclude-newer-package] block

The lock lost its [options.exclude-newer-package] manifest (pydantic-ai-slim
= false, ...) -- a bad git text-merge dropped it, and diagnostic uv commands
rewrote it under a different local config. Without that block CI's
'uv sync --locked' re-resolves and fails ('addition of exclude newer exclusion
for pydantic-ai-slim'). Restore origin/main's exact lock.

* fix: restore uv.lock [options.exclude-newer-package] block

A pre-commit hook was rewriting uv.lock under the local uv config, stripping
the [options.exclude-newer-package] manifest (pydantic-ai-slim = false, ...).
Without it CI's 'uv sync --locked' re-resolves and fails. Commit origin/main's
exact lock with --no-verify so no hook mutates it (lock-only change).

* test: cover the docs-parity helper edge cases (100% coverage)

The strengthened helpers added defensive branches (missing frontmatter close,
fenced code before the lead, missing/forbidden/ClassName H1, lead running to
EOF) that no real doc exercises. Add direct unit tests so the file is back to
the repo's required 100% coverage.

* docs: link every capability README to its source module + enforce it

CodeRabbit re-flagged planning/README.md for a missing source link. Only
guardrails had one, so add the source-module link to all 15 remaining
capability READMEs (matching the doc pages) and add a parity test so the
requirement is mechanical and cannot silently regress.

* docs(agents): drop stale folder tree; fix flat docs path + guard names

AGENTS.md's File-structure ASCII tree and capability-authoring's doc paths
still showed docs/capabilities// docs/experimental/ (flattened in this PR) and
the old /docs/harness URL. Delete the tree rather than redraw it -- the layout
is discoverable by listing the repo; keep only the non-obvious conventions
(flat docs/, the README<->doc parity requirement). Also fix the Vocabulary
guard examples (InputGuard/OutputGuard, not the nonexistent InputGuardrail/
CostGuard).

* test: statically validate doc snippets exist and parse

Every Python snippet in the capability READMEs and docs/*.md pages is now
checked for the two failures a reader hits immediately: it does not parse
(syntax), or it imports a pydantic_ai_harness symbol that does not exist (stale
module path or renamed name -- the class of bug behind the experimental.* import
drift). Static only: no model/network execution, so it needs no mocking. The
four illustrative API-signature blocks opt out with a {test="skip"} fence
(read by pytest-examples, stripped-safe for the unified-docs render).

* test: don't fail doc-snippet check on a missing optional extra

The static check imported capability modules to resolve their symbols, but in
the slim CI job (no extras) importing e.g. pydantic_ai_harness.experimental.acp
raises ModuleNotFoundError for the absent third-party 'acp' package -- the
harness module exists, its extra just isn't installed. Distinguish a genuinely
missing harness module (fail) from a missing extra (skip) by the ImportError's
module name.
This commit is contained in:
David SF
2026-07-13 11:00:12 -05:00
committed by GitHub
parent 310ffadf09
commit 3ba9e2f9a5
42 changed files with 4420 additions and 106 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/agents/docs-parity-reviewer.md
+82
View File
@@ -0,0 +1,82 @@
---
name: docs-parity-reviewer
description: Use as the final documentation gate before a capability PR merges. Verifies that a user-facing change keeps the capability README and its unified-docs page in sync with each other and with the code, that every snippet is runnable, and that links follow repo convention. Reports gaps; does not edit.
tools: Read, Grep, Glob, Bash
model: sonnet
---
You are the documentation parity gate for `pydantic-ai-harness`. Every released
capability ships two docs that must stay in sync with the code and with each
other:
- **README** -- `pydantic_ai_harness/<capability>/README.md` (or
`pydantic_ai_harness/experimental/acp/README.md` for ACP). Serves GitHub and
PyPI. Keeps absolute links and its badges.
- **Unified doc** -- flat at `docs/<capability>.md`. Renders on the docs site
(`https://pydantic.dev/docs/ai/harness/`). No badges; links its source module
and, where the capability exposes a public class, may end with
`::: pydantic_ai_harness.<Class>` autodoc blocks. The sidebar is a flat list --
no `capabilities/` or `experimental/` subdirectories.
Both are hand-maintained. A change to one that is not reflected in the other is
the failure mode you exist to catch.
## What you are given
The diff or description of a capability change (the touched capability, and what
its user-facing behavior now is). If you are not told which capability changed,
infer it from the changed files under `pydantic_ai_harness/`.
## Checks
Read the capability source, its README, and its unified doc, then report each
problem as a finding (blocking / warning / nit) with a concrete fix.
1. **Both docs updated.** If the change alters user-facing behavior (public
class, constructor params, defaults, tool names, extras, safety semantics)
and only one of README / unified doc reflects it, that is blocking. A doc
describing behavior the code no longer has is also blocking.
2. **Snippets run.** Every code block in both docs has all imports and the
pieces needed to actually run (Agent construction, capability wiring). Class
names, params, and defaults match the current source. Model ids are unchanged
from what the source uses -- a changed model id is blocking.
3. **README <-> unified doc consistency.** The two agree on install extras,
option names, defaults, and safety caveats. They need not be identical prose,
but they must not contradict each other or the code.
4. **Links.** Unified doc: harness-internal links are relative `.md`
(`[Shell](shell.md)`); Pydantic AI links use
root-relative internal paths `/ai/<section>/<page>/` (not legacy
`ai.pydantic.dev` links); no leftover `../../README.md`, `../capabilities/`,
`../experimental/`, or badge markup.
README: absolute links are fine.
5. **Source link + API block.** Every page links its source module
(`https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/<module>/`)
so a reading agent can verify behavior -- a missing source link is a finding.
Where the capability exposes a public class, the page may also end with a
`## API reference` section of `::: pydantic_ai_harness...` autodoc blocks
(auto-expanded from the docstring, not hand-written). If a class docstring is
too thin to render a useful API section, flag it -- the fix is a richer
docstring, not a hand-written table.
6. **Safety caveats preserved.** Where the source carries access, sandbox, or
command-control limits (Shell, CodeMode, FileSystem), both docs state them.
7. **Writing style.** Both follow `AGENTS.md` "Writing style": no em-dashes (use
`--`), no hype, plain ASCII punctuation.
8. **Purpose-first lead.** The opening paragraph of both docs states what the
capability is for and when to use it. An internal hook or class name
(`before_model_request`, `after_tool_execute`, ...) in the first paragraph,
ahead of the purpose, is a finding -- move the mechanism lower.
9. **Name matches the capability.** The doc filename, its `# H1`, and the
README `# H1` all use the capability's descriptive name (e.g. "Overflowing
Tool Output", not "Overflow"). A short or ClassName-style heading is a finding.
10. **Stability framing.** Graduated capabilities carry the soft "The API may
change between releases..." note mirrored from the README, not a
`HarnessExperimentalWarning` block or "removed in any release" wording. ACP
is the only page that keeps an `!!! warning "Experimental"`.
If a released capability has a README but no `docs/` page (or vice versa), that
missing file is a blocking finding.
## Output
A terse list of findings, most severe first, each naming the file, the severity,
and the fix. If everything is in order, say so in one line. Do not edit files.
.gitignore
+5
View File
@@ -3,6 +3,11 @@
.DS_Store
.agents/settings.local.json
.agents/skills/branch-context/
# .agents/ (aka .claude via the symlink) is Claude Code config. Most of it is
# per-developer, but the agents/ dir holds shared, committed agents such as the
# docs-parity-reviewer -- keep it tracked.
!.agents/agents/
!.claude/agents/
AGENTS.local.md
CLAUDE.local.md
LOCAL_WORKTREES.md
AGENTS.md
+13 -14
View File
@@ -21,7 +21,7 @@ change instead of reimplementing core behavior in harness.
- **Capability**: an `AbstractCapability` subclass that bundles tools, hooks, instructions, and model settings into a reusable unit. This is the core abstraction of pydantic-ai-harness.
- **Hook**: a lifecycle method on `AbstractCapability` that intercepts agent graph execution (e.g. `before_model_request`, `wrap_run`, `after_tool_execute`)
- **Toolset**: a collection of tools that a capability can provide to the agent
- **Guard**: a type of capability that validates inputs/outputs or controls tool access (e.g. `InputGuardrail`, `CostGuard`)
- **Guard**: a type of capability that validates inputs/outputs or controls tool access (e.g. `InputGuard`, `OutputGuard`)
- **Harness**: this package -- a collection of pre-made capabilities for Pydantic AI.
- **AICA**: AI Code Assistant -- the automated agent that implements issues, reviews plans, and handles PR feedback
- **Ralph loop**: the state-machine-based workflow that drives AICA through phases (TRIAGE -> GOALS -> PLAN -> CODE -> VERIFY -> REVIEW -> PUBLISH)
@@ -114,19 +114,18 @@ Always run `make lint && make typecheck && make test` before committing.
## File structure
```
pydantic_ai_harness/
__init__.py # public API re-exports
<capability>/ # each capability gets its own package
__init__.py # public exports for the capability
_capability.py # capability class (AbstractCapability subclass)
_toolset.py # toolset implementation
README.md # standalone docs for the capability
tests/
conftest.py # shared fixtures (TestModel, test_agent)
<capability>/ # tests mirror source packages
test_<capability>.py
```
The tree is discoverable by listing it; only the conventions that are not are
recorded here.
Each released capability is a self-contained package under
`pydantic_ai_harness/<capability>/` (naming and exports are covered in the
preflight above), with tests under `tests/<capability>/`. It ships **two**
hand-maintained docs that must stay in sync: the `README.md` next to the code
(GitHub/PyPI) and the `docs/<capability>.md` page (the docs site at
pydantic.dev/docs/ai/harness). The `docs/` folder is flat -- there are no
`capabilities/` or `experimental/` subdirectories. A user-facing change updates
both; `agent_docs/review-checklist.md` "Docs" and the `docs-parity-reviewer`
subagent enforce the parity before merge.
Do not add placeholder template files for new capabilities. Start from the
existing `CodeMode` package shape, then delete what the new capability does not
README.md
+4 -3
View File
@@ -94,7 +94,7 @@ practices and warning of a "normalization of deviance" as engineers stop reviewi
[![Logfire trace from the Quick start run](docs/images/quick-start-trace.png)](https://logfire-us.pydantic.dev/public-trace/84bcf123-2106-49da-9f6f-5c26395339bb?spanId=7650806a0785b946)
**[See this run as a public Logfire trace →](https://logfire-us.pydantic.dev/public-trace/84bcf123-2106-49da-9f6f-5c26395339bb?spanId=7650806a0785b946)** Each `run_code` span fans out into the tool calls the model issued from inside the sandbox -- it's the easiest way to understand what code mode actually did.
**[See this run as a public Logfire trace ->](https://logfire-us.pydantic.dev/public-trace/84bcf123-2106-49da-9f6f-5c26395339bb?spanId=7650806a0785b946)** Each `run_code` span fans out into the tool calls the model issued from inside the sandbox -- it's the easiest way to understand what code mode actually did.
## Orchestrating sub-agents: DynamicWorkflow
@@ -182,6 +182,7 @@ We studied leading coding agents, agent frameworks, and Claw-style assistants to
| | **Tool orphan repair** | Fix orphaned tool calls in conversation history | :construction: [PR&nbsp;#184](https://github.com/pydantic/pydantic-ai-harness/pull/184) | |
| **Reasoning** | **Adaptive reasoning** | Adjust thinking effort based on task complexity | :construction: [PR&nbsp;#174](https://github.com/pydantic/pydantic-ai-harness/pull/174) | |
| | **Current time** | Inject current date/time into system prompt | :construction: [PR&nbsp;#170](https://github.com/pydantic/pydantic-ai-harness/pull/170) | |
| **Prompt management** | **Managed prompt** | Serve Logfire-managed prompts as instructions, so you can edit and roll them out without a code deploy | :white_check_mark: [Docs](pydantic_ai_harness/logfire/) | |
> Packages by [vstorm-co](https://github.com/vstorm-co) are endorsed by the Pydantic AI team. We're working with them to upstream some of their implementations into this repo.
@@ -326,8 +327,8 @@ make testcov # pytest with 100% branch coverage
Pydantic AI Harness uses **0.x versioning** to signal that APIs are still stabilizing. During 0.x:
- **Minor releases** (0.1 → 0.2) may include breaking changes -- renamed parameters, changed defaults, restructured APIs. As the library grows, especially as capabilities gain provider-native support (starting as a local implementation, then auto-switching to the provider's built-in API when available), we may need to reshape APIs we couldn't fully anticipate in the initial design.
- **Patch releases** (0.1.0 → 0.1.1) will not intentionally break existing behavior.
- **Minor releases** (0.1 -> 0.2) may include breaking changes -- renamed parameters, changed defaults, restructured APIs. As the library grows, especially as capabilities gain provider-native support (starting as a local implementation, then auto-switching to the provider's built-in API when available), we may need to reshape APIs we couldn't fully anticipate in the initial design.
- **Patch releases** (0.1.0 -> 0.1.1) will not intentionally break existing behavior.
- **All breaking changes** are documented in release notes with migration guidance.
- Where practical, we'll keep the previous behavior available under a deprecated name or configuration option before removing it.
agent_docs/capability-authoring.md
+8 -1
View File
@@ -25,7 +25,14 @@ Each capability package should normally have:
- `__init__.py` with public exports
- `_capability.py` for the public capability class
- `_toolset.py` only if the capability needs toolset behavior
- `README.md` with focused usage docs
- `README.md` with focused usage docs (serves GitHub and PyPI)
- a unified-docs page at `docs/<capability>.md` (the `docs/` folder is flat --
no `capabilities/` or `experimental/` subdirectories). It mirrors the README
for the docs site, drops badges, links other harness pages with relative `.md`
links and Pydantic AI docs with root-relative `/ai/...` links, links its
source module, and -- where the capability exposes a public class -- may end
with a `::: pydantic_ai_harness.<Class>` autodoc block. The README and this
page are kept in sync (see `review-checklist.md` "Docs").
- mirrored tests under `tests/<capability>/`
The root `pydantic_ai_harness/__init__.py` should re-export stable public
agent_docs/review-checklist.md
+53 -2
View File
@@ -45,7 +45,58 @@ well before now, or that was built against unreleased Pydantic AI changes.
## Docs
- Capability README or root README is updated for user-facing behavior.
- Examples match declared extras.
Every released capability ships two hand-maintained docs that must stay in sync
with the code and with each other:
- the **README** next to the implementation (`pydantic_ai_harness/<capability>/README.md`,
or `pydantic_ai_harness/experimental/<capability>/README.md` for ACP), which
serves GitHub and PyPI, and
- the **unified doc** on the docs site, flat under `docs/<capability>.md`. The
sidebar is a flat list under "Pydantic AI Harness" -- no `capabilities/` or
`experimental/` subdirectories.
Checks:
- Both the README and the unified doc are updated for any user-facing change
(public class, params, defaults, tool names, extras, safety semantics). A
change reflected in only one of them is a defect, not a follow-up.
- The two do not contradict each other or the source on extras, option names,
defaults, or safety caveats.
- Every snippet in both docs is runnable: all imports present, class/param names
match the source, model ids unchanged from what the source uses. Imports use
the canonical module path (never `pydantic_ai_harness.experimental.*` for a
graduated capability).
- **Purpose-first lead.** The opening paragraph of each page and README states
what the capability is for and when to reach for it -- no internal hook or
class name (`before_model_request`, `after_tool_execute`, ...) before the
purpose. Mechanism belongs lower down.
- **Name matches the capability.** The doc filename, its `# H1`, and the
README's `# H1` all use the capability's descriptive name (e.g.
"Overflowing Tool Output", not "Overflow"; "Runtime Authoring", not
"Authoring").
- **Source link.** Each page links its source module
(`https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/<module>/`)
so a reading agent can verify behavior. Where the capability exposes a public
class, the page may also end with a `## API reference` section of
`::: pydantic_ai_harness...` autodoc blocks (auto-expanded from the docstring,
not hand-written).
- **Stability framing.** Graduated capabilities carry the soft note "The API may
change between releases..." mirrored from their README -- NOT a
`HarnessExperimentalWarning` block or "removed in any release" wording. ACP is
the only page that keeps an `!!! warning "Experimental"` (it may still be
removed).
- Links: harness-internal links are relative `.md`; Pydantic AI docs use
root-relative internal links `/ai/<section>/<page>/` (verify the route resolves
on the live `pydantic.dev/docs` site before using it).
- Docs explain composition constraints and safety implications.
- The PR links an issue.
The mechanical half of these checks (README present + linked, flat page present,
source link present, name matches, no experimental strings on non-ACP pages, no
hook name in the lead) is enforced by `tests/test_docs_parity.py`. The semantic
half (does the prose match the code, are snippets truly runnable) is what the
reviewer below is for.
This is the last documentation gate before merge. Run the `docs-parity-reviewer`
subagent (`.agents/agents/docs-parity-reviewer.md`) on the change as the final
review step; treat its blocking findings as merge blockers.
docs/acp.md
+246
View File
@@ -0,0 +1,246 @@
---
title: ACP (Agent Client Protocol)
description: Serve a Pydantic AI agent to editors and terminal UIs over the Agent Client Protocol -- streamed text, diff-rendered file edits, human-in-the-loop tool approval, and per-workspace sessions.
---
# ACP (Agent Client Protocol)
Editors like [Zed](https://zed.dev/docs/ai/external-agents) speak ACP: a stdio JSON-RPC protocol that lets a TUI or editor drive an external coding agent -- streaming its text, rendering its file edits as diffs, and prompting the user to approve sensitive tool calls. Reach for this capability when you want a Pydantic AI `Agent` to appear as a first-class agent inside one of those editors, without implementing the ACP server side yourself.
!!! warning "Experimental"
Unlike the graduated capabilities in these docs, ACP itself may still be **removed**, not just changed -- it lives under `pydantic_ai_harness.experimental` and may change or be removed in any release, without a deprecation period. Import it from the experimental path -- there is no top-level export:
```python
from pydantic_ai_harness.experimental.acp import run_acp_stdio_sync
```
Importing any experimental capability emits a `HarnessExperimentalWarning`. Silence **all** harness experimental warnings with a single filter (no per-capability lines needed):
```python
import warnings
from pydantic_ai_harness.experimental import HarnessExperimentalWarning
warnings.filterwarnings('ignore', category=HarnessExperimentalWarning)
```
## The problem
To plug a Pydantic AI agent into an ACP editor you would otherwise have to implement the ACP server side by hand -- chunking streamed text under the wire limit, rendering tool calls as diffs, mapping the protocol's permission requests onto the agent's tools, and managing per-workspace sessions.
## The solution
`run_acp_stdio` serves any Pydantic AI `Agent` as an ACP agent over stdin/stdout. The editor launches your script as a subprocess and talks to it; the adapter translates between ACP and the agent's run loop:
| ACP needs | The adapter provides |
|---|---|
| Streamed assistant text and reasoning | Agent text/thinking deltas, chunked under the wire limit |
| Rich tool calls (`kind`, file `locations`, diffs) | A presenter that recognizes `FileSystem`/`Shell` tool calls |
| Human-in-the-loop tool approval | Maps ACP permission requests to Pydantic AI's deferred-approval tools |
| Per-workspace sessions | A `session_config` hook to root tools at the client's working directory |
| Cancellation, multi-turn history, session close | Handled per session |
## Installation
```bash
uv add "pydantic-ai-harness[acp]"
```
This pulls in the [`agent-client-protocol`](https://pypi.org/project/agent-client-protocol/) SDK. The rest of the harness does not depend on it -- only `pydantic_ai_harness.experimental.acp` does.
## Quick start
Write a script that builds your agent and serves it:
```python
# my_acp_agent.py
from pydantic_ai import Agent
from pydantic_ai_harness.experimental.acp import run_acp_stdio_sync
def build_agent() -> Agent[None, str]:
return Agent('anthropic:claude-sonnet-4-6', instructions='You are a coding assistant.')
if __name__ == '__main__':
run_acp_stdio_sync(build_agent())
```
`run_acp_stdio_sync` blocks for the lifetime of the connection -- it is the `main()` of an agent the editor launches. Inside an existing event loop, use the async `run_acp_stdio` instead.
## Connecting from an editor
ACP clients launch the agent as a subprocess. In Zed, register it as an [external agent](https://zed.dev/docs/ai/external-agents) in `settings.json`:
```json
{
"agent_servers": {
"My Pydantic AI Agent": {
"type": "custom",
"command": "python",
"args": ["/absolute/path/to/my_acp_agent.py"],
"env": { "ANTHROPIC_API_KEY": "..." }
}
}
}
```
Any ACP-compatible client works the same way -- point it at `python my_acp_agent.py`.
The provider environment must be available to the launched subprocess. GUI editors and SDK-based test wrappers may not source your interactive shell startup files. If a real-model agent exits before initialize or fails provider auth, first verify that the command's process can see variables such as `ANTHROPIC_API_KEY`.
## Rooting tools at the workspace
A coding agent should read and write files in the workspace the editor opened, not wherever the subprocess started. ACP gives each session a working directory (`cwd`); a `session_config` factory turns that into per-session tools:
```python
from pydantic_ai import Agent
from pydantic_ai_harness.experimental.acp import AcpSession, AcpSessionConfig, run_acp_stdio_sync
from pydantic_ai_harness.filesystem import FileSystem
from pydantic_ai_harness.shell import Shell
agent = Agent('anthropic:claude-sonnet-4-6')
def session_config(session: AcpSession) -> AcpSessionConfig[None]:
# Root file and shell tools at the workspace the client opened.
return AcpSessionConfig(
deps=None,
toolsets=[
FileSystem[None](root_dir=session.cwd).get_toolset(),
Shell[None](cwd=session.cwd).get_toolset(),
],
)
if __name__ == '__main__':
run_acp_stdio_sync(agent, session_config=session_config)
```
The factory runs once per session with the client's `AcpSession` setup (its `cwd`, `mcp_servers`, and capabilities) and returns an `AcpSessionConfig` whose `deps` and `toolsets` apply to every run in that session. This is correct across multiple concurrent sessions in one process, where a single static `FileSystem` could not be.
## Editor-native filesystem and shell (optional)
The local [`FileSystem`](filesystem.md) and [`Shell`](shell.md) above operate on the agent process's own disk and subprocesses. An editor's source of truth is different: unsaved buffers, its own idea of the workspace layout, and -- for a remote or containerized editor -- the machine the code actually lives on. When the client advertises support, `acp_filesystem` and `acp_terminal` give the agent `read_file`/`write_file`/`run_command` tools that route through the client, so it acts where the user is:
```python
from pydantic_ai_harness.experimental.acp import AcpSession, AcpSessionConfig, acp_filesystem, acp_terminal
from pydantic_ai_harness.filesystem import FileSystem
from pydantic_ai_harness.shell import Shell
def session_config(session: AcpSession) -> AcpSessionConfig[None]:
# Use the editor's filesystem/terminal when offered; otherwise fall back to local.
fs = acp_filesystem(session) or FileSystem[None](root_dir=session.cwd).get_toolset()
shell = acp_terminal(session) or Shell[None](cwd=session.cwd).get_toolset()
return AcpSessionConfig(deps=None, toolsets=[fs, shell])
```
Each helper returns `None` when the client did not advertise the capability, so the `or` falls back to local and the agent works either way. The tool names match the local `FileSystem`/`Shell`, so rich rendering stays identical.
## Tool approval
Mark a tool to require approval and ACP relays the decision to the client, which shows the user an approve/reject prompt:
```python
@agent.tool_plain(requires_approval=True)
def delete_file(path: str) -> str:
...
```
The lifecycle the client sees is `pending` (awaiting approval) -> `in_progress` (granted, running) -> `completed`/`failed`, so an unapproved action is never shown as already running. "Always allow"/"always reject" decisions are remembered for the session, scoped by default to the exact call (tool name plus arguments) so approving one call never silently approves a different one. Pass `permission_policy` to widen or narrow that scope.
## Rich tool rendering
By default the adapter recognizes the harness `FileSystem` and `Shell` tool calls by name and annotates them with an ACP `kind` (`read`/`edit`/`search`/`execute`), the file `locations` they touch, and an inline diff for edits -- so the editor renders click-to-file links and diff views instead of opaque JSON. Pass `tool_presenter` to add rendering for your own tools (optionally with `chain_presenters` ahead of the default `default_coding_presenter`), or `lambda _call: None` to disable it.
## MCP servers
An ACP client may offer MCP servers during session setup. This adapter does not connect them itself; a `session_config` is the place to turn `session.mcp_servers` into Pydantic AI toolsets (for example with `pydantic_ai.mcp.MCPServerStdio`). If a client sends MCP servers and no `session_config` is installed to consume them, the session request is rejected rather than silently ignoring them. A spec-following client only sends HTTP/SSE MCP servers when the agent advertises support during `initialize`; when your `session_config` connects them, say so:
```python
from acp import schema
from pydantic_ai_harness.experimental.acp import PydanticAIACPAgent
PydanticAIACPAgent(
agent,
session_config=connect_mcp_servers,
mcp_capabilities=schema.McpCapabilities(http=True, sse=True),
)
```
## Prompt content types
The agent advertises which prompt content it accepts. The default is **text only**, so a client is not invited to send blocks a text model cannot handle. Enable the kinds your model supports:
```python
from acp import schema
run_acp_stdio_sync(agent, prompt_capabilities=schema.PromptCapabilities(image=True, embedded_context=True))
```
## Session persistence
Pass a `session_store` to let a client reopen a past conversation with `session/load`. Each committed turn is persisted as two parts -- the model's message history and the client-visible transcript -- and reopening restores the history into the agent and replays the transcript to the client, so its UI is rebuilt as the user last saw it. Without a store, `session/load` is advertised as unsupported.
```python
from pydantic_ai_harness.experimental.acp import InMemorySessionStore
run_acp_stdio_sync(agent, session_store=InMemorySessionStore())
```
`InMemorySessionStore` keeps sessions for the lifetime of the process. Implement the `SessionStore` protocol (`save`/`load` a `StoredSession`) over a file or database to make them survive a restart -- the stored values are Pydantic models, so they serialize with Pydantic. Session persistence is for *reopening a conversation*; it is orthogonal to per-run durability. To also make individual turns crash-resilient, add a [step-durability capability](step-persistence.md) -- each ACP turn is one agent run, so the two layers compose with no glue.
## Model selection
Pass `models` to advertise a stable ACP session config option named `model` (using Pydantic AI [model names](/ai/models/)). The first is each session's default. A selection is applied as a per-run override -- the shared agent is never mutated -- and is persisted with the session when a `session_store` is set.
```python
run_acp_stdio_sync(agent, models=['anthropic:claude-sonnet-4-6', 'anthropic:claude-opus-4-8', 'openai:gpt-4o'])
```
A model id is any string a Pydantic AI model accepts, so newer models not yet in `KnownModelName` work too. Pass `models='all'` to offer every model Pydantic AI knows. To advertise ids `infer_model` does not understand (OAuth or subscription models), pass `model_resolver` to map the selected id to a prebuilt `Model`.
## Cancellation and limitations
- **Cancellation.** `session/cancel` and `session/close` cancel the in-flight turn; close waits for it to unwind before returning. Cooperative async tools stop promptly. A synchronous tool already running in a worker thread cannot be force-stopped, so prefer async tools for cancellation-sensitive work.
- **Approval detection.** Tools that require approval are recognized when they live in a `FunctionToolset` (which the harness `FileSystem`/`Shell` and `@agent.tool` all use). A tool whose approval requirement is decided dynamically per call (by raising `ApprovalRequired` from its body) starts as `in_progress`, and any side effects it ran before raising have already happened -- use an `ApprovalRequiredToolset` for actions that must not partially execute before approval.
- **Overwrite diffs.** `write_file` renders an overwrite as if creating a new file, so the diff understates what it replaced.
- **Live terminal panes.** `acp_terminal` returns a command's captured output; it does not embed a live terminal pane in the tool call.
- **Images.** Prompt image blocks are off by default and must be enabled via `prompt_capabilities` with a model that accepts them.
- **Slash commands.** The adapter does not yet advertise any commands, so no slash commands appear in the client. Planned.
## API
```python {test="skip"}
run_acp_stdio( # async; serve until the client disconnects
agent,
*,
deps=None,
name=None, # advertised name; defaults to the agent's name
version='0.1.0',
session_config=None, # per-session deps/toolsets from the client's setup
permission_policy=None, # scope of remembered "always" approval decisions
prompt_capabilities=None, # defaults to text-only
mcp_capabilities=None, # MCP transports to advertise; needs a session_config to connect them
tool_presenter=None, # defaults to the FileSystem/Shell presenter
session_store=None, # enables session/load by persisting each session
models=None, # models offered as the `model` config option ('all' for every known model)
model_resolver=None, # maps an advertised model id to the Model used for the run
usage_limits=None, # per-run request/token ceilings
)
run_acp_stdio_sync(...) # synchronous wrapper, same arguments
PydanticAIACPAgent(agent, *, ...) # the ACP agent object, to embed in a custom server
```
The module also exports the session types (`AcpSession`, `AcpSessionConfig`, `McpServer`), the store types (`SessionStore`, `StoredSession`, `InMemorySessionStore`), the client toolsets (`AcpFileSystemToolset`, `AcpTerminalToolset`, `acp_filesystem`, `acp_terminal`), the permission types (`ToolCallPermission`, `default_permission_scope`), and the presentation helpers (`ToolCallPresentation`, `chain_presenters`, `default_coding_presenter`).
Source: [`pydantic_ai_harness/experimental/acp/`](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/experimental/acp/).
## Further reading
- [Agent Client Protocol](https://agentclientprotocol.com) -- protocol specification
- [Zed external agents](https://zed.dev/docs/ai/external-agents) -- editor-side configuration
- [Human-in-the-loop tool approval](/ai/tools-toolsets/deferred-tools/#human-in-the-loop-tool-approval) (Pydantic AI)
- [Pydantic AI capabilities](/ai/core-concepts/capabilities/)
docs/code-mode.md
+327
View File
@@ -0,0 +1,327 @@
---
title: Code Mode
description: Wrap an agent's tools into a single sandboxed run_code tool so the model orchestrates many calls in one Python program instead of many round-trips.
---
# Code Mode
`CodeMode` replaces individual tool calls with a single sandboxed Python execution environment. Instead of the model issuing one tool call per action, it writes a Python program that calls your tools as functions -- with loops, conditionals, variables, and `asyncio.gather` -- all inside a sandboxed [Monty](https://github.com/pydantic/monty) runtime.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/code_mode/)
## The problem
Standard tool calling costs one model round-trip per tool call. An agent that needs to fetch 10 items and process each one makes 11+ model calls -- slow, expensive, and heavy on context. The conversation history grows with every intermediate result, and everything runs sequentially unless the model deliberately batches parallel tool calls.
## The solution
`CodeMode` wraps the agent's tools into a single `run_code` tool. The model writes one Python snippet that orchestrates many tool calls locally: fan them out with `asyncio.gather`, filter and transform results in plain Python, and return only what matters.
| Standard tool calling | Code mode |
|---|---|
| 1 model call per tool | 1 model call for N tools |
| Sequential by default | Parallel via `asyncio.gather` |
| No local computation | Filter, transform, aggregate in code |
| Large conversation history | Compact -- fewer messages |
## Installation
Code mode requires the Monty sandbox, available via the `codemode` extra (the `code-mode` extra is an equivalent alias):
```bash
uv add "pydantic-ai-harness[codemode]"
```
## Usage
Construct an `Agent` with `CodeMode()` in its `capabilities`, then register tools as usual. Every tool becomes callable from inside `run_code`:
```python
from pydantic_ai import Agent
from pydantic_ai_harness import CodeMode
agent = Agent('anthropic:claude-sonnet-4-6', capabilities=[CodeMode()])
@agent.tool_plain
def get_weather(city: str) -> dict:
"""Get current weather for a city."""
return {'city': city, 'temp_f': 72, 'condition': 'sunny'}
@agent.tool_plain
def convert_temp(fahrenheit: float) -> float:
"""Convert Fahrenheit to Celsius."""
return round((fahrenheit - 32) * 5 / 9, 1)
result = agent.run_sync("What's the weather in Paris and Tokyo, in Celsius?")
print(result.output)
```
Inside a single `run_code` call, the model writes code like the following (illustrative -- the exact code the model emits will vary):
```python
import asyncio
paris, tokyo = await asyncio.gather(
get_weather(city='Paris'),
get_weather(city='Tokyo'),
)
paris_c = await convert_temp(fahrenheit=paris['temp_f'])
tokyo_c = await convert_temp(fahrenheit=tokyo['temp_f'])
{'paris': paris_c, 'tokyo': tokyo_c}
```
Both weather lookups run in parallel, the conversions run locally, and the whole thing collapses into one model round-trip instead of five.
## Selective tool sandboxing
By default, `CodeMode(tools='all')` sandboxes every tool. The `tools` field is a Pydantic AI `ToolSelector`, so you can control precisely which tools go through the sandbox. Tools that match the selector become callables inside `run_code`; non-matching tools stay visible to the model as regular tool calls.
```python
from pydantic_ai_harness import CodeMode
# By name -- only these tools are available inside run_code
CodeMode(tools=['search', 'fetch'])
# By predicate -- (ctx, tool_def) -> bool | Awaitable[bool]
CodeMode(tools=lambda ctx, td: td.name != 'dangerous_tool')
# By metadata -- combine with SetToolMetadata or a toolset's .with_metadata()
CodeMode(tools={'code_mode': True})
```
### Metadata-based selection
Use metadata when the decision should travel with a tool or toolset, rather than with one `CodeMode` instance. This suits shared toolsets: the toolset author tags the tools that are safe and useful to call from generated code, and each agent opts into that tag with `CodeMode(tools={...})`.
`CodeMode(tools={'code_mode': True})` uses the standard Pydantic AI [`ToolSelector`](/ai/api/pydantic-ai/tools/) metadata form. A tool is sandboxed when its `ToolDefinition.metadata` contains all of the selector's key-value pairs. Extra metadata on the tool is fine, and nested dictionaries are matched by deep inclusion.
The common pattern is to tag an entire toolset with `.with_metadata(...)`:
```python
from pydantic_ai import Agent
from pydantic_ai.toolsets import FunctionToolset
from pydantic_ai_harness import CodeMode
def search(query: str) -> str:
"""Search the web."""
return f'results for {query}'
def fetch(url: str) -> str:
"""Fetch a URL."""
return f'contents of {url}'
search_tools = FunctionToolset(tools=[search, fetch]).with_metadata(code_mode=True)
agent = Agent(
'anthropic:claude-sonnet-4-6',
toolsets=[search_tools],
capabilities=[CodeMode(tools={'code_mode': True})],
)
```
Here `search` and `fetch` are removed from the model-facing tool list and become callable functions inside `run_code`. Tools without `metadata['code_mode'] == True` stay visible as regular tool calls.
## Tool Search interaction
When you mark tools or whole toolsets `defer_loading=True` ([Tool Search](/ai/tools-toolsets/tools-advanced/#tool-search)), `CodeMode` keeps them out of `run_code` while they're undiscovered -- they pass straight through, so Tool Search drives them as usual (sent on the wire with `defer_loading` on providers with native tool search; otherwise dropped until discovered, with a `search_tools` tool alongside `run_code`). Once the model discovers a tool it comes back with `defer_loading=False`, and from then on `CodeMode` folds it into `run_code` like any other tool, so it's callable from generated code.
That fold-in grows `run_code`'s description, which invalidates the prompt-cache prefix once at the moment of discovery (turns with no discovery stay cache-warm). Two ways to avoid the bust:
- Pass `dynamic_catalog=True` to keep `run_code`'s description static across discoveries. The catalog of sandboxed-tool signatures moves into the agent instructions (as a dynamic [`InstructionPart`](/ai/api/pydantic-ai/messages/#pydantic_ai.messages.InstructionPart)) and newly-discovered tools are announced via [`ctx.enqueue`](/ai/api/pydantic-ai/tools/#pydantic_ai.tools.RunContext.enqueue) instead of by rebuilding the description:
```python
from pydantic_ai_harness import CodeMode
CodeMode(dynamic_catalog=True)
```
This pays off when paired with Tool Search: the tool-definitions block stays byte-stable so the prefix cache survives discoveries, at the cost of a larger (but cache-friendly) system prompt. With a fixed toolset and no Tool Search, the default keeps the system prompt shorter and is the better choice.
- To instead keep a Tool Search corpus fully native -- never folded into `run_code`, but not callable from inside it -- exclude it with a `tools` selector; corpus members carry `with_native` set to the managing native tool:
```python
from pydantic_ai_harness import CodeMode
CodeMode(tools=lambda ctx, td: td.with_native is None)
```
## Return values
The last expression in the snippet is automatically captured as the return value -- the model does not need to `print()`. Reserve `print()` for supplementary logging: printed text is surfaced separately, wrapped alongside the last-expression result.
| Scenario | Return |
|---|---|
| No print output | Last expression value |
| With print output | `{'output': '<printed text>', 'result': <last expression>}` |
| Multimodal content (e.g. images) | Returned natively for model processing |
## REPL state
State persists between `run_code` calls within the same agent run -- variables, imports, and function definitions carry over. Pass `restart: true` in the tool call to reset state.
## Observability
Nested tool calls inside `run_code` produce their own spans when instrumented with [Logfire](https://pydantic.dev/logfire) or any OpenTelemetry backend -- the easiest way to understand what code mode actually did, since each `run_code` span fans out into the tool calls the model issued from inside the sandbox. See the [Pydantic AI Logfire docs](/ai/integrations/logfire/) for setup.
The `run_code` tool return also carries metadata with every nested call, keyed by call id:
```python
from pydantic_ai import Agent
from pydantic_ai.messages import ToolReturnPart
from pydantic_ai_harness import CodeMode
agent = Agent('anthropic:claude-sonnet-4-6', capabilities=[CodeMode()])
@agent.tool_plain
def get_weather(city: str) -> dict:
"""Get current weather for a city."""
return {'city': city, 'temp_f': 72}
result = agent.run_sync("What's the weather in Paris?")
for msg in result.all_messages():
for part in msg.parts:
if isinstance(part, ToolReturnPart) and part.tool_name == 'run_code':
metadata = part.metadata or {}
tool_calls = metadata['tool_calls'] # dict[str, ToolCallPart]
tool_returns = metadata['tool_returns'] # dict[str, ToolReturnPart]
```
## In practice
A representative run wires `CodeMode` up against an MCP server and a web search and asks it to find the most-discussed Hacker News story across three feeds, pull the comment thread and the submitter's profile, and search the web for follow-up coverage. `CodeMode` collapses that into two `run_code` calls: the first fetches all three feeds in parallel via `asyncio.gather`, dedupes by id, filters by score, and ranks by comment count -- in plain Python; the second batches the three follow-up calls (`hn_get_thread`, `hn_get_user`, `duckduckgo_search`) together.
[![CodeMode's first run_code: parallel asyncio.gather over three HN feeds, then a dedupe and a score filter](images/code-mode-trace.png)](https://logfire-us.pydantic.dev/public-trace/84bcf123-2106-49da-9f6f-5c26395339bb?spanId=7650806a0785b946)
**[See the full Logfire trace ->](https://logfire-us.pydantic.dev/public-trace/84bcf123-2106-49da-9f6f-5c26395339bb?spanId=7650806a0785b946)** Each `run_code` span fans out into the tool calls the model issued from inside the sandbox.
## Filesystem and OS access
Sandboxed code runs with no access to the host's files, environment, or clock. Two parameters grant it access -- reach for them only when the agent's task genuinely needs the host.
Both parameters are fixed when the capability is built, so construct `CodeMode` per request to scope host access to that request.
### `mount` -- share host directories
Reach for `mount` when the agent works with real files: analyzing a dataset you've dropped in a folder and writing a report back, editing a checkout, or processing a batch of documents. Sandboxed `pathlib` code reads and writes under the mounted path. (For environment variables or the clock, use `os_access` instead.)
```python
from pydantic_ai import Agent
from pydantic_monty import MountDir
from pydantic_ai_harness import CodeMode
# The agent can read /work/data.csv and write /work/summary.md back to the host:
agent = Agent(
'anthropic:claude-sonnet-4-6',
capabilities=[CodeMode(mount=MountDir('/work', '/tmp/agent-workspace', mode='read-write'))],
)
```
A `MountDir` defaults to copy-on-write `mode='overlay'`: the sandbox reads host files and sees its own writes, but those writes do **not** reach the host. Pass `mode='read-write'` to persist them, or `mode='read-only'` to forbid writes. `mount` also accepts a list of `MountDir` for multiple mount points.
### `os_access` -- answer the sandbox's OS calls yourself
Reach for `os_access` when the agent needs environment variables, the current date and time, or filesystem behavior you control. Hand it a ready-made OS implementation (`AbstractOS`), or a callback that decides each call -- so you can inject just the secrets it needs, pin "now" for reproducible runs, or route file access to your own store.
```python
from pydantic_ai import Agent
from pydantic_monty import OSAccess
from pydantic_ai_harness import CodeMode
# Give the agent a fixed set of environment values:
agent = Agent(
'anthropic:claude-sonnet-4-6',
capabilities=[CodeMode(os_access=OSAccess(environ={'API_BASE': 'https://api.example.com'}))],
)
```
A callback receives each OS call and decides its fate:
```python
from pydantic_ai import Agent
from pydantic_monty import NOT_HANDLED
from pydantic_ai_harness import CodeMode
allowed_env = {'API_KEY': 'sk-...'}
def my_os(fn, args, kwargs):
if fn == 'os.getenv':
# Answer the call: allow-listed keys resolve, every other key reads back
# as None -- absent, exactly like a real unset variable.
return allowed_env.get(args[0])
# Refuse everything else: NOT_HANDLED makes the call fail in the sandbox.
return NOT_HANDLED
agent = Agent('anthropic:claude-sonnet-4-6', capabilities=[CodeMode(os_access=my_os)])
```
Your callback's return value decides the call's fate, and the two outcomes are easy to confuse:
- **Return any value** -- including `None`, `''`, or `0` -- and that becomes the result the sandbox sees. `os.getenv` returning `None` looks exactly like a normal unset variable, so the agent's code keeps running. This is how you *hide* something: answer with an empty value.
- **Return `NOT_HANDLED`** and the call is treated as unsupported: it raises inside the sandbox and the model gets a retry. This *refuses* a capability outright -- use it to block, not to say "no value". Returning `NOT_HANDLED` for a key the agent reasonably expects will burn retries.
!!! warning "Both expose the real host to model-written code"
`mount` and `os_access` hand model-generated Python real access to your filesystem and environment. Grant only what the task needs, and prefer constructing `CodeMode` per request so the granted access is scoped to that request.
!!! note "Monty-specific types"
These hooks use Monty's `AbstractOS`/`MountDir` types from `pydantic_monty`.
## Sandbox restrictions
Code runs inside [Monty](https://github.com/pydantic/monty), a sandboxed Python subset. Key restrictions:
- No class definitions.
- No third-party imports. Allowed stdlib modules: `sys`, `typing`, `asyncio`, `math`, `json`, `re`, `datetime`, `os`, `pathlib` (each must be imported before use).
- No wall-clock or timing primitives by default: `asyncio.sleep`, `datetime.datetime.now()`, `datetime.date.today()`, and the `time` module. `datetime.datetime.now()` / `datetime.date.today()` become available with an `os_access` handler (above); `asyncio.sleep` and `time` never do.
- No `import *`.
- Filesystem I/O needs an `os_access` handler or a `mount`; `os.getenv` / `os.environ` need an `os_access` handler.
- Tools requiring approval or with deferred (`CallDeferred`) execution are sandboxed like any other tool; without a `HandleDeferredToolCalls` (or equivalent) capability on the agent to resolve them inline, calling one from `run_code` raises an error that surfaces to the model as a retry.
## Agent spec (YAML/JSON)
`CodeMode` works with Pydantic AI's [agent spec](/ai/core-concepts/agent-spec/) feature for defining agents in YAML or JSON:
```yaml
# agent.yaml
model: anthropic:claude-sonnet-4-6
capabilities:
- CodeMode: {}
```
```python
from pydantic_ai import Agent
from pydantic_ai_harness import CodeMode
agent = Agent.from_file('agent.yaml', custom_capability_types=[CodeMode])
result = agent.run_sync('...')
print(result.output)
```
Pass `custom_capability_types` so the spec loader knows how to instantiate `CodeMode`. Arguments can be passed in the YAML too:
```yaml
capabilities:
- CodeMode:
tools: ['search', 'fetch']
max_retries: 5
```
## Further reading
- [Tool use via code](https://www.anthropic.com/engineering/code-execution-with-mcp) (Anthropic)
- [Code mode in production](https://blog.cloudflare.com/code-mode/) (Cloudflare)
- [Pydantic AI capabilities](/ai/core-concepts/capabilities/)
## API reference
::: pydantic_ai_harness.CodeMode
docs/compaction.md
+264
View File
@@ -0,0 +1,264 @@
---
title: Compaction
description: A menu of strategies -- clear, dedupe, trim, or summarize -- for keeping an agent's conversation history within the model's context window.
---
# Compaction
Compaction is a menu of strategies for keeping an agent's conversation history within a model's context window. Each strategy is a Pydantic AI `Capability` that edits the message history just before each request goes out. The edits **persist** into the run's message history, so a trim, clear, or summary carries forward to later steps -- it is not recomputed from the full history every turn.
All strategies preserve tool-call / tool-return **pairing**. Core does not validate this, and a provider rejects an orphaned pair, so the pairing guarantee is what makes these safe to drop into an agent. The zero-LLM strategies never call a model; only `SummarizingCompaction` (and `TieredCompaction` when it escalates that far) spends tokens.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/compaction/)
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
## The problem
An agent that runs for many turns accumulates history: tool outputs, file reads, model reasoning, repeated content. Left unchecked, that history outgrows the model's context window and the next request fails. Compaction keeps the history bounded, and the right strategy depends on where the bloat lives and how much you can afford to spend reclaiming it.
## The menu
| Capability | Cost | What it does | Reach for it when |
|---|---|---|---|
| `ClampOversizedMessages` | zero-LLM | Head/tail-truncates a single oversized part (response text, tool-call args) | One runaway generation blew past the context cap and no other strategy can reach it |
| `SlidingWindow` | zero-LLM | Drops the oldest whole messages down to a tail | You only need the recent turns and can discard old context entirely |
| `ClearToolResults` | zero-LLM | Blanks the content of old tool *results* in place, keeping the last `keep_pairs` | Tool outputs dominate context and can be re-fetched on demand (the cheap first tier) |
| `DeduplicateFileReads` | zero-LLM | Blanks every file read superseded by a newer read of the same file | The agent re-reads files and only the latest version matters |
| `SummarizingCompaction` | one LLM call | Summarizes older messages into a structured summary, keeping the recent tail | Old context still matters but must be compressed; use behind the cheap tiers |
| `TieredCompaction` | escalates | Runs cheap passes first, summarizes only if still over `target_tokens` | You want a sensible default: spend the expensive summary only when needed |
| `LimitWarner` | zero-LLM | Injects an URGENT/CRITICAL warning as limits approach | You want the agent to wrap up rather than have its history rewritten |
## Triggers
Every size-based strategy triggers on `max_messages` and/or `max_tokens` (estimated). Token counts use a ~4-chars-per-token heuristic by default; pass a `tokenizer` callable (for example `tiktoken`) for accuracy. `DeduplicateFileReads` runs on every request when no trigger is set (it is cheap and near-lossless). `TieredCompaction` triggers and stops on a single `target_tokens` budget. `ClampOversizedMessages` triggers per *part* (`max_part_tokens` / `max_part_chars`), not on the whole history -- the failure it targets is one oversized part, not a large total.
## The recommended default: `TieredCompaction`
The field consensus (Anthropic, OpenCode, Letta) is to clear and dedupe first, and summarize only when that is not enough. Summarization turns input tokens into output tokens, which are billed at a premium and generated serially, so it is genuinely expensive. The zero-LLM strategies touch only the cheaper input side.
`TieredCompaction` encodes that escalation: it runs each tier in order, re-measures the token count after each, and stops as soon as the conversation fits `target_tokens`. Order the tiers cheap-to-expensive so the expensive summarization tier is only reached when the cheap passes cannot reclaim enough.
```python
from pydantic_ai import Agent
from pydantic_ai_harness.compaction import (
ClearToolResults,
DeduplicateFileReads,
SummarizingCompaction,
TieredCompaction,
)
from pydantic_ai.messages import ToolCallPart
def my_file_key(call: ToolCallPart) -> str | None:
if call.tool_name != 'read_file':
return None
return call.args_as_dict().get('path')
agent = Agent(
'openai:gpt-4o',
capabilities=[
TieredCompaction(
tiers=[
DeduplicateFileReads(file_key=my_file_key),
ClearToolResults(max_tokens=1, keep_pairs=3),
SummarizingCompaction(max_messages=1, keep_messages=20), # model inherits the run's
],
target_tokens=120_000,
)
],
)
```
A tier inside `TieredCompaction` is driven directly by the orchestrator, which re-measures after each tier and stops once under `target_tokens`. A tier's own `max_*` trigger is therefore irrelevant when it runs inside `TieredCompaction` -- set it to anything valid (for example `ClearToolResults(max_tokens=1)`). Any object with `async def compact(messages, ctx) -> list[ModelMessage]` (the `CompactionStrategy` protocol) can be a tier, so you can plug in your own.
## `ClampOversizedMessages`: surviving a runaway generation
A single model response of repeated whitespace, or a single tool call with a giant payload, can produce one part so large the *next* request exceeds the provider's context cap. None of the other strategies can reach it: `SlidingWindow` drops the oldest messages but the offender is the newest; `ClearToolResults` only touches tool *results*; `LimitWarner` never edits history; and feeding the history to `SummarizingCompaction` hits the same cap.
`ClampOversizedMessages` truncates the offending part in place, keeping a head slice and a tail slice with a `[clamped: removed N of M characters]` marker between them. Degenerate generations are low-entropy repetition, so a head/tail slice loses little.
```python
from pydantic_ai import Agent
from pydantic_ai_harness.compaction import ClampOversizedMessages
agent = Agent(
'openai:gpt-4o',
capabilities=[
ClampOversizedMessages(max_part_tokens=50_000, keep_head_chars=2_000, keep_tail_chars=2_000)
],
)
```
A part is clamped only when it is oversized *and* the clamp actually shrinks it, so keep `keep_head_chars + keep_tail_chars` well below your per-part threshold.
It clamps two kinds of part inside each `ModelResponse`:
- **Response text** (`TextPart`) -- the critical case, a runaway model-response text part.
- **Tool-call args** (`ToolCallPart`), when `clamp_tool_call_args=True` (the default) -- the same failure shape for a giant payload (for example a runaway `write_plan`). The args are replaced with a small JSON object `{"_clamped": "<head>...<tail>"}` so they stay valid function arguments; the original call already executed, so this only shrinks the history copy. Set `clamp_tool_call_args=False` to clamp response text only.
Request-side parts (user prompts, tool *returns*, system prompts) are deliberately out of scope: user input should not be silently rewritten, and oversized tool returns are the job of `ClearToolResults`.
Use it as the first tier of `TieredCompaction`, before `ClearToolResults`:
```python
from pydantic_ai_harness.compaction import (
ClampOversizedMessages,
ClearToolResults,
TieredCompaction,
)
TieredCompaction(
tiers=[
ClampOversizedMessages(max_part_tokens=50_000),
ClearToolResults(max_tokens=1, keep_pairs=3),
],
target_tokens=120_000,
)
```
## `ClearToolResults`: the cheap first tier
Tool outputs typically dominate an agent's context, and the agent can usually re-run a tool if it needs the data again. `ClearToolResults` replaces the content of the oldest tool *results* with a short placeholder while keeping the most recent `keep_pairs` tool-call / tool-return pairs intact. The tool calls stay paired with their now-blanked results, so the history stays valid.
```python
from pydantic_ai import Agent
from pydantic_ai_harness.compaction import ClearToolResults
agent = Agent(
'openai:gpt-4o',
capabilities=[ClearToolResults(max_tokens=100_000, keep_pairs=3)],
)
```
Set `clear_tool_inputs=True` to also blank the arguments of the cleared calls, and `exclude_tools` to a set of tool names whose results are never cleared.
## `DeduplicateFileReads`: drop superseded reads
When the same file is read more than once, only the latest read keeps its content; earlier reads are blanked with a placeholder, with pairing preserved.
There is no default `file_key`: identifying a file read is agent-specific, and a wrong guess would drop live data. Supply a callable mapping a `ToolCallPart` to a stable file key, or `None` when the call is not a file read:
```python
from pydantic_ai import Agent
from pydantic_ai.messages import ToolCallPart
from pydantic_ai_harness.compaction import DeduplicateFileReads
def file_key(call: ToolCallPart) -> str | None:
if call.tool_name != 'read_file':
return None
return call.args_as_dict().get('path')
agent = Agent('openai:gpt-4o', capabilities=[DeduplicateFileReads(file_key=file_key)])
```
With no `max_messages` or `max_tokens` trigger set, `DeduplicateFileReads` runs on every request. It is cheap and near-lossless, so that default is usually what you want.
## `SlidingWindow`: keep only the recent tail
When the conversation exceeds the configured threshold, `SlidingWindow` discards the oldest whole messages down to a tail, preserving tool-call / tool-return pairs. Reach for it when you only need the recent turns and can discard old context entirely.
```python
from pydantic_ai import Agent
from pydantic_ai_harness.compaction import SlidingWindow
agent = Agent(
'openai:gpt-4o',
capabilities=[SlidingWindow(max_messages=80, keep_messages=40)],
)
```
By default `preserve_first_user_message=True` keeps the first user turn (in addition to system prompts) even when it falls outside the window, so the agent does not lose the original task. Pass `keep_tokens` instead of `keep_messages` to trim to a token budget rather than a message count.
## `SummarizingCompaction`: compress, do not discard
When old context still matters but must be compressed, `SummarizingCompaction` summarizes the older messages with a dedicated model call and replaces them with a single structured summary, preserving the recent tail and tool-call integrity. It is the expensive tier, so it is best used behind the cheaper passes (see `TieredCompaction`).
```python
from pydantic_ai import Agent
from pydantic_ai_harness.compaction import SummarizingCompaction
agent = Agent(
'openai:gpt-4o',
capabilities=[
SummarizingCompaction(
model='openai:gpt-4o-mini',
max_messages=60,
keep_messages=20,
)
],
)
```
`model` accepts a model name or a `Model`; when left `None` it inherits the running agent's model. No token caps are imposed on the summary call. By default `incremental=True` extends any existing summary from a prior compaction rather than regenerating it from scratch.
### Usage accounting
The summary call is a real request to the model, so its full usage -- tokens **and** the request itself -- is folded into the run's `ctx.usage`. This is deliberate: it keeps cost honest, keeps the request count consistent (a model request that did not count as one would be the surprise), and lets a `UsageLimits` request limit catch a runaway compaction. A run-request or iteration limiter will therefore see compaction calls among its requests.
## `LimitWarner`: warn instead of rewrite
`LimitWarner` never edits history. As the run approaches a configured limit, it injects an URGENT (then CRITICAL) warning as a trailing user turn, so the model wraps up rather than having its context rewritten under it. Models tend to pay more attention to user messages than system messages, which is why the warning is a user turn. Previous warnings from this capability are stripped before deciding whether to inject a new one.
```python
from pydantic_ai import Agent
from pydantic_ai_harness.compaction import LimitWarner
agent = Agent(
'openai:gpt-4o',
capabilities=[
LimitWarner(
max_iterations=40,
max_context_tokens=100_000,
)
],
)
```
Warnings begin at `warning_threshold` (default `0.7`, a fraction of the limit) and become CRITICAL for iterations once the remaining request count drops to `critical_remaining_iterations` (default `3`). It watches three kinds of limit -- `max_iterations`, `max_context_tokens`, and `max_total_tokens` -- and by default warns on whichever are configured; narrow that with `warn_on`.
## Cache tradeoff
Clearing, deduplicating, clamping, and summarizing all rewrite message content, which invalidates the provider's prompt cache from the edit point onward -- the next request pays a cache-write. For `ClearToolResults`, use `min_clear_tokens` to skip clearing that reclaims too little to be worth busting the cache. For `ClampOversizedMessages` the cache bust is unavoidable, because the alternative is a failed request.
## Tracing
When core instrumentation is active (the `Instrumentation` capability, `agent.instrument`, or `Agent.instrument_all()`), each strategy emits a `compact_messages` span on the run's tracer the moment it actually compacts -- that is, in `before_model_request`, once the strategy's threshold is exceeded (`ClampOversizedMessages` emits only when a part is actually clamped). `TieredCompaction` emits a single span for the whole escalation rather than one per tier, because it drives each tier's `compact` directly. Without instrumentation the tracer is a no-op, so the span adds no overhead.
The span name is the static `compact_messages`; the strategy is an attribute, not part of the name, to keep span cardinality low. Attributes:
| Attribute | Type | Meaning |
|---|---|---|
| `gen_ai.conversation.compacted` | bool | Always `true`; the OpenTelemetry GenAI convention's flag for a compacted context |
| `compaction.strategy` | str | Strategy class name (for example `SlidingWindow`, `SummarizingCompaction`) |
| `compaction.messages_before` | int | Message count before compaction |
| `compaction.messages_after` | int | Message count after compaction |
| `compaction.tokens_before` | int | Estimated token count before compaction |
| `compaction.tokens_after` | int | Estimated token count after compaction |
`gen_ai.conversation.compacted` is the GenAI semantic convention's flag; the rest is harness-specific. Token counts use the strategy's `tokenizer` when set, otherwise the ~4-chars-per-token heuristic. Raw message content is not recorded.
## Out of scope
These strategies compress or drop context *inside* the window. Moving large tool outputs *out* of the window -- overflowing them to a file the agent (or a subagent) can query on demand -- is a separate capability ([overflowing tool output](overflowing-tool-output.md)), not lossy truncation. Prefer it over capping individual tool outputs.
## API reference
The recommended default is `TieredCompaction`; the other strategies below can be used standalone or plugged in as its tiers.
::: pydantic_ai_harness.compaction.TieredCompaction
::: pydantic_ai_harness.compaction.ClampOversizedMessages
::: pydantic_ai_harness.compaction.ClearToolResults
::: pydantic_ai_harness.compaction.DeduplicateFileReads
::: pydantic_ai_harness.compaction.SlidingWindow
::: pydantic_ai_harness.compaction.SummarizingCompaction
::: pydantic_ai_harness.compaction.LimitWarner
docs/context.md
+117
View File
@@ -0,0 +1,117 @@
---
title: Context
description: Discover and load a repo's accumulated coding-assistant context engineering -- instruction files, skills, sub-agents, and hooks.
---
# Context
`RepoContext` discovers and loads a repo's accumulated coding-assistant context engineering (CE): the instruction files (`CLAUDE.md`/`AGENTS.md`) scattered across the tree and the assets under `.claude`/`.agents`/`.codex`/`.grok` (skills, sub-agents, hooks).
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/context/)
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
## The problem
A repo accumulates CE for whatever coding assistant worked in it: instruction files (`CLAUDE.md`/`AGENTS.md`) scattered across the tree, and assets under `.claude`/`.agents`/`.codex`/`.grok` (skills, sub-agents, hooks). An agent that loads only the top-level instruction file misses the ancestor context and has no idea the rest of the setup exists, so it can neither honor it nor translate it.
## The solution
`RepoContext` bundles three strategies, each independently toggleable. Construct it with `RepoContext(...)` in an `Agent`'s `capabilities`, anchored at the deepest directory the agent works in:
```python
from pathlib import Path
from pydantic_ai import Agent
from pydantic_ai_harness.context import RepoContext
agent = Agent(
'anthropic:claude-sonnet-4-6',
capabilities=[RepoContext(workspace_dir=Path('.'), home_dir=Path.home())],
)
result = agent.run_sync('Summarize the coding-assistant setup in this repo.')
print(result.output)
```
### 1. Walk-up instruction autoload (on by default)
Loads `CLAUDE.md`/`AGENTS.md` from `workspace_dir` and every ancestor up to `home_dir` (inclusive). Precedence is ancestor-first, workspace-last: broadest context first, most specific last. Files are deduped by resolved real path and by content hash, so a symlinked `AGENTS.md -> CLAUDE.md` or two ancestors sharing identical content load once.
When `home_dir` is `None` (the default), only `workspace_dir` is scanned -- no walk-up. Pass `home_dir=Path.home()` to walk up to your home directory.
### 2. Asset inventory (on by default)
Exposes one tool, `inventory_agent_context()`, that reports where the repo's CE assets live -- the `.claude`/`.agents`/`.codex`/`.grok` roots and, within each, the `skills/` (`SKILL.md`), `agents/` (`.md`), and `settings.json` (hooks) it contains. It returns a structured `AgentContextInventory`; it locates assets and does not parse them, leaving translation to the orchestrator.
Rename the tool with `inventory_tool_name`, or scope which roots it scans with `asset_roots`.
### 3. Nested-on-traversal (off by default)
When the model lists or reads a directory, surface that directory's `CLAUDE.md`/`AGENTS.md`. This couples to the host's list/read tools, so it is opt-in and configurable:
```python
from pathlib import Path
from pydantic_ai import Agent
from pydantic_ai_harness import FileSystem
from pydantic_ai_harness.context import RepoContext
agent = Agent(
'anthropic:claude-sonnet-4-6',
capabilities=[
FileSystem(root_dir='.'),
RepoContext(
workspace_dir=Path('.'),
nested_traversal=True,
traversal_tool_names=frozenset({'list_directory', 'read_file'}), # the FileSystem tool names to hook
traversal_path_arg='path', # the path arg key
nested_inject='pointer', # or 'contents'
)
],
)
```
`nested_inject='pointer'` (default) appends a one-line note pointing at the file; `'contents'` inlines the file body. Each directory is surfaced at most once per run.
## Cache cost
Injecting file contents into the system prompt costs prompt-cache stability: a changed prefix re-bills the whole cached region. `RepoContext` keeps the two cache-relevant paths separate:
- Strategy 1 reads its files once at run start and injects them as static system instructions, so the cached prefix stays byte-identical across turns.
- Strategy 3 is volatile (it depends on which directory was just touched), so its note is appended to the tool result in the message tail -- never to the system prompt -- and cannot invalidate the cached prefix.
## Configuration
```python
RepoContext(
workspace_dir, # Path -- the deepest dir the agent works in (required)
home_dir=None, # Path | None -- shallowest dir to stop walk-up at, inclusive
filenames=('CLAUDE.md', 'AGENTS.md'),
autoload_instructions=True, # Strategy 1
expose_inventory_tool=True, # Strategy 2
inventory_tool_name='inventory_agent_context',
nested_traversal=False, # Strategy 3
nested_inject='pointer', # 'pointer' | 'contents'
traversal_tool_names=frozenset({'list_directory', 'read_file'}),
traversal_path_arg='path',
asset_roots=('.claude', '.agents', '.codex', '.grok'),
)
```
## Scope
`RepoContext` locates and loads CE; it does not parse skill/sub-agent frontmatter or hook bodies, and it does not rewrite or translate assets. Strategy 1 reads its files once per run, so mid-run edits to those files are not reloaded.
## Further reading
- [Pydantic AI capabilities](/ai/core-concepts/capabilities/)
- [Pydantic AI hooks](/ai/core-concepts/hooks/)
## API reference
::: pydantic_ai_harness.context.RepoContext
::: pydantic_ai_harness.context.AgentContextInventory
::: pydantic_ai_harness.context.AssetRoot
docs/dynamic-workflow.md
+265
View File
@@ -0,0 +1,265 @@
---
title: Dynamic Workflow
description: Let an orchestrator agent coordinate a catalog of sub-agents by writing one sandboxed Python script -- fan-out, chaining, voting, and retry loops in a single tool call.
---
# Dynamic Workflow
`DynamicWorkflow` is for the case where the coordination *between* sub-agents is the actual work. Say you have a few specialists -- one reviews code, one summarizes findings, one writes the final note. Each is easy to call on its own; the hard part is the choreography: review three files at once, keep only the reports that found something, summarize those, and hand the summary to the writer. Reach for this capability when that orchestration involves fan-out, chaining, voting, or retry loops that you do not want to run one model turn at a time, with every intermediate result flowing back through the orchestrator's context.
!!! note "Import path"
Import this capability from its submodule -- there is no top-level `pydantic_ai_harness` re-export:
```python
from pydantic_ai_harness.dynamic_workflow import DynamicWorkflow
```
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
## The idea
The usual way to coordinate sub-agents is one tool call per step. The agent calls the reviewer and waits, reads the result, calls the reviewer again, waits again, and so on. Every intermediate result travels back into the agent's context, and every step that depends on the previous one is a separate model turn.
`DynamicWorkflow` takes a different route. You hand it a catalog of named sub-agents, and it gives the model a single tool, `run_workflow`. Inside that tool the model writes ordinary Python: each of your sub-agents is an `async` function it can call, loop over, and combine. The script runs to completion in one tool call, and only its final value comes back to the model. The choreography moves out of the conversation and into code.
If you have met [Code Mode](code-mode.md), this will feel familiar -- the same [Monty](https://github.com/pydantic/monty) sandbox and the same idea: write a script instead of many tool calls. The difference is what the script gets to call. In Code Mode it calls the agent's own tools; here it calls whole sub-agents.
## How this relates to Subagents
The harness has two delegation capabilities. They trade in the same currency -- named, isolated sub-agent runs -- but at different altitudes:
- [`SubAgents`](subagents.md) exposes one `delegate_task(agent_name, task)` tool. Each delegation is its own tool call and its own model turn. It is the right fit when delegations are occasional, or when each result needs the parent's judgment before the next one.
- `DynamicWorkflow` moves the choreography into a script. Fan-out, chaining, voting, and retry loops all run inside one tool call, and intermediate results never enter the parent's context.
Start with `SubAgents` if you are not sure. A `delegate_task` orchestrator converts to a workflow catalog without changing the sub-agents themselves.
## Installation
The script runs inside the Monty sandbox, so install the extra:
```bash
uv add "pydantic-ai-harness[dynamic-workflow]"
```
## Your first workflow
Two sub-agents, one orchestrator:
```python
from pydantic_ai import Agent
from pydantic_ai_harness.dynamic_workflow import DynamicWorkflow
reviewer = Agent('openai:gpt-5', name='reviewer', description='Reviews code for bugs.')
summarizer = Agent('openai:gpt-5', name='summarizer', description='Summarizes findings.')
orchestrator = Agent(
'openai:gpt-5',
capabilities=[DynamicWorkflow(agents=[reviewer, summarizer])],
)
```
`reviewer` and `summarizer` are plain agents -- the same `Agent` you already know. Their `name` becomes the function name the model calls in the script, so pick names that are valid Python identifiers. Their `description` tells the model what each one is for; write it the way you would document a function. `DynamicWorkflow(agents=[...])` bundles them into one capability and hands the orchestrator a single `run_workflow` tool.
## What the model does with it
When the orchestrator decides to use the tool, it does not call your sub-agents one at a time. It writes a script:
```python
import asyncio
reports = await asyncio.gather(
reviewer(task="Review auth.py for bugs:\n<file contents>"),
reviewer(task="Review parser.py for bugs:\n<file contents>"),
)
await summarizer(task="Summarize these review findings:\n" + "\n\n".join(reports))
```
The parts that matter:
- Each sub-agent is an `async` function. You call it with `await`.
- You pass the work as a single keyword argument, `task`. Always by keyword -- `reviewer(task="...")`, not `reviewer("...")`.
- `asyncio.gather(...)` runs the two reviews concurrently instead of one after the other.
- The last expression's value becomes the result the model sees. The intermediate `reports` list never leaves the sandbox.
Each call is a full `Agent.run`, with its own model loop, message history, tools, and typed output. Two things follow: calls are **isolated** (a sub-agent remembers nothing from an earlier call, so put everything it needs into `task`), and calls **cost tokens and take time** (which is why this capability gives you budgets, below).
## Sub-agents can return structured data
A sub-agent returns whatever its `output_type` produces. The default is a string, but give a sub-agent a Pydantic model and the script receives a `dict`:
```python
from pydantic import BaseModel
class Score(BaseModel):
value: int
reason: str
critic = Agent('openai:gpt-5', name='critic', description='Scores an answer 0-10.', output_type=Score)
```
Inside the script the model reads the fields by subscript, the way it would read a JSON object:
```python
result = await critic(task="Score this answer: ...")
result["value"] # not result.value
```
The catalog the model sees renders each output type as a `TypedDict`, so it knows the fields and reads them by subscript on its own.
## How results come back
The value of the script's last expression becomes the tool result -- the model does not `print()` it.
| The script... | The model receives |
| --- | --- |
| ends in a value, no print | that value directly (or `{}` if it is `None`) |
| prints and ends in a value | `{"output": "<printed text>", "result": <value>}` |
| prints and ends in `None` | `{"output": "<printed text>"}` |
`print()` is for debug logging; it stringifies, so let the last expression carry the real result.
## Choosing sub-agent models
By default each sub-agent uses the model it was constructed with. Set `inherit_model=True` when the host passes a per-run model override to the parent agent (for example from a `/model` command) and every sub-agent dispatch should follow that resolved parent model. Leave it `False` when a sub-agent is deliberately pinned to a different model.
## Keeping it safe: budgets
A sub-agent is non-deterministic, costs tokens, and can fan out into more sub-agents. `DynamicWorkflow` gives you a hard count ceiling, token budgets, and a guard against runaway sandbox scripts.
### `max_agent_calls` -- an exact count
```python
DynamicWorkflow(agents=[...], max_agent_calls=50) # 50 is the default
```
A hard, host-enforced ceiling on the number of sub-agent runs in one parent run. It is one budget shared across every `run_workflow` call in that run, and it holds exactly even when the script fans out with `asyncio.gather`. When the budget runs out, the workflow stops calling sub-agents and returns a terminal result that includes the sub-agent results that did complete, so nothing you already paid for is wasted. This is the only knob that bounds the number of runs exactly.
### `sub_agent_usage_limits` and `forward_usage` -- bounding cost
`sub_agent_usage_limits` is a `UsageLimits` applied to each sub-agent run. `forward_usage` controls whether the whole tree shares one usage counter:
| `forward_usage` | Counter | What the limit means |
| --- | --- | --- |
| `True` (default) | the parent's `usage` is shared across the tree | a tree-wide cap. Under concurrent fan-out it is best-effort: several sub-agents can pass the check before any of them adds to the count. |
| `False` | each sub-agent run counts on its own | per-run limits. A per-run `total_tokens_limit` of `T` with `max_agent_calls` of `N` bounds the tree to roughly `N * T` tokens. |
!!! warning "The parent `run()` usage limit is not forwarded"
The `usage_limits` you pass to the parent `run()` is not forwarded into sub-agents -- it is re-checked only at the parent's own request boundaries. To bound sub-agents, set `sub_agent_usage_limits`; for an exact ceiling on the number of runs, use `max_agent_calls`.
### `resource_limits` -- guarding the script itself
These limits guard the orchestration script's own memory and allocations, not the sub-agents it calls. The default backstop is 256 MB and 50 million allocations, with no time limit.
```python
DynamicWorkflow(agents=[...], resource_limits={'max_duration_secs': 30})
```
`max_duration_secs` measures the time your script spends running sandbox code, not wall-clock time. While the script waits on a sub-agent it is suspended and that time does not count, so the cap will not fire on a normal workflow no matter how long the sub-agents take. Its one job is catching a pure-CPU runaway -- a `while True:` loop that never awaits, which none of the sub-agent budgets can stop because it never calls a sub-agent. Pass `'unlimited'` to remove every limit, or a partial dict that merges onto the backstop so you override only the caps you name.
### Workflows do not nest
A sub-agent cannot start its own workflow; a nested `run_workflow` call returns a terminal error instead of running. The practical rule: do not give the sub-agents in your catalog the `DynamicWorkflow` capability. They are the leaves of the orchestration, not orchestrators.
## Renaming a sub-agent: `WorkflowAgent`
By default a sub-agent shows up under its own `name` and `description`. To give it a different name or description for one workflow without editing the agent itself, wrap it in a `WorkflowAgent`:
```python
from pydantic_ai_harness.dynamic_workflow import WorkflowAgent
DynamicWorkflow(
agents=[
WorkflowAgent(
reviewer,
name='check',
description='Checks one code change and returns actionable review findings.',
),
],
)
```
Now the model calls `check(task=...)`. Passing a bare agent is shorthand for wrapping it in a `WorkflowAgent` with no overrides.
## Adding sub-agents mid-run: `reveal()`
The catalog is fixed when a run starts, which keeps it in the prompt-cache prefix across turns. To make a new sub-agent available during a run (say once a fixer agent has been provisioned), keep a reference to the `DynamicWorkflow` instance and call `reveal()`:
```python
workflow = DynamicWorkflow(agents=[reviewer])
orchestrator = Agent('openai:gpt-5', deps_type=MyDeps, capabilities=[workflow])
# later, from the host or from another tool:
workflow.reveal(fixer)
```
The revealed sub-agent becomes callable on the next step; the model learns about it through a short announcement message that carries the new function's signature. The `run_workflow` description itself stays frozen at the agents present when the run started, so a runtime reveal never moves the prompt-cache prefix. `reveal()` is append-only and validates immediately -- a missing name, an invalid identifier, a reserved keyword, or a name collision raises `UserError` at the call site.
## Loading it only when needed: `defer_loading`
`DynamicWorkflow` carries a fair amount of instruction text, and most turns do not need it. Keep it collapsed to a one-line entry until the model actually loads it:
```python
DynamicWorkflow(
agents=[reviewer, summarizer],
id='workflow',
defer_loading=True,
)
```
`defer_loading=True` needs a stable `id`. See [on-demand capabilities](/ai/core-concepts/capabilities/#on-demand-capabilities) for the full picture.
## What runs in the sandbox
The script runs in Monty, a subset of Python. Knowing the edges matters:
- No class definitions, and no third-party libraries.
- Useful standard-library modules: `asyncio`, `math`, `json`, `re`, `typing`. Import what you use; other modules are unavailable or stubbed.
- No wall-clock or timing primitives -- no `asyncio.sleep`, no `datetime.now()`, no `time`.
- `asyncio.gather(...)` runs sub-agents concurrently but does not support `return_exceptions=True`.
Before a script runs it is statically type-checked against the sub-agent signatures. A misspelled function, a positional `task`, or a wrong-typed argument costs one retry, but no sub-agent budget and no sandbox execution.
!!! warning "Errors abort the whole script"
A sub-agent that raises cannot be caught inside the script -- one failure aborts the whole script and the model retries it. Write scripts where sub-agents do not depend on catching each other's errors. If a script fails after some sub-agents already finished, the retry prompt lists those completed results, so the model can reuse them as plain values instead of paying for the same calls again.
## Observability
The [Logfire](https://pydantic.dev/logfire) trace is the best way to see what a workflow did. Each sub-agent run appears nested under the `run_workflow` span, and the span carries the exact `code` argument the model wrote, so you can read the script it actually ran. Until first-class progress streaming ships, set `event_stream_handler` on each sub-agent `Agent` to watch sub-agent runs inside the one tool call.
## API
```python
DynamicWorkflow( # all parameters are keyword-only
agents=[...], # Sequence[AbstractAgent | WorkflowAgent], required
tool_name='run_workflow',
max_agent_calls=50,
max_retries=3,
forward_usage=True,
inherit_model=False, # True -> sub-agents run with the parent run's resolved model
sub_agent_usage_limits=None, # UsageLimits per sub-agent run; None -> pydantic-ai default
resource_limits=None, # None -> backstop (256 MB, 50M allocs, no time cap);
# 'unlimited' -> off; a dict is merged onto the backstop
id=None, # required when defer_loading=True
description=None, # one-line catalog entry shown while deferred
defer_loading=False,
)
workflow.reveal(agent) # AbstractAgent | WorkflowAgent; validates before appending
WorkflowAgent(
agent, # Agent, required, positional
name=None, # sandbox function name; falls back to agent.name
description=None, # function docstring; falls back to agent.description
)
```
`DynamicWorkflowToolset` and `WorkflowResourceLimits` are also exported from the module for advanced use.
Source: [`pydantic_ai_harness/dynamic_workflow/`](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/dynamic_workflow/).
## Further reading
- [Code Mode](code-mode.md) -- the same sandbox, calling the agent's own tools instead of sub-agents.
- [Subagents](subagents.md) -- one-delegation-per-tool-call sub-agents, without the scripted choreography.
- [Rewriting Bun in Rust](https://bun.com/blog/bun-in-rust) (Bun) -- the same pattern at scale, via Claude Code's dynamic workflows.
- [Capabilities](/ai/core-concepts/capabilities/) and [on-demand capabilities](/ai/core-concepts/capabilities/#on-demand-capabilities).
docs/filesystem.md
+188
View File
@@ -0,0 +1,188 @@
---
title: FileSystem
description: Give a Pydantic AI agent sandboxed, glob-filtered file access scoped to a single directory tree, with symlink-safe containment checks.
---
# FileSystem
`FileSystem` gives an agent a fixed set of file tools -- read, write, edit, list,
search, find, create, and inspect -- all scoped to a single `root_dir`. Every path is
resolved and containment-checked (symlinks included) before any I/O, and access
is filtered through allow / deny / protected glob patterns.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/filesystem/)
## The problem
Letting an agent touch the filesystem directly is risky: path traversal
(`../../etc/passwd`), symlinks that escape the project, clobbering `.git`, or
leaking `.env` secrets. Hand-rolling the guards around every tool call is
repetitive and easy to get subtly wrong.
`FileSystem` centralizes those guards. It exposes one bounded, sandboxed
toolset so you configure the boundary once and reuse it across agents.
## Usage
Add `FileSystem` to your agent's `capabilities` with a `root_dir`. Everything
the agent reads or writes is confined to that directory.
```python
from pydantic_ai import Agent
from pydantic_ai_harness import FileSystem
agent = Agent(
'anthropic:claude-sonnet-4-6',
capabilities=[FileSystem(root_dir='./workspace')],
)
result = agent.run_sync('Read config.toml and tell me the package name.')
print(result.output)
```
`root_dir` defaults to the current directory (`.`), but passing an explicit
workspace path is the recommended practice -- the sandbox is only as tight as
the root you give it.
## Tools
`FileSystem` contributes eight tools, all path-scoped to `root_dir`:
| Tool | Purpose |
|---|---|
| `read_file` | Read a text file with line numbers and a content hash. Binary files are detected and not dumped. Supports `offset`/`limit` paging. |
| `write_file` | Create or overwrite a file. Optional `expected_hash` rejects stale writes (optimistic concurrency). |
| `edit_file` | Exact-string replacement; `old_text` must match exactly once. Optional `expected_hash`. |
| `list_directory` | List a directory's entries with type indicators and sizes. |
| `search_files` | Regex search over file contents, optionally narrowed by an `include_glob`. |
| `find_files` | Glob search over file names (e.g. `*.py`, `**/*.json`). |
| `create_directory` | Create a directory and any missing parents. |
| `file_info` | Metadata for a file or directory (size, type, line count, hash, symlink target). |
Tool errors the model can correct -- a missing file, a denied path, a stale
edit -- are surfaced as
[`ModelRetry`](/ai/core-concepts/agent/#reflection-and-self-correction),
so the agent gets the error message back and can adjust rather than aborting
the run.
## Security model
- **Containment.** Paths resolve relative to `root_dir`; anything resolving
outside -- via `..`, an absolute path, or a symlink -- is rejected. Symlinks
are resolved with `os.path.realpath` *before* the containment check, closing
the TOCTTOU window.
- **Binary detection.** `read_file` returns a placeholder instead of dumping
binary bytes into the model context.
- **Optimistic concurrency.** `write_file`/`edit_file` accept an
`expected_hash` so an agent operating on a stale read is told to re-read
rather than silently overwriting newer content.
## Pattern filtering
Three independent glob lists control access. Patterns are matched with
`fnmatch`, whose `*` spans `/`, so `*.py` matches `src/main.py` and you rarely
need `**`.
| Field | Effect |
|---|---|
| `allowed_patterns` | If non-empty, only matching paths are accessible (allowlist). |
| `denied_patterns` | Matching paths are always rejected (denylist). |
| `protected_patterns` | Matching paths are read-only -- reads succeed, writes are rejected. |
`protected_patterns` defaults to `.git/*`, `.env`, `.env.*`, `*.pem`, `*.key`,
and `**/secrets*`. Pass an empty list to disable protection.
```python
from pydantic_ai import Agent
from pydantic_ai_harness import FileSystem
agent = Agent(
'anthropic:claude-sonnet-4-6',
capabilities=[
FileSystem(
root_dir='./workspace',
allowed_patterns=['*.py', '*.toml'],
denied_patterns=['**/node_modules/*'],
),
],
)
```
### Direct access vs. walkers
The three rules apply at two different granularities:
- **Direct access** (`read_file`, `write_file`, `edit_file`, `file_info`,
`create_directory`) gates the operation's target path. You must name a path
that the patterns permit.
- **Walkers** (`list_directory`, `search_files`, `find_files`) gate their root
by deny/protected patterns, but **not** by `allowed_patterns` -- a directory
root like `.` never matches a file pattern such as `src/*.py`, so requiring
it to would make every listing fail. Instead, the root is always walked and
each **entry** is filtered against all three lists. A directory listing can
never surface a path the agent couldn't otherwise read or write.
So with `allowed_patterns=['*.py']`, `list_directory('.')` succeeds and shows
only the `.py` entries; `read_file('notes.md')` is rejected.
Note that the walkers filter entries with write-level access, so
`protected_patterns` matches are omitted from `list_directory`, `search_files`,
and `find_files` output even though those exact paths remain directly readable
via `read_file`/`file_info`.
!!! note
Dotfiles and dot-directories (`.git`, `.env`, `.github`, ...) are skipped by
all three walkers -- `list_directory`, `search_files`, and `find_files` --
regardless of patterns.
## Configuration
```python
from pydantic_ai_harness import FileSystem
FileSystem(
root_dir='.', # str | Path -- sandbox root
allowed_patterns=[], # allowlist globs (empty = allow all)
denied_patterns=[], # denylist globs
protected_patterns=[...], # read-only globs (defaults to secrets/.git)
max_read_lines=2000, # cap for a single read_file
max_search_results=1000, # cap for search_files
max_find_results=1000, # cap for find_files
)
```
The three integer limits must be positive; they are validated at construction
and raise `ValueError` otherwise.
## Agent spec (YAML/JSON)
`FileSystem` works with Pydantic AI's
[agent spec](/ai/core-concepts/agent-spec/):
```yaml
model: anthropic:claude-sonnet-4-6
capabilities:
- FileSystem:
root_dir: ./workspace
allowed_patterns: ['*.py', '*.toml']
```
```python
from pydantic_ai import Agent
from pydantic_ai_harness import FileSystem
agent = Agent.from_file('agent.yaml', custom_capability_types=[FileSystem])
```
Pass `custom_capability_types` so the spec loader knows how to instantiate
`FileSystem`.
## Further reading
- [Pydantic AI capabilities](/ai/core-concepts/capabilities/)
- [Toolsets](/ai/tools-toolsets/toolsets/)
- [the capabilities overview](index.md)
## API reference
::: pydantic_ai_harness.FileSystem
docs/guardrails.md
+180
View File
@@ -0,0 +1,180 @@
---
title: Input & Output Guardrails
description: Validate the user prompt before it reaches the model and the model output before it reaches the caller, with allow/block/replace/retry verdicts and optional parallel execution.
---
# Input & Output Guardrails
Guardrails put a validation layer on the two edges of an agent run: the prompt on its way *in* to the model, and the output on its way *out* to the caller. Reach for them when unstructured input or output must be screened before it is acted on -- a prompt-injection attempt you never want to send, PII you must redact, an off-topic request you want to refuse cheaply, or an answer that must cite its sources before you show it. Without a guardrail the framework sends whatever the user typed and returns whatever the model produced, verbatim; a guardrail interposes a callable you control that gets the final say.
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
## The problem
Agents take unstructured input from users and return unstructured output to callers. On its own the framework does not reason about "this is unsafe to send" or "this is unsafe to show" -- a prompt-injection attempt reaches the model as-is, and any output the model produces is returned untouched. You need a place to inspect the value and decide what happens next.
## The solution
Two capabilities -- `InputGuard` and `OutputGuard` -- each wrap a `guard` callable you supply. The guard inspects a value (the prompt, or the output) and returns one of four outcomes:
| Outcome | `InputGuard` | `OutputGuard` |
|---|---|---|
| **allow** | send the prompt to the model | return the output to the caller |
| **block** | skip the model call; a refusal message becomes the response | raise `OutputBlocked` |
| **replace** | rewrite the prompt sent to the model (redaction) | substitute a sanitized output |
| **retry** | -- (not valid for input) | send the output back to the model to try again |
The asymmetry between input `block` and output `block` is deliberate. Blocking the input spends no tokens, so a graceful refusal is almost always right. Blocking the output means the model already produced something you do not want exposed, so raising forces the caller to decide what to do next.
Both `InputGuard`, `OutputGuard`, and their supporting types are top-level exports:
```python
from pydantic_ai import Agent
from pydantic_ai_harness import GuardResult, InputGuard, OutputGuard
def no_secrets(prompt: str) -> bool:
return 'api_key' not in prompt.lower()
def no_pii(output: object) -> GuardResult:
if 'SSN' in str(output):
return GuardResult.block('The response contained personal data.')
return GuardResult.allow()
agent = Agent(
'openai:gpt-5.4',
capabilities=[
InputGuard(guard=no_secrets),
OutputGuard(guard=no_pii),
],
)
```
A guard returns a bare `bool` (`True` = allow, `False` = block) for the simple case, or a `GuardResult` for the richer outcomes. Guards may also be async -- return an awaitable `bool`/`GuardResult`, for example to call a moderation API.
`OutputGuard` receives the output unchanged -- no automatic stringification. For a string output the guard reads it directly; for a typed (Pydantic model) output the guard gets the model instance, so pick the serialization that fits the check (read a field, or call `output.model_dump_json()` for JSON text). This avoids the trap of `str(MyModel(...))` producing a `MyModel(field=...)` repr that hides field contents from regex-based checks.
## `GuardResult`
Construct a `GuardResult` with its classmethods, not the raw fields:
```python
from pydantic_ai_harness import GuardResult
GuardResult.allow() # let the value through
GuardResult.block('reason') # refuse; `reason` is optional (a default is used otherwise)
GuardResult.replace(cleaned_value) # substitute a sanitized value and continue
GuardResult.retry('instruction') # OutputGuard only: ask the model to redo the output
```
The block/retry message is produced at the moment the guard decides, so it can carry the guard's own reasoning rather than a string frozen at construction time.
## Redaction (`replace`)
Return `GuardResult.replace(value)` to sanitize rather than refuse. `InputGuard` rewrites the prompt sent to the model; `OutputGuard` substitutes the output returned to the caller.
```python
def scrub_emails(text: str) -> GuardResult:
cleaned = EMAIL_RE.sub('[email]', text)
return GuardResult.replace(cleaned) if cleaned != text else GuardResult.allow()
agent = Agent(
'openai:gpt-5.4',
capabilities=[
InputGuard(guard=scrub_emails), # strip PII before it reaches the model
OutputGuard(guard=scrub_emails), # strip PII before it reaches the caller
],
)
```
Input redaction requires sequential mode -- it is incompatible with `parallel=True`, since a parallel guard runs alongside a model call that has already started with the original prompt.
## Retry (`retry`)
`OutputGuard` can send a bad output back to the model instead of blocking it. Return `GuardResult.retry(instruction)` -- the instruction is the retry prompt the model sees. This reuses pydantic-ai's normal retry machinery and counts against the run's output-retry budget.
```python
def must_cite_sources(output: object) -> GuardResult:
if not has_citations(output):
return GuardResult.retry('Include at least one source citation.')
return GuardResult.allow()
OutputGuard(guard=must_cite_sources)
```
## Accessing run context
A guard may take a `RunContext` as its first parameter when it needs run state -- `deps` for tenant- or role-aware policy, message history for conversation-aware checks. The parameter is detected from the signature, so prompt-only guards need not declare it:
```python
from pydantic_ai import RunContext
from pydantic_ai_harness import InputGuard
def tenant_policy(ctx: RunContext[MyDeps], prompt: str) -> bool:
return ctx.deps.tier == 'pro' or 'advanced-feature' not in prompt
InputGuard(guard=tenant_policy)
```
## Parallel input guards
A slow guard (an LLM classifier, a network call) run sequentially adds its latency to every turn. Set `parallel=True` to run the guard concurrently with the model call instead, overlapping the two so the guard adds no latency on the pass path. The model call is cancelled the moment the guard reports a violation.
```python
InputGuard(guard=slow_async_classifier, parallel=True)
```
Parallel mode trades tokens for latency: sequential mode never calls the model when the guard blocks, but parallel mode has already started the model call -- if the guard trips only after the model has responded, those tokens were spent. For fast local checks (regex, keyword lookup) sequential is the better default. `replace` is not available under `parallel=True`.
## Hard-fail path
`block` is the graceful path. To make the caller see an exception instead, raise from the guard:
```python
from pydantic_ai_harness import InputBlocked
def strict_guard(prompt: str) -> bool:
if contains_credentials(prompt):
raise InputBlocked('credentials detected')
return True
```
Any exception raised by the guard propagates as-is -- use `InputBlocked` / `OutputBlocked` from this module, or your own exception types.
## Streaming
`OutputGuard` inspects the **final** output only -- during `run_stream()` partial chunks reach the caller before the guard runs, so a `block` or `replace` verdict cannot un-send content already streamed. Use `run()` / `run_sync()` when the output must be screened before any of it is exposed. `GuardResult.retry()` is **not** supported under `run_stream()` and surfaces there as `UnexpectedModelBehavior`. `InputGuard` (including `parallel=True`) works the same in streamed and non-streamed runs.
## Tracing
`replace` and `block` are recorded as spans on the active OpenTelemetry tracer, so a redaction or refusal shows up in [Logfire](https://pydantic.dev/logfire) traces (`guardrail redacted input`, `guardrail blocked output`, and so on) with `guardrail.*` attributes. Content attributes -- the original/replacement values for a redaction and the refusal `message` for a block -- are attached **only** when `RunContext.trace_include_content` is enabled, since these can quote the very content the guard exists to keep out of traces.
`OutputGuard` positions its block/redact spans so they are always captured by an enclosing `Instrumentation` span regardless of capability order, while `InputGuard` runs innermost so any capability that morphs messages (a prompt rewriter, a context manager) runs first and the guard sees the final prompt the model will receive.
## Relationship to `pydantic-ai-shields`
[`pydantic-ai-shields`](https://github.com/vstorm-co/pydantic-ai-shields) provides opinionated implementations on top of these primitives (prompt-injection detectors, PII scrubbers, keyword blocklists). Use the guardrails here when you want to plug in your own validation logic; reach for shields when you need a batteries-included detector.
## API
```python
InputGuard(
guard, # Callable[..., bool | GuardResult | Awaitable[bool | GuardResult]]
parallel=False, # run concurrently with the model call
)
OutputGuard(
guard, # Callable[..., bool | GuardResult | Awaitable[bool | GuardResult]]
)
```
The guard callable takes the inspected value -- the prompt for `InputGuard`, the output for `OutputGuard` -- optionally preceded by a `RunContext`. `InputGuardFunc` and `OutputGuardFunc` are the exported signature aliases; `GuardrailError` is the base for `InputBlocked` and `OutputBlocked`.
Source: [`pydantic_ai_harness/guardrails/`](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/guardrails/).
docs/index.md
+149
View File
@@ -0,0 +1,149 @@
---
title: Pydantic AI Harness
description: The official capability library for Pydantic AI -- pick-and-choose batteries that turn your agent into a coding agent, research assistant, or anything else.
---
# Pydantic AI Harness
**The batteries for your [Pydantic AI](/ai/) agent.**
Pydantic AI's [capabilities](/ai/core-concepts/capabilities/) and [hooks](/ai/core-concepts/hooks/) API is how you give an agent its harness -- bundles of tools, lifecycle hooks, instructions, and model settings that extend what the agent can do without any framework changes.
**Pydantic AI Harness** is the official capability library for Pydantic AI, maintained by the [Pydantic AI](https://github.com/pydantic/pydantic-ai) team. Pydantic AI core ships the capabilities that require model or framework support, plus the ones fundamental to every agent -- [web search](/ai/core-concepts/capabilities/#provider-adaptive-tools), [tool search](/ai/tools-toolsets/deferred-tools/), [thinking](/ai/core-concepts/capabilities/#thinking). Everything else lives here: standalone building blocks you pick and choose to turn your agent into a coding agent, a research assistant, or anything else. This is also where new capabilities start -- as they stabilize and prove themselves broadly essential, they can graduate into core.
## What goes where?
Pydantic AI core ships the agent loop, model providers, the capabilities/hooks abstraction, and two kinds of capabilities:
- **Capabilities that require model or framework support** -- anything backed by provider native tools (like [image generation](/ai/core-concepts/capabilities/#provider-adaptive-tools)), provider-specific APIs (like compaction via the OpenAI or Anthropic APIs), or deep agent graph integration. These go hand-in-hand with model class code and need to ship together.
- **Capabilities that are fundamental to the agent experience** -- things nearly every agent benefits from, like [web search](/ai/core-concepts/capabilities/#provider-adaptive-tools), [tool search](/ai/tools-toolsets/deferred-tools/), and [thinking](/ai/core-concepts/capabilities/#thinking). These feel like qualities of the agent itself, not accessories.
**Pydantic AI Harness** is where everything else lives: standalone capabilities that make specific categories of agents powerful, or that are still finding their final shape. Context management, memory, guardrails, file system access, code execution, multi-agent orchestration -- these are the building blocks you pick and choose based on what your agent needs to do.
The harness is also where new capabilities *start*. It ships as a separate package so capabilities can iterate faster without the strict backward-compatibility requirements of core. As a capability stabilizes and proves itself broadly essential, it can graduate into core -- [code mode](code-mode.md) is an early candidate.
Many capabilities benefit from a "fall up" pattern: they typically start as a local implementation that works with every model, then gain provider-native support that uses the provider's built-in API when available -- auto-switching between the two. This is how [web search](/ai/core-concepts/capabilities/#provider-adaptive-tools), [web fetch](/ai/core-concepts/capabilities/#provider-adaptive-tools), and [image generation](/ai/core-concepts/capabilities/#provider-adaptive-tools) already work in core, and the same approach is coming for skills, code mode, and context compaction.
## Installation
```bash
uv add pydantic-ai-harness
```
Some capabilities need an extra to pull in their optional dependencies:
```bash
uv add "pydantic-ai-harness[codemode]" # Code Mode (adds the Monty sandbox)
uv add "pydantic-ai-harness[dynamic-workflow]" # Dynamic Workflow (adds the Monty sandbox)
uv add "pydantic-ai-harness[logfire]" # Managed Prompt (Logfire-managed prompts)
uv add "pydantic-ai-harness[acp]" # ACP (Agent Client Protocol SDK)
```
The `code-mode` extra is also supported as an alias for `codemode`.
Requires Python 3.10+ and `pydantic-ai-slim>=2.1.0`.
## Quick start
Install the harness alongside the Pydantic AI extras this example uses:
```bash
uv add "pydantic-ai-slim[anthropic,mcp,duckduckgo,logfire]" "pydantic-ai-harness[code-mode]"
```
```python
import logfire
from pydantic_ai import Agent
from pydantic_ai.capabilities import MCP, WebSearch
from pydantic_ai_harness import CodeMode
# See https://pydantic.dev/docs/ai/integrations/logfire/ for setup details.
logfire.configure()
logfire.instrument_pydantic_ai()
agent = Agent(
'anthropic:claude-opus-4-7',
capabilities=[
# Wraps every tool into a single run_code tool, sandboxed by Monty
# (https://github.com/pydantic/monty -- pulled in by the [code-mode] extra).
# The model writes Python that calls multiple tools with loops, conditionals,
# asyncio.gather, and local filtering -- one model round-trip for N tool calls.
CodeMode(),
# Connect to any MCP server -- here, the open-source Hacker News server
# (https://github.com/cyanheads/hn-mcp-server). native=False forces the
# local MCP toolset so CodeMode can wrap the tools; without it,
# providers that natively support MCP server connectors execute the tools
# server-side and bypass the sandbox.
MCP('https://hn.caseyjhand.com/mcp', native=False),
# Provider-adaptive web search; native=False routes through the local
# DuckDuckGo fallback (the [duckduckgo] extra above) so CodeMode can batch
# web searches alongside the HN calls in a single run_code.
WebSearch(native=False),
],
)
result = agent.run_sync(
"Across the top, best, and 'show HN' Hacker News feeds, find the most-discussed "
"story with at least 100 points. Pull its comment thread, its submitter's profile, "
"and any web coverage. Summarize what you find in one paragraph."
)
print(result.output)
"""
The most-discussed HN story across top/best/show clearing 100 points is "Vibe coding
and agentic engineering are getting closer than I'd like" by Simon Willison (748 points,
853 comments, on the Best feed), submitted by long-time HNer e12e. The piece argues
that the two modes Willison once kept mentally separate -- throwaway "vibe coding" and
disciplined "agentic engineering" -- are blurring, since agents like Claude Code now
reliably handle non-trivial tasks like "build a JSON API endpoint that runs a SQL query"
with tests and docs on the first pass. The HN thread is unusually substantive, with
commenters debating whether LLMs created or merely *exposed* sloppy engineering
practices and warning of a "normalization of deviance" as engineers stop reviewing diffs.
"""
```
[![Logfire trace from the Quick start run](images/quick-start-trace.png)](https://logfire-us.pydantic.dev/public-trace/84bcf123-2106-49da-9f6f-5c26395339bb?spanId=7650806a0785b946)
**[See this run as a public Logfire trace ->](https://logfire-us.pydantic.dev/public-trace/84bcf123-2106-49da-9f6f-5c26395339bb?spanId=7650806a0785b946)** Each `run_code` span fans out into the tool calls the model issued from inside the sandbox -- it's the easiest way to understand what code mode actually did.
## Capabilities
Each capability is a self-contained battery you drop into an agent's `capabilities=[...]` list. They compose with each other and with Pydantic AI's [built-in capabilities](/ai/core-concepts/capabilities/).
| Capability | What it does | Extra |
|---|---|---|
| [Code Mode](code-mode.md) | Wraps the agent's tools into a single `run_code` tool, sandboxed by [Monty](https://github.com/pydantic/monty). The model writes Python that calls the tools as functions -- with loops, conditionals, `asyncio.gather`, and local filtering -- collapsing N tool calls into one model round-trip. | `codemode` |
| [FileSystem](filesystem.md) | Sandboxed file access scoped to a root directory: read, write, edit, search, and find files. Rejects path traversal above the root, resolves symlinks before authorizing, and keeps `.git/`, `.env`, key files, and secrets read-only by default. | -- |
| [Shell](shell.md) | Command execution in a subprocess rooted at a working directory, gated by allowlists, denylists, timeouts, and optional environment-variable stripping (including a preset for common LLM provider credentials). | -- |
| [Context](context.md) | Auto-loads repo context -- `CLAUDE.md`/`AGENTS.md` and repository structure -- so the agent starts a run already oriented in the project. | -- |
| [Pydantic AI Docs](pydantic-ai-docs.md) | An on-demand `read_pyai_docs` tool that pulls Pydantic AI documentation into the run when the agent needs it, instead of preloading it. | -- |
| [Compaction](compaction.md) | Keeps a run within token limits: sliding-window trimming, LLM-powered summarization of older messages, and warnings before the context or iteration ceiling is hit. | -- |
| [Overflowing Tool Output](overflowing-tool-output.md) | Reduces an oversized tool return when it is produced -- truncate, spill to a queryable file, or summarize -- so a large payload does not persist in history and get re-sent every request. | -- |
| [Step Persistence](step-persistence.md) | Saves and restores full conversation state; snapshot, resume (`continue_run`), and fork (`fork_run`) a run. | -- |
| [Media](media.md) | Offloads large `BinaryContent` to content-addressed stores (local or S3) so big media does not bloat message history. | -- |
| [Subagents](subagents.md) | Delegates subtasks to specialized child agents through a delegate tool. | -- |
| [Dynamic Workflow](dynamic-workflow.md) | Orchestrates sub-agents from a model-written Python script -- fan-out, chaining, and voting in a single tool call. | `dynamic-workflow` |
| [Planning](planning.md) | Breaks a complex task into a structured plan before execution and tracks progress against it. | -- |
| [Runtime Authoring](runtime-authoring.md) | Lets an agent author, validate, and load real capabilities at runtime. | -- |
| [Guardrails](guardrails.md) | Validates user input before a run starts and model output after it completes -- block or redact, with structured results. | -- |
| [Managed Prompt](managed-prompt.md) | Backs an agent's instructions with a [Logfire-managed prompt](https://logfire.pydantic.dev/docs/reference/advanced/prompt-management/), so you can version, label, and roll out prompt changes from the Logfire UI without redeploying -- with a code default that keeps the agent working when no remote value is available. | `logfire` |
| [ACP](acp.md) *(experimental)* | Serves an agent to editors (Zed, etc.) over the [Agent Client Protocol](https://agentclientprotocol.com) -- streamed text, diff-rendered edits, and tool approval. | `acp` |
Most capabilities are stable within the [version policy](#version-policy) below. [ACP](acp.md) is the exception -- it is still experimental, imported from `pydantic_ai_harness.experimental.acp`, and may change or be removed in a future release.
## Build your own
[Capabilities](/ai/core-concepts/capabilities/#building-custom-capabilities) are the primary extension point for Pydantic AI. Any of the capabilities in this library can serve as a reference for building your own.
Publishing as a standalone package? Use the `pydantic-ai-<name>` naming convention -- see [Publishing capability packages](/ai/guides/extensibility/#publishing-capability-packages).
## Version policy
Pydantic AI Harness uses **0.x versioning** to signal that APIs are still stabilizing. During 0.x, minor releases (0.1 -> 0.2) may include breaking changes -- renamed parameters, changed defaults, restructured APIs -- while patch releases (0.1.0 -> 0.1.1) will not intentionally break existing behavior. All breaking changes are documented in release notes with migration guidance. This is why the harness is a separate package from [Pydantic AI](https://github.com/pydantic/pydantic-ai), which has a [stricter version policy](/ai/project/version-policy/). As the core capabilities stabilize, the library will move toward 1.0 with matching stability guarantees.
## Pydantic AI references
- [Capabilities](/ai/core-concepts/capabilities/) -- what capabilities are, built-in capabilities, building your own
- [Hooks](/ai/core-concepts/hooks/) -- lifecycle hooks reference, ordering, error handling
- [Extensibility](/ai/guides/extensibility/) -- publishing packages, third-party ecosystem
- [Toolsets](/ai/tools-toolsets/toolsets/) -- building tools for capabilities
- [API reference](/ai/api/pydantic-ai/capabilities/) -- full API docs
docs/managed-prompt.md
+227
View File
@@ -0,0 +1,227 @@
---
title: Managed Prompt
description: Back a Pydantic AI agent's instructions with a Logfire-managed prompt so you can version, label, and roll it out without redeploying.
---
# Managed Prompt
`ManagedPrompt` backs an agent's instructions with a
[Logfire-managed prompt](https://logfire.pydantic.dev/docs/reference/advanced/prompt-management/),
so you can iterate on your system prompt from the Logfire UI -- versioned, labelled, and rolled
out -- without touching code or redeploying. It's a Pydantic AI [capability](index.md), so you
wire it in through the `capabilities=` parameter on `Agent`.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/logfire/)
Install the `logfire` extra:
```bash
uv add "pydantic-ai-harness[logfire]"
```
!!! note "A first-party `Managed` capability is in flight"
A broader, first-party `Managed` capability is being built in
[pydantic-ai#5107](https://github.com/pydantic/pydantic-ai/pull/5107) and will eventually be
importable as `pydantic_ai.managed.logfire.Managed` -- covering instructions, model settings,
and whole-spec variables. Until then, `ManagedPrompt` is the supported path for backing an
agent's instructions with a Logfire-managed prompt.
## The problem it solves
Prompts are critical to agent behavior, but iterating on them through the normal
edit -> review -> deploy loop is slow. You can't easily A/B test a change, and you can't roll it
back the moment it misbehaves in production without shipping a new build.
`ManagedPrompt` moves the prompt out of your codebase and into Logfire's managed-variable store.
It declares the backing managed variable for you and resolves it **once per run**, feeding the
resolved value into the agent's instructions. Resolution happens inside the run's
[`wrap_run`](/ai/api/pydantic-ai/capabilities/#pydantic_ai.capabilities.AbstractCapability.wrap_run)
hook, using the
[`ResolvedVariable`](https://logfire.pydantic.dev/docs/reference/advanced/managed-variables/) as a
context manager that stays open for the whole run -- so the selected label and version are attached
as baggage to every child span of the agent run. You get a direct correlation between a run's
behavior and the exact prompt version that produced it, plus instant iteration and rollback from
the Logfire UI.
## Usage
Pass the prompt name and a default value. The name `support_agent` is declared as the managed
variable `prompt__support_agent` -- the naming Logfire's Prompt management uses (hyphens in a name
become underscores). The `default` keeps the agent working until a remote value is published, so
your code always runs even before you create the prompt in Logfire.
```python
import logfire
from pydantic_ai import Agent
from pydantic_ai_harness.logfire import ManagedPrompt
logfire.configure()
agent = Agent(
'openai:gpt-5',
capabilities=[
ManagedPrompt(
'support_agent',
default='You are a helpful customer support agent. Be friendly and concise.',
label='production',
)
],
)
result = agent.run_sync('My order never arrived.')
print(result.output)
```
Pinning `label='production'` is the recommended default: the resolved value only changes on a
deliberate prompt rollout, which keeps the provider prompt cache hot (see
[Prompt-cache trade-off](#prompt-cache-trade-off) below).
## Targeting
For deterministic A/B assignment (the same user always sees the same label), pass a
`targeting_key`. It can be a static string or a callable that derives the key from the
[`RunContext`](/ai/api/pydantic-ai/tools/#pydantic_ai.tools.RunContext) -- handy when the
key lives in your agent's `deps`:
```python
from dataclasses import dataclass
from pydantic_ai import Agent
from pydantic_ai_harness.logfire import ManagedPrompt
@dataclass
class Deps:
user_id: str
agent = Agent(
'openai:gpt-5',
deps_type=Deps,
capabilities=[
ManagedPrompt(
'support_agent',
default='You are a helpful customer support agent.',
targeting_key=lambda ctx: ctx.deps.user_id,
),
],
)
```
Pass `attributes` (a mapping, or a callable returning one) for condition-based targeting rules.
When `label` is omitted, the variable's rollout and targeting rules pick the label. When both
`targeting_key` and `attributes` are omitted, Logfire falls back to its own targeting context and
then to the active trace id.
For Logfire-side targeting that lives outside the agent (e.g. set once per request handler), use
Logfire's
[`targeting_context`](https://logfire.pydantic.dev/docs/reference/advanced/managed-variables/) in
an outer scope; `ManagedPrompt` only needs `targeting_key` / `attributes` when the key comes from
the agent's `RunContext`.
## Templating with deps
By default the resolved prompt is used verbatim. Pass `render_template=True` to render it as a
Handlebars template against the agent's `deps` -- the same mechanism as
[`TemplateStr`](/ai/api/pydantic-ai/agent/) -- so `{{field}}` is filled
from `deps`:
```python
from dataclasses import dataclass
from pydantic_ai import Agent
from pydantic_ai_harness.logfire import ManagedPrompt
@dataclass
class Deps:
customer_name: str
agent = Agent(
'openai:gpt-5',
deps_type=Deps,
capabilities=[
ManagedPrompt(
'support_agent',
default='You are helping {{customer_name}}. Be friendly and concise.',
render_template=True,
),
],
)
```
Rendering requires `pydantic-handlebars` (install `pydantic-ai-slim[spec]`). It is off by default.
## Prompt-cache trade-off
The resolved value lands in the agent's **system instructions**. Provider prompt caches (Anthropic,
OpenAI, etc.) key strictly by prefix -- `tools -> system -> messages` -- so any change to the system
block invalidates the cached prefix for the affected runs.
| Mode | Cache impact |
| --- | --- |
| Pinned `label='production'`, no rollout split | **Cache-stable.** The value only changes on a deliberate prompt rollout, which is the same cost as a redeploy. |
| Percentage rollout across labels (no `label=`) | Different runs land on different labels -> splits the cache into one lane per label. |
| `targeting_key` per user/tenant with multiple labels in play | Cache lanes per assigned label; deterministic per key but still N lanes overall. |
| Mid-traffic label flip in the Logfire UI | One-shot cold-invalidation for everyone on that label. |
In short: pinning a `label` keeps the cache hot; using `ManagedPrompt` as an A/B platform is opt-in
cache cost. If you don't need rollouts, `label='production'` is the recommended default.
## Bringing your own variable
Declaring the same name more than once is fine -- each `ManagedPrompt` builds its own backing
variable, so sharing a prompt across several agents just works. Pass an existing
[`logfire.variables.Variable`](https://logfire.pydantic.dev/docs/reference/advanced/managed-variables/)
as the first argument instead of a name when you want to declare the variable yourself -- for
example a template variable, or one registered for `variables_push`:
```python
import logfire
from pydantic_ai import Agent
from pydantic_ai_harness.logfire import ManagedPrompt
logfire.configure()
support_prompt = logfire.var(
name='prompt__support_agent',
type=str,
default='You are a helpful customer support agent. Be friendly and concise.',
)
agent = Agent('openai:gpt-5', capabilities=[ManagedPrompt(support_prompt, label='production')])
```
When `name` is a prompt name (not a `Variable`), pass `logfire_instance=` to declare the variable
on a specific Logfire instance instead of the module-level default. `default` is required when
`name` is a prompt name and is ignored when you pass a `Variable` (which already carries its own
default and instance).
## How it composes
- **Resolves once per run.** A label flip or rollout change that lands in Logfire mid-run is not
picked up until the next run starts -- the trade-off for run-stable instructions and a single
baggage scope across all child spans.
- **Runs outermost.** The capability wraps
[`Instrumentation`](/ai/api/pydantic-ai/capabilities/#pydantic_ai.capabilities.Instrumentation)
so the resolved variable's baggage covers the agent run span as well as its children. On recent
Logfire versions both the selected label and the version are propagated as separate baggage
attributes.
- **Concurrency-safe.** Resolution is isolated per run via a context variable, so a single
capability instance is safe to share across concurrent runs.
- **Inspectable mid-run.** `ManagedPrompt.resolved` exposes the active run's `ResolvedVariable`
(`value`, `label`, `version`, `reason`) for inspection -- e.g. from inside a tool. It is `None`
outside a run.
## API reference
The resolved prompt is a `str`. Pass the bare prompt name (the `prompt__` prefix and
hyphen-to-underscore normalization are applied for you) and a `default`, then use `label`,
`targeting_key`, `attributes`, `render_template`, and `logfire_instance` to control resolution.
::: pydantic_ai_harness.ManagedPrompt
docs/media.md
+151
View File
@@ -0,0 +1,151 @@
---
title: Media Externalization
description: Content-addressed stores and walker helpers that move large BinaryContent payloads out of message history into deduplicated storage and put them back on demand.
---
# Media Externalization
A conversation that carries images, audio, or other `BinaryContent` inlines those bytes into every message. Persist that history and each snapshot re-serializes the payloads; the same image referenced by ten messages is ten copies of the bytes. Media externalization solves that: content-addressed stores write each payload once, keyed by its own hash, and leave a short `media+sha256://` URI in its place. Reach for it whenever binary payloads would otherwise balloon what you store or send.
!!! note "Import path"
Import these helpers from their submodule -- there is no top-level `pydantic_ai_harness` re-export:
```python
from pydantic_ai_harness.media import (
DiskMediaStore,
S3MediaStore,
SqliteMediaStore,
externalize_media,
restore_media,
)
```
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
## Building blocks, not a capability
These are building blocks. There is no class you add to `Agent(capabilities=[...])` yet. [`StepPersistence`](step-persistence.md) already uses them to keep snapshots small when messages carry `BinaryContent`, and a forthcoming `MediaExternalizer` capability will reuse the same stores to rewrite `BinaryContent` into URL parts before the model sees them.
## Why content-addressing
The URI is derived from the payload hash, so identical bytes deduplicate automatically. The same bytes are stored once no matter how many messages or snapshots reference them, and moving the underlying storage is a one-line swap because the URI does not change.
## Stores
Every store implements the `MediaStore` protocol -- `put`, `get`, `exists`, `public_url`, and `get_metadata`, all async and content-addressed.
| Store | Backed by | Use when |
|---|---|---|
| `DiskMediaStore(directory=...)` | A directory on disk | Local runs and tests |
| `SqliteMediaStore(database=...)` | A SQLite database | A single-file store that travels with the data |
| `S3MediaStore(bucket=, endpoint=, region=, ...)` | S3 or an S3-compatible bucket | Shared or production storage |
`S3MediaStore` uses path-style URLs plus handrolled SigV4, so it is compatible with AWS S3, Cloudflare R2 (`region='auto'`), MinIO, and other S3-compatible providers. `SqliteMediaStore` also accepts `connection=` instead of `database=` to share a `sqlite3.Connection`.
## Walker helpers
`externalize_media` and `restore_media` walk a message node and swap payloads for URIs and back:
```python
from pydantic_ai_harness.media import DiskMediaStore, externalize_media, restore_media
store = DiskMediaStore(directory='./media')
# Replace BinaryContent larger than the threshold with media+sha256:// URIs.
lean = await externalize_media(message, media_store=store, threshold_bytes=32_000)
# Later, rehydrate the URIs back into BinaryContent.
full = await restore_media(lean, media_store=store)
```
`externalize_media` only externalizes payloads over `threshold_bytes`; smaller ones stay inline. Round-trip is transparent -- `restore_media` returns `BinaryContent` with the original bytes. If you need to key media yourself, `media_uri_for` and `parse_media_uri` give you the raw URI round-trip.
## Public URLs
When a store is fronted by a CDN, a local HTTP server, or a signed-URL service, pass a `public_url=` resolver (or use `make_static_public_url`) to turn a stored `media+sha256://` URI into a URL the model can fetch directly. Without a resolver, `public_url(...)` returns `None`.
A static base URL, for a public bucket or CDN:
```python
from pydantic_ai_harness.media import S3MediaStore, make_static_public_url
store = S3MediaStore(
bucket='my-bucket',
endpoint='https://<acc>.r2.cloudflarestorage.com',
region='auto',
access_key_id=..., secret_access_key=...,
key_prefix='media/',
public_url=make_static_public_url('https://pub-abc.r2.dev', key_prefix='media/'),
)
```
A presigned or rotating-signature URL -- pass any async callable that takes `(uri, MediaContext)`:
```python
from pydantic_ai_harness.media import MediaContext, S3MediaStore
async def presign(uri: str, ctx: MediaContext) -> str:
key = 'media/' + uri.removeprefix('media+sha256://') + '.bin'
return await my_signer.generate(key, ttl=3600, content_type=ctx.media_type)
store = S3MediaStore(..., public_url=presign)
```
This is what the forthcoming `MediaExternalizer` will use to swap `BinaryContent` parts for `ImageUrl` / `AudioUrl` / other URL parts before the model sees the message, letting providers fetch big media over the wire without re-encoding bytes into the request body. Emitting a URL is always safe: pydantic-ai providers transparently download the bytes when the target model does not natively accept that URL type, so you only ever lose wire savings, never correctness.
## `MediaContext`
Every store method and both user-supplied callables (`PublicUrlResolver`, `KeyStrategy`) accept a `MediaContext` -- an extensible per-operation bag:
```python
from collections.abc import Mapping
from dataclasses import dataclass, field
@dataclass(frozen=True, kw_only=True)
class MediaContext:
media_type: str | None = None # e.g. 'image/png'
filename: str | None = None # original filename, when known
metadata: Mapping[str, str] = field(default_factory=dict) # user-supplied tags
```
All fields default, so you pass what you have and ignore the rest; new fields are added non-breakingly as use cases emerge. `get_metadata(uri)` round-trips the user-supplied `metadata` mapping on all three stores; `media_type` is persisted separately (as the byte payload's `Content-Type`).
## `KeyStrategy`
The default on-store key layout is `<sha256>.bin`. `DiskMediaStore` and `S3MediaStore` accept a `key_strategy=` override to fit an existing layout; `SqliteMediaStore` does not, since its primary key is the digest:
```python
from pydantic_ai_harness.media import DiskMediaStore, MediaContext
def by_media_type(uri: str, ctx: MediaContext) -> str:
digest = uri.removeprefix('media+sha256://')
ext = {'image/png': '.png', 'image/jpeg': '.jpg'}.get(ctx.media_type or '', '.bin')
return f'images/{digest}{ext}'
store = DiskMediaStore('runs', key_strategy=by_media_type)
```
If your strategy depends on `ctx.media_type`, the same context must be supplied at read time for `get`/`exists` to find the blob. `DiskMediaStore` rejects strategies that produce absolute paths or `..` segments, to keep writes inside the store directory. `default_key_strategy` is exported if you want to build on it.
## API
| Symbol | Purpose |
|---|---|
| `MediaStore` | Async content-addressed store protocol (`put` / `get` / `exists` / `public_url` / `get_metadata`) |
| `DiskMediaStore`, `SqliteMediaStore`, `S3MediaStore` | Concrete stores |
| `MediaContext` | Per-operation context (media type, filename, tags) threaded through store operations |
| `KeyStrategy`, `default_key_strategy` | On-store key layout |
| `PublicUrlResolver`, `make_static_public_url` | Resolve a stored URI to a public URL |
| `externalize_media`, `restore_media` | Walk a message node to externalize / rehydrate payloads |
| `media_uri_for`, `parse_media_uri` | Compute and parse a `media+sha256://` URI |
Source: [`pydantic_ai_harness/media/`](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/media/).
## Related
- [Step Persistence](step-persistence.md) -- the first consumer of these stores, externalizing `BinaryContent` in run snapshots.
docs/nav.json
+24
View File
@@ -0,0 +1,24 @@
[
{
"label": "Pydantic AI Harness",
"items": [
{ "label": "Overview", "slug": "index" },
{ "label": "Code Mode", "slug": "code-mode" },
{ "label": "FileSystem", "slug": "filesystem" },
{ "label": "Shell", "slug": "shell" },
{ "label": "Context", "slug": "context" },
{ "label": "Pydantic AI Docs", "slug": "pydantic-ai-docs" },
{ "label": "Compaction", "slug": "compaction" },
{ "label": "Overflowing Tool Output", "slug": "overflowing-tool-output" },
{ "label": "Step Persistence", "slug": "step-persistence" },
{ "label": "Media", "slug": "media" },
{ "label": "Subagents", "slug": "subagents" },
{ "label": "Dynamic Workflow", "slug": "dynamic-workflow" },
{ "label": "Planning", "slug": "planning" },
{ "label": "Runtime Authoring", "slug": "runtime-authoring" },
{ "label": "Guardrails", "slug": "guardrails" },
{ "label": "Managed Prompt", "slug": "managed-prompt" },
{ "label": "ACP", "slug": "acp" }
]
}
]
docs/overflowing-tool-output.md
+250
View File
@@ -0,0 +1,250 @@
---
title: Overflowing Tool Output
description: Reduce oversized tool returns when they are produced -- truncate, spill to a queryable file, or summarize -- so a large payload does not persist in history.
---
# Overflowing Tool Output
`OverflowingToolOutput` reduces a tool return that is large enough to dominate the context
window. Tool returns persist in history as `ToolReturnPart`s, so an oversized one is re-sent
on every later model request, paying its token cost for the rest of the run. This capability
intercepts a return when it is produced, reduces it once, and lets the reduced form persist --
the reduction is not recomputed per request.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/overflowing_tool_output/)
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
## The problem
A tool can return a payload large enough to dominate the context window: a big file read, a
verbose log, a large JSON document. Because tool returns persist in history, an oversized one
is re-sent on every later model request, paying its token cost for the rest of the run.
This is the overflow-to-file follow-up the [compaction](compaction.md) capability names as out
of scope: it moves large tool outputs *out* of the window at production time, rather than
compressing or dropping context already inside it.
## The three modes
| Mode | Cost | Lossy? | What the model gets |
|---|---|---|---|
| `Truncate` | zero-LLM | yes | A head / tail / head+tail clamp of the text |
| `Spill` | zero-LLM | no | A handle + preview + shape sketch; full payload read back on demand |
| `Summarize` | one LLM call | yes | A size-gated summary (inherits the run's model by default) |
`Spill` is lossless: the full payload is persisted and the model reads slices of it through
the registered `read_tool_result(handle, offset, limit, from_end, pattern)` tool (the Claude
Code pattern, the core [#4352](https://github.com/pydantic/pydantic-ai/issues/4352) design).
That tool is bounded: `offset >= 0`, `limit` clamped to a built-in line cap, the joined output
capped, and `pattern` is a literal substring (not a regex), so a model-supplied value cannot
hang the host with catastrophic backtracking.
## Usage
Construct an `Agent` with `OverflowingToolOutput()` in its `capabilities`. With no arguments it
uses the default band: spill returns of 10,000 characters or more, with a bounded truncation
fallback if the store cannot accept the write.
```python
from pydantic_ai import Agent
from pydantic_ai_harness.overflowing_tool_output import OverflowingToolOutput
agent = Agent('openai:gpt-4o', capabilities=[OverflowingToolOutput()])
```
The capability registers a single `read_tool_result` tool so the model can page back into any
spilled payload. Its own returns are exempt from reduction.
## Bands: combine the modes
Configure an ordered list of size `bands`. Each band is a `(over, action)` pair: when a
return's measured size reaches `over`, its action runs. The band with the largest threshold
that fits wins; anything below the smallest threshold passes through.
```python
from pydantic_ai import Agent
from pydantic_ai_harness.overflowing_tool_output import (
Band,
OverflowingToolOutput,
Spill,
Summarize,
Truncate,
)
agent = Agent(
'openai:gpt-4o',
capabilities=[
OverflowingToolOutput(
bands=[
Band(over=100_000, action=Spill()), # huge: keep losslessly, read back on demand
Band(over=20_000, action=Summarize()), # large: compress with the run's model
Band(over=5_000, action=Truncate()), # medium: cheap clamp
],
# below 5,000: passthrough
)
],
)
```
The default band, when you pass no `bands`, is `Spill(then=Truncate())` at a 10,000-character
threshold: lossless when a store accepts the write, a bounded truncation otherwise -- zero LLM
cost and no silent drop.
`Passthrough()` is an explicit no-op action for `bands` or `per_tool` lists, leaving matching
returns untouched.
### Fallbacks with `then`
Every action takes an optional `then`, applied when the action cannot run: a `Spill` whose
store errors, a `Truncate` / `Summarize` on a binary payload, a `Summarize` whose model call
raises. `then` chains, so `Summarize(then=Spill(then=Truncate()))` degrades summarize ->
spill -> truncate.
### Per-tool overrides and filtering
`per_tool` replaces the global band list for named tools (file reads to `head`, logs to
`tail`); `tool_filter` (a `ToolSelector`) scopes which tools the capability touches at all.
```python
from pydantic_ai import Agent
from pydantic_ai_harness.overflowing_tool_output import (
Band,
OverflowingToolOutput,
Truncate,
TruncationStrategy,
)
agent = Agent(
'openai:gpt-4o',
capabilities=[
OverflowingToolOutput(
per_tool={
'read_file': [Band(over=8_000, action=Truncate(strategy=TruncationStrategy.head))],
'run_shell': [Band(over=8_000, action=Truncate(strategy=TruncationStrategy.tail))],
},
tool_filter=['read_file', 'run_shell', 'search'],
)
],
)
```
`TruncationStrategy` has three members: `head` (keep the first characters, good for headers and
schemas), `tail` (keep the last characters, good for build and test output where errors land
last), and `head_tail` (keep both ends, elide the middle -- the default).
## Both `return_value` and `content` are reduced
A `ToolReturn` carries a `return_value` and an optional `content` that core renders as a
separate, model-visible part which also persists in history. This capability measures and
reduces both with the same band logic (they spill to distinct handles). Text `content` is
reduced in place; non-text `content` (multimodal parts) that overflows is left unreduced with
a `warnings.warn`, since it cannot be safely truncated.
## Size unit
Thresholds are measured in characters by default. Set `over_tokens=True` to measure in
estimated tokens (the same ~4-chars-per-token heuristic as [compaction](compaction.md)); pass a
`tokenizer` callable for accuracy. `Truncate.max_chars` is always characters -- truncation is a
character operation regardless of the threshold unit. Set `strip_ansi=True` to strip ANSI
escape sequences from text returns before measuring and reducing.
## Spill store
Spilled payloads go through the narrow `OverflowStore` protocol. The default `LocalFileStore`
writes one file per `(run_id, tool_call_id, retry)` under a stable root directory and keeps it
after the run, so a later `read_tool_result` -- in this run or a subsequent agent/run -- can
still reach it. The handle is backend-addressable (a relative key), not an absolute local
path, so a durable backend (Temporal, a blob store, or the core `ExecutionEnvironment`
workspace once #4352 lands) can resolve the same handle in another process. Supply your own
backend with `store=...`.
```python
from typing import Protocol
class OverflowStore(Protocol):
async def write(self, key: str, data: bytes) -> str: ... # returns a handle
async def read(self, handle: str) -> bytes: ...
```
### Security model (shared root, not isolation)
The store root is stable and shareable on purpose -- spilled files must be readable by a later
agent or run -- so security does not come from per-instance isolation. It comes from two
mechanisms: the root is created with `0700` (owner-only) permissions, and `read` resolves the
target (following symlinks) and rejects anything that escapes the root via symlink, `..`, or
an absolute path. Handle segments are also sanitized so a crafted handle cannot traverse out.
### Cleanup: keep-forever by default, opt-in TTL pruning
By default the store keeps spilled files forever -- deleting on run end would break a later
agent that still wants to read a spill. To bound disk use, opt into age-based pruning:
```python
from datetime import timedelta
from pydantic_ai import Agent
from pydantic_ai_harness.overflowing_tool_output import LocalFileStore, OverflowingToolOutput
store = LocalFileStore(cleanup_after=timedelta(hours=6)) # default: None = keep forever
agent = Agent('openai:gpt-4o', capabilities=[OverflowingToolOutput(store=store)])
```
When set, a `write` schedules a background prune (a daemon thread, off the hot path) that
deletes files whose modification time (`st_mtime`) is older than `cleanup_after`. Pruning is
non-blocking and non-erroring: any failure is caught and surfaced via `warnings.warn`, never
propagated into the agent run, so cleanup can never fail a run or block the hot path.
Last-read time (`st_atime`) is unreliable on `noatime`/`relatime` mounts and is not used.
Prefer external cleanup (cron, a sweeper) over the in-process TTL? Point it at the store root
and delete by mtime:
```python
import time
from pathlib import Path
root = Path('/tmp/pyai_harness_overflow') # or your configured base_dir
cutoff = time.time() - 6 * 3600
for path in root.rglob('*'):
if path.is_file() and path.stat().st_mtime < cutoff:
path.unlink(missing_ok=True)
```
## Usage accounting
A `Summarize` call is a real request to the model, so its full usage -- tokens and the
request itself -- folds into the run's `ctx.usage`, exactly like `SummarizingCompaction`. No
token caps are imposed on the summary call. A `UsageLimits` request limit will see it.
By default `Summarize` inherits the running agent's model (`ctx.model`). Pass a model id or
instance to `Summarize(model=...)` to override, or a `summarize` callable to bypass the
built-in prompt entirely. The `summary_prompt` template on the capability must contain both
`{tool_name}` and `{output}` placeholders.
## Edge cases
- Binary returns spill verbatim and are never stringify-truncated; `Truncate` / `Summarize`
on binary fall through to `then`.
- Structured / nested returns spill (or summarize) by preference -- truncating JSON produces
invalid JSON. `Spill` includes a one-line shape sketch of the top level.
- `ModelRetry` and tool errors never reach this hook (they are raised, not returned), so the
model always gets the full error it needs to recover.
- A large `ToolReturn.content` is reduced with the same bands as `return_value`; non-text
content that overflows is left unreduced with a warning.
- Multiple oversized returns in one step get distinct handles (keyed per `tool_call_id`);
retries get distinct handles too (keyed per `retry`), so a retried call never clobbers the
earlier attempt's spill.
## Relationship to other capabilities
- Distinct from [compaction](compaction.md), which compresses or drops context already inside
the window; this capability moves large tool outputs out of the window at production time.
- Consumes core [#4352](https://github.com/pydantic/pydantic-ai/issues/4352) (the canonical
queryable-file primitive) through the `OverflowStore` seam once it lands.
- Distinct from `ClampOversizedMessages`, which clamps runaway model responses, not tool
returns.
## API reference
::: pydantic_ai_harness.overflowing_tool_output.OverflowingToolOutput
docs/planning.md
+143
View File
@@ -0,0 +1,143 @@
---
title: Planning
description: Give an agent a structured, self-updating task plan through a single write_plan tool, without ever invalidating the prompt cache.
---
# Planning
`Planning` gives the model a structured, self-updating task plan through a single `write_plan` tool -- and surfaces the current plan back to the model every turn without ever invalidating the prompt cache.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/planning/)
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
## The problem
Long agentic runs drift: the model loses track of what it set out to do and what's left. The usual fix -- keep a running plan and re-inject it into the system prompt each turn -- invalidates the prompt cache. The system prompt sits at the front of the request, so every plan edit changes the cached prefix and forces the whole conversation to be re-processed at full token price.
## The solution
`Planning` gives the model one tool, `write_plan`, that owns the plan (whole-plan replacement -- pass the full list every call, no indices). The current plan is surfaced back to the model as an ephemeral reminder appended to the tail of each request, behind a cache breakpoint:
- The reminder is added in `wrap_model_request`, which runs *after* the durable history is persisted, so it reaches the model but is never written to `message_history`. No reminders accumulate across turns.
- A `CachePoint` is placed immediately *before* the reminder, so the cached prefix (tools + system + real conversation) stays byte-identical turn over turn. Only the reminder falls outside the cache.
So the plan stays current in the model's view while the cached prefix is never invalidated; the only added cost is re-reading the reminder each turn.
## Usage
Construct an `Agent` with `Planning()` in its `capabilities`. The `write_plan` tool is registered automatically, and the static usage guidance is added to the system prompt:
```python
from pydantic_ai import Agent
from pydantic_ai_harness.planning import Planning
agent = Agent('anthropic:claude-sonnet-4-6', capabilities=[Planning()])
result = agent.run_sync('Refactor the auth module and add tests.')
print(result.output)
```
## The tool
| Tool | Purpose |
|---|---|
| `write_plan(items)` | Create or replace the full plan. The model passes the entire ordered list every time, including unchanged, completed, and cancelled steps. |
Each item is a `content` string plus a `status` (`pending`, `in_progress`, `completed`, `cancelled`). The convention -- stated in the guidance and noted in the tool's reply -- is to keep exactly one step `in_progress`.
There is no `get_plan` tool: the current plan is already in the model's context via the tail reminder every turn.
## Why whole-plan replacement
Addressing steps by mutable integer index (insert/remove/reorder) is error-prone for both the code (index bookkeeping) and the model (indices it just saw can go stale within a turn). Restating the whole plan each call removes that: there are no indices to track, and a later call can't corrupt partial state. For short plans the token cost is negligible.
## Caching guarantee
The plan is never injected into the system prompt or instructions. Static usage guidance goes there (cache-stable); only the mutable plan rides the ephemeral tail reminder. Across turns:
- the durable history grows append-only and is replayed byte-identically, so the whole prefix is a cache hit;
- the reminder and its `CachePoint` live only in the per-request copy, so they can't invalidate anything and aren't persisted.
`CachePoint` is supported on Anthropic and Amazon Bedrock; on providers without prompt caching it's simply ignored (nothing to bust).
## Configuration
```python
from pydantic_ai_harness.planning import Planning
Planning(
guidance=None, # static system-prompt guidance; None = default, '' = omit
cache_ttl='5m', # TTL for the cache breakpoint before the reminder ('5m' | '1h')
)
```
- `guidance` -- static planning guidance added to the system prompt. It is identical on every request, so it stays cache-stable. Leave it as `None` for the built-in default, or set `''` to omit guidance entirely.
- `cache_ttl` -- TTL for the cache breakpoint placed before the plan reminder. One of `'5m'` or `'1h'`.
## Observing the plan
Plan state is per-run (a fresh, isolated plan each run via `for_run`), so it doesn't live on the `Planning()` instance you construct. To see the final plan, read the most recent `write_plan` tool return from the run's messages -- its content is the rendered plan:
```python
from pydantic_ai import Agent
from pydantic_ai.messages import ToolReturnPart
from pydantic_ai_harness.planning import Planning
agent = Agent('anthropic:claude-sonnet-4-6', capabilities=[Planning()])
result = agent.run_sync('Refactor the auth module and add tests.')
plans = [
part.content
for message in result.all_messages()
for part in message.parts
if isinstance(part, ToolReturnPart) and part.tool_name == 'write_plan'
]
latest_plan = plans[-1] if plans else None
print(latest_plan)
```
## Composition
`Planning` contributes a single leaf toolset (`write_plan`), some static instructions, and a `wrap_model_request` hook. It does not wrap or intercept other toolsets, so it composes cleanly alongside other capabilities and your own tools in the same `Agent(..., capabilities=[...])`.
The tail reminder is only appended when the last message in the request is a `ModelRequest` and the plan is non-empty, so an empty plan adds nothing to the request. Because the reminder and its `CachePoint` live only in the per-request copy and never enter the durable history, `Planning` is safe with message-history replay and does not accumulate stale reminders across turns.
## Agent spec (YAML/JSON)
`Planning` works with Pydantic AI's [agent spec](/ai/core-concepts/agent-spec/) feature for defining agents in YAML or JSON:
```yaml
# agent.yaml
model: anthropic:claude-sonnet-4-6
capabilities:
- Planning: {}
```
```python
from pydantic_ai import Agent
from pydantic_ai_harness.planning import Planning
agent = Agent.from_file('agent.yaml', custom_capability_types=[Planning])
result = agent.run_sync('...')
print(result.output)
```
Pass `custom_capability_types` so the spec loader knows how to instantiate `Planning`. Arguments can be passed in the YAML too:
```yaml
capabilities:
- Planning:
cache_ttl: '1h'
```
## Further reading
- [Pydantic AI capabilities](/ai/core-concepts/capabilities/)
- [Hooks](/ai/core-concepts/hooks/) -- `wrap_model_request` is the ephemeral injection point used here
- [Anthropic prompt caching](https://docs.claude.com/en/docs/build-with-claude/prompt-caching)
- [Code Mode](code-mode.md) -- another prompt-cache-aware harness capability
## API reference
::: pydantic_ai_harness.planning.Planning
docs/pydantic-ai-docs.md
+90
View File
@@ -0,0 +1,90 @@
---
title: Pydantic AI Docs
description: Give an agent a tool that locates and returns Pydantic AI documentation on demand instead of preloading it into the system prompt.
---
# Pydantic AI Docs
`PyaiDocs` gives an agent a single tool, `read_pyai_docs(topic)`, that locates a Pydantic AI documentation page and returns it verbatim. Nothing is bundled into context up front. Each call resolves the topic from a configured local checkout first, then falls back to fetching the page from `pydantic/pydantic-ai:main`, so it works whether or not you have a local checkout (the remote fallback needs network access).
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/docs/)
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
## The problem
An agent that authors Pydantic AI capabilities, hooks, tools, or toolsets needs the current docs for those APIs. Preloading the docs into the system prompt spends context the agent rarely needs in full, and pins a snapshot that drifts from `main`.
## The solution
`PyaiDocs` exposes one tool, `read_pyai_docs(topic)`, that locates the requested page and returns it verbatim. Each call resolves the topic from a configured local checkout first, then falls back to fetching the page from `pydantic/pydantic-ai:main`, so it works whether or not you have a local checkout (the remote fallback needs network access).
The available topics are `capabilities`, `hooks`, `tools`, `tools-advanced`, `toolsets`, and `agent`.
## Usage
Construct an `Agent` with `PyaiDocs()` in its `capabilities`. Point `local_docs_path` at a local Pydantic AI docs checkout to read from disk first, or omit it to always fetch from the remote source:
```python
from pathlib import Path
from pydantic_ai import Agent
from pydantic_ai_harness.docs import PyaiDocs
agent = Agent(
'anthropic:claude-sonnet-4-6',
capabilities=[PyaiDocs(local_docs_path=Path('~/pydantic/ai/base/docs').expanduser())],
)
result = agent.run_sync('Read the toolsets docs, then explain how to build a FunctionToolset.')
print(result.output)
```
The capability also adds a short static instruction telling the model that the `read_pyai_docs` tool exists and to read the relevant topic before authoring or modifying a Pydantic AI capability, hook, tool, or toolset, rather than relying on memory. The instruction is cache-stable, so it does not invalidate the prompt-cache prefix between turns.
## Resolution order
Each call resolves in this order:
1. **Local checkout** -- when `local_docs_path` (or the `PYDANTIC_AI_HARNESS_DOCS_PATH` env var) is set and `{path}/{topic}.md` exists, that file is read and returned.
2. **Remote fetch** -- otherwise the page is fetched from `https://raw.githubusercontent.com/pydantic/pydantic-ai/main/docs/{topic}.md`.
3. **Neither resolves** -- a descriptive error naming the local path tried and the URL.
The capability never runs git. Keep the local checkout current yourself; the remote path always reads `main`, so it is the fresh fallback.
`local_docs_path` takes precedence over the `PYDANTIC_AI_HARNESS_DOCS_PATH` env var. Both have `~` expanded, so a raw `~/...` path resolves to the local checkout instead of silently falling through to the remote source. With neither set, every call goes straight to the remote source.
## Configuration
| Option | Default | Purpose |
| --- | --- | --- |
| `local_docs_path` | `None` | Local pyai docs checkout to read first. Falls back to the `PYDANTIC_AI_HARNESS_DOCS_PATH` env var, then to the remote source. |
| `cache` | `True` | Memoize each returned doc in-process for the capability's lifetime, so a topic is read or fetched at most once. |
Caching lives on the capability instance and is shared across the toolsets it builds, so a memoized topic survives multiple agent runs that reuse the same `PyaiDocs`. Set `cache=False` to re-read or re-fetch on every call -- useful when the local checkout changes underneath a long-lived capability.
## Agent spec (YAML/JSON)
`PyaiDocs` works with Pydantic AI's [agent spec](/ai/core-concepts/agent-spec/) feature for defining agents in YAML or JSON. Its serialization name is `PyaiDocs`:
```yaml
# agent.yaml
model: anthropic:claude-sonnet-4-6
capabilities:
- PyaiDocs: {}
```
```python
from pydantic_ai import Agent
from pydantic_ai_harness.docs import PyaiDocs
agent = Agent.from_file('agent.yaml', custom_capability_types=[PyaiDocs])
result = agent.run_sync('...')
print(result.output)
```
Pass `custom_capability_types` so the spec loader knows how to instantiate `PyaiDocs`.
## API reference
::: pydantic_ai_harness.docs.PyaiDocs
docs/runtime-authoring.md
+95
View File
@@ -0,0 +1,95 @@
---
title: Runtime Authoring
description: Let an agent write, validate, and persist real pydantic-ai capabilities at runtime, live on the next run.
---
# Runtime Authoring
`RuntimeAuthoring` lets an agent author, validate, and persist real pydantic-ai capabilities while it runs. It exposes three tools that let the model write a capability class to disk as Python source, validate it immediately, and manage the set of authored capabilities. Each authored capability is a real `pydantic_ai.capabilities.AbstractCapability` subclass, so it can contribute instructions, model settings, a toolset, native tools, or a lifecycle hook.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/runtime_authoring/)
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
## The problem
A coding agent often discovers, mid-task, that it wants a behavior its host does not yet have: a guardrail, an extra instruction, a tool, a request hook. The capability surface to express that already exists -- but normally only a developer can write a capability class, wire it into the agent, and restart. The agent itself cannot extend its own host while it runs.
## The solution
`RuntimeAuthoring` exposes three tools:
- `author_capability(name, code)` -- write `code` to `<directory>/<name>.py`, import it, and validate it. Validation requires exactly one `pydantic_ai.capabilities.AbstractCapability` subclass that constructs with no arguments; the side-effect-free static getters (`get_instructions`, `get_toolset`, `get_native_tools`, `get_model_settings`, `get_serialization_name`) are exercised. The async lifecycle hooks are not run -- they need a live `RunContext`.
- `list_authored_capabilities()` -- list authored capabilities with their status and any validation error.
- `disable_authored_capability(name)` -- stop a capability from being injected on the next run.
A "hook" is not a standalone object in pydantic-ai -- it is a method on a capability. So authoring a hook means authoring a capability that overrides one lifecycle method. A single overridden hook is a valid capability.
## Usage
Construct `RuntimeAuthoring` with a `directory` for the authored files, then add it to the agent's `capabilities`:
```python
from pathlib import Path
from pydantic_ai import Agent
from pydantic_ai_harness.runtime_authoring import RuntimeAuthoring
authoring = RuntimeAuthoring(directory=Path('.authored'))
agent = Agent('anthropic:claude-sonnet-4-6', capabilities=[authoring])
```
The agent can now call `author_capability`, `list_authored_capabilities`, and `disable_authored_capability`. `RuntimeAuthoring` also contributes static, cache-stable system-prompt guidance explaining these tools. Leave `guidance=None` for the default text, or pass your own string; set `guidance=''` to omit it entirely.
## Activation boundary
A capability **cannot** be added to a live, already-executing run. pydantic-ai resolves the effective capability set once at the start of each run (the run's root capability is fixed; there is no setter). So an authored capability is live on the **next** `agent.run(...)`, not the run that authored it. Authoring writes and validates the capability immediately, but its tools and hooks only exist once the next run's toolset and capability chain are assembled at run start.
### Integration contract
The orchestrator drives the loop, so it owns the one-line contract: thread the store's active capabilities into each run via `agent.run(..., capabilities=...)`. With that in place, the authored capability is live on the very next loop iteration -- no process restart:
```python
from pathlib import Path
from pydantic_ai import Agent
from pydantic_ai_harness.runtime_authoring import RuntimeAuthoring
authoring = RuntimeAuthoring(directory=Path('.authored'))
agent = Agent('anthropic:claude-sonnet-4-6', capabilities=[authoring])
history = None
done = False
next_prompt = 'Start the task.'
while not done:
extra = authoring.store.load_active()
result = await agent.run(next_prompt, message_history=history, capabilities=extra)
history = result.all_messages()
# ... decide `next_prompt` and `done` from `result` ...
```
`authoring.store` is the disk-backed `CapabilityStore` over the same `directory`. `store.load_active()` re-imports and re-constructs every active authored capability for injection into the next run. Entries that fail to load (corrupt source, construction error) are skipped, not raised, so one bad capability never blocks the rest.
## Persistence
Authored capabilities persist to disk: each is one `<directory>/<name>.py` file, indexed by a sibling `manifest.json`. A fresh process picks them up by constructing a new `RuntimeAuthoring` over the same `directory` and calling `store.load_active()`.
`manifest.json` records each capability's name, module file, class name, status (`active` or `disabled`), and last validation error. That is the surface a UI can read to show what the agent has authored. The manifest is written atomically (temp file plus `os.replace`), so a crash mid-write never leaves a partial file that reads back as "no capabilities".
Capability names must be lowercase letters, digits, and underscores, starting with a letter. Reusing a name replaces the previous capability of that name. A code that imports but fails validation is still written to disk (so it can be inspected) and recorded with its `last_error` set; `load_active()` skips it.
## Trust boundary
`RuntimeAuthoring` executes arbitrary Python in-process at import, construction, and run time. That is the same trust boundary an agent that already runs shell commands and edits files operates under, which is the deliberate choice here. Do not point it at a directory whose contents you would not run yourself, and treat authored capabilities as code the agent is executing on your host.
Because authored capabilities hold live code, they are not spec-serializable (`get_serialization_name()` returns `None`) and are persisted as source rather than as an [agent spec](/ai/core-concepts/agent-spec/).
## Typing
Imported authored code is dynamic, but nothing typed `Any` crosses back into the harness: every value pulled from an authored module is narrowed with `isinstance`/`issubclass` before use, and loaded instances are typed `AbstractCapability[object]`. Because `AgentDepsT` is contravariant, an `AbstractCapability[object]` is accepted by any agent's `capabilities=` parameter.
## API reference
::: pydantic_ai_harness.runtime_authoring.RuntimeAuthoring
::: pydantic_ai_harness.runtime_authoring.CapabilityStore
docs/shell.md
+255
View File
@@ -0,0 +1,255 @@
---
title: Shell
description: Give a Pydantic AI agent shell command execution with allow/deny controls, environment scrubbing, and managed background processes.
---
# Shell
`Shell` gives an agent the ability to run shell commands, with allow/deny
controls, environment scrubbing, and managed background processes. It exposes
command-execution tools rooted at a working directory and cleans up any
background processes automatically when the agent run ends.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/shell/)
## The problem
Agents frequently need to run a build, a test suite, a linter, or a quick
`grep`. Wiring up subprocess handling -- streaming output, timeouts, truncation,
killing runaway processes, and cleaning up background jobs at the end of a run --
is fiddly boilerplate that every agent reinvents.
`Shell` bundles that plumbing into a single [capability](/ai/core-concepts/capabilities/):
configurable allow/deny lists, output truncation tuned to keep the useful tail,
optional sticky working directory, environment control that can keep host
secrets out of spawned commands, and automatic cleanup of background processes
when the run finishes.
## Usage
Construct `Shell` with a working directory and pass it to an `Agent` via the
`capabilities` parameter:
```python
from pydantic_ai import Agent
from pydantic_ai_harness import Shell
agent = Agent(
'anthropic:claude-sonnet-4-6',
capabilities=[Shell(cwd='./workspace', allowed_commands=['ls', 'cat', 'rg'])],
)
result = agent.run_sync('List the Python files and summarize the largest one.')
print(result.output)
```
By default `Shell` runs in the current directory with the built-in destructive-command
denylist active -- `Shell()` alone is a working (if permissive) configuration.
## Tools
`Shell` contributes four tools to the agent:
| Tool | Purpose |
|---|---|
| `run_command` | Run a command synchronously and return labelled stdout/stderr plus exit code. Honors a per-call or default timeout. |
| `start_command` | Launch a long-running command (server, watcher) in the background; returns an ID. |
| `check_command` | Report the status and accumulated output of a background command. |
| `stop_command` | Terminate a background command and return its final output. |
`run_command` accepts an optional `timeout_seconds` argument that overrides
`default_timeout` for a single call. `check_command` and `stop_command` take the
`command_id` string returned by `start_command`.
Output is labelled with `[stdout]` / `[stderr]` markers and an `[exit code: N]`
line on non-zero exit. When it exceeds `max_output_chars` the **tail** is kept
(the head is dropped), so errors, stack traces, and the `[stderr]` section --
which all land at the end -- survive truncation.
## Command controls
Two mutually exclusive lists decide which executables may run, plus filters for
shell operators and interactive commands:
| Field | Effect |
|---|---|
| `allowed_commands` | If non-empty, only these executables may run (allowlist). |
| `denied_commands` | These executables are always rejected (denylist). |
| `denied_operators` | Shell operators (e.g. `>`, `>>`, `\|`) that are rejected when present. |
| `allow_interactive` | If `False` (default), commands that expect a TTY (`vi`, `sudo`, `ssh`, ...) are blocked. |
`allowed_commands` and `denied_commands` are mutually exclusive -- set one, not
both. Setting both raises a `ValueError` at construction. `denied_commands`
defaults to a list of destructive commands (`rm`, `rmdir`, `mkfs`, `dd`,
`format`, `shutdown`, `reboot`, `halt`, `poweroff`, `init`); pass an empty list
to disable it. The executable name is extracted with `shlex`, so arguments don't
bypass the check.
A denied command surfaces to the model as a
[`ModelRetry`](/ai/tools-toolsets/tools-advanced/#tool-retries), not a hard error:
the run continues and the model can pick an allowed command instead.
!!! warning "Best-effort, not a security boundary"
These command checks are best-effort. A sufficiently motivated agent can
defeat them (e.g. `bash -c '...'`, env-var indirection). For hard
guarantees, run the agent inside OS-level isolation -- a container or
sandbox.
## Environment control
By default a spawned command inherits the agent process's full environment. In a
sandbox that holds LLM API keys, tokens, or other secrets, a command the model
writes can read them. Two fields control what the subprocess sees:
| Field | Effect |
|---|---|
| `env` | Explicit environment that replaces inheritance entirely. The subprocess sees exactly these variables and nothing else. |
| `denied_env_patterns` | Glob patterns (`fnmatch`) for variable names stripped from the base environment. Mirrors `denied_commands`. |
`env` is a hard boundary for inherited environment variables: set it and inherited secrets cannot reach the
subprocess at all (you supply `PATH` and anything else the command needs).
`denied_env_patterns` is a denylist over the inherited environment -- lighter to
configure when you only need to drop a few known-sensitive names. The two
compose: when both are set, patterns also filter the explicit `env`. Leaving
both unset preserves the inherit-everything default.
```python
import os
from pydantic_ai_harness import Shell
from pydantic_ai_harness.shell import LLM_API_KEY_ENV_PATTERNS
# Strip provider credentials from the inherited environment.
Shell(cwd='./repo', denied_env_patterns=LLM_API_KEY_ENV_PATTERNS)
# Or hand the subprocess a fixed environment, inheriting nothing.
Shell(cwd='./repo', env={'PATH': os.environ['PATH'], 'HOME': os.environ['HOME']})
```
`LLM_API_KEY_ENV_PATTERNS` covers common provider prefixes (`ANTHROPIC_*`,
`GATEWAY_*`, `GEMINI_*`, `GOOGLE_*`, `OPENAI_*`, `OPENROUTER_*`) plus
`PYDANTIC_AI_GATEWAY_API_KEY`. It targets LLM credentials only -- it does not
cover other host secrets (a `LOGFIRE_TOKEN`, a GitHub token, cloud
credentials), and its prefixes are coarse, so `GOOGLE_*` also strips
non-credential vars like `GOOGLE_APPLICATION_CREDENTIALS`. Treat it as a
starting point and add your own patterns. It is not the default: stripping
environment variables silently would break agents that rely on inherited
credentials, so it is opt-in.
`env` is enforced at spawn, not applied as a post-hoc filter on a running
process: the subprocess starts with exactly the resolved environment (your
`env`, minus anything `denied_env_patterns` removes from it). That makes it a
real boundary for inherited environment variables, unlike the best-effort command denylist. It is not a full
security boundary: a command running under the same OS identity can still read
host files -- use OS-level isolation for that. The flip side is that a
pattern broad enough to strip `PATH` or `HOME`, or an `env` that omits them, can
break command resolution. External commands may still run via the shell's
built-in default `PATH` on some systems, but don't rely on it -- set `PATH`
explicitly when you replace the environment.
## Background processes
`start_command` writes stdout/stderr to temp files and returns a short ID. Use
`check_command(command_id)` to poll and `stop_command(command_id)` to terminate
and collect final output. Processes are launched in their own session
(`start_new_session`) so the whole process group can be signalled -- `SIGTERM`,
escalating to `SIGKILL` after a grace period.
On run end, the toolset's cleanup terminates every still-running background
process and deletes its temp files. The agent runtime enters toolsets via an
`AsyncExitStack`, so this cleanup runs whether the run succeeds or raises -- an
agent that forgets to call `stop_command` won't leak processes.
```python
from pydantic_ai import Agent
from pydantic_ai_harness import Shell
agent = Agent(
'anthropic:claude-sonnet-4-6',
capabilities=[Shell(cwd='./app', allowed_commands=['npm', 'curl'])],
)
result = agent.run_sync(
'Start the dev server with `npm run dev`, wait for it to boot, '
'then curl http://localhost:3000/health and report the status.'
)
print(result.output)
```
## Working directory
By default each command runs in `cwd` and `cd` has no lasting effect. Set
`persist_cwd=True` to make `cd` sticky across calls: each command is wrapped so
that after it runs, its final working directory is recorded to a private temp
file, and that directory is carried into subsequent calls. The path is only
updated when the command exits `0`, and the record is written out-of-band (not
to stdout) so command output can never spoof the tracked directory.
```python
from pydantic_ai import Agent
from pydantic_ai_harness import Shell
agent = Agent(
'anthropic:claude-sonnet-4-6',
capabilities=[Shell(cwd='.', persist_cwd=True, allowed_commands=['cd', 'ls', 'pwd'])],
)
```
Each run gets a fresh toolset instance, so the tracked directory and any
background processes are isolated between concurrent runs and always start back
at the configured `cwd`.
## Configuration
Every field of `Shell` with its default:
```python
from pydantic_ai_harness import Shell
Shell(
cwd='.', # str | Path -- working directory
allowed_commands=[], # allowlist (mutually exclusive with denied)
denied_commands=[...], # denylist (defaults to destructive commands)
denied_operators=[], # blocked shell operators
default_timeout=30.0, # seconds, per run_command
max_output_chars=50_000, # output cap returned to the model
persist_cwd=False, # make cd sticky across calls
allow_interactive=False, # allow TTY-style commands
env=None, # explicit env, replacing inheritance (None = inherit)
denied_env_patterns=[], # glob patterns stripped from the env
)
```
## Agent spec (YAML/JSON)
`Shell` works with Pydantic AI's
[agent spec](/ai/core-concepts/agent-spec/), so you can declare it in a
config file instead of Python:
```yaml
# agent.yaml
model: anthropic:claude-sonnet-4-6
capabilities:
- Shell:
cwd: ./workspace
allowed_commands: ['ls', 'cat', 'rg', 'pytest']
```
```python
from pydantic_ai import Agent
from pydantic_ai_harness import Shell
agent = Agent.from_file('agent.yaml', custom_capability_types=[Shell])
```
Pass `custom_capability_types` so the spec loader knows how to instantiate
`Shell`.
## Further reading
- [Pydantic AI capabilities](/ai/core-concepts/capabilities/)
- [Toolsets](/ai/tools-toolsets/toolsets/)
## API reference
::: pydantic_ai_harness.Shell
docs/step-persistence.md
+447
View File
@@ -0,0 +1,447 @@
---
title: Step Persistence
description: Record what an agent did at each boundary, save provider-valid snapshots to resume or fork from, and track tool side effects across crashes.
---
# Step Persistence
`StepPersistence` records what an agent did at each boundary, separate from whether the run can be safely resumed. It is the persistence substrate for orchestrators that delegate to sub-agents -- for example, an AICA orchestrator that spawns a `code_librarian` to investigate one symbol, then continues that delegate's investigation with a follow-up question.
It is not a full graph-state checkpoint. Capability-state restore, workspace snapshots, and graph-node resume are out of scope and tracked separately (see `pydantic-ai-harness` issues #149 and #196).
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/step_persistence/)
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
## What it gives you
1. **Append-only step events.** Every interesting boundary (run start/end, model request, tool call, failure) appends a `StepEvent`. A run killed mid-tool-call still leaves a usable event trail.
2. **Continuable snapshots.** A `ContinuableSnapshot` is saved only at boundaries where the message history is provider-valid: every `ToolCallPart` has a matching `ToolReturnPart` or `RetryPromptPart`, with no orphan, duplicate, or out-of-order returns. Pass the snapshot's `messages` back to `Agent.run(message_history=...)` to continue or fork.
3. **Tool-effect ledger.** Every tool call's lifecycle (`started`, `completed`, `failed`) is recorded against `(run_id, tool_call_id)`. After a crash, a tool with a `started` record and no terminal update should be treated as `unknown_after_crash`: the side effect may or may not have happened.
4. **Lineage metadata.** `conversation_id` (sequence) and `parent_run_id` (hierarchy) are independent axes. See [Three-level identity](#three-level-identity).
## Quick start
```python
import asyncio
from pydantic_ai import Agent
from pydantic_ai_harness.step_persistence import StepPersistence, InMemoryStepStore
store = InMemoryStepStore()
librarian = Agent(
'openai:gpt-5',
capabilities=[StepPersistence(store=store, agent_name='code_librarian')],
)
async def main():
await librarian.run('Find ThinkingPartDelta and confirm the callable allowance')
asyncio.run(main())
```
That is the whole setup. `run_id` is always per-`Agent.run` call, matching pydantic_ai's `RunContext.run_id`. For multi-turn logical grouping use `conversation_id=` -- that is the pydantic_ai-native primitive for it (see [Three-level identity](#three-level-identity)).
`run_id` resolution per call:
- **Explicit `run_id='libr-1'`** becomes the id for this one call. This suits single-shot use cases (a deterministic id for testing, replay, debugging, or a one-off scripted run). Reusing one capability instance with the same explicit `run_id` across multiple `.run()` calls raises `ValueError` in `before_run`. The tool-effect ledger is keyed by `(run_id, tool_call_id)` and providers reuse deterministic tool-call ids, so a silent collision would erase the `unknown_after_crash` signal. Use `conversation_id=` for multi-turn grouping instead.
- **`agent_name` set, `run_id` unset** derives `'{agent_name}-{8-char-hex}'`, freshly materialised in `for_run` per `.run()` call. Reusing one capability instance across runs yields distinct ids (`code_librarian-a3b2`, `code_librarian-c9d1`, and so on). This is the recommended default for delegate capabilities.
- **Neither set** falls back to `ctx.run_id` (pydantic_ai's auto-generated id) per `.run()` call, and to a UUID4 if that is absent.
The orchestrator pattern -- one logical agent serving many turns -- uses `conversation_id`, not a shared `run_id`:
```python
import asyncio
from pydantic_ai import Agent
from pydantic_ai_harness.step_persistence import StepPersistence, InMemoryStepStore
store = InMemoryStepStore()
orchestrator = Agent(
'openai:gpt-5',
capabilities=[StepPersistence(store=store, agent_name='orchestrator')],
)
async def main():
for turn in turns:
await orchestrator.run(turn, conversation_id='orch-conv')
# All turns of this orchestrator, chronological:
records = await store.list_runs(conversation_id='orch-conv')
asyncio.run(main())
```
## Three-level identity
The capability mirrors pydantic_ai's identity stack:
| Concept | Definition | Granularity |
| --- | --- | --- |
| `conversation_id` | The dialogue. Resolved by pydantic_ai from the `conversation_id=` argument to `Agent.run`, or the most recent `conversation_id` on `message_history`, or a fresh UUID7. | sequence of runs |
| `run_id` | One `Agent.run` invocation. | one step in the sequence |
| `step_index` | Graph-node count within a run (`ctx.run_step`). | one node within one run |
`StepEvent.conversation_id` and `RunRecord.conversation_id` are populated from `ctx.conversation_id`. So three `.run()` calls sharing one `conversation_id` produce three distinct `run_id`s, all queryable as a group:
```python
import asyncio
async def main():
runs = await store.list_runs(conversation_id='conv-abc') # 3 records, chronological
asyncio.run(main())
```
## Continuing a delegate's investigation
pydantic_ai already has `message_history=` for "carry on with this prior context". `StepPersistence` does not introduce a parallel mechanism. It exposes one helper that loads the most recent provider-valid snapshot:
```python
import asyncio
from pydantic_ai import Agent
from pydantic_ai_harness.step_persistence import (
StepPersistence,
InMemoryStepStore,
continue_run,
)
store = InMemoryStepStore()
librarian = Agent(
'openai:gpt-5',
capabilities=[StepPersistence(store=store, agent_name='code_librarian')],
)
async def main():
# Earlier: tag the first turn with a conversation id so the follow-up can find it.
await librarian.run(
'Find ThinkingPartDelta and confirm the callable allowance',
conversation_id='libr-conv',
)
# Later (possibly a different process):
prior_run = (await store.list_runs(conversation_id='libr-conv'))[-1].run_id
history = await continue_run(store, run_id=prior_run)
await librarian.run(
'Read _apply_provider_details_delta and check the path',
message_history=history,
conversation_id='libr-conv', # keep the conversation grouping
)
asyncio.run(main())
```
`fork_run(store, run_id=...)` returns the same shape but is intended when the caller wants a branched logical run from that snapshot point (the new run gets a fresh `run_id` and probably a fresh `conversation_id`).
### What "safe to continue from" means
`continue_run` only returns the messages of the latest provider-valid snapshot for that `run_id`. Snapshots are written at two boundaries:
- after every `CallToolsNode` completes (all tool calls returned), and
- at `after_run`, as a fallback if the run reached no such boundary.
A run that crashed mid-tool-call has events (`tool_call_started`) but no snapshot for that point. `continue_run` returns the snapshot from the previous safe boundary, not the failed step. If no continuable snapshot exists at all, `continue_run` raises `LookupError`.
## Run lineage: `parent_run_id`
`parent_run_id` is a lineage label, not a functional dependency. It does two things:
- Every `StepEvent` and `RunRecord` carries it, so you can filter and group.
- `store.list_runs(parent_run_id='orch-1')` returns every delegate run pointing at that orchestrator.
It is auto-inferred for in-process delegation: when an orchestrator's tool synchronously calls a delegate's `Agent.run(...)`, the delegate's `StepPersistence` picks up the orchestrator's `run_id` via a `ContextVar` that the orchestrator's `wrap_run` set. No threading required:
```python
import asyncio
from pydantic_ai import Agent
from pydantic_ai_harness.step_persistence import StepPersistence, InMemoryStepStore
store = InMemoryStepStore()
orchestrator = Agent(
'openai:gpt-5',
capabilities=[StepPersistence(store=store, agent_name='orchestrator')],
)
librarian = Agent(
'openai:gpt-5',
capabilities=[StepPersistence(store=store, agent_name='code_librarian')],
)
@orchestrator.tool_plain
async def ask_librarian(question: str) -> str:
result = await librarian.run(question) # parent_run_id auto-filled
return result.output
async def main():
# Tag the orchestrator turn so the lookup below can find its run_id.
await orchestrator.run(
'Where is ThinkingPartDelta defined?',
conversation_id='orch-conv',
)
# All librarian runs now point at the orchestrator's run_id:
orch_run_id = (await store.list_runs(conversation_id='orch-conv'))[-1].run_id
delegates = await store.list_runs(parent_run_id=orch_run_id)
asyncio.run(main())
```
Set `parent_run_id=` explicitly to override (for example, cross-process delegation where `ContextVar`s do not propagate).
`parent_run_id` is distinct from `conversation_id`. The orchestrator and delegate usually live in different conversations (the orchestrator talks to a user; the delegate talks to itself). But they share a parent-child link.
## Inspecting a run tree
`list_runs` returns matches sorted by `started_at` ascending across all backends -- pick the most recent with `[-1]`.
```python
import asyncio
async def main():
# Every delegate of one orchestrator run (chronological)
delegates = await store.list_runs(parent_run_id='orch-3f2a')
# Every run in one dialogue (multi-turn conversation across many .run() calls)
turns = await store.list_runs(conversation_id='conv-abc')
latest_turn = turns[-1]
# Filters combine (AND):
focused = await store.list_runs(
parent_run_id='orch-3f2a',
conversation_id='libr-conv',
)
# Detail per run:
events = await store.list_events(run_id=delegates[0].run_id)
snapshot = await store.latest_snapshot(run_id=delegates[0].run_id)
unresolved = await store.list_unresolved_tool_effects(run_id=delegates[0].run_id)
asyncio.run(main())
```
## Failure recovery
```python
import asyncio
async def main():
# An earlier delegate run died mid-investigation.
events = await store.list_events(run_id='libr-3f2a')
unresolved = await store.list_unresolved_tool_effects(run_id='libr-3f2a')
for record in unresolved:
# status == 'started' with no terminal update -- unknown_after_crash.
print(f'tool {record.tool_name} ({record.tool_call_id}) may or may not have run')
print(f' idempotency_key={record.idempotency_key} '
f'effect_summary={record.effect_summary}')
# Decide whether to resume or branch:
history = await continue_run(store, run_id='libr-3f2a')
# If the unresolved tools were read-only and safe to redo:
await librarian.run('continue investigating', message_history=history,
conversation_id='libr-conv')
# If side effects might have happened and the orchestrator wants a fresh attempt:
history = await fork_run(store, run_id='libr-3f2a')
# ... pass to a new delegate run with a different agent_name / conversation_id.
asyncio.run(main())
```
Side-effect deduplication is the orchestrator's responsibility. Tools that write external state should annotate their in-flight `ToolEffectRecord` via `annotate_tool_effect`:
```python
from pydantic_ai import RunContext
from pydantic_ai_harness.step_persistence import annotate_tool_effect
@orchestrator.tool
async def set_label(ctx: RunContext[Deps], issue: int, label: str) -> str:
await annotate_tool_effect(
store,
ctx,
idempotency_key=f'issue-{issue}::label::{label}',
effect_summary=f'set label {label!r} on issue #{issue}',
)
await github.set_label(issue, label) # the actual side effect
return 'ok'
```
The helper reads the active `run_id` from the `StepPersistence` `ContextVar` and `tool_call_id` / `tool_name` from `ctx`, then merges the metadata into the prior record. It is a no-op when called outside a step-persistence-wrapped tool call. `after_tool_execute` preserves both fields when it writes the terminal `completed` / `failed` entry.
## Backends
- `InMemoryStepStore` -- process-local; great for tests.
- `FileStepStore(directory)` -- directory layout under `<directory>/<run_id>/`:
- `run.json` -- `RunRecord` (lineage)
- `events.jsonl` -- append-only `StepEvent`s
- `tool_effects.jsonl` -- append-only `ToolEffectRecord`s, scoped to this run
- `snapshots/{seq}.json` -- `ContinuableSnapshot`s, named by a per-run monotonic counter (not `step_index`, which would collide when the same `run_id` is reused across `Agent.run` calls, since `ctx.run_step` resets to 0 each call).
- `SqliteStepStore(database='runs.db')` -- single SQLite file with tables `runs`, `events`, `snapshots`, `tool_effects`, and a sibling `media` table for externalized blobs (see [Persisting media](#persisting-media) below). WAL mode is enabled; `tool_effects` upserts per `(run_id, tool_call_id)` so the latest state wins; snapshots use `AUTOINCREMENT seq` to mirror `FileStepStore._next_snapshot_seq`. Pass `connection=` instead of `database=` to share a `sqlite3.Connection` with the rest of your application; the connection must be opened with `check_same_thread=False` because hook calls are dispatched onto a worker thread.
All three implement the same async `StepStore` protocol, so capability hooks never block the event loop on the file/sqlite backends (I/O is dispatched via `anyio.to_thread`).
`FileStepStore` validates `run_id` against `[A-Za-z0-9_.-]{1,200}` (and rejects `..`) to prevent path traversal. Callers passing user-controlled IDs should still sanitise first.
## Persisting media
`BinaryContent` payloads (images, audio, documents, video) inlined as base64 inside a snapshot would balloon every file or row containing the message. Both `FileStepStore` and `SqliteStepStore` externalize any `BinaryContent.data` at or above **64 KiB** through a configured `MediaStore`, leaving a URI reference in the snapshot. Round-trip is transparent: `latest_snapshot(...).messages[*]` returns `BinaryContent` with the original bytes.
| StepStore | Default `media_store` | Where blobs live |
| ------------------- | --------------------------------------- | ------------------------------------- |
| `InMemoryStepStore` | not applicable | bytes stay in the in-memory snapshot |
| `FileStepStore` | `DiskMediaStore(<root>/media/)` | `<root>/media/<sha256>.bin` |
| `SqliteStepStore` | `SqliteMediaStore(database=<same db>)` | sibling `media` table in the same DB |
Override the destination by passing your own `MediaStore`:
```python
from pydantic_ai_harness.step_persistence import FileStepStore
from pydantic_ai_harness.media import S3MediaStore
store = FileStepStore(
'runs',
media_store=S3MediaStore(
bucket='my-bucket',
endpoint='https://<account>.r2.cloudflarestorage.com',
region='auto',
access_key_id=...,
secret_access_key=...,
),
media_threshold_bytes=64 * 1024, # raise or lower if you want
)
```
Opt out entirely (keep bytes inline in the snapshot JSON/row):
```python
from pydantic_ai_harness.step_persistence import FileStepStore, SqliteStepStore
FileStepStore('runs', media_store=None)
SqliteStepStore(database='runs.db', media_store=None)
```
URIs are `media+sha256://<hex>`, content-addressed. The same blob written through any `MediaStore` resolves the same way, so dedup is automatic and moving the underlying storage is a one-line swap. The shipped implementations are:
- `DiskMediaStore(directory)` -- one file per blob at `<directory>/<sha256>.bin`.
- `SqliteMediaStore(database=...)` or `SqliteMediaStore(connection=...)` -- one row per blob (`INSERT OR IGNORE` for content-addressed dedup).
- `S3MediaStore(bucket=, endpoint=, region=, access_key_id=, secret_access_key=)` -- path-style URLs plus handrolled SigV4. Compatible with AWS S3, Cloudflare R2 (`region='auto'`), MinIO, and other S3-compatible providers. PUT/GET/HEAD only -- no multipart, lifecycle, or listing in v1.
### Exposing externalized bytes as URLs
Each store accepts a `public_url=` callable that turns the canonical `media+sha256://<hex>` URI into a URL the model can fetch directly. The forthcoming `MediaExternalizer` capability will use this to swap `BinaryContent` parts for `ImageUrl` / `AudioUrl` / other URL parts before the model sees the message, letting providers fetch big media over the wire without re-encoding bytes into the request body.
Static base URL (public R2 bucket, CDN):
```python
from pydantic_ai_harness.media import S3MediaStore, make_static_public_url
store = S3MediaStore(
bucket='my-bucket',
endpoint='https://<acc>.r2.cloudflarestorage.com',
region='auto',
access_key_id=..., secret_access_key=...,
key_prefix='media/',
public_url=make_static_public_url('https://pub-abc.r2.dev', key_prefix='media/'),
)
```
Presigned or rotating-signature URL -- pass any async callable that takes `(uri, MediaContext)`:
```python
from pydantic_ai_harness.media import MediaContext, S3MediaStore
async def presign(uri: str, ctx: MediaContext) -> str:
key = 'media/' + uri.removeprefix('media+sha256://') + '.bin'
return await my_signer.generate(key, ttl=3600, content_type=ctx.media_type)
store = S3MediaStore(..., public_url=presign)
```
### `MediaContext`, an extensible per-operation bag
Every `MediaStore` method (`put`, `get`, `exists`, `public_url`, `get_metadata`) and both user-supplied callables (`PublicUrlResolver`, `KeyStrategy`) accept a `MediaContext`:
```python
from collections.abc import Mapping
from dataclasses import dataclass, field
@dataclass(frozen=True, kw_only=True)
class MediaContext:
media_type: str | None = None # e.g. 'image/png'
filename: str | None = None # original filename, when known
metadata: Mapping[str, str] = field(default_factory=dict) # user-supplied tags
```
All fields default; new fields are added non-breakingly as use cases emerge. Pass what you have, ignore the rest.
**Persistence by store.** `get_metadata(uri)` round-trips the user-supplied `metadata` mapping on all three stores. `media_type` is also persisted but is not part of what `get_metadata` returns (it is stored for the byte payload itself, for example as the `Content-Type`).
- `SqliteMediaStore` writes `metadata` to a JSON column and `media_type` to a dedicated column.
- `S3MediaStore` sends `metadata` as signed `x-amz-meta-*` headers (ASCII alphanumeric plus dash key names) and `media_type` as `Content-Type`; `get_metadata` reads the `x-amz-meta-*` values back from the HEAD response.
- `DiskMediaStore` writes a sidecar JSON file (`<resolved>.meta.json`) alongside each blob, atomic via tmp plus rename. Sidecars are absent only when the put carried no metadata.
### `key_strategy`: controlling the backend storage path
Default is `<sha256>.bin`. `DiskMediaStore` and `S3MediaStore` accept overrides to fit existing layouts; `SqliteMediaStore` does not (its primary key is the digest, so a user-chosen key would either break dedup or be a no-op):
```python
from pydantic_ai_harness.media import DiskMediaStore, MediaContext
def by_media_type(uri: str, ctx: MediaContext) -> str:
digest = uri.removeprefix('media+sha256://')
ext = {'image/png': '.png', 'image/jpeg': '.jpg'}.get(ctx.media_type or '', '.bin')
return f'images/{digest}{ext}'
store = DiskMediaStore('runs', key_strategy=by_media_type)
```
**Caveat**: if your strategy depends on `context.media_type` (for example, to pick an extension), `get(uri)` and `exists(uri)` will not find the blob unless the same context is supplied at read time. For pure path-organisation strategies (no context dependency) the constraint does not apply.
`DiskMediaStore` rejects strategies that produce absolute paths or paths containing `..` segments, to prevent escaping the store directory.
Separately, all three stores accept a `public_url=` resolver, useful when a CDN, local HTTP server, or signed-URL service fronts the bytes. Without it `public_url(...)` returns `None` (the model never sees a URL unless a resolver is configured and it returns a string).
pydantic_ai providers transparently download bytes from a URL when the target model does not natively accept that URL type, so emitting a URL is always safe: you only ever lose wire savings, never correctness.
!!! note "The future `MediaExternalizer` capability"
When it lands, the composition will be `Agent(capabilities=[MediaExternalizer(store), StepPersistence(...)])` and `StepPersistence` will see already-URL-ified messages, so the externalize walk becomes a no-op. The existing API does not change.
### Persisting to unsupported backends
DynamoDB, Postgres, Redis, GCS, and other backends are out of scope for this release. Write your own `StepStore` (about ten methods on a Protocol) or your own `MediaStore` (three methods) and pass it via `store=` / `media_store=`. Please open an issue if you ship one -- we want to feed the eventual shared adapter layer with N >= 3 real implementations before abstracting.
## What this capability does not do
- It does not restore capability per-run state, graph-node state, retry counters, or in-flight streaming responses.
- It does not deduplicate replayed side effects automatically. Tools that write artifacts, labels, PRs, or external state should call `annotate_tool_effect(store, ctx, ...)` (see [Failure recovery](#failure-recovery)) so the orchestrator can decide whether replay is safe.
- It does not clean up old snapshots or events. Retention is the caller's responsibility.
- It does not emit OpenTelemetry spans. pydantic_ai's `Instrumentation` capability already spans `agent run` / `chat` / `running tool` and populates `gen_ai.agent.name`, `gen_ai.agent.call.id`, `gen_ai.conversation.id` via baggage. A future change may add step-persistence attributes to the active span; that is tracked as a follow-up issue.
## Related
- [Capabilities overview](index.md)
- [Code Mode](code-mode.md)
## API reference
::: pydantic_ai_harness.step_persistence.StepPersistence
docs/subagents.md
+227
View File
@@ -0,0 +1,227 @@
---
title: Subagents
description: Let an agent delegate self-contained tasks to named child agents via a single delegate_task tool, with per-delegate budgets and failure handling.
---
# Subagents
`SubAgents` lets an agent delegate self-contained tasks to named child agents. It takes a sequence of `SubAgent` entries and exposes a single `delegate_task(agent_name, task)` tool. Each delegation runs the chosen sub-agent in its own run -- with its own message history, so it never sees the parent conversation -- and returns its output to the parent.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/subagents/)
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
## The problem
A single agent that does everything accumulates a large tool set and a long context. Splitting the work across specialized sub-agents keeps each context focused, but wiring up delegation by hand means writing a tool per agent, forwarding deps, threading usage limits, and telling the model what it can delegate to.
## The solution
`SubAgents` takes a sequence of `SubAgent` entries and exposes a single `delegate_task(agent_name, task)` tool. Each delegation runs the chosen sub-agent in its own run -- with its own message history, so it never sees the parent conversation -- and returns its output to the parent. The available sub-agents are listed in the system prompt as a static instruction, so the listing stays in the cached prefix.
```python
from pydantic_ai import Agent
from pydantic_ai_harness.subagents import SubAgent, SubAgents
researcher = Agent('anthropic:claude-sonnet-4-6', name='researcher', description='Researches a topic and reports findings')
writer = Agent('anthropic:claude-sonnet-4-6', name='writer', description='Turns notes into polished prose')
orchestrator = Agent(
'anthropic:claude-opus-4-7',
capabilities=[SubAgents(agents=[SubAgent(researcher), SubAgent(writer)])],
)
result = orchestrator.run_sync('Research the history of TLS and write a one-paragraph summary.')
print(result.output)
```
A delegate's name -- how the parent model refers to it, and how it is listed in the prompt -- is the agent's own `name`, or a `SubAgent(name=...)` override. Two delegates resolving to the same name is an error, and an agent with no name and no override is rejected.
## The tool
| Tool | Purpose |
|---|---|
| `delegate_task(agent_name, task)` | Run the named sub-agent on a self-contained task and return its output. |
- The sub-agent runs with its own message history, so `task` must be self-contained.
- An unknown `agent_name` raises `ModelRetry`, so the model can correct itself.
- The result returned to the parent is `str(result.output)`.
## Deps, usage, tools, and capabilities
- **Deps are forwarded.** The parent run's `deps` are passed to each sub-agent, so sub-agents share the parent's `AgentDepsT` (enforced by the type signature -- every sub-agent is an `AbstractAgent[AgentDepsT, Any]`).
- **Usage is shared by default.** The parent's `usage` is passed to each sub-agent run, so token usage aggregates and a parent `usage_limits` applies across the whole agent tree. Set `forward_usage=False` to give each sub-agent run its own accounting.
- **Tools can be inherited.** With `inherit_tools=True`, the parent agent's own tools (registered directly or via `toolsets`) are added to each sub-agent run, on top of the sub-agent's own. Tools contributed by the parent's capabilities are not inherited: they are bound to capability instances registered in the parent run, and would arrive without the hooks and instructions they depend on. Use `shared_capabilities` to give sub-agents a capability. This also excludes the delegate tool itself, so a sub-agent can't recurse into further delegation. Off by default.
- **Capabilities can be shared.** `shared_capabilities` are applied to every sub-agent run -- e.g. give all sub-agents a common guardrail, memory, or planning capability without rebuilding each `Agent`.
- **Sub-agent events can be streamed.** Pass an `event_stream_handler` and it's forwarded to each sub-agent run, so the sub-agent's model-streaming and tool events surface to the caller (the handler receives the sub-agent's own `RunContext`).
## Per-delegate run controls
Each `SubAgent` carries its own budgets, so one delegate's controls do not touch the others. A `SubAgent` with no controls set runs with the `SubAgents` defaults.
```python
from pydantic_ai import Agent
from pydantic_ai.usage import UsageLimits
from pydantic_ai_harness.subagents import SubAgent, SubAgents
reproducer = Agent('anthropic:claude-sonnet-4-6', instructions='Reproduce the reported bug from a minimal script.')
librarian = Agent('anthropic:claude-sonnet-4-6', instructions='Find relevant docs, issues, and prior art.')
orchestrator = Agent(
'anthropic:claude-opus-4-7',
capabilities=[
SubAgents(
agents=[
SubAgent(reproducer, usage_limits=UsageLimits(request_limit=35), timeout_seconds=600, max_calls=1),
SubAgent(librarian, usage_limits=UsageLimits(request_limit=18), timeout_seconds=300, max_calls=2),
]
)
],
)
```
| Field | Effect |
|---|---|
| `usage_limits` | A request/token budget for one delegation. The child runs with its own usage accounting, so the budget counts only that child's requests and tokens (not the parent's or siblings'), even when `forward_usage=True`. The tradeoff: that child's tokens no longer aggregate into the parent's `usage`. Reaching the budget is a soft outcome (see below), not a run-stopping `UsageLimitExceeded`. |
| `timeout_seconds` | A wall-clock budget for one delegation. When the child exceeds it, its run is cancelled and the parent gets a soft steering message instead of hanging on the child. The cancelled child's `event_stream_handler` (if any) stops receiving events without a terminal event. |
| `max_calls` | The maximum number of delegations to this sub-agent per parent run. Once reached, further delegations return a soft budget-exhausted message without running the child. Counts are scoped to one `Agent.run` (a `run_id`) and cleared when it ends, so each parent run and each level of a nested tree budgets independently. |
| `on_failure` | A steering message returned to the parent for any soft degradation of this delegate, in place of the built-in default. Setting it also makes child failures soft (see below). |
| `contain_errors` | Whether an unexpected crash in this delegate is caught and returned to the parent as a bounded `ModelRetry` instead of aborting the parent run (see below). Unset inherits the `SubAgents(contain_errors=...)` default (off). |
## Failure handling
A *soft outcome* returns a steering message to the parent as a normal tool result, so its model reads the message and decides what to do next (rather than immediately re-delegating, which a `ModelRetry` invites). A timeout, a reached `usage_limits` budget, and an exhausted `max_calls` budget are always soft. When `on_failure` is set, the message it carries replaces the built-in default for these outcomes.
A sub-agent run that fails with a *soft model error* (`ModelRetry`, `UnexpectedModelBehavior`, e.g. it exhausted its own retries) is, by default, converted into a `ModelRetry` for the parent -- so the parent's model sees `Sub-agent '<name>' failed: ...` and can react by re-delegating. The delegate tool defaults to `tool_retries=2`, so the parent aborts only after that many consecutive delegate failures; the counter resets after any successful delegation. Raise `tool_retries` to tolerate a flakier sub-agent, or set `None` to inherit the parent agent's default tool retries. Set `on_failure` for a delegate to make its failures soft instead: the child error returns the `on_failure` message as a normal tool result.
Hard errors propagate to stop the whole run. A `UsageLimitExceeded` from a child that has *no* per-delegate `usage_limits` (so it shares the parent's accounting) means the whole tree is out of budget and propagates; a child reaching its *own* `usage_limits` is soft, as above.
An *unexpected crash* -- any other exception the child raises, such as a provider `ModelAPIError`/`FallbackExceptionGroup` or a plain `ValueError` from a bad tool argument -- propagates by default and aborts the parent run. Set `contain_errors=True` (per delegate, or as the `SubAgents` default) to catch it and return it to the parent as a bounded `ModelRetry` instead, so one delegate crash cannot kill the whole run. Containment stays loud: the exception rides the retry message (`Sub-agent '<name>' crashed: ...`), it is logged via the standard `logging` module, and `tool_retries` still bounds consecutive crashes into an abort. This is orthogonal to `on_failure` -- a contained crash always raises the loud retry, never the soft `on_failure` return, so a genuine bug is never masked as success. Cancellation, a shared `UsageLimitExceeded`, pydantic-ai control-flow signals (`CallDeferred`, `ApprovalRequired`, the `Skip*` signals), and `UserError` always propagate regardless of `contain_errors`.
## Discovery
The sub-agents are listed in the system prompt via `get_instructions`, using each agent's `description` (or a `SubAgent(description=...)` override). A sub-agent with no description is listed by name alone.
## Loading sub-agents from disk
A repo's markdown agent definitions become delegates without writing any `Agent` code. By default every `*.md` file under the conventional folders is loaded as a sub-agent, alongside the explicitly-passed `agents`.
```python
from pydantic_ai import Agent
from pydantic_ai_harness.subagents import SubAgents
orchestrator = Agent(
'anthropic:claude-opus-4-7',
capabilities=[SubAgents(inherit_tools=True)], # auto-loads ./.agents/agents/ and ~/.agents/agents/
)
```
`agent_folders` controls where definitions come from. It defaults to `'agents'`, the conventional layout:
- A folder-name `str` (the default `'agents'`): for the project root (cwd) then the home root, load from `<root>/.agents/<name>/`, falling back to `<root>/.claude/<name>/` when `<root>/.agents/` is absent.
- A sequence of paths loads from exactly those folders, in order.
- `None` disables disk loading, exposing only the explicitly-passed `agents`.
### Definition format
A definition is a markdown file with optional frontmatter:
```markdown
---
name: researcher
description: Researches a topic and reports findings
tools: Read, Grep
---
You research topics. Report your findings, each with a source.
```
- `name` is the delegate name (how the parent refers to it and how it is listed). It falls back to the filename stem when absent.
- `description` drives the prompt listing.
- The markdown body becomes the agent's instructions.
- `tools` (or `allowed-tools`) is a comma-separated string or a YAML block list. See "Tools" below.
- `model` and `color` are ignored: the model is inherited from the parent (see below), and `color` has no pyai equivalent.
Frontmatter is read by a small, dependency-free parser limited to those keys (`pyyaml` is not a harness dependency). Full YAML frontmatter is not supported.
### Models and effort
Disk agents inherit the parent run's model by default. Per agent, the caller can override the model and set a thinking/effort level via `agent_overrides`, keyed by the agent's name:
```python
from pydantic_ai_harness.subagents import AgentOverride, SubAgents
SubAgents(
agent_folders='agents',
agent_overrides={'researcher': AgentOverride(model='anthropic:claude-sonnet-4-6', effort='high')},
)
```
Every agent the capability builds runs at a minimum thinking-effort floor. `MINIMUM_EFFORT_FLOOR` and the `clamp_effort(level, floor=...)` helper are exported so an orchestrator can apply the same floor to its own agents (that orchestrator-side application is the caller's responsibility). `clamp_effort` maps `None`/`False` to the floor, leaves `True` (provider-default effort) unchanged, and raises a concrete level below the floor up to it. Effort is applied through pyai's `ModelSettings.thinking`.
### Tools
A disk agent gets no tools by default (`inherit_tools` is `False`); set `inherit_tools=True` to expose the parent's tools to it through the `inherit_tools` mechanism, in which case its `tools` frontmatter is ignored. To map the frontmatter tool names to specific toolsets instead, pass a `tool_resolver`: it receives each tool name (so it can honor entries like `Bash(git:*)`) and returns the toolsets that provide it, or `None` for an unknown name, which is skipped with a warning.
```python
from pydantic_ai_harness.subagents import SubAgents
def resolve(tool_name: str):
return TOOLSETS.get(tool_name) # -> Sequence[AgentToolset[object]] | None
SubAgents(agent_folders='agents', tool_resolver=resolve)
```
### Precedence
When the same name appears in more than one source, the higher-precedence one wins and the others are skipped with a warning: explicitly-passed `agents` first, then the project folder, then the home folder (and, for an explicit path sequence, earlier paths before later ones). A duplicate name within the explicitly-passed `agents` list is still an error.
## Configuration
```python
SubAgents(
agents=(), # Sequence[SubAgent[AgentDepsT]] -- each pairs an agent with its run controls
agent_folders='agents',# folder-name str (convention) | Sequence[Path] | None (disable)
agent_overrides={}, # Mapping[str, AgentOverride] -- per-disk-agent model/effort override
tool_resolver=None, # Callable[[str], Sequence[AgentToolset[object]] | None] -- disk-agent tool mapping
forward_usage=True, # share the parent's usage with sub-agent runs
inherit_tools=False, # expose the parent's own tools to sub-agents (capability tools excluded)
shared_capabilities=(),# capabilities applied to every sub-agent run
event_stream_handler=None, # forwarded to each sub-agent run to stream its events
tool_name='delegate_task',
tool_retries=2, # extra delegate-tool attempts after a sub-agent error before aborting (None inherits the agent default)
contain_errors=False, # default for SubAgent.contain_errors: contain an unexpected crash as a bounded retry
)
```
```python
SubAgent(
agent, # AbstractAgent[AgentDepsT, Any] -- the child agent to run
name=None, # delegate name; defaults to the agent's own `name`
description=None, # prompt-listing description; defaults to the agent's own `description`
usage_limits=None, # per-delegation request/token budget (isolated accounting)
timeout_seconds=None, # per-delegation wall-clock budget
max_calls=None, # max delegations to this sub-agent per parent run
on_failure=None, # steering message for soft degradations of this delegate
contain_errors=None, # contain an unexpected crash as a bounded retry; None inherits the SubAgents default
)
```
`SubAgents` is not serializable via the [agent spec](/ai/core-concepts/agent-spec/) (it holds live `Agent` instances), so `get_serialization_name()` returns `None`.
## Notes
- Sub-agents can themselves have `SubAgents`, forming a tree. Share `usage` (the default) and set a `usage_limits` on the top-level run to bound the whole tree.
- Delegations the model issues in parallel run as independent sub-agent runs.
## Further reading
- [Pydantic AI capabilities](/ai/core-concepts/capabilities/)
- [Multi-agent applications](/ai/guides/multi-agent-applications/)
## API reference
::: pydantic_ai_harness.subagents.SubAgents
::: pydantic_ai_harness.subagents.SubAgent
::: pydantic_ai_harness.subagents.AgentOverride
pydantic_ai_harness/code_mode/README.md
+7 -4
View File
@@ -2,6 +2,8 @@
Replace individual tool calls with a single sandboxed Python execution environment.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/code_mode/)
## The problem
Standard tool calling requires one model round-trip per tool call. An agent that needs to fetch 10 items and process each one makes 11+ model calls -- slow, expensive, and context-heavy.
@@ -57,7 +59,7 @@ The [harness Quick start](../../README.md#quick-start) wires `CodeMode` up again
[![CodeMode's first run_code: parallel asyncio.gather over three HN feeds, then a dedupe and a score filter](../../docs/images/code-mode-trace.png)](https://logfire-us.pydantic.dev/public-trace/84bcf123-2106-49da-9f6f-5c26395339bb?spanId=7650806a0785b946)
**[See the full Logfire trace →](https://logfire-us.pydantic.dev/public-trace/84bcf123-2106-49da-9f6f-5c26395339bb?spanId=7650806a0785b946)** Each `run_code` span fans out into the tool calls the model issued from inside the sandbox -- the easiest way to understand what code mode actually did. See the [Pydantic AI Logfire docs](https://ai.pydantic.dev/logfire/) for setup details.
**[See the full Logfire trace ->](https://logfire-us.pydantic.dev/public-trace/84bcf123-2106-49da-9f6f-5c26395339bb?spanId=7650806a0785b946)** Each `run_code` span fans out into the tool calls the model issued from inside the sandbox -- the easiest way to understand what code mode actually did. See the [Pydantic AI Logfire docs](https://ai.pydantic.dev/logfire/) for setup details.
## Installation
@@ -239,19 +241,20 @@ Code runs inside [Monty](https://github.com/pydantic/monty), a sandboxed Python
- No class definitions
- No third-party imports (allowed stdlib: `sys`, `typing`, `asyncio`, `math`, `json`, `re`, `datetime`, `os`, `pathlib`)
- No wall-clock or timing primitives by default (`asyncio.sleep`, `datetime.now()`, `date.today()`, `time`) -- `datetime.now()`/`date.today()` become available with an `os_access` handler (above); `asyncio.sleep`/`time` never do
- No wall-clock or timing primitives by default (`asyncio.sleep`, `datetime.datetime.now()`, `datetime.date.today()`, `time`) -- `datetime.datetime.now()`/`datetime.date.today()` become available with an `os_access` handler (above); `asyncio.sleep`/`time` never do
- No `import *`
- Filesystem I/O needs an `os_access` handler or a `mount`; `os.getenv`/`os.environ` need an `os_access` handler
- Tools requiring approval or with deferred execution are excluded from the sandbox
- Tools requiring approval or with deferred (`CallDeferred`) execution are sandboxed like any other tool; without a `HandleDeferredToolCalls` (or equivalent) capability on the agent to resolve them inline, calling one from `run_code` raises an error that surfaces to the model as a retry
## API
```python
```python {test="skip"}
CodeMode(
tools: ToolSelector = 'all', # 'all', list[str], callable, or dict
max_retries: int = 3, # retries on sandbox execution errors
os_access: CodeModeOS | None = None, # host handler for env vars, clock, and file I/O
mount: CodeModeMount | None = None, # host directories to share with the sandbox
dynamic_catalog: bool = False, # keep run_code's description cache-stable; catalog moves into instructions
)
```
pydantic_ai_harness/compaction/README.md
+28 -4
View File
@@ -1,4 +1,4 @@
# Compaction capabilities
# Compaction
> [!NOTE]
> Import these capabilities from their submodule -- there is no top-level `pydantic_ai_harness` re-export:
@@ -10,13 +10,15 @@
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
A menu of strategies for keeping an agent's conversation history within a model's context
window. Each is a Pydantic AI `Capability` that runs in the `before_model_request` hook; edits
**persist** into the run's message history, so a trim/clear/summary carries forward to later
window. Each is a Pydantic AI `Capability` that edits the message history just before each
request goes out; edits **persist** into the run's message history, so a trim/clear/summary carries forward to later
steps (it is not recomputed from the full history every turn).
All strategies preserve tool-call / tool-return **pairing** -- core does not validate this, and a
provider rejects an orphaned pair. The zero-LLM strategies never call a model.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/compaction/)
## The menu
| Capability | Cost | What it does | Reach for it when |
@@ -26,7 +28,7 @@ provider rejects an orphaned pair. The zero-LLM strategies never call a model.
| `ClearToolResults` | zero-LLM | Blanks the content of old tool *results* in place, keeping the last `keep_pairs` | Tool outputs dominate context and can be re-fetched on demand (the cheap first tier) |
| `DeduplicateFileReads` | zero-LLM | Blanks every file read superseded by a newer read of the same file | The agent re-reads files and only the latest version matters |
| `SummarizingCompaction` | one LLM call | Summarizes older messages into a structured summary, keeping the recent tail | Old context still matters but must be compressed; use behind the cheap tiers |
| `TieredCompaction` | escalates | Runs cheap passes first, summarizes only if still over `target_tokens` | You want the SOTA default: spend the expensive summary only when needed |
| `TieredCompaction` | escalates | Runs cheap passes first, summarizes only if still over `target_tokens` | You want a sensible default: spend the expensive summary only when needed |
| `LimitWarner` | zero-LLM | Injects an URGENT/CRITICAL warning as limits approach | You want the agent to wrap up rather than have its history rewritten |
## Triggers
@@ -94,6 +96,23 @@ TieredCompaction(
)
```
## `SlidingWindow` and `ClearToolResults` options
`SlidingWindow` keeps the last `keep_messages` down to a tail; pass `keep_tokens` instead for a token
budget rather than a message count. By default `preserve_first_user_message=True` keeps the first user
turn even when it falls outside the window, so the agent does not lose the original task.
`ClearToolResults` keeps the last `keep_pairs` intact. Set `clear_tool_inputs=True` to also blank the
arguments of the cleared calls, and `exclude_tools` to a set of tool names whose results are never
cleared.
## `LimitWarner` thresholds
Warnings begin at `warning_threshold` (default `0.7`, a fraction of the limit) and escalate to CRITICAL
for iterations once the remaining request count drops to `critical_remaining_iterations` (default `3`).
It watches `max_iterations`, `max_context_tokens`, and `max_total_tokens`, warning on whichever are
configured; narrow that with `warn_on`.
## Cost: why summarization is the last resort
Summarization turns input tokens into output tokens, which are billed at a premium and generated
@@ -141,6 +160,11 @@ from the edit point onward -- the next request pays a cache-write. Use `ClearToo
`SummarizingCompaction(model=...)` accepts a model name or `Model`; when left `None` it inherits the
running agent's model. No token caps are imposed on the summary call.
By default `incremental=True` extends an existing summary from a prior compaction rather than
regenerating it from scratch, and `preserve_first_user_message=True` keeps the original task turn even
when it falls outside the window. Pass `keep_tokens` to trim the retained tail to a token budget instead
of `keep_messages`.
## Usage accounting
The summary call is a real request to the model, so its full usage -- tokens **and** the request
pydantic_ai_harness/context/README.md
+28 -8
View File
@@ -1,4 +1,4 @@
# RepoContext
# Context
> [!NOTE]
> Import this capability from its submodule -- there is no top-level `pydantic_ai_harness` re-export:
@@ -11,6 +11,8 @@
Discover and load a repo's accumulated coding-assistant context engineering (CE).
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/context/)
## The problem
A repo accumulates CE for whatever coding assistant worked in it: instruction
@@ -43,6 +45,9 @@ context first, most specific last. Files are deduped by resolved real path and b
content hash, so a symlinked `AGENTS.md -> CLAUDE.md` or two ancestors sharing
identical content load once.
When `home_dir` is `None` (the default), only `workspace_dir` is scanned -- no
walk-up. Pass `home_dir=Path.home()` to walk up to your home directory.
### 2. Asset inventory (on by default)
Exposes one tool, `inventory_agent_context()`, that reports where the repo's CE
@@ -51,6 +56,9 @@ the `skills/` (SKILL.md), `agents/` (`.md`), and `settings.json` (hooks) it
contains. It returns a structured `AgentContextInventory`; it locates assets and
does not parse them, leaving translation to the orchestrator.
Rename the tool with `inventory_tool_name`, or scope which roots it scans with
`asset_roots`.
### 3. Nested-on-traversal (off by default)
When the model lists or reads a directory, surface that directory's
@@ -58,12 +66,24 @@ When the model lists or reads a directory, surface that directory's
opt-in and configurable:
```python
RepoContext(
workspace_dir=Path('.'),
nested_traversal=True,
traversal_tool_names=frozenset({'list_dir', 'read_file'}), # match your tools
traversal_path_arg='path', # the path arg key
nested_inject='pointer', # or 'contents'
from pathlib import Path
from pydantic_ai import Agent
from pydantic_ai_harness import FileSystem
from pydantic_ai_harness.context import RepoContext
agent = Agent(
'anthropic:claude-sonnet-4-6',
capabilities=[
FileSystem(root_dir='.'),
RepoContext(
workspace_dir=Path('.'),
nested_traversal=True,
traversal_tool_names=frozenset({'list_directory', 'read_file'}), # the FileSystem tool names to hook
traversal_path_arg='path', # the path arg key
nested_inject='pointer', # or 'contents'
)
],
)
```
@@ -110,4 +130,4 @@ files once per run, so mid-run edits to those files are not reloaded.
## Further reading
- [Pydantic AI capabilities](https://ai.pydantic.dev/capabilities/)
- [Pydantic AI hooks](https://ai.pydantic.dev/capabilities/#lifecycle-hooks)
- [Pydantic AI hooks](https://ai.pydantic.dev/hooks/)
pydantic_ai_harness/docs/README.md
+5 -2
View File
@@ -1,4 +1,4 @@
# PyaiDocs
# Pydantic AI Docs
> [!NOTE]
> Import this capability from its submodule -- there is no top-level `pydantic_ai_harness` re-export:
@@ -11,6 +11,8 @@
Give an agent a tool that locates and returns Pydantic AI documentation on demand.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/docs/)
## The problem
An agent that authors Pydantic AI capabilities, hooks, tools, or toolsets needs the
@@ -22,7 +24,8 @@ the agent rarely needs in full and pins a snapshot that drifts from `main`.
`PyaiDocs` exposes one tool, `read_pyai_docs(topic)`, that locates the requested page and
returns it verbatim -- nothing is bundled into context up front. Each call resolves the
topic from a configured local checkout first, then falls back to fetching the page from
`pydantic/pydantic-ai:main`, so it works in any environment.
`pydantic/pydantic-ai:main`, so it works whether or not you have a local checkout (the
remote fallback needs network access).
Topics: `capabilities`, `hooks`, `tools`, `tools-advanced`, `toolsets`, `agent`.
pydantic_ai_harness/dynamic_workflow/README.md
+2
View File
@@ -13,6 +13,8 @@ Let one agent coordinate a whole team of sub-agents by writing a small Python sc
> The extensions planned in [What is coming](#what-is-coming), structured sub-agent inputs and durable
> workflows, touch the sub-agent call contract itself.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/dynamic_workflow/)
## The idea
Say you have a few specialist agents. One reviews code. One summarizes findings. One writes the
pydantic_ai_harness/experimental/acp/README.md
+3 -1
View File
@@ -21,6 +21,8 @@
Expose a Pydantic AI agent to editors and terminal UIs over the [Agent Client Protocol](https://agentclientprotocol.com).
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/experimental/acp/)
## The problem
Editors like [Zed](https://zed.dev/docs/ai/external-agents) speak ACP: a stdio JSON-RPC protocol that lets a TUI or editor drive an external coding agent -- streaming its text, rendering its file edits as diffs, and prompting the user to approve sensitive tool calls. To plug a Pydantic AI agent into one of these editors you would otherwise have to implement the ACP server side yourself.
@@ -223,7 +225,7 @@ Each completed turn reports its token counts (input/output/total, plus cached to
## API
```python
```python {test="skip"}
run_acp_stdio( # async; serve until the client disconnects
agent,
*,
pydantic_ai_harness/filesystem/README.md
+12 -3
View File
@@ -2,6 +2,8 @@
Give an agent sandboxed, pattern-filtered access to a directory tree.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/filesystem/)
## The problem
Letting an agent touch the filesystem directly is risky: path traversal
@@ -33,7 +35,7 @@ print(result.output)
| Tool | Purpose |
|---|---|
| `read_file` | Read a text file with line numbers and a content hash. Binary files are detected and not dumped. |
| `read_file` | Read a text file with line numbers and a content hash. Binary files are detected and not dumped. Supports `offset`/`limit` paging. |
| `write_file` | Create or overwrite a file. Optional `expected_hash` rejects stale writes (optimistic concurrency). |
| `edit_file` | Exact-string replacement; `old_text` must match exactly once. Optional `expected_hash`. |
| `list_directory` | List a directory's entries with type indicators and sizes. |
@@ -66,7 +68,7 @@ need `**`.
| `denied_patterns` | Matching paths are always rejected (denylist). |
| `protected_patterns` | Matching paths are read-only -- reads succeed, writes are rejected. |
`protected_patterns` defaults to `.git/`, `.env`/`.env.*`, `*.pem`, `*.key`,
`protected_patterns` defaults to `.git/*`, `.env`/`.env.*`, `*.pem`, `*.key`,
and `**/secrets*`. Pass an empty list to disable protection.
### Direct access vs. walkers
@@ -86,13 +88,20 @@ The three rules apply at two different granularities:
So with `allowed_patterns=['*.py']`, `list_directory('.')` succeeds and shows
only the `.py` entries; `read_file('notes.md')` is rejected.
> Dotfiles and dot-directories (`.git`, `.env`, `.github`, …) are skipped by
Note that the walkers filter entries with write-level access, so
`protected_patterns` matches are omitted from `list_directory`, `search_files`,
and `find_files` output even though those exact paths remain directly readable
via `read_file`/`file_info`.
> Dotfiles and dot-directories (`.git`, `.env`, `.github`, ...) are skipped by
> all three walkers -- `list_directory`, `search_files`, and `find_files` --
> regardless of patterns.
## Configuration
```python
from pydantic_ai_harness import FileSystem
FileSystem(
root_dir='.', # str | Path -- sandbox root
allowed_patterns=[], # allowlist globs (empty = allow all)
pydantic_ai_harness/guardrails/README.md
+19 -14
View File
@@ -1,21 +1,26 @@
# Guardrails
# Input & Output Guardrails
Validate the user prompt before it reaches the model, and the model output before it reaches the caller.
> [!NOTE]
> The API may change between releases. Where practical, breaking changes ship with a deprecation warning.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/guardrails/)
## The problem
Agents take unstructured input from users and return unstructured output to callers. Without a validation layer, a prompt injection attempt, PII-laden message, or off-topic question goes to the model as-is, and any output the model produces is returned verbatim. The framework does not reason about "this is unsafe to send" or "this is unsafe to show".
## The solution
Two capabilities — `InputGuard` and `OutputGuard` — each backed by a `guard` callable you supply. The guard inspects a value (the prompt, or the output) and returns one of four outcomes:
Two capabilities -- `InputGuard` and `OutputGuard` -- each backed by a `guard` callable you supply. The guard inspects a value (the prompt, or the output) and returns one of four outcomes:
| Outcome | `InputGuard` | `OutputGuard` |
|---|---|---|
| **allow** | send the prompt to the model | return the output to the caller |
| **block** | skip the model call; a refusal message becomes the response (`SkipModelRequest`) | raise `OutputBlocked` |
| **replace** | rewrite the prompt sent to the model (redaction) | substitute a sanitized output |
| **retry** | — (not valid for input) | send the output back to the model to try again (`ModelRetry`) |
| **retry** | -- (not valid for input) | send the output back to the model to try again (`ModelRetry`) |
A guard that raises an exception instead propagates it as a hard failure. The asymmetry between input `block` and output `block` is intentional: blocking the input spends no tokens, so a graceful refusal is almost always right; blocking the output means the model already produced something you do not want exposed, so raising forces the caller to decide what to do next.
@@ -47,9 +52,9 @@ agent = Agent(
)
```
`OutputGuard` receives the output unchanged — no automatic stringification. For a string output the guard reads it directly; for a typed (Pydantic model) output the guard gets the model instance, so pick the serialization that fits the check (read a field, or call `output.model_dump_json()` for JSON text). This avoids the trap of `str(MyModel(...))` producing a `MyModel(field=...)` repr that hides field contents from regex-based checks.
`OutputGuard` receives the output unchanged -- no automatic stringification. For a string output the guard reads it directly; for a typed (Pydantic model) output the guard gets the model instance, so pick the serialization that fits the check (read a field, or call `output.model_dump_json()` for JSON text). This avoids the trap of `str(MyModel(...))` producing a `MyModel(field=...)` repr that hides field contents from regex-based checks.
Guards may also be async — return an awaitable `bool`/`GuardResult`, e.g. to call a moderation API.
Guards may also be async -- return an awaitable `bool`/`GuardResult`, e.g. to call a moderation API.
## `GuardResult`
@@ -85,11 +90,11 @@ agent = Agent(
)
```
Input redaction requires sequential mode — it is incompatible with `parallel=True`, since a parallel guard runs alongside a model call that has already started with the original prompt.
Input redaction requires sequential mode -- it is incompatible with `parallel=True`, since a parallel guard runs alongside a model call that has already started with the original prompt.
## Retry (`retry`)
`OutputGuard` can send a bad output back to the model instead of blocking it. Return `GuardResult.retry(instruction)` — the instruction is the retry prompt the model sees. This reuses pydantic-ai's normal retry machinery and counts against the run's output-retry budget.
`OutputGuard` can send a bad output back to the model instead of blocking it. Return `GuardResult.retry(instruction)` -- the instruction is the retry prompt the model sees. This reuses pydantic-ai's normal retry machinery and counts against the run's output-retry budget.
```python
def must_cite_sources(output: object) -> GuardResult:
@@ -103,11 +108,11 @@ OutputGuard(guard=must_cite_sources)
## Streaming
`OutputGuard` inspects the **final** output only — during `run_stream()` partial chunks reach the caller before the guard runs, so a `block` or `replace` verdict cannot un-send content already streamed. Use `run()` / `run_sync()` when the output must be screened before any of it is exposed. `GuardResult.retry()` is **not** supported under `run_stream()` — pydantic-ai does not retry output during streaming, and a `retry` verdict there surfaces as `UnexpectedModelBehavior`. `InputGuard` (including `parallel=True`) works the same in streamed and non-streamed runs.
`OutputGuard` inspects the **final** output only -- during `run_stream()` partial chunks reach the caller before the guard runs, so a `block` or `replace` verdict cannot un-send content already streamed. Use `run()` / `run_sync()` when the output must be screened before any of it is exposed. `GuardResult.retry()` is **not** supported under `run_stream()` -- pydantic-ai does not retry output during streaming, and a `retry` verdict there surfaces as `UnexpectedModelBehavior`. `InputGuard` (including `parallel=True`) works the same in streamed and non-streamed runs.
## Tracing
`replace` and `block` are recorded as spans on the active OpenTelemetry tracer, so a redaction or refusal shows up in Logfire traces (`guardrail redacted input`, `guardrail blocked output`, etc.) with `guardrail.*` attributes. Content attributes — the original/replacement values for a redaction and the refusal `message` for a block — are attached **only** when `RunContext.trace_include_content` is enabled, since these can quote the very content the guard exists to keep out of traces. `retry` needs no special tracing: the retried model request appears in the trace on its own.
`replace` and `block` are recorded as spans on the active OpenTelemetry tracer, so a redaction or refusal shows up in Logfire traces (`guardrail redacted input`, `guardrail blocked output`, etc.) with `guardrail.*` attributes. Content attributes -- the original/replacement values for a redaction and the refusal `message` for a block -- are attached **only** when `RunContext.trace_include_content` is enabled, since these can quote the very content the guard exists to keep out of traces. `retry` needs no special tracing: the retried model request appears in the trace on its own.
`OutputGuard` declares `position='outermost', wrapped_by=[Instrumentation]` so its block/redact spans are always captured by an enclosing `Instrumentation` span regardless of how the user orders capabilities. `InputGuard` declares `position='innermost'` so any capability that morphs messages (a prompt rewriter, a context manager) runs first and the guard sees the final prompt the model will receive.
@@ -119,11 +124,11 @@ A slow guard (an LLM classifier, a network call) run sequentially adds its laten
InputGuard(guard=slow_async_classifier, parallel=True)
```
Parallel mode trades tokens for latency: sequential mode never calls the model when the guard blocks, but parallel mode has already started the model call — if the guard trips only after the model has responded, those tokens were spent. For fast local checks (regex, keyword lookup) sequential is the better default. `replace` is not available under `parallel=True` (see [Redaction](#redaction-replace)).
Parallel mode trades tokens for latency: sequential mode never calls the model when the guard blocks, but parallel mode has already started the model call -- if the guard trips only after the model has responded, those tokens were spent. For fast local checks (regex, keyword lookup) sequential is the better default. `replace` is not available under `parallel=True` (see [Redaction](#redaction-replace)).
## Accessing run context
A guard may take a `RunContext` as its first parameter when it needs run state — `deps` for tenant- or role-aware policy, message history for conversation-aware checks. The parameter is detected from the signature, so prompt-only guards need not declare it:
A guard may take a `RunContext` as its first parameter when it needs run state -- `deps` for tenant- or role-aware policy, message history for conversation-aware checks. The parameter is detected from the signature, so prompt-only guards need not declare it:
```python
from pydantic_ai import RunContext
@@ -151,11 +156,11 @@ def strict_guard(prompt: str) -> bool:
return True
```
Any exception raised by the guard propagates as-is — use `InputBlocked` / `OutputBlocked` from this module, or your own exception types.
Any exception raised by the guard propagates as-is -- use `InputBlocked` / `OutputBlocked` from this module, or your own exception types.
## API
```python
```python {test="skip"}
@dataclass
class GuardResult:
action: Literal['allow', 'block', 'replace', 'retry']
@@ -174,7 +179,7 @@ OutputGuard(
)
```
The guard callable takes the inspected value — the prompt for `InputGuard`, the output for `OutputGuard` — optionally preceded by a `RunContext`.
The guard callable takes the inspected value -- the prompt for `InputGuard`, the output for `OutputGuard` -- optionally preceded by a `RunContext`.
## Relationship to `pydantic-ai-shields`
pydantic_ai_harness/logfire/README.md
+5 -3
View File
@@ -9,6 +9,8 @@ Install the extra:
pip install 'pydantic-ai-harness[logfire]'
```
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/logfire/)
## `ManagedPrompt`
Back an agent's instructions with a Logfire-managed
@@ -23,7 +25,7 @@ Back an agent's instructions with a Logfire-managed
### The problem
Prompts are critical to agent behavior, but iterating on them through the normal
edit → review → deploy loop is slow, and you can't easily A/B test a change or roll it
edit -> review -> deploy loop is slow, and you can't easily A/B test a change or roll it
back the moment it misbehaves in production.
### The solution
@@ -142,13 +144,13 @@ Rendering requires `pydantic-handlebars` (install `pydantic-ai-slim[spec]`). It
### Prompt-cache trade-off
The resolved value lands in the agent's **system instructions**. Provider prompt caches (Anthropic,
OpenAI, etc.) key strictly by prefix -- `tools → system → messages` -- so any change to the system
OpenAI, etc.) key strictly by prefix -- `tools -> system -> messages` -- so any change to the system
block invalidates the cached prefix for the affected runs.
| Mode | Cache impact |
| --- | --- |
| Pinned `label='production'`, no rollout split | **Cache-stable.** The value only changes on a deliberate prompt rollout, which is the same cost as a redeploy. |
| Percentage rollout across labels (no `label=`) | Different runs land on different labels → splits the cache into one lane per label. |
| Percentage rollout across labels (no `label=`) | Different runs land on different labels -> splits the cache into one lane per label. |
| `targeting_key` per user/tenant with multiple labels in play | Cache lanes per assigned label; deterministic per key but still N lanes overall. |
| Mid-traffic label flip in the Logfire UI | One-shot cold-invalidation for everyone on that label. |
pydantic_ai_harness/media/README.md
+3 -1
View File
@@ -1,4 +1,4 @@
# Media
# Media Externalization
> [!NOTE]
> Import these helpers from their submodule -- there is no top-level `pydantic_ai_harness` re-export:
@@ -19,6 +19,8 @@ Content-addressed stores and walker helpers that move large binary payloads out
These are building blocks, not a capability. There is no class you add to `Agent(capabilities=[...])` yet. [`StepPersistence`](../step_persistence/) uses them to keep snapshots small when messages carry `BinaryContent`. A forthcoming `MediaExternalizer` capability will reuse the same stores to rewrite `BinaryContent` into URL parts before the model sees them.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/media/)
## Why
A conversation that carries images, audio, or other `BinaryContent` inlines those bytes into every message. Persist that history and each snapshot re-serializes the payloads. Content-addressed storage writes each payload once, keyed by its own hash, and leaves a short `media://` URI in its place. The same bytes are stored once no matter how many messages or snapshots reference them.
pydantic_ai_harness/overflowing_tool_output/README.md
+42 -10
View File
@@ -1,4 +1,4 @@
# Overflow capability
# Overflowing Tool Output
> [!NOTE]
> Import this capability from its submodule -- there is no top-level `pydantic_ai_harness` re-export:
@@ -12,13 +12,15 @@
A tool can return a payload large enough to dominate the context window. Tool returns
persist in history as `ToolReturnPart`s, so an oversized one is re-sent on every later
model request -- paying its token cost for the rest of the run. `OverflowingToolOutput`
intercepts a return in the `after_tool_execute` hook, reduces it once, and lets the reduced
intercepts a return when it is produced, reduces it once, and lets the reduced
form persist. The reduction is not recomputed per request.
This is the overflow-to-file follow-up the `compaction` README names as out of scope: it
moves large tool outputs *out* of the window at production time, rather than compressing or
dropping context already inside it.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/overflowing_tool_output/)
## The three modes
| Mode | Cost | Lossy? | What the model gets |
@@ -32,7 +34,8 @@ the registered `read_tool_result(handle, offset, limit, from_end, pattern)` tool
Code pattern, the core [#4352](https://github.com/pydantic/pydantic-ai/issues/4352) design).
That tool is bounded: `offset >= 0`, `limit` clamped to a built-in line cap, the joined output
capped, and `pattern` is a literal substring (not a regex), so a model-supplied value cannot
hang the host with catastrophic backtracking.
hang the host with catastrophic backtracking. The read-back tool's own returns are exempt from
reduction, so a `read_tool_result` result is never itself spilled or truncated.
### Both `return_value` and `content` are reduced
@@ -76,6 +79,9 @@ agent = Agent(
The default band, when you pass no `bands`, is `Spill(then=Truncate())`: lossless when a
store accepts the write, a bounded truncation otherwise -- zero LLM cost and no silent drop.
`Passthrough()` is an explicit no-op action for `bands` or `per_tool` lists, leaving matching
returns untouched.
### Fallbacks with `then`
Every action takes an optional `then`, applied when the action cannot run: a `Spill` whose
@@ -89,12 +95,25 @@ spill -> truncate.
`tail`); `tool_filter` (a `ToolSelector`) scopes which tools the capability touches at all.
```python
OverflowingToolOutput(
per_tool={
'read_file': [Band(over=8_000, action=Truncate(strategy=TruncationStrategy.head))],
'run_shell': [Band(over=8_000, action=Truncate(strategy=TruncationStrategy.tail))],
},
tool_filter=['read_file', 'run_shell', 'search'],
from pydantic_ai import Agent
from pydantic_ai_harness.experimental.overflow import (
Band,
OverflowingToolOutput,
Truncate,
TruncationStrategy,
)
agent = Agent(
'openai:gpt-4o',
capabilities=[
OverflowingToolOutput(
per_tool={
'read_file': [Band(over=8_000, action=Truncate(strategy=TruncationStrategy.head))],
'run_shell': [Band(over=8_000, action=Truncate(strategy=TruncationStrategy.tail))],
},
tool_filter=['read_file', 'run_shell', 'search'],
)
],
)
```
@@ -103,7 +122,8 @@ OverflowingToolOutput(
Thresholds are measured in characters by default. Set `over_tokens=True` to measure in
estimated tokens (the same ~4-chars-per-token heuristic as `compaction`); pass a `tokenizer`
callable for accuracy. `Truncate.max_chars` is always characters -- truncation is a
character operation regardless of the threshold unit.
character operation regardless of the threshold unit. Set `strip_ansi=True` to strip ANSI
escape sequences from text returns before measuring and reducing.
## Spill store
@@ -116,6 +136,9 @@ workspace once #4352 lands) can resolve the same handle in another process. Supp
backend with `store=...`.
```python
from typing import Protocol
class OverflowStore(Protocol):
async def write(self, key: str, data: bytes) -> str: ... # returns a handle
async def read(self, handle: str) -> bytes: ...
@@ -137,7 +160,11 @@ agent that still wants to read a spill. To bound disk use, opt into age-based pr
```python
from datetime import timedelta
from pydantic_ai import Agent
from pydantic_ai_harness.experimental.overflow import LocalFileStore, OverflowingToolOutput
store = LocalFileStore(cleanup_after=timedelta(hours=6)) # default: None = keep forever
agent = Agent('openai:gpt-4o', capabilities=[OverflowingToolOutput(store=store)])
```
When set, a `write` schedules a background prune (a daemon thread, off the hot path) that
@@ -166,6 +193,11 @@ A `Summarize` call is a real request to the model, so its full usage -- tokens a
request itself -- folds into the run's `ctx.usage`, exactly like `SummarizingCompaction`. No
token caps are imposed on the summary call. A `UsageLimits` request limit will see it.
By default `Summarize` inherits the running agent's model (`ctx.model`). Pass a model id or
instance to `Summarize(model=...)` to override, or a `summarize` callable to bypass the
built-in prompt entirely. The `summary_prompt` template on the capability must contain both
`{tool_name}` and `{output}` placeholders.
## Edge cases
- Binary returns spill verbatim and are never stringify-truncated; `Truncate` / `Summarize`
pydantic_ai_harness/planning/README.md
+4
View File
@@ -11,6 +11,8 @@
Give an agent a structured, self-updating task plan -- without ever invalidating the prompt cache.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/planning/)
## The problem
Long agentic runs drift: the model loses track of what it set out to do and what's left. The usual fix -- keep a running plan and re-inject it into the system prompt each turn -- invalidates the prompt cache. The system prompt sits at the front of the request, so every plan edit changes the cached prefix and forces the whole conversation to be re-processed at full token price.
@@ -60,6 +62,8 @@ The plan is never injected into the system prompt or instructions. Static usage
## Configuration
```python
from pydantic_ai_harness.planning import Planning
Planning(
guidance=None, # static system-prompt guidance; None = default, '' = omit
cache_ttl='5m', # TTL for the cache breakpoint before the reminder ('5m' | '1h')
pydantic_ai_harness/runtime_authoring/README.md
+28 -2
View File
@@ -1,4 +1,4 @@
# RuntimeAuthoring
# Runtime Authoring
> [!NOTE]
> Import this capability from its submodule -- there is no top-level `pydantic_ai_harness` re-export:
@@ -11,6 +11,8 @@
Let an agent author, validate, and persist real pydantic-ai capabilities at runtime.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/runtime_authoring/)
## The problem
A coding agent often discovers, mid-task, that it wants a behavior its host does not
@@ -46,6 +48,10 @@ authoring = RuntimeAuthoring(directory=Path('.authored'))
agent = Agent('anthropic:claude-sonnet-4-6', capabilities=[authoring])
```
`RuntimeAuthoring` also contributes static, cache-stable system-prompt guidance
explaining these tools. Leave `guidance=None` for the default text, or pass your own
string; set `guidance=''` to omit it entirely.
## Activation boundary
A capability **cannot** be added to a live, already-executing run. pydantic-ai resolves
@@ -64,11 +70,22 @@ active capabilities into each run. With `agent.run(..., capabilities=...)`, the
capability is live on the very next loop iteration -- no process restart.
```python
from pathlib import Path
from pydantic_ai import Agent
from pydantic_ai_harness.experimental.authoring import RuntimeAuthoring
authoring = RuntimeAuthoring(directory=Path('.authored'))
agent = Agent('anthropic:claude-sonnet-4-6', capabilities=[authoring])
history = None
done = False
next_prompt = 'Start the task.'
while not done:
extra = authoring.store.load_active()
result = await agent.run(next_prompt, message_history=history, capabilities=extra)
history = result.all_messages()
# ... decide `next_prompt` and `done` from `result` ...
```
Because the capabilities also persist to disk (`<directory>/<name>.py` plus a
@@ -78,11 +95,20 @@ Because the capabilities also persist to disk (`<directory>/<name>.py` plus a
`manifest.json` records each capability's name, module file, class name, status, and last
validation error -- the surface a UI can read to show what the agent has authored.
Capability names must be lowercase letters, digits, and underscores, starting with a
letter; reusing a name replaces the previous capability of that name.
## Trust boundary and the sandboxed alternative
Authoring executes arbitrary Python in-process at import, construction, and run time. That
is the same trust boundary an agent that already runs shell commands and edits files
operates under, which is the deliberate choice here.
operates under, which is the deliberate choice here. Do not point it at a directory whose
contents you would not run yourself, and treat authored capabilities as code the agent is
executing on your host.
Because authored capabilities hold live code, they are not spec-serializable
(`get_serialization_name()` returns `None`) and are persisted as source rather than as an
agent spec.
The sandboxed alternative is the dormant `pa` Monty hook-slot registration system in the
Loopy tree (`pa/slots.py`, `pa/registration_tools.py`, `pa/capability.py`
pydantic_ai_harness/shell/README.md
+20 -12
View File
@@ -3,6 +3,8 @@
Give an agent the ability to run shell commands, with allow/deny controls and
managed background processes.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/shell/)
## The problem
Agents frequently need to run a build, a test suite, a linter, or a quick
@@ -50,13 +52,16 @@ which all land at the end -- survive truncation.
| `allowed_commands` | If non-empty, only these executables may run (allowlist). |
| `denied_commands` | These executables are always rejected (denylist). |
| `denied_operators` | Shell operators (e.g. `>`, `>>`, `|`) that are rejected when present. |
| `allow_interactive` | If `False` (default), commands that expect a TTY (`vi`, `sudo`, `ssh`, …) are blocked. |
| `allow_interactive` | If `False` (default), commands that expect a TTY (`vi`, `sudo`, `ssh`, ...) are blocked. |
`allowed_commands` and `denied_commands` are mutually exclusive -- set one, not
both. `denied_commands` defaults to a list of destructive commands (`rm`,
`rmdir`, `mkfs`, `dd`, `shutdown`, `reboot`, …); pass an empty list to disable.
The executable name is extracted with `shlex`, so arguments don't bypass the
check.
`rmdir`, `mkfs`, `dd`, `format`, `shutdown`, `reboot`, `halt`, `poweroff`,
`init`); pass an empty list to disable. The executable name is extracted with
`shlex`, so arguments don't bypass the check.
A denied or blocked command surfaces to the model as a `ModelRetry` (the model
can retry with an allowed command) rather than aborting the run.
> **These checks are best-effort, not a security boundary.** A sufficiently
> motivated agent can defeat them (e.g. `bash -c '...'`, env-var indirection).
@@ -74,7 +79,7 @@ writes can read them. Two fields control what the subprocess sees:
| `env` | Explicit environment that replaces inheritance entirely. The subprocess sees exactly these variables and nothing else. |
| `denied_env_patterns` | Glob patterns (`fnmatch`) for variable names stripped from the base environment. Mirrors `denied_commands`. |
`env` is a hard boundary: set it and inherited secrets cannot reach the
`env` is a hard boundary for inherited environment variables: set it and inherited secrets cannot reach the
subprocess at all (you supply `PATH` and anything else the command needs).
`denied_env_patterns` is a denylist over the inherited environment -- lighter to
configure when you only need to drop a few known-sensitive names. The two
@@ -106,7 +111,9 @@ credentials, so it is opt-in.
`env` is enforced at spawn, not applied as a post-hoc filter on a running
process: the subprocess starts with exactly the resolved environment (your
`env`, minus anything `denied_env_patterns` removes from it). That makes it a
real boundary, unlike the best-effort command denylist. The flip side is that a
real boundary for inherited environment variables, unlike the best-effort command denylist. It is not a full
security boundary: a command running under the same OS identity can still read
host files -- use OS-level isolation for that. The flip side is that a
pattern broad enough to strip `PATH` or `HOME`, or an `env` that omits them, can
break command resolution. External commands may still run via the shell's
built-in default `PATH` on some systems, but don't rely on it -- set `PATH`
@@ -115,8 +122,8 @@ explicitly when you replace the environment.
## Background processes
`start_command` writes stdout/stderr to temp files and returns a short ID. Use
`check_command(id)` to poll and `stop_command(id)` to terminate and collect
final output. Processes are launched in their own session (`start_new_session`)
`check_command(command_id)` to poll and `stop_command(command_id)` to terminate
and collect final output. Processes are launched in their own session (`start_new_session`)
so the whole process group can be signalled -- `SIGTERM`, escalating to
`SIGKILL` after a grace period.
@@ -128,10 +135,11 @@ agent that forgets to call `stop_command` won't leak processes.
## Working directory
By default each command runs in `cwd` and `cd` has no lasting effect. Set
`persist_cwd=True` to make `cd` sticky: the toolset appends a `pwd` sentinel to
successful commands, parses the result, and carries the new directory into
subsequent calls. Commands containing `;` skip the sentinel injection so the
`&&`-gated sentinel can't be bypassed.
`persist_cwd=True` to make `cd` sticky across calls: each command is wrapped so
that after it runs, its final working directory is recorded to a private temp
file, and that directory is carried into subsequent calls. The path is only
updated when the command exits `0`, and the record is written out-of-band (not
to stdout) so command output can never spoof the tracked directory.
## Configuration
pydantic_ai_harness/step_persistence/README.md
+24 -19
View File
@@ -1,4 +1,4 @@
# StepPersistence
# Step Persistence
> [!NOTE]
> Import `StepPersistence` and the `media` stores from their submodules -- there is no top-level
@@ -21,6 +21,8 @@ It is not a full graph-state checkpoint. Capability-state restore, workspace
snapshots, and graph-node resume are out of scope and tracked separately
(see `pydantic-ai-harness` issues #149 and #196).
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/step_persistence/)
## What it gives you
1. **Append-only step events** -- every interesting boundary (run start/end,
@@ -61,21 +63,21 @@ primitive for it (see [Three-level identity](#three-level-identity)).
`run_id` resolution per call:
- **Explicit `run_id='libr-1'`** → used as-is for this one call.
- **Explicit `run_id='libr-1'`** becomes the id for this one call.
Single-shot use cases (deterministic id for testing, replay, debugging,
a one-off scripted run). Reusing one capability instance with the same
explicit `run_id` across multiple `.run()` calls **raises `ValueError`
in `before_run`** -- the tool-effect ledger is keyed by
explicit `run_id` across multiple `.run()` calls raises `ValueError`
in `before_run` -- the tool-effect ledger is keyed by
`(run_id, tool_call_id)` and providers reuse deterministic tool-call
ids, so a silent collision would erase the `unknown_after_crash`
signal. Use `conversation_id=` for multi-turn grouping instead.
- **`agent_name` set, `run_id` unset** → `'{agent_name}-{8-char-hex}'`,
- **`agent_name` set, `run_id` unset** derives `'{agent_name}-{8-char-hex}'`,
freshly materialised in `for_run` per `.run()` call. Reusing one
capability instance across runs yields distinct ids
(`code_librarian-a3b2`, `code_librarian-c9d1`, …). This is the
(`code_librarian-a3b2`, `code_librarian-c9d1`, and so on). This is the
recommended default for delegate capabilities.
- **Neither set** → `ctx.run_id` (pydantic_ai's auto-generated id) per
`.run()` call, falling back to a UUID4.
- **Neither set** falls back to `ctx.run_id` (pydantic_ai's auto-generated
id) per `.run()` call, and to a UUID4 if that is absent.
The orchestrator pattern -- one logical agent serving many turns -- uses
`conversation_id`, not a shared `run_id`:
@@ -143,15 +145,16 @@ run gets a fresh `run_id` and probably a fresh `conversation_id`).
### What "safe to continue from" means
`continue_run` only returns the messages of the **latest provider-valid
snapshot** for that `run_id`. Snapshots are written at two boundaries:
`continue_run` only returns the messages of the latest provider-valid
snapshot for that `run_id`. Snapshots are written at two boundaries:
- after every `CallToolsNode` completes (all tool calls returned), and
- at `after_run`.
- at `after_run`, as a fallback if the run reached no such boundary.
A run that crashed mid-tool-call has events (`tool_call_started`) but no
snapshot for that point. `continue_run` returns the snapshot from the
**previous** safe boundary, not the failed step.
previous safe boundary, not the failed step. If no continuable snapshot
exists at all, `continue_run` raises `LookupError`.
## Run lineage -- `parent_run_id`
@@ -162,7 +165,7 @@ two things:
- `store.list_runs(parent_run_id='orch-1')` returns every delegate run
pointing at that orchestrator.
It is **auto-inferred for in-process delegation**: when an orchestrator's
It is auto-inferred for in-process delegation: when an orchestrator's
tool synchronously calls a delegate's `Agent.run(...)`, the delegate's
`StepPersistence` picks up the orchestrator's `run_id` via a `ContextVar`
that the orchestrator's `wrap_run` set. No threading required:
@@ -196,7 +199,7 @@ delegates = await store.list_runs(parent_run_id=orch_run_id)
Set `parent_run_id=` explicitly to override (e.g. cross-process delegation
where `ContextVar`s do not propagate).
`parent_run_id` is **distinct from `conversation_id`**. The orchestrator
`parent_run_id` is distinct from `conversation_id`. The orchestrator
and delegate usually live in *different* conversations (the orchestrator
talks to a user; the delegate talks to itself). But they share a
parent-child link.
@@ -253,6 +256,7 @@ write external state should annotate their in-flight `ToolEffectRecord`
via `annotate_tool_effect`:
```python
from pydantic_ai import RunContext
from pydantic_ai_harness.step_persistence import annotate_tool_effect
@orchestrator.tool
@@ -269,7 +273,8 @@ async def set_label(ctx: RunContext[Deps], issue: int, label: str) -> str:
The helper reads the active `run_id` from the `StepPersistence`
`ContextVar` and `tool_call_id` / `tool_name` from `ctx`, then merges the
metadata into the prior record. `after_tool_execute` preserves both
metadata into the prior record. It is a no-op when called outside a
step-persistence-wrapped tool call. `after_tool_execute` preserves both
fields when it writes the terminal `completed` / `failed` entry.
## Backends
@@ -307,7 +312,7 @@ IDs should still sanitise first.
`BinaryContent` payloads (images, audio, documents, video) inline as
base64 inside a snapshot would balloon every file/row containing the
message. Both `FileStepStore` and `SqliteStepStore` externalize any
`BinaryContent.data` ≥ **64 KiB** through a configured `MediaStore`,
`BinaryContent.data` at or above 64 KiB through a configured `MediaStore`,
leaving a URI reference in the snapshot. Round-trip is transparent --
`latest_snapshot(...).messages[*]` returns `BinaryContent` with the
original bytes.
@@ -472,13 +477,13 @@ always safe -- you only ever lose wire savings, never correctness.
### Persisting unsupported backends
DynamoDB, Postgres, Redis, GCS, and other backends are out of scope for
this release. Write your own `StepStore` (≈ ten methods on a Protocol) or
this release. Write your own `StepStore` (about ten methods on a Protocol) or
your own `MediaStore` (three methods) and pass it via `store=` /
`media_store=`. Please open an issue if you ship one -- we want to feed
the eventual shared adapter layer with N≥3 real implementations before
the eventual shared adapter layer with N >= 3 real implementations before
abstracting.
## What this capability does **not** do
## What this capability does not do
- It does not restore capability per-run state, graph-node state, retry
counters, or in-flight streaming responses.
pydantic_ai_harness/subagents/README.md
+9 -2
View File
@@ -1,4 +1,4 @@
# SubAgents
# Subagents
> [!NOTE]
> Import this capability from its submodule -- there is no top-level `pydantic_ai_harness` re-export:
@@ -11,6 +11,8 @@
Let an agent delegate self-contained tasks to named child agents.
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/subagents/)
## The problem
A single agent that does everything accumulates a large tool set and a long context. Splitting the work across specialized sub-agents keeps each context focused, but wiring up delegation by hand means writing a tool per agent, forwarding deps, threading usage limits, and telling the model what it can delegate to.
@@ -60,10 +62,13 @@ A delegate's name -- how the parent model refers to it, and how it is listed in
Each `SubAgent` carries its own budgets, so one delegate's controls do not touch the others. A `SubAgent` with no controls set runs with the `SubAgents` defaults.
```python
from pydantic_ai import Agent
from pydantic_ai.usage import UsageLimits
from pydantic_ai_harness.subagents import SubAgent, SubAgents
# reproducer and librarian are Agent instances, as in the example above.
reproducer = Agent('anthropic:claude-sonnet-4-6', instructions='Reproduce the reported bug from a minimal script.')
librarian = Agent('anthropic:claude-sonnet-4-6', instructions='Find relevant docs, issues, and prior art.')
orchestrator = Agent(
'anthropic:claude-opus-4-7',
capabilities=[
@@ -160,6 +165,8 @@ Every agent the capability builds runs at a minimum thinking-effort floor. `MINI
A disk agent gets no tools by default (`inherit_tools` is `False`); set `inherit_tools=True` to expose the parent's tools to it through the `inherit_tools` mechanism, in which case its `tools` frontmatter is ignored. To map the frontmatter tool names to specific toolsets instead, pass a `tool_resolver`: it receives each tool name (so it can honor entries like `Bash(git:*)`) and returns the toolsets that provide it, or `None` for an unknown name, which is skipped with a warning.
```python
from pydantic_ai_harness.experimental.subagents import SubAgents
def resolve(tool_name: str):
return TOOLSETS.get(tool_name) # -> Sequence[AgentToolset[object]] | None
tests/test_doc_snippets.py
+138
View File
@@ -0,0 +1,138 @@
"""Static validation of the code snippets shown in the docs.
Every Python snippet in a capability `README.md` (GitHub/PyPI) and in the flat
`docs/<capability>.md` pages (the unified docs site) is checked for the two
failure modes a reader hits immediately:
- **it does not parse** -- a syntax error means the snippet cannot run at all;
- **it imports a harness symbol that does not exist** -- a stale module path or a
renamed/removed name (e.g. a snippet still importing from
`pydantic_ai_harness.experimental.<graduated>`).
This is the *static* half of doc-snippet testing. It deliberately does not
execute the snippets -- most build an `Agent` and call `.run()`, which needs a
model -- so it stays fast and needs no mocking. Running snippets against a mocked
model is a separate concern (see `test_readme_quick_start.py` for that shape).
Illustrative signature blocks (API-reference pseudo-code with type annotations or
a bare `*`, which is not runnable Python) opt out with a `{test="skip"}` fence
directive.
"""
from __future__ import annotations as _annotations
import ast
import importlib
import os
import warnings
from collections.abc import Iterable
from pathlib import Path
import pytest
from _pytest.mark import ParameterSet
from pytest_examples import CodeExample, find_examples
_ROOT = Path(__file__).parent.parent
_HARNESS = 'pydantic_ai_harness'
def _harness_import_targets(tree: ast.AST) -> Iterable[tuple[str, str | None]]:
"""`(module, name)` for every `pydantic_ai_harness` symbol a snippet imports.
`name` is `None` for a plain `import pydantic_ai_harness.x` or a star import,
where only the module's existence can be checked.
"""
for node in ast.walk(tree):
if isinstance(node, ast.ImportFrom):
module = node.module or ''
if module == _HARNESS or module.startswith(f'{_HARNESS}.'):
for alias in node.names:
yield module, None if alias.name == '*' else alias.name
elif isinstance(node, ast.Import):
for alias in node.names:
if alias.name == _HARNESS or alias.name.startswith(f'{_HARNESS}.'):
yield alias.name, None
def _is_missing_harness_module(exc_name: str | None) -> bool:
"""True when an ImportError is a genuinely absent harness module, not a missing extra.
A missing optional dependency (e.g. `acp` in the `slim` CI job) raises
`ModuleNotFoundError` naming the third-party package, not the harness module --
the harness module exists, its extra just isn't installed.
"""
return exc_name is not None and exc_name.startswith(_HARNESS)
def _snippet_problem(source: str) -> str | None:
"""Return why a snippet is invalid, or `None` if it parses and its harness imports resolve."""
try:
tree = ast.parse(source)
except SyntaxError as exc:
return f'does not parse: {exc.msg} (line {exc.lineno})'
for module, name in _harness_import_targets(tree):
try:
with warnings.catch_warnings():
warnings.simplefilter('ignore') # a deprecated shim path still resolves; existence is what we check
imported = importlib.import_module(module)
except ImportError as exc:
if _is_missing_harness_module(exc.name):
return f'imports `{module}`, which does not exist: {exc}'
continue # missing optional extra in this environment; the harness module exists
if name is not None and not hasattr(imported, name):
return f'imports `{name}` from `{module}`, but that name does not exist'
return None
def _doc_snippets() -> Iterable[ParameterSet]:
# `find_examples` yields only Python fenced blocks and wants paths relative to
# the cwd, so pin it to the repo root (matches `test_skill_examples.py`).
os.chdir(_ROOT)
readmes = sorted(str(p.relative_to(_ROOT)) for p in _ROOT.glob(f'{_HARNESS}/**/README.md'))
for ex in find_examples(*readmes, 'docs'):
yield pytest.param(ex, id=f'{ex.path}:{ex.start_line}')
@pytest.mark.parametrize('example', _doc_snippets())
def test_doc_snippet_valid(example: CodeExample) -> None:
if example.prefix_settings().get('test', '').startswith('skip'):
pytest.skip('illustrative signature block; not runnable Python')
problem = _snippet_problem(example.source)
assert problem is None, (
f'{example.path}:{example.start_line} {problem}. '
'Fix the snippet, or mark the fence `{test="skip"}` if it is illustrative signature pseudo-code.'
)
def test_doc_snippets_discovered() -> None:
# Guard against a discovery break silently making the check vacuous.
assert sum(1 for _ in _doc_snippets()) >= 100
def test_snippet_problem_detects_each_failure_mode() -> None:
# Valid: harness imports that resolve, star imports, plain imports, and non-harness imports.
assert _snippet_problem('from pydantic_ai_harness import CodeMode') is None
assert _snippet_problem('import pydantic_ai_harness.code_mode') is None
assert _snippet_problem('from pydantic_ai_harness.overflowing_tool_output import *') is None
assert _snippet_problem('from os import path\nimport sys') is None
# Invalid: syntax, a module that does not exist, and a name that does not exist.
assert 'does not parse' in (_snippet_problem('def (:') or '')
assert 'does not exist' in (_snippet_problem('from pydantic_ai_harness.nope import X') or '')
assert 'does not exist' in (_snippet_problem('from pydantic_ai_harness import NoSuchCapability') or '')
def test_missing_harness_module_classification() -> None:
assert _is_missing_harness_module('pydantic_ai_harness.experimental.nope') is True
assert _is_missing_harness_module('acp') is False # a missing extra, not a harness module
assert _is_missing_harness_module(None) is False
def test_missing_optional_extra_is_not_a_failure(monkeypatch: pytest.MonkeyPatch) -> None:
# Simulate the `slim` environment: the harness module exists, but importing it
# fails because its third-party extra is absent. That is not a broken snippet.
def _extra_missing(module: str) -> object:
raise ModuleNotFoundError("No module named 'acp'", name='acp')
monkeypatch.setattr(importlib, 'import_module', _extra_missing)
assert _snippet_problem('from pydantic_ai_harness.experimental.acp import run_acp_stdio') is None
tests/test_docs_parity.py
+233 -1
View File
@@ -9,6 +9,7 @@ prose match the code as written) is a review-time concern, not a unit test.
from __future__ import annotations
import re
from pathlib import Path
import pytest
@@ -67,8 +68,239 @@ def test_capability_has_readme(package: Path) -> None:
def test_capability_linked_from_top_readme(package: Path) -> None:
top_readme = (_ROOT / 'README.md').read_text(encoding='utf-8')
link_target = f'{package.relative_to(_ROOT).as_posix()}/'
assert link_target in top_readme, (
# Require an actual Markdown link to the package, not just the path appearing
# anywhere (prose or an unrelated URL would otherwise satisfy the check).
linked = any(t.startswith(link_target) for t in _markdown_link_targets(top_readme))
assert linked, (
f'{package.relative_to(_ROOT)} is not linked from the top-level README.md. '
f'Add a row for it (linking `{link_target}`) to the "What\'s available today" or "Roadmap" tables '
'so the README stays in step with the code.'
)
# --- Unified-docs page checks (docs/*.md) -----------------------------------
#
# The flat pages under `docs/` render on the unified site. These mechanical
# checks encode the capability-authoring rules agreed in the 2026-07-10 team
# sync: purpose-first leads, a source link on every page, names that match the
# capability, and no leftover "experimental" framing on graduated capabilities.
# ACP is the one page that stays experimental.
_DOCS_DIR = _ROOT / 'docs'
_NON_CAPABILITY_PAGES = {'index.md', 'mutation-testing.md'}
_ACP_PAGE = 'acp.md'
_SOURCE_LINK = 'github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/'
# Framing that must not appear on a graduated (non-ACP) capability page.
_EXPERIMENTAL_MARKERS = ('HarnessExperimentalWarning', 'removed in any release', '!!! warning "Experimental')
# Lifecycle hook names must not lead a page -- mechanism goes below the purpose.
_LEAD_HOOK_NAMES = ('before_model_request', 'after_model_request', 'before_tool_execute', 'after_tool_execute')
# ClassName-style headings are a smell, except where the class name IS the name.
_ALLOWED_CLASSNAME_HEADINGS = {'FileSystem'}
_FORBIDDEN_HEADINGS = {'overflow', 'authoring', 'overflow capability', 'compaction capabilities'}
def _capability_doc_pages() -> list[Path]:
return [p for p in sorted(_DOCS_DIR.glob('*.md')) if p.name not in _NON_CAPABILITY_PAGES]
_CAPABILITY_DOC_PAGES = _capability_doc_pages()
# Each capability doc page maps to its source module (for the source-link check)
# and its expected H1 (for the heading check). Keeping this explicit makes the
# checks page-specific: a page that links the wrong module, or carries a generic
# or empty heading, fails instead of passing on a substring match.
_CAPABILITY_PAGE_META = {
'code-mode.md': ('code_mode', 'Code Mode'),
'filesystem.md': ('filesystem', 'FileSystem'),
'shell.md': ('shell', 'Shell'),
'managed-prompt.md': ('logfire', 'Managed Prompt'),
'context.md': ('context', 'Context'),
'pydantic-ai-docs.md': ('docs', 'Pydantic AI Docs'),
'compaction.md': ('compaction', 'Compaction'),
'overflowing-tool-output.md': ('overflowing_tool_output', 'Overflowing Tool Output'),
'step-persistence.md': ('step_persistence', 'Step Persistence'),
'media.md': ('media', 'Media Externalization'),
'subagents.md': ('subagents', 'Subagents'),
'dynamic-workflow.md': ('dynamic_workflow', 'Dynamic Workflow'),
'planning.md': ('planning', 'Planning'),
'runtime-authoring.md': ('runtime_authoring', 'Runtime Authoring'),
'guardrails.md': ('guardrails', 'Input & Output Guardrails'),
'acp.md': ('experimental/acp', 'ACP (Agent Client Protocol)'),
}
def _markdown_link_targets(text: str) -> list[str]:
"""Every `](target)` destination in the text -- so a bare path mention is not a link."""
return re.findall(r'\]\(([^)\s]+)\)', text)
def _strip_frontmatter(text: str) -> str:
if text.startswith('---\n'):
end = text.find('\n---', 4)
if end != -1:
return text[end + 4 :]
return text
def _heading_problem(h1: str) -> str | None:
"""Return why an H1 fails the name rule, or None if it is fine."""
name = h1[2:].strip() if h1.startswith('# ') else h1.strip()
if not name:
return 'missing H1 heading'
if name.lower() in _FORBIDDEN_HEADINGS:
return f'"{name}" is a short/legacy form -- use the full capability name'
for word in name.split():
if word in _ALLOWED_CLASSNAME_HEADINGS:
continue
if re.search(r'[a-z][A-Z]', word):
return f'"{name}" is ClassName-style ("{word}") -- use spaced words'
return None
def _h1(text: str) -> str:
for line in _strip_frontmatter(text).splitlines():
if line.startswith('# '):
return line
return ''
def _lead_paragraph(text: str) -> str:
"""The first prose paragraph after the H1, skipping links, notes, and admonitions."""
lines = _strip_frontmatter(text).splitlines()
start = next((i + 1 for i, ln in enumerate(lines) if ln.startswith('# ')), 0)
collected: list[str] = []
in_fence = False
for line in lines[start:]:
stripped = line.strip()
if in_fence:
if stripped.startswith('```'):
in_fence = False
continue
if not collected:
if not stripped:
continue
if stripped.startswith('```'):
in_fence = True
continue
# Skip non-prose preamble: headings, blockquotes, admonitions,
# links/images, tables, and indented admonition bodies.
if stripped.startswith(('#', '>', '!!!', '[', '![', '|')) or line.startswith(' '):
continue
collected.append(stripped)
elif not stripped:
break
else:
collected.append(stripped)
return ' '.join(collected)
def test_strip_frontmatter_handles_missing_close() -> None:
assert _strip_frontmatter('---\ntitle: x\n---\nbody') == '\nbody'
# Opened but never closed -- returned unchanged rather than swallowing the file.
assert _strip_frontmatter('---\nnot closed') == '---\nnot closed'
assert _strip_frontmatter('no frontmatter') == 'no frontmatter'
def test_h1_missing_returns_empty() -> None:
assert _h1('# Title\nbody') == '# Title'
assert _h1('just prose, no heading') == ''
def test_heading_problem_flags_each_failure_mode() -> None:
assert _heading_problem('') == 'missing H1 heading'
assert _heading_problem('# Overflow') is not None # forbidden short form
assert _heading_problem('# SubAgents') is not None # ClassName-style
assert _heading_problem('# FileSystem') is None # allowlisted ClassName
assert _heading_problem('# Code Mode') is None
def test_lead_paragraph_skips_preamble_and_fences() -> None:
doc = (
'---\ntitle: x\n---\n'
'# Title\n\n'
'```python\ncode\n```\n\n'
'> a note\n'
'[Source](x)\n'
' indented body\n\n'
'The real lead.\n\n'
'second paragraph\n'
)
# Fenced block, blockquote, link, and indented lines are skipped; collection
# stops at the blank line after the first prose paragraph.
assert _lead_paragraph(doc) == 'The real lead.'
# A lead that runs to EOF exercises multi-line collection and loop exhaustion.
assert _lead_paragraph('# T\n\nline one\nline two') == 'line one line two'
def test_capability_doc_pages_discovered() -> None:
# Guard against a moved docs root making every check below vacuously pass.
assert len(_CAPABILITY_DOC_PAGES) >= 12
# A new capability page must declare its module + expected heading here, so
# the source-link and heading checks below stay page-specific.
unmapped = sorted(p.name for p in _CAPABILITY_DOC_PAGES if p.name not in _CAPABILITY_PAGE_META)
assert not unmapped, f'add {unmapped} to _CAPABILITY_PAGE_META (source module + expected heading)'
@pytest.mark.parametrize('page', _CAPABILITY_DOC_PAGES, ids=lambda p: p.name)
def test_doc_page_links_its_source(page: Path) -> None:
module, _ = _CAPABILITY_PAGE_META[page.name]
expected = f'{_SOURCE_LINK}{module}/'
targets = _markdown_link_targets(page.read_text(encoding='utf-8'))
assert any(expected in t for t in targets), (
f'{page.relative_to(_ROOT)} must link its own source module as a Markdown link '
f'(target containing `{expected}`), not just mention the prefix or link a different module.'
)
@pytest.mark.parametrize('page', _CAPABILITY_DOC_PAGES, ids=lambda p: p.name)
def test_doc_page_heading_matches_capability(page: Path) -> None:
_, expected_name = _CAPABILITY_PAGE_META[page.name]
h1 = _h1(page.read_text(encoding='utf-8'))
assert h1, f'{page.relative_to(_ROOT)} has no H1 heading.'
actual = h1[2:].strip()
assert actual == expected_name, (
f'{page.relative_to(_ROOT)} H1 is "{actual}"; expected "{expected_name}" '
'(doc filename, H1, and capability name must agree).'
)
@pytest.mark.parametrize('page', _CAPABILITY_DOC_PAGES, ids=lambda p: p.name)
def test_doc_page_lead_is_purpose_first(page: Path) -> None:
lead = _lead_paragraph(page.read_text(encoding='utf-8'))
hit = next((h for h in _LEAD_HOOK_NAMES if h in lead), None)
assert hit is None, (
f'{page.relative_to(_ROOT)}: opening paragraph names the `{hit}` hook. '
'Lead with the purpose (what it is for, when to use it); move mechanism lower.'
)
@pytest.mark.parametrize(
'page',
[p for p in _CAPABILITY_DOC_PAGES if p.name != _ACP_PAGE],
ids=lambda p: p.name,
)
def test_graduated_doc_page_has_no_experimental_framing(page: Path) -> None:
text = page.read_text(encoding='utf-8')
hit = next((m for m in _EXPERIMENTAL_MARKERS if m in text), None)
assert hit is None, (
f'{page.relative_to(_ROOT)}: graduated capability still carries experimental framing ({hit!r}). '
'Only ACP keeps an experimental note; soften the rest to the README stability note.'
)
@pytest.mark.parametrize('package', _CAPABILITY_PACKAGES, ids=lambda p: str(p.relative_to(_ROOT)))
def test_capability_readme_heading_matches_capability(package: Path) -> None:
problem = _heading_problem(_h1((package / 'README.md').read_text(encoding='utf-8')))
assert problem is None, f'{package.relative_to(_ROOT) / "README.md"}: {problem}'
@pytest.mark.parametrize('package', _CAPABILITY_PACKAGES, ids=lambda p: str(p.relative_to(_ROOT)))
def test_capability_readme_links_source(package: Path) -> None:
module = package.relative_to(_ROOT / 'pydantic_ai_harness').as_posix()
expected = f'{_SOURCE_LINK}{module}/'
targets = _markdown_link_targets((package / 'README.md').read_text(encoding='utf-8'))
assert any(expected in t for t in targets), (
f'{package.relative_to(_ROOT) / "README.md"} must link its own source module '
f'(a Markdown link containing `{expected}`), so parity tooling can find the implementation.'
)