furyhawk/pydantic-ai-harness
1
0
Fork 0
mirror of https://github.com/pydantic/pydantic-ai-harness.git synced 2026-07-21 02:45:34 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: reconcile README with shipped capabilities; enforce parity in CI (#342)

Browse Source 
* docs: reconcile README with shipped capabilities; enforce parity in CI

The README undersold the package: the "capability matrix" still listed
capabilities as unmerged PRs after they shipped as experimental/*, the
version floor was three minors stale, and the ecosystem example used an
API (SubAgentConfig) that no longer exists. Readers could not tell what
actually ships.

Restructure around what ships today (4 stable + 10 experimental, each true
to its exported classes) vs a roadmap of only unshipped work, and add a
mechanical docs-parity test so a capability can no longer land without
showing up in the README. Also add the missing media README and refresh
the agent skill's capability list and version floor.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test: mark unreachable non-package guard as no-cover

The __init__.py guard in _capability_packages() never triggers in a clean
tree (__pycache__ is already filtered by the underscore-prefix check), so the
branch was uncovered and dropped total coverage below the 100% floor.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: keep README's capability matrix, add shipped rows

Revert the README to main's structure and make the update additive: flip
the newly-shipped capabilities to shipped in the existing capability matrix
and add rows for the ones that were not listed (ManagedPrompt, PyaiDocs,
RuntimeAuthoring, Media), rather than restructuring the page. Bump the
pydantic-ai-slim floor to 2.1.0. Fix the SKILL.md anchors that pointed at
the removed sections.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: add Dynamic Workflows (#273) roadmap row to matrix

Reconcile against the 7/7 harness state sweep: #273 is complete but open
(not merged, no package on main), so it belongs in the matrix as a tracked
roadmap row. It was the one item from the sweep still missing after the
shipped-row updates.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Aditya Vardhan
2026-07-09 15:20:17 +05:30
committed by GitHub
co-authored by Claude Opus 4.8
parent b4365440b5
commit 77c08faeb7
4 changed files with 159 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+15 -10
View File
@@ -33,7 +33,7 @@ uv add "pydantic-ai-harness[acp]" # ACP (serve an agent to editors over
The `code-mode` extra is also supported as an alias.
Requires Python 3.10+ and `pydantic-ai-slim>=1.95.1`.
Requires Python 3.10+ and `pydantic-ai-slim>=2.1.0`.
## Quick start
@@ -107,20 +107,25 @@ We studied leading coding agents, agent frameworks, and Claw-style assistants to
| | **Tool search** | Progressive tool discovery for large tool sets | :white_check_mark: [Pydantic&nbsp;AI](https://pydantic.dev/docs/ai/tools-toolsets/toolsets/#deferred-loading) | |
| | **File system** | Read, write, edit, search files with path traversal prevention | :white_check_mark: [Docs](pydantic_ai_harness/filesystem/) | [pydantic-ai-backend](https://github.com/vstorm-co/pydantic-ai-backend) (vstorm&#8209;co) |
| | **Shell** | Execute commands with allowlists, denylists, and timeouts | :white_check_mark: [Docs](pydantic_ai_harness/shell/) | [pydantic-ai-backend](https://github.com/vstorm-co/pydantic-ai-backend) (vstorm&#8209;co) |
| | **Repo context injection** | Auto-load CLAUDE.md/AGENTS.md and repo structure | :construction: [PR&nbsp;#175](https://github.com/pydantic/pydantic-ai-harness/pull/175) | [pydantic-deep](https://github.com/vstorm-co/pydantic-deepagents) (vstorm&#8209;co) |
| | **Repo context injection** | Auto-load CLAUDE.md/AGENTS.md and repo structure | :white_check_mark: [Docs](pydantic_ai_harness/experimental/context/) (experimental) | [pydantic-deep](https://github.com/vstorm-co/pydantic-deepagents) (vstorm&#8209;co) |
| | **Docs lookup** | On-demand `read_pyai_docs` tool for Pydantic AI docs | :white_check_mark: [Docs](pydantic_ai_harness/experimental/docs/) (experimental) | |
| | **Verification loop** | Run tests after edits, auto-fix failures | :construction: [PR&nbsp;#169](https://github.com/pydantic/pydantic-ai-harness/pull/169) | |
| **Editor integration** | **ACP** | Serve an agent to editors (Zed, etc.) over the [Agent Client Protocol](https://agentclientprotocol.com) -- streamed text, diff-rendered edits, tool approval | :white_check_mark: [Docs](pydantic_ai_harness/experimental/acp/) (experimental) | |
| **Context management** | **Sliding window** | Trim conversation history to stay within token limits | :construction: [PR&nbsp;#191](https://github.com/pydantic/pydantic-ai-harness/pull/191) | [summarization-pydantic-ai](https://github.com/vstorm-co/summarization-pydantic-ai) (vstorm&#8209;co) |
| | **Context compaction** | LLM-powered summarization of older messages | :construction: [PR&nbsp;#191](https://github.com/pydantic/pydantic-ai-harness/pull/191) | [summarization-pydantic-ai](https://github.com/vstorm-co/summarization-pydantic-ai) (vstorm&#8209;co) |
| | **Limit warnings** | Warn agent before hitting context/iteration limits | :construction: [PR&nbsp;#191](https://github.com/pydantic/pydantic-ai-harness/pull/191) | [summarization-pydantic-ai](https://github.com/vstorm-co/summarization-pydantic-ai) (vstorm&#8209;co) |
| | **Tool output management** | Truncate, summarize, or spill large tool outputs | :construction: [PR&nbsp;#185](https://github.com/pydantic/pydantic-ai-harness/pull/185) | |
| **Prompt management** | **Managed prompt** | Back an agent's instructions with a [Logfire](https://pydantic.dev/logfire)-managed prompt, editable without shipping code | :white_check_mark: [Docs](pydantic_ai_harness/logfire/) | |
| **Context management** | **Sliding window** | Trim conversation history to stay within token limits | :white_check_mark: [Docs](pydantic_ai_harness/experimental/compaction/) (experimental) | [summarization-pydantic-ai](https://github.com/vstorm-co/summarization-pydantic-ai) (vstorm&#8209;co) |
| | **Context compaction** | LLM-powered summarization of older messages | :white_check_mark: [Docs](pydantic_ai_harness/experimental/compaction/) (experimental) | [summarization-pydantic-ai](https://github.com/vstorm-co/summarization-pydantic-ai) (vstorm&#8209;co) |
| | **Limit warnings** | Warn agent before hitting context/iteration limits | :white_check_mark: [Docs](pydantic_ai_harness/experimental/compaction/) (experimental) | [summarization-pydantic-ai](https://github.com/vstorm-co/summarization-pydantic-ai) (vstorm&#8209;co) |
| | **Tool output management** | Truncate, summarize, or spill large tool outputs | :white_check_mark: [Docs](pydantic_ai_harness/experimental/overflow/) (experimental) | |
| | **System reminders** | Inject periodic reminders to counteract instruction drift | :construction: [PR&nbsp;#181](https://github.com/pydantic/pydantic-ai-harness/pull/181) | |
| **Memory &&nbsp;persistence** | **Memory** | Persistent key-value memory across sessions | :construction: [PR&nbsp;#179](https://github.com/pydantic/pydantic-ai-harness/pull/179) | [pydantic-deep](https://github.com/vstorm-co/pydantic-deepagents) (vstorm&#8209;co) |
| | **Session persistence** | Save and restore full conversation state | :construction: [PR&nbsp;#176](https://github.com/pydantic/pydantic-ai-harness/pull/176) | |
| | **Checkpointing** | Save, rewind, and fork conversation state | :memo: [#196](https://github.com/pydantic/pydantic-ai-harness/issues/196) | [pydantic-deep](https://github.com/vstorm-co/pydantic-deepagents) (vstorm&#8209;co) |
| **Agent orchestration** | **Sub-agents** | Delegate subtasks to specialized child agents | :construction: [PR&nbsp;#178](https://github.com/pydantic/pydantic-ai-harness/pull/178) | [subagents-pydantic-ai](https://github.com/vstorm-co/subagents-pydantic-ai) (vstorm&#8209;co) |
| | **Session persistence** | Save and restore full conversation state | :white_check_mark: [Docs](pydantic_ai_harness/experimental/step_persistence/) (experimental) | |
| | **Checkpointing** | Snapshot, resume (`continue_run`), and fork (`fork_run`) a run | :white_check_mark: [Docs](pydantic_ai_harness/experimental/step_persistence/) (experimental) | [pydantic-deep](https://github.com/vstorm-co/pydantic-deepagents) (vstorm&#8209;co) |
| | **Media externalization** | Offload large `BinaryContent` to content-addressed stores (building blocks) | :white_check_mark: [Docs](pydantic_ai_harness/experimental/media/) (experimental) | |
| **Agent orchestration** | **Sub-agents** | Delegate subtasks to specialized child agents | :white_check_mark: [Docs](pydantic_ai_harness/experimental/subagents/) (experimental) | [subagents-pydantic-ai](https://github.com/vstorm-co/subagents-pydantic-ai) (vstorm&#8209;co) |
| | **Dynamic workflows** | Author and run typed multi-agent workflows as sandboxed code | :construction: [PR&nbsp;#273](https://github.com/pydantic/pydantic-ai-harness/pull/273) | |
| | **Skills** | Progressive tool loading -- search, activate, deactivate | :construction: [PR&nbsp;#183](https://github.com/pydantic/pydantic-ai-harness/pull/183) | [pydantic-ai-skills](https://github.com/DougTrajano/pydantic-ai-skills) (DougTrajano), [pydantic-deep](https://github.com/vstorm-co/pydantic-deepagents) (vstorm&#8209;co) |
| | **Planning** | Break complex tasks into structured plans before execution | :construction: [PR&nbsp;#180](https://github.com/pydantic/pydantic-ai-harness/pull/180) | |
| | **Planning** | Break complex tasks into structured plans before execution | :white_check_mark: [Docs](pydantic_ai_harness/experimental/planning/) (experimental) | |
| | **Runtime authoring** | Let an agent author, validate, and load real capabilities at runtime | :white_check_mark: [Docs](pydantic_ai_harness/experimental/authoring/) (experimental) | |
| | **Task tracking** | Track tasks, subtasks, and dependencies | :memo: [#65](https://github.com/pydantic/pydantic-ai-harness/issues/65) | [pydantic-ai-todo](https://github.com/vstorm-co/pydantic-ai-todo) (vstorm&#8209;co) |
| | **Teams** | Multi-agent teams with shared state and message bus | :memo: [#195](https://github.com/pydantic/pydantic-ai-harness/issues/195) | [pydantic-deep](https://github.com/vstorm-co/pydantic-deepagents) (vstorm&#8209;co) |
| **Safety &&nbsp;guardrails** | **Input guardrails** | Validate user input before the agent run starts | :construction: [PR&nbsp;#182](https://github.com/pydantic/pydantic-ai-harness/pull/182) | [pydantic-ai-shields](https://github.com/vstorm-co/pydantic-ai-shields) (vstorm&#8209;co) |
pydantic_ai_harness/.agents/skills/pydantic-ai-harness/SKILL.md
+17 -6
View File
@@ -1,8 +1,8 @@
---
name: pydantic-ai-harness
description: Extend Pydantic AI agents with batteries-included capabilities from pydantic-ai-harness -- currently Code Mode, which collapses many tool calls into one sandboxed Python execution. Use when the user mentions pydantic-ai-harness, CodeMode, Monty, code mode, or tool sandboxing, when they want an agent to run agent-written Python, or when a Pydantic AI agent would benefit from orchestrating multiple tool calls in a single sandboxed script.
description: Extend Pydantic AI agents with batteries-included capabilities from pydantic-ai-harness -- Code Mode (collapse many tool calls into one sandboxed Python execution), a filesystem and shell, sub-agents, planning, context compaction, and more. Use when the user mentions pydantic-ai-harness, CodeMode, Monty, code mode, or tool sandboxing, when they want first-party filesystem/shell/sub-agent/planning/compaction capabilities for a Pydantic AI agent, when they want an agent to run agent-written Python, or when a Pydantic AI agent would benefit from orchestrating multiple tool calls in a single sandboxed script.
license: MIT
compatibility: Requires Python 3.10+ and pydantic-ai-slim>=1.95.1
compatibility: Requires Python 3.10+ and pydantic-ai-slim>=2.1.0
metadata:
version: "0.1.0"
author: pydantic
@@ -33,13 +33,24 @@ Do **not** use this skill for:
## Supported Capabilities
`CodeMode` has a full reference below; it is the flagship capability and the one this skill goes deep on.
The rest ship today and each has its own README with API and examples.
**Stable** (imported from `pydantic_ai_harness`):
| Capability | Description | Reference |
|---|---|---|
| `CodeMode` | Wraps eligible tools into a single sandboxed `run_code` tool so the model orchestrates them in Python | [Code Mode](./references/CODE-MODE.md) |
| `FileSystem` | Read, write, edit, and search files under a root directory, with traversal prevention | [README](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/filesystem) |
| `Shell` | Run commands in a subprocess with allowlists, a default denylist, timeouts, and env masking | [README](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/shell) |
| `ManagedPrompt` | Back an agent's instructions with a Logfire-managed prompt | [README](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/logfire) |
More capability areas are tracked in the
[capability matrix](https://github.com/pydantic/pydantic-ai-harness#capability-matrix); as they stabilize,
this skill grows to cover them.
**Experimental** (imported from `pydantic_ai_harness.experimental`, may change in any release): `SubAgents`,
`Planning`, a compaction family (`SlidingWindow`, `SummarizingCompaction`, ...), `OverflowingToolOutput`,
`RepoContext`, `StepPersistence`, `PyaiDocs`, `RuntimeAuthoring`, and an ACP server adapter.
The full, current list with links and status is in the
[capability matrix](https://github.com/pydantic/pydantic-ai-harness#capability-matrix).
## Install
@@ -53,7 +64,7 @@ Each capability declares its own extra. Code Mode needs the Monty sandbox:
uv add "pydantic-ai-harness[codemode]" # `code-mode` is also accepted as an alias
```
Requires Python 3.10+ and `pydantic-ai-slim>=1.95.1`.
Requires Python 3.10+ and `pydantic-ai-slim>=2.1.0`.
## Quick Start
pydantic_ai_harness/experimental/media/README.md
+61
View File
@@ -0,0 +1,61 @@
# Media
> **Experimental.** Imported from `pydantic_ai_harness.experimental.media`, this module can change or be removed in any release. Importing it emits a `HarnessExperimentalWarning`.
```python
from pydantic_ai_harness.experimental.media import (
DiskMediaStore,
S3MediaStore,
SqliteMediaStore,
externalize_media,
restore_media,
)
```
Content-addressed stores and walker helpers that move large binary payloads out of message history and put them back on demand.
These are building blocks, not a capability. There is no class you add to `Agent(capabilities=[...])` yet. [`StepPersistence`](../step_persistence/) uses them to keep snapshots small when messages carry `BinaryContent`. A forthcoming `MediaExternalizer` capability will reuse the same stores to rewrite `BinaryContent` into URL parts before the model sees them.
## Why
A conversation that carries images, audio, or other `BinaryContent` inlines those bytes into every message. Persist that history and each snapshot re-serializes the payloads. Content-addressed storage writes each payload once, keyed by its own hash, and leaves a short `media://` URI in its place. The same bytes are stored once no matter how many messages or snapshots reference them.
## Stores
Every store implements the `MediaStore` protocol: `put`, `get`, `exists`, `public_url`, and `get_metadata`, all async and content-addressed (the URI is derived from the payload hash, so identical bytes deduplicate).
| Store | Backed by | Use when |
|---|---|---|
| `DiskMediaStore(directory=...)` | A directory on disk | Local runs and tests |
| `SqliteMediaStore(...)` | A SQLite database | A single-file store that travels with the data |
| `S3MediaStore(...)` | S3 or an S3-compatible bucket | Shared or production storage |
A `KeyStrategy` controls the on-store layout, and a `PublicUrlResolver` (or `make_static_public_url`) turns a stored URI into a public URL when the store is served over HTTP.
## Walker helpers
`externalize_media` and `restore_media` walk a message node and swap payloads for URIs and back.
```python
store = DiskMediaStore(directory='./media')
# Replace BinaryContent larger than the threshold with media:// URIs.
lean = await externalize_media(message, media_store=store, threshold_bytes=32_000)
# Later, rehydrate the URIs back into BinaryContent.
full = await restore_media(lean, media_store=store)
```
`externalize_media` only externalizes payloads over `threshold_bytes`; smaller ones stay inline. `media_uri_for` and `parse_media_uri` give you the raw URI round-trip if you need to key media yourself.
## API
| Symbol | Purpose |
|---|---|
| `MediaStore` | Async content-addressed store protocol (`put` / `get` / `exists` / `public_url` / `get_metadata`) |
| `DiskMediaStore`, `SqliteMediaStore`, `S3MediaStore` | Concrete stores |
| `MediaContext` | Per-call context (e.g. tenant) threaded through store operations |
| `KeyStrategy`, `default_key_strategy` | On-store key layout |
| `PublicUrlResolver`, `make_static_public_url` | Resolve a stored URI to a public URL |
| `externalize_media`, `restore_media` | Walk a message node to externalize / rehydrate payloads |
| `media_uri_for`, `parse_media_uri` | Compute and parse a `media://` URI |
tests/test_docs_parity.py
+66
View File
@@ -0,0 +1,66 @@
"""Keep the README honest about what ships.
Every capability package must document itself with a `README.md` and be linked
from the top-level `README.md`. A capability cannot land without showing up in
the docs, so the "what's available today" tables cannot silently fall behind the
code. This is the mechanical half of docs parity; the semantic half (does the
prose match the code as written) is a review-time concern, not a unit test.
"""
from __future__ import annotations
from pathlib import Path
import pytest
_ROOT = Path(__file__).parent.parent
_PACKAGE = _ROOT / 'pydantic_ai_harness'
# The `experimental` package is a namespace/warning shim, not a capability, so it
# has no standalone README and is not listed in the top-level tables.
_NAMESPACE_PACKAGES = {_PACKAGE / 'experimental'}
def _capability_packages() -> list[Path]:
"""Directories that are importable packages and represent a capability's public surface."""
candidates: list[Path] = []
for parent in (_PACKAGE, _PACKAGE / 'experimental'):
for child in sorted(parent.iterdir()):
if not child.is_dir() or child.name.startswith(('_', '.')):
continue
# A non-package dir under the capability roots does not occur in a clean tree, so this guard stays uncovered.
if not (child / '__init__.py').exists(): # pragma: no cover
continue
if child in _NAMESPACE_PACKAGES:
continue
candidates.append(child)
return candidates
_CAPABILITY_PACKAGES = _capability_packages()
def test_capability_packages_discovered() -> None:
# Guard against the discovery silently finding nothing (e.g. a moved package root),
# which would make the parametrized checks below vacuously pass.
assert len(_CAPABILITY_PACKAGES) >= 10
@pytest.mark.parametrize('package', _CAPABILITY_PACKAGES, ids=lambda p: str(p.relative_to(_ROOT)))
def test_capability_has_readme(package: Path) -> None:
readme = package / 'README.md'
assert readme.exists(), (
f'{package.relative_to(_ROOT)} is an importable capability package but has no README.md. '
'Add one (start from an existing capability README) so its public surface is documented.'
)
@pytest.mark.parametrize('package', _CAPABILITY_PACKAGES, ids=lambda p: str(p.relative_to(_ROOT)))
def test_capability_linked_from_top_readme(package: Path) -> None:
top_readme = (_ROOT / 'README.md').read_text(encoding='utf-8')
link_target = f'{package.relative_to(_ROOT).as_posix()}/'
assert link_target in top_readme, (
f'{package.relative_to(_ROOT)} is not linked from the top-level README.md. '
f'Add a row for it (linking `{link_target}`) to the "What\'s available today" or "Roadmap" tables '
'so the README stays in step with the code.'
)