From 77c08faeb73a43da7563e91b09b8386520cf1771 Mon Sep 17 00:00:00 2001 From: Aditya Vardhan <76904033+adtyavrdhn@users.noreply.github.com> Date: Thu, 9 Jul 2026 15:20:17 +0530 Subject: [PATCH] docs: reconcile README with shipped capabilities; enforce parity in CI (#342) * docs: reconcile README with shipped capabilities; enforce parity in CI The README undersold the package: the "capability matrix" still listed capabilities as unmerged PRs after they shipped as experimental/*, the version floor was three minors stale, and the ecosystem example used an API (SubAgentConfig) that no longer exists. Readers could not tell what actually ships. Restructure around what ships today (4 stable + 10 experimental, each true to its exported classes) vs a roadmap of only unshipped work, and add a mechanical docs-parity test so a capability can no longer land without showing up in the README. Also add the missing media README and refresh the agent skill's capability list and version floor. Co-Authored-By: Claude Opus 4.8 (1M context) * test: mark unreachable non-package guard as no-cover The __init__.py guard in _capability_packages() never triggers in a clean tree (__pycache__ is already filtered by the underscore-prefix check), so the branch was uncovered and dropped total coverage below the 100% floor. Co-Authored-By: Claude Opus 4.8 (1M context) * docs: keep README's capability matrix, add shipped rows Revert the README to main's structure and make the update additive: flip the newly-shipped capabilities to shipped in the existing capability matrix and add rows for the ones that were not listed (ManagedPrompt, PyaiDocs, RuntimeAuthoring, Media), rather than restructuring the page. Bump the pydantic-ai-slim floor to 2.1.0. Fix the SKILL.md anchors that pointed at the removed sections. Co-Authored-By: Claude Opus 4.8 (1M context) * docs: add Dynamic Workflows (#273) roadmap row to matrix Reconcile against the 7/7 harness state sweep: #273 is complete but open (not merged, no package on main), so it belongs in the matrix as a tracked roadmap row. It was the one item from the sweep still missing after the shipped-row updates. Co-Authored-By: Claude Opus 4.8 (1M context) --------- Co-authored-by: Claude Opus 4.8 (1M context) --- README.md | 25 ++++--- .../skills/pydantic-ai-harness/SKILL.md | 23 +++++-- .../experimental/media/README.md | 61 +++++++++++++++++ tests/test_docs_parity.py | 66 +++++++++++++++++++ 4 files changed, 159 insertions(+), 16 deletions(-) create mode 100644 pydantic_ai_harness/experimental/media/README.md create mode 100644 tests/test_docs_parity.py diff --git a/README.md b/README.md index 5967778..59e5df8 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,7 @@ uv add "pydantic-ai-harness[acp]" # ACP (serve an agent to editors over The `code-mode` extra is also supported as an alias. -Requires Python 3.10+ and `pydantic-ai-slim>=1.95.1`. +Requires Python 3.10+ and `pydantic-ai-slim>=2.1.0`. ## Quick start @@ -107,20 +107,25 @@ We studied leading coding agents, agent frameworks, and Claw-style assistants to | | **Tool search** | Progressive tool discovery for large tool sets | :white_check_mark: [Pydantic AI](https://pydantic.dev/docs/ai/tools-toolsets/toolsets/#deferred-loading) | | | | **File system** | Read, write, edit, search files with path traversal prevention | :white_check_mark: [Docs](pydantic_ai_harness/filesystem/) | [pydantic-ai-backend](https://github.com/vstorm-co/pydantic-ai-backend) (vstorm‑co) | | | **Shell** | Execute commands with allowlists, denylists, and timeouts | :white_check_mark: [Docs](pydantic_ai_harness/shell/) | [pydantic-ai-backend](https://github.com/vstorm-co/pydantic-ai-backend) (vstorm‑co) | -| | **Repo context injection** | Auto-load CLAUDE.md/AGENTS.md and repo structure | :construction: [PR #175](https://github.com/pydantic/pydantic-ai-harness/pull/175) | [pydantic-deep](https://github.com/vstorm-co/pydantic-deepagents) (vstorm‑co) | +| | **Repo context injection** | Auto-load CLAUDE.md/AGENTS.md and repo structure | :white_check_mark: [Docs](pydantic_ai_harness/experimental/context/) (experimental) | [pydantic-deep](https://github.com/vstorm-co/pydantic-deepagents) (vstorm‑co) | +| | **Docs lookup** | On-demand `read_pyai_docs` tool for Pydantic AI docs | :white_check_mark: [Docs](pydantic_ai_harness/experimental/docs/) (experimental) | | | | **Verification loop** | Run tests after edits, auto-fix failures | :construction: [PR #169](https://github.com/pydantic/pydantic-ai-harness/pull/169) | | | **Editor integration** | **ACP** | Serve an agent to editors (Zed, etc.) over the [Agent Client Protocol](https://agentclientprotocol.com) -- streamed text, diff-rendered edits, tool approval | :white_check_mark: [Docs](pydantic_ai_harness/experimental/acp/) (experimental) | | -| **Context management** | **Sliding window** | Trim conversation history to stay within token limits | :construction: [PR #191](https://github.com/pydantic/pydantic-ai-harness/pull/191) | [summarization-pydantic-ai](https://github.com/vstorm-co/summarization-pydantic-ai) (vstorm‑co) | -| | **Context compaction** | LLM-powered summarization of older messages | :construction: [PR #191](https://github.com/pydantic/pydantic-ai-harness/pull/191) | [summarization-pydantic-ai](https://github.com/vstorm-co/summarization-pydantic-ai) (vstorm‑co) | -| | **Limit warnings** | Warn agent before hitting context/iteration limits | :construction: [PR #191](https://github.com/pydantic/pydantic-ai-harness/pull/191) | [summarization-pydantic-ai](https://github.com/vstorm-co/summarization-pydantic-ai) (vstorm‑co) | -| | **Tool output management** | Truncate, summarize, or spill large tool outputs | :construction: [PR #185](https://github.com/pydantic/pydantic-ai-harness/pull/185) | | +| **Prompt management** | **Managed prompt** | Back an agent's instructions with a [Logfire](https://pydantic.dev/logfire)-managed prompt, editable without shipping code | :white_check_mark: [Docs](pydantic_ai_harness/logfire/) | | +| **Context management** | **Sliding window** | Trim conversation history to stay within token limits | :white_check_mark: [Docs](pydantic_ai_harness/experimental/compaction/) (experimental) | [summarization-pydantic-ai](https://github.com/vstorm-co/summarization-pydantic-ai) (vstorm‑co) | +| | **Context compaction** | LLM-powered summarization of older messages | :white_check_mark: [Docs](pydantic_ai_harness/experimental/compaction/) (experimental) | [summarization-pydantic-ai](https://github.com/vstorm-co/summarization-pydantic-ai) (vstorm‑co) | +| | **Limit warnings** | Warn agent before hitting context/iteration limits | :white_check_mark: [Docs](pydantic_ai_harness/experimental/compaction/) (experimental) | [summarization-pydantic-ai](https://github.com/vstorm-co/summarization-pydantic-ai) (vstorm‑co) | +| | **Tool output management** | Truncate, summarize, or spill large tool outputs | :white_check_mark: [Docs](pydantic_ai_harness/experimental/overflow/) (experimental) | | | | **System reminders** | Inject periodic reminders to counteract instruction drift | :construction: [PR #181](https://github.com/pydantic/pydantic-ai-harness/pull/181) | | | **Memory & persistence** | **Memory** | Persistent key-value memory across sessions | :construction: [PR #179](https://github.com/pydantic/pydantic-ai-harness/pull/179) | [pydantic-deep](https://github.com/vstorm-co/pydantic-deepagents) (vstorm‑co) | -| | **Session persistence** | Save and restore full conversation state | :construction: [PR #176](https://github.com/pydantic/pydantic-ai-harness/pull/176) | | -| | **Checkpointing** | Save, rewind, and fork conversation state | :memo: [#196](https://github.com/pydantic/pydantic-ai-harness/issues/196) | [pydantic-deep](https://github.com/vstorm-co/pydantic-deepagents) (vstorm‑co) | -| **Agent orchestration** | **Sub-agents** | Delegate subtasks to specialized child agents | :construction: [PR #178](https://github.com/pydantic/pydantic-ai-harness/pull/178) | [subagents-pydantic-ai](https://github.com/vstorm-co/subagents-pydantic-ai) (vstorm‑co) | +| | **Session persistence** | Save and restore full conversation state | :white_check_mark: [Docs](pydantic_ai_harness/experimental/step_persistence/) (experimental) | | +| | **Checkpointing** | Snapshot, resume (`continue_run`), and fork (`fork_run`) a run | :white_check_mark: [Docs](pydantic_ai_harness/experimental/step_persistence/) (experimental) | [pydantic-deep](https://github.com/vstorm-co/pydantic-deepagents) (vstorm‑co) | +| | **Media externalization** | Offload large `BinaryContent` to content-addressed stores (building blocks) | :white_check_mark: [Docs](pydantic_ai_harness/experimental/media/) (experimental) | | +| **Agent orchestration** | **Sub-agents** | Delegate subtasks to specialized child agents | :white_check_mark: [Docs](pydantic_ai_harness/experimental/subagents/) (experimental) | [subagents-pydantic-ai](https://github.com/vstorm-co/subagents-pydantic-ai) (vstorm‑co) | +| | **Dynamic workflows** | Author and run typed multi-agent workflows as sandboxed code | :construction: [PR #273](https://github.com/pydantic/pydantic-ai-harness/pull/273) | | | | **Skills** | Progressive tool loading -- search, activate, deactivate | :construction: [PR #183](https://github.com/pydantic/pydantic-ai-harness/pull/183) | [pydantic-ai-skills](https://github.com/DougTrajano/pydantic-ai-skills) (DougTrajano), [pydantic-deep](https://github.com/vstorm-co/pydantic-deepagents) (vstorm‑co) | -| | **Planning** | Break complex tasks into structured plans before execution | :construction: [PR #180](https://github.com/pydantic/pydantic-ai-harness/pull/180) | | +| | **Planning** | Break complex tasks into structured plans before execution | :white_check_mark: [Docs](pydantic_ai_harness/experimental/planning/) (experimental) | | +| | **Runtime authoring** | Let an agent author, validate, and load real capabilities at runtime | :white_check_mark: [Docs](pydantic_ai_harness/experimental/authoring/) (experimental) | | | | **Task tracking** | Track tasks, subtasks, and dependencies | :memo: [#65](https://github.com/pydantic/pydantic-ai-harness/issues/65) | [pydantic-ai-todo](https://github.com/vstorm-co/pydantic-ai-todo) (vstorm‑co) | | | **Teams** | Multi-agent teams with shared state and message bus | :memo: [#195](https://github.com/pydantic/pydantic-ai-harness/issues/195) | [pydantic-deep](https://github.com/vstorm-co/pydantic-deepagents) (vstorm‑co) | | **Safety & guardrails** | **Input guardrails** | Validate user input before the agent run starts | :construction: [PR #182](https://github.com/pydantic/pydantic-ai-harness/pull/182) | [pydantic-ai-shields](https://github.com/vstorm-co/pydantic-ai-shields) (vstorm‑co) | diff --git a/pydantic_ai_harness/.agents/skills/pydantic-ai-harness/SKILL.md b/pydantic_ai_harness/.agents/skills/pydantic-ai-harness/SKILL.md index af7d3d9..3e86ab9 100644 --- a/pydantic_ai_harness/.agents/skills/pydantic-ai-harness/SKILL.md +++ b/pydantic_ai_harness/.agents/skills/pydantic-ai-harness/SKILL.md @@ -1,8 +1,8 @@ --- name: pydantic-ai-harness -description: Extend Pydantic AI agents with batteries-included capabilities from pydantic-ai-harness -- currently Code Mode, which collapses many tool calls into one sandboxed Python execution. Use when the user mentions pydantic-ai-harness, CodeMode, Monty, code mode, or tool sandboxing, when they want an agent to run agent-written Python, or when a Pydantic AI agent would benefit from orchestrating multiple tool calls in a single sandboxed script. +description: Extend Pydantic AI agents with batteries-included capabilities from pydantic-ai-harness -- Code Mode (collapse many tool calls into one sandboxed Python execution), a filesystem and shell, sub-agents, planning, context compaction, and more. Use when the user mentions pydantic-ai-harness, CodeMode, Monty, code mode, or tool sandboxing, when they want first-party filesystem/shell/sub-agent/planning/compaction capabilities for a Pydantic AI agent, when they want an agent to run agent-written Python, or when a Pydantic AI agent would benefit from orchestrating multiple tool calls in a single sandboxed script. license: MIT -compatibility: Requires Python 3.10+ and pydantic-ai-slim>=1.95.1 +compatibility: Requires Python 3.10+ and pydantic-ai-slim>=2.1.0 metadata: version: "0.1.0" author: pydantic @@ -33,13 +33,24 @@ Do **not** use this skill for: ## Supported Capabilities +`CodeMode` has a full reference below; it is the flagship capability and the one this skill goes deep on. +The rest ship today and each has its own README with API and examples. + +**Stable** (imported from `pydantic_ai_harness`): + | Capability | Description | Reference | |---|---|---| | `CodeMode` | Wraps eligible tools into a single sandboxed `run_code` tool so the model orchestrates them in Python | [Code Mode](./references/CODE-MODE.md) | +| `FileSystem` | Read, write, edit, and search files under a root directory, with traversal prevention | [README](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/filesystem) | +| `Shell` | Run commands in a subprocess with allowlists, a default denylist, timeouts, and env masking | [README](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/shell) | +| `ManagedPrompt` | Back an agent's instructions with a Logfire-managed prompt | [README](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/logfire) | -More capability areas are tracked in the -[capability matrix](https://github.com/pydantic/pydantic-ai-harness#capability-matrix); as they stabilize, -this skill grows to cover them. +**Experimental** (imported from `pydantic_ai_harness.experimental`, may change in any release): `SubAgents`, +`Planning`, a compaction family (`SlidingWindow`, `SummarizingCompaction`, ...), `OverflowingToolOutput`, +`RepoContext`, `StepPersistence`, `PyaiDocs`, `RuntimeAuthoring`, and an ACP server adapter. + +The full, current list with links and status is in the +[capability matrix](https://github.com/pydantic/pydantic-ai-harness#capability-matrix). ## Install @@ -53,7 +64,7 @@ Each capability declares its own extra. Code Mode needs the Monty sandbox: uv add "pydantic-ai-harness[codemode]" # `code-mode` is also accepted as an alias ``` -Requires Python 3.10+ and `pydantic-ai-slim>=1.95.1`. +Requires Python 3.10+ and `pydantic-ai-slim>=2.1.0`. ## Quick Start diff --git a/pydantic_ai_harness/experimental/media/README.md b/pydantic_ai_harness/experimental/media/README.md new file mode 100644 index 0000000..7fc25cd --- /dev/null +++ b/pydantic_ai_harness/experimental/media/README.md @@ -0,0 +1,61 @@ +# Media + +> **Experimental.** Imported from `pydantic_ai_harness.experimental.media`, this module can change or be removed in any release. Importing it emits a `HarnessExperimentalWarning`. + +```python +from pydantic_ai_harness.experimental.media import ( + DiskMediaStore, + S3MediaStore, + SqliteMediaStore, + externalize_media, + restore_media, +) +``` + +Content-addressed stores and walker helpers that move large binary payloads out of message history and put them back on demand. + +These are building blocks, not a capability. There is no class you add to `Agent(capabilities=[...])` yet. [`StepPersistence`](../step_persistence/) uses them to keep snapshots small when messages carry `BinaryContent`. A forthcoming `MediaExternalizer` capability will reuse the same stores to rewrite `BinaryContent` into URL parts before the model sees them. + +## Why + +A conversation that carries images, audio, or other `BinaryContent` inlines those bytes into every message. Persist that history and each snapshot re-serializes the payloads. Content-addressed storage writes each payload once, keyed by its own hash, and leaves a short `media://` URI in its place. The same bytes are stored once no matter how many messages or snapshots reference them. + +## Stores + +Every store implements the `MediaStore` protocol: `put`, `get`, `exists`, `public_url`, and `get_metadata`, all async and content-addressed (the URI is derived from the payload hash, so identical bytes deduplicate). + +| Store | Backed by | Use when | +|---|---|---| +| `DiskMediaStore(directory=...)` | A directory on disk | Local runs and tests | +| `SqliteMediaStore(...)` | A SQLite database | A single-file store that travels with the data | +| `S3MediaStore(...)` | S3 or an S3-compatible bucket | Shared or production storage | + +A `KeyStrategy` controls the on-store layout, and a `PublicUrlResolver` (or `make_static_public_url`) turns a stored URI into a public URL when the store is served over HTTP. + +## Walker helpers + +`externalize_media` and `restore_media` walk a message node and swap payloads for URIs and back. + +```python +store = DiskMediaStore(directory='./media') + +# Replace BinaryContent larger than the threshold with media:// URIs. +lean = await externalize_media(message, media_store=store, threshold_bytes=32_000) + +# Later, rehydrate the URIs back into BinaryContent. +full = await restore_media(lean, media_store=store) +``` + +`externalize_media` only externalizes payloads over `threshold_bytes`; smaller ones stay inline. `media_uri_for` and `parse_media_uri` give you the raw URI round-trip if you need to key media yourself. + +## API + +| Symbol | Purpose | +|---|---| +| `MediaStore` | Async content-addressed store protocol (`put` / `get` / `exists` / `public_url` / `get_metadata`) | +| `DiskMediaStore`, `SqliteMediaStore`, `S3MediaStore` | Concrete stores | +| `MediaContext` | Per-call context (e.g. tenant) threaded through store operations | +| `KeyStrategy`, `default_key_strategy` | On-store key layout | +| `PublicUrlResolver`, `make_static_public_url` | Resolve a stored URI to a public URL | +| `externalize_media`, `restore_media` | Walk a message node to externalize / rehydrate payloads | +| `media_uri_for`, `parse_media_uri` | Compute and parse a `media://` URI | diff --git a/tests/test_docs_parity.py b/tests/test_docs_parity.py new file mode 100644 index 0000000..a652d3c --- /dev/null +++ b/tests/test_docs_parity.py @@ -0,0 +1,66 @@ +"""Keep the README honest about what ships. + +Every capability package must document itself with a `README.md` and be linked +from the top-level `README.md`. A capability cannot land without showing up in +the docs, so the "what's available today" tables cannot silently fall behind the +code. This is the mechanical half of docs parity; the semantic half (does the +prose match the code as written) is a review-time concern, not a unit test. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +_ROOT = Path(__file__).parent.parent +_PACKAGE = _ROOT / 'pydantic_ai_harness' + +# The `experimental` package is a namespace/warning shim, not a capability, so it +# has no standalone README and is not listed in the top-level tables. +_NAMESPACE_PACKAGES = {_PACKAGE / 'experimental'} + + +def _capability_packages() -> list[Path]: + """Directories that are importable packages and represent a capability's public surface.""" + candidates: list[Path] = [] + for parent in (_PACKAGE, _PACKAGE / 'experimental'): + for child in sorted(parent.iterdir()): + if not child.is_dir() or child.name.startswith(('_', '.')): + continue + # A non-package dir under the capability roots does not occur in a clean tree, so this guard stays uncovered. + if not (child / '__init__.py').exists(): # pragma: no cover + continue + if child in _NAMESPACE_PACKAGES: + continue + candidates.append(child) + return candidates + + +_CAPABILITY_PACKAGES = _capability_packages() + + +def test_capability_packages_discovered() -> None: + # Guard against the discovery silently finding nothing (e.g. a moved package root), + # which would make the parametrized checks below vacuously pass. + assert len(_CAPABILITY_PACKAGES) >= 10 + + +@pytest.mark.parametrize('package', _CAPABILITY_PACKAGES, ids=lambda p: str(p.relative_to(_ROOT))) +def test_capability_has_readme(package: Path) -> None: + readme = package / 'README.md' + assert readme.exists(), ( + f'{package.relative_to(_ROOT)} is an importable capability package but has no README.md. ' + 'Add one (start from an existing capability README) so its public surface is documented.' + ) + + +@pytest.mark.parametrize('package', _CAPABILITY_PACKAGES, ids=lambda p: str(p.relative_to(_ROOT))) +def test_capability_linked_from_top_readme(package: Path) -> None: + top_readme = (_ROOT / 'README.md').read_text(encoding='utf-8') + link_target = f'{package.relative_to(_ROOT).as_posix()}/' + assert link_target in top_readme, ( + f'{package.relative_to(_ROOT)} is not linked from the top-level README.md. ' + f'Add a row for it (linking `{link_target}`) to the "What\'s available today" or "Roadmap" tables ' + 'so the README stays in step with the code.' + )