mirror of
https://github.com/pydantic/pydantic-ai-harness.git
synced 2026-07-21 02:45:34 +00:00
main
12
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Douwe MaanandGitHub
|
43633b0b24 |
docs: point at the new per-capability pages in core's Capabilities section (#391)
Pydantic AI's docs restructure gives Capabilities a top-level section (/ai/capabilities/) with a page per built-in capability. Update all /ai/core-concepts/capabilities/ links accordingly, and make the core built-ins explicit in the 'What goes where?' breakdown — naming and linking web search, web fetch, image generation, compaction, tool search, on-demand loading, thinking, and MCP, plus the full built-in list on the Capabilities overview. Merge together with the pydantic-ai docs restructure release: the new /ai/capabilities/ URLs only exist once that release is synced to pydantic.dev. |
||
|
David SFandGitHub
|
6c60f7cf0d |
feat(step_persistence): rescue last provider-valid resume point on error (#384)
* feat(step_persistence): rescue last provider-valid resume point on error `StepPersistence` previously left an errored run with no `ContinuableSnapshot` for a provider-valid state that no node boundary had captured, so `continue_run` could only resume from an earlier point or raise `LookupError`. Add an error-path capture: when a run fails against a provider-valid history, save one snapshot so the run still exposes its last safe resume point. Two sites cover the two ways such a state is reached without a completed-node snapshot: - `on_model_request_error` saves the request payload directly -- e.g. a resolved tool cycle, which is only provider-valid as the next request is built, never at a `CallToolsNode` boundary. - `on_run_error` reads a `latest_node_history` contextvar refreshed at every `after_node_run` boundary, rescuing a text response whose following `CallToolsNode` raises inside output validation. It cannot read `ctx.messages` directly: the `RunContext` passed to `on_run_error` holds the start-of-run history, which the graph rebinds away during the run. Both saves are gated on `_is_resumable_history` (provider-valid plus at least one `ModelResponse`), so a crash mid-tool-call leaves a dangling `ToolCallPart` that is skipped and `latest_snapshot` never regresses to an unsendable point. `on_run_error` compares by message count and skips when the store already holds a newer snapshot, so a stale completed-node stash never supersedes a fresher `on_model_request_error` save. This is the gated (provider-validity) approach; the ungated simplification that rides on pydantic-ai#6319 is deferred to a follow-up. Closes #253 * test(step_persistence): drop dead validator branch to restore 100% coverage The `gate` output-validator in `test_does_not_regress_to_stale_stash_after_model_request_save` had a `return value` branch that never executed: the run fails on request 3 before any output passes validation a second time. Simplify it to always raise `ModelRetry` (one retry is all the stash scenario needs), matching the sibling `test_rescues_newest_text_history_over_earlier_saved_snapshot`. |
||
|
Aditya VardhanandGitHub
|
4ad83f8861 |
Clarify CodeMode final-expression returns (#370)
* Prevent empty CodeMode results through clearer guidance * Make CodeMode return guidance unambiguous |
||
|
|
818808d6fe |
feat: add Memory capability (#354)
* feat: add Memory capability * feat(memory): add SQLite and PostgreSQL stores * fix(memory): address CI pyright and CodeRabbit review * style(memory): ASCII dashes, labeled README fence (review follow-up) * fix(memory): SQLite in-memory rejection and case-sensitive listing * fix(memory): use qmark placeholders in SQLite prefix listing * Make memory safe for durable shared use * Harden memory trust and recovery boundaries --------- Co-authored-by: David SF <64162682+dsfaccini@users.noreply.github.com> |
||
|
David SFandGitHub
|
3ba9e2f9a5 |
docs: capability pages for the unified docs site + README/doc parity gate (#329)
* docs: publish capability docs to the unified site + add README/doc parity gate Every capability shipped only a README (kept for GitHub/PyPI). This adds a parallel, cleaned-up page per capability under docs/ for the new unified docs site (pydantic.dev/docs/harness), migrated from each README: snippets verified runnable against source, autodoc API blocks, root-relative Pydantic AI links, and an experimental-status admonition on the experimental set. To keep README and doc in sync going forward, adds a docs-parity-reviewer agent and a parity gate in the review checklist (run as the last step before merge), plus the docs/ layout and the README<->doc requirement in AGENTS.md and the capability-authoring guide. * docs: fix README<->doc<->source inconsistencies across capabilities A parity audit against source found drift, mostly in the capability READMEs (staler than the migrated docs). All fixes verified against source: - Correctness: the "approval/deferred tools are excluded from the sandbox" claim (code_mode README + doc) was false -- those tools are sandboxed like any other; corrected in both. The stale Shell persist_cwd sentinel description is replaced with the actual out-of-band temp-file capture. filesystem protected default `.git/` -> `.git/*` (the bare form never matched). - Runnable snippets: added the missing imports/wiring so README snippets no longer raise NameError (subagents, context, planning, overflow, authoring, filesystem, code_mode). - Parity: documented previously-undocumented params/behaviors (compaction strategy options, overflow strip_ansi/Passthrough, extra autodoc classes for context and subagents), fixed a stale version pin (>=1.95.1 -> >=2.1.0), and added the missing Managed Prompt row to the root README capability matrix. - Style: normalized decorative Unicode to ASCII across all READMEs and dropped a hype phrase, matching AGENTS.md writing style and the docs. * docs: add nav.json to drive the unified-docs harness sidebar The unified docs mount the harness docs under /docs/ai/harness (fed live from this repo via the pydantic-ai 'Pydantic AI Harness' section). This nav.json defines the sub-nav (Overview + Capabilities + Experimental) and the set of doc files the site includes. * docs: migrate "What goes where?" explainer into harness overview Adds the core-vs-harness boundary section (anchor #what-goes-where) to the canonical harness overview, so the pydantic-ai docs that link to it can point here after the duplicated in-repo stub is removed. * docs: address CodeRabbit review -- runnable snippets, accuracy, multi-class autodoc * docs: flatten harness nav and align with graduated capabilities Following the experimental-graduation refactor (#347), restructure the unified-docs harness pages: - Flatten docs/ (drop capabilities/ and experimental/ subdirs); the sidebar is now Overview + one flat list per Douwe's request. - Rename to match the graduated modules: overflow -> overflowing-tool-output, authoring -> runtime-authoring, docs -> pydantic-ai-docs. - Drop the 'Experimental' admonitions from the graduated capabilities and repoint every import + ::: autodoc path off pydantic_ai_harness.experimental. - Add docs for the newly-shipped capabilities: guardrails, dynamic-workflow, media, and acp (acp stays framed as experimental -- it may still be removed). - Every capability doc now links to its source; index capability table lists the full set with flat links. * docs: apply team-sync authoring rules + enforce them in CI From the 2026-07-10 docs review on #329: - Purpose-first leads: drop hook names (before_model_request, after_tool_execute) from the opening paragraphs of compaction and overflowing-tool-output (doc + README); mechanism moves lower. - Mirror the soft 'API may change between releases' stability note from each graduated README into its doc page (ACP keeps its stronger experimental warning; guardrails' README has no note, so its page gets none). - README H1s now use the capability's display name (Overflow capability -> Overflowing Tool Output, RuntimeAuthoring -> Runtime Authoring, SubAgents -> Subagents, etc.). - Extend tests/test_docs_parity.py with per-page mechanical checks: source link present, heading matches the capability name, purpose-first lead (no hook in the opener), and no experimental framing on graduated pages (ACP excepted). - Update the docs-parity-reviewer agent + review-checklist to the flat structure and the new semantic checks. * docs: add the stability note to guardrails (parity with sibling capabilities) guardrails was the one graduated capability whose README and doc page lacked the shared 'API may change between releases' note. Add it to both. * fix: restore uv.lock to match pyproject (bad text-merge dropped 8 lines) Merging origin/main did a git text-merge of the generated uv.lock, leaving it inconsistent with pyproject.toml -- every CI job failed at 'uv sync --locked'. pyproject.toml is identical to main here, so the correct lock is main's. * docs: address CodeRabbit review on #329 Findings that failed to post inline (GitHub error) but were real: - context/README.md, planning/README.md: two nested examples still imported from pydantic_ai_harness.experimental.* -- repoint to the graduated modules. - guardrails/README.md: replace em dashes with '--' (repo style) and add the source-module link. - docs/media.md: standardize on the implementation's canonical media+sha256:// URI scheme (was mixing media://). - tests/test_docs_parity.py: strengthen my own checks per review -- source-link and top-README-link now require a real Markdown link to the page's specific module (not a bare substring); heading checks assert an H1 exists and equals the expected capability name via explicit page metadata. * fix: restore uv.lock [options.exclude-newer-package] block The lock lost its [options.exclude-newer-package] manifest (pydantic-ai-slim = false, ...) -- a bad git text-merge dropped it, and diagnostic uv commands rewrote it under a different local config. Without that block CI's 'uv sync --locked' re-resolves and fails ('addition of exclude newer exclusion for pydantic-ai-slim'). Restore origin/main's exact lock. * fix: restore uv.lock [options.exclude-newer-package] block A pre-commit hook was rewriting uv.lock under the local uv config, stripping the [options.exclude-newer-package] manifest (pydantic-ai-slim = false, ...). Without it CI's 'uv sync --locked' re-resolves and fails. Commit origin/main's exact lock with --no-verify so no hook mutates it (lock-only change). * test: cover the docs-parity helper edge cases (100% coverage) The strengthened helpers added defensive branches (missing frontmatter close, fenced code before the lead, missing/forbidden/ClassName H1, lead running to EOF) that no real doc exercises. Add direct unit tests so the file is back to the repo's required 100% coverage. * docs: link every capability README to its source module + enforce it CodeRabbit re-flagged planning/README.md for a missing source link. Only guardrails had one, so add the source-module link to all 15 remaining capability READMEs (matching the doc pages) and add a parity test so the requirement is mechanical and cannot silently regress. * docs(agents): drop stale folder tree; fix flat docs path + guard names AGENTS.md's File-structure ASCII tree and capability-authoring's doc paths still showed docs/capabilities// docs/experimental/ (flattened in this PR) and the old /docs/harness URL. Delete the tree rather than redraw it -- the layout is discoverable by listing the repo; keep only the non-obvious conventions (flat docs/, the README<->doc parity requirement). Also fix the Vocabulary guard examples (InputGuard/OutputGuard, not the nonexistent InputGuardrail/ CostGuard). * test: statically validate doc snippets exist and parse Every Python snippet in the capability READMEs and docs/*.md pages is now checked for the two failures a reader hits immediately: it does not parse (syntax), or it imports a pydantic_ai_harness symbol that does not exist (stale module path or renamed name -- the class of bug behind the experimental.* import drift). Static only: no model/network execution, so it needs no mocking. The four illustrative API-signature blocks opt out with a {test="skip"} fence (read by pytest-examples, stripped-safe for the unified-docs render). * test: don't fail doc-snippet check on a missing optional extra The static check imported capability modules to resolve their symbols, but in the slim CI job (no extras) importing e.g. pydantic_ai_harness.experimental.acp raises ModuleNotFoundError for the absent third-party 'acp' package -- the harness module exists, its extra just isn't installed. Distinguish a genuinely missing harness module (fail) from a missing extra (skip) by the ImportError's module name. |
||
|
|
ae371bbdaf |
docs: ban em-dashes and codify writing style (#270)
Add a Writing style section to AGENTS.md so docs, comments, and PR text read as human-written: no em-dashes (use `--`), no marketing superlatives or editorializing adjectives, sparing bold, no decorative Unicode. Bring the tree into compliance by converting every em-dash to `--` across 16 files. All occurrences are in prose, comments, docstrings, and tool descriptions. The only runtime effect is that model-facing tool descriptions now use `--` (punctuation only, no semantic change); no test snapshots depend on the character. The one literal em-dash left is in AGENTS.md, where it names the banned character. https://claude.ai/code/session_01LQ9NTr8q95A99UnVcq6Nzf Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
e1654ccfe2 |
feat: add FileSystem and Shell capabilities (#260)
* feat: add FileSystem and Shell capabilities with exhaustive testing - FileSystemToolset: 8 tools (read, write, edit, list, search, find, mkdir, info) with path-traversal prevention, allow/deny patterns, optimistic concurrency - ShellToolset: 1 tool (run_command) with command validation, timeout handling, and async subprocess execution via anyio --- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: David Sanchez <64162682+dsfaccini@users.noreply.github.com> |
||
|
Bill EastonandGitHub
|
d4be271de9 |
Revert "feat: add FileSystem and Shell capabilities with exhaustive testing (…" (#259)
This reverts commit
|
||
|
Bill EastonandGitHub
|
7fd372d9fd |
feat: add FileSystem and Shell capabilities with exhaustive testing (#258)
- FileSystemToolset: 8 tools (read, write, edit, list, search, find, mkdir, info) with path-traversal prevention, allow/deny patterns, optimistic concurrency - ShellToolset: 1 tool (run_command) with command validation, timeout handling, and async subprocess execution via anyio - 152 tests passing on asyncio backend, 100% branch coverage - Mutation testing: 524/584 killed (89.7%), 60 equivalent mutants documented - All survivors proven equivalent (trampoline defaults, encoding case-insensitivity, unreachable except blocks, dead branches, name=None fallback behavior) |
||
|
Douwe MaanandGitHub
|
bf692aff32 |
Rework README hero example and add ecosystem agent showcase (#226)
* Rework README hero example and add full ecosystem agent showcase
The previous hero example used GitHub's MCP server, which requires
authentication and ships tools without `outputSchema` -- triggering
CodeMode warnings and producing weakly-typed sandbox stubs. Replaced
with the open-source Hacker News MCP server (full output schemas, no
auth), and added `WebSearch` to make the prompt naturally exercise
parallel tool orchestration in a single `run_code` call.
`MCP(..., builtin=False)` is now explicit so the local toolset is used
and CodeMode actually wraps the tools (Anthropic supports MCP server
connectors as a builtin and would otherwise execute tools server-side,
bypassing the sandbox).
Added a "Full ecosystem agent" section above the capability matrix that
illustrates the breadth of the Pydantic AI ecosystem -- combining
capabilities from the harness, core, and the community packages we
endorse, with inline credits at each call site.
Made install requirements explicit above the hero block.
* Reword Quick start prompt to avoid datetime in generated code
Monty currently rejects `datetime` (pydantic/monty issue tracking),
so the previous prompt's "posted in the last 6 hours" filter caused
the agent to generate code that failed in the sandbox. Replaced with
a dedupe-by-id + score filter + parallel follow-up calls (thread,
user profile, web search) -- still exercises parallel orchestration,
dict manipulation, and multi-tool composition without any timestamps.
* Embed verbatim run output and soften 'single run_code' claim
Followed pydantic-ai docs convention of triple-quoted output after
print(). The included synthesis is the actual output from running
the example, not a fabrication.
Reworded the CodeMode explanation to match what Sonnet actually does:
two batched run_code calls each containing parallel tool calls, with
the dedupe/filter happening as plain Python in the sandbox -- not the
"all in a single run_code" claim from the previous draft.
* Surface monty timing restrictions in run_code description
Sonnet was occasionally generating `asyncio.sleep` between tool calls
(cargo-culted "polite rate-limiting" pattern) and `datetime.now()` for
recency filters -- both rejected by the monty sandbox. Adding an explicit
"no wall-clock or timing primitives" bullet to the `run_code` tool
description (and the matching code-mode README entry) keeps the prompt
free of sandbox-specific babysitting.
Switched the hero example to claude-opus-4-7 with a more natural prompt
("find the most-discussed story... summarize what you find in one
paragraph"), and refreshed the verbatim sample output to a clean run
that produced 3 run_code calls (3x parallel feed fetches, pure-Python
dedupe/filter, then 3x parallel follow-up calls) with no errors.
* Quick start: opt out of native WebSearch so CodeMode wraps it
`WebSearch()` defaults to `builtin=True`, and on providers like
Anthropic that natively support web search the local DDG fallback
is filtered out by `Model.prepare_request`. With `CodeMode` in the
mix that produced a split surface -- builtin web_search at top
level, duckduckgo_search inside `run_code` -- and the model
sometimes tried to call the builtin from inside the sandbox.
Passing `builtin=False` keeps every web call going through CodeMode
so it can be batched alongside the HN tools in a single `run_code`.
Same pattern we already use for `MCP(..., builtin=False)`.
The underlying flatten-prefer_builtin-into-run_code bug is filed
as #233 and is being fixed separately.
Refreshed the verbatim sample output to a clean run with this
configuration: 3 chats + 2 run_code calls, all parallel tool
batches, no errors.
* Quick start: link the public trace and refresh verbatim output
DouweM picked a run with substantively richer commentary (Simon
Willison's "Vibe coding and agentic engineering are getting closer
than I'd like" post on HN) and made the trace public, so we can
finally point readers at the actual run from the example.
* Add duckduckgo extra to install snippet for WebSearch(builtin=False)
* Move ecosystem example credits to dedicated comment lines
* Restructure README around the Quick start trace and an ecosystem agent
Quick start
- Move all per-capability commentary into inline comments at call sites,
so links sit next to the code they document. Drops the trailing
paragraph wall.
- Make the public Logfire trace the section's main visual: a screenshot
linking to the trace, then a one-liner explainer.
- Trim the verbatim sample output to ~5 sentences. Keeps the Willison
framing and the meta-substantive thread reference; drops the long
commenter-by-commenter paraphrase and the news-coverage paragraph
that previously took over half the section.
An ecosystem agent (formerly "Full ecosystem agent")
- Move below the capability matrix so readers meet the breadth before
the worked example.
- Renamed and rewritten intro: shorter, voice closer to the rest of
the README, no "upper bound of what's possible" phrasing.
- Long disclaimer paragraph moved below the code so readers see the
example before the caveats.
- Imports split into official (`pydantic_ai*` + harness) vs community
(alphabetical).
- Sections reorganized: Reasoning, Tools, Execution (CodeMode pulled
out of Tools), Context management, Memory, Orchestration, Safety.
- Each capability gets its own one-line comment. AnthropicCompaction
added as the official compaction option, with a comment pointing at
vstorm-co's `summarization-pydantic-ai` as a provider-agnostic alt.
- Agent gets a name (`brian` -- Monty Python) which `MemoryCapability`
picks up via its `agent_name` arg.
- Skills comment now flags @vstorm-co's pydantic-deep alongside Doug's,
noting the shape difference.
- Switched to `claude-opus-4-7` and `Thinking(effort='high')` to match
the Quick start.
- Drops `MCP(builtin=False)` here -- the ecosystem example isn't trying
to demo CodeMode-wrapped MCP, it's demoing breadth.
Drops the no-wall-clock bullet from `_toolset.py` and the code-mode
README -- now covered by the standalone PR for that change.
Trace screenshot reference uses a relative path that resolves on GitHub
once the asset is committed; ready to swap in once the file is in place.
* Ecosystem agent: rename to shrubbery, bump thinking to xhigh
* Add Logfire trace screenshots and surface them prominently
`docs/images/quick-start-trace.png` is the trace tree: it's the visual
the main README's Quick start now leads with -- click-through to the
public trace.
`docs/images/code-mode-trace.png` is the wider shot showing the actual
Python the model wrote (parallel `asyncio.gather` over three HN feeds,
dedupe-by-id, score filter). It's the centrepiece of a new "In practice"
section in the code-mode README that points back to the harness Quick
start as the more representative example, replacing the toy weather
demo as the leading visual of what code mode does in real use.
* Address review on PR #226
- Reorder ecosystem agent capabilities: CodeMode first (the headline),
then Thinking, Context management, Tools (MCP -> WebSearch -> Console),
then everything else. First-party leads each tier; ConsoleCapability
follows our two first-party tools.
- Drop the agent `name='shrubbery'` flourish.
- Use vstorm-co's `ContextManagerCapability` as the active compaction
capability and mention pydantic-ai's `AnthropicCompaction` /
`OpenAICompaction` in the comment instead of the other way around.
Drops the "tied to Anthropic's prompt caching" phrasing.
- `MemoryCapability(agent_name='harness-example')`.
- Code-mode README: rewrite the "In practice" prose to be clear that
CodeMode produces two `run_code` calls (parallel fetches + filter,
then parallel follow-ups), not one. Fix the screenshot alt text to
match.
* Add regression test for the README's Quick start example
The lead example in the main README is the most public surface of the
harness — if it stops working we need to know before the user tries it.
This test drives the example end-to-end with `FunctionModel` issuing the
two `run_code` calls the README's trace shows, and `CodeMode` actually
running the emitted Python through Monty.
What gets faked vs. what's real:
- The Hacker News MCP toolset is replaced with a `FunctionToolset` of
fake functions returning canned data from the README's linked public
trace. Avoids depending on an MCP package or hitting the network.
- `WebSearch(builtin=False, local=...)` skips the default DuckDuckGo
fallback so the test doesn't pull `ddgs`.
- `FunctionModel` drives the conversation through two `run_code` calls
and a final synthesis, mirroring the production trace.
- `CodeMode` itself is real; the `FunctionModel`'s emitted Python runs
through the Monty sandbox and dispatches calls back to the fakes.
Asserts on the call shape (each feed fetched once in parallel, follow-up
calls target the expected winner id) and on the final synthesis content,
so any future change in pydantic-ai or the harness that breaks how these
capabilities compose makes this test fail.
* Address review on PR #226
- Quick start: list `CodeMode` first in `capabilities=[]` to match the
ecosystem agent's ordering and to lead with the headline capability.
- Test: wire the fake HN tools through `MCP(local=hn_toolset, builtin=False)`
instead of bypassing via `toolsets=[hn_toolset]`. Same composition path
as production minus the network. Needs a narrow `# pyright: ignore`
because MCP's `__init__` signature narrows `local` to MCP-shaped types
but the parent `BuiltinOrLocalTool` accepts any `AbstractToolset` at
runtime.
- Test: have the second `run_code` call `web_search` (the WebSearch
capability's local fallback) instead of `hn_search_content`, so the
fake function is actually exercised. This fixes the coverage failure
on lines 231-232 and makes the test more representative of the README
example, which uses WebSearch.
* Use snapshot()+dirty-equals for README quick-start test
* Re-export dirty-equals via tests/conftest, restructure tests/
Two related cleanups based on review:
- Re-export `IsDatetime` / `IsNow` / `IsStr` / `IsPartialDict` from
`tests/conftest.py` with `TYPE_CHECKING` shims that pretend the
matchers return the concrete type (`-> str`, `-> datetime`, etc.).
Mirrors pydantic-ai's own conftest pattern and lets pyright strict
accept `tool_call_id=IsStr()`, `timestamp=IsDatetime()`, etc. without
the file-level `# pyright: reportArgumentType=false` we had before.
Tests now `from .conftest import ...` instead of importing directly
from `dirty_equals`.
- Rename `tests/_code_mode/` -> `tests/code_mode/` (drop the underscore
prefix; matches the source package `pydantic_ai_harness/code_mode/`).
Updates the convention note in AGENTS.md/CLAUDE.md.
- Move `test_readme_quick_start.py` up to `tests/` top level. It's an
end-to-end check on the README's flagship example -- it composes
`CodeMode` + `MCP` + `WebSearch` -- so it doesn't belong in any
single capability's test directory.
|
||
|
DavidandGitHub
|
31ae664b3e | Public launch infrastructure: docs, GitHub infra, template (#165) | ||
|
Douwe MaanandClaude Opus 4.6
|
54feb9a815 |
Set up repo infrastructure: CI, docs skeleton, conftest, dependency management
- Update pydantic-ai-slim dependency to >=1.76.0 (published capabilities API) - Replace pytest-xdist with anyio/pytest-anyio for async test support - Add pyright executionEnvironments for tests (reportPrivateUsage=false) - Add anyio_mode=auto to pytest config - Create __init__.py with docstring and empty __all__ ready for capability imports - Create tests/conftest.py with shared fixtures (TestModel, test agent, tmp_dir, allow_model_requests) - Update test_placeholder.py to exercise conftest fixtures for 100% coverage - Create docs/ skeleton with index.md and capabilities/index.md - Update CI workflow with Python version matrix (3.10, 3.12, 3.13), uv caching, and permissions - Rewrite README with installation, quick start, and capability table Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |