mirror of
https://github.com/pydantic/pydantic-ai-harness.git
synced 2026-07-20 10:25:35 +00:00
main
93
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Aditya VardhanandGitHub
|
4ad83f8861 |
Clarify CodeMode final-expression returns (#370)
* Prevent empty CodeMode results through clearer guidance * Make CodeMode return guidance unambiguous |
||
|
|
f2389a893c |
Add .macroscope/ignore.md to skip mechanical files in review (#387)
* Add .macroscope/ignore.md to skip mechanical files in code review Macroscope's managed review bills per KB of diff reviewed. This drops the mechanical, non-hand-reviewed files from the reviewed/billed diff: uv.lock (large on dependency bumps), recorded VCR cassettes, and doc images. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Preserve Macroscope default ignores; keep test code reviewed A custom .macroscope/ignore.md replaces Macroscope's built-in defaults rather than extending them. Copy their default base patterns verbatim and add this repo's VCR cassettes on top. The default test-file patterns are omitted so Macroscope still reviews test code; cassettes stay ignored via the explicit rule. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Bill Easton <bill@pydantic.dev> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
e4839c654b |
Keep already-empty ModelRequests in LimitWarner._strip_old_warnings (#311)
LimitWarner rebuilds history each request, dropping any ModelRequest left with no parts after warning markers are stripped. An already-empty ModelRequest -- which pydantic-ai's empty-response retry appends when a model returns an empty response against a non-optional output type -- has no parts to begin with, so it was dropped too. That left history ending on a ModelResponse, and the next request failed the _agent_graph precondition with `UserError: Processed history must end with a ModelRequest`, killing the run. Keep a request that was already empty; only drop one that became empty because every part was a warning marker. Regression test covers the empty-response-retry tail. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>v0.7.1 |
||
|
Aditya VardhanandGitHub
|
67f8d870a5 | Prevent search result format from becoming a return type (#377) | ||
|
Aditya VardhanandGitHub
|
73513a60bc |
Remove third-party MCP endpoint from pydantic-ai-harness skill (#368)
* Use local tools in harness skill quick start * Align CodeMode skill security guidance |
||
|
|
818808d6fe |
feat: add Memory capability (#354)
* feat: add Memory capability * feat(memory): add SQLite and PostgreSQL stores * fix(memory): address CI pyright and CodeRabbit review * style(memory): ASCII dashes, labeled README fence (review follow-up) * fix(memory): SQLite in-memory rejection and case-sensitive listing * fix(memory): use qmark placeholders in SQLite prefix listing * Make memory safe for durable shared use * Harden memory trust and recovery boundaries --------- Co-authored-by: David SF <64162682+dsfaccini@users.noreply.github.com>v0.7.0 |
||
|
David SFandGitHub
|
39b4c569cb | Stop attributing every AICA comment to Claude (#362) | ||
|
Aditya VardhanandGitHub
|
4ee2b3291f |
ci: drop the pydantic-ai v2 beta early-warning job (#359)
The v2-beta job was a non-blocking visibility signal, not a required check. Removing it since it is no longer needed. |
||
|
David SFandGitHub
|
3ba9e2f9a5 |
docs: capability pages for the unified docs site + README/doc parity gate (#329)
* docs: publish capability docs to the unified site + add README/doc parity gate Every capability shipped only a README (kept for GitHub/PyPI). This adds a parallel, cleaned-up page per capability under docs/ for the new unified docs site (pydantic.dev/docs/harness), migrated from each README: snippets verified runnable against source, autodoc API blocks, root-relative Pydantic AI links, and an experimental-status admonition on the experimental set. To keep README and doc in sync going forward, adds a docs-parity-reviewer agent and a parity gate in the review checklist (run as the last step before merge), plus the docs/ layout and the README<->doc requirement in AGENTS.md and the capability-authoring guide. * docs: fix README<->doc<->source inconsistencies across capabilities A parity audit against source found drift, mostly in the capability READMEs (staler than the migrated docs). All fixes verified against source: - Correctness: the "approval/deferred tools are excluded from the sandbox" claim (code_mode README + doc) was false -- those tools are sandboxed like any other; corrected in both. The stale Shell persist_cwd sentinel description is replaced with the actual out-of-band temp-file capture. filesystem protected default `.git/` -> `.git/*` (the bare form never matched). - Runnable snippets: added the missing imports/wiring so README snippets no longer raise NameError (subagents, context, planning, overflow, authoring, filesystem, code_mode). - Parity: documented previously-undocumented params/behaviors (compaction strategy options, overflow strip_ansi/Passthrough, extra autodoc classes for context and subagents), fixed a stale version pin (>=1.95.1 -> >=2.1.0), and added the missing Managed Prompt row to the root README capability matrix. - Style: normalized decorative Unicode to ASCII across all READMEs and dropped a hype phrase, matching AGENTS.md writing style and the docs. * docs: add nav.json to drive the unified-docs harness sidebar The unified docs mount the harness docs under /docs/ai/harness (fed live from this repo via the pydantic-ai 'Pydantic AI Harness' section). This nav.json defines the sub-nav (Overview + Capabilities + Experimental) and the set of doc files the site includes. * docs: migrate "What goes where?" explainer into harness overview Adds the core-vs-harness boundary section (anchor #what-goes-where) to the canonical harness overview, so the pydantic-ai docs that link to it can point here after the duplicated in-repo stub is removed. * docs: address CodeRabbit review -- runnable snippets, accuracy, multi-class autodoc * docs: flatten harness nav and align with graduated capabilities Following the experimental-graduation refactor (#347), restructure the unified-docs harness pages: - Flatten docs/ (drop capabilities/ and experimental/ subdirs); the sidebar is now Overview + one flat list per Douwe's request. - Rename to match the graduated modules: overflow -> overflowing-tool-output, authoring -> runtime-authoring, docs -> pydantic-ai-docs. - Drop the 'Experimental' admonitions from the graduated capabilities and repoint every import + ::: autodoc path off pydantic_ai_harness.experimental. - Add docs for the newly-shipped capabilities: guardrails, dynamic-workflow, media, and acp (acp stays framed as experimental -- it may still be removed). - Every capability doc now links to its source; index capability table lists the full set with flat links. * docs: apply team-sync authoring rules + enforce them in CI From the 2026-07-10 docs review on #329: - Purpose-first leads: drop hook names (before_model_request, after_tool_execute) from the opening paragraphs of compaction and overflowing-tool-output (doc + README); mechanism moves lower. - Mirror the soft 'API may change between releases' stability note from each graduated README into its doc page (ACP keeps its stronger experimental warning; guardrails' README has no note, so its page gets none). - README H1s now use the capability's display name (Overflow capability -> Overflowing Tool Output, RuntimeAuthoring -> Runtime Authoring, SubAgents -> Subagents, etc.). - Extend tests/test_docs_parity.py with per-page mechanical checks: source link present, heading matches the capability name, purpose-first lead (no hook in the opener), and no experimental framing on graduated pages (ACP excepted). - Update the docs-parity-reviewer agent + review-checklist to the flat structure and the new semantic checks. * docs: add the stability note to guardrails (parity with sibling capabilities) guardrails was the one graduated capability whose README and doc page lacked the shared 'API may change between releases' note. Add it to both. * fix: restore uv.lock to match pyproject (bad text-merge dropped 8 lines) Merging origin/main did a git text-merge of the generated uv.lock, leaving it inconsistent with pyproject.toml -- every CI job failed at 'uv sync --locked'. pyproject.toml is identical to main here, so the correct lock is main's. * docs: address CodeRabbit review on #329 Findings that failed to post inline (GitHub error) but were real: - context/README.md, planning/README.md: two nested examples still imported from pydantic_ai_harness.experimental.* -- repoint to the graduated modules. - guardrails/README.md: replace em dashes with '--' (repo style) and add the source-module link. - docs/media.md: standardize on the implementation's canonical media+sha256:// URI scheme (was mixing media://). - tests/test_docs_parity.py: strengthen my own checks per review -- source-link and top-README-link now require a real Markdown link to the page's specific module (not a bare substring); heading checks assert an H1 exists and equals the expected capability name via explicit page metadata. * fix: restore uv.lock [options.exclude-newer-package] block The lock lost its [options.exclude-newer-package] manifest (pydantic-ai-slim = false, ...) -- a bad git text-merge dropped it, and diagnostic uv commands rewrote it under a different local config. Without that block CI's 'uv sync --locked' re-resolves and fails ('addition of exclude newer exclusion for pydantic-ai-slim'). Restore origin/main's exact lock. * fix: restore uv.lock [options.exclude-newer-package] block A pre-commit hook was rewriting uv.lock under the local uv config, stripping the [options.exclude-newer-package] manifest (pydantic-ai-slim = false, ...). Without it CI's 'uv sync --locked' re-resolves and fails. Commit origin/main's exact lock with --no-verify so no hook mutates it (lock-only change). * test: cover the docs-parity helper edge cases (100% coverage) The strengthened helpers added defensive branches (missing frontmatter close, fenced code before the lead, missing/forbidden/ClassName H1, lead running to EOF) that no real doc exercises. Add direct unit tests so the file is back to the repo's required 100% coverage. * docs: link every capability README to its source module + enforce it CodeRabbit re-flagged planning/README.md for a missing source link. Only guardrails had one, so add the source-module link to all 15 remaining capability READMEs (matching the doc pages) and add a parity test so the requirement is mechanical and cannot silently regress. * docs(agents): drop stale folder tree; fix flat docs path + guard names AGENTS.md's File-structure ASCII tree and capability-authoring's doc paths still showed docs/capabilities// docs/experimental/ (flattened in this PR) and the old /docs/harness URL. Delete the tree rather than redraw it -- the layout is discoverable by listing the repo; keep only the non-obvious conventions (flat docs/, the README<->doc parity requirement). Also fix the Vocabulary guard examples (InputGuard/OutputGuard, not the nonexistent InputGuardrail/ CostGuard). * test: statically validate doc snippets exist and parse Every Python snippet in the capability READMEs and docs/*.md pages is now checked for the two failures a reader hits immediately: it does not parse (syntax), or it imports a pydantic_ai_harness symbol that does not exist (stale module path or renamed name -- the class of bug behind the experimental.* import drift). Static only: no model/network execution, so it needs no mocking. The four illustrative API-signature blocks opt out with a {test="skip"} fence (read by pytest-examples, stripped-safe for the unified-docs render). * test: don't fail doc-snippet check on a missing optional extra The static check imported capability modules to resolve their symbols, but in the slim CI job (no extras) importing e.g. pydantic_ai_harness.experimental.acp raises ModuleNotFoundError for the absent third-party 'acp' package -- the harness module exists, its extra just isn't installed. Distinguish a genuinely missing harness module (fail) from a missing extra (skip) by the ImportError's module name. |
||
|
David SFandGitHub
|
310ffadf09 |
refactor: graduate capabilities out of experimental (ACP excepted) (#347)
* refactor: graduate capabilities out of `experimental` (ACP excepted) The `experimental` label suppressed the try->fail->report->improve loop we want from real usage, so drop it from the harness. Ten capabilities move to top-level submodules; ACP stays experimental (it may still be reshaped or moved to core, and can't be generic across agents). - Old `pydantic_ai_harness.experimental.<name>` paths keep working as DeprecationWarning shims (via `warn_moved`), so existing imports don't break. - Renames to match capability names: authoring -> runtime_authoring, overflow -> overflowing_tool_output. - Capabilities stay in individual submodules with no top-level re-export, so importing the root package never pulls in a capability's optional deps. - Docs drop the experimental framing and the "may be removed in any release" language in favor of "API subject to change". * docs: tell capability authors to ask the user when unsure about a name Per PR review: a name is a public commitment once shipped, so ask rather than guess. |
||
|
David SFandGitHub
|
d13cdae9cb |
Fix docs-parity gate and OutputGuard retry test after guardrails merge (#348)
* docs: link shipped guardrails package from README (fix docs-parity CI) * test: use FunctionModel for OutputGuard retry test (survives pyai #6403 text-recovery removal) |
||
|
|
5ab8dd3d81 |
feat: input and output guardrails (block, redact, retry) (#249)
* feat: add input and output guardrails * chore: use single backticks * fix: run InputGuard only on the first model request * fix: replace list[Any] with Sequence[ModelMessage] * fix: pass raw output to OutputGuard, not str(result.output) * refactor: organize tests into TestCapabilityName classes * fix: drain cancelled tasks in InputGuard parallel finally * fix: re-raise task exceptions via await instead of .exception() * refactor: consolidate parallel cancel/drain into single finally * feat: support callable block_message; collapse InputGuard hooks block_message now accepts a callable so the refusal text can reflect the prompt/output that tripped the guard, rather than being frozen at construction time. InputGuard's sequential path moves from before_model_request into wrap_model_request, so a single hook covers both sequential and parallel modes instead of two hooks each branching on `parallel`. Tests move to tests/guardrails/ to match the tests/<capability>/ layout. * refactor: guards return bool | GuardResult; drop block_message A guard now returns either a bare bool or a GuardResult carrying a refusal message, replacing the separate block_message constructor field. The message is produced when the guard decides, so it can reflect the guard's own reasoning rather than a string frozen at construction time. Guards (and the GuardResult path) may optionally take a RunContext as a first parameter, detected from the signature like pydantic-ai's output validators, so deps- and history-aware guards are possible without closing over globals. Prompt/output-only guards are unchanged. * feat: guard outcomes — allow/block/replace/retry with OTel spans A guard now reports one of four outcomes via GuardResult classmethods (bool shorthand still works): allow, block, replace, retry. - replace lets a guard redact rather than refuse — InputGuard rewrites the prompt sent to the model, OutputGuard substitutes the output. - retry lets OutputGuard send a bad output back to the model; OutputGuard moves from after_run to after_output_process so it can raise ModelRetry and return a modified output. - replace and block emit spans on the run tracer so a redaction or refusal is visible in Logfire; redacted content is included only when RunContext.trace_include_content is set. InputGuard replace requires sequential mode and retry is rejected as a usage error, since neither is meaningful for input. * test: cover guardrail test helpers for the 100% gate The coverage gate measures test files too; the _prompt_text helper had unreached branches. Drop it and assert on message parts inline. * docs: scope streaming behavior; add streaming tests A pydantic-ai-correctness review surfaced two streaming points. Verified both empirically: - InputGuard(parallel=True) works under run_stream() — no deadlock. - OutputGuard GuardResult.retry() is unsupported under run_stream(): pydantic-ai does not retry output while streaming, so a retry verdict surfaces as UnexpectedModelBehavior. Document the retry limitation and that OutputGuard screens only the final output (partial chunks reach the caller first while streaming). Note that input redaction also rewrites persisted history and targets text prompts. Add streaming tests for both guards to lock the behavior in. * test: make the parallel-streaming test a real regression guard The test proving InputGuard(parallel=True) does not deadlock under run_stream() had no timeout — a reintroduced deadlock would hang CI instead of failing. Wrap it in asyncio.wait_for and document the reviewed concern it guards against. * feat: guardrail capability ordering + GuardResult hardening Address @adtyavrdhn's review on #249. - `InputGuard.get_ordering()` → `position='innermost'` so any message-morphing capability runs first and the guard sees the final prompt the model will receive. - `OutputGuard.get_ordering()` → `position='outermost', wrapped_by= [Instrumentation]` so the guard's block/redact spans are always captured by an enclosing `Instrumentation` span regardless of user list order. - `GuardResult` is `frozen=True, kw_only=True` with a `__post_init__` that rejects field combinations the four-outcome contract does not allow (e.g. `replace` without a replacement). `block` with no message stays valid — the default kicks in at the use site. - `_run_guard` and `after_output_process` dispatch via `match action:` with `assert_never` exhaustiveness guards. - `_trace_block` gates the refusal `message` attribute behind `ctx.trace_include_content`, matching `_trace_redaction` — the message can quote sensitive content from the guarded value. - New tests: ordering declarations, `__post_init__` validation, frozen enforcement, outer-cancellation no-leak regression guard (which confirms `asyncio.shield` around the cleanup `gather` is not needed — the outer cancel is already consumed by `asyncio.wait`). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump pydantic-ai-slim to 2.4.0 in lock CI resolves pydantic packages to newest (uv.toml marks them exclude-newer=false); the merged lock was stale at 2.1.0 so uv sync --locked failed. Regenerate to 2.4.0. * test(code_mode): drop removed ToolSearchMatch.description field pydantic-ai 2.4.0 narrowed ToolSearchMatch to carry only `name`; the full ToolDefinition (with description) is surfaced separately. Update the test fixtures to the new shape. --------- Co-authored-by: Kacper Włodarczyk <kacperwlodarczyk@protonmail.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Aditya VardhanandGitHub
|
1229858a57 |
Announce harness releases on Twitter (#341)
* Announce harness releases on Twitter Mirror pydantic-ai's post-release tweet so a PyPI publish also posts to the shared @pydantic account, keeping release visibility consistent across the workspace. Version is read from the tag ref since the release job emits no version output. Requires the TWITTER_* secrets to be provisioned on this repo. * Scope send-tweet secrets to the release environment zizmor's secrets-outside-env audit fails the lint job because the tweet job reads the TWITTER_* secrets without a dedicated environment. Bind it to the release environment, matching the release job's convention for secret-using jobs (the harness has no zizmor ignore config like pydantic-ai does).v0.6.0 |
||
|
|
545a703227 |
feat: Adds Dynamic Workflows (#273)
* feat(dynamic-workflow): orchestrate sub-agents from a model-written script Turn-by-turn sub-agent delegation makes every composition step (chain, vote, loop) a separate model turn and pushes each intermediate result back through the orchestrator's context. DynamicWorkflow is Code Mode with sub-agents as the callables: the model writes one Monty-sandboxed script that composes the sub-agents with ordinary control flow, and only the final result returns to the model. Extracts the shared Monty execution loop into _monty_exec, reused by code_mode. The pydantic-ai bump the capability needs (defer_loading, ctx.enqueue) lands separately, per the repo policy on dependency changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(deps): bump pydantic-ai 1.95.1 -> 1.105.0 DynamicWorkflow needs defer_loading (>=1.97) and ctx.enqueue (1.105) for on-demand loading and cache-stable runtime sub-agent reveal. The lock exempts the pydantic-ai family from its exclude-newer cutoff to pull the newer release. Collateral: xfail a code_mode ToolSearch test that relied on the old discovered-tool tracking, and regenerate the managed_prompt snapshot for the new tool-definition fields. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(examples): tidy example docs and keep them fully linted Drop the blanket 'examples/**' pydocstyle ignore — it masked a single missing docstring. Add the one-line docstring to fork_and_resume's main() instead, so examples stay under the same docstring rules as library code. Also drop the redundant "Runs / needs a key" column from the examples README. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(dynamic-workflow): render catalog via core FunctionSignature, reject keyword names Addresses PR #273 review. Render the sub-agent catalog through pydantic_ai's FunctionSignature (the same renderer code_mode uses) instead of a hand-rolled f-string. This is the real dedup with code_mode and fixes two latent bugs: the signature now forces keyword-only `task` (`async def name(*, task: str)`) to match dispatch, which reads kwargs['task']; and a description containing a newline or quote now renders as a safe docstring block instead of breaking the f-string. Reject sub-agent names that are Python keywords. `'class'.isidentifier()` is True, so a keyword name passed validation but the model could never call it (`await class(...)` is a syntax error) — a silently-uncallable agent. Validate non-keyword identifiers at construction, via a shared in-file helper. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(dynamic-workflow): harden public API ahead of release Adversarial review of the dynamic-workflow surface surfaced contracts that would be painful to change once released; fix them while cheap: - Correct the "usage_limits bounds the tree" claim — the parent's limit is never forwarded to sub-agents. Add `sub_agent_usage_limits` so a per-sub-agent limit plus `max_agent_calls` give a real tree-wide token ceiling, and document what `forward_usage` actually does. - Own `WorkflowResourceLimits` instead of leaking pydantic_monty's `ResourceLimits` onto the public field; add an explicit `'unlimited'` sentinel and merge a partial dict onto the backstop (no silent drop). - Narrow `agents` to `list` so the append-to-reveal contract is enforced by the type rather than explained in prose. - Drop the `description` -> `agent.name` fallback that rendered a useless self-referential docstring to the model. - Teach the model-facing tool description what the README already knew: keyword-only `task`, no `return_exceptions`/abort-on-raise, dict (not attribute) access for structured output, and the print envelope. Also refresh the runnable examples (audit + migrate) and keep the README and tests in sync. lint/typecheck clean, 100% branch coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * removing time limit from monty * docs(examples): consolidate into one verified DynamicWorkflow showcase Replace the overlapping audit and migrate examples with a single example that exercises the full capability range — parallel fan-out, read/write confinement, an adversarial review, a feedback loop, typed fan-in, and a Logfire trace — so a reader sees what the capability is for without wading through two near-identical fan-out demos. Fixes issues an adversarial review surfaced: the retry loop now threads reviewer issues back to the migrator (so it can actually converge), the synthesizer's typed report survives to the caller via the orchestrator's output_type, and several claims (budget ceiling, confinement, run command portability) are corrected to match the code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(deps): raise pydantic-ai-slim floor to 1.101.0 for RunContext.enqueue The dynamic_workflow reveal path calls ctx.enqueue to announce a sub-agent added mid-run, but RunContext.enqueue did not exist until pydantic-ai 1.101.0 (verified: absent in 1.100.0, present in 1.101.0). The declared floor was still >=1.95.1, so the test-floor job (--resolution lowest-direct) and any install at the floor would resolve a RunContext without enqueue and raise AttributeError when revealing an agent instead of announcing it. Bump the runtime dep, the logfire extra, and the dev group to >=1.101.0, and update the README requirement line to match. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(dynamic-workflow): move capability under experimental DynamicWorkflow ships as experimental so its API can change without a deprecation period, like the compaction menu. Importing it now emits a HarnessExperimentalWarning, silenceable category-wide. _monty_exec stays at the package root: it is shared with code_mode, which is stable, so it must not move into experimental. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(dynamic-workflow): correct stdlib import guidance for the sandbox The tool description and README presented `asyncio, math, json, re, typing` as the complete set of importable modules. In Monty, `datetime`/`os`/`sys` also import, but they are inert in DynamicWorkflow (no os/path hooks are registered, so their calls resolve to NameError). Reword to a curated, non-exhaustive list so the model-facing text is accurate without steering toward the stubbed dead ends. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(dynamic-workflow): harden sandbox boundary and reveal/config validation Adversarial review of the run_workflow path surfaced several silent-failure and crash modes, all reachable from model-authored scripts or host config: - A model script awaiting the same sub-agent call twice in one asyncio.gather made the Monty VM raise pyo3_runtime.PanicException (a BaseException), which escaped both Monty error handlers and tore down the whole agent run. Catch it by name and convert to a retry; non-panic BaseExceptions still propagate. - Sub-agent calls silently dropped extra keyword args and accepted a non-string task (a dict/list was smeared into message parts), running on wrong input with no signal. Reject both at dispatch, before the budget is touched. - A runtime reveal whose name was invalid or collided with an existing agent was silently dropped, keeping the old agent while the host believed it had swapped one in. Distinguish idempotent re-reveal (object identity) from a real conflict, warn once on the latter, never swap silently. - max_agent_calls < 1 produced a dead-on-arrival tool with a misleading "conclude with results gathered" message; a typo'd resource_limits key was silently ignored, disabling the only guard against a pure-CPU loop. Both now raise UserError at construction. Also pins the headline guarantee with a concurrent fan-out test asserting max_agent_calls admits exactly the budget under asyncio.gather (previously only covered sequentially). _toolset.py stays at 100% line+branch coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: await cancelled dispatch tasks and survive bad mid-run reveals Cancelling a Monty execution previously fired task.cancel() and returned immediately, leaving in-flight tool calls and sub-agent runs unwinding in the background — they could mutate the shared usage accumulator after the tool call had already reported its result, and teardown errors were never retrieved. The executor now awaits the cancelled tasks so nothing outlives the call that spawned it. This changes code_mode's cancellation timing too (flagged in the PR description): teardown now waits for in-flight calls to unwind instead of abandoning them. Also: create the sequential dispatch coroutine after the pending-task barrier so cancellation at the barrier cannot leak a never-awaited coroutine, and make `for_run` index the agent list leniently — an invalid WorkflowAgent appended mid-run (which `_reveal_pending` deliberately tolerates) no longer hard-fails every subsequent run via `replace`'s re-validation. A new test pins that host-raised exceptions cannot be caught inside the sandbox: the budget-exhausted terminal result relies on it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(dynamic-workflow): document experimental status and actual return shapes The README never said the import warns or how to silence it, omitted the print()/None result shaping the tool actually performs, and the API block hid `description` and the WorkflowResourceLimits type. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(dynamic-workflow): cover deps forwarding, message isolation, and shared-counter limits An adversarial tests-vs-spec pass found three documented behaviors with no assertion behind them: parent deps reaching sub-agents, sub-agents never seeing the parent conversation, and sub_agent_usage_limits checked against the shared counter under forward_usage=True. Dropping any of them would have passed the suite. Also pins the documented backstop values and the served tool's max_retries, asserts the reveal announcement names the actual tool, and fixes two comments that described a duration backstop / a dataclasses.replace that don't exist. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(dynamic-workflow): record the core usage_limits TOCTOU race behind max_agent_calls The exact host-side counter exists because core's limit enforcement races under concurrent fan-out and RunContext can't forward the parent's limits; pin that context (and the upstream TODO) where the discipline lives so it isn't lost once the PR ships. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(examples): replace the generated example with a feature-led README Review feedback on #273: the example will be rewritten by hand, and the README should lead with the Anthropic dynamic-workflows post and the feature itself rather than narrating one example run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(dynamic-workflow): add a dedicated install extra Installing DynamicWorkflow's sandbox dependency via the code-mode extra conflated two capabilities; each now names its own extra (both resolve to pydantic-monty). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(code_mode): convert sandbox VM panics to retries A model-written script that awaits the same call twice in one asyncio.gather panics the Monty VM; pyo3 surfaces that as a BaseException that nothing caught, tearing down the whole agent run. DynamicWorkflow already guarded this; the guard now lives in _monty_exec and code_mode applies it too, dropping its REPL since the aborted VM's accumulated state can't be trusted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(dynamic-workflow): remove duplication that could drift Single-source the long-form field docs on the capability (the toolset docstrings now point there), derive the resource-limit key set from the TypedDict instead of a parallel frozenset, and freeze the rendered tool description itself rather than a catalog dict re-rendered every step -- making the cache-stability claim structural. Drops the for_run_step override that duplicated the base default, plus redundant tests and a stateless test fixture. No behavior or public API change; coverage stays at 100%. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: narrow the shared executor seam and drop dead type unions Name sanitization is a code_mode concern, so the mapping moves into its dispatch closure and MontyExecutor stops knowing about it. The empty pending_call_ids guard fell through to an identical resume call, so it goes. Concrete return types on get_toolset/for_run let tests drop four isinstance asserts that existed only to narrow types pyright can now see directly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: fix a comment stating a false invariant; unshadow a closure param _by_name is mutated during a run (reveals fold into it each step); the stable-registry invariant only holds while a script executes, which is what the executor relies on. The dispatch closure's name param shadowed call_tool's own, a future-edit hazard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(dynamic-workflow): restructure README for faster comprehension Lead with a minimal runnable snippet and state the context-saving idea once instead of repeating it across Why/What/blockquote. Move Installation above the full example. Turn the sub_agent_usage_limits paragraph into a forward_usage True-vs-False table so the N*T hard ceiling and the best-effort concurrency caveat are scannable. No technical claims changed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(dynamic-workflow): address CodeRabbit review - Fix broken relative link to Code Mode README: the capability lives two levels deep under experimental/, so `../code_mode/` resolved to a non-existent experimental/code_mode/. Use `../../code_mode/`. - Add missing `-> None` return types to all four test_warnings methods (the bot flagged one; the whole file was missing them). - Narrow `_workflow_result`'s unconstrained passthrough from `Any` to `object`. Left the other flagged `Any`s in place: the `call` override and `dispatch` must match `AbstractToolset.call_tool` / `_monty_exec.DispatchFn`, and `AbstractAgent[..., Any]` carries a heterogeneous sub-agent output type that `object` would reject on assignment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(dynamic-workflow): harden error surfaces, render output schemas, add reveal() API Findings from an end-to-end adversarial review (PR #273 branch), fixes authored with codex exec: - Serialize sub-agent output inside the dispatch error wrapper: a non-serializable output leaked host class paths into the retry text and failed identically on every retry. - Render each sub-agent's output schema in the catalog and reveal announcements (TypedDict-style, with cross-catalog type-name conflict handling). The model previously saw `-> Any` and had to be told the fields in prose. - Add DynamicWorkflow.reveal() as the supported runtime reveal API; list append remains the underlying mechanism. An aliased-list-only contract cannot survive the planned durable resume. - Return terminal {'error': ...} results for permanent conditions (nesting refusal, budget exhaustion) instead of ModelRetry, which burned retries into UnexpectedModelBehavior. The budget result now carries last_error and completed. - Salvage completed sub-agent results into retry messages and the budget-terminal dict so a retry can reuse them instead of re-spending the budget; the sandbox-panic path does the same. - Pass through model-safe exception messages (UsageLimitExceeded) so the model can react; keep the opaque form for everything else. - State the call budget and sub-agent statelessness in the tool description; warn instead of failing when a reveal has no pending message queue to announce into. - Docs: correct the N*T "hard ceiling" claim (core checks token limits after a response arrives), fix the examples README script (it subscripted str outputs), pin model-facing text with snapshots. The old "budget error cannot mask another failure" comment was disproven under asyncio.gather; the true invariant is documented. Refs: https://www.anthropic.com/engineering/code-execution-with-mcp, https://code.claude.com/docs/en/workflows Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(dynamic-workflow): align tests with pydantic-ai v2 deps typing Merging main raised the pydantic-ai-slim floor to >=2.1, whose Agent `deps_type` default became `object` (was `NoneType`). Constructing an `Agent(..., capabilities=[Cap[None](...)])` then fails overload matching: the capabilities pin `AgentDepsT=None` while the default supplies `object`. Switch the deps typevar to `object` throughout, matching the convention the code_mode and subagents tests already use. Also make test_max_agent_calls_exact_under_concurrent_fan_out order- independent. The completed list was asserted in exact order, but which three of eight concurrent calls win the budget and the order they finish are both scheduler-dependent (3.14 admits them in a different order than 3.13). Assert the invariant that matters: exactly the budget ran, each once, well-formed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: stop tracking .agents/scheduled_tasks.lock This is a machine-local runtime lock for Claude Code's scheduled-task runner (pid/session mutex), not project content. It was committed by accident. Remove it from the tree and gitignore it alongside the other .agents machine-local files. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(dynamic-workflow): accept raw agents, make reveal() the strict mutation API Pydantic AI 2.x agents carry their own name and description, so wrapping every entry in WorkflowAgent was boilerplate: agents=[reviewer, summarizer] now works, following the OpenAI-handoffs union pattern where a raw agent means "use the agent's own metadata" and the wrapper is a per-use-site override. WorkflowAgent gains the agent.description fallback and positional agent that the sibling SubAgent already had; the divergence had no rationale. Making reveal() the only post-construction channel (eager validation, UserError at the call site, internal catalog list) removes the reason the toolset needed lenient re-validation: the strict/lenient _rebuild duality, _reveal_warned, and the warn-and-skip paths are deleted, and an entry that reaches the toolset invalid now fails fast as a contract violation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(dynamic-workflow): rewrite README as a progressive tutorial The reference-style README front-loaded precise semantics before a reader had a mental model to hang them on. Reshape it into a learning arc -- smallest working example first, one concept per section, dense tables tucked into admonition boxes -- so the same facts land in the order a reader needs them. All technical claims are carried over unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(dynamic-workflow): use raw-agent shorthand in the examples README The migration example wrapped each sub-agent in WorkflowAgent solely to supply a catalog description, which read as pre-refactor mandatory wrapping and contradicted the prose above it ("documented by its description"). An agent's description is a property of the agent, not of one workflow, and the field has other consumers (agent spec, OTel gen_ai.agent.description), so its natural home is the Agent. Move the descriptions onto the agents and pass them raw; WorkflowAgent stays documented in the capability README's override section for the cases that need it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(dynamic-workflow): drop internal decisions log from the PR DYNAMIC_WORKFLOW_DECISIONS.md was an internal working-notes log (branch codenames, v1/maybe-v2 planning, open problems). It doesn't belong in the shipped repo and duplicates the capability README. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: fix pydantic-ai-slim version pin and tighten DynamicWorkflow docstrings Correct the required version in README.md to `pydantic-ai-slim>=2.1.0` (matches pyproject.toml; the previous `>=1.101.0` was wrong), and trim the verbose DynamicWorkflow capability/field docstrings to say the same thing more concisely. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(dynamic-workflow): promote out of experimental DynamicWorkflow graduates to a released top-level capability after a hardening round driven by live model runs and adversarial review: - Live-exercised with a real Opus orchestrator across ten scenarios (fan-out, structured output, budget exhaustion, error salvage, runtime reveal, defer_loading, nesting refusal, open-ended research). The one model-facing trip point found -- reading the sub-agent call budget as per-script rather than per-run -- is fixed in the tool description. - Reveals on a deferred capability no longer leak: get_tools holds reveal folding and announcements until the capability is loaded. Ownership is resolved by identity through the run's capability registry, unwrapping wrapper chains, because ids cannot identify the owner (an id-less capability registers under a generated key, and a wrapper registers in place of what it wraps). - Public annotations erase sub-agent outputs as `object`, not `Any`. - README drops the experimental banner, documents the per-run budget and keyword-only construction, and loses a stray editing artifact that had been committed at the end of the file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(dynamic-workflow): let sub-agents inherit the run model * test(dynamic-workflow): exclude never-run parent model fn from coverage The parent model callback in test_inherit_model_off_keeps_sub_agent_bound_model is asserted never to run, so its body is unreachable and broke fail-under=100. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(dynamic-workflow): merge into CodeMode when both are on one agent With both capabilities registered, CodeMode used to fold run_workflow into the run_code sandbox as a plain function taking a code string, so the model had to write a script containing a second script as a string literal, run in a nested sandbox, with no return schema. The two now compose instead: when a CodeMode capability is present, DynamicWorkflow exposes each sub-agent as its own (task) tool carrying the agent's output schema, and CodeMode renders them as typed async functions inside run_code, callable alongside the agent's regular tools in one script. Budget, usage forwarding, and the nesting guard moved into a shared call path so both modes enforce the same rules; the sub-agent guidance run_workflow's description carried moves into instructions in merged mode. No new public API: the merge is automatic. Standalone run_workflow scripts are now statically type-checked against the sub-agent signatures before execution (every call is a fresh sandbox, so the check is always sound); a misspelled function or positional task costs a retry but no sub-agent budget. The runtime kwarg guards remain, because Any-typed values (e.g. json.loads results) evade the checker, and the tests now exercise them through exactly that route instead of reaching into private helpers. Also: budget-sharing across multiple run_workflow calls in one run gets a test (the tool description's headline claim was untested), the examples/ README folds into the capability README, and the unrelated .gitignore line is dropped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: explain DynamicWorkflow in the main README The main README showed CodeMode end to end but only gave DynamicWorkflow a matrix row, so a reader had no way to see the point of the capability without opening its package README. Add a compact section after the Quick start: the turn-by-turn delegation problem, the two-line setup, the script the model writes, and how it composes with CodeMode, budgets, and deferred loading. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(dynamic-workflow): move under experimental New capabilities start under pydantic_ai_harness.experimental per repo convention, and the planned extensions (structured sub-agent inputs, durable workflows) touch the sub-agent call contract, so the API needs room to move before names go stable. This reverses the earlier top-level promotion on top of everything built since: importing the capability now emits HarnessExperimentalWarning and there is no top-level export. The README additionally explains the experimental status, positions DynamicWorkflow against SubAgents (one-delegation-per-turn vs scripted choreography), and documents the merged-mode tool-name collision. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(dynamic-workflow): separate from CodeMode instead of merging The CodeMode merge coupled two independent capabilities, and when their composition did not line up -- a restrictive CodeMode `tool_selector`, or running out of the sub-agent budget mid-script -- it degraded in ways the merge could not recover cleanly. Drop the merge: DynamicWorkflow always exposes its `run_workflow` tool and no longer inspects the run's capabilities. `run_workflow` is itself a code-execution sandbox, so CodeMode must not fold it into `run_code` -- that would make the model pass a script as a string argument to a function inside another script. CodeMode now keeps any tool carrying `code_arg_name` metadata native; `run_workflow` already declares it, so with both capabilities the model sees `run_code` and `run_workflow` as two independent tools. The rule only affects code-execution tools, so CodeMode is unchanged for every other tool. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(dynamic-workflow): correct max_duration_secs semantics and refresh README The docs claimed `max_duration_secs` counts wall-clock, including time awaiting sub-agents fanned out with `asyncio.gather`, and justified having no default cap on that basis. Empirically that is false across every monty the capability supports: the timer is a per-bytecode-step check, so it measures in-sandbox execution time and excludes time the script spends suspended on the host awaiting sub-agents (sequential or gathered). Correct the claim in `_toolset.py`, `_capability.py`, and the README, and describe what the cap is actually for: a pure-CPU `while True` runaway, the one thing the sub-agent budgets cannot catch. Add a regression test that pins the corrected behavior so the stale claim cannot silently return: slow sub-agents under a small cap still complete. Rewrite the README prose in a plainer voice with no em-dashes, and cite Jarred Sumner's Bun Zig-to-Rust rewrite as a real-world example of the same pattern (at Claude Code's session scale, not this specific API). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
77c08faeb7 |
docs: reconcile README with shipped capabilities; enforce parity in CI (#342)
* docs: reconcile README with shipped capabilities; enforce parity in CI The README undersold the package: the "capability matrix" still listed capabilities as unmerged PRs after they shipped as experimental/*, the version floor was three minors stale, and the ecosystem example used an API (SubAgentConfig) that no longer exists. Readers could not tell what actually ships. Restructure around what ships today (4 stable + 10 experimental, each true to its exported classes) vs a roadmap of only unshipped work, and add a mechanical docs-parity test so a capability can no longer land without showing up in the README. Also add the missing media README and refresh the agent skill's capability list and version floor. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: mark unreachable non-package guard as no-cover The __init__.py guard in _capability_packages() never triggers in a clean tree (__pycache__ is already filtered by the underscore-prefix check), so the branch was uncovered and dropped total coverage below the 100% floor. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: keep README's capability matrix, add shipped rows Revert the README to main's structure and make the update additive: flip the newly-shipped capabilities to shipped in the existing capability matrix and add rows for the ones that were not listed (ManagedPrompt, PyaiDocs, RuntimeAuthoring, Media), rather than restructuring the page. Bump the pydantic-ai-slim floor to 2.1.0. Fix the SKILL.md anchors that pointed at the removed sections. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: add Dynamic Workflows (#273) roadmap row to matrix Reconcile against the 7/7 harness state sweep: #273 is complete but open (not merged, no package on main), so it belongs in the matrix as a tracked roadmap row. It was the one item from the sweep still missing after the shipped-row updates. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
b4365440b5 |
feat: serve a Pydantic AI agent to editors over ACP (experimental) (#274)
* feat: add ACP capability to serve agents to editors Editors and TUIs that speak the Agent Client Protocol (Zed and others) can drive an external coding agent, but plugging a Pydantic AI agent into one previously meant implementing the ACP server side by hand. run_acp_stdio serves any Agent over stdio: streamed text and thinking, file edits rendered as diffs, human-in-the-loop tool approval mapped to deferred-approval tools, per-workspace sessions via a session_config hook, model switching, cancellation, and optional session persistence. FileSystem.get_toolset/Shell.get_toolset now return their concrete toolset types so the ACP presenter can recognize their tool calls by name and annotate them with kinds, locations, and diffs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: skip ACP tests at collection when the acp extra is not installed The slim CI jobs sync without extras, so `agent-client-protocol` is absent and the ACP test modules failed at import during collection. Ignore them via conftest when `acp` can't be found; `test_packaging.py` stays collected since package metadata holds on base installs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(acp): enable unstable routing and bound streamed updates by byte size session/set_model and session/close are advertised at initialize, but the ACP SDK router rejects them as unstable unless run_agent is given use_unstable_protocol=True -- so the model picker and session-close affordance returned method_not_found over a real connection. Enable the flag. Streamed text was chunked by character count, but the SDK serializes with ensure_ascii=True, so a non-ASCII code point expands up to 12 bytes (a surrogate pair) inside the JSON string. A single agent_message_chunk of emoji/CJK could exceed the client's 64 KiB read buffer and drop the connection. Chunk by escaped byte length instead. Both gaps were invisible to the suite: the close/set_model tests called the adapter directly (bypassing the router) and the large-output test used only ASCII. Add a through-the-router stdio test and a non-ASCII chunking test as guards. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(acp): per-turn usage, read-resilient terminal cleanup, persistence + native-toolset tests Address review follow-ups on the ACP capability: - Report per-turn token usage on `PromptResponse.usage`, summed across approval passes (UNSTABLE ACP field; clients that don't support it ignore it). - Suppress client errors during the shielded terminal kill/release so a failing cleanup call can't mask the in-flight CancelledError (the spec requires the turn to end with a `cancelled` stop reason). - Cover the persistence guarantees that previously had no store-active test: a cancelled turn commits nothing, an approval-resume turn persists each update once (no duplicate tool-call start), and `StoredSession` round-trips through Pydantic across the full `SessionUpdate` union. - Add a through-the-wire stdio test that the editor-native fs/terminal toolsets route to the client when mounted per session. - Document the dynamic-`ApprovalRequired` partial-side-effect nuance, per-turn usage, and that session persistence composes with per-run step durability. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(acp): editor-native reads for read-only fs clients, writes delegated locally A client that advertises filesystem reads but not writes previously got no editor-native filesystem at all (acp_filesystem returned None, so callers fell back to a fully-local toolset and lost the editor's live view, e.g. unsaved buffers). It now returns a combined toolset: reads route through the editor, and writes are delegated to the local FileSystem capability rooted at the session cwd (reusing its path sandboxing rather than reimplementing writes). This is coherent only when the agent shares the workspace disk with the editor (same machine, or an agent inside the editor's container); for a remote editor the writes land on the agent's disk. Documented as such at the helper and README. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(acp): point _all_known_model_names at the now-merged known_model_names() pydantic-ai#5803 added `pydantic_ai.models.known_model_names()`, the public replacement for the `KnownModelName.__value__` introspection. It isn't in a released `pydantic-ai-slim` yet, so the swap (and the floor bump it requires) waits for that release; update the breadcrumb so it's actioned then. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(acp): replay user turns on session/load, advertise MCP capabilities Two spec-conformance gaps from PR review: - The transcript only ever recorded agent-direction updates, so a reopened session replayed a one-sided conversation. ACP requires session/load to replay the entire conversation, user turns included. The prompt's content blocks are now seeded into the turn's update list -- recorded for replay but never sent live (the client renders its own prompt; the prompt-turn spec sends no user echo) -- riding the existing commit path so a cancelled turn still rolls its user message back. - initialize left mcp_capabilities at the default (http/sse false), and the spec forbids clients from sending HTTP/SSE MCP servers that are not advertised -- making the documented session_config -> mcp_servers path unreachable for conforming clients over those transports (stdio is not gated). Advertising is now an explicit opt-in mirroring prompt_capabilities, since only the embedder knows which transports their session_config connects; setting it without a session_config fails at construction rather than inviting servers that session/new would reject. Also adds the guard test from review follow-up: _all_known_model_names fails loudly if pyai ever recomposes the KnownModelName alias. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(acp): make the collection guard coverable in a single environment The acp-extra collection guard was an `if` statement whose true arm only runs on slim installs, so local `make testcov` (always all-extras) could never reach 100% -- only CI's combined slim+all-extras matrix could. A conditional expression has no statement arc for branch coverage to miss, so the documented local gate works again without excluding a line CI genuinely covers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(acp): record why _all_known_model_names avoids the unreleased public API The breadcrumb read as if known_model_names() were already usable; make the deliberate avoidance explicit. The public enumerator is merged upstream but not in any released pydantic-ai-slim, and the test-floor CI job runs against the floor release, so calling it would break there. Move the rationale from the docstring to a code comment (docstring stays contract-only) and spell out the swap-and-delete trigger: a release that ships it plus a floor bump. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(acp): add spec-conformance suite driven over a real in-memory wire The existing tests invoke adapter methods directly, below the SDK's JSON-RPC router and serialization -- a boundary that cannot see two bug classes that already bit this adapter: a method the router gates before the adapter runs (the use_unstable_protocol reachability bug) and a frame whose serialized bytes overrun the client buffer (the ensure_ascii chunking bug). `tests/acp/_wire.py` closes that gap without a subprocess: a real ClientSideConnection talks to acp.run_agent across a socket.socketpair in one event loop, so every request crosses the router and codec and every update arrives as bytes the client re-parses, with asyncio's default 64 KiB reader limit standing in for the stdio buffer. `tests/acp/test_conformance.py` is organized by spec clause, not by adapter method, each with an oracle built from the spec/input rather than the adapter's own output: - version negotiation echoes the requested version (not a literal) - capabilities are advertised iff supported (load_session, mcp, auth_methods, session list/fork/resume, modes, config options) -- read back off the wire - unstable methods route through with the flag and are method_not_found without it (the load-bearing reason run_acp_stdio enables it) - error codes distinguish method_not_found from invalid_params - session/load replays the entire conversation including user turns - large non-ASCII output reassembles intact within the client's read buffer CONFORMANCE.md records the clause-to-test matrix, the open gaps (none a known bug), the two verified deviations, and the N/A ledger, derived from a spec-page sweep with adversarial verification of every candidate finding. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(acp): stop surfacing unconsumed additionalDirectories The adapter accepted the client's additionalDirectories and forwarded them to the session_config AcpSession, but never advertised sessionCapabilities.additionalDirectories -- so a conformant client never sent them, and nothing downstream consumed them (the FileSystem capability is single-root). The ACP capability is also still UNSTABLE. Drop the dead surface: remove the field from AcpSession and stop forwarding it. The acp.Agent base signature still carries the parameter, so the session methods accept and ignore it. Supporting it properly (multi-root filesystem + advertisement) is recorded as a deferred feature. Also trims CONFORMANCE.md to a public supported-capability summary. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(acp): keep inline image bytes and let run_acp_stdio advertise MCP An ACP image block always carries inline `data`; `uri` is only an optional source reference, so preferring the URL dropped the bytes the client actually sent in favour of a link the model may be unable to fetch. Prefer the inline data, matching the reference ACP agents. `run_acp_stdio`/`run_acp_stdio_sync` forwarded every adapter option except `mcp_capabilities`, so advertising MCP transports was only reachable by constructing `PydanticAIACPAgent` by hand. Forward it for parity. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(acp): reliable approval scope, mapped stop reasons, safe session reload "Always allow"/"always reject" keyed the scope on the raw tool arguments, which a streaming model (and OpenAI by default) delivers as a JSON string; re-ordered keys then produced a different key and silently re-prompted for a call already decided. Canonicalize via `args_as_dict()` so the same logical call shares one remembered decision. Completed turns always reported `end_turn`, hiding `max_tokens`/`refusal` from the client. Map the model's finish reason to the ACP stop reason. `session/load` overwrote a still-open session without cancelling its in-flight turn, leaking a task that could later persist stale state over the restored transcript. Tear it down first, sharing the close_session teardown path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(acp): correct the API reference and capability notes The README API block omitted `session_store`/`models` (and now `mcp_capabilities`), the limitations list contradicted the documented MCP-via-`session_config` support, and the overwrite-diff note misdescribed what the code emits. Also drop a dangling internal-doc reference from a source comment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(acp): record the unhandled SessionStore failure contract A durable SessionStore can fail on save or return a corrupt payload on load, but the adapter currently propagates those exceptions rather than handling them. Document the gap on the Protocol where store implementers will see it, so the behaviour is a known boundary rather than a surprise -- without adding handling we have no consumer for yet. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(acp): correct the SessionStore failure note to match observed behaviour The previous note assumed store exceptions surface "raw". Verified against the SDK (connection.py `_run_request`) and over a real in-memory wire with a raising stub store: they are converted to JSON-RPC errors before reaching the client -- a `pydantic.ValidationError` to `invalid_params` (-32602), anything else to a generic internal error (-32603). Record what the client actually receives rather than what was guessed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(acp): handle SessionStore failures instead of leaking them `session_store` is a public, documented extension point, so a durable store that fails on save or returns a corrupt payload on load is reachable today -- not hypothetical. A failed save previously errored the very turn the user had already watched stream to completion; a corrupt load leaked raw pydantic/IO detail to the client. Make save failures non-fatal: log and swallow them so the turn (or session) that already committed in memory still succeeds, and the next save catches the store up. Make load failures fail `session/load` with a clear internal error rather than a leaked exception, since a session that cannot be read cannot be reopened. Verified over the in-memory wire with a failing stub store. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(acp): close out in-flight tool calls when a turn is cancelled A tool call announced as pending/in_progress was never driven to a terminal status when its turn was cancelled (via session/cancel or a dismissed permission dialog), so a client kept rendering it as running after the turn ended cancelled. On the cancel unwind, fail any tool call still lacking a terminal result -- sent live (a cancelled turn never commits its transcript) and shielded so the asyncio cancellation cannot abort the send. Verified over the in-memory wire. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(acp): reject prompts queued behind a session that closed or reloaded A prompt waiting on the session's turn lock held a reference to the old SessionState; close_session/load_session only cancel the *active* turn, so the queued prompt would run against the discarded state - invisible to session/cancel - and persist its orphaned history over the closed (or just restored) session. Re-checking liveness after acquiring the lock turns that zombie turn into the invalid_params the client already gets for a prompt sent after close. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(acp): never swallow a handler's own teardown cancellation prompt() and _cancel_active_turn awaited the turn task bare, so the CancelledError from connection teardown was indistinguishable from the turn's own cancellation and got converted into a response (or, via a dismissed permission dialog, replaced by the internal _TurnCancelled). The SDK cancels each handler task exactly once on shutdown and its sender is already closed by then, so a handler that survives that one cancel and tries to answer hangs Connection.close() forever. Awaiting through asyncio.shield keeps the two cancellation sources apart: the turn's end is read off the (done) task, while the handler's own cancellation propagates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(acp): don't report rollback when a cancel lands in the post-commit save The store save after a turn commits was the one suspension point left between commit and return: a session/cancel delivered there escaped the turn task and the prompt answered 'cancelled' (the rollback signal, no user_message_id) while the session's in-memory history and transcript already contained the whole turn - the next prompt would build on history the client believed was discarded. The InMemorySessionStore never suspends, which is why the existing cancel-persistence test could not see this; any real (file/db) store can. A cancel that arrives after commit has simply lost the race, so the interrupted save is treated like the write failures _persist already swallows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(acp): don't leak a terminal when run_command is cancelled mid-create The terminal/create call sat before the cleanup try-block, so a cancellation landing there unwound without a kill or release - but the request may already be on the wire (the SDK sender flushes queued payloads even when the awaiting future is cancelled), leaving the command running client-side with nobody holding its id. Running the create as a shielded task lets the cleanup await the response late, learn the id, and kill/release as usual. Folding the late-id case into one kill path and one release path also keeps the existing guarantee that a failing kill never skips the release. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(acp): end limit-hit turns with max_turn_requests, close out tool calls on errors pydantic-ai's default UsageLimits(request_limit=50) means a long tool loop routinely raises UsageLimitExceeded - which escaped the turn as a JSON-RPC internal_error, the exact condition ACP defines the max_turn_requests stop reason for (token limits map to max_tokens). The raising run's partial messages are not retrievable, so such a turn rolls back like a cancellation and says so (no user_message_id, no usage). Turns failing with any other exception now also close out their announced tool calls before the error reaches the client, matching the cancellation path, so the client does not render them as running next to the turn's failure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(acp): resolve workspace-relative paths before they reach fs/read|write ACP requires absolute paths on fs/read_text_file and fs/write_text_file, but models routinely emit workspace-relative ones - the local FileSystem tools document relative paths and share these tools' names, and the presenter layer already absolutizes for exactly that reason. The client-backed toolset forwarded them raw, producing non-conformant requests the client may reject. acp_filesystem now hands the toolset the session cwd so relative paths resolve before the wire; absolute paths and directly-constructed cwd-less toolsets are untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(acp): true up the conformance claims, comments, and doc links The stop-reason note predated the finish-reason mapping and said the opposite of what the code (and its tests) do; the prompt-capabilities bullet read as inbound enforcement when the spec puts that restriction on the client; the stdio MCP gap (the spec's unconditional MUST) was undisclosed; and two code comments claimed validation/rejection the SDK router does not actually perform - the adapter's own raises are the load-bearing behavior. Doc links now use the canonical pydantic.dev paths both old URLs redirect to. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(acp): shield the terminal create in place instead of via a side task The ensure_future + asyncio.shield shape from the previous commit hit Python 3.12+'s shield behavior: once the outer await is cancelled, a late failure of the inner create is reported to the loop exception handler even when the cleanup retrieves it, which spams production logs and fails under pytest-anyio. An anyio shield around the create await itself is simpler and leans on the same anyio-mediated cancellation the kill/release cleanup already depends on: the cancellation defers to the next await, by which point the terminal id is known and the existing cleanup kills and releases as usual. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(acp): small type and surface cleanups from review - ToolCallPermission.args was typed object although the only construction site passes args_as_dict(); Mapping[str, object] lets a custom policy scope by an argument without isinstance gymnastics. - default_permission_scope is referenced by the permission_policy docs as the fallback to compose with, so it must be importable: exported. - McpServer names the per-server union once instead of spelling it in two places; McpServers gains the TypeAlias marker its sibling had. - _model_state folds the None case so both call sites drop a repeated conditional; the load_session store guard's no-cover pragma was wrong (the SDK router routes session/load regardless of the advertisement, and test_persistence already executes the branch). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(acp): pin claimed behavior the suite asserted too weakly or not at all - The unsupported-method test asserted only RequestError despite name and CONFORMANCE.md promising method_not_found; codes now pinned (-32601, and -32602 for the unknown-session prompt). - The model-override test was tautological: both models answered successfully either way. It now asserts the override's distinct canned output (mutation-checked: dropping the per-run override fails it). - Usage summation across approval passes had no test; an approval turn's output tokens must exceed the resume pass alone (mutation-checked against 'usage = result.usage()'). - New protocol-contract coverage: cancel racing an unanswered permission dialog (turn ends cancelled, pending call driven to failed), double cancel idempotency, image block arriving in model history as decoded BinaryContent, the denial message reaching the failed update's raw_output, chunk_text's exact-budget boundary, and a set_model save failure being swallowed like every other persist failure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(acp): survive raw cancellation during terminal create; answer cancelled after a post-commit cancel An adversarial re-review of this branch's own fixes caught a regression: the anyio shield around terminal/create only blocks anyio-mediated cancellation, but the adapter and pydantic-ai deliver raw task.cancel() (turn.cancel, cancel_and_drain), which pierces it - reintroducing the leaked-terminal window end to end. The create now runs as its own task awaited via asyncio.wait: a raw cancel hits the waiter, not the create, and unlike asyncio.shield (3.12+) a late create failure is not reported to the loop exception handler when the cleanup retrieves it. The leak scenario is pinned at both the toolset level and through the real adapter/agent path. Also from the re-review: a cancel that lands after the turn committed (inside the store save) now answers 'cancelled' as the spec requires - while keeping the committed signals (user_message_id, usage, history) - and the one-tick teardown-vs-turn ambiguity that 3.10 cannot resolve is documented at both await sites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(acp): record the post-commit-cancel semantics in CONFORMANCE Also drops an unbounded poll loop in the permission-dialog race test for a single deterministic tick (prompt sets active_turn before its first suspension point), which was the one partial branch left repo-wide. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(acp): let the turn build its own response and drop the commit-flag tuple _run_turn returned (usage, stop_reason, committed) so prompt() could reassemble a PromptResponse it had all the pieces for; building the response in the turn removes that protocol and the committed flag. Recording turn.updates is now unconditional (only the commit gate matters), and the presenter-fields/acp_filesystem branches collapse to single construction paths. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(acp): appease codespell (unparseable -> unparsable) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Support Pydantic AI 2.x in ACP * feat(acp): forward usage limits into agent runs Let hosts bound each ACP run segment with Pydantic AI usage ceilings while preserving the default run behavior when no explicit limits are configured. * feat(acp): resolve selected session models Let embedders map advertised ACP model ids to concrete Pydantic AI models and re-surface a resident session's current model state. * feat(acp): expose read-only session history Hosts can gauge context with the same estimate_token_count path the compaction tiers use by reading committed resident-session history. The returned list is a shallow snapshot, and callers must treat the shared ModelMessage objects as read-only. * Track ACP 0.11 session config models * chore(acp): fix coverage, trim redundancy, clarify module names Get the ACP PR to green CI and simpler to review, without changing behavior. - Coverage: the only uncovered line was a dead `return None` branch in the `_model_option` test helper (source was already 100%). Assert the option is present instead, dropping the untested branch and its now-redundant guards. - Remove CONFORMANCE.md: its supported/not-supported matrix duplicated the package README's feature sections and "Cancellation and limitations". Point the two references at the README. - Dedup the test ACP clients: RecordingClient, FakeClient, and WireClient each re-spelled the same ~13-method "unused capability" stub block to satisfy the SDK Client interface. Extract it once into RecordingClientBase; each client now subclasses it and overrides only what it exercises. Interface drift is one edit. - Rename two opaque modules for clarity (files only; public symbols unchanged): _present.py -> _presentation.py (matches its ToolCallPresentation exports) and _native.py -> _client_toolsets.py (client-routed filesystem/shell toolsets), with tests/acp/test_native.py -> test_client_toolsets.py to mirror the source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(acp): move under experimental and shrink the public surface New capabilities start under pydantic_ai_harness.experimental, so ACP follows: importing warns HarnessExperimentalWarning and the API may change without deprecation. __all__ drops the typing-only aliases (25 to 18 names); default_permission_scope and McpServer stay because policies compose with the default scope and AcpSession.mcp_servers is typed by McpServer. Tests move to tests/experimental/acp to mirror the source tree; the experimental warning test lives in the SDK-gated test_acp.py so slim (no-extras) installs stay green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(acp): expose model_resolver and usage_limits on the stdio entry points PydanticAIACPAgent already accepted both, so wanting a per-run token ceiling or host-defined model ids forced users off run_acp_stdio onto the class and a hand-rolled acp.run_agent call, where forgetting use_unstable_protocol=True silently loses session/close. Forward the two params so the one-liner covers every adapter option. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: David SF <david.sanchez@pydantic.dev> |
||
|
David SFandGitHub
|
b5b93704c3 |
feat(subagents): contain unexpected sub-agent crashes as bounded retries (#326)
A delegate that crashes with anything other than an expected soft degradation (a provider ModelAPIError/FallbackExceptionGroup, a plain ValueError from a bad tool argument) aborts the whole parent run. For an orchestrator fanning out to flaky delegates that is too fragile: one crash kills unrelated in-flight work. Add opt-in SubAgent(contain_errors=True) / SubAgents(contain_errors=False) default. When on, the crash is caught and returned to the parent as a bounded ModelRetry instead of propagating. It stays loud: the exception rides the retry message, it is logged, and tool_retries still bounds consecutive crashes into an abort -- a genuine bug is never masked as success. Orthogonal to on_failure (which only sets the message for expected soft degradations). Cancellation, a shared UsageLimitExceeded, pydantic-ai control-flow signals, and UserError always propagate. |
||
|
David SFandGitHub
|
069d1ffacf |
feat(subagents): configurable delegate-tool retries (#324)
* feat(subagents): configurable delegate-tool retries A sub-agent that errors (e.g. exhausts its own output retries) surfaces as a ToolRetryError on the parent's delegate_task tool, which defaults to retries=1 -- so one flaky sub-agent run crashes the whole parent run with no chance to re-delegate. Add a `tool_retries` field on `SubAgents` (default `None`, inherits the parent's default, so behavior is unchanged) that sets the delegate tool's retry budget, letting callers give the parent a chance to recover from a transient sub-agent failure. * feat(subagents): resilient default for delegate-tool retries Default `tool_retries` to 2 (not None/inherit) so a repeated flaky sub-agent does not abort the parent run on its first repeat. The retry counter resets after any successful delegation, so this bounds consecutive failures. `None` still inherits the parent agent's default. |
||
|
Aditya VardhanandGitHub
|
3c5ec34f9e |
Bump vcrpy to 8.2.1 to fix YAML deserialization RCE (#300)
Resolves Dependabot alert GHSA-rpj2-4hq8-938g: vcrpy < 8.2.1 loads cassette files with PyYAML's unsafe loader, allowing arbitrary code execution on cassette load (e.g. in CI). 8.2.1 switches to SafeLoader. |
||
|
Aditya VardhanandGitHub
|
6eb91dfe86 | Bump pydantic-ai floor to 2.1 (#303) v0.5.0 | ||
|
Aditya VardhanandGitHub
|
51ba5693bf |
Revert "fix(code_mode): tighten run_code prompt against phantom imports and…" (#317)
This reverts commit
|
||
|
|
326c3d648d |
Emit a span when compaction happens (#312)
* Emit compaction telemetry for observability Compaction silently rewrites a run's message history, so an operator could not see when or how much it reclaimed. Each compacting strategy now emits a `compact_messages` span (with before/after message and token counts) when it actually runs, so compaction is visible alongside the rest of a run's traces. The span is emitted in `before_model_request` rather than `compact` so that `TieredCompaction` produces one span for the escalation instead of one per tier. It rides on the run's tracer, which is a no-op without instrumentation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015kRjwKq2AmcuWZcDfJ4WHs * Suppress compaction span on no-op and uninstrumented runs A trigger can fire while compact() returns the history unchanged (most notably DeduplicateFileReads, which has no threshold), so wrapping compact() inside the span emitted a compact_messages span on ordinary turns with identical before/after counts. Run compact() first and only start the span when the returned history actually changed. Building the token attributes also ran estimate_token_count (and any user tokenizer) over the full before/after histories on every compaction, even when ctx.tracer is a no-op tracer and the span is discarded. Gate the attribute computation on span.is_recording() so uninstrumented runs pay nothing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015kRjwKq2AmcuWZcDfJ4WHs * Make compact_with_span config params keyword-only Forcing strategy/messages/compact/tokenizer to be passed by keyword prevents argument-order mistakes at the six strategy call sites, where several same-typed callables/lists are passed positionally. * making compact_with_span * only after ctx * Type compaction test tracer field with opentelemetry Tracer The fake RunContext tracer field was typed as Any. opentelemetry provides the proper type and core's RunContext uses it, so type the field as opentelemetry.trace.Tracer to match. * Type compaction test fake context model with pydantic_ai Model The fake RunContext model field was typed as Any and backed by a hand-rolled stub carrying only model_id. Back it with TestModel, a real Model subclass, and annotate the field as pydantic_ai.models.Model so the fake conforms to the type the production code expects from ctx.model. * Return original messages on no-op compaction The no-op early-return branch of compact_with_span only runs when the history is unchanged, so returning the original messages instead of the compacted list makes the no-op explicit at the call site without changing behavior. * redundant list length check * history covers would_clamp * Set gen_ai.conversation.compacted on the compaction span The OpenTelemetry GenAI semantic convention models context compaction as the `gen_ai.conversation.compacted` boolean, set true-only. The span emits solely when the history actually changed, so the flag is only ever true. The `compaction.*` attributes have no convention equivalent and stay harness-specific. --------- Co-authored-by: Aditya Vardhan <adtyavrdhn@gmail.com> |
||
|
|
4e21f7bcbf |
docs(agents): discourage private-helper imports in tests (#316)
Keep tests pinned to each capability's public surface so they keep passing across internal refactors of the `_`-prefixed modules, and point genuinely unreachable branches at `# pragma: no cover` instead of reaching into private helpers. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
David SFandGitHub
|
f460861435 |
Add experimental CE-discovery capabilities: context, subagents disk-load, docs, authoring (#304)
RepoContext (instruction autoload + asset inventory), SubAgents disk-load, PyaiDocs (on-demand pyai docs), RuntimeAuthoring (runtime capability authoring). Reviewed + hardened; validated under pydantic-ai 1.107 and 2.0.0. |
||
|
|
0418280d1e |
test(logfire): tolerate pydantic-ai 2.0.0 agent-span rename (#305)
pydantic-ai 2.0.0 reworked Instrumentation: the agent run span was renamed from `agent run` to `invoke_agent agent`, the tool span from `running tool` to `execute_tool noop`, and several span attribute keys were renamed. The `test on latest pydantic-ai` CI job (newest pyai = 2.0.0) failed on the two span-name assertions while the locked/floor 1.x jobs stayed green, so a static snapshot update would just flip the failure onto the 1.x jobs. Harness still supports the 1.x floor (`pydantic-ai-slim>=1.105.0`), so make the two tests version-tolerant instead of bumping the floor: - Add `_PYDANTIC_AI_GE_2` from `importlib.metadata.version`. - `test_provider_backed_resolution_tags_v1_instrumentation_spans`: branch the expected agent-span name so it stays live on both versions. - `test_baggage_propagates_to_run_and_child_spans`: skipif on 2.0.0 since its full span-tree snapshot needs a separate 2.0.0 refresh. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
bb6da9948d |
Keep framework tools native in CodeMode (#296)
* Keep framework tools native in CodeMode * test(code_mode): cover load_capability bootstrap through Agent The existing unit test exercises the native/sandbox split via a static toolset. Add an Agent-level test for the actual issue #276 scenario: a deferred capability plus CodeMode(tools='all'), asserting load_capability stays a native call and the deferred member tool keeps defer_loading=True instead of being folded into run_code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(code_mode): route framework tools native off tool_kind alone The name fallback (`name == _SEARCH_TOOLS_NAME`) was redundant: pydantic-ai has set `tool_kind` on the `search_tools` ToolDefinition since 1.95.0, and the dependency floor is 1.105.0, so `tool_kind is not None` already catches it. Routing keys purely off the framework discriminator now. Also switch the #276 bootstrap test from FunctionModel to TestModel (per the test guideline) and rescope its assertions to provider-agnostic facts -- the old assertion only held because FunctionModel's profile happens to support native tool search. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(code_mode): mark unreached deferred-tool body no cover The #276 bootstrap test keeps `demo_tool` hidden, so its body never runs. Mark it `# pragma: no cover` to satisfy the 100% coverage gate, matching the sibling deferred-tool tests. --------- Co-authored-by: Aditya Vardhan <adtyavrdhn@gmail.com> |
||
|
|
6232e0762b |
feat(subagents): replace name-keyed mappings with a SubAgent list (#298)
* feat(subagents): replace name-keyed mappings with a SubAgent list Referencing sub-agents by dict keys across `agents`, `descriptions`, and `limits` meant the same name was repeated in several places and a typo in one mapping silently desynced from the others. A single `SubAgent` entry now pairs an agent with its name/description overrides and per-delegate run controls, so `SubAgents(agents=[SubAgent(...), ...])` keeps everything for one delegate in one place. A delegate's name defaults to the agent's own `name` (overridable via `SubAgent(name=...)`); two delegates resolving to the same name, or an agent with no name and no override, is rejected at construction. `SubAgentLimits` is folded into `SubAgent` and removed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HkzP9cq8KvuTDLLf7BJZFx * refactor(subagents): keep toolset constructor kwarg named `agents` Per review feedback, SubAgentToolset keeps its `agents` constructor kwarg rather than renaming to `sub_agents`, matching the public SubAgents(agents=...) API and the internal _by_name source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Aditya Vardhan <adtyavrdhn@gmail.com> |
||
|
|
50638595fd |
Add "Part of the Pydantic Stack" footer to README (#297)
Links the marketing pages for Pydantic AI, Pydantic Logfire, and Logfire AI Gateway so readers can discover the rest of the stack in one place. UTM params attribute traffic back to this repo. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
3f4cbd0a66 |
fix(code_mode): tighten run_code prompt against phantom imports and bad inputs (#295)
* fix(code_mode): tighten run_code prompt against phantom imports and bad inputs CodeMode agents repeatedly waste run_code turns on avoidable mistakes -- importing unavailable modules (textwrap) or a phantom `functions`/`sandbox_tools` module, pasting oversized one-line literals (unterminated-literal errors), indexing typed tool results as dicts, and echoing `...`/`??` truncation placeholders back as arguments. Each failure burns a model turn and pushes agents toward their request limits. The prompt now closes the importable-module allow-list (anything outside it fails; the listed functions are pre-bound, do not import them) and adds a short caution: no oversized literals, results are typed objects (read fields, don't json.loads or index a string), and never pass a truncation placeholder as an argument. Prose-only; 100% branch coverage retained. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(code_mode): make the run_code prompt replaceable via `instructions` CodeMode had no way to customize the run_code prompt -- you had to fork. Add an `instructions` parameter: a str replaces the built-in prose, or a callable receives the built-in (host-aware) prose and returns the replacement (append/prepend/rewrite). The auto-generated tool catalog is always appended afterward, so run_code stays callable regardless. This is the right seam for project-specific guidance instead of baking it into the shared default. Also drop the version-fragile module examples from the import-allow-list note (e.g. `collections` is unavailable today but could land in a future Monty); the list itself is already in the prompt, and the prose is now customizable anyway. 100% branch coverage retained; tests cover the str and callable forms (catalog preserved). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(code_mode): `instructions` is a plain str + export the default prose Drop the callable form of `instructions` (overkill). It is now `str | None`: a str replaces the run_code base prose, `None` keeps the built-in. To build on the default instead of replacing it, import the newly-exported `default_run_code_instructions()` and append/prepend/edit, then pass the result -- no callable needed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(code_mode): append via `instructions`, replace via `dangerously_replace_instructions` Keeps the Monty-tuned base prose by default so a customization can't silently desync `run_code` from the sandbox semantics. `instructions` now appends to the built-in prose (before the catalog); a separate, clearly-labeled `dangerously_replace_instructions` is the escape hatch for full replacement. The two are mutually exclusive (UserError on both set). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
1b709d8e30 |
fix(overflow): read_tool_result returns on a missing handle instead of raising (#293)
* fix(overflow): read_tool_result returns on a missing handle instead of raising A wrong handle (e.g. a model passing a tool-call id) or a result no longer in the store made `read_tool_result` raise `ModelRetry`, which consumes a tool retry and, once the budget is spent, escalates to a fatal `UnexpectedModelBehavior` that kills the run. This took down a pydanty reproducer run (the model passed `functions.read_tool_result_44.1` as a handle; max retries was 1). `_read_slice` now returns a guiding message on a missing handle -- pointing the model at the exact handle string from the spill marker, or to re-run the original tool -- so a bad handle can never crash a run. The `offset`/`limit` input validations stay `ModelRetry` (deterministic caller errors the model corrects). 100% branch coverage retained. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(overflow): don't echo the store error in the missing-handle message A FileNotFoundError (and custom-store errors) can carry the resolved filesystem path or other backend detail; the model only needs the handle and recovery guidance. Drop ": {exc}" and assert the store path is not in the returned message. Addresses review feedback on #293. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
e6dfd4c303 |
ci: add pydantic-ai v2 beta early-warning job and make code_mode v1/v2 compatible (#291)
* ci: add non-blocking pydantic-ai v2 beta test job The harness targets the pydantic-ai v1 line, but the `>=1.105.0` floor also admits v2 prereleases once prerelease resolution is enabled. Nothing in CI exercised that path, so v2-breaking changes were invisible until release. This adds a `test-v2-beta` job that resolves the latest v2 beta and runs the suite against it. It surfaces real breakage today: v2 dropped the `calls` argument from `ToolManager.get_parallel_execution_mode`, which CodeMode still calls with the v1 signature, so `code_mode` raises `TypeError` under v2. The job is intentionally kept out of the `check` gate so an expected red result on the unsupported v2 line never blocks a merge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: cap the v2-beta pytest step so a hang fails fast The v2 beta job is expected to surface breakage, but some v2 breaks hang instead of failing cleanly. When code mode raises an unhandled exception inside a DBOS workflow, DBOS's background recovery thread stays alive and the Python process never exits, so the step rode to the 20-minute job timeout and burned a full runner slot on every push. Wrap the pytest invocation in `timeout -k 30 300` so any such hang fails fast (exit 124). A per-test timeout plugin would not help: the hang happens during interpreter shutdown, after the test body completes, so the cap has to be on the process. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(code_mode): call get_parallel_execution_mode compatibly across v1/v2 pydantic-ai v2 (#5339, shipped in 2.0.0b1) dropped the `calls` argument from `ToolManager.get_parallel_execution_mode`: it now reads the run-scoped mode from a context var and applies per-tool `sequential` barriers separately. The harness passed `[]` specifically to isolate the context var from per-tool flags, which is exactly what the no-arg v2 call returns, so the two are equivalent. Inspect the method arity and call the matching shape. Inspecting rather than catching TypeError avoids swallowing a genuine TypeError raised inside the method. The `Callable[...]` annotation erases the bound signature so both call shapes typecheck whichever major's stubs pyright resolves. Without this, every code_mode run under v2 raises TypeError; inside a DBOS workflow that unhandled error also wedged the process (a non-daemon recovery thread blocked interpreter shutdown), which is what hung the v2-beta CI job. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(code_mode): cover both arities of the v1/v2 mode dispatch The arity-based dispatch added an `else` branch that the v1-pinned coverage gate never executes (the v2 no-arg call only runs under pydantic-ai v2), so total branch coverage fell to 99% and failed the `fail_under=100` gate. Extract the dispatch into `_global_mode_is_sequential` and unit-test both call shapes directly, so both branches are exercised whichever major is installed. This is honest coverage rather than a `# pragma: no cover` that would hide a branch that does run (in the v2-beta job). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: deselect v1-only OTel assertions from the v2-beta job Two instrumentation tests in test_managed_prompt.py assert pydantic-ai v1's OpenTelemetry attribute and span names. v2 deliberately renamed these (aggregated-usage attributes, GenAI-semconv span names), so the tests are expected-red on v2 and carry no signal in this job. Deselecting them keeps the v2-beta job a meaningful early-warning for capability breakage (code mode, durable execution) instead of going red on documented instrumentation drift. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: keep managed prompt v2 signal --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: David SF <david.sanchez@pydantic.dev>v0.4.0 |
||
|
David SFandGitHub
|
caf50108fe | Explain code mode metadata selection (#292) | ||
|
|
24fd30e70c |
docs: stale-PR re-eval, experimental-export, and background-coder review guidance (#288)
* docs: add stale-PR re-eval, experimental-export, and consumer-lens guidance Adopting a months-old PR (#243) surfaced three gaps in the AICA review/authoring guides: - no discipline for re-evaluating a stale or pre-merge PR (retire temporary uv.sources pins once upstream lands; re-check drift vs current main) - the experimental-vs-top-level export convention was undocumented, so the code_mode exemplar steered new-capability authors toward top-level exports - the review checklist named "dogfooding need" but no reference consumer or the cache/cost/reliability lens a long-running background coder cares about Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: key export-stability rule on shipped-in-release, not on a symbol list Only CodeMode has shipped in a published release (v0.3.0 exports just CodeMode); FileSystem, ManagedPrompt, and Shell are top-level on main but pre-release. The earlier wording labeled all four "already-released", which is inaccurate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: drop the reference-consumer lens bullet Keep the two clearest gaps (stale-PR re-eval, experimental-vs-released exports); drop the consumer-lens addition, which named pydanty (a separate repo) in the harness review checklist and was the most likely to read as out-of-place. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: David SF <david.sanchez@pydantic.dev> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
2924277b17 |
Add CodeMode dynamic_catalog flag for cache-stable tool disclosure (#243)
* feat(code_mode): add CodeModeDynamicCatalog for cache-stable tool disclosure Moves CodeMode's per-tool signature catalog out of `run_code.description` (which lives in the prompt-cache-keyed tool-definitions block) and into agent instructions as a dynamic `InstructionPart`, then announces newly- discovered tools via `RunContext.enqueue` rather than by mutating the cached description. The result: the tool-defs block stays byte-stable across Tool Search discoveries / per-step toolset swaps, only the dynamic instructions and append-only system-prompt announcements grow. This is the Tier-2 reshape from pydantic/pydantic-ai-harness#232. Opt-in (default `CodeMode` keeps the catalog in `run_code.description`, which is slightly cheaper on prompt size when the toolset never changes); pair with `ToolSearch` or a churning toolset for the cache win. Depends on pydantic/pydantic-ai#4980 for the pending message queue; pinned to the `background-tools` branch of pydantic-ai-slim until that lands. Cross-provider cache safety completes once pydantic/pydantic-ai#5437 (XML-wrapped mid-conversation SystemPromptPart mapping) ships. * Re-lock pydantic-ai-slim to background-tools HEAD after main merge * Adapt to pydantic-ai refactor: PendingMessage.payload→request * Add e2e test exercising eager tool through run_code (closes coverage gap) * Fold CodeModeDynamicCatalog into CodeMode as a `dynamic_catalog` flag Replace the separate `CodeModeDynamicCatalog` capability (which surgically wrapped `CodeMode`'s assembled toolset) with a `dynamic_catalog: bool` flag on `CodeMode` itself. When set, `CodeModeToolset` keeps only the static base prose in `run_code.description` and surfaces the sandboxed-tool catalog as a dynamic `InstructionPart` via `get_instructions`, and `CodeMode` announces newly discovered tools by enqueuing a `SystemPromptPart` through `ctx.enqueue`. This drops the shim package and its dedicated test module, merging the behavior and its tests into `code_mode`. The search addendum now stays in `run_code.description` in both modes (it's cache-stable), an improvement over the prior wrapper which dropped it. * Enqueue discovery announcement as a bare SystemPromptPart Re-lock pydantic-ai-slim to the latest background-tools HEAD, which now accepts `ModelRequestPart`s directly in `enqueue` and renders mid-conversation `SystemPromptPart`s inline rather than hoisting them (pydantic/pydantic-ai#5509). Drop the `ModelRequest` wrapper workaround and enqueue the `SystemPromptPart` directly; on the wire it renders as an XML-wrapped user prompt, so the catalog cache stays intact across providers. * deps: re-lock pydantic-ai-slim; give test RunContext a live pending-message queue `RunContext.enqueue` now raises when the context has no pending-message queue (pydantic-ai removed the silent-drop path). Pass `pending_messages=[]` in the test helper so the discovery-announcement unit tests still exercise enqueue, and narrow `ctx.pending_messages` before reading it. Real runs wire the queue automatically. * code_mode: drop cast/Any from discovered-name extraction _extract_discovered_names walked the tool-search return with cast() over Any, which the no-typecast rule disallows. Validate the untrusted return through two lenient pydantic TypeAdapters instead: a malformed catalog yields [] and a malformed entry is skipped, matching the prior behavior with a fully-typed path. Also rewrite the one em-dash in the get_instructions docstring to --. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * code_mode: import TypedDict from typing_extensions for py<3.12 pydantic rejects `typing.TypedDict` on Python < 3.12 (PydanticUserError at TypeAdapter construction), which broke the 3.10/3.11 CI jobs. Match _toolset.py and use `typing_extensions.TypedDict`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: David SF <david.sanchez@pydantic.dev> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
9821aab0d5 |
feat(compaction): add ClampOversizedMessages for runaway generations (#286)
A single runaway model response (degenerate whitespace) or giant tool-call payload can produce one part large enough to blow the next request past the provider context cap. No existing strategy can reach it: SlidingWindow drops the oldest messages, ClearToolResults only touches tool results, and feeding the history to SummarizingCompaction hits the same cap. ClampOversizedMessages truncates the offending ModelResponse part in place, keeping a head/tail slice with a marker. Covers response text (critical case) and tool-call args (clamp_tool_call_args, default on); request-side parts stay out of scope. Composes as the first zero-LLM tier of TieredCompaction. Closes #285 Co-authored-by: David SF <david.sanchez@pydantic.dev> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
3ac6034f7e |
feat(experimental): OverflowingToolOutput capability for oversized tool returns (#290)
* feat(experimental): OverflowingToolOutput capability for oversized tool returns An oversized tool return persists in history and is re-sent on every later model request, paying its token cost for the rest of the run. This capability intercepts a return in `after_tool_execute`, reduces it once, and lets the reduced form persist instead of recomputing it per request. Three freely-combinable modes selected by an ordered list of size bands: truncate (cheap clamp), spill (lossless persist + `read_tool_result` handle for on-demand slice/grep/tail), and summarize (size-gated LLM summary inheriting the run's model). The default band is `Spill(then=Truncate())`: lossless when the store accepts the write, a bounded truncation otherwise, never a silent drop. Spilled payloads go through a narrow `OverflowStore` protocol (local-file default) so a durable backend or core #4352's queryable-file primitive can slot in behind the same handle. Mirrors the experimental/compaction package shape, export pattern, and model-inherit path; truncation / ANSI / binary helpers are harvested from PR #185, whose one-way spill this supersedes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(overflow): address review -- store hardening, content overflow, bounds, no-Any, TTL cleanup S1: harden LocalFileStore while preserving cross-agent sharing. Keep a stable, configurable root (so a later run can read an earlier spill), but create it with 0700 perms and verify on read that the resolved target stays within the root, rejecting symlink/`..`/absolute escapes. Sharing is the goal; isolation is not the security mechanism. C1: large ToolReturn.content bypassed reduction -- core renders it as a separate model-visible part that also persists in history. Measure and reduce content with the same band logic (distinct spill handle); non-text content that overflows is left unreduced with a warning. C2: bound read_tool_result -- require offset >= 0 and limit >= 1, clamp limit to a line cap, cap the joined output, and make `pattern` a literal substring instead of a regex so a model-supplied value cannot trigger catastrophic backtracking. A1: drop avoidable Any -- typed tokenizer, narrowed payload/metadata handling, and removed the reportUnknownArgumentType ignores via bool guards. M1: summarize tests now assert the real claims (ctx.model inheritance, explicit model override, usage=ctx.usage threading) with a FunctionModel instead of a wholesale Agent mock. Cleanup feature: opt-in LocalFileStore(cleanup_after=timedelta) prunes files older than the TTL in a background daemon thread; keep-forever stays the default. Pruning is non-blocking and swallows all errors into warnings, never failing a run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(overflow): convert remaining avoidable Any to precise types (A1) `_is_mapping`/`_is_text_sequence` are now `TypeGuard`s, so `json_sketch` takes `object` and the sketch helpers take `Mapping[object, object]` / `Sequence[object]` -- no `Any`, no `Unknown`. In the capability, `original` is `object`, `metadata` / `existing` are `object`, `_copy_mapping` / `_with_handles` return `dict[str, object]`, and the assembled `ToolReturn` locals and `_assemble` return type drop `Any` for `ToolReturn[object]` / `object`. The only `Any` left is the core-inherited `after_tool_execute` hook signature (`args`/`result`/return). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: David SF <david.sanchez@pydantic.dev> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
c56a4b42f8 |
Add host-backed OS and filesystem access to CodeMode sandbox (#262)
* feat(code_mode): expose host-backed OS access to the sandbox Sandboxed `run_code` had no way to reach the filesystem, environment, or wall clock: Monty supports it through an OS callback / `AbstractOS` and directory mounts, but `CodeMode` never threaded `os`/`mount` into `feed_start` or the snapshot resume loop, so callers couldn't enable it. Add `os` and `mount` options on `CodeMode`/`CodeModeToolset`, thread them through `feed_start` and every `resume` site (OS auto-dispatch stops the moment a resume omits them), and make the `run_code` description reflect whether host-backed access is configured. * test(code_mode): harden OS-access tests around the threading invariants Add edge cases that pin the behaviours most likely to regress: OS access surviving across REPL-persisted `run_code` calls, a raising `os` callback degrading to `ModelRetry` instead of crashing the loop, and `mount` accepting a `list[MountDir]`. Hoist the never-invoked callback used by the description/forwarding assertions into one shared helper. * docs(code_mode): tighten and verify the filesystem/OS access section Trim the host-access docs to the essentials and make the example self-contained (drop the undefined helper). The snippet and the documented `mount`/callback constructions are run end-to-end to confirm they work. * docs(code_mode): correct per-request scoping wording `os`/`mount` are static capability fields (no per-run resolver), so the "stateful AbstractOS rooted at a per-user directory" guidance over-claimed. Reword to: build CodeMode per request to scope access. Every other doc line was re-checked empirically against pydantic-monty 0.0.17. * fix(code_mode): don't advertise env/clock for mount-only sandboxes A `mount` only exposes filesystem paths; `os.getenv`/`os.environ` and `datetime.now()`/`date.today()` still require an `os` handler. The description used one host-access note for both, so mount-only agents were told env/clock were routed to the host and would emit calls that fail and burn run_code retries (verified against pydantic-monty 0.0.17). Split the description into three states (none / mount-only filesystem / os), and correct the README and docstrings that conflated the two. * docs(code_mode): correct two run_code description claims verified against monty Audited every statement in the run_code description, docstrings, and README against pydantic-monty 0.0.17. Two were imprecise: - "imported at the top of your snippet" -- mid-snippet imports work, so the rule is just "before use". - OS-enabled note said calls route "to the host environment", but an in-memory AbstractOS (e.g. OSAccess) handles them too -- it's the configured OS handler, not necessarily the host. * docs(code_mode): clarify overlay-mode write persistence and fix wording The mount docs implied writes reach the host, but MountDir defaults to copy-on-write overlay mode, so writes stay in the sandbox unless mode is 'read-write'. Also tighten two awkward/redundant doc lines. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(code_mode): rename public OS/mount surface to be backend-neutral The public type aliases leaked the Monty backend name into a surface we can't rename later. Rename them to match the existing CodeMode/CodeModeToolset convention, and rename the os= parameter to os_access= so it stops shadowing the stdlib os module that sandboxed code itself uses. - MontyOS -> CodeModeOS, MontyOSCallback -> CodeModeOSCallback, MontyMount -> CodeModeMount - CodeMode/CodeModeToolset param os= -> os_access= (mount unchanged) - internal resume()/feed_start() forwarding keeps Monty's literal os= kwarg Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(code_mode): stop shadowing the os module in the execution loop The OS/mount threading named its parameter `os`, shadowing the stdlib module inside the execution-loop helpers. Rename the variable to `os_access` (matching the public field) while keeping Monty's required `os=` keyword only at the resume/feed_start call sites. Also inline the single-use restriction-line helper into `_base_description`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(code_mode): make CodeMode config fields keyword-only The option list keeps growing; pin tools/max_retries as the only positional args and force os_access/mount (and future config) to be passed by name via a KW_ONLY sentinel, so adding options can't silently shift positional meaning. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(code_mode): make os_access/mount docs clear on first read Public docs should let a reader grasp the host-access surface without reverse-engineering it. Reframe the docstrings and README around when to reach for each primitive instead of what is switched off, drop the type-restating prose the annotations already carry, and lead with concrete tasks (share a dataset; inject just the secrets the agent needs). Tighten the os-access test sweep so each test asserts exactly its invariant: drop redundant negative description asserts (one note is interpolated, so the positive phrase alone proves selection), drop an assertion already owned by another test, and type the tmp_path fixtures. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(code_mode): clarify os_access callback return semantics The raw-callback example claimed non-allow-listed keys "stay hidden" by returning NOT_HANDLED. Verified against Monty: NOT_HANDLED *refuses* the call (raises in the sandbox -> model retry), it does not return None. A model probing for an optional secret would crash and burn retries. Distinguish the two return modes explicitly so users don't pick the wrong one: return a value (incl. None) to answer/hide, NOT_HANDLED to refuse a capability outright. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(code_mode): lock in os_access value-vs-NOT_HANDLED semantics Returning a value (including None) from an os_access callback answers the call -- a None reads back like an unset env var, so the sandbox keeps running. Returning NOT_HANDLED refuses the call, raising in the sandbox and surfacing as ModelRetry. These two paths are easy to confuse and silently regress, so pin both. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * code_mode: warn default run_code prompt about unavailable fs/env; drop Any from CodeModeOSCallback The PR's dynamic restriction note covered the mount and os-enabled states but not the default state, which still listed os/pathlib as importable without saying their I/O fails -- the exact wasted-retry case this change reduces. Extend the default note to state filesystem, env, and clock calls are unavailable without a mount or OS handler. Also type the public CodeModeOSCallback alias with object instead of Any: the harness only forwards the callback to Monty and never calls it, so object typechecks cleanly (pyright strict, 0 errors) and keeps Any out of the public API. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: David SF <david.sanchez@pydantic.dev> |
||
|
|
05ae737182 |
bumping pydantic ai (#280)
* bumping pydantic ai * Fix breakage from pydantic-ai 1.107: capability toolsets no longer cross runs All three failures came from pydantic-ai #5230 (on-demand capabilities): - SubAgents inherit_tools transplanted capability-contributed toolsets into sub-agent runs where the owning capability is not registered, which now fails CapabilityOwnedToolset's ownership resolution. Inherit only the parent's own toolsets; capability sharing is shared_capabilities' job. This also drops the delegate tool, replacing the name-based filter. - Tool search discovery moved from message-scanning to RunContext.discovered_tool_names, so the code_mode test now derives it via parse_discovered_tools like the agent graph does. - ToolDefinition gained capability_id, refreshing the logfire snapshot. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Test latest pydantic-ai in CI and set the floor to the verified minimum The lock-resolved test matrix never exercises pydantic-ai releases newer than the lock, and pydantic-ai's harness-compat job only covers harness code that exists when a pydantic-ai PR merges. Harness code merged after a core change (SubAgents, ManagedPrompt vs core #5230) was therefore never tested against it until a manual bump. The new test-latest job re-locks pydantic-ai-slim/pydantic-graph to the newest published versions on every run, so release breakage surfaces immediately. With latest covered by CI, the pydantic-ai-slim floor no longer needs to chase releases: set it to 1.105.0, the first release with capability ownership semantics (core #5230) that the harness now relies on, and verified against the full suite via the test-floor resolution. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Mirror pydantic-ai's lowest-versions matrix and unfreeze its resolution The test-floor job was not testing the floor: under the workflow-level UV_FROZEN=1, `uv sync --resolution lowest-direct` installs the locked (latest) versions, so the job duplicated the regular matrix while claiming lowest-version coverage. pydantic-ai's test-lowest-versions job sets UV_FROZEN=0 on the sync step for exactly this reason; do the same, and adopt the rest of its shape (full Python matrix, fail-fast off, job timeouts) so floor regressions that only reproduce on some Python version get caught. A genuinely-lowest resolution surfaced one over-coupled snapshot: the `logfire.metrics` span attribute only exists on logfire releases newer than the extra's 4.31.0 floor, so treat it as volatile in the ManagedPrompt span snapshots instead of raising the floor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Replace UV_FROZEN with UV_LOCKED in CI Frozen mode installs the lockfile blindly: lock drift goes unnoticed and resolution flags become silent no-ops, which is how the lowest-versions job ended up testing the locked versions. Locked mode validates the lockfile against pyproject.toml and fails loudly on any mismatch, so a job that cannot do what it claims turns red instead of green. The two jobs that deviate from the lock on purpose (lowest-versions resolution, latest pydantic-ai upgrade) opt out per step, each with a comment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Keep filtering the delegate tool when SubAgentToolset is used directly The capability-ownership filter only drops the delegate tool when the toolset arrives via the SubAgents capability, since that is the path that wraps it in CapabilityOwnedToolset. SubAgentToolset is publicly exported, and registered directly in Agent(toolsets=[...]) nothing wraps it, so inherit_tools=True forwarded delegate_task to sub-agents and re-enabled recursive delegation. Restore the name filter on top of the capability filter so both registration paths stay non-recursive. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Adapt to per-delegate run controls from main PR #283 made limits and call_counts required on SubAgentToolset, which broke the direct-construction test from the previous commit once CI merged the branch with main. The merge also stranded the CapabilityOwnedToolset import below the new SubAgentLimits dataclass; moved it back into the import block. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
e34c30ae54 |
feat(shell): control subprocess environment to stop secret inheritance (#284)
* feat(shell): control subprocess environment to stop secret inheritance Shell spawned every command with the inherited parent environment, so an agent running model-written commands in a sandbox could read the host's LLM API keys and tokens. This blocked replacing Pydanty's hand-rolled runner with Shell. Add two composable controls: `env` replaces inheritance entirely (a hard boundary -- the subprocess sees only what you pass), and `denied_env_patterns` strips glob-matched names from the base environment (mirrors `denied_commands`). When both are set, patterns also filter the explicit `env`. The default stays inherit-everything. Export `LLM_API_KEY_ENV_PATTERNS` for the common provider-credential denylist; opt-in, since stripping env silently would break agents that rely on inherited credentials. Closes #281 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(shell): tighten env-control wording after adversarial review Three-lens self-review (API breadth, correctness, docs/tests) surfaced doc-precision gaps and one missing end-to-end test; no correctness defects. - denied_env_patterns matches by glob, not exact match like denied_commands: correct the "mirrors denied_commands" claim to name the real parallel (the denied_* naming convention). - LLM_API_KEY_ENV_PATTERNS targets LLM credentials only and uses coarse prefixes (GOOGLE_* also strips GOOGLE_APPLICATION_CREDENTIALS); say so, and note it does not cover other host secrets like LOGFIRE/GitHub tokens. - env is enforced at spawn; clarify that with denied_env_patterns it is the resolved (filtered) env, and warn that stripping PATH/HOME breaks command resolution. - Add an end-to-end test proving env and denied_env_patterns compose at spawn. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: David SF <david.sanchez@pydantic.dev> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
34dcbf8352 |
feat(subagents): per-delegate run controls (usage limits, timeout, call budget, soft-failure) (#283)
* feat(subagents): per-delegate run controls via SubAgentLimits Delegations through SubAgents ran with no per-child budgets, so an orchestrator that needs one child capped at N requests, another bounded by a wall-clock deadline, or a per-run delegation cap had to keep a parallel hand-rolled delegation path (motivating consumer: Pydanty's run_delegate). Add a `limits: Mapping[str, SubAgentLimits]` knob. `SubAgentLimits` carries `usage_limits` (isolated child accounting so the budget is the child's own, reached softly rather than via a propagating UsageLimitExceeded), `timeout_seconds` (wall-clock cancel with a soft steering message), `max_calls` (per-run delegation budget, counts keyed by run_id and cleared in wrap_run), and `on_failure` (a single steering message that overrides the default soft text and flips child failures from a parent ModelRetry to a soft tool result). Absent names keep current behavior. Closes #282 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * review: validate limits keys, exclude run-state from eq, cover parallel budget Adversarial self-review follow-ups on the per-delegate controls: - SubAgents.__post_init__ raises when `limits` names a sub-agent absent from `agents`. A typo'd key would otherwise silently run that delegate unbudgeted, the exact failure the budgets exist to prevent. - `_call_counts` is marked compare=False so accumulating delegation counts during a run don't change SubAgents equality. - Add a test for two delegations issued in one parent step: the synchronous count still caps them at max_calls. Document that max_calls is scoped per run_id (so nested trees budget independently) and that a timed-out child's event stream is cut without a terminal event. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: David SF <david.sanchez@pydantic.dev> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
6f2aa11438 |
Add StepPersistence capability for step-event durability across delegates (#251)
* Add StepPersistence capability for step-event durability across delegates Supersedes PR #176 (SessionPersistence): orchestrators like pydanty need visible event trails for delegate runs that may time out before a "save full session after the run" hook can fire, and need to continue or fork a delegate's prior investigation without rediscovering context. A single after-run snapshot is too coarse for that use case. The capability now records (a) append-only StepEvents at every boundary (run/model-request/tool-call start, completion, failure), (b) a ContinuableSnapshot only when message history is provider-valid (every ToolCallPart has a matching ToolReturnPart / RetryPromptPart) — saved mid-run after CallToolsNode and at after_run, and (c) a ToolEffectRecord ledger so a run killed between before_tool_execute and after_tool_execute leaves an `unknown_after_crash`-style record rather than a falsely-continuable snapshot. Lineage metadata (parent_run_id, agent_name) ties delegate runs back to their orchestrator. `continue_run` / `fork_run` helpers load the latest continuable snapshot for a run. Backends: InMemoryStepStore (tests) and FileStepStore (JSONL events + JSON snapshots, with run_id path-traversal validation and anyio.to_thread for blocking I/O). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Address pydanty review + ergonomics: tighten correctness and identity model Correctness fixes from pydanty's PR review: - FileStepStore: snapshot filenames are now a per-run monotonic counter, not `ctx.run_step` — `run_step` resets each Agent.run, so re-using a `run_id` across calls would let an earlier run's higher step-index snapshot mask a later run's lower-step-index one. - StepStore.get_tool_effect now takes both `run_id` and `tool_call_id`. TestModel and other providers can reuse deterministic tool-call ids across runs; the previous unscoped lookup let one run's effect leak into another's record (including `started_at`). - is_provider_valid now rejects orphan, duplicate, and out-of-order tool returns — the old `set.discard` pattern silently accepted any return regardless of whether a matching call was open. Identity model: - `run_id` resolution: explicit > `{agent_name}-{8-char-hex}` > UUID. Materialised per Agent.run in `for_run`, so reusing one capability instance never silently merges runs. - `parent_run_id` auto-inferred via a module-level ContextVar set in `wrap_run`, so an orchestrator's tool that synchronously calls `delegate.run(...)` produces a delegate `RunRecord.parent_run_id` pointing at the orchestrator's `run_id` with zero threading. Explicit `parent_run_id=` still wins. - `conversation_id` propagated to `StepEvent` and `RunRecord`; `store.list_runs(conversation_id=..., parent_run_id=...)` supports filtering by either or both. Mirrors pydantic_ai's three-level identity (conversation -> run -> step) so "run 1, run 2, run 3" of one dialogue is queryable as a group via `conversation_id`. - `continue_from=` field dropped from the capability. Continuation is now only via `continue_run(store, run_id=...)` -> standard `Agent.run(message_history=...)`. One way to pass history into pydantic_ai, no parallel capability flag. README rewritten around the final API. New sections: three-level identity, run lineage with auto-inferred parent, inspecting a run tree, failure recovery. Tests: 168 total (up from 64), 100% branch coverage on the package. New coverage for the snapshot seq counter, cross-run tool-effect isolation, orphan/duplicate/out-of-order return rejection, ContextVar parent inference across nested agent.run, conversation_id propagation, and the agent_name-derived run_id default. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Address pydanty round-2 review: retry validity, list ordering, effect metadata Correctness: - is_provider_valid no longer rejects non-tool RetryPromptParts. Pydantic AI emits `RetryPromptPart(tool_name=None)` for output-validation failures and providers map those as plain user messages, not tool results. The previous check required every RetryPromptPart to resolve an open tool call, so a run with one output retry produced no final continuable snapshot despite being fully valid. - StepStore.list_runs now guarantees chronological (started_at ascending) ordering across both backends. FileStepStore was previously returning directory-name order (lexicographic), so the README's `[-1]` pattern for "latest run in conversation" could pick the older run when run ids did not sort by recency. - after_tool_execute and on_tool_execute_error preserve idempotency_key and effect_summary from the prior `started` record. Previously the terminal record was written without those fields, so any annotation the tool body wrote was lost on completion. - from_spec raises ValueError for unknown backends instead of silently falling back to in-memory storage. For a persistence capability, turning a typo into accidental non-durability is the wrong failure mode. API: - New annotate_tool_effect(store, ctx, *, idempotency_key=None, effect_summary=None) helper. Tool bodies that write external state call it to attach idempotency + effect metadata to the in-flight ToolEffectRecord without knowing the (run_id, tool_call_id) plumbing. Resolves run_id from a ContextVar set by wrap_run; reads tool_call_id / tool_name from RunContext. - ContextVar moved from `_capability.py` into a new `_context.py` module so the helper and the capability can share it without circular imports and without crossing the private-name barrier. Docs: README fixes a non-existent `list_runs(agent_name=None)` call, documents the chronological-ordering guarantee, and replaces the hand-wavy "populate fields on the ToolEffectRecord" line with a concrete `annotate_tool_effect` example. Tests: 178 total (was 168), 100% branch coverage on the package. Added coverage for non-tool retry acceptance, chronological list_runs on both backends, metadata preservation across completed/failed transitions, annotate_tool_effect under realistic agent.tool, and from_spec backend validation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(step_persistence): align FileStepStore docstring with seq-counter layout The class docstring still showed snapshots/<step_index>.json from the pre-fix layout, but both the README and _next_snapshot_seq document the monotonic counter. Bring the class docstring in line. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(step_persistence): fix broken README examples + clarify run_id sharing Pydanty round-3 review: - README continuation and lineage examples queried `list_runs(conversation_id=...)` on conversations the earlier `.run(...)` calls never set, so the examples crashed with IndexError on `[-1]`. Pass the conversation_id to the earlier calls so the lookup actually works. - The capability docstring claimed reusing a `StepPersistence` instance across `Agent.run` calls does NOT share the id. That is true only for the auto-derived (`agent_name`-prefixed or `ctx.run_id`) cases — an explicit `run_id=` is shared across every `.run()` by design, since that is the orchestrator pattern where the caller owns one logical identity across turns. Rewrite the resolution-order docs to spell out which cases share and which don't, and when to pick each. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(step_persistence): align run_id semantics with pydantic_ai (per-call, not shared) Pydanty round-4 review: the prior round documented explicit `run_id` as shared across `.run()` calls on one capability instance — that framing caused real correctness gaps. The `ToolEffectRecord` ledger is keyed by `(run_id, tool_call_id)` and providers reuse deterministic tool-call ids (e.g. `TestModel` emits `pyd_ai_tool_call_id__{name}`), so a second `.run()` overwrites the first's effect record under the same key — the `unknown_after_crash` signal from turn 1 disappears when turn 2 lands. Realign: - `run_id` is per-`Agent.run`, matching `pydantic_ai.RunContext.run_id`. - For multi-turn logical grouping, use `conversation_id=` on `Agent.run(...)` — that is the pyai-native primitive. The orchestrator pattern is `conversation_id='orch'` with each turn auto-deriving its own `run_id`. - Explicit `run_id=` remains supported but is documented as single-shot (testing, replay, debugging). Reusing it across calls is a caller contract violation, not an implementation feature. Code is unchanged — the implementation was already correct under the right contract. Only the docs were misleading. Tests: - `TestRunIdIsPerCall::test_multi_turn_orchestrator_uses_conversation_id` exercises the recommended pattern: three turns sharing a `conversation_id`, three distinct auto-derived `run_id`s, all queryable as a group. - `TestRunIdIsPerCall::test_explicit_run_id_reuse_collides_ledger` locks down the misuse contract: reusing one explicit `run_id` across two `.run()` calls produces colliding effect records under the `(run_id, tool_call_id)` key. The behavior is documented; the test exists so a future refactor cannot silently change it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(step_persistence): explain why for_run uses a local ContextVar Pyai-aligned review flagged this as a P3 explainer: pydantic_ai already has three single-slot cross-run signals (RUN_ID_BAGGAGE_KEY, ctx.run_id, _CURRENT_RUN_CONTEXT). All three get overwritten by the inner Instrumentation.wrap_run before any nested capability can see the parent identity. A separate harness-local ContextVar, snapshotted before our own wrap_run rebinds it, is the only correct mechanism today. Spell this out so the next reader doesn't try to 'simplify' it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(step_persistence): enforce explicit-run_id reuse with ValueError Pydanty round-5 review accepted the docs-only contract but flagged that "documented but not enforced" is a soft spot. Enforce it: `before_run` calls `store.get_run(run_id=...)` when the user supplied an explicit `run_id`, and raises `ValueError` if a record with that id already exists. The auto-derived cases cannot trigger this check (each call materialises a fresh id in `for_run`). The check is one extra store read per Agent.run when an explicit run_id is set, only. The error message points the caller at `conversation_id` for multi-turn grouping. Test renamed from `test_explicit_run_id_reuse_collides_ledger` to `test_explicit_run_id_reuse_raises` — asserts the second `.run()` raises and the first run's records survive untouched. README + capability docstring updated: the misuse path is now "raises" not "caller's contract." Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: gitignore branch-context skill artifacts + AGENTS.local.md Two patterns that match the existing CLAUDE.local.md ignore convention: - AGENTS.local.md — canonical local-instructions file (CLAUDE.local.md is symlinked to it where the worktree follows the same AGENTS.md/CLAUDE.md symlink pattern). - .agents/skills/branch-context/ — per-worktree decisions log (`pr-decisions.md`) and the skill's local SKILL.md. Pattern lifted from `~/pydantic/ai/base/.claude/skills/branch-context/` where pyai uses an identical setup. Neither is intended to land in PRs — they record cross-iteration design calls so future AICA sessions in this worktree don't silently undo them. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(step_persistence): add SqliteStepStore + MediaStore externalization Adds a new `pydantic_ai_harness.media` package (MediaStore protocol + DiskMediaStore / SqliteMediaStore / S3MediaStore) and wires it into the file/sqlite step-persistence backends so large BinaryContent payloads get externalized out of snapshot JSON / table rows by default. Defaults are zero-config: FileStepStore writes blobs under `<root>/media/<sha256>.bin`; SqliteStepStore writes them to a sibling `media` table in the same DB. Threshold is 64 KiB and URI scheme is `media+sha256://<hex>` so blobs are content-addressed across stores. Pass `media_store=None` to keep bytes inline, or a custom `MediaStore` to redirect (e.g. `S3MediaStore` for R2 / AWS / MinIO). S3MediaStore handrolls SigV4 over httpx to avoid a botocore/boto3 dependency. Verified working against Cloudflare R2. `StepPersistence.from_spec(backend='sqlite', database=...)` now resolves. 180 → 261 tests, 100% branch coverage maintained. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(media): VCR cassettes for S3MediaStore against R2 Adds replay-driven tests under `tests/media/test_s3_cassettes.py` that exercise `S3MediaStore.put/get/exists` against pre-recorded Cloudflare R2 responses. CI runs them without any S3 creds via the committed cassettes under `tests/media/cassettes/`. Sanitisation policy: - `before_record_request`/`before_record_response` swap the real R2 account-id subdomain and bucket name for fixed placeholders (`account.r2.cloudflarestorage.com`, `harness-test-bucket`) - `Authorization` and `x-amz-date` filtered to `REDACTED` - CF-RAY, x-amz-version-id, x-amz-checksum-*, x-amz-request-id headers dropped (none load-bearing for tests; some carry identifying info) - Non-2xx response bodies blanked (R2's gzipped XML error envelope leaks the bucket name; our code only checks status code) The `s3_credentials` fixture uses `os.environ.get(NAME, PLACEHOLDER)` per field, so real R2 creds are used when recording locally with `.env` loaded, and the placeholder constants match the scrubbed cassettes during replay. Because the placeholders are fixed, any scrubber miss during a future re-record shows up as a replay URL mismatch — built-in canary against credential / private-data leakage in committed cassettes. Adds `pytest-recording` (pulls `vcrpy`) to the dev deps. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(media): public_url resolver — protocol method + per-store callable Adds `MediaStore.public_url(uri) -> str | None` plus a `public_url=` constructor parameter on every concrete store. The parameter accepts a sync or async callable; the store auto-detects and awaits if needed. This is the bottom-layer primitive for the forthcoming `MediaExternalizer` capability — that capability will call `store.public_url(...)` per externalized blob and swap `BinaryContent` for `ImageUrl` / `AudioUrl` parts before the model sees the message. The callable shape covers both static URLs (public bucket / CDN — use `make_static_public_url` helper) and dynamic URLs (presigned, per-request signing — pass any async callable with TTL captured in its closure). Why a callable rather than a static config: a public bucket's URL host is not derivable from the bucket creds (R2 public buckets use `pub-<hash>.r2.dev`, AWS public buckets use a different scheme than the path-style endpoint we sign for). The URL is always user-supplied information, so a callable is the right primitive — same shape for the static and presigned cases, and `get` stays untouched (it serves the harness's internal byte fetch, not the model's external HTTP fetch). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(media): introduce MediaContext + key_strategy for extensible operations Adds `MediaContext` (frozen, kw-only dataclass with `media_type`, `filename`, `metadata`) and threads it through every `MediaStore` method and both user callables (`PublicUrlResolver`, `KeyStrategy`). New context fields can be added non-breakingly; existing call sites and resolvers keep working. Also adds: - `KeyStrategy = Callable[[str, MediaContext], str]` for per-store layout control. Default `default_key_strategy` produces `<sha256>.bin`. Disk store validates the result against `..` traversal. - `metadata` persistence on `SqliteMediaStore` (new JSON column) and `S3MediaStore` (signed `x-amz-meta-*` headers, ASCII key validation). Disk store explicitly does NOT persist metadata in v1 — sidecar / xattr options each have load-bearing drawbacks; we ship nothing rather than a half-true persistence promise. - `make_static_public_url(...)` updated to the new `(uri, ctx)` signature. The shift is motivated by the same principle as pydantic_ai's `RunContext`: extension via fields on a context bag rather than via breaking signature changes. Every new requirement (TTL hints for presigned URLs, audit ids, response-header overrides, etc.) becomes a field addition, not an API revision. Cassettes from the previous commit replay unchanged — match-on does not include the signed headers and the request URLs are stable. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(media): close metadata round-trip; drop vestigial sqlite key_strategy Adds `MediaStore.get_metadata(uri) -> Mapping[str, str]` to the protocol and implements it on all three concrete stores: - `DiskMediaStore`: writes a sidecar `<resolved>.meta.json` alongside the blob on put (atomic tmp + rename), reads it back on `get_metadata`. Returns `{}` when no metadata was supplied. v1 had documented this as a deliberate gap — sidecar JSON is straightforward and the xattr / ADS drawbacks don't apply. - `SqliteMediaStore`: `SELECT metadata FROM <table> WHERE sha256=?` + `json.loads`. Raises `FileNotFoundError` for unknown URIs. - `S3MediaStore`: HEAD + collects `x-amz-meta-*` response headers, strips the prefix. Reuses the existing 404 / non-2xx error shape. Drops `key_strategy=` from `SqliteMediaStore`. The digest is the primary key by content-addressing construction — a user-chosen key would either break dedup or be a no-op. Kept on Disk + S3 where bucket / directory layout is a real concern. README + branch-context entries updated to reflect: all three stores round-trip metadata; key_strategy is Disk + S3 only. Coverage stays at 100% branch. * fix(media): reject absolute paths from DiskMediaStore key_strategy `Path(root) / absolute_path` returns `absolute_path` — the root is silently discarded — so a custom `key_strategy` returning `/etc/passwd` (or similar) escapes the store directory even though the previous check only blocked `..`. Tighten the validator to reject both shapes. Caught by pydanty during its #251 integration review. * fix(step_persistence): dedupe terminal snapshot; document sqlite thread-affinity The terminal CallToolsNode already saves the final provider-valid snapshot with the correct step_index. after_run was re-saving the same tail stamped with step_index=0 (ctx.run_step is reset by then), so latest_snapshot reported a misleading step and every run wrote a duplicate. Track whether a node snapshot was taken via a task-local ContextVar and make after_run a fallback that only fires when the run reached no provider-valid boundary. Also document that a caller-owned sqlite connection= must set check_same_thread=False (store SQL runs on anyio worker threads), on both SqliteStepStore and SqliteMediaStore, and correct the WAL-on-every-connection claim. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(media): SigV4 wire path matches signed path; walker preserves unknown fields S3MediaStore signed `_canonical_uri(path)` (each segment percent-encoded) but sent the raw path, letting httpx apply looser encoding. A custom key_prefix / key_strategy emitting reserved chars (`@`, `(`, `=`, ...) diverged from the signed path -> SignatureDoesNotMatch. Send the canonical bytes via httpx `raw_path` so signer and sender agree. Default `<hex>.bin` keys are unaffected. The externalize/restore walker hardcoded the BinaryContent key set, silently dropping any field pydantic_ai adds upstream. Copy the node and swap only `data` <-> marker keys so unknown fields round-trip. Adds tests for reserved-char path agreement, unknown-field preservation, and restore over a pruned blob. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: gate step_persistence + media behind experimental namespace Move both packages under `pydantic_ai_harness.experimental`, matching the convention introduced for compaction/planning/subagents (#191): the old `pydantic_ai_harness.step_persistence` / `.media` paths and the top-level re-exports are gone, so the only import path is now from pydantic_ai_harness.experimental.step_persistence import StepPersistence from pydantic_ai_harness.experimental.media import S3MediaStore Both package __init__s call `warn_experimental(...)`, so importing either emits a `HarnessExperimentalWarning` (silenced category-wide by one filter). This keeps us from committing to a public surface before the capability has real usage. README gains the standard experimental banner; warning tests cover both new packages. 100% branch coverage retained. * docs(media): fix metadata/key_strategy drift, convert em-dashes Subagent review of the experimental move surfaced doc drift: - README "Persistence by store" implied `get_metadata` returns `media_type`; it returns only the user `metadata` mapping. Reworded. - `KeyStrategy` docstring still listed `SqliteMediaStore` / "DB primary key" as a user; sqlite has no `key_strategy`. Dropped it. - README understated `DiskMediaStore` traversal protection (it rejects absolute paths as well as `..`). - README had a stale paragraph that read as `key_strategy` but described `public_url`, and omitted S3. Rewritten to name `public_url` and all three stores. - README `MediaContext` method list now includes `get_metadata`. - Converted em-dashes to `--` across the step_persistence + media trees per the writing-style rule (#270). Other experimental packages' pre-existing em-dashes are left for a separate sweep. No code behavior change; 100% branch coverage retained. * docs(experimental): add warning banner to planning/subagents, finish em-dash sweep Uniformity pass across the experimental packages: - Add the standard `[!WARNING]` experimental banner to planning/README.md and subagents/README.md (compaction and step_persistence already had it; these two lacked it). - Convert remaining em-dashes to `--` in the compaction package and the shared `_warn.py`, per the writing-style rule (#270). The whole experimental source tree is now em-dash-free. Docs only; 100% branch coverage retained. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: David SF <david.sanchez@pydantic.dev> |
||
|
|
d347d08e3c |
feat(subagents): add SubAgents capability for delegating to child agents (#267)
SubAgents exposes a single delegate_task(agent_name, task) tool that runs a named child agent in a fresh, isolated run, with the available agents listed in the system prompt as a cache-stable instruction. - deps are forwarded to every sub-agent; usage is shared by default (forward_usage) so usage limits apply across the whole agent tree - inherit_tools (opt-in) exposes the parent's tools to sub-agents, with the delegate tool filtered out so they can't recurse into delegation - shared_capabilities are applied to every sub-agent run - event_stream_handler is forwarded to every sub-agent run so sub-agent events surface to the caller - soft sub-agent failures (ModelRetry / UnexpectedModelBehavior) convert into a parent ModelRetry it can react to; hard limits propagate Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
ae371bbdaf |
docs: ban em-dashes and codify writing style (#270)
Add a Writing style section to AGENTS.md so docs, comments, and PR text read as human-written: no em-dashes (use `--`), no marketing superlatives or editorializing adjectives, sparing bold, no decorative Unicode. Bring the tree into compliance by converting every em-dash to `--` across 16 files. All occurrences are in prose, comments, docstrings, and tool descriptions. The only runtime effect is that model-facing tool descriptions now use `--` (punctuation only, no semantic change); no test snapshots depend on the character. The one literal em-dash left is in AGENTS.md, where it names the banned character. https://claude.ai/code/session_01LQ9NTr8q95A99UnVcq6Nzf Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
3d3247110c |
feat(planning): add Planning capability with cache-safe plan reminders (#266)
Planning gives the model a single write_plan(items) tool that owns the plan via whole-plan replacement (no fragile indices). The current plan is surfaced back as an ephemeral reminder appended to the request tail in wrap_model_request, behind a CachePoint: - the reminder reaches the model but is never written to the durable message history, so no stale copies accumulate - the CachePoint sits before the reminder, so the cached prefix (tools + system + real conversation) stays byte-identical across turns and only the small reminder falls outside the cache Static usage guidance goes in the system prompt (cache-stable); the mutable plan is never injected there. Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Douwe MaanandGitHub
|
442e9ff493 |
Add experimental compaction menu under pydantic_ai_harness.experimental (#191)
Compaction capabilities (SlidingWindow, ClearToolResults, DeduplicateFileReads, SummarizingCompaction, TieredCompaction, LimitWarner) shipped as experimental, with a HarnessExperimentalWarning + single category-wide silence filter. 100% coverage; ruff + pyright strict. |
||
|
|
bee3f02ca7 |
feat(logfire): add ManagedPrompt capability (#257)
* feat(logfire): add ManagedPrompt capability Back an agent's instructions with a Logfire-managed prompt, resolved once per run inside wrap_run so the selected label/version are attached as baggage to every child span. Lets prompts be iterated, versioned, and rolled out from the Logfire UI without redeploying, with a code default as the offline safety net. Lives in pydantic_ai_harness.logfire; adds a 'logfire' optional dependency extra. The test directory is named logfire_variables to avoid shadowing the third-party logfire package under pyright's tests execution-environment root. * fix: satisfy codespell on managed prompt comment * docs(logfire): make ManagedPrompt examples runnable instead of skipped The examples fall back to their code default with no Logfire backend and run clean under a mocked model, matching the main README's unmarked quick-start. * refactor(logfire): rename ManagedPrompt `prompt` param to `name` Logfire calls a managed prompt by its name, so `ManagedPrompt(name=...)` reads better than `ManagedPrompt(prompt=...)`. Also clarifies the `label` and `targeting_key` docstrings per review. * review fixes: cache framing, public imports, [spec] in logfire extra - README: prompt-cache trade-off section, notes on once-per-run resolution, label+version baggage semantics, and the targeting_context outer setter. Callout pointing at pydantic-ai#5107 so users know a first-party `Managed` capability is in flight. - Switch to the public `logfire.variables` import paths (was reaching into `logfire.variables.variable` / `.abstract`). - Pull `pydantic-ai-slim[spec]` from the `logfire` extra so `render_template=True` works without a separate install. - Warn when `logfire_instance` is passed alongside a prebuilt `Variable` (silently ignored before). - Scrub `code.lineno` from span snapshots and expand the volatile-attributes comment so the snapshot doesn't rot on line shifts. - Wrap the provider-backed test in a context manager that restores the module's baseline Logfire config in `finally`, so future tests don't inherit a provider. --------- Co-authored-by: David Sanchez <64162682+dsfaccini@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Bill Easton <williamseaston@gmail.com> Co-authored-by: David SF <david.sanchez@pydantic.dev> |
||
|
|
e1654ccfe2 |
feat: add FileSystem and Shell capabilities (#260)
* feat: add FileSystem and Shell capabilities with exhaustive testing - FileSystemToolset: 8 tools (read, write, edit, list, search, find, mkdir, info) with path-traversal prevention, allow/deny patterns, optimistic concurrency - ShellToolset: 1 tool (run_command) with command validation, timeout handling, and async subprocess execution via anyio --- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: David Sanchez <64162682+dsfaccini@users.noreply.github.com> |
||
|
gordonblackadderandGitHub
|
ad1a2014ef |
add SKILL for harness, including a reference to code mode (#228)
Add a harness agent skill Documents Code Mode for Pydantic AI, following the agentskills.io spec and matching the existing building-pydantic-ai-agents skill. - Skill directory matches the name field (spec requirement) - Deps aligned with pyproject.toml: pydantic-ai-slim>=1.95.1 plus the code-mode install extra - MCP quick-start passes native=False so CodeMode wraps the tools - Documents the Monty timing-primitive restriction; adds a Code Mode API section - Examples linted via pytest-examples (model-free ones executed) so they can't go stale |
||
|
Bill EastonandGitHub
|
d4be271de9 |
Revert "feat: add FileSystem and Shell capabilities with exhaustive testing (…" (#259)
This reverts commit
|
||
|
Bill EastonandGitHub
|
7fd372d9fd |
feat: add FileSystem and Shell capabilities with exhaustive testing (#258)
- FileSystemToolset: 8 tools (read, write, edit, list, search, find, mkdir, info) with path-traversal prevention, allow/deny patterns, optimistic concurrency - ShellToolset: 1 tool (run_command) with command validation, timeout handling, and async subprocess execution via anyio - 152 tests passing on asyncio backend, 100% branch coverage - Mutation testing: 524/584 killed (89.7%), 60 equivalent mutants documented - All survivors proven equivalent (trampoline defaults, encoding case-insensitivity, unreachable except blocks, dead branches, name=None fallback behavior) |