25 Commits

Include renames

Author	SHA1	Message	Date
4 people	5ab8dd3d81	feat: input and output guardrails (block, redact, retry) (#249) ... * feat: add input and output guardrails * chore: use single backticks * fix: run InputGuard only on the first model request * fix: replace list[Any] with Sequence[ModelMessage] * fix: pass raw output to OutputGuard, not str(result.output) * refactor: organize tests into TestCapabilityName classes * fix: drain cancelled tasks in InputGuard parallel finally * fix: re-raise task exceptions via await instead of .exception() * refactor: consolidate parallel cancel/drain into single finally * feat: support callable block_message; collapse InputGuard hooks block_message now accepts a callable so the refusal text can reflect the prompt/output that tripped the guard, rather than being frozen at construction time. InputGuard's sequential path moves from before_model_request into wrap_model_request, so a single hook covers both sequential and parallel modes instead of two hooks each branching on `parallel`. Tests move to tests/guardrails/ to match the tests/<capability>/ layout. * refactor: guards return bool | GuardResult; drop block_message A guard now returns either a bare bool or a GuardResult carrying a refusal message, replacing the separate block_message constructor field. The message is produced when the guard decides, so it can reflect the guard's own reasoning rather than a string frozen at construction time. Guards (and the GuardResult path) may optionally take a RunContext as a first parameter, detected from the signature like pydantic-ai's output validators, so deps- and history-aware guards are possible without closing over globals. Prompt/output-only guards are unchanged. * feat: guard outcomes — allow/block/replace/retry with OTel spans A guard now reports one of four outcomes via GuardResult classmethods (bool shorthand still works): allow, block, replace, retry. - replace lets a guard redact rather than refuse — InputGuard rewrites the prompt sent to the model, OutputGuard substitutes the output. - retry lets OutputGuard send a bad output back to the model; OutputGuard moves from after_run to after_output_process so it can raise ModelRetry and return a modified output. - replace and block emit spans on the run tracer so a redaction or refusal is visible in Logfire; redacted content is included only when RunContext.trace_include_content is set. InputGuard replace requires sequential mode and retry is rejected as a usage error, since neither is meaningful for input. * test: cover guardrail test helpers for the 100% gate The coverage gate measures test files too; the _prompt_text helper had unreached branches. Drop it and assert on message parts inline. * docs: scope streaming behavior; add streaming tests A pydantic-ai-correctness review surfaced two streaming points. Verified both empirically: - InputGuard(parallel=True) works under run_stream() — no deadlock. - OutputGuard GuardResult.retry() is unsupported under run_stream(): pydantic-ai does not retry output while streaming, so a retry verdict surfaces as UnexpectedModelBehavior. Document the retry limitation and that OutputGuard screens only the final output (partial chunks reach the caller first while streaming). Note that input redaction also rewrites persisted history and targets text prompts. Add streaming tests for both guards to lock the behavior in. * test: make the parallel-streaming test a real regression guard The test proving InputGuard(parallel=True) does not deadlock under run_stream() had no timeout — a reintroduced deadlock would hang CI instead of failing. Wrap it in asyncio.wait_for and document the reviewed concern it guards against. * feat: guardrail capability ordering + GuardResult hardening Address @adtyavrdhn's review on #249. - `InputGuard.get_ordering()` → `position='innermost'` so any message-morphing capability runs first and the guard sees the final prompt the model will receive. - `OutputGuard.get_ordering()` → `position='outermost', wrapped_by= [Instrumentation]` so the guard's block/redact spans are always captured by an enclosing `Instrumentation` span regardless of user list order. - `GuardResult` is `frozen=True, kw_only=True` with a `__post_init__` that rejects field combinations the four-outcome contract does not allow (e.g. `replace` without a replacement). `block` with no message stays valid — the default kicks in at the use site. - `_run_guard` and `after_output_process` dispatch via `match action:` with `assert_never` exhaustiveness guards. - `_trace_block` gates the refusal `message` attribute behind `ctx.trace_include_content`, matching `_trace_redaction` — the message can quote sensitive content from the guarded value. - New tests: ordering declarations, `__post_init__` validation, frozen enforcement, outer-cancellation no-leak regression guard (which confirms `asyncio.shield` around the cleanup `gather` is not needed — the outer cancel is already consumed by `asyncio.wait`). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump pydantic-ai-slim to 2.4.0 in lock CI resolves pydantic packages to newest (uv.toml marks them exclude-newer=false); the merged lock was stale at 2.1.0 so uv sync --locked failed. Regenerate to 2.4.0. * test(code_mode): drop removed ToolSearchMatch.description field pydantic-ai 2.4.0 narrowed ToolSearchMatch to carry only `name`; the full ToolDefinition (with description) is surfaced separately. Update the test fixtures to the new shape. --------- Co-authored-by: Kacper Włodarczyk <kacperwlodarczyk@protonmail.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-07-10 14:21:23 -05:00
3 people	545a703227	feat: Adds Dynamic Workflows (#273) ... * feat(dynamic-workflow): orchestrate sub-agents from a model-written script Turn-by-turn sub-agent delegation makes every composition step (chain, vote, loop) a separate model turn and pushes each intermediate result back through the orchestrator's context. DynamicWorkflow is Code Mode with sub-agents as the callables: the model writes one Monty-sandboxed script that composes the sub-agents with ordinary control flow, and only the final result returns to the model. Extracts the shared Monty execution loop into _monty_exec, reused by code_mode. The pydantic-ai bump the capability needs (defer_loading, ctx.enqueue) lands separately, per the repo policy on dependency changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(deps): bump pydantic-ai 1.95.1 -> 1.105.0 DynamicWorkflow needs defer_loading (>=1.97) and ctx.enqueue (1.105) for on-demand loading and cache-stable runtime sub-agent reveal. The lock exempts the pydantic-ai family from its exclude-newer cutoff to pull the newer release. Collateral: xfail a code_mode ToolSearch test that relied on the old discovered-tool tracking, and regenerate the managed_prompt snapshot for the new tool-definition fields. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(examples): tidy example docs and keep them fully linted Drop the blanket 'examples/**' pydocstyle ignore — it masked a single missing docstring. Add the one-line docstring to fork_and_resume's main() instead, so examples stay under the same docstring rules as library code. Also drop the redundant "Runs / needs a key" column from the examples README. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(dynamic-workflow): render catalog via core FunctionSignature, reject keyword names Addresses PR #273 review. Render the sub-agent catalog through pydantic_ai's FunctionSignature (the same renderer code_mode uses) instead of a hand-rolled f-string. This is the real dedup with code_mode and fixes two latent bugs: the signature now forces keyword-only `task` (`async def name(*, task: str)`) to match dispatch, which reads kwargs['task']; and a description containing a newline or quote now renders as a safe docstring block instead of breaking the f-string. Reject sub-agent names that are Python keywords. `'class'.isidentifier()` is True, so a keyword name passed validation but the model could never call it (`await class(...)` is a syntax error) — a silently-uncallable agent. Validate non-keyword identifiers at construction, via a shared in-file helper. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(dynamic-workflow): harden public API ahead of release Adversarial review of the dynamic-workflow surface surfaced contracts that would be painful to change once released; fix them while cheap: - Correct the "usage_limits bounds the tree" claim — the parent's limit is never forwarded to sub-agents. Add `sub_agent_usage_limits` so a per-sub-agent limit plus `max_agent_calls` give a real tree-wide token ceiling, and document what `forward_usage` actually does. - Own `WorkflowResourceLimits` instead of leaking pydantic_monty's `ResourceLimits` onto the public field; add an explicit `'unlimited'` sentinel and merge a partial dict onto the backstop (no silent drop). - Narrow `agents` to `list` so the append-to-reveal contract is enforced by the type rather than explained in prose. - Drop the `description` -> `agent.name` fallback that rendered a useless self-referential docstring to the model. - Teach the model-facing tool description what the README already knew: keyword-only `task`, no `return_exceptions`/abort-on-raise, dict (not attribute) access for structured output, and the print envelope. Also refresh the runnable examples (audit + migrate) and keep the README and tests in sync. lint/typecheck clean, 100% branch coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * removing time limit from monty * docs(examples): consolidate into one verified DynamicWorkflow showcase Replace the overlapping audit and migrate examples with a single example that exercises the full capability range — parallel fan-out, read/write confinement, an adversarial review, a feedback loop, typed fan-in, and a Logfire trace — so a reader sees what the capability is for without wading through two near-identical fan-out demos. Fixes issues an adversarial review surfaced: the retry loop now threads reviewer issues back to the migrator (so it can actually converge), the synthesizer's typed report survives to the caller via the orchestrator's output_type, and several claims (budget ceiling, confinement, run command portability) are corrected to match the code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(deps): raise pydantic-ai-slim floor to 1.101.0 for RunContext.enqueue The dynamic_workflow reveal path calls ctx.enqueue to announce a sub-agent added mid-run, but RunContext.enqueue did not exist until pydantic-ai 1.101.0 (verified: absent in 1.100.0, present in 1.101.0). The declared floor was still >=1.95.1, so the test-floor job (--resolution lowest-direct) and any install at the floor would resolve a RunContext without enqueue and raise AttributeError when revealing an agent instead of announcing it. Bump the runtime dep, the logfire extra, and the dev group to >=1.101.0, and update the README requirement line to match. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(dynamic-workflow): move capability under experimental DynamicWorkflow ships as experimental so its API can change without a deprecation period, like the compaction menu. Importing it now emits a HarnessExperimentalWarning, silenceable category-wide. _monty_exec stays at the package root: it is shared with code_mode, which is stable, so it must not move into experimental. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(dynamic-workflow): correct stdlib import guidance for the sandbox The tool description and README presented `asyncio, math, json, re, typing` as the complete set of importable modules. In Monty, `datetime`/`os`/`sys` also import, but they are inert in DynamicWorkflow (no os/path hooks are registered, so their calls resolve to NameError). Reword to a curated, non-exhaustive list so the model-facing text is accurate without steering toward the stubbed dead ends. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(dynamic-workflow): harden sandbox boundary and reveal/config validation Adversarial review of the run_workflow path surfaced several silent-failure and crash modes, all reachable from model-authored scripts or host config: - A model script awaiting the same sub-agent call twice in one asyncio.gather made the Monty VM raise pyo3_runtime.PanicException (a BaseException), which escaped both Monty error handlers and tore down the whole agent run. Catch it by name and convert to a retry; non-panic BaseExceptions still propagate. - Sub-agent calls silently dropped extra keyword args and accepted a non-string task (a dict/list was smeared into message parts), running on wrong input with no signal. Reject both at dispatch, before the budget is touched. - A runtime reveal whose name was invalid or collided with an existing agent was silently dropped, keeping the old agent while the host believed it had swapped one in. Distinguish idempotent re-reveal (object identity) from a real conflict, warn once on the latter, never swap silently. - max_agent_calls < 1 produced a dead-on-arrival tool with a misleading "conclude with results gathered" message; a typo'd resource_limits key was silently ignored, disabling the only guard against a pure-CPU loop. Both now raise UserError at construction. Also pins the headline guarantee with a concurrent fan-out test asserting max_agent_calls admits exactly the budget under asyncio.gather (previously only covered sequentially). _toolset.py stays at 100% line+branch coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: await cancelled dispatch tasks and survive bad mid-run reveals Cancelling a Monty execution previously fired task.cancel() and returned immediately, leaving in-flight tool calls and sub-agent runs unwinding in the background — they could mutate the shared usage accumulator after the tool call had already reported its result, and teardown errors were never retrieved. The executor now awaits the cancelled tasks so nothing outlives the call that spawned it. This changes code_mode's cancellation timing too (flagged in the PR description): teardown now waits for in-flight calls to unwind instead of abandoning them. Also: create the sequential dispatch coroutine after the pending-task barrier so cancellation at the barrier cannot leak a never-awaited coroutine, and make `for_run` index the agent list leniently — an invalid WorkflowAgent appended mid-run (which `_reveal_pending` deliberately tolerates) no longer hard-fails every subsequent run via `replace`'s re-validation. A new test pins that host-raised exceptions cannot be caught inside the sandbox: the budget-exhausted terminal result relies on it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(dynamic-workflow): document experimental status and actual return shapes The README never said the import warns or how to silence it, omitted the print()/None result shaping the tool actually performs, and the API block hid `description` and the WorkflowResourceLimits type. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(dynamic-workflow): cover deps forwarding, message isolation, and shared-counter limits An adversarial tests-vs-spec pass found three documented behaviors with no assertion behind them: parent deps reaching sub-agents, sub-agents never seeing the parent conversation, and sub_agent_usage_limits checked against the shared counter under forward_usage=True. Dropping any of them would have passed the suite. Also pins the documented backstop values and the served tool's max_retries, asserts the reveal announcement names the actual tool, and fixes two comments that described a duration backstop / a dataclasses.replace that don't exist. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(dynamic-workflow): record the core usage_limits TOCTOU race behind max_agent_calls The exact host-side counter exists because core's limit enforcement races under concurrent fan-out and RunContext can't forward the parent's limits; pin that context (and the upstream TODO) where the discipline lives so it isn't lost once the PR ships. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(examples): replace the generated example with a feature-led README Review feedback on #273: the example will be rewritten by hand, and the README should lead with the Anthropic dynamic-workflows post and the feature itself rather than narrating one example run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(dynamic-workflow): add a dedicated install extra Installing DynamicWorkflow's sandbox dependency via the code-mode extra conflated two capabilities; each now names its own extra (both resolve to pydantic-monty). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(code_mode): convert sandbox VM panics to retries A model-written script that awaits the same call twice in one asyncio.gather panics the Monty VM; pyo3 surfaces that as a BaseException that nothing caught, tearing down the whole agent run. DynamicWorkflow already guarded this; the guard now lives in _monty_exec and code_mode applies it too, dropping its REPL since the aborted VM's accumulated state can't be trusted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(dynamic-workflow): remove duplication that could drift Single-source the long-form field docs on the capability (the toolset docstrings now point there), derive the resource-limit key set from the TypedDict instead of a parallel frozenset, and freeze the rendered tool description itself rather than a catalog dict re-rendered every step -- making the cache-stability claim structural. Drops the for_run_step override that duplicated the base default, plus redundant tests and a stateless test fixture. No behavior or public API change; coverage stays at 100%. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: narrow the shared executor seam and drop dead type unions Name sanitization is a code_mode concern, so the mapping moves into its dispatch closure and MontyExecutor stops knowing about it. The empty pending_call_ids guard fell through to an identical resume call, so it goes. Concrete return types on get_toolset/for_run let tests drop four isinstance asserts that existed only to narrow types pyright can now see directly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: fix a comment stating a false invariant; unshadow a closure param _by_name is mutated during a run (reveals fold into it each step); the stable-registry invariant only holds while a script executes, which is what the executor relies on. The dispatch closure's name param shadowed call_tool's own, a future-edit hazard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(dynamic-workflow): restructure README for faster comprehension Lead with a minimal runnable snippet and state the context-saving idea once instead of repeating it across Why/What/blockquote. Move Installation above the full example. Turn the sub_agent_usage_limits paragraph into a forward_usage True-vs-False table so the N*T hard ceiling and the best-effort concurrency caveat are scannable. No technical claims changed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(dynamic-workflow): address CodeRabbit review - Fix broken relative link to Code Mode README: the capability lives two levels deep under experimental/, so `../code_mode/` resolved to a non-existent experimental/code_mode/. Use `../../code_mode/`. - Add missing `-> None` return types to all four test_warnings methods (the bot flagged one; the whole file was missing them). - Narrow `_workflow_result`'s unconstrained passthrough from `Any` to `object`. Left the other flagged `Any`s in place: the `call` override and `dispatch` must match `AbstractToolset.call_tool` / `_monty_exec.DispatchFn`, and `AbstractAgent[..., Any]` carries a heterogeneous sub-agent output type that `object` would reject on assignment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(dynamic-workflow): harden error surfaces, render output schemas, add reveal() API Findings from an end-to-end adversarial review (PR #273 branch), fixes authored with codex exec: - Serialize sub-agent output inside the dispatch error wrapper: a non-serializable output leaked host class paths into the retry text and failed identically on every retry. - Render each sub-agent's output schema in the catalog and reveal announcements (TypedDict-style, with cross-catalog type-name conflict handling). The model previously saw `-> Any` and had to be told the fields in prose. - Add DynamicWorkflow.reveal() as the supported runtime reveal API; list append remains the underlying mechanism. An aliased-list-only contract cannot survive the planned durable resume. - Return terminal {'error': ...} results for permanent conditions (nesting refusal, budget exhaustion) instead of ModelRetry, which burned retries into UnexpectedModelBehavior. The budget result now carries last_error and completed. - Salvage completed sub-agent results into retry messages and the budget-terminal dict so a retry can reuse them instead of re-spending the budget; the sandbox-panic path does the same. - Pass through model-safe exception messages (UsageLimitExceeded) so the model can react; keep the opaque form for everything else. - State the call budget and sub-agent statelessness in the tool description; warn instead of failing when a reveal has no pending message queue to announce into. - Docs: correct the N*T "hard ceiling" claim (core checks token limits after a response arrives), fix the examples README script (it subscripted str outputs), pin model-facing text with snapshots. The old "budget error cannot mask another failure" comment was disproven under asyncio.gather; the true invariant is documented. Refs: https://www.anthropic.com/engineering/code-execution-with-mcp, https://code.claude.com/docs/en/workflows Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(dynamic-workflow): align tests with pydantic-ai v2 deps typing Merging main raised the pydantic-ai-slim floor to >=2.1, whose Agent `deps_type` default became `object` (was `NoneType`). Constructing an `Agent(..., capabilities=[Cap[None](...)])` then fails overload matching: the capabilities pin `AgentDepsT=None` while the default supplies `object`. Switch the deps typevar to `object` throughout, matching the convention the code_mode and subagents tests already use. Also make test_max_agent_calls_exact_under_concurrent_fan_out order- independent. The completed list was asserted in exact order, but which three of eight concurrent calls win the budget and the order they finish are both scheduler-dependent (3.14 admits them in a different order than 3.13). Assert the invariant that matters: exactly the budget ran, each once, well-formed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: stop tracking .agents/scheduled_tasks.lock This is a machine-local runtime lock for Claude Code's scheduled-task runner (pid/session mutex), not project content. It was committed by accident. Remove it from the tree and gitignore it alongside the other .agents machine-local files. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(dynamic-workflow): accept raw agents, make reveal() the strict mutation API Pydantic AI 2.x agents carry their own name and description, so wrapping every entry in WorkflowAgent was boilerplate: agents=[reviewer, summarizer] now works, following the OpenAI-handoffs union pattern where a raw agent means "use the agent's own metadata" and the wrapper is a per-use-site override. WorkflowAgent gains the agent.description fallback and positional agent that the sibling SubAgent already had; the divergence had no rationale. Making reveal() the only post-construction channel (eager validation, UserError at the call site, internal catalog list) removes the reason the toolset needed lenient re-validation: the strict/lenient _rebuild duality, _reveal_warned, and the warn-and-skip paths are deleted, and an entry that reaches the toolset invalid now fails fast as a contract violation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(dynamic-workflow): rewrite README as a progressive tutorial The reference-style README front-loaded precise semantics before a reader had a mental model to hang them on. Reshape it into a learning arc -- smallest working example first, one concept per section, dense tables tucked into admonition boxes -- so the same facts land in the order a reader needs them. All technical claims are carried over unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(dynamic-workflow): use raw-agent shorthand in the examples README The migration example wrapped each sub-agent in WorkflowAgent solely to supply a catalog description, which read as pre-refactor mandatory wrapping and contradicted the prose above it ("documented by its description"). An agent's description is a property of the agent, not of one workflow, and the field has other consumers (agent spec, OTel gen_ai.agent.description), so its natural home is the Agent. Move the descriptions onto the agents and pass them raw; WorkflowAgent stays documented in the capability README's override section for the cases that need it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(dynamic-workflow): drop internal decisions log from the PR DYNAMIC_WORKFLOW_DECISIONS.md was an internal working-notes log (branch codenames, v1/maybe-v2 planning, open problems). It doesn't belong in the shipped repo and duplicates the capability README. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: fix pydantic-ai-slim version pin and tighten DynamicWorkflow docstrings Correct the required version in README.md to `pydantic-ai-slim>=2.1.0` (matches pyproject.toml; the previous `>=1.101.0` was wrong), and trim the verbose DynamicWorkflow capability/field docstrings to say the same thing more concisely. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(dynamic-workflow): promote out of experimental DynamicWorkflow graduates to a released top-level capability after a hardening round driven by live model runs and adversarial review: - Live-exercised with a real Opus orchestrator across ten scenarios (fan-out, structured output, budget exhaustion, error salvage, runtime reveal, defer_loading, nesting refusal, open-ended research). The one model-facing trip point found -- reading the sub-agent call budget as per-script rather than per-run -- is fixed in the tool description. - Reveals on a deferred capability no longer leak: get_tools holds reveal folding and announcements until the capability is loaded. Ownership is resolved by identity through the run's capability registry, unwrapping wrapper chains, because ids cannot identify the owner (an id-less capability registers under a generated key, and a wrapper registers in place of what it wraps). - Public annotations erase sub-agent outputs as `object`, not `Any`. - README drops the experimental banner, documents the per-run budget and keyword-only construction, and loses a stray editing artifact that had been committed at the end of the file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(dynamic-workflow): let sub-agents inherit the run model * test(dynamic-workflow): exclude never-run parent model fn from coverage The parent model callback in test_inherit_model_off_keeps_sub_agent_bound_model is asserted never to run, so its body is unreachable and broke fail-under=100. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(dynamic-workflow): merge into CodeMode when both are on one agent With both capabilities registered, CodeMode used to fold run_workflow into the run_code sandbox as a plain function taking a code string, so the model had to write a script containing a second script as a string literal, run in a nested sandbox, with no return schema. The two now compose instead: when a CodeMode capability is present, DynamicWorkflow exposes each sub-agent as its own (task) tool carrying the agent's output schema, and CodeMode renders them as typed async functions inside run_code, callable alongside the agent's regular tools in one script. Budget, usage forwarding, and the nesting guard moved into a shared call path so both modes enforce the same rules; the sub-agent guidance run_workflow's description carried moves into instructions in merged mode. No new public API: the merge is automatic. Standalone run_workflow scripts are now statically type-checked against the sub-agent signatures before execution (every call is a fresh sandbox, so the check is always sound); a misspelled function or positional task costs a retry but no sub-agent budget. The runtime kwarg guards remain, because Any-typed values (e.g. json.loads results) evade the checker, and the tests now exercise them through exactly that route instead of reaching into private helpers. Also: budget-sharing across multiple run_workflow calls in one run gets a test (the tool description's headline claim was untested), the examples/ README folds into the capability README, and the unrelated .gitignore line is dropped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: explain DynamicWorkflow in the main README The main README showed CodeMode end to end but only gave DynamicWorkflow a matrix row, so a reader had no way to see the point of the capability without opening its package README. Add a compact section after the Quick start: the turn-by-turn delegation problem, the two-line setup, the script the model writes, and how it composes with CodeMode, budgets, and deferred loading. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(dynamic-workflow): move under experimental New capabilities start under pydantic_ai_harness.experimental per repo convention, and the planned extensions (structured sub-agent inputs, durable workflows) touch the sub-agent call contract, so the API needs room to move before names go stable. This reverses the earlier top-level promotion on top of everything built since: importing the capability now emits HarnessExperimentalWarning and there is no top-level export. The README additionally explains the experimental status, positions DynamicWorkflow against SubAgents (one-delegation-per-turn vs scripted choreography), and documents the merged-mode tool-name collision. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(dynamic-workflow): separate from CodeMode instead of merging The CodeMode merge coupled two independent capabilities, and when their composition did not line up -- a restrictive CodeMode `tool_selector`, or running out of the sub-agent budget mid-script -- it degraded in ways the merge could not recover cleanly. Drop the merge: DynamicWorkflow always exposes its `run_workflow` tool and no longer inspects the run's capabilities. `run_workflow` is itself a code-execution sandbox, so CodeMode must not fold it into `run_code` -- that would make the model pass a script as a string argument to a function inside another script. CodeMode now keeps any tool carrying `code_arg_name` metadata native; `run_workflow` already declares it, so with both capabilities the model sees `run_code` and `run_workflow` as two independent tools. The rule only affects code-execution tools, so CodeMode is unchanged for every other tool. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(dynamic-workflow): correct max_duration_secs semantics and refresh README The docs claimed `max_duration_secs` counts wall-clock, including time awaiting sub-agents fanned out with `asyncio.gather`, and justified having no default cap on that basis. Empirically that is false across every monty the capability supports: the timer is a per-bytecode-step check, so it measures in-sandbox execution time and excludes time the script spends suspended on the host awaiting sub-agents (sequential or gathered). Correct the claim in `_toolset.py`, `_capability.py`, and the README, and describe what the cap is actually for: a pure-CPU `while True` runaway, the one thing the sub-agent budgets cannot catch. Add a regression test that pins the corrected behavior so the stale claim cannot silently return: slow sub-agents under a small cap still complete. Rewrite the README prose in a plainer voice with no em-dashes, and cite Jarred Sumner's Bun Zig-to-Rust rewrite as a real-world example of the same pattern (at Claude Code's session scale, not this specific API). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-09 16:53:02 +05:30
4 people	b4365440b5	feat: serve a Pydantic AI agent to editors over ACP (experimental) (#274) ... * feat: add ACP capability to serve agents to editors Editors and TUIs that speak the Agent Client Protocol (Zed and others) can drive an external coding agent, but plugging a Pydantic AI agent into one previously meant implementing the ACP server side by hand. run_acp_stdio serves any Agent over stdio: streamed text and thinking, file edits rendered as diffs, human-in-the-loop tool approval mapped to deferred-approval tools, per-workspace sessions via a session_config hook, model switching, cancellation, and optional session persistence. FileSystem.get_toolset/Shell.get_toolset now return their concrete toolset types so the ACP presenter can recognize their tool calls by name and annotate them with kinds, locations, and diffs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: skip ACP tests at collection when the acp extra is not installed The slim CI jobs sync without extras, so `agent-client-protocol` is absent and the ACP test modules failed at import during collection. Ignore them via conftest when `acp` can't be found; `test_packaging.py` stays collected since package metadata holds on base installs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(acp): enable unstable routing and bound streamed updates by byte size session/set_model and session/close are advertised at initialize, but the ACP SDK router rejects them as unstable unless run_agent is given use_unstable_protocol=True -- so the model picker and session-close affordance returned method_not_found over a real connection. Enable the flag. Streamed text was chunked by character count, but the SDK serializes with ensure_ascii=True, so a non-ASCII code point expands up to 12 bytes (a surrogate pair) inside the JSON string. A single agent_message_chunk of emoji/CJK could exceed the client's 64 KiB read buffer and drop the connection. Chunk by escaped byte length instead. Both gaps were invisible to the suite: the close/set_model tests called the adapter directly (bypassing the router) and the large-output test used only ASCII. Add a through-the-router stdio test and a non-ASCII chunking test as guards. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(acp): per-turn usage, read-resilient terminal cleanup, persistence + native-toolset tests Address review follow-ups on the ACP capability: - Report per-turn token usage on `PromptResponse.usage`, summed across approval passes (UNSTABLE ACP field; clients that don't support it ignore it). - Suppress client errors during the shielded terminal kill/release so a failing cleanup call can't mask the in-flight CancelledError (the spec requires the turn to end with a `cancelled` stop reason). - Cover the persistence guarantees that previously had no store-active test: a cancelled turn commits nothing, an approval-resume turn persists each update once (no duplicate tool-call start), and `StoredSession` round-trips through Pydantic across the full `SessionUpdate` union. - Add a through-the-wire stdio test that the editor-native fs/terminal toolsets route to the client when mounted per session. - Document the dynamic-`ApprovalRequired` partial-side-effect nuance, per-turn usage, and that session persistence composes with per-run step durability. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(acp): editor-native reads for read-only fs clients, writes delegated locally A client that advertises filesystem reads but not writes previously got no editor-native filesystem at all (acp_filesystem returned None, so callers fell back to a fully-local toolset and lost the editor's live view, e.g. unsaved buffers). It now returns a combined toolset: reads route through the editor, and writes are delegated to the local FileSystem capability rooted at the session cwd (reusing its path sandboxing rather than reimplementing writes). This is coherent only when the agent shares the workspace disk with the editor (same machine, or an agent inside the editor's container); for a remote editor the writes land on the agent's disk. Documented as such at the helper and README. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(acp): point _all_known_model_names at the now-merged known_model_names() pydantic-ai#5803 added `pydantic_ai.models.known_model_names()`, the public replacement for the `KnownModelName.__value__` introspection. It isn't in a released `pydantic-ai-slim` yet, so the swap (and the floor bump it requires) waits for that release; update the breadcrumb so it's actioned then. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(acp): replay user turns on session/load, advertise MCP capabilities Two spec-conformance gaps from PR review: - The transcript only ever recorded agent-direction updates, so a reopened session replayed a one-sided conversation. ACP requires session/load to replay the entire conversation, user turns included. The prompt's content blocks are now seeded into the turn's update list -- recorded for replay but never sent live (the client renders its own prompt; the prompt-turn spec sends no user echo) -- riding the existing commit path so a cancelled turn still rolls its user message back. - initialize left mcp_capabilities at the default (http/sse false), and the spec forbids clients from sending HTTP/SSE MCP servers that are not advertised -- making the documented session_config -> mcp_servers path unreachable for conforming clients over those transports (stdio is not gated). Advertising is now an explicit opt-in mirroring prompt_capabilities, since only the embedder knows which transports their session_config connects; setting it without a session_config fails at construction rather than inviting servers that session/new would reject. Also adds the guard test from review follow-up: _all_known_model_names fails loudly if pyai ever recomposes the KnownModelName alias. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(acp): make the collection guard coverable in a single environment The acp-extra collection guard was an `if` statement whose true arm only runs on slim installs, so local `make testcov` (always all-extras) could never reach 100% -- only CI's combined slim+all-extras matrix could. A conditional expression has no statement arc for branch coverage to miss, so the documented local gate works again without excluding a line CI genuinely covers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(acp): record why _all_known_model_names avoids the unreleased public API The breadcrumb read as if known_model_names() were already usable; make the deliberate avoidance explicit. The public enumerator is merged upstream but not in any released pydantic-ai-slim, and the test-floor CI job runs against the floor release, so calling it would break there. Move the rationale from the docstring to a code comment (docstring stays contract-only) and spell out the swap-and-delete trigger: a release that ships it plus a floor bump. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(acp): add spec-conformance suite driven over a real in-memory wire The existing tests invoke adapter methods directly, below the SDK's JSON-RPC router and serialization -- a boundary that cannot see two bug classes that already bit this adapter: a method the router gates before the adapter runs (the use_unstable_protocol reachability bug) and a frame whose serialized bytes overrun the client buffer (the ensure_ascii chunking bug). `tests/acp/_wire.py` closes that gap without a subprocess: a real ClientSideConnection talks to acp.run_agent across a socket.socketpair in one event loop, so every request crosses the router and codec and every update arrives as bytes the client re-parses, with asyncio's default 64 KiB reader limit standing in for the stdio buffer. `tests/acp/test_conformance.py` is organized by spec clause, not by adapter method, each with an oracle built from the spec/input rather than the adapter's own output: - version negotiation echoes the requested version (not a literal) - capabilities are advertised iff supported (load_session, mcp, auth_methods, session list/fork/resume, modes, config options) -- read back off the wire - unstable methods route through with the flag and are method_not_found without it (the load-bearing reason run_acp_stdio enables it) - error codes distinguish method_not_found from invalid_params - session/load replays the entire conversation including user turns - large non-ASCII output reassembles intact within the client's read buffer CONFORMANCE.md records the clause-to-test matrix, the open gaps (none a known bug), the two verified deviations, and the N/A ledger, derived from a spec-page sweep with adversarial verification of every candidate finding. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(acp): stop surfacing unconsumed additionalDirectories The adapter accepted the client's additionalDirectories and forwarded them to the session_config AcpSession, but never advertised sessionCapabilities.additionalDirectories -- so a conformant client never sent them, and nothing downstream consumed them (the FileSystem capability is single-root). The ACP capability is also still UNSTABLE. Drop the dead surface: remove the field from AcpSession and stop forwarding it. The acp.Agent base signature still carries the parameter, so the session methods accept and ignore it. Supporting it properly (multi-root filesystem + advertisement) is recorded as a deferred feature. Also trims CONFORMANCE.md to a public supported-capability summary. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(acp): keep inline image bytes and let run_acp_stdio advertise MCP An ACP image block always carries inline `data`; `uri` is only an optional source reference, so preferring the URL dropped the bytes the client actually sent in favour of a link the model may be unable to fetch. Prefer the inline data, matching the reference ACP agents. `run_acp_stdio`/`run_acp_stdio_sync` forwarded every adapter option except `mcp_capabilities`, so advertising MCP transports was only reachable by constructing `PydanticAIACPAgent` by hand. Forward it for parity. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(acp): reliable approval scope, mapped stop reasons, safe session reload "Always allow"/"always reject" keyed the scope on the raw tool arguments, which a streaming model (and OpenAI by default) delivers as a JSON string; re-ordered keys then produced a different key and silently re-prompted for a call already decided. Canonicalize via `args_as_dict()` so the same logical call shares one remembered decision. Completed turns always reported `end_turn`, hiding `max_tokens`/`refusal` from the client. Map the model's finish reason to the ACP stop reason. `session/load` overwrote a still-open session without cancelling its in-flight turn, leaking a task that could later persist stale state over the restored transcript. Tear it down first, sharing the close_session teardown path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(acp): correct the API reference and capability notes The README API block omitted `session_store`/`models` (and now `mcp_capabilities`), the limitations list contradicted the documented MCP-via-`session_config` support, and the overwrite-diff note misdescribed what the code emits. Also drop a dangling internal-doc reference from a source comment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(acp): record the unhandled SessionStore failure contract A durable SessionStore can fail on save or return a corrupt payload on load, but the adapter currently propagates those exceptions rather than handling them. Document the gap on the Protocol where store implementers will see it, so the behaviour is a known boundary rather than a surprise -- without adding handling we have no consumer for yet. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(acp): correct the SessionStore failure note to match observed behaviour The previous note assumed store exceptions surface "raw". Verified against the SDK (connection.py `_run_request`) and over a real in-memory wire with a raising stub store: they are converted to JSON-RPC errors before reaching the client -- a `pydantic.ValidationError` to `invalid_params` (-32602), anything else to a generic internal error (-32603). Record what the client actually receives rather than what was guessed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(acp): handle SessionStore failures instead of leaking them `session_store` is a public, documented extension point, so a durable store that fails on save or returns a corrupt payload on load is reachable today -- not hypothetical. A failed save previously errored the very turn the user had already watched stream to completion; a corrupt load leaked raw pydantic/IO detail to the client. Make save failures non-fatal: log and swallow them so the turn (or session) that already committed in memory still succeeds, and the next save catches the store up. Make load failures fail `session/load` with a clear internal error rather than a leaked exception, since a session that cannot be read cannot be reopened. Verified over the in-memory wire with a failing stub store. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(acp): close out in-flight tool calls when a turn is cancelled A tool call announced as pending/in_progress was never driven to a terminal status when its turn was cancelled (via session/cancel or a dismissed permission dialog), so a client kept rendering it as running after the turn ended cancelled. On the cancel unwind, fail any tool call still lacking a terminal result -- sent live (a cancelled turn never commits its transcript) and shielded so the asyncio cancellation cannot abort the send. Verified over the in-memory wire. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(acp): reject prompts queued behind a session that closed or reloaded A prompt waiting on the session's turn lock held a reference to the old SessionState; close_session/load_session only cancel the *active* turn, so the queued prompt would run against the discarded state - invisible to session/cancel - and persist its orphaned history over the closed (or just restored) session. Re-checking liveness after acquiring the lock turns that zombie turn into the invalid_params the client already gets for a prompt sent after close. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(acp): never swallow a handler's own teardown cancellation prompt() and _cancel_active_turn awaited the turn task bare, so the CancelledError from connection teardown was indistinguishable from the turn's own cancellation and got converted into a response (or, via a dismissed permission dialog, replaced by the internal _TurnCancelled). The SDK cancels each handler task exactly once on shutdown and its sender is already closed by then, so a handler that survives that one cancel and tries to answer hangs Connection.close() forever. Awaiting through asyncio.shield keeps the two cancellation sources apart: the turn's end is read off the (done) task, while the handler's own cancellation propagates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(acp): don't report rollback when a cancel lands in the post-commit save The store save after a turn commits was the one suspension point left between commit and return: a session/cancel delivered there escaped the turn task and the prompt answered 'cancelled' (the rollback signal, no user_message_id) while the session's in-memory history and transcript already contained the whole turn - the next prompt would build on history the client believed was discarded. The InMemorySessionStore never suspends, which is why the existing cancel-persistence test could not see this; any real (file/db) store can. A cancel that arrives after commit has simply lost the race, so the interrupted save is treated like the write failures _persist already swallows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(acp): don't leak a terminal when run_command is cancelled mid-create The terminal/create call sat before the cleanup try-block, so a cancellation landing there unwound without a kill or release - but the request may already be on the wire (the SDK sender flushes queued payloads even when the awaiting future is cancelled), leaving the command running client-side with nobody holding its id. Running the create as a shielded task lets the cleanup await the response late, learn the id, and kill/release as usual. Folding the late-id case into one kill path and one release path also keeps the existing guarantee that a failing kill never skips the release. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(acp): end limit-hit turns with max_turn_requests, close out tool calls on errors pydantic-ai's default UsageLimits(request_limit=50) means a long tool loop routinely raises UsageLimitExceeded - which escaped the turn as a JSON-RPC internal_error, the exact condition ACP defines the max_turn_requests stop reason for (token limits map to max_tokens). The raising run's partial messages are not retrievable, so such a turn rolls back like a cancellation and says so (no user_message_id, no usage). Turns failing with any other exception now also close out their announced tool calls before the error reaches the client, matching the cancellation path, so the client does not render them as running next to the turn's failure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(acp): resolve workspace-relative paths before they reach fs/read|write ACP requires absolute paths on fs/read_text_file and fs/write_text_file, but models routinely emit workspace-relative ones - the local FileSystem tools document relative paths and share these tools' names, and the presenter layer already absolutizes for exactly that reason. The client-backed toolset forwarded them raw, producing non-conformant requests the client may reject. acp_filesystem now hands the toolset the session cwd so relative paths resolve before the wire; absolute paths and directly-constructed cwd-less toolsets are untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(acp): true up the conformance claims, comments, and doc links The stop-reason note predated the finish-reason mapping and said the opposite of what the code (and its tests) do; the prompt-capabilities bullet read as inbound enforcement when the spec puts that restriction on the client; the stdio MCP gap (the spec's unconditional MUST) was undisclosed; and two code comments claimed validation/rejection the SDK router does not actually perform - the adapter's own raises are the load-bearing behavior. Doc links now use the canonical pydantic.dev paths both old URLs redirect to. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(acp): shield the terminal create in place instead of via a side task The ensure_future + asyncio.shield shape from the previous commit hit Python 3.12+'s shield behavior: once the outer await is cancelled, a late failure of the inner create is reported to the loop exception handler even when the cleanup retrieves it, which spams production logs and fails under pytest-anyio. An anyio shield around the create await itself is simpler and leans on the same anyio-mediated cancellation the kill/release cleanup already depends on: the cancellation defers to the next await, by which point the terminal id is known and the existing cleanup kills and releases as usual. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(acp): small type and surface cleanups from review - ToolCallPermission.args was typed object although the only construction site passes args_as_dict(); Mapping[str, object] lets a custom policy scope by an argument without isinstance gymnastics. - default_permission_scope is referenced by the permission_policy docs as the fallback to compose with, so it must be importable: exported. - McpServer names the per-server union once instead of spelling it in two places; McpServers gains the TypeAlias marker its sibling had. - _model_state folds the None case so both call sites drop a repeated conditional; the load_session store guard's no-cover pragma was wrong (the SDK router routes session/load regardless of the advertisement, and test_persistence already executes the branch). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(acp): pin claimed behavior the suite asserted too weakly or not at all - The unsupported-method test asserted only RequestError despite name and CONFORMANCE.md promising method_not_found; codes now pinned (-32601, and -32602 for the unknown-session prompt). - The model-override test was tautological: both models answered successfully either way. It now asserts the override's distinct canned output (mutation-checked: dropping the per-run override fails it). - Usage summation across approval passes had no test; an approval turn's output tokens must exceed the resume pass alone (mutation-checked against 'usage = result.usage()'). - New protocol-contract coverage: cancel racing an unanswered permission dialog (turn ends cancelled, pending call driven to failed), double cancel idempotency, image block arriving in model history as decoded BinaryContent, the denial message reaching the failed update's raw_output, chunk_text's exact-budget boundary, and a set_model save failure being swallowed like every other persist failure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(acp): survive raw cancellation during terminal create; answer cancelled after a post-commit cancel An adversarial re-review of this branch's own fixes caught a regression: the anyio shield around terminal/create only blocks anyio-mediated cancellation, but the adapter and pydantic-ai deliver raw task.cancel() (turn.cancel, cancel_and_drain), which pierces it - reintroducing the leaked-terminal window end to end. The create now runs as its own task awaited via asyncio.wait: a raw cancel hits the waiter, not the create, and unlike asyncio.shield (3.12+) a late create failure is not reported to the loop exception handler when the cleanup retrieves it. The leak scenario is pinned at both the toolset level and through the real adapter/agent path. Also from the re-review: a cancel that lands after the turn committed (inside the store save) now answers 'cancelled' as the spec requires - while keeping the committed signals (user_message_id, usage, history) - and the one-tick teardown-vs-turn ambiguity that 3.10 cannot resolve is documented at both await sites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(acp): record the post-commit-cancel semantics in CONFORMANCE Also drops an unbounded poll loop in the permission-dialog race test for a single deterministic tick (prompt sets active_turn before its first suspension point), which was the one partial branch left repo-wide. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(acp): let the turn build its own response and drop the commit-flag tuple _run_turn returned (usage, stop_reason, committed) so prompt() could reassemble a PromptResponse it had all the pieces for; building the response in the turn removes that protocol and the committed flag. Recording turn.updates is now unconditional (only the commit gate matters), and the presenter-fields/acp_filesystem branches collapse to single construction paths. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(acp): appease codespell (unparseable -> unparsable) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Support Pydantic AI 2.x in ACP * feat(acp): forward usage limits into agent runs Let hosts bound each ACP run segment with Pydantic AI usage ceilings while preserving the default run behavior when no explicit limits are configured. * feat(acp): resolve selected session models Let embedders map advertised ACP model ids to concrete Pydantic AI models and re-surface a resident session's current model state. * feat(acp): expose read-only session history Hosts can gauge context with the same estimate_token_count path the compaction tiers use by reading committed resident-session history. The returned list is a shallow snapshot, and callers must treat the shared ModelMessage objects as read-only. * Track ACP 0.11 session config models * chore(acp): fix coverage, trim redundancy, clarify module names Get the ACP PR to green CI and simpler to review, without changing behavior. - Coverage: the only uncovered line was a dead `return None` branch in the `_model_option` test helper (source was already 100%). Assert the option is present instead, dropping the untested branch and its now-redundant guards. - Remove CONFORMANCE.md: its supported/not-supported matrix duplicated the package README's feature sections and "Cancellation and limitations". Point the two references at the README. - Dedup the test ACP clients: RecordingClient, FakeClient, and WireClient each re-spelled the same ~13-method "unused capability" stub block to satisfy the SDK Client interface. Extract it once into RecordingClientBase; each client now subclasses it and overrides only what it exercises. Interface drift is one edit. - Rename two opaque modules for clarity (files only; public symbols unchanged): _present.py -> _presentation.py (matches its ToolCallPresentation exports) and _native.py -> _client_toolsets.py (client-routed filesystem/shell toolsets), with tests/acp/test_native.py -> test_client_toolsets.py to mirror the source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(acp): move under experimental and shrink the public surface New capabilities start under pydantic_ai_harness.experimental, so ACP follows: importing warns HarnessExperimentalWarning and the API may change without deprecation. __all__ drops the typing-only aliases (25 to 18 names); default_permission_scope and McpServer stay because policies compose with the default scope and AcpSession.mcp_servers is typed by McpServer. Tests move to tests/experimental/acp to mirror the source tree; the experimental warning test lives in the SDK-gated test_acp.py so slim (no-extras) installs stay green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(acp): expose model_resolver and usage_limits on the stdio entry points PydanticAIACPAgent already accepted both, so wanting a per-run token ceiling or host-defined model ids forced users off run_acp_stdio onto the class and a hand-rolled acp.run_agent call, where forgetting use_unstable_protocol=True silently loses session/close. Forward the two params so the one-liner covers every adapter option. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: David SF <david.sanchez@pydantic.dev>	2026-07-07 23:36:09 +05:30
Aditya VardhanandGitHub	3c5ec34f9e	Bump vcrpy to 8.2.1 to fix YAML deserialization RCE (#300) ... Resolves Dependabot alert GHSA-rpj2-4hq8-938g: vcrpy < 8.2.1 loads cassette files with PyYAML's unsafe loader, allowing arbitrary code execution on cassette load (e.g. in CI). 8.2.1 switches to SafeLoader.	2026-07-01 17:50:40 +05:30
Aditya VardhanandGitHub	6eb91dfe86	Bump pydantic-ai floor to 2.1 (#303)	2026-07-01 00:02:14 +05:30
3 people	05ae737182	bumping pydantic ai (#280) ... * bumping pydantic ai * Fix breakage from pydantic-ai 1.107: capability toolsets no longer cross runs All three failures came from pydantic-ai #5230 (on-demand capabilities): - SubAgents inherit_tools transplanted capability-contributed toolsets into sub-agent runs where the owning capability is not registered, which now fails CapabilityOwnedToolset's ownership resolution. Inherit only the parent's own toolsets; capability sharing is shared_capabilities' job. This also drops the delegate tool, replacing the name-based filter. - Tool search discovery moved from message-scanning to RunContext.discovered_tool_names, so the code_mode test now derives it via parse_discovered_tools like the agent graph does. - ToolDefinition gained capability_id, refreshing the logfire snapshot. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Test latest pydantic-ai in CI and set the floor to the verified minimum The lock-resolved test matrix never exercises pydantic-ai releases newer than the lock, and pydantic-ai's harness-compat job only covers harness code that exists when a pydantic-ai PR merges. Harness code merged after a core change (SubAgents, ManagedPrompt vs core #5230) was therefore never tested against it until a manual bump. The new test-latest job re-locks pydantic-ai-slim/pydantic-graph to the newest published versions on every run, so release breakage surfaces immediately. With latest covered by CI, the pydantic-ai-slim floor no longer needs to chase releases: set it to 1.105.0, the first release with capability ownership semantics (core #5230) that the harness now relies on, and verified against the full suite via the test-floor resolution. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Mirror pydantic-ai's lowest-versions matrix and unfreeze its resolution The test-floor job was not testing the floor: under the workflow-level UV_FROZEN=1, `uv sync --resolution lowest-direct` installs the locked (latest) versions, so the job duplicated the regular matrix while claiming lowest-version coverage. pydantic-ai's test-lowest-versions job sets UV_FROZEN=0 on the sync step for exactly this reason; do the same, and adopt the rest of its shape (full Python matrix, fail-fast off, job timeouts) so floor regressions that only reproduce on some Python version get caught. A genuinely-lowest resolution surfaced one over-coupled snapshot: the `logfire.metrics` span attribute only exists on logfire releases newer than the extra's 4.31.0 floor, so treat it as volatile in the ManagedPrompt span snapshots instead of raising the floor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Replace UV_FROZEN with UV_LOCKED in CI Frozen mode installs the lockfile blindly: lock drift goes unnoticed and resolution flags become silent no-ops, which is how the lowest-versions job ended up testing the locked versions. Locked mode validates the lockfile against pyproject.toml and fails loudly on any mismatch, so a job that cannot do what it claims turns red instead of green. The two jobs that deviate from the lock on purpose (lowest-versions resolution, latest pydantic-ai upgrade) opt out per step, each with a comment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Keep filtering the delegate tool when SubAgentToolset is used directly The capability-ownership filter only drops the delegate tool when the toolset arrives via the SubAgents capability, since that is the path that wraps it in CapabilityOwnedToolset. SubAgentToolset is publicly exported, and registered directly in Agent(toolsets=[...]) nothing wraps it, so inherit_tools=True forwarded delegate_task to sub-agents and re-enabled recursive delegation. Restore the name filter on top of the capability filter so both registration paths stay non-recursive. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Adapt to per-delegate run controls from main PR #283 made limits and call_counts required on SubAgentToolset, which broke the direct-construction test from the previous commit once CI merged the branch with main. The merge also stranded the CapabilityOwnedToolset import below the new SubAgentLimits dataclass; moved it back into the import block. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-06-12 16:55:22 +05:30
4 people	6f2aa11438	Add StepPersistence capability for step-event durability across delegates (#251) ... * Add StepPersistence capability for step-event durability across delegates Supersedes PR #176 (SessionPersistence): orchestrators like pydanty need visible event trails for delegate runs that may time out before a "save full session after the run" hook can fire, and need to continue or fork a delegate's prior investigation without rediscovering context. A single after-run snapshot is too coarse for that use case. The capability now records (a) append-only StepEvents at every boundary (run/model-request/tool-call start, completion, failure), (b) a ContinuableSnapshot only when message history is provider-valid (every ToolCallPart has a matching ToolReturnPart / RetryPromptPart) — saved mid-run after CallToolsNode and at after_run, and (c) a ToolEffectRecord ledger so a run killed between before_tool_execute and after_tool_execute leaves an `unknown_after_crash`-style record rather than a falsely-continuable snapshot. Lineage metadata (parent_run_id, agent_name) ties delegate runs back to their orchestrator. `continue_run` / `fork_run` helpers load the latest continuable snapshot for a run. Backends: InMemoryStepStore (tests) and FileStepStore (JSONL events + JSON snapshots, with run_id path-traversal validation and anyio.to_thread for blocking I/O). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Address pydanty review + ergonomics: tighten correctness and identity model Correctness fixes from pydanty's PR review: - FileStepStore: snapshot filenames are now a per-run monotonic counter, not `ctx.run_step` — `run_step` resets each Agent.run, so re-using a `run_id` across calls would let an earlier run's higher step-index snapshot mask a later run's lower-step-index one. - StepStore.get_tool_effect now takes both `run_id` and `tool_call_id`. TestModel and other providers can reuse deterministic tool-call ids across runs; the previous unscoped lookup let one run's effect leak into another's record (including `started_at`). - is_provider_valid now rejects orphan, duplicate, and out-of-order tool returns — the old `set.discard` pattern silently accepted any return regardless of whether a matching call was open. Identity model: - `run_id` resolution: explicit > `{agent_name}-{8-char-hex}` > UUID. Materialised per Agent.run in `for_run`, so reusing one capability instance never silently merges runs. - `parent_run_id` auto-inferred via a module-level ContextVar set in `wrap_run`, so an orchestrator's tool that synchronously calls `delegate.run(...)` produces a delegate `RunRecord.parent_run_id` pointing at the orchestrator's `run_id` with zero threading. Explicit `parent_run_id=` still wins. - `conversation_id` propagated to `StepEvent` and `RunRecord`; `store.list_runs(conversation_id=..., parent_run_id=...)` supports filtering by either or both. Mirrors pydantic_ai's three-level identity (conversation -> run -> step) so "run 1, run 2, run 3" of one dialogue is queryable as a group via `conversation_id`. - `continue_from=` field dropped from the capability. Continuation is now only via `continue_run(store, run_id=...)` -> standard `Agent.run(message_history=...)`. One way to pass history into pydantic_ai, no parallel capability flag. README rewritten around the final API. New sections: three-level identity, run lineage with auto-inferred parent, inspecting a run tree, failure recovery. Tests: 168 total (up from 64), 100% branch coverage on the package. New coverage for the snapshot seq counter, cross-run tool-effect isolation, orphan/duplicate/out-of-order return rejection, ContextVar parent inference across nested agent.run, conversation_id propagation, and the agent_name-derived run_id default. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Address pydanty round-2 review: retry validity, list ordering, effect metadata Correctness: - is_provider_valid no longer rejects non-tool RetryPromptParts. Pydantic AI emits `RetryPromptPart(tool_name=None)` for output-validation failures and providers map those as plain user messages, not tool results. The previous check required every RetryPromptPart to resolve an open tool call, so a run with one output retry produced no final continuable snapshot despite being fully valid. - StepStore.list_runs now guarantees chronological (started_at ascending) ordering across both backends. FileStepStore was previously returning directory-name order (lexicographic), so the README's `[-1]` pattern for "latest run in conversation" could pick the older run when run ids did not sort by recency. - after_tool_execute and on_tool_execute_error preserve idempotency_key and effect_summary from the prior `started` record. Previously the terminal record was written without those fields, so any annotation the tool body wrote was lost on completion. - from_spec raises ValueError for unknown backends instead of silently falling back to in-memory storage. For a persistence capability, turning a typo into accidental non-durability is the wrong failure mode. API: - New annotate_tool_effect(store, ctx, *, idempotency_key=None, effect_summary=None) helper. Tool bodies that write external state call it to attach idempotency + effect metadata to the in-flight ToolEffectRecord without knowing the (run_id, tool_call_id) plumbing. Resolves run_id from a ContextVar set by wrap_run; reads tool_call_id / tool_name from RunContext. - ContextVar moved from `_capability.py` into a new `_context.py` module so the helper and the capability can share it without circular imports and without crossing the private-name barrier. Docs: README fixes a non-existent `list_runs(agent_name=None)` call, documents the chronological-ordering guarantee, and replaces the hand-wavy "populate fields on the ToolEffectRecord" line with a concrete `annotate_tool_effect` example. Tests: 178 total (was 168), 100% branch coverage on the package. Added coverage for non-tool retry acceptance, chronological list_runs on both backends, metadata preservation across completed/failed transitions, annotate_tool_effect under realistic agent.tool, and from_spec backend validation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(step_persistence): align FileStepStore docstring with seq-counter layout The class docstring still showed snapshots/<step_index>.json from the pre-fix layout, but both the README and _next_snapshot_seq document the monotonic counter. Bring the class docstring in line. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(step_persistence): fix broken README examples + clarify run_id sharing Pydanty round-3 review: - README continuation and lineage examples queried `list_runs(conversation_id=...)` on conversations the earlier `.run(...)` calls never set, so the examples crashed with IndexError on `[-1]`. Pass the conversation_id to the earlier calls so the lookup actually works. - The capability docstring claimed reusing a `StepPersistence` instance across `Agent.run` calls does NOT share the id. That is true only for the auto-derived (`agent_name`-prefixed or `ctx.run_id`) cases — an explicit `run_id=` is shared across every `.run()` by design, since that is the orchestrator pattern where the caller owns one logical identity across turns. Rewrite the resolution-order docs to spell out which cases share and which don't, and when to pick each. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(step_persistence): align run_id semantics with pydantic_ai (per-call, not shared) Pydanty round-4 review: the prior round documented explicit `run_id` as shared across `.run()` calls on one capability instance — that framing caused real correctness gaps. The `ToolEffectRecord` ledger is keyed by `(run_id, tool_call_id)` and providers reuse deterministic tool-call ids (e.g. `TestModel` emits `pyd_ai_tool_call_id__{name}`), so a second `.run()` overwrites the first's effect record under the same key — the `unknown_after_crash` signal from turn 1 disappears when turn 2 lands. Realign: - `run_id` is per-`Agent.run`, matching `pydantic_ai.RunContext.run_id`. - For multi-turn logical grouping, use `conversation_id=` on `Agent.run(...)` — that is the pyai-native primitive. The orchestrator pattern is `conversation_id='orch'` with each turn auto-deriving its own `run_id`. - Explicit `run_id=` remains supported but is documented as single-shot (testing, replay, debugging). Reusing it across calls is a caller contract violation, not an implementation feature. Code is unchanged — the implementation was already correct under the right contract. Only the docs were misleading. Tests: - `TestRunIdIsPerCall::test_multi_turn_orchestrator_uses_conversation_id` exercises the recommended pattern: three turns sharing a `conversation_id`, three distinct auto-derived `run_id`s, all queryable as a group. - `TestRunIdIsPerCall::test_explicit_run_id_reuse_collides_ledger` locks down the misuse contract: reusing one explicit `run_id` across two `.run()` calls produces colliding effect records under the `(run_id, tool_call_id)` key. The behavior is documented; the test exists so a future refactor cannot silently change it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(step_persistence): explain why for_run uses a local ContextVar Pyai-aligned review flagged this as a P3 explainer: pydantic_ai already has three single-slot cross-run signals (RUN_ID_BAGGAGE_KEY, ctx.run_id, _CURRENT_RUN_CONTEXT). All three get overwritten by the inner Instrumentation.wrap_run before any nested capability can see the parent identity. A separate harness-local ContextVar, snapshotted before our own wrap_run rebinds it, is the only correct mechanism today. Spell this out so the next reader doesn't try to 'simplify' it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(step_persistence): enforce explicit-run_id reuse with ValueError Pydanty round-5 review accepted the docs-only contract but flagged that "documented but not enforced" is a soft spot. Enforce it: `before_run` calls `store.get_run(run_id=...)` when the user supplied an explicit `run_id`, and raises `ValueError` if a record with that id already exists. The auto-derived cases cannot trigger this check (each call materialises a fresh id in `for_run`). The check is one extra store read per Agent.run when an explicit run_id is set, only. The error message points the caller at `conversation_id` for multi-turn grouping. Test renamed from `test_explicit_run_id_reuse_collides_ledger` to `test_explicit_run_id_reuse_raises` — asserts the second `.run()` raises and the first run's records survive untouched. README + capability docstring updated: the misuse path is now "raises" not "caller's contract." Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: gitignore branch-context skill artifacts + AGENTS.local.md Two patterns that match the existing CLAUDE.local.md ignore convention: - AGENTS.local.md — canonical local-instructions file (CLAUDE.local.md is symlinked to it where the worktree follows the same AGENTS.md/CLAUDE.md symlink pattern). - .agents/skills/branch-context/ — per-worktree decisions log (`pr-decisions.md`) and the skill's local SKILL.md. Pattern lifted from `~/pydantic/ai/base/.claude/skills/branch-context/` where pyai uses an identical setup. Neither is intended to land in PRs — they record cross-iteration design calls so future AICA sessions in this worktree don't silently undo them. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(step_persistence): add SqliteStepStore + MediaStore externalization Adds a new `pydantic_ai_harness.media` package (MediaStore protocol + DiskMediaStore / SqliteMediaStore / S3MediaStore) and wires it into the file/sqlite step-persistence backends so large BinaryContent payloads get externalized out of snapshot JSON / table rows by default. Defaults are zero-config: FileStepStore writes blobs under `<root>/media/<sha256>.bin`; SqliteStepStore writes them to a sibling `media` table in the same DB. Threshold is 64 KiB and URI scheme is `media+sha256://<hex>` so blobs are content-addressed across stores. Pass `media_store=None` to keep bytes inline, or a custom `MediaStore` to redirect (e.g. `S3MediaStore` for R2 / AWS / MinIO). S3MediaStore handrolls SigV4 over httpx to avoid a botocore/boto3 dependency. Verified working against Cloudflare R2. `StepPersistence.from_spec(backend='sqlite', database=...)` now resolves. 180 → 261 tests, 100% branch coverage maintained. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(media): VCR cassettes for S3MediaStore against R2 Adds replay-driven tests under `tests/media/test_s3_cassettes.py` that exercise `S3MediaStore.put/get/exists` against pre-recorded Cloudflare R2 responses. CI runs them without any S3 creds via the committed cassettes under `tests/media/cassettes/`. Sanitisation policy: - `before_record_request`/`before_record_response` swap the real R2 account-id subdomain and bucket name for fixed placeholders (`account.r2.cloudflarestorage.com`, `harness-test-bucket`) - `Authorization` and `x-amz-date` filtered to `REDACTED` - CF-RAY, x-amz-version-id, x-amz-checksum-*, x-amz-request-id headers dropped (none load-bearing for tests; some carry identifying info) - Non-2xx response bodies blanked (R2's gzipped XML error envelope leaks the bucket name; our code only checks status code) The `s3_credentials` fixture uses `os.environ.get(NAME, PLACEHOLDER)` per field, so real R2 creds are used when recording locally with `.env` loaded, and the placeholder constants match the scrubbed cassettes during replay. Because the placeholders are fixed, any scrubber miss during a future re-record shows up as a replay URL mismatch — built-in canary against credential / private-data leakage in committed cassettes. Adds `pytest-recording` (pulls `vcrpy`) to the dev deps. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(media): public_url resolver — protocol method + per-store callable Adds `MediaStore.public_url(uri) -> str | None` plus a `public_url=` constructor parameter on every concrete store. The parameter accepts a sync or async callable; the store auto-detects and awaits if needed. This is the bottom-layer primitive for the forthcoming `MediaExternalizer` capability — that capability will call `store.public_url(...)` per externalized blob and swap `BinaryContent` for `ImageUrl` / `AudioUrl` parts before the model sees the message. The callable shape covers both static URLs (public bucket / CDN — use `make_static_public_url` helper) and dynamic URLs (presigned, per-request signing — pass any async callable with TTL captured in its closure). Why a callable rather than a static config: a public bucket's URL host is not derivable from the bucket creds (R2 public buckets use `pub-<hash>.r2.dev`, AWS public buckets use a different scheme than the path-style endpoint we sign for). The URL is always user-supplied information, so a callable is the right primitive — same shape for the static and presigned cases, and `get` stays untouched (it serves the harness's internal byte fetch, not the model's external HTTP fetch). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(media): introduce MediaContext + key_strategy for extensible operations Adds `MediaContext` (frozen, kw-only dataclass with `media_type`, `filename`, `metadata`) and threads it through every `MediaStore` method and both user callables (`PublicUrlResolver`, `KeyStrategy`). New context fields can be added non-breakingly; existing call sites and resolvers keep working. Also adds: - `KeyStrategy = Callable[[str, MediaContext], str]` for per-store layout control. Default `default_key_strategy` produces `<sha256>.bin`. Disk store validates the result against `..` traversal. - `metadata` persistence on `SqliteMediaStore` (new JSON column) and `S3MediaStore` (signed `x-amz-meta-*` headers, ASCII key validation). Disk store explicitly does NOT persist metadata in v1 — sidecar / xattr options each have load-bearing drawbacks; we ship nothing rather than a half-true persistence promise. - `make_static_public_url(...)` updated to the new `(uri, ctx)` signature. The shift is motivated by the same principle as pydantic_ai's `RunContext`: extension via fields on a context bag rather than via breaking signature changes. Every new requirement (TTL hints for presigned URLs, audit ids, response-header overrides, etc.) becomes a field addition, not an API revision. Cassettes from the previous commit replay unchanged — match-on does not include the signed headers and the request URLs are stable. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(media): close metadata round-trip; drop vestigial sqlite key_strategy Adds `MediaStore.get_metadata(uri) -> Mapping[str, str]` to the protocol and implements it on all three concrete stores: - `DiskMediaStore`: writes a sidecar `<resolved>.meta.json` alongside the blob on put (atomic tmp + rename), reads it back on `get_metadata`. Returns `{}` when no metadata was supplied. v1 had documented this as a deliberate gap — sidecar JSON is straightforward and the xattr / ADS drawbacks don't apply. - `SqliteMediaStore`: `SELECT metadata FROM <table> WHERE sha256=?` + `json.loads`. Raises `FileNotFoundError` for unknown URIs. - `S3MediaStore`: HEAD + collects `x-amz-meta-*` response headers, strips the prefix. Reuses the existing 404 / non-2xx error shape. Drops `key_strategy=` from `SqliteMediaStore`. The digest is the primary key by content-addressing construction — a user-chosen key would either break dedup or be a no-op. Kept on Disk + S3 where bucket / directory layout is a real concern. README + branch-context entries updated to reflect: all three stores round-trip metadata; key_strategy is Disk + S3 only. Coverage stays at 100% branch. * fix(media): reject absolute paths from DiskMediaStore key_strategy `Path(root) / absolute_path` returns `absolute_path` — the root is silently discarded — so a custom `key_strategy` returning `/etc/passwd` (or similar) escapes the store directory even though the previous check only blocked `..`. Tighten the validator to reject both shapes. Caught by pydanty during its #251 integration review. * fix(step_persistence): dedupe terminal snapshot; document sqlite thread-affinity The terminal CallToolsNode already saves the final provider-valid snapshot with the correct step_index. after_run was re-saving the same tail stamped with step_index=0 (ctx.run_step is reset by then), so latest_snapshot reported a misleading step and every run wrote a duplicate. Track whether a node snapshot was taken via a task-local ContextVar and make after_run a fallback that only fires when the run reached no provider-valid boundary. Also document that a caller-owned sqlite connection= must set check_same_thread=False (store SQL runs on anyio worker threads), on both SqliteStepStore and SqliteMediaStore, and correct the WAL-on-every-connection claim. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(media): SigV4 wire path matches signed path; walker preserves unknown fields S3MediaStore signed `_canonical_uri(path)` (each segment percent-encoded) but sent the raw path, letting httpx apply looser encoding. A custom key_prefix / key_strategy emitting reserved chars (`@`, `(`, `=`, ...) diverged from the signed path -> SignatureDoesNotMatch. Send the canonical bytes via httpx `raw_path` so signer and sender agree. Default `<hex>.bin` keys are unaffected. The externalize/restore walker hardcoded the BinaryContent key set, silently dropping any field pydantic_ai adds upstream. Copy the node and swap only `data` <-> marker keys so unknown fields round-trip. Adds tests for reserved-char path agreement, unknown-field preservation, and restore over a pruned blob. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: gate step_persistence + media behind experimental namespace Move both packages under `pydantic_ai_harness.experimental`, matching the convention introduced for compaction/planning/subagents (#191): the old `pydantic_ai_harness.step_persistence` / `.media` paths and the top-level re-exports are gone, so the only import path is now from pydantic_ai_harness.experimental.step_persistence import StepPersistence from pydantic_ai_harness.experimental.media import S3MediaStore Both package __init__s call `warn_experimental(...)`, so importing either emits a `HarnessExperimentalWarning` (silenced category-wide by one filter). This keeps us from committing to a public surface before the capability has real usage. README gains the standard experimental banner; warning tests cover both new packages. 100% branch coverage retained. * docs(media): fix metadata/key_strategy drift, convert em-dashes Subagent review of the experimental move surfaced doc drift: - README "Persistence by store" implied `get_metadata` returns `media_type`; it returns only the user `metadata` mapping. Reworded. - `KeyStrategy` docstring still listed `SqliteMediaStore` / "DB primary key" as a user; sqlite has no `key_strategy`. Dropped it. - README understated `DiskMediaStore` traversal protection (it rejects absolute paths as well as `..`). - README had a stale paragraph that read as `key_strategy` but described `public_url`, and omitted S3. Rewritten to name `public_url` and all three stores. - README `MediaContext` method list now includes `get_metadata`. - Converted em-dashes to `--` across the step_persistence + media trees per the writing-style rule (#270). Other experimental packages' pre-existing em-dashes are left for a separate sweep. No code behavior change; 100% branch coverage retained. * docs(experimental): add warning banner to planning/subagents, finish em-dash sweep Uniformity pass across the experimental packages: - Add the standard `[!WARNING]` experimental banner to planning/README.md and subagents/README.md (compaction and step_persistence already had it; these two lacked it). - Convert remaining em-dashes to `--` in the compaction package and the shared `_warn.py`, per the writing-style rule (#270). The whole experimental source tree is now em-dash-free. Docs only; 100% branch coverage retained. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: David SF <david.sanchez@pydantic.dev>	2026-06-10 23:55:06 -05:00
6 people	bee3f02ca7	feat(logfire): add ManagedPrompt capability (#257) ... * feat(logfire): add ManagedPrompt capability Back an agent's instructions with a Logfire-managed prompt, resolved once per run inside wrap_run so the selected label/version are attached as baggage to every child span. Lets prompts be iterated, versioned, and rolled out from the Logfire UI without redeploying, with a code default as the offline safety net. Lives in pydantic_ai_harness.logfire; adds a 'logfire' optional dependency extra. The test directory is named logfire_variables to avoid shadowing the third-party logfire package under pyright's tests execution-environment root. * fix: satisfy codespell on managed prompt comment * docs(logfire): make ManagedPrompt examples runnable instead of skipped The examples fall back to their code default with no Logfire backend and run clean under a mocked model, matching the main README's unmarked quick-start. * refactor(logfire): rename ManagedPrompt `prompt` param to `name` Logfire calls a managed prompt by its name, so `ManagedPrompt(name=...)` reads better than `ManagedPrompt(prompt=...)`. Also clarifies the `label` and `targeting_key` docstrings per review. * review fixes: cache framing, public imports, [spec] in logfire extra - README: prompt-cache trade-off section, notes on once-per-run resolution, label+version baggage semantics, and the targeting_context outer setter. Callout pointing at pydantic-ai#5107 so users know a first-party `Managed` capability is in flight. - Switch to the public `logfire.variables` import paths (was reaching into `logfire.variables.variable` / `.abstract`). - Pull `pydantic-ai-slim[spec]` from the `logfire` extra so `render_template=True` works without a separate install. - Warn when `logfire_instance` is passed alongside a prebuilt `Variable` (silently ignored before). - Scrub `code.lineno` from span snapshots and expand the volatile-attributes comment so the snapshot doesn't rot on line shifts. - Wrap the provider-backed test in a context manager that restores the module's baseline Logfire config in `finally`, so future tests don't inherit a provider. --------- Co-authored-by: David Sanchez <64162682+dsfaccini@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Bill Easton <williamseaston@gmail.com> Co-authored-by: David SF <david.sanchez@pydantic.dev>	2026-06-01 22:19:30 -05:00
4 people	e1654ccfe2	feat: add FileSystem and Shell capabilities (#260) ... * feat: add FileSystem and Shell capabilities with exhaustive testing - FileSystemToolset: 8 tools (read, write, edit, list, search, find, mkdir, info) with path-traversal prevention, allow/deny patterns, optimistic concurrency - ShellToolset: 1 tool (run_command) with command validation, timeout handling, and async subprocess execution via anyio --- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: David Sanchez <64162682+dsfaccini@users.noreply.github.com>	2026-06-01 20:47:07 -05:00
gordonblackadderandGitHub	ad1a2014ef	add SKILL for harness, including a reference to code mode (#228) ... Add a harness agent skill Documents Code Mode for Pydantic AI, following the agentskills.io spec and matching the existing building-pydantic-ai-agents skill. - Skill directory matches the name field (spec requirement) - Deps aligned with pyproject.toml: pydantic-ai-slim>=1.95.1 plus the code-mode install extra - MCP quick-start passes native=False so CodeMode wraps the tools - Documents the Monty timing-primitive restriction; adds a Code Mode API section - Examples linted via pytest-examples (model-free ones executed) so they can't go stale	2026-06-01 12:26:17 +05:30
Bill EastonandGitHub	d4be271de9	Revert "feat: add FileSystem and Shell capabilities with exhaustive testing (…" (#259) ... This reverts commit 7fd372d9fd.	2026-05-26 20:01:42 -05:00
Bill EastonandGitHub	7fd372d9fd	feat: add FileSystem and Shell capabilities with exhaustive testing (#258) ... - FileSystemToolset: 8 tools (read, write, edit, list, search, find, mkdir, info) with path-traversal prevention, allow/deny patterns, optimistic concurrency - ShellToolset: 1 tool (run_command) with command validation, timeout handling, and async subprocess execution via anyio - 152 tests passing on asyncio backend, 100% branch coverage - Mutation testing: 524/584 killed (89.7%), 60 equivalent mutants documented - All survivors proven equivalent (trampoline defaults, encoding case-insensitivity, unreachable except blocks, dead branches, name=None fallback behavior)	2026-05-26 19:56:47 -05:00
3 people	cda9f873f5	deps: bump pydantic-ai-slim>=1.95.1, drop temporary git pin (#241) ... pydantic-ai 1.95.1 is on PyPI with the eager-import fix from pydantic/pydantic-ai#5422 (`current_otel_traceparent` no longer trips Temporal's workflow sandbox). Remove the temporary `[tool.uv.sources]` git pin and bump the floor. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-13 13:06:41 -06:00
3 people	59e65b7a12	fix(code_mode): honor Tool Search's deferred-loading contract (#240) ... * fix(code_mode): honor Tool Search's deferred-loading contract CodeMode flattened `defer_loading=True` tools into `run_code`'s description regardless of discovery state — defeating progressive disclosure, and on providers with native tool search also double-listing the same tool on the wire (top-level `tools[]` with `defer_loading: true` *and* its signature in `run_code.description`), plus busting the prompt cache on every discovery. Keep `defer_loading=True` tools as native pass-through so ToolSearchToolset's `defer_loading` / `with_native` flags reach `Model.prepare_request` unaltered; they fold into `run_code` only once discovered (`defer_loading=False`). Migrate the sibling `prefer_builtin` filter to its renamed `unless_native` form. Sources pydantic-ai from `main` for pydantic/pydantic-ai#5143 (native Tool Search); drop `[tool.uv.sources]` and bump the `pydantic-ai-slim` floor once that ships in a release. The same pydantic-ai changes rename the `builtin=` capability kwarg to `native=`, so the README and quick-start test are updated to match. Closes #232 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * deps: bump pydantic-ai-slim floor to >=1.95.0, drop temporary git pin pydantic-ai 1.95.0 ships pydantic/pydantic-ai#5143 (native Tool Search), so remove the `[tool.uv.sources]` git override and bump the floor to it. 1.95.0 also deprecates `Agent(instrument=...)` in favor of an `Instrumentation` capability, so the OTel-spans test switches to `capabilities=[..., Instrumentation(...)]`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: say "local MCP toolset" not "FastMCP" in the CodeMode + MCP example Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: add ToolSearch to the ecosystem-agent README example Show progressive tool discovery alongside CodeMode in the full ecosystem showcase — deferred tools fold into `run_code` once discovered. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: group ToolSearch with CodeMode as a meta-capability in the README example Both transform the toolset before it reaches the model, so they sit together under "Tool execution & discovery" rather than alongside the tool providers. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: trim the ToolSearch comment to one line, matching the others Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: clarify CodeMode + Tool Search behavior and the cache escape hatch Rewrite the code-mode README's Tool Search section to cover both the native and local-fallback paths accurately, spell out that a discovered tool folds into `run_code` (busting the prompt cache once at discovery), and document the `tools=` selector workaround (`td.with_native is None`) for keeping a Tool Search corpus fully native. Mirror the escape-hatch note in the toolset docstring. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ci(compat): set UV_FROZEN=1 so the editable overlay isn't clobbered by uv run `compat-test.yml` overlays `pydantic-ai-slim` / `pydantic-graph` from the local checkout via `uv pip install --no-deps -e ...`, but doesn't set `UV_FROZEN`. The next `uv run ruff/pyright/pytest` calls then auto-re-resolve against PyPI ("Ignoring existing lockfile due to change in resolution mode: `lowest-direct` vs. `highest`"), silently uninstall the overlay, and install whatever `pydantic-ai-slim` is on PyPI — so the compat check has been a no-op against the actual pydantic-ai checkout we wanted to validate. Caught when pydantic-ai 1.95.0 landed `current_otel_traceparent` with lazy imports that trip the Temporal sandbox; harness-compat against the merge commit went green (because it was testing 1.94.0 from PyPI, not the merge commit), and only failed once 1.95.0 was published to PyPI. `main.yml` already pins `UV_FROZEN: '1'` at workflow scope; do the same here. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * deps: re-pin pydantic-ai-slim to git main for the Temporal sandbox fix Released `1.95.0` (the lock-resolved version under `pydantic-ai-slim>=1.95.0`) still has the `current_otel_traceparent` lazy-import bug that hangs Temporal workflows on the `test_code_mode_runs_in_temporal_workflow` path — the `all-extras` jobs sat at 1h+ before this. Repin to pai `main` (now at `cf0b9077e`, which has pydantic/pydantic-ai#5422 merged) until `1.95.1` is on PyPI. Drop this section + bump the floor in a follow-up once the release lands. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-13 12:00:15 -06:00
Douwe MaanandGitHub	bf692aff32	Rework README hero example and add ecosystem agent showcase (#226) ... * Rework README hero example and add full ecosystem agent showcase The previous hero example used GitHub's MCP server, which requires authentication and ships tools without `outputSchema` -- triggering CodeMode warnings and producing weakly-typed sandbox stubs. Replaced with the open-source Hacker News MCP server (full output schemas, no auth), and added `WebSearch` to make the prompt naturally exercise parallel tool orchestration in a single `run_code` call. `MCP(..., builtin=False)` is now explicit so the local toolset is used and CodeMode actually wraps the tools (Anthropic supports MCP server connectors as a builtin and would otherwise execute tools server-side, bypassing the sandbox). Added a "Full ecosystem agent" section above the capability matrix that illustrates the breadth of the Pydantic AI ecosystem -- combining capabilities from the harness, core, and the community packages we endorse, with inline credits at each call site. Made install requirements explicit above the hero block. * Reword Quick start prompt to avoid datetime in generated code Monty currently rejects `datetime` (pydantic/monty issue tracking), so the previous prompt's "posted in the last 6 hours" filter caused the agent to generate code that failed in the sandbox. Replaced with a dedupe-by-id + score filter + parallel follow-up calls (thread, user profile, web search) -- still exercises parallel orchestration, dict manipulation, and multi-tool composition without any timestamps. * Embed verbatim run output and soften 'single run_code' claim Followed pydantic-ai docs convention of triple-quoted output after print(). The included synthesis is the actual output from running the example, not a fabrication. Reworded the CodeMode explanation to match what Sonnet actually does: two batched run_code calls each containing parallel tool calls, with the dedupe/filter happening as plain Python in the sandbox -- not the "all in a single run_code" claim from the previous draft. * Surface monty timing restrictions in run_code description Sonnet was occasionally generating `asyncio.sleep` between tool calls (cargo-culted "polite rate-limiting" pattern) and `datetime.now()` for recency filters -- both rejected by the monty sandbox. Adding an explicit "no wall-clock or timing primitives" bullet to the `run_code` tool description (and the matching code-mode README entry) keeps the prompt free of sandbox-specific babysitting. Switched the hero example to claude-opus-4-7 with a more natural prompt ("find the most-discussed story... summarize what you find in one paragraph"), and refreshed the verbatim sample output to a clean run that produced 3 run_code calls (3x parallel feed fetches, pure-Python dedupe/filter, then 3x parallel follow-up calls) with no errors. * Quick start: opt out of native WebSearch so CodeMode wraps it `WebSearch()` defaults to `builtin=True`, and on providers like Anthropic that natively support web search the local DDG fallback is filtered out by `Model.prepare_request`. With `CodeMode` in the mix that produced a split surface -- builtin web_search at top level, duckduckgo_search inside `run_code` -- and the model sometimes tried to call the builtin from inside the sandbox. Passing `builtin=False` keeps every web call going through CodeMode so it can be batched alongside the HN tools in a single `run_code`. Same pattern we already use for `MCP(..., builtin=False)`. The underlying flatten-prefer_builtin-into-run_code bug is filed as #233 and is being fixed separately. Refreshed the verbatim sample output to a clean run with this configuration: 3 chats + 2 run_code calls, all parallel tool batches, no errors. * Quick start: link the public trace and refresh verbatim output DouweM picked a run with substantively richer commentary (Simon Willison's "Vibe coding and agentic engineering are getting closer than I'd like" post on HN) and made the trace public, so we can finally point readers at the actual run from the example. * Add duckduckgo extra to install snippet for WebSearch(builtin=False) * Move ecosystem example credits to dedicated comment lines * Restructure README around the Quick start trace and an ecosystem agent Quick start - Move all per-capability commentary into inline comments at call sites, so links sit next to the code they document. Drops the trailing paragraph wall. - Make the public Logfire trace the section's main visual: a screenshot linking to the trace, then a one-liner explainer. - Trim the verbatim sample output to ~5 sentences. Keeps the Willison framing and the meta-substantive thread reference; drops the long commenter-by-commenter paraphrase and the news-coverage paragraph that previously took over half the section. An ecosystem agent (formerly "Full ecosystem agent") - Move below the capability matrix so readers meet the breadth before the worked example. - Renamed and rewritten intro: shorter, voice closer to the rest of the README, no "upper bound of what's possible" phrasing. - Long disclaimer paragraph moved below the code so readers see the example before the caveats. - Imports split into official (`pydantic_ai*` + harness) vs community (alphabetical). - Sections reorganized: Reasoning, Tools, Execution (CodeMode pulled out of Tools), Context management, Memory, Orchestration, Safety. - Each capability gets its own one-line comment. AnthropicCompaction added as the official compaction option, with a comment pointing at vstorm-co's `summarization-pydantic-ai` as a provider-agnostic alt. - Agent gets a name (`brian` -- Monty Python) which `MemoryCapability` picks up via its `agent_name` arg. - Skills comment now flags @vstorm-co's pydantic-deep alongside Doug's, noting the shape difference. - Switched to `claude-opus-4-7` and `Thinking(effort='high')` to match the Quick start. - Drops `MCP(builtin=False)` here -- the ecosystem example isn't trying to demo CodeMode-wrapped MCP, it's demoing breadth. Drops the no-wall-clock bullet from `_toolset.py` and the code-mode README -- now covered by the standalone PR for that change. Trace screenshot reference uses a relative path that resolves on GitHub once the asset is committed; ready to swap in once the file is in place. * Ecosystem agent: rename to shrubbery, bump thinking to xhigh * Add Logfire trace screenshots and surface them prominently `docs/images/quick-start-trace.png` is the trace tree: it's the visual the main README's Quick start now leads with -- click-through to the public trace. `docs/images/code-mode-trace.png` is the wider shot showing the actual Python the model wrote (parallel `asyncio.gather` over three HN feeds, dedupe-by-id, score filter). It's the centrepiece of a new "In practice" section in the code-mode README that points back to the harness Quick start as the more representative example, replacing the toy weather demo as the leading visual of what code mode does in real use. * Address review on PR #226 - Reorder ecosystem agent capabilities: CodeMode first (the headline), then Thinking, Context management, Tools (MCP -> WebSearch -> Console), then everything else. First-party leads each tier; ConsoleCapability follows our two first-party tools. - Drop the agent `name='shrubbery'` flourish. - Use vstorm-co's `ContextManagerCapability` as the active compaction capability and mention pydantic-ai's `AnthropicCompaction` / `OpenAICompaction` in the comment instead of the other way around. Drops the "tied to Anthropic's prompt caching" phrasing. - `MemoryCapability(agent_name='harness-example')`. - Code-mode README: rewrite the "In practice" prose to be clear that CodeMode produces two `run_code` calls (parallel fetches + filter, then parallel follow-ups), not one. Fix the screenshot alt text to match. * Add regression test for the README's Quick start example The lead example in the main README is the most public surface of the harness — if it stops working we need to know before the user tries it. This test drives the example end-to-end with `FunctionModel` issuing the two `run_code` calls the README's trace shows, and `CodeMode` actually running the emitted Python through Monty. What gets faked vs. what's real: - The Hacker News MCP toolset is replaced with a `FunctionToolset` of fake functions returning canned data from the README's linked public trace. Avoids depending on an MCP package or hitting the network. - `WebSearch(builtin=False, local=...)` skips the default DuckDuckGo fallback so the test doesn't pull `ddgs`. - `FunctionModel` drives the conversation through two `run_code` calls and a final synthesis, mirroring the production trace. - `CodeMode` itself is real; the `FunctionModel`'s emitted Python runs through the Monty sandbox and dispatches calls back to the fakes. Asserts on the call shape (each feed fetched once in parallel, follow-up calls target the expected winner id) and on the final synthesis content, so any future change in pydantic-ai or the harness that breaks how these capabilities compose makes this test fail. * Address review on PR #226 - Quick start: list `CodeMode` first in `capabilities=[]` to match the ecosystem agent's ordering and to lead with the headline capability. - Test: wire the fake HN tools through `MCP(local=hn_toolset, builtin=False)` instead of bypassing via `toolsets=[hn_toolset]`. Same composition path as production minus the network. Needs a narrow `# pyright: ignore` because MCP's `__init__` signature narrows `local` to MCP-shaped types but the parent `BuiltinOrLocalTool` accepts any `AbstractToolset` at runtime. - Test: have the second `run_code` call `web_search` (the WebSearch capability's local fallback) instead of `hn_search_content`, so the fake function is actually exercised. This fixes the coverage failure on lines 231-232 and makes the test more representative of the README example, which uses WebSearch. * Use snapshot()+dirty-equals for README quick-start test * Re-export dirty-equals via tests/conftest, restructure tests/ Two related cleanups based on review: - Re-export `IsDatetime` / `IsNow` / `IsStr` / `IsPartialDict` from `tests/conftest.py` with `TYPE_CHECKING` shims that pretend the matchers return the concrete type (`-> str`, `-> datetime`, etc.). Mirrors pydantic-ai's own conftest pattern and lets pyright strict accept `tool_call_id=IsStr()`, `timestamp=IsDatetime()`, etc. without the file-level `# pyright: reportArgumentType=false` we had before. Tests now `from .conftest import ...` instead of importing directly from `dirty_equals`. - Rename `tests/_code_mode/` -> `tests/code_mode/` (drop the underscore prefix; matches the source package `pydantic_ai_harness/code_mode/`). Updates the convention note in AGENTS.md/CLAUDE.md. - Move `test_readme_quick_start.py` up to `tests/` top level. It's an end-to-end check on the README's flagship example -- it composes `CodeMode` + `MCP` + `WebSearch` -- so it doesn't belong in any single capability's test directory.	2026-05-08 18:05:42 -06:00
Douwe MaanandGitHub	cd28cf24d1	ci: harness-compat reusable workflow + drop pydantic-ai-slim git source; cover approved-tool ApprovalRequired re-raise (#223) ... * test(code_mode): cover approved-tool re-raising `ApprovalRequired` pydantic/pydantic-ai#5275 documents on `_resolve_single_deferred` that a re-raised `CallDeferred` / `ApprovalRequired` from the post-approval tool body bubbles up without re-invoking the handler. The harness's existing `except (CallDeferred, ApprovalRequired)` clause then converts it to a `UserError` → `MontyRuntimeError` → `ModelRetry`. Lock that contract in with a regression test so we notice if the upstream behavior shifts. Refresh `uv.lock` to pick up #5275 from pydantic-ai main, which restored `wrap_validation_errors` on `ToolManager.handle_call`. No harness code change is needed — the existing call site already uses the kwarg. * ci: add `compat-test.yml` reusable workflow for pydantic-ai → harness compat checks Called from pydantic-ai's CI (via `workflow_call`) to verify that an arbitrary pydantic-ai ref doesn't break the harness's lint / typecheck / test suite. The workflow: - checks out harness@main and pydantic-ai @ the input ref (with optional `pydantic-ai-repo` for fork PRs) - rewrites `[tool.uv.sources]` to point `pydantic-ai-slim` at the local checkout, then `uv lock --upgrade-package` to refresh the lock - runs `ruff format --check`, `ruff check`, `pyright`, `pytest` Runs without secrets and with `contents: read` only — safe to invoke from fork-PR contexts (the calling pydantic-ai workflow gates fork PRs behind an approval label before invoking, as defense-in-depth). * chore: bump `pydantic-ai-slim` floor to `>=1.89.1` 1.89.1 ships pydantic/pydantic-ai#5275 (restored `wrap_validation_errors` on `ToolManager.handle_call`). Pinning the floor there so harness users on the broken 1.86–1.89.0 window get a clean resolver error instead of a runtime `TypeError: ToolManager.handle_call() got an unexpected keyword argument 'wrap_validation_errors'`. `uv.lock` is unchanged: `tool.uv.sources` already pulls pydantic-ai-slim from `main`, which is past 1.89.1. * ci: bump `test-floor` pin to match new `>=1.89.1` floor Companion to ae9428e: that commit bumped the floor in `pyproject.toml` but left this job pinning 1.80.0, which (a) no longer matches the documented floor and (b) would have started failing because 1.80.0 predates #5275 and the harness's existing `handle_call(wrap_validation_errors=...)` call site would TypeError against it. * ci+chore: drop `[tool.uv.sources]` for pydantic-ai-slim; auto-floor `test-floor`; simplify compat-test - Drop `[tool.uv.sources]` git override for pydantic-ai-slim so the lock resolves from PyPI like users would. The harness's `test` matrix and install path now mirror what users actually get; bleeding-edge compat with pydantic-ai's main is verified by pydantic-ai's `harness compat` job (called from its CI on PRs, main pushes, and tags). `uv.lock` refreshed. - Refactor `test-floor` to read the floor version from `pyproject.toml` at runtime instead of hardcoding it. Bumping the floor in pyproject is now the single source of truth — no separate CI pin to keep in sync. - Simplify `compat-test.yml`: with no `[tool.uv.sources]` to fight, the workflow just `uv sync`s the lock then `uv pip install --no-deps -e`'s the local pydantic-ai checkout. Same pattern test-floor uses, just pointing at a path instead of a PyPI version. * ci: switch `test-floor` to `uv sync --resolution lowest-direct`; install `pydantic-graph` from checkout in compat-test - Replace the bespoke "read floor from pyproject + pin slim only" Python script in `test-floor` with `uv sync --resolution lowest-direct`, matching pydantic-ai's `test-lowest-versions` job. Resolves *all* direct deps to their floors so the job exercises the full claimed compatibility envelope, not just slim's floor. - Add explicit floors for the previously-unfloored dev deps so `lowest-direct` doesn't drag pre-Python-3 versions: `pytest>=9.0.0`, `anyio[trio]>=4.11.0` (where the pytest plugin started registering the `anyio_mode` ini option), `coverage>=7.10.7`. Match pydantic-ai's discipline. `pytest-anyio` has only 0.0.0 on PyPI so no floor. - `compat-test.yml` now also `uv pip install -e`'s `pydantic-graph` from the local pydantic-ai checkout alongside slim. The two are sibling packages that version-track together; testing slim from-source against a PyPI `pydantic-graph` would mask cross-package issues (e.g. a slim PR depending on an unreleased graph change).	2026-05-05 16:02:56 -06:00
Aditya VardhanandGitHub	fb62396335	Support codemode dependency group alias (#224)	2026-05-04 14:22:55 -06:00
Douwe MaanandGitHub	fe9a587bd4	feat(code_mode): resolve deferred/approval-required tool calls via HandleDeferredToolCalls (#220) ... * feat(code_mode): resolve deferred tool calls via HandleDeferredToolCalls Tools with `kind='external'` or `'unapproved'` (and tools that raise ApprovalRequired/CallDeferred at runtime) are no longer excluded from the sandbox and promoted back to native tools. They now take the normal sandboxed path, and a HandleDeferredToolCalls capability on the agent can resolve them inline — so the model sees the resolved return value instead of having the deferral bounce out as a separate native tool call. - Remove the td.defer filter in _partition_callable_tools (no more native fallback for deferred tools). - Drop the native_fallbacks return value and the corresponding deferred-tool warning. - Update the sandbox UserError message when no handler is configured to point users at HandleDeferredToolCalls. - Update the deferred_execution test to assert sandbox inclusion and the approval-retry test to match the new error message. Depends on pydantic/pydantic-ai#5142 landing and being released; once it does, bump the pydantic-ai-slim lower bound. * code_mode: record outcome='denied' on nested ToolReturnPart for handler denials When the `HandleDeferredToolCalls` handler denies a tool call, `handle_call` now raises `ToolDeniedError` (on pydantic-ai-slim once released) instead of returning the denial message as a plain string. CodeMode catches it, records a `ToolReturnPart(outcome='denied')` in `nested_returns` so message history reflects the denial correctly, and re-raises so the sandbox surfaces the denial as an exception rather than as what would look like a successful tool return. The `ToolDeniedError` import is gated behind a compat shim so this module still loads against the currently released pydantic-ai-slim (which lacks the exception); the shim resolves to a placeholder class that never matches a real exception, leaving the except clause inert until a release ships `ToolDeniedError`. Depends on pydantic/pydantic-ai#5142. * code_mode: handle denials via `ToolDenied` return value, not exception `ToolManager.handle_call` no longer raises a (now-removed) `ToolDeniedError` on handler denial — it returns the `ToolDenied` value the handler produced. Drop the compat shim, import `ToolDenied` directly, and switch the dispatch to inspect the return value: record the denial as `outcome='denied'` on the nested `ToolReturnPart` and raise a `RuntimeError` inside the sandbox so the script can't mistake the denial message for a regular string return. * Bump pydantic-ai-slim lock + cover denial path Now that the slim PR has merged to main, refresh the lockfile to pick up the `HandleDeferredToolCalls` capability and `handle_call`'s `ToolDenied` return value. Add a denial test that asserts the denied-call flow surfaces as `ModelRetry` with the original denial message preserved in the trace. Notes on the test: - The handler returns `ToolDenied('nope')`; the harness records `outcome='denied'` on the nested `ToolReturnPart` and raises `RuntimeError` inside the sandbox. - The script doesn't catch the RuntimeError, so Monty surfaces it as `MontyRuntimeError`, which the harness converts back to `ModelRetry`. The retry message preserves the denial message so the model knows what went wrong. * ci: test against floor pydantic-ai-slim (1.80.0) in addition to main The default `test` matrix uses the `[tool.uv.sources]` override pinning slim to its main branch, so it never exercises the published-PyPI install path. Add a `test-floor` job that overrides slim to the lowest version declared in `pyproject.toml` (>=1.80.0) and runs the test suite, so we catch any accidental dependency on unreleased slim features in code paths that should be backward-compatible. Gate the new HandleDeferredToolCalls denial test with `pytest.skip` when the capability isn't importable — currently the only test that requires a post-1.80.0 slim, but the pattern can be reused if more land later. * fix coverage: pragma the floor-only skip in the denial test The `except ImportError → pytest.skip` branch only fires when running against the slim floor (1.80.0) where `HandleDeferredToolCalls` doesn't exist yet. The default test matrix runs against slim main, so coverage counted those two lines as uncovered. Mark the branch `# pragma: no cover` since it's an explicit skip path that the floor-slim CI job exercises but isn't included in the coverage report (the floor job doesn't gate on coverage by design).	2026-04-24 18:57:49 -06:00
3 people	9b1a121a92	Fix Monty's snapshot resume API (#216) ... * fix(code-mode): match Monty's current snapshot resume API * bumping monty * single quotes * uv.lock * docstring: use single backticks per project convention Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * revert unrelated test docstring rewording Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-21 19:50:02 +05:30
3 people	987b7312d5	Rename package to pydantic-ai-harness (#207) ... - Package name: pydantic-harness -> pydantic-ai-harness - Python module: pydantic_harness -> pydantic_ai_harness - Display name: Pydantic Harness -> Pydantic AI Harness - All URLs updated to github.com/pydantic/pydantic-ai-harness Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-13 14:33:28 -06:00
3 people	13841dd82e	Bump pydantic-ai-slim to >=1.80.0, fix memory community link (#200) ... Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-10 18:08:08 -06:00
4 people	729a28a9fd	Feature: Code Mode Capability with Monty (#167) ... Co-authored-by: Aditya Vardhan <adtyavrdhn@gmail.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-10 16:57:29 -06:00
Douwe MaanandClaude Opus 4.6	54feb9a815	Set up repo infrastructure: CI, docs skeleton, conftest, dependency management ... - Update pydantic-ai-slim dependency to >=1.76.0 (published capabilities API) - Replace pytest-xdist with anyio/pytest-anyio for async test support - Add pyright executionEnvironments for tests (reportPrivateUsage=false) - Add anyio_mode=auto to pytest config - Create __init__.py with docstring and empty __all__ ready for capability imports - Create tests/conftest.py with shared fixtures (TestModel, test agent, tmp_dir, allow_model_requests) - Update test_placeholder.py to exercise conftest fixtures for 100% coverage - Create docs/ skeleton with index.md and capabilities/index.md - Update CI workflow with Python version matrix (3.10, 3.12, 3.13), uv caching, and permissions - Rewrite README with installation, quick start, and capability table Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-02 05:22:43 +00:00
David Montague	74c18109a3	Use uv-dynamic-versioning to derive version from git tags ... No need to manually keep version in pyproject.toml in sync with tags — the version is now determined automatically from git, matching pydantic-ai.	2026-03-22 20:39:28 -06:00
David Montague	1b823c3fce	Add Python package skeleton with CI/CD release pipeline ... Set up pydantic-harness as a publishable Python package using hatchling, with CI (lint, test across Python 3.10-3.13) and automated PyPI releases on tag pushes via OIDC trusted publishing — mirroring pydantic-handlebars.	2026-03-22 20:35:16 -06:00