mirror of
https://github.com/pydantic/pydantic-ai-harness.git
synced 2026-07-21 02:45:34 +00:00
main
19
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Aditya VardhanandGitHub
|
4ad83f8861 |
Clarify CodeMode final-expression returns (#370)
* Prevent empty CodeMode results through clearer guidance * Make CodeMode return guidance unambiguous |
||
|
David SFandGitHub
|
3ba9e2f9a5 |
docs: capability pages for the unified docs site + README/doc parity gate (#329)
* docs: publish capability docs to the unified site + add README/doc parity gate Every capability shipped only a README (kept for GitHub/PyPI). This adds a parallel, cleaned-up page per capability under docs/ for the new unified docs site (pydantic.dev/docs/harness), migrated from each README: snippets verified runnable against source, autodoc API blocks, root-relative Pydantic AI links, and an experimental-status admonition on the experimental set. To keep README and doc in sync going forward, adds a docs-parity-reviewer agent and a parity gate in the review checklist (run as the last step before merge), plus the docs/ layout and the README<->doc requirement in AGENTS.md and the capability-authoring guide. * docs: fix README<->doc<->source inconsistencies across capabilities A parity audit against source found drift, mostly in the capability READMEs (staler than the migrated docs). All fixes verified against source: - Correctness: the "approval/deferred tools are excluded from the sandbox" claim (code_mode README + doc) was false -- those tools are sandboxed like any other; corrected in both. The stale Shell persist_cwd sentinel description is replaced with the actual out-of-band temp-file capture. filesystem protected default `.git/` -> `.git/*` (the bare form never matched). - Runnable snippets: added the missing imports/wiring so README snippets no longer raise NameError (subagents, context, planning, overflow, authoring, filesystem, code_mode). - Parity: documented previously-undocumented params/behaviors (compaction strategy options, overflow strip_ansi/Passthrough, extra autodoc classes for context and subagents), fixed a stale version pin (>=1.95.1 -> >=2.1.0), and added the missing Managed Prompt row to the root README capability matrix. - Style: normalized decorative Unicode to ASCII across all READMEs and dropped a hype phrase, matching AGENTS.md writing style and the docs. * docs: add nav.json to drive the unified-docs harness sidebar The unified docs mount the harness docs under /docs/ai/harness (fed live from this repo via the pydantic-ai 'Pydantic AI Harness' section). This nav.json defines the sub-nav (Overview + Capabilities + Experimental) and the set of doc files the site includes. * docs: migrate "What goes where?" explainer into harness overview Adds the core-vs-harness boundary section (anchor #what-goes-where) to the canonical harness overview, so the pydantic-ai docs that link to it can point here after the duplicated in-repo stub is removed. * docs: address CodeRabbit review -- runnable snippets, accuracy, multi-class autodoc * docs: flatten harness nav and align with graduated capabilities Following the experimental-graduation refactor (#347), restructure the unified-docs harness pages: - Flatten docs/ (drop capabilities/ and experimental/ subdirs); the sidebar is now Overview + one flat list per Douwe's request. - Rename to match the graduated modules: overflow -> overflowing-tool-output, authoring -> runtime-authoring, docs -> pydantic-ai-docs. - Drop the 'Experimental' admonitions from the graduated capabilities and repoint every import + ::: autodoc path off pydantic_ai_harness.experimental. - Add docs for the newly-shipped capabilities: guardrails, dynamic-workflow, media, and acp (acp stays framed as experimental -- it may still be removed). - Every capability doc now links to its source; index capability table lists the full set with flat links. * docs: apply team-sync authoring rules + enforce them in CI From the 2026-07-10 docs review on #329: - Purpose-first leads: drop hook names (before_model_request, after_tool_execute) from the opening paragraphs of compaction and overflowing-tool-output (doc + README); mechanism moves lower. - Mirror the soft 'API may change between releases' stability note from each graduated README into its doc page (ACP keeps its stronger experimental warning; guardrails' README has no note, so its page gets none). - README H1s now use the capability's display name (Overflow capability -> Overflowing Tool Output, RuntimeAuthoring -> Runtime Authoring, SubAgents -> Subagents, etc.). - Extend tests/test_docs_parity.py with per-page mechanical checks: source link present, heading matches the capability name, purpose-first lead (no hook in the opener), and no experimental framing on graduated pages (ACP excepted). - Update the docs-parity-reviewer agent + review-checklist to the flat structure and the new semantic checks. * docs: add the stability note to guardrails (parity with sibling capabilities) guardrails was the one graduated capability whose README and doc page lacked the shared 'API may change between releases' note. Add it to both. * fix: restore uv.lock to match pyproject (bad text-merge dropped 8 lines) Merging origin/main did a git text-merge of the generated uv.lock, leaving it inconsistent with pyproject.toml -- every CI job failed at 'uv sync --locked'. pyproject.toml is identical to main here, so the correct lock is main's. * docs: address CodeRabbit review on #329 Findings that failed to post inline (GitHub error) but were real: - context/README.md, planning/README.md: two nested examples still imported from pydantic_ai_harness.experimental.* -- repoint to the graduated modules. - guardrails/README.md: replace em dashes with '--' (repo style) and add the source-module link. - docs/media.md: standardize on the implementation's canonical media+sha256:// URI scheme (was mixing media://). - tests/test_docs_parity.py: strengthen my own checks per review -- source-link and top-README-link now require a real Markdown link to the page's specific module (not a bare substring); heading checks assert an H1 exists and equals the expected capability name via explicit page metadata. * fix: restore uv.lock [options.exclude-newer-package] block The lock lost its [options.exclude-newer-package] manifest (pydantic-ai-slim = false, ...) -- a bad git text-merge dropped it, and diagnostic uv commands rewrote it under a different local config. Without that block CI's 'uv sync --locked' re-resolves and fails ('addition of exclude newer exclusion for pydantic-ai-slim'). Restore origin/main's exact lock. * fix: restore uv.lock [options.exclude-newer-package] block A pre-commit hook was rewriting uv.lock under the local uv config, stripping the [options.exclude-newer-package] manifest (pydantic-ai-slim = false, ...). Without it CI's 'uv sync --locked' re-resolves and fails. Commit origin/main's exact lock with --no-verify so no hook mutates it (lock-only change). * test: cover the docs-parity helper edge cases (100% coverage) The strengthened helpers added defensive branches (missing frontmatter close, fenced code before the lead, missing/forbidden/ClassName H1, lead running to EOF) that no real doc exercises. Add direct unit tests so the file is back to the repo's required 100% coverage. * docs: link every capability README to its source module + enforce it CodeRabbit re-flagged planning/README.md for a missing source link. Only guardrails had one, so add the source-module link to all 15 remaining capability READMEs (matching the doc pages) and add a parity test so the requirement is mechanical and cannot silently regress. * docs(agents): drop stale folder tree; fix flat docs path + guard names AGENTS.md's File-structure ASCII tree and capability-authoring's doc paths still showed docs/capabilities// docs/experimental/ (flattened in this PR) and the old /docs/harness URL. Delete the tree rather than redraw it -- the layout is discoverable by listing the repo; keep only the non-obvious conventions (flat docs/, the README<->doc parity requirement). Also fix the Vocabulary guard examples (InputGuard/OutputGuard, not the nonexistent InputGuardrail/ CostGuard). * test: statically validate doc snippets exist and parse Every Python snippet in the capability READMEs and docs/*.md pages is now checked for the two failures a reader hits immediately: it does not parse (syntax), or it imports a pydantic_ai_harness symbol that does not exist (stale module path or renamed name -- the class of bug behind the experimental.* import drift). Static only: no model/network execution, so it needs no mocking. The four illustrative API-signature blocks opt out with a {test="skip"} fence (read by pytest-examples, stripped-safe for the unified-docs render). * test: don't fail doc-snippet check on a missing optional extra The static check imported capability modules to resolve their symbols, but in the slim CI job (no extras) importing e.g. pydantic_ai_harness.experimental.acp raises ModuleNotFoundError for the absent third-party 'acp' package -- the harness module exists, its extra just isn't installed. Distinguish a genuinely missing harness module (fail) from a missing extra (skip) by the ImportError's module name. |
||
|
|
545a703227 |
feat: Adds Dynamic Workflows (#273)
* feat(dynamic-workflow): orchestrate sub-agents from a model-written script Turn-by-turn sub-agent delegation makes every composition step (chain, vote, loop) a separate model turn and pushes each intermediate result back through the orchestrator's context. DynamicWorkflow is Code Mode with sub-agents as the callables: the model writes one Monty-sandboxed script that composes the sub-agents with ordinary control flow, and only the final result returns to the model. Extracts the shared Monty execution loop into _monty_exec, reused by code_mode. The pydantic-ai bump the capability needs (defer_loading, ctx.enqueue) lands separately, per the repo policy on dependency changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(deps): bump pydantic-ai 1.95.1 -> 1.105.0 DynamicWorkflow needs defer_loading (>=1.97) and ctx.enqueue (1.105) for on-demand loading and cache-stable runtime sub-agent reveal. The lock exempts the pydantic-ai family from its exclude-newer cutoff to pull the newer release. Collateral: xfail a code_mode ToolSearch test that relied on the old discovered-tool tracking, and regenerate the managed_prompt snapshot for the new tool-definition fields. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(examples): tidy example docs and keep them fully linted Drop the blanket 'examples/**' pydocstyle ignore — it masked a single missing docstring. Add the one-line docstring to fork_and_resume's main() instead, so examples stay under the same docstring rules as library code. Also drop the redundant "Runs / needs a key" column from the examples README. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(dynamic-workflow): render catalog via core FunctionSignature, reject keyword names Addresses PR #273 review. Render the sub-agent catalog through pydantic_ai's FunctionSignature (the same renderer code_mode uses) instead of a hand-rolled f-string. This is the real dedup with code_mode and fixes two latent bugs: the signature now forces keyword-only `task` (`async def name(*, task: str)`) to match dispatch, which reads kwargs['task']; and a description containing a newline or quote now renders as a safe docstring block instead of breaking the f-string. Reject sub-agent names that are Python keywords. `'class'.isidentifier()` is True, so a keyword name passed validation but the model could never call it (`await class(...)` is a syntax error) — a silently-uncallable agent. Validate non-keyword identifiers at construction, via a shared in-file helper. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(dynamic-workflow): harden public API ahead of release Adversarial review of the dynamic-workflow surface surfaced contracts that would be painful to change once released; fix them while cheap: - Correct the "usage_limits bounds the tree" claim — the parent's limit is never forwarded to sub-agents. Add `sub_agent_usage_limits` so a per-sub-agent limit plus `max_agent_calls` give a real tree-wide token ceiling, and document what `forward_usage` actually does. - Own `WorkflowResourceLimits` instead of leaking pydantic_monty's `ResourceLimits` onto the public field; add an explicit `'unlimited'` sentinel and merge a partial dict onto the backstop (no silent drop). - Narrow `agents` to `list` so the append-to-reveal contract is enforced by the type rather than explained in prose. - Drop the `description` -> `agent.name` fallback that rendered a useless self-referential docstring to the model. - Teach the model-facing tool description what the README already knew: keyword-only `task`, no `return_exceptions`/abort-on-raise, dict (not attribute) access for structured output, and the print envelope. Also refresh the runnable examples (audit + migrate) and keep the README and tests in sync. lint/typecheck clean, 100% branch coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * removing time limit from monty * docs(examples): consolidate into one verified DynamicWorkflow showcase Replace the overlapping audit and migrate examples with a single example that exercises the full capability range — parallel fan-out, read/write confinement, an adversarial review, a feedback loop, typed fan-in, and a Logfire trace — so a reader sees what the capability is for without wading through two near-identical fan-out demos. Fixes issues an adversarial review surfaced: the retry loop now threads reviewer issues back to the migrator (so it can actually converge), the synthesizer's typed report survives to the caller via the orchestrator's output_type, and several claims (budget ceiling, confinement, run command portability) are corrected to match the code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(deps): raise pydantic-ai-slim floor to 1.101.0 for RunContext.enqueue The dynamic_workflow reveal path calls ctx.enqueue to announce a sub-agent added mid-run, but RunContext.enqueue did not exist until pydantic-ai 1.101.0 (verified: absent in 1.100.0, present in 1.101.0). The declared floor was still >=1.95.1, so the test-floor job (--resolution lowest-direct) and any install at the floor would resolve a RunContext without enqueue and raise AttributeError when revealing an agent instead of announcing it. Bump the runtime dep, the logfire extra, and the dev group to >=1.101.0, and update the README requirement line to match. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(dynamic-workflow): move capability under experimental DynamicWorkflow ships as experimental so its API can change without a deprecation period, like the compaction menu. Importing it now emits a HarnessExperimentalWarning, silenceable category-wide. _monty_exec stays at the package root: it is shared with code_mode, which is stable, so it must not move into experimental. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(dynamic-workflow): correct stdlib import guidance for the sandbox The tool description and README presented `asyncio, math, json, re, typing` as the complete set of importable modules. In Monty, `datetime`/`os`/`sys` also import, but they are inert in DynamicWorkflow (no os/path hooks are registered, so their calls resolve to NameError). Reword to a curated, non-exhaustive list so the model-facing text is accurate without steering toward the stubbed dead ends. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(dynamic-workflow): harden sandbox boundary and reveal/config validation Adversarial review of the run_workflow path surfaced several silent-failure and crash modes, all reachable from model-authored scripts or host config: - A model script awaiting the same sub-agent call twice in one asyncio.gather made the Monty VM raise pyo3_runtime.PanicException (a BaseException), which escaped both Monty error handlers and tore down the whole agent run. Catch it by name and convert to a retry; non-panic BaseExceptions still propagate. - Sub-agent calls silently dropped extra keyword args and accepted a non-string task (a dict/list was smeared into message parts), running on wrong input with no signal. Reject both at dispatch, before the budget is touched. - A runtime reveal whose name was invalid or collided with an existing agent was silently dropped, keeping the old agent while the host believed it had swapped one in. Distinguish idempotent re-reveal (object identity) from a real conflict, warn once on the latter, never swap silently. - max_agent_calls < 1 produced a dead-on-arrival tool with a misleading "conclude with results gathered" message; a typo'd resource_limits key was silently ignored, disabling the only guard against a pure-CPU loop. Both now raise UserError at construction. Also pins the headline guarantee with a concurrent fan-out test asserting max_agent_calls admits exactly the budget under asyncio.gather (previously only covered sequentially). _toolset.py stays at 100% line+branch coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: await cancelled dispatch tasks and survive bad mid-run reveals Cancelling a Monty execution previously fired task.cancel() and returned immediately, leaving in-flight tool calls and sub-agent runs unwinding in the background — they could mutate the shared usage accumulator after the tool call had already reported its result, and teardown errors were never retrieved. The executor now awaits the cancelled tasks so nothing outlives the call that spawned it. This changes code_mode's cancellation timing too (flagged in the PR description): teardown now waits for in-flight calls to unwind instead of abandoning them. Also: create the sequential dispatch coroutine after the pending-task barrier so cancellation at the barrier cannot leak a never-awaited coroutine, and make `for_run` index the agent list leniently — an invalid WorkflowAgent appended mid-run (which `_reveal_pending` deliberately tolerates) no longer hard-fails every subsequent run via `replace`'s re-validation. A new test pins that host-raised exceptions cannot be caught inside the sandbox: the budget-exhausted terminal result relies on it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(dynamic-workflow): document experimental status and actual return shapes The README never said the import warns or how to silence it, omitted the print()/None result shaping the tool actually performs, and the API block hid `description` and the WorkflowResourceLimits type. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(dynamic-workflow): cover deps forwarding, message isolation, and shared-counter limits An adversarial tests-vs-spec pass found three documented behaviors with no assertion behind them: parent deps reaching sub-agents, sub-agents never seeing the parent conversation, and sub_agent_usage_limits checked against the shared counter under forward_usage=True. Dropping any of them would have passed the suite. Also pins the documented backstop values and the served tool's max_retries, asserts the reveal announcement names the actual tool, and fixes two comments that described a duration backstop / a dataclasses.replace that don't exist. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(dynamic-workflow): record the core usage_limits TOCTOU race behind max_agent_calls The exact host-side counter exists because core's limit enforcement races under concurrent fan-out and RunContext can't forward the parent's limits; pin that context (and the upstream TODO) where the discipline lives so it isn't lost once the PR ships. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(examples): replace the generated example with a feature-led README Review feedback on #273: the example will be rewritten by hand, and the README should lead with the Anthropic dynamic-workflows post and the feature itself rather than narrating one example run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(dynamic-workflow): add a dedicated install extra Installing DynamicWorkflow's sandbox dependency via the code-mode extra conflated two capabilities; each now names its own extra (both resolve to pydantic-monty). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(code_mode): convert sandbox VM panics to retries A model-written script that awaits the same call twice in one asyncio.gather panics the Monty VM; pyo3 surfaces that as a BaseException that nothing caught, tearing down the whole agent run. DynamicWorkflow already guarded this; the guard now lives in _monty_exec and code_mode applies it too, dropping its REPL since the aborted VM's accumulated state can't be trusted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(dynamic-workflow): remove duplication that could drift Single-source the long-form field docs on the capability (the toolset docstrings now point there), derive the resource-limit key set from the TypedDict instead of a parallel frozenset, and freeze the rendered tool description itself rather than a catalog dict re-rendered every step -- making the cache-stability claim structural. Drops the for_run_step override that duplicated the base default, plus redundant tests and a stateless test fixture. No behavior or public API change; coverage stays at 100%. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: narrow the shared executor seam and drop dead type unions Name sanitization is a code_mode concern, so the mapping moves into its dispatch closure and MontyExecutor stops knowing about it. The empty pending_call_ids guard fell through to an identical resume call, so it goes. Concrete return types on get_toolset/for_run let tests drop four isinstance asserts that existed only to narrow types pyright can now see directly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: fix a comment stating a false invariant; unshadow a closure param _by_name is mutated during a run (reveals fold into it each step); the stable-registry invariant only holds while a script executes, which is what the executor relies on. The dispatch closure's name param shadowed call_tool's own, a future-edit hazard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(dynamic-workflow): restructure README for faster comprehension Lead with a minimal runnable snippet and state the context-saving idea once instead of repeating it across Why/What/blockquote. Move Installation above the full example. Turn the sub_agent_usage_limits paragraph into a forward_usage True-vs-False table so the N*T hard ceiling and the best-effort concurrency caveat are scannable. No technical claims changed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(dynamic-workflow): address CodeRabbit review - Fix broken relative link to Code Mode README: the capability lives two levels deep under experimental/, so `../code_mode/` resolved to a non-existent experimental/code_mode/. Use `../../code_mode/`. - Add missing `-> None` return types to all four test_warnings methods (the bot flagged one; the whole file was missing them). - Narrow `_workflow_result`'s unconstrained passthrough from `Any` to `object`. Left the other flagged `Any`s in place: the `call` override and `dispatch` must match `AbstractToolset.call_tool` / `_monty_exec.DispatchFn`, and `AbstractAgent[..., Any]` carries a heterogeneous sub-agent output type that `object` would reject on assignment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(dynamic-workflow): harden error surfaces, render output schemas, add reveal() API Findings from an end-to-end adversarial review (PR #273 branch), fixes authored with codex exec: - Serialize sub-agent output inside the dispatch error wrapper: a non-serializable output leaked host class paths into the retry text and failed identically on every retry. - Render each sub-agent's output schema in the catalog and reveal announcements (TypedDict-style, with cross-catalog type-name conflict handling). The model previously saw `-> Any` and had to be told the fields in prose. - Add DynamicWorkflow.reveal() as the supported runtime reveal API; list append remains the underlying mechanism. An aliased-list-only contract cannot survive the planned durable resume. - Return terminal {'error': ...} results for permanent conditions (nesting refusal, budget exhaustion) instead of ModelRetry, which burned retries into UnexpectedModelBehavior. The budget result now carries last_error and completed. - Salvage completed sub-agent results into retry messages and the budget-terminal dict so a retry can reuse them instead of re-spending the budget; the sandbox-panic path does the same. - Pass through model-safe exception messages (UsageLimitExceeded) so the model can react; keep the opaque form for everything else. - State the call budget and sub-agent statelessness in the tool description; warn instead of failing when a reveal has no pending message queue to announce into. - Docs: correct the N*T "hard ceiling" claim (core checks token limits after a response arrives), fix the examples README script (it subscripted str outputs), pin model-facing text with snapshots. The old "budget error cannot mask another failure" comment was disproven under asyncio.gather; the true invariant is documented. Refs: https://www.anthropic.com/engineering/code-execution-with-mcp, https://code.claude.com/docs/en/workflows Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(dynamic-workflow): align tests with pydantic-ai v2 deps typing Merging main raised the pydantic-ai-slim floor to >=2.1, whose Agent `deps_type` default became `object` (was `NoneType`). Constructing an `Agent(..., capabilities=[Cap[None](...)])` then fails overload matching: the capabilities pin `AgentDepsT=None` while the default supplies `object`. Switch the deps typevar to `object` throughout, matching the convention the code_mode and subagents tests already use. Also make test_max_agent_calls_exact_under_concurrent_fan_out order- independent. The completed list was asserted in exact order, but which three of eight concurrent calls win the budget and the order they finish are both scheduler-dependent (3.14 admits them in a different order than 3.13). Assert the invariant that matters: exactly the budget ran, each once, well-formed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: stop tracking .agents/scheduled_tasks.lock This is a machine-local runtime lock for Claude Code's scheduled-task runner (pid/session mutex), not project content. It was committed by accident. Remove it from the tree and gitignore it alongside the other .agents machine-local files. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(dynamic-workflow): accept raw agents, make reveal() the strict mutation API Pydantic AI 2.x agents carry their own name and description, so wrapping every entry in WorkflowAgent was boilerplate: agents=[reviewer, summarizer] now works, following the OpenAI-handoffs union pattern where a raw agent means "use the agent's own metadata" and the wrapper is a per-use-site override. WorkflowAgent gains the agent.description fallback and positional agent that the sibling SubAgent already had; the divergence had no rationale. Making reveal() the only post-construction channel (eager validation, UserError at the call site, internal catalog list) removes the reason the toolset needed lenient re-validation: the strict/lenient _rebuild duality, _reveal_warned, and the warn-and-skip paths are deleted, and an entry that reaches the toolset invalid now fails fast as a contract violation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(dynamic-workflow): rewrite README as a progressive tutorial The reference-style README front-loaded precise semantics before a reader had a mental model to hang them on. Reshape it into a learning arc -- smallest working example first, one concept per section, dense tables tucked into admonition boxes -- so the same facts land in the order a reader needs them. All technical claims are carried over unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(dynamic-workflow): use raw-agent shorthand in the examples README The migration example wrapped each sub-agent in WorkflowAgent solely to supply a catalog description, which read as pre-refactor mandatory wrapping and contradicted the prose above it ("documented by its description"). An agent's description is a property of the agent, not of one workflow, and the field has other consumers (agent spec, OTel gen_ai.agent.description), so its natural home is the Agent. Move the descriptions onto the agents and pass them raw; WorkflowAgent stays documented in the capability README's override section for the cases that need it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(dynamic-workflow): drop internal decisions log from the PR DYNAMIC_WORKFLOW_DECISIONS.md was an internal working-notes log (branch codenames, v1/maybe-v2 planning, open problems). It doesn't belong in the shipped repo and duplicates the capability README. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: fix pydantic-ai-slim version pin and tighten DynamicWorkflow docstrings Correct the required version in README.md to `pydantic-ai-slim>=2.1.0` (matches pyproject.toml; the previous `>=1.101.0` was wrong), and trim the verbose DynamicWorkflow capability/field docstrings to say the same thing more concisely. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(dynamic-workflow): promote out of experimental DynamicWorkflow graduates to a released top-level capability after a hardening round driven by live model runs and adversarial review: - Live-exercised with a real Opus orchestrator across ten scenarios (fan-out, structured output, budget exhaustion, error salvage, runtime reveal, defer_loading, nesting refusal, open-ended research). The one model-facing trip point found -- reading the sub-agent call budget as per-script rather than per-run -- is fixed in the tool description. - Reveals on a deferred capability no longer leak: get_tools holds reveal folding and announcements until the capability is loaded. Ownership is resolved by identity through the run's capability registry, unwrapping wrapper chains, because ids cannot identify the owner (an id-less capability registers under a generated key, and a wrapper registers in place of what it wraps). - Public annotations erase sub-agent outputs as `object`, not `Any`. - README drops the experimental banner, documents the per-run budget and keyword-only construction, and loses a stray editing artifact that had been committed at the end of the file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(dynamic-workflow): let sub-agents inherit the run model * test(dynamic-workflow): exclude never-run parent model fn from coverage The parent model callback in test_inherit_model_off_keeps_sub_agent_bound_model is asserted never to run, so its body is unreachable and broke fail-under=100. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(dynamic-workflow): merge into CodeMode when both are on one agent With both capabilities registered, CodeMode used to fold run_workflow into the run_code sandbox as a plain function taking a code string, so the model had to write a script containing a second script as a string literal, run in a nested sandbox, with no return schema. The two now compose instead: when a CodeMode capability is present, DynamicWorkflow exposes each sub-agent as its own (task) tool carrying the agent's output schema, and CodeMode renders them as typed async functions inside run_code, callable alongside the agent's regular tools in one script. Budget, usage forwarding, and the nesting guard moved into a shared call path so both modes enforce the same rules; the sub-agent guidance run_workflow's description carried moves into instructions in merged mode. No new public API: the merge is automatic. Standalone run_workflow scripts are now statically type-checked against the sub-agent signatures before execution (every call is a fresh sandbox, so the check is always sound); a misspelled function or positional task costs a retry but no sub-agent budget. The runtime kwarg guards remain, because Any-typed values (e.g. json.loads results) evade the checker, and the tests now exercise them through exactly that route instead of reaching into private helpers. Also: budget-sharing across multiple run_workflow calls in one run gets a test (the tool description's headline claim was untested), the examples/ README folds into the capability README, and the unrelated .gitignore line is dropped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: explain DynamicWorkflow in the main README The main README showed CodeMode end to end but only gave DynamicWorkflow a matrix row, so a reader had no way to see the point of the capability without opening its package README. Add a compact section after the Quick start: the turn-by-turn delegation problem, the two-line setup, the script the model writes, and how it composes with CodeMode, budgets, and deferred loading. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(dynamic-workflow): move under experimental New capabilities start under pydantic_ai_harness.experimental per repo convention, and the planned extensions (structured sub-agent inputs, durable workflows) touch the sub-agent call contract, so the API needs room to move before names go stable. This reverses the earlier top-level promotion on top of everything built since: importing the capability now emits HarnessExperimentalWarning and there is no top-level export. The README additionally explains the experimental status, positions DynamicWorkflow against SubAgents (one-delegation-per-turn vs scripted choreography), and documents the merged-mode tool-name collision. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(dynamic-workflow): separate from CodeMode instead of merging The CodeMode merge coupled two independent capabilities, and when their composition did not line up -- a restrictive CodeMode `tool_selector`, or running out of the sub-agent budget mid-script -- it degraded in ways the merge could not recover cleanly. Drop the merge: DynamicWorkflow always exposes its `run_workflow` tool and no longer inspects the run's capabilities. `run_workflow` is itself a code-execution sandbox, so CodeMode must not fold it into `run_code` -- that would make the model pass a script as a string argument to a function inside another script. CodeMode now keeps any tool carrying `code_arg_name` metadata native; `run_workflow` already declares it, so with both capabilities the model sees `run_code` and `run_workflow` as two independent tools. The rule only affects code-execution tools, so CodeMode is unchanged for every other tool. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(dynamic-workflow): correct max_duration_secs semantics and refresh README The docs claimed `max_duration_secs` counts wall-clock, including time awaiting sub-agents fanned out with `asyncio.gather`, and justified having no default cap on that basis. Empirically that is false across every monty the capability supports: the timer is a per-bytecode-step check, so it measures in-sandbox execution time and excludes time the script spends suspended on the host awaiting sub-agents (sequential or gathered). Correct the claim in `_toolset.py`, `_capability.py`, and the README, and describe what the cap is actually for: a pure-CPU `while True` runaway, the one thing the sub-agent budgets cannot catch. Add a regression test that pins the corrected behavior so the stale claim cannot silently return: slow sub-agents under a small cap still complete. Rewrite the README prose in a plainer voice with no em-dashes, and cite Jarred Sumner's Bun Zig-to-Rust rewrite as a real-world example of the same pattern (at Claude Code's session scale, not this specific API). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Aditya VardhanandGitHub
|
51ba5693bf |
Revert "fix(code_mode): tighten run_code prompt against phantom imports and…" (#317)
This reverts commit
|
||
|
|
bb6da9948d |
Keep framework tools native in CodeMode (#296)
* Keep framework tools native in CodeMode * test(code_mode): cover load_capability bootstrap through Agent The existing unit test exercises the native/sandbox split via a static toolset. Add an Agent-level test for the actual issue #276 scenario: a deferred capability plus CodeMode(tools='all'), asserting load_capability stays a native call and the deferred member tool keeps defer_loading=True instead of being folded into run_code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(code_mode): route framework tools native off tool_kind alone The name fallback (`name == _SEARCH_TOOLS_NAME`) was redundant: pydantic-ai has set `tool_kind` on the `search_tools` ToolDefinition since 1.95.0, and the dependency floor is 1.105.0, so `tool_kind is not None` already catches it. Routing keys purely off the framework discriminator now. Also switch the #276 bootstrap test from FunctionModel to TestModel (per the test guideline) and rescope its assertions to provider-agnostic facts -- the old assertion only held because FunctionModel's profile happens to support native tool search. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(code_mode): mark unreached deferred-tool body no cover The #276 bootstrap test keeps `demo_tool` hidden, so its body never runs. Mark it `# pragma: no cover` to satisfy the 100% coverage gate, matching the sibling deferred-tool tests. --------- Co-authored-by: Aditya Vardhan <adtyavrdhn@gmail.com> |
||
|
|
3f4cbd0a66 |
fix(code_mode): tighten run_code prompt against phantom imports and bad inputs (#295)
* fix(code_mode): tighten run_code prompt against phantom imports and bad inputs CodeMode agents repeatedly waste run_code turns on avoidable mistakes -- importing unavailable modules (textwrap) or a phantom `functions`/`sandbox_tools` module, pasting oversized one-line literals (unterminated-literal errors), indexing typed tool results as dicts, and echoing `...`/`??` truncation placeholders back as arguments. Each failure burns a model turn and pushes agents toward their request limits. The prompt now closes the importable-module allow-list (anything outside it fails; the listed functions are pre-bound, do not import them) and adds a short caution: no oversized literals, results are typed objects (read fields, don't json.loads or index a string), and never pass a truncation placeholder as an argument. Prose-only; 100% branch coverage retained. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(code_mode): make the run_code prompt replaceable via `instructions` CodeMode had no way to customize the run_code prompt -- you had to fork. Add an `instructions` parameter: a str replaces the built-in prose, or a callable receives the built-in (host-aware) prose and returns the replacement (append/prepend/rewrite). The auto-generated tool catalog is always appended afterward, so run_code stays callable regardless. This is the right seam for project-specific guidance instead of baking it into the shared default. Also drop the version-fragile module examples from the import-allow-list note (e.g. `collections` is unavailable today but could land in a future Monty); the list itself is already in the prompt, and the prose is now customizable anyway. 100% branch coverage retained; tests cover the str and callable forms (catalog preserved). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(code_mode): `instructions` is a plain str + export the default prose Drop the callable form of `instructions` (overkill). It is now `str | None`: a str replaces the run_code base prose, `None` keeps the built-in. To build on the default instead of replacing it, import the newly-exported `default_run_code_instructions()` and append/prepend/edit, then pass the result -- no callable needed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(code_mode): append via `instructions`, replace via `dangerously_replace_instructions` Keeps the Monty-tuned base prose by default so a customization can't silently desync `run_code` from the sandbox semantics. `instructions` now appends to the built-in prose (before the catalog); a separate, clearly-labeled `dangerously_replace_instructions` is the escape hatch for full replacement. The two are mutually exclusive (UserError on both set). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
e6dfd4c303 |
ci: add pydantic-ai v2 beta early-warning job and make code_mode v1/v2 compatible (#291)
* ci: add non-blocking pydantic-ai v2 beta test job The harness targets the pydantic-ai v1 line, but the `>=1.105.0` floor also admits v2 prereleases once prerelease resolution is enabled. Nothing in CI exercised that path, so v2-breaking changes were invisible until release. This adds a `test-v2-beta` job that resolves the latest v2 beta and runs the suite against it. It surfaces real breakage today: v2 dropped the `calls` argument from `ToolManager.get_parallel_execution_mode`, which CodeMode still calls with the v1 signature, so `code_mode` raises `TypeError` under v2. The job is intentionally kept out of the `check` gate so an expected red result on the unsupported v2 line never blocks a merge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: cap the v2-beta pytest step so a hang fails fast The v2 beta job is expected to surface breakage, but some v2 breaks hang instead of failing cleanly. When code mode raises an unhandled exception inside a DBOS workflow, DBOS's background recovery thread stays alive and the Python process never exits, so the step rode to the 20-minute job timeout and burned a full runner slot on every push. Wrap the pytest invocation in `timeout -k 30 300` so any such hang fails fast (exit 124). A per-test timeout plugin would not help: the hang happens during interpreter shutdown, after the test body completes, so the cap has to be on the process. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(code_mode): call get_parallel_execution_mode compatibly across v1/v2 pydantic-ai v2 (#5339, shipped in 2.0.0b1) dropped the `calls` argument from `ToolManager.get_parallel_execution_mode`: it now reads the run-scoped mode from a context var and applies per-tool `sequential` barriers separately. The harness passed `[]` specifically to isolate the context var from per-tool flags, which is exactly what the no-arg v2 call returns, so the two are equivalent. Inspect the method arity and call the matching shape. Inspecting rather than catching TypeError avoids swallowing a genuine TypeError raised inside the method. The `Callable[...]` annotation erases the bound signature so both call shapes typecheck whichever major's stubs pyright resolves. Without this, every code_mode run under v2 raises TypeError; inside a DBOS workflow that unhandled error also wedged the process (a non-daemon recovery thread blocked interpreter shutdown), which is what hung the v2-beta CI job. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(code_mode): cover both arities of the v1/v2 mode dispatch The arity-based dispatch added an `else` branch that the v1-pinned coverage gate never executes (the v2 no-arg call only runs under pydantic-ai v2), so total branch coverage fell to 99% and failed the `fail_under=100` gate. Extract the dispatch into `_global_mode_is_sequential` and unit-test both call shapes directly, so both branches are exercised whichever major is installed. This is honest coverage rather than a `# pragma: no cover` that would hide a branch that does run (in the v2-beta job). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: deselect v1-only OTel assertions from the v2-beta job Two instrumentation tests in test_managed_prompt.py assert pydantic-ai v1's OpenTelemetry attribute and span names. v2 deliberately renamed these (aggregated-usage attributes, GenAI-semconv span names), so the tests are expected-red on v2 and carry no signal in this job. Deselecting them keeps the v2-beta job a meaningful early-warning for capability breakage (code mode, durable execution) instead of going red on documented instrumentation drift. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: keep managed prompt v2 signal --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: David SF <david.sanchez@pydantic.dev> |
||
|
David SFandGitHub
|
caf50108fe | Explain code mode metadata selection (#292) | ||
|
|
2924277b17 |
Add CodeMode dynamic_catalog flag for cache-stable tool disclosure (#243)
* feat(code_mode): add CodeModeDynamicCatalog for cache-stable tool disclosure Moves CodeMode's per-tool signature catalog out of `run_code.description` (which lives in the prompt-cache-keyed tool-definitions block) and into agent instructions as a dynamic `InstructionPart`, then announces newly- discovered tools via `RunContext.enqueue` rather than by mutating the cached description. The result: the tool-defs block stays byte-stable across Tool Search discoveries / per-step toolset swaps, only the dynamic instructions and append-only system-prompt announcements grow. This is the Tier-2 reshape from pydantic/pydantic-ai-harness#232. Opt-in (default `CodeMode` keeps the catalog in `run_code.description`, which is slightly cheaper on prompt size when the toolset never changes); pair with `ToolSearch` or a churning toolset for the cache win. Depends on pydantic/pydantic-ai#4980 for the pending message queue; pinned to the `background-tools` branch of pydantic-ai-slim until that lands. Cross-provider cache safety completes once pydantic/pydantic-ai#5437 (XML-wrapped mid-conversation SystemPromptPart mapping) ships. * Re-lock pydantic-ai-slim to background-tools HEAD after main merge * Adapt to pydantic-ai refactor: PendingMessage.payload→request * Add e2e test exercising eager tool through run_code (closes coverage gap) * Fold CodeModeDynamicCatalog into CodeMode as a `dynamic_catalog` flag Replace the separate `CodeModeDynamicCatalog` capability (which surgically wrapped `CodeMode`'s assembled toolset) with a `dynamic_catalog: bool` flag on `CodeMode` itself. When set, `CodeModeToolset` keeps only the static base prose in `run_code.description` and surfaces the sandboxed-tool catalog as a dynamic `InstructionPart` via `get_instructions`, and `CodeMode` announces newly discovered tools by enqueuing a `SystemPromptPart` through `ctx.enqueue`. This drops the shim package and its dedicated test module, merging the behavior and its tests into `code_mode`. The search addendum now stays in `run_code.description` in both modes (it's cache-stable), an improvement over the prior wrapper which dropped it. * Enqueue discovery announcement as a bare SystemPromptPart Re-lock pydantic-ai-slim to the latest background-tools HEAD, which now accepts `ModelRequestPart`s directly in `enqueue` and renders mid-conversation `SystemPromptPart`s inline rather than hoisting them (pydantic/pydantic-ai#5509). Drop the `ModelRequest` wrapper workaround and enqueue the `SystemPromptPart` directly; on the wire it renders as an XML-wrapped user prompt, so the catalog cache stays intact across providers. * deps: re-lock pydantic-ai-slim; give test RunContext a live pending-message queue `RunContext.enqueue` now raises when the context has no pending-message queue (pydantic-ai removed the silent-drop path). Pass `pending_messages=[]` in the test helper so the discovery-announcement unit tests still exercise enqueue, and narrow `ctx.pending_messages` before reading it. Real runs wire the queue automatically. * code_mode: drop cast/Any from discovered-name extraction _extract_discovered_names walked the tool-search return with cast() over Any, which the no-typecast rule disallows. Validate the untrusted return through two lenient pydantic TypeAdapters instead: a malformed catalog yields [] and a malformed entry is skipped, matching the prior behavior with a fully-typed path. Also rewrite the one em-dash in the get_instructions docstring to --. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * code_mode: import TypedDict from typing_extensions for py<3.12 pydantic rejects `typing.TypedDict` on Python < 3.12 (PydanticUserError at TypeAdapter construction), which broke the 3.10/3.11 CI jobs. Match _toolset.py and use `typing_extensions.TypedDict`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: David SF <david.sanchez@pydantic.dev> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
c56a4b42f8 |
Add host-backed OS and filesystem access to CodeMode sandbox (#262)
* feat(code_mode): expose host-backed OS access to the sandbox Sandboxed `run_code` had no way to reach the filesystem, environment, or wall clock: Monty supports it through an OS callback / `AbstractOS` and directory mounts, but `CodeMode` never threaded `os`/`mount` into `feed_start` or the snapshot resume loop, so callers couldn't enable it. Add `os` and `mount` options on `CodeMode`/`CodeModeToolset`, thread them through `feed_start` and every `resume` site (OS auto-dispatch stops the moment a resume omits them), and make the `run_code` description reflect whether host-backed access is configured. * test(code_mode): harden OS-access tests around the threading invariants Add edge cases that pin the behaviours most likely to regress: OS access surviving across REPL-persisted `run_code` calls, a raising `os` callback degrading to `ModelRetry` instead of crashing the loop, and `mount` accepting a `list[MountDir]`. Hoist the never-invoked callback used by the description/forwarding assertions into one shared helper. * docs(code_mode): tighten and verify the filesystem/OS access section Trim the host-access docs to the essentials and make the example self-contained (drop the undefined helper). The snippet and the documented `mount`/callback constructions are run end-to-end to confirm they work. * docs(code_mode): correct per-request scoping wording `os`/`mount` are static capability fields (no per-run resolver), so the "stateful AbstractOS rooted at a per-user directory" guidance over-claimed. Reword to: build CodeMode per request to scope access. Every other doc line was re-checked empirically against pydantic-monty 0.0.17. * fix(code_mode): don't advertise env/clock for mount-only sandboxes A `mount` only exposes filesystem paths; `os.getenv`/`os.environ` and `datetime.now()`/`date.today()` still require an `os` handler. The description used one host-access note for both, so mount-only agents were told env/clock were routed to the host and would emit calls that fail and burn run_code retries (verified against pydantic-monty 0.0.17). Split the description into three states (none / mount-only filesystem / os), and correct the README and docstrings that conflated the two. * docs(code_mode): correct two run_code description claims verified against monty Audited every statement in the run_code description, docstrings, and README against pydantic-monty 0.0.17. Two were imprecise: - "imported at the top of your snippet" -- mid-snippet imports work, so the rule is just "before use". - OS-enabled note said calls route "to the host environment", but an in-memory AbstractOS (e.g. OSAccess) handles them too -- it's the configured OS handler, not necessarily the host. * docs(code_mode): clarify overlay-mode write persistence and fix wording The mount docs implied writes reach the host, but MountDir defaults to copy-on-write overlay mode, so writes stay in the sandbox unless mode is 'read-write'. Also tighten two awkward/redundant doc lines. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(code_mode): rename public OS/mount surface to be backend-neutral The public type aliases leaked the Monty backend name into a surface we can't rename later. Rename them to match the existing CodeMode/CodeModeToolset convention, and rename the os= parameter to os_access= so it stops shadowing the stdlib os module that sandboxed code itself uses. - MontyOS -> CodeModeOS, MontyOSCallback -> CodeModeOSCallback, MontyMount -> CodeModeMount - CodeMode/CodeModeToolset param os= -> os_access= (mount unchanged) - internal resume()/feed_start() forwarding keeps Monty's literal os= kwarg Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(code_mode): stop shadowing the os module in the execution loop The OS/mount threading named its parameter `os`, shadowing the stdlib module inside the execution-loop helpers. Rename the variable to `os_access` (matching the public field) while keeping Monty's required `os=` keyword only at the resume/feed_start call sites. Also inline the single-use restriction-line helper into `_base_description`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(code_mode): make CodeMode config fields keyword-only The option list keeps growing; pin tools/max_retries as the only positional args and force os_access/mount (and future config) to be passed by name via a KW_ONLY sentinel, so adding options can't silently shift positional meaning. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(code_mode): make os_access/mount docs clear on first read Public docs should let a reader grasp the host-access surface without reverse-engineering it. Reframe the docstrings and README around when to reach for each primitive instead of what is switched off, drop the type-restating prose the annotations already carry, and lead with concrete tasks (share a dataset; inject just the secrets the agent needs). Tighten the os-access test sweep so each test asserts exactly its invariant: drop redundant negative description asserts (one note is interpolated, so the positive phrase alone proves selection), drop an assertion already owned by another test, and type the tmp_path fixtures. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(code_mode): clarify os_access callback return semantics The raw-callback example claimed non-allow-listed keys "stay hidden" by returning NOT_HANDLED. Verified against Monty: NOT_HANDLED *refuses* the call (raises in the sandbox -> model retry), it does not return None. A model probing for an optional secret would crash and burn retries. Distinguish the two return modes explicitly so users don't pick the wrong one: return a value (incl. None) to answer/hide, NOT_HANDLED to refuse a capability outright. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(code_mode): lock in os_access value-vs-NOT_HANDLED semantics Returning a value (including None) from an os_access callback answers the call -- a None reads back like an unset env var, so the sandbox keeps running. Returning NOT_HANDLED refuses the call, raising in the sandbox and surfacing as ModelRetry. These two paths are easy to confuse and silently regress, so pin both. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * code_mode: warn default run_code prompt about unavailable fs/env; drop Any from CodeModeOSCallback The PR's dynamic restriction note covered the mount and os-enabled states but not the default state, which still listed os/pathlib as importable without saying their I/O fails -- the exact wasted-retry case this change reduces. Extend the default note to state filesystem, env, and clock calls are unavailable without a mount or OS handler. Also type the public CodeModeOSCallback alias with object instead of Any: the harness only forwards the callback to Monty and never calls it, so object typechecks cleanly (pyright strict, 0 errors) and keeps Any out of the public API. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: David SF <david.sanchez@pydantic.dev> |
||
|
|
ae371bbdaf |
docs: ban em-dashes and codify writing style (#270)
Add a Writing style section to AGENTS.md so docs, comments, and PR text read as human-written: no em-dashes (use `--`), no marketing superlatives or editorializing adjectives, sparing bold, no decorative Unicode. Bring the tree into compliance by converting every em-dash to `--` across 16 files. All occurrences are in prose, comments, docstrings, and tool descriptions. The only runtime effect is that model-facing tool descriptions now use `--` (punctuation only, no semantic change); no test snapshots depend on the character. The one literal em-dash left is in AGENTS.md, where it names the banned character. https://claude.ai/code/session_01LQ9NTr8q95A99UnVcq6Nzf Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
59e65b7a12 |
fix(code_mode): honor Tool Search's deferred-loading contract (#240)
* fix(code_mode): honor Tool Search's deferred-loading contract CodeMode flattened `defer_loading=True` tools into `run_code`'s description regardless of discovery state — defeating progressive disclosure, and on providers with native tool search also double-listing the same tool on the wire (top-level `tools[]` with `defer_loading: true` *and* its signature in `run_code.description`), plus busting the prompt cache on every discovery. Keep `defer_loading=True` tools as native pass-through so ToolSearchToolset's `defer_loading` / `with_native` flags reach `Model.prepare_request` unaltered; they fold into `run_code` only once discovered (`defer_loading=False`). Migrate the sibling `prefer_builtin` filter to its renamed `unless_native` form. Sources pydantic-ai from `main` for pydantic/pydantic-ai#5143 (native Tool Search); drop `[tool.uv.sources]` and bump the `pydantic-ai-slim` floor once that ships in a release. The same pydantic-ai changes rename the `builtin=` capability kwarg to `native=`, so the README and quick-start test are updated to match. Closes #232 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * deps: bump pydantic-ai-slim floor to >=1.95.0, drop temporary git pin pydantic-ai 1.95.0 ships pydantic/pydantic-ai#5143 (native Tool Search), so remove the `[tool.uv.sources]` git override and bump the floor to it. 1.95.0 also deprecates `Agent(instrument=...)` in favor of an `Instrumentation` capability, so the OTel-spans test switches to `capabilities=[..., Instrumentation(...)]`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: say "local MCP toolset" not "FastMCP" in the CodeMode + MCP example Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: add ToolSearch to the ecosystem-agent README example Show progressive tool discovery alongside CodeMode in the full ecosystem showcase — deferred tools fold into `run_code` once discovered. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: group ToolSearch with CodeMode as a meta-capability in the README example Both transform the toolset before it reaches the model, so they sit together under "Tool execution & discovery" rather than alongside the tool providers. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: trim the ToolSearch comment to one line, matching the others Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: clarify CodeMode + Tool Search behavior and the cache escape hatch Rewrite the code-mode README's Tool Search section to cover both the native and local-fallback paths accurately, spell out that a discovered tool folds into `run_code` (busting the prompt cache once at discovery), and document the `tools=` selector workaround (`td.with_native is None`) for keeping a Tool Search corpus fully native. Mirror the escape-hatch note in the toolset docstring. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ci(compat): set UV_FROZEN=1 so the editable overlay isn't clobbered by uv run `compat-test.yml` overlays `pydantic-ai-slim` / `pydantic-graph` from the local checkout via `uv pip install --no-deps -e ...`, but doesn't set `UV_FROZEN`. The next `uv run ruff/pyright/pytest` calls then auto-re-resolve against PyPI ("Ignoring existing lockfile due to change in resolution mode: `lowest-direct` vs. `highest`"), silently uninstall the overlay, and install whatever `pydantic-ai-slim` is on PyPI — so the compat check has been a no-op against the actual pydantic-ai checkout we wanted to validate. Caught when pydantic-ai 1.95.0 landed `current_otel_traceparent` with lazy imports that trip the Temporal sandbox; harness-compat against the merge commit went green (because it was testing 1.94.0 from PyPI, not the merge commit), and only failed once 1.95.0 was published to PyPI. `main.yml` already pins `UV_FROZEN: '1'` at workflow scope; do the same here. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * deps: re-pin pydantic-ai-slim to git main for the Temporal sandbox fix Released `1.95.0` (the lock-resolved version under `pydantic-ai-slim>=1.95.0`) still has the `current_otel_traceparent` lazy-import bug that hangs Temporal workflows on the `test_code_mode_runs_in_temporal_workflow` path — the `all-extras` jobs sat at 1h+ before this. Repin to pai `main` (now at `cf0b9077e`, which has pydantic/pydantic-ai#5422 merged) until `1.95.1` is on PyPI. Drop this section + bump the floor in a follow-up once the release lands. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Douwe MaanandGitHub
|
bf692aff32 |
Rework README hero example and add ecosystem agent showcase (#226)
* Rework README hero example and add full ecosystem agent showcase
The previous hero example used GitHub's MCP server, which requires
authentication and ships tools without `outputSchema` -- triggering
CodeMode warnings and producing weakly-typed sandbox stubs. Replaced
with the open-source Hacker News MCP server (full output schemas, no
auth), and added `WebSearch` to make the prompt naturally exercise
parallel tool orchestration in a single `run_code` call.
`MCP(..., builtin=False)` is now explicit so the local toolset is used
and CodeMode actually wraps the tools (Anthropic supports MCP server
connectors as a builtin and would otherwise execute tools server-side,
bypassing the sandbox).
Added a "Full ecosystem agent" section above the capability matrix that
illustrates the breadth of the Pydantic AI ecosystem -- combining
capabilities from the harness, core, and the community packages we
endorse, with inline credits at each call site.
Made install requirements explicit above the hero block.
* Reword Quick start prompt to avoid datetime in generated code
Monty currently rejects `datetime` (pydantic/monty issue tracking),
so the previous prompt's "posted in the last 6 hours" filter caused
the agent to generate code that failed in the sandbox. Replaced with
a dedupe-by-id + score filter + parallel follow-up calls (thread,
user profile, web search) -- still exercises parallel orchestration,
dict manipulation, and multi-tool composition without any timestamps.
* Embed verbatim run output and soften 'single run_code' claim
Followed pydantic-ai docs convention of triple-quoted output after
print(). The included synthesis is the actual output from running
the example, not a fabrication.
Reworded the CodeMode explanation to match what Sonnet actually does:
two batched run_code calls each containing parallel tool calls, with
the dedupe/filter happening as plain Python in the sandbox -- not the
"all in a single run_code" claim from the previous draft.
* Surface monty timing restrictions in run_code description
Sonnet was occasionally generating `asyncio.sleep` between tool calls
(cargo-culted "polite rate-limiting" pattern) and `datetime.now()` for
recency filters -- both rejected by the monty sandbox. Adding an explicit
"no wall-clock or timing primitives" bullet to the `run_code` tool
description (and the matching code-mode README entry) keeps the prompt
free of sandbox-specific babysitting.
Switched the hero example to claude-opus-4-7 with a more natural prompt
("find the most-discussed story... summarize what you find in one
paragraph"), and refreshed the verbatim sample output to a clean run
that produced 3 run_code calls (3x parallel feed fetches, pure-Python
dedupe/filter, then 3x parallel follow-up calls) with no errors.
* Quick start: opt out of native WebSearch so CodeMode wraps it
`WebSearch()` defaults to `builtin=True`, and on providers like
Anthropic that natively support web search the local DDG fallback
is filtered out by `Model.prepare_request`. With `CodeMode` in the
mix that produced a split surface -- builtin web_search at top
level, duckduckgo_search inside `run_code` -- and the model
sometimes tried to call the builtin from inside the sandbox.
Passing `builtin=False` keeps every web call going through CodeMode
so it can be batched alongside the HN tools in a single `run_code`.
Same pattern we already use for `MCP(..., builtin=False)`.
The underlying flatten-prefer_builtin-into-run_code bug is filed
as #233 and is being fixed separately.
Refreshed the verbatim sample output to a clean run with this
configuration: 3 chats + 2 run_code calls, all parallel tool
batches, no errors.
* Quick start: link the public trace and refresh verbatim output
DouweM picked a run with substantively richer commentary (Simon
Willison's "Vibe coding and agentic engineering are getting closer
than I'd like" post on HN) and made the trace public, so we can
finally point readers at the actual run from the example.
* Add duckduckgo extra to install snippet for WebSearch(builtin=False)
* Move ecosystem example credits to dedicated comment lines
* Restructure README around the Quick start trace and an ecosystem agent
Quick start
- Move all per-capability commentary into inline comments at call sites,
so links sit next to the code they document. Drops the trailing
paragraph wall.
- Make the public Logfire trace the section's main visual: a screenshot
linking to the trace, then a one-liner explainer.
- Trim the verbatim sample output to ~5 sentences. Keeps the Willison
framing and the meta-substantive thread reference; drops the long
commenter-by-commenter paraphrase and the news-coverage paragraph
that previously took over half the section.
An ecosystem agent (formerly "Full ecosystem agent")
- Move below the capability matrix so readers meet the breadth before
the worked example.
- Renamed and rewritten intro: shorter, voice closer to the rest of
the README, no "upper bound of what's possible" phrasing.
- Long disclaimer paragraph moved below the code so readers see the
example before the caveats.
- Imports split into official (`pydantic_ai*` + harness) vs community
(alphabetical).
- Sections reorganized: Reasoning, Tools, Execution (CodeMode pulled
out of Tools), Context management, Memory, Orchestration, Safety.
- Each capability gets its own one-line comment. AnthropicCompaction
added as the official compaction option, with a comment pointing at
vstorm-co's `summarization-pydantic-ai` as a provider-agnostic alt.
- Agent gets a name (`brian` -- Monty Python) which `MemoryCapability`
picks up via its `agent_name` arg.
- Skills comment now flags @vstorm-co's pydantic-deep alongside Doug's,
noting the shape difference.
- Switched to `claude-opus-4-7` and `Thinking(effort='high')` to match
the Quick start.
- Drops `MCP(builtin=False)` here -- the ecosystem example isn't trying
to demo CodeMode-wrapped MCP, it's demoing breadth.
Drops the no-wall-clock bullet from `_toolset.py` and the code-mode
README -- now covered by the standalone PR for that change.
Trace screenshot reference uses a relative path that resolves on GitHub
once the asset is committed; ready to swap in once the file is in place.
* Ecosystem agent: rename to shrubbery, bump thinking to xhigh
* Add Logfire trace screenshots and surface them prominently
`docs/images/quick-start-trace.png` is the trace tree: it's the visual
the main README's Quick start now leads with -- click-through to the
public trace.
`docs/images/code-mode-trace.png` is the wider shot showing the actual
Python the model wrote (parallel `asyncio.gather` over three HN feeds,
dedupe-by-id, score filter). It's the centrepiece of a new "In practice"
section in the code-mode README that points back to the harness Quick
start as the more representative example, replacing the toy weather
demo as the leading visual of what code mode does in real use.
* Address review on PR #226
- Reorder ecosystem agent capabilities: CodeMode first (the headline),
then Thinking, Context management, Tools (MCP -> WebSearch -> Console),
then everything else. First-party leads each tier; ConsoleCapability
follows our two first-party tools.
- Drop the agent `name='shrubbery'` flourish.
- Use vstorm-co's `ContextManagerCapability` as the active compaction
capability and mention pydantic-ai's `AnthropicCompaction` /
`OpenAICompaction` in the comment instead of the other way around.
Drops the "tied to Anthropic's prompt caching" phrasing.
- `MemoryCapability(agent_name='harness-example')`.
- Code-mode README: rewrite the "In practice" prose to be clear that
CodeMode produces two `run_code` calls (parallel fetches + filter,
then parallel follow-ups), not one. Fix the screenshot alt text to
match.
* Add regression test for the README's Quick start example
The lead example in the main README is the most public surface of the
harness — if it stops working we need to know before the user tries it.
This test drives the example end-to-end with `FunctionModel` issuing the
two `run_code` calls the README's trace shows, and `CodeMode` actually
running the emitted Python through Monty.
What gets faked vs. what's real:
- The Hacker News MCP toolset is replaced with a `FunctionToolset` of
fake functions returning canned data from the README's linked public
trace. Avoids depending on an MCP package or hitting the network.
- `WebSearch(builtin=False, local=...)` skips the default DuckDuckGo
fallback so the test doesn't pull `ddgs`.
- `FunctionModel` drives the conversation through two `run_code` calls
and a final synthesis, mirroring the production trace.
- `CodeMode` itself is real; the `FunctionModel`'s emitted Python runs
through the Monty sandbox and dispatches calls back to the fakes.
Asserts on the call shape (each feed fetched once in parallel, follow-up
calls target the expected winner id) and on the final synthesis content,
so any future change in pydantic-ai or the harness that breaks how these
capabilities compose makes this test fail.
* Address review on PR #226
- Quick start: list `CodeMode` first in `capabilities=[]` to match the
ecosystem agent's ordering and to lead with the headline capability.
- Test: wire the fake HN tools through `MCP(local=hn_toolset, builtin=False)`
instead of bypassing via `toolsets=[hn_toolset]`. Same composition path
as production minus the network. Needs a narrow `# pyright: ignore`
because MCP's `__init__` signature narrows `local` to MCP-shaped types
but the parent `BuiltinOrLocalTool` accepts any `AbstractToolset` at
runtime.
- Test: have the second `run_code` call `web_search` (the WebSearch
capability's local fallback) instead of `hn_search_content`, so the
fake function is actually exercised. This fixes the coverage failure
on lines 231-232 and makes the test more representative of the README
example, which uses WebSearch.
* Use snapshot()+dirty-equals for README quick-start test
* Re-export dirty-equals via tests/conftest, restructure tests/
Two related cleanups based on review:
- Re-export `IsDatetime` / `IsNow` / `IsStr` / `IsPartialDict` from
`tests/conftest.py` with `TYPE_CHECKING` shims that pretend the
matchers return the concrete type (`-> str`, `-> datetime`, etc.).
Mirrors pydantic-ai's own conftest pattern and lets pyright strict
accept `tool_call_id=IsStr()`, `timestamp=IsDatetime()`, etc. without
the file-level `# pyright: reportArgumentType=false` we had before.
Tests now `from .conftest import ...` instead of importing directly
from `dirty_equals`.
- Rename `tests/_code_mode/` -> `tests/code_mode/` (drop the underscore
prefix; matches the source package `pydantic_ai_harness/code_mode/`).
Updates the convention note in AGENTS.md/CLAUDE.md.
- Move `test_readme_quick_start.py` up to `tests/` top level. It's an
end-to-end check on the README's flagship example -- it composes
`CodeMode` + `MCP` + `WebSearch` -- so it doesn't belong in any
single capability's test directory.
|
||
|
Douwe MaanandGitHub
|
e56c60313a |
Document monty's missing wall-clock and timing primitives in run_code description (#236)
`asyncio.sleep`, `datetime.datetime.now()`, `datetime.date.today()`, and the `time` module are gated by the sandbox (timing primitives are restricted to avoid timing-based side channels). Without this hint in the `run_code` description, models occasionally generate sandbox-rejected code patterns -- `asyncio.sleep` between tool calls (cargo-culted "polite rate-limiting") and `datetime.now()` for recency filters. |
||
|
Douwe MaanandGitHub
|
0fcd4c0aea |
fix(code_mode): filter prefer_builtin tools out of sandboxing (#234)
`CodeModeToolset.get_tools` was sandboxing local-fallback tools annotated with `prefer_builtin`, rendering them inside `run_code.description` before `Model.prepare_request` ever got a chance to drop them when the provider supports the matching builtin. Mirror the existing `_SEARCH_TOOLS_NAME` guard so these tools stay native and the model sees a single, non-duplicated tool surface. Closes #233. |
||
|
Aditya VardhanandGitHub
|
fb62396335 |
Support codemode dependency group alias (#224)
|
||
|
Douwe MaanandGitHub
|
fe9a587bd4 |
feat(code_mode): resolve deferred/approval-required tool calls via HandleDeferredToolCalls (#220)
* feat(code_mode): resolve deferred tool calls via HandleDeferredToolCalls Tools with `kind='external'` or `'unapproved'` (and tools that raise ApprovalRequired/CallDeferred at runtime) are no longer excluded from the sandbox and promoted back to native tools. They now take the normal sandboxed path, and a HandleDeferredToolCalls capability on the agent can resolve them inline — so the model sees the resolved return value instead of having the deferral bounce out as a separate native tool call. - Remove the td.defer filter in _partition_callable_tools (no more native fallback for deferred tools). - Drop the native_fallbacks return value and the corresponding deferred-tool warning. - Update the sandbox UserError message when no handler is configured to point users at HandleDeferredToolCalls. - Update the deferred_execution test to assert sandbox inclusion and the approval-retry test to match the new error message. Depends on pydantic/pydantic-ai#5142 landing and being released; once it does, bump the pydantic-ai-slim lower bound. * code_mode: record outcome='denied' on nested ToolReturnPart for handler denials When the `HandleDeferredToolCalls` handler denies a tool call, `handle_call` now raises `ToolDeniedError` (on pydantic-ai-slim once released) instead of returning the denial message as a plain string. CodeMode catches it, records a `ToolReturnPart(outcome='denied')` in `nested_returns` so message history reflects the denial correctly, and re-raises so the sandbox surfaces the denial as an exception rather than as what would look like a successful tool return. The `ToolDeniedError` import is gated behind a compat shim so this module still loads against the currently released pydantic-ai-slim (which lacks the exception); the shim resolves to a placeholder class that never matches a real exception, leaving the except clause inert until a release ships `ToolDeniedError`. Depends on pydantic/pydantic-ai#5142. * code_mode: handle denials via `ToolDenied` return value, not exception `ToolManager.handle_call` no longer raises a (now-removed) `ToolDeniedError` on handler denial — it returns the `ToolDenied` value the handler produced. Drop the compat shim, import `ToolDenied` directly, and switch the dispatch to inspect the return value: record the denial as `outcome='denied'` on the nested `ToolReturnPart` and raise a `RuntimeError` inside the sandbox so the script can't mistake the denial message for a regular string return. * Bump pydantic-ai-slim lock + cover denial path Now that the slim PR has merged to main, refresh the lockfile to pick up the `HandleDeferredToolCalls` capability and `handle_call`'s `ToolDenied` return value. Add a denial test that asserts the denied-call flow surfaces as `ModelRetry` with the original denial message preserved in the trace. Notes on the test: - The handler returns `ToolDenied('nope')`; the harness records `outcome='denied'` on the nested `ToolReturnPart` and raises `RuntimeError` inside the sandbox. - The script doesn't catch the RuntimeError, so Monty surfaces it as `MontyRuntimeError`, which the harness converts back to `ModelRetry`. The retry message preserves the denial message so the model knows what went wrong. * ci: test against floor pydantic-ai-slim (1.80.0) in addition to main The default `test` matrix uses the `[tool.uv.sources]` override pinning slim to its main branch, so it never exercises the published-PyPI install path. Add a `test-floor` job that overrides slim to the lowest version declared in `pyproject.toml` (>=1.80.0) and runs the test suite, so we catch any accidental dependency on unreleased slim features in code paths that should be backward-compatible. Gate the new HandleDeferredToolCalls denial test with `pytest.skip` when the capability isn't importable — currently the only test that requires a post-1.80.0 slim, but the pattern can be reused if more land later. * fix coverage: pragma the floor-only skip in the denial test The `except ImportError → pytest.skip` branch only fires when running against the slim floor (1.80.0) where `HandleDeferredToolCalls` doesn't exist yet. The default test matrix runs against slim main, so coverage counted those two lines as uncovered. Mark the branch `# pragma: no cover` since it's an explicit skip path that the floor-slim CI job exercises but isn't included in the coverage report (the floor job doesn't gate on coverage by design). |
||
|
|
9b1a121a92 |
Fix Monty's snapshot resume API (#216)
* fix(code-mode): match Monty's current snapshot resume API * bumping monty * single quotes * uv.lock * docstring: use single backticks per project convention Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * revert unrelated test docstring rewording Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
987b7312d5 |
Rename package to pydantic-ai-harness (#207)
- Package name: pydantic-harness -> pydantic-ai-harness - Python module: pydantic_harness -> pydantic_ai_harness - Display name: Pydantic Harness -> Pydantic AI Harness - All URLs updated to github.com/pydantic/pydantic-ai-harness Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |