2 Commits

Include renames

Author	SHA1	Message	Date
David SFandGitHub	3ba9e2f9a5	docs: capability pages for the unified docs site + README/doc parity gate (#329) ... * docs: publish capability docs to the unified site + add README/doc parity gate Every capability shipped only a README (kept for GitHub/PyPI). This adds a parallel, cleaned-up page per capability under docs/ for the new unified docs site (pydantic.dev/docs/harness), migrated from each README: snippets verified runnable against source, autodoc API blocks, root-relative Pydantic AI links, and an experimental-status admonition on the experimental set. To keep README and doc in sync going forward, adds a docs-parity-reviewer agent and a parity gate in the review checklist (run as the last step before merge), plus the docs/ layout and the README<->doc requirement in AGENTS.md and the capability-authoring guide. * docs: fix README<->doc<->source inconsistencies across capabilities A parity audit against source found drift, mostly in the capability READMEs (staler than the migrated docs). All fixes verified against source: - Correctness: the "approval/deferred tools are excluded from the sandbox" claim (code_mode README + doc) was false -- those tools are sandboxed like any other; corrected in both. The stale Shell persist_cwd sentinel description is replaced with the actual out-of-band temp-file capture. filesystem protected default `.git/` -> `.git/*` (the bare form never matched). - Runnable snippets: added the missing imports/wiring so README snippets no longer raise NameError (subagents, context, planning, overflow, authoring, filesystem, code_mode). - Parity: documented previously-undocumented params/behaviors (compaction strategy options, overflow strip_ansi/Passthrough, extra autodoc classes for context and subagents), fixed a stale version pin (>=1.95.1 -> >=2.1.0), and added the missing Managed Prompt row to the root README capability matrix. - Style: normalized decorative Unicode to ASCII across all READMEs and dropped a hype phrase, matching AGENTS.md writing style and the docs. * docs: add nav.json to drive the unified-docs harness sidebar The unified docs mount the harness docs under /docs/ai/harness (fed live from this repo via the pydantic-ai 'Pydantic AI Harness' section). This nav.json defines the sub-nav (Overview + Capabilities + Experimental) and the set of doc files the site includes. * docs: migrate "What goes where?" explainer into harness overview Adds the core-vs-harness boundary section (anchor #what-goes-where) to the canonical harness overview, so the pydantic-ai docs that link to it can point here after the duplicated in-repo stub is removed. * docs: address CodeRabbit review -- runnable snippets, accuracy, multi-class autodoc * docs: flatten harness nav and align with graduated capabilities Following the experimental-graduation refactor (#347), restructure the unified-docs harness pages: - Flatten docs/ (drop capabilities/ and experimental/ subdirs); the sidebar is now Overview + one flat list per Douwe's request. - Rename to match the graduated modules: overflow -> overflowing-tool-output, authoring -> runtime-authoring, docs -> pydantic-ai-docs. - Drop the 'Experimental' admonitions from the graduated capabilities and repoint every import + ::: autodoc path off pydantic_ai_harness.experimental. - Add docs for the newly-shipped capabilities: guardrails, dynamic-workflow, media, and acp (acp stays framed as experimental -- it may still be removed). - Every capability doc now links to its source; index capability table lists the full set with flat links. * docs: apply team-sync authoring rules + enforce them in CI From the 2026-07-10 docs review on #329: - Purpose-first leads: drop hook names (before_model_request, after_tool_execute) from the opening paragraphs of compaction and overflowing-tool-output (doc + README); mechanism moves lower. - Mirror the soft 'API may change between releases' stability note from each graduated README into its doc page (ACP keeps its stronger experimental warning; guardrails' README has no note, so its page gets none). - README H1s now use the capability's display name (Overflow capability -> Overflowing Tool Output, RuntimeAuthoring -> Runtime Authoring, SubAgents -> Subagents, etc.). - Extend tests/test_docs_parity.py with per-page mechanical checks: source link present, heading matches the capability name, purpose-first lead (no hook in the opener), and no experimental framing on graduated pages (ACP excepted). - Update the docs-parity-reviewer agent + review-checklist to the flat structure and the new semantic checks. * docs: add the stability note to guardrails (parity with sibling capabilities) guardrails was the one graduated capability whose README and doc page lacked the shared 'API may change between releases' note. Add it to both. * fix: restore uv.lock to match pyproject (bad text-merge dropped 8 lines) Merging origin/main did a git text-merge of the generated uv.lock, leaving it inconsistent with pyproject.toml -- every CI job failed at 'uv sync --locked'. pyproject.toml is identical to main here, so the correct lock is main's. * docs: address CodeRabbit review on #329 Findings that failed to post inline (GitHub error) but were real: - context/README.md, planning/README.md: two nested examples still imported from pydantic_ai_harness.experimental.* -- repoint to the graduated modules. - guardrails/README.md: replace em dashes with '--' (repo style) and add the source-module link. - docs/media.md: standardize on the implementation's canonical media+sha256:// URI scheme (was mixing media://). - tests/test_docs_parity.py: strengthen my own checks per review -- source-link and top-README-link now require a real Markdown link to the page's specific module (not a bare substring); heading checks assert an H1 exists and equals the expected capability name via explicit page metadata. * fix: restore uv.lock [options.exclude-newer-package] block The lock lost its [options.exclude-newer-package] manifest (pydantic-ai-slim = false, ...) -- a bad git text-merge dropped it, and diagnostic uv commands rewrote it under a different local config. Without that block CI's 'uv sync --locked' re-resolves and fails ('addition of exclude newer exclusion for pydantic-ai-slim'). Restore origin/main's exact lock. * fix: restore uv.lock [options.exclude-newer-package] block A pre-commit hook was rewriting uv.lock under the local uv config, stripping the [options.exclude-newer-package] manifest (pydantic-ai-slim = false, ...). Without it CI's 'uv sync --locked' re-resolves and fails. Commit origin/main's exact lock with --no-verify so no hook mutates it (lock-only change). * test: cover the docs-parity helper edge cases (100% coverage) The strengthened helpers added defensive branches (missing frontmatter close, fenced code before the lead, missing/forbidden/ClassName H1, lead running to EOF) that no real doc exercises. Add direct unit tests so the file is back to the repo's required 100% coverage. * docs: link every capability README to its source module + enforce it CodeRabbit re-flagged planning/README.md for a missing source link. Only guardrails had one, so add the source-module link to all 15 remaining capability READMEs (matching the doc pages) and add a parity test so the requirement is mechanical and cannot silently regress. * docs(agents): drop stale folder tree; fix flat docs path + guard names AGENTS.md's File-structure ASCII tree and capability-authoring's doc paths still showed docs/capabilities// docs/experimental/ (flattened in this PR) and the old /docs/harness URL. Delete the tree rather than redraw it -- the layout is discoverable by listing the repo; keep only the non-obvious conventions (flat docs/, the README<->doc parity requirement). Also fix the Vocabulary guard examples (InputGuard/OutputGuard, not the nonexistent InputGuardrail/ CostGuard). * test: statically validate doc snippets exist and parse Every Python snippet in the capability READMEs and docs/*.md pages is now checked for the two failures a reader hits immediately: it does not parse (syntax), or it imports a pydantic_ai_harness symbol that does not exist (stale module path or renamed name -- the class of bug behind the experimental.* import drift). Static only: no model/network execution, so it needs no mocking. The four illustrative API-signature blocks opt out with a {test="skip"} fence (read by pytest-examples, stripped-safe for the unified-docs render). * test: don't fail doc-snippet check on a missing optional extra The static check imported capability modules to resolve their symbols, but in the slim CI job (no extras) importing e.g. pydantic_ai_harness.experimental.acp raises ModuleNotFoundError for the absent third-party 'acp' package -- the harness module exists, its extra just isn't installed. Distinguish a genuinely missing harness module (fail) from a missing extra (skip) by the ImportError's module name.	2026-07-13 11:00:12 -05:00
4 people	5ab8dd3d81	feat: input and output guardrails (block, redact, retry) (#249) ... * feat: add input and output guardrails * chore: use single backticks * fix: run InputGuard only on the first model request * fix: replace list[Any] with Sequence[ModelMessage] * fix: pass raw output to OutputGuard, not str(result.output) * refactor: organize tests into TestCapabilityName classes * fix: drain cancelled tasks in InputGuard parallel finally * fix: re-raise task exceptions via await instead of .exception() * refactor: consolidate parallel cancel/drain into single finally * feat: support callable block_message; collapse InputGuard hooks block_message now accepts a callable so the refusal text can reflect the prompt/output that tripped the guard, rather than being frozen at construction time. InputGuard's sequential path moves from before_model_request into wrap_model_request, so a single hook covers both sequential and parallel modes instead of two hooks each branching on `parallel`. Tests move to tests/guardrails/ to match the tests/<capability>/ layout. * refactor: guards return bool | GuardResult; drop block_message A guard now returns either a bare bool or a GuardResult carrying a refusal message, replacing the separate block_message constructor field. The message is produced when the guard decides, so it can reflect the guard's own reasoning rather than a string frozen at construction time. Guards (and the GuardResult path) may optionally take a RunContext as a first parameter, detected from the signature like pydantic-ai's output validators, so deps- and history-aware guards are possible without closing over globals. Prompt/output-only guards are unchanged. * feat: guard outcomes — allow/block/replace/retry with OTel spans A guard now reports one of four outcomes via GuardResult classmethods (bool shorthand still works): allow, block, replace, retry. - replace lets a guard redact rather than refuse — InputGuard rewrites the prompt sent to the model, OutputGuard substitutes the output. - retry lets OutputGuard send a bad output back to the model; OutputGuard moves from after_run to after_output_process so it can raise ModelRetry and return a modified output. - replace and block emit spans on the run tracer so a redaction or refusal is visible in Logfire; redacted content is included only when RunContext.trace_include_content is set. InputGuard replace requires sequential mode and retry is rejected as a usage error, since neither is meaningful for input. * test: cover guardrail test helpers for the 100% gate The coverage gate measures test files too; the _prompt_text helper had unreached branches. Drop it and assert on message parts inline. * docs: scope streaming behavior; add streaming tests A pydantic-ai-correctness review surfaced two streaming points. Verified both empirically: - InputGuard(parallel=True) works under run_stream() — no deadlock. - OutputGuard GuardResult.retry() is unsupported under run_stream(): pydantic-ai does not retry output while streaming, so a retry verdict surfaces as UnexpectedModelBehavior. Document the retry limitation and that OutputGuard screens only the final output (partial chunks reach the caller first while streaming). Note that input redaction also rewrites persisted history and targets text prompts. Add streaming tests for both guards to lock the behavior in. * test: make the parallel-streaming test a real regression guard The test proving InputGuard(parallel=True) does not deadlock under run_stream() had no timeout — a reintroduced deadlock would hang CI instead of failing. Wrap it in asyncio.wait_for and document the reviewed concern it guards against. * feat: guardrail capability ordering + GuardResult hardening Address @adtyavrdhn's review on #249. - `InputGuard.get_ordering()` → `position='innermost'` so any message-morphing capability runs first and the guard sees the final prompt the model will receive. - `OutputGuard.get_ordering()` → `position='outermost', wrapped_by= [Instrumentation]` so the guard's block/redact spans are always captured by an enclosing `Instrumentation` span regardless of user list order. - `GuardResult` is `frozen=True, kw_only=True` with a `__post_init__` that rejects field combinations the four-outcome contract does not allow (e.g. `replace` without a replacement). `block` with no message stays valid — the default kicks in at the use site. - `_run_guard` and `after_output_process` dispatch via `match action:` with `assert_never` exhaustiveness guards. - `_trace_block` gates the refusal `message` attribute behind `ctx.trace_include_content`, matching `_trace_redaction` — the message can quote sensitive content from the guarded value. - New tests: ordering declarations, `__post_init__` validation, frozen enforcement, outer-cancellation no-leak regression guard (which confirms `asyncio.shield` around the cleanup `gather` is not needed — the outer cancel is already consumed by `asyncio.wait`). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump pydantic-ai-slim to 2.4.0 in lock CI resolves pydantic packages to newest (uv.toml marks them exclude-newer=false); the merged lock was stale at 2.1.0 so uv sync --locked failed. Regenerate to 2.4.0. * test(code_mode): drop removed ToolSearchMatch.description field pydantic-ai 2.4.0 narrowed ToolSearchMatch to carry only `name`; the full ToolDefinition (with description) is surfaced separately. Update the test fixtures to the new shape. --------- Co-authored-by: Kacper Włodarczyk <kacperwlodarczyk@protonmail.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-07-10 14:21:23 -05:00