mirror of https://github.com/pydantic/pydantic-ai-harness.git synced 2026-07-21 02:45:34 +00:00

Files

T

David SFandGitHub 3ba9e2f9a5 docs: capability pages for the unified docs site + README/doc parity gate (#329 )

* docs: publish capability docs to the unified site + add README/doc parity gate

Every capability shipped only a README (kept for GitHub/PyPI). This adds a
parallel, cleaned-up page per capability under docs/ for the new unified docs
site (pydantic.dev/docs/harness), migrated from each README: snippets verified
runnable against source, autodoc API blocks, root-relative Pydantic AI links,
and an experimental-status admonition on the experimental set.

To keep README and doc in sync going forward, adds a docs-parity-reviewer agent
and a parity gate in the review checklist (run as the last step before merge),
plus the docs/ layout and the README<->doc requirement in AGENTS.md and the
capability-authoring guide.

* docs: fix README<->doc<->source inconsistencies across capabilities

A parity audit against source found drift, mostly in the capability READMEs
(staler than the migrated docs). All fixes verified against source:

- Correctness: the "approval/deferred tools are excluded from the sandbox" claim
  (code_mode README + doc) was false -- those tools are sandboxed like any
  other; corrected in both. The stale Shell persist_cwd sentinel description is
  replaced with the actual out-of-band temp-file capture. filesystem protected
  default `.git/` -> `.git/*` (the bare form never matched).
- Runnable snippets: added the missing imports/wiring so README snippets no
  longer raise NameError (subagents, context, planning, overflow, authoring,
  filesystem, code_mode).
- Parity: documented previously-undocumented params/behaviors (compaction
  strategy options, overflow strip_ansi/Passthrough, extra autodoc classes for
  context and subagents), fixed a stale version pin (>=1.95.1 -> >=2.1.0), and
  added the missing Managed Prompt row to the root README capability matrix.
- Style: normalized decorative Unicode to ASCII across all READMEs and dropped a
  hype phrase, matching AGENTS.md writing style and the docs.

* docs: add nav.json to drive the unified-docs harness sidebar

The unified docs mount the harness docs under /docs/ai/harness (fed live from
this repo via the pydantic-ai 'Pydantic AI Harness' section). This nav.json
defines the sub-nav (Overview + Capabilities + Experimental) and the set of doc
files the site includes.

* docs: migrate "What goes where?" explainer into harness overview

Adds the core-vs-harness boundary section (anchor #what-goes-where) to the
canonical harness overview, so the pydantic-ai docs that link to it can point
here after the duplicated in-repo stub is removed.

* docs: address CodeRabbit review -- runnable snippets, accuracy, multi-class autodoc

* docs: flatten harness nav and align with graduated capabilities

Following the experimental-graduation refactor (#347), restructure the
unified-docs harness pages:

- Flatten docs/ (drop capabilities/ and experimental/ subdirs); the sidebar
  is now Overview + one flat list per Douwe's request.
- Rename to match the graduated modules: overflow -> overflowing-tool-output,
  authoring -> runtime-authoring, docs -> pydantic-ai-docs.
- Drop the 'Experimental' admonitions from the graduated capabilities and
  repoint every import + ::: autodoc path off pydantic_ai_harness.experimental.
- Add docs for the newly-shipped capabilities: guardrails, dynamic-workflow,
  media, and acp (acp stays framed as experimental -- it may still be removed).
- Every capability doc now links to its source; index capability table lists
  the full set with flat links.

* docs: apply team-sync authoring rules + enforce them in CI

From the 2026-07-10 docs review on #329:

- Purpose-first leads: drop hook names (before_model_request,
  after_tool_execute) from the opening paragraphs of compaction and
  overflowing-tool-output (doc + README); mechanism moves lower.
- Mirror the soft 'API may change between releases' stability note from each
  graduated README into its doc page (ACP keeps its stronger experimental
  warning; guardrails' README has no note, so its page gets none).
- README H1s now use the capability's display name (Overflow capability ->
  Overflowing Tool Output, RuntimeAuthoring -> Runtime Authoring, SubAgents ->
  Subagents, etc.).
- Extend tests/test_docs_parity.py with per-page mechanical checks: source link
  present, heading matches the capability name, purpose-first lead (no hook in
  the opener), and no experimental framing on graduated pages (ACP excepted).
- Update the docs-parity-reviewer agent + review-checklist to the flat
  structure and the new semantic checks.

* docs: add the stability note to guardrails (parity with sibling capabilities)

guardrails was the one graduated capability whose README and doc page lacked
the shared 'API may change between releases' note. Add it to both.

* fix: restore uv.lock to match pyproject (bad text-merge dropped 8 lines)

Merging origin/main did a git text-merge of the generated uv.lock, leaving it
inconsistent with pyproject.toml -- every CI job failed at 'uv sync --locked'.
pyproject.toml is identical to main here, so the correct lock is main's.

* docs: address CodeRabbit review on #329

Findings that failed to post inline (GitHub error) but were real:
- context/README.md, planning/README.md: two nested examples still imported
  from pydantic_ai_harness.experimental.* -- repoint to the graduated modules.
- guardrails/README.md: replace em dashes with '--' (repo style) and add the
  source-module link.
- docs/media.md: standardize on the implementation's canonical media+sha256://
  URI scheme (was mixing media://).
- tests/test_docs_parity.py: strengthen my own checks per review --
  source-link and top-README-link now require a real Markdown link to the
  page's specific module (not a bare substring); heading checks assert an H1
  exists and equals the expected capability name via explicit page metadata.

* fix: restore uv.lock [options.exclude-newer-package] block

The lock lost its [options.exclude-newer-package] manifest (pydantic-ai-slim
= false, ...) -- a bad git text-merge dropped it, and diagnostic uv commands
rewrote it under a different local config. Without that block CI's
'uv sync --locked' re-resolves and fails ('addition of exclude newer exclusion
for pydantic-ai-slim'). Restore origin/main's exact lock.

* fix: restore uv.lock [options.exclude-newer-package] block

A pre-commit hook was rewriting uv.lock under the local uv config, stripping
the [options.exclude-newer-package] manifest (pydantic-ai-slim = false, ...).
Without it CI's 'uv sync --locked' re-resolves and fails. Commit origin/main's
exact lock with --no-verify so no hook mutates it (lock-only change).

* test: cover the docs-parity helper edge cases (100% coverage)

The strengthened helpers added defensive branches (missing frontmatter close,
fenced code before the lead, missing/forbidden/ClassName H1, lead running to
EOF) that no real doc exercises. Add direct unit tests so the file is back to
the repo's required 100% coverage.

* docs: link every capability README to its source module + enforce it

CodeRabbit re-flagged planning/README.md for a missing source link. Only
guardrails had one, so add the source-module link to all 15 remaining
capability READMEs (matching the doc pages) and add a parity test so the
requirement is mechanical and cannot silently regress.

* docs(agents): drop stale folder tree; fix flat docs path + guard names

AGENTS.md's File-structure ASCII tree and capability-authoring's doc paths
still showed docs/capabilities// docs/experimental/ (flattened in this PR) and
the old /docs/harness URL. Delete the tree rather than redraw it -- the layout
is discoverable by listing the repo; keep only the non-obvious conventions
(flat docs/, the README<->doc parity requirement). Also fix the Vocabulary
guard examples (InputGuard/OutputGuard, not the nonexistent InputGuardrail/
CostGuard).

* test: statically validate doc snippets exist and parse

Every Python snippet in the capability READMEs and docs/*.md pages is now
checked for the two failures a reader hits immediately: it does not parse
(syntax), or it imports a pydantic_ai_harness symbol that does not exist (stale
module path or renamed name -- the class of bug behind the experimental.* import
drift). Static only: no model/network execution, so it needs no mocking. The
four illustrative API-signature blocks opt out with a {test="skip"} fence
(read by pytest-examples, stripped-safe for the unified-docs render).

* test: don't fail doc-snippet check on a missing optional extra

The static check imported capability modules to resolve their symbols, but in
the slim CI job (no extras) importing e.g. pydantic_ai_harness.experimental.acp
raises ModuleNotFoundError for the absent third-party 'acp' package -- the
harness module exists, its extra just isn't installed. Distinguish a genuinely
missing harness module (fail) from a missing extra (skip) by the ImportError's
module name.

2026-07-13 11:00:12 -05:00

4.9 KiB

Raw Blame History

Review Checklist

Use this before opening a PR or reviewing a capability change.

Product Fit

The capability has a clear user or dogfooding need.
The behavior belongs in harness, not Pydantic AI core.
The public API is small and named around user concepts.
The capability composes with relevant existing capabilities.

Implementation

Public exports are intentional.
Private helpers stay private.
Types are precise; new public signatures do not use Any.
No casts are used to paper over type design.
The implementation uses Pydantic AI hooks/toolsets instead of duplicating core runtime behavior.
Capability ordering is justified when present.
Dependency changes were made through uv and have a clear reason.

Stale Or Pre-Merge PRs

Run these checks when adopting, rebasing, or re-reviewing a PR that was opened well before now, or that was built against unreleased Pydantic AI changes.

Temporary [tool.uv.sources] pins to a branch or git ref are removed once the upstream change they waited on has landed in a released pydantic-ai-slim.
Each upstream Pydantic AI PR or branch the change rode on has merged. Link the upstream PR and its merge state.
The touched surface has not drifted: re-check the capability, hook, and toolset signatures it depends on against current main, not against the state at fork time.
Behavior the PR worked around because a primitive was missing is reconsidered if that primitive now exists in core.

Tests

Tests cover the public Agent(..., capabilities=[...]) path where possible.
Lower-level tests cover lifecycle, schemas, retries, and metadata when needed.
Error paths and important option combinations are covered.
Relevant protocol-shaped output is snapshotted.
make lint, make typecheck, and make test pass before handoff.

Docs

Every released capability ships two hand-maintained docs that must stay in sync with the code and with each other:

the README next to the implementation (pydantic_ai_harness/<capability>/README.md, or pydantic_ai_harness/experimental/<capability>/README.md for ACP), which serves GitHub and PyPI, and
the unified doc on the docs site, flat under docs/<capability>.md. The sidebar is a flat list under "Pydantic AI Harness" -- no capabilities/ or experimental/ subdirectories.

Checks:

Both the README and the unified doc are updated for any user-facing change (public class, params, defaults, tool names, extras, safety semantics). A change reflected in only one of them is a defect, not a follow-up.
The two do not contradict each other or the source on extras, option names, defaults, or safety caveats.
Every snippet in both docs is runnable: all imports present, class/param names match the source, model ids unchanged from what the source uses. Imports use the canonical module path (never pydantic_ai_harness.experimental.* for a graduated capability).
Purpose-first lead. The opening paragraph of each page and README states what the capability is for and when to reach for it -- no internal hook or class name (before_model_request, after_tool_execute, ...) before the purpose. Mechanism belongs lower down.
Name matches the capability. The doc filename, its # H1, and the README's # H1 all use the capability's descriptive name (e.g. "Overflowing Tool Output", not "Overflow"; "Runtime Authoring", not "Authoring").
Source link. Each page links its source module (https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/<module>/) so a reading agent can verify behavior. Where the capability exposes a public class, the page may also end with a ## API reference section of ::: pydantic_ai_harness... autodoc blocks (auto-expanded from the docstring, not hand-written).
Stability framing. Graduated capabilities carry the soft note "The API may change between releases..." mirrored from their README -- NOT a HarnessExperimentalWarning block or "removed in any release" wording. ACP is the only page that keeps an !!! warning "Experimental" (it may still be removed).
Links: harness-internal links are relative .md; Pydantic AI docs use root-relative internal links /ai/<section>/<page>/ (verify the route resolves on the live pydantic.dev/docs site before using it).
Docs explain composition constraints and safety implications.
The PR links an issue.

The mechanical half of these checks (README present + linked, flat page present, source link present, name matches, no experimental strings on non-ACP pages, no hook name in the lead) is enforced by tests/test_docs_parity.py. The semantic half (does the prose match the code, are snippets truly runnable) is what the reviewer below is for.

This is the last documentation gate before merge. Run the docs-parity-reviewer subagent (.agents/agents/docs-parity-reviewer.md) on the change as the final review step; treat its blocking findings as merge blockers.

4.9 KiB Raw Blame History