mirror of https://github.com/pydantic/pydantic-ai-harness.git synced 2026-07-21 02:45:34 +00:00

Files

T

David SFandGitHub 3ba9e2f9a5 docs: capability pages for the unified docs site + README/doc parity gate (#329 )

* docs: publish capability docs to the unified site + add README/doc parity gate

Every capability shipped only a README (kept for GitHub/PyPI). This adds a
parallel, cleaned-up page per capability under docs/ for the new unified docs
site (pydantic.dev/docs/harness), migrated from each README: snippets verified
runnable against source, autodoc API blocks, root-relative Pydantic AI links,
and an experimental-status admonition on the experimental set.

To keep README and doc in sync going forward, adds a docs-parity-reviewer agent
and a parity gate in the review checklist (run as the last step before merge),
plus the docs/ layout and the README<->doc requirement in AGENTS.md and the
capability-authoring guide.

* docs: fix README<->doc<->source inconsistencies across capabilities

A parity audit against source found drift, mostly in the capability READMEs
(staler than the migrated docs). All fixes verified against source:

- Correctness: the "approval/deferred tools are excluded from the sandbox" claim
  (code_mode README + doc) was false -- those tools are sandboxed like any
  other; corrected in both. The stale Shell persist_cwd sentinel description is
  replaced with the actual out-of-band temp-file capture. filesystem protected
  default `.git/` -> `.git/*` (the bare form never matched).
- Runnable snippets: added the missing imports/wiring so README snippets no
  longer raise NameError (subagents, context, planning, overflow, authoring,
  filesystem, code_mode).
- Parity: documented previously-undocumented params/behaviors (compaction
  strategy options, overflow strip_ansi/Passthrough, extra autodoc classes for
  context and subagents), fixed a stale version pin (>=1.95.1 -> >=2.1.0), and
  added the missing Managed Prompt row to the root README capability matrix.
- Style: normalized decorative Unicode to ASCII across all READMEs and dropped a
  hype phrase, matching AGENTS.md writing style and the docs.

* docs: add nav.json to drive the unified-docs harness sidebar

The unified docs mount the harness docs under /docs/ai/harness (fed live from
this repo via the pydantic-ai 'Pydantic AI Harness' section). This nav.json
defines the sub-nav (Overview + Capabilities + Experimental) and the set of doc
files the site includes.

* docs: migrate "What goes where?" explainer into harness overview

Adds the core-vs-harness boundary section (anchor #what-goes-where) to the
canonical harness overview, so the pydantic-ai docs that link to it can point
here after the duplicated in-repo stub is removed.

* docs: address CodeRabbit review -- runnable snippets, accuracy, multi-class autodoc

* docs: flatten harness nav and align with graduated capabilities

Following the experimental-graduation refactor (#347), restructure the
unified-docs harness pages:

- Flatten docs/ (drop capabilities/ and experimental/ subdirs); the sidebar
  is now Overview + one flat list per Douwe's request.
- Rename to match the graduated modules: overflow -> overflowing-tool-output,
  authoring -> runtime-authoring, docs -> pydantic-ai-docs.
- Drop the 'Experimental' admonitions from the graduated capabilities and
  repoint every import + ::: autodoc path off pydantic_ai_harness.experimental.
- Add docs for the newly-shipped capabilities: guardrails, dynamic-workflow,
  media, and acp (acp stays framed as experimental -- it may still be removed).
- Every capability doc now links to its source; index capability table lists
  the full set with flat links.

* docs: apply team-sync authoring rules + enforce them in CI

From the 2026-07-10 docs review on #329:

- Purpose-first leads: drop hook names (before_model_request,
  after_tool_execute) from the opening paragraphs of compaction and
  overflowing-tool-output (doc + README); mechanism moves lower.
- Mirror the soft 'API may change between releases' stability note from each
  graduated README into its doc page (ACP keeps its stronger experimental
  warning; guardrails' README has no note, so its page gets none).
- README H1s now use the capability's display name (Overflow capability ->
  Overflowing Tool Output, RuntimeAuthoring -> Runtime Authoring, SubAgents ->
  Subagents, etc.).
- Extend tests/test_docs_parity.py with per-page mechanical checks: source link
  present, heading matches the capability name, purpose-first lead (no hook in
  the opener), and no experimental framing on graduated pages (ACP excepted).
- Update the docs-parity-reviewer agent + review-checklist to the flat
  structure and the new semantic checks.

* docs: add the stability note to guardrails (parity with sibling capabilities)

guardrails was the one graduated capability whose README and doc page lacked
the shared 'API may change between releases' note. Add it to both.

* fix: restore uv.lock to match pyproject (bad text-merge dropped 8 lines)

Merging origin/main did a git text-merge of the generated uv.lock, leaving it
inconsistent with pyproject.toml -- every CI job failed at 'uv sync --locked'.
pyproject.toml is identical to main here, so the correct lock is main's.

* docs: address CodeRabbit review on #329

Findings that failed to post inline (GitHub error) but were real:
- context/README.md, planning/README.md: two nested examples still imported
  from pydantic_ai_harness.experimental.* -- repoint to the graduated modules.
- guardrails/README.md: replace em dashes with '--' (repo style) and add the
  source-module link.
- docs/media.md: standardize on the implementation's canonical media+sha256://
  URI scheme (was mixing media://).
- tests/test_docs_parity.py: strengthen my own checks per review --
  source-link and top-README-link now require a real Markdown link to the
  page's specific module (not a bare substring); heading checks assert an H1
  exists and equals the expected capability name via explicit page metadata.

* fix: restore uv.lock [options.exclude-newer-package] block

The lock lost its [options.exclude-newer-package] manifest (pydantic-ai-slim
= false, ...) -- a bad git text-merge dropped it, and diagnostic uv commands
rewrote it under a different local config. Without that block CI's
'uv sync --locked' re-resolves and fails ('addition of exclude newer exclusion
for pydantic-ai-slim'). Restore origin/main's exact lock.

* fix: restore uv.lock [options.exclude-newer-package] block

A pre-commit hook was rewriting uv.lock under the local uv config, stripping
the [options.exclude-newer-package] manifest (pydantic-ai-slim = false, ...).
Without it CI's 'uv sync --locked' re-resolves and fails. Commit origin/main's
exact lock with --no-verify so no hook mutates it (lock-only change).

* test: cover the docs-parity helper edge cases (100% coverage)

The strengthened helpers added defensive branches (missing frontmatter close,
fenced code before the lead, missing/forbidden/ClassName H1, lead running to
EOF) that no real doc exercises. Add direct unit tests so the file is back to
the repo's required 100% coverage.

* docs: link every capability README to its source module + enforce it

CodeRabbit re-flagged planning/README.md for a missing source link. Only
guardrails had one, so add the source-module link to all 15 remaining
capability READMEs (matching the doc pages) and add a parity test so the
requirement is mechanical and cannot silently regress.

* docs(agents): drop stale folder tree; fix flat docs path + guard names

AGENTS.md's File-structure ASCII tree and capability-authoring's doc paths
still showed docs/capabilities// docs/experimental/ (flattened in this PR) and
the old /docs/harness URL. Delete the tree rather than redraw it -- the layout
is discoverable by listing the repo; keep only the non-obvious conventions
(flat docs/, the README<->doc parity requirement). Also fix the Vocabulary
guard examples (InputGuard/OutputGuard, not the nonexistent InputGuardrail/
CostGuard).

* test: statically validate doc snippets exist and parse

Every Python snippet in the capability READMEs and docs/*.md pages is now
checked for the two failures a reader hits immediately: it does not parse
(syntax), or it imports a pydantic_ai_harness symbol that does not exist (stale
module path or renamed name -- the class of bug behind the experimental.* import
drift). Static only: no model/network execution, so it needs no mocking. The
four illustrative API-signature blocks opt out with a {test="skip"} fence
(read by pytest-examples, stripped-safe for the unified-docs render).

* test: don't fail doc-snippet check on a missing optional extra

The static check imported capability modules to resolve their symbols, but in
the slim CI job (no extras) importing e.g. pydantic_ai_harness.experimental.acp
raises ModuleNotFoundError for the absent third-party 'acp' package -- the
harness module exists, its extra just isn't installed. Distinguish a genuinely
missing harness module (fail) from a missing extra (skip) by the ImportError's
module name.

2026-07-13 11:00:12 -05:00

4.6 KiB

Raw Permalink Blame History

Capability Authoring

Harness capabilities should be small, composable batteries built on Pydantic AI primitives.

Choose The Abstraction

Use AbstractCapability when the feature contributes instructions, model settings, toolsets, native tools, or lifecycle hooks.
Use a WrapperToolset when the feature changes how an existing toolset is presented or called.
Use a leaf AbstractToolset when the feature owns a new collection of tools.
Use hooks when behavior belongs at a specific point in the agent lifecycle.
Use capability ordering only when composition semantics require it. Keep the reason visible in the code or docstring.

If the feature changes provider wire behavior, normalized message structure, tool execution semantics, output selection, or durable execution primitives, it probably belongs in Pydantic AI core first.

Public Shape

Each capability package should normally have:

__init__.py with public exports
_capability.py for the public capability class
_toolset.py only if the capability needs toolset behavior
README.md with focused usage docs (serves GitHub and PyPI)
a unified-docs page at docs/<capability>.md (the docs/ folder is flat -- no capabilities/ or experimental/ subdirectories). It mirrors the README for the docs site, drops badges, links other harness pages with relative .md links and Pydantic AI docs with root-relative /ai/... links, links its source module, and -- where the capability exposes a public class -- may end with a ::: pydantic_ai_harness.<Class> autodoc block. The README and this page are kept in sync (see review-checklist.md "Docs").
mirrored tests under tests/<capability>/

The root pydantic_ai_harness/__init__.py should re-export stable public capabilities. Keep implementation helpers private unless users need them.

Capability Submodules And Exports

The experimental tier is retired. ACP is the sole remaining experimental capability (pydantic_ai_harness/experimental/acp/); do not add new capabilities there.

New capabilities land as a top-level submodule pydantic_ai_harness/<name>/. They are not re-exported from the root pydantic_ai_harness/__init__.py: each capability keeps its own optional dependencies, so importing the root package must not pull in a capability's extras. Users import a capability from its submodule (from pydantic_ai_harness.<name> import ...).

Naming: the module name is the capability name, one module per capability or strategy. Prefer a longer descriptive name over a terse one (e.g. overflowing_tool_output, not overflow). A known term is fine as-is (e.g. compaction). If you are unsure what to name a capability, ask the user (via the ask-user tool) rather than guessing -- a name is a public commitment once shipped.

When a capability's module path changes, keep the old path working as a DeprecationWarning shim so existing imports do not break.

Top-level re-exports in pydantic_ai_harness/__init__.py (CodeMode, FileSystem, Shell, ManagedPrompt) are the exception, not the rule. Once an export has shipped in a published release it is a backward-compatibility commitment: do not move, rename, or break it. Do not add new top-level re-exports.

APIs are subject to change between releases; breaking changes ship deprecation warnings where practical.

API Design

Prefer a small dataclass capability with typed fields.
Name fields by the user concept, not the implementation mechanism.
Accept the most generic useful input types.
Avoid Any in new public signatures.
Avoid casts. Fix the type shape instead.
Keep defaults conservative and easy to explain.
Do not add package dependencies without a clear issue and package-manager command.

Composition Checks

Before treating a capability as done, check how it composes with:

other capabilities in the same Agent(..., capabilities=[...])
toolsets and wrapper toolsets
ToolSearch
deferred tools and approval flows
provider-native versus local fallback tools
streaming/event behavior when the capability emits or wraps events
durable execution when the capability affects tool calls, context, serialization, retries, or lifecycle ordering

CodeMode is a useful reference for wrapper-toolset composition, tool selection, ToolSearch interaction, public docs, and test depth.

Docs

Each user-facing capability needs docs close to the code. Explain:

what problem it solves
minimal usage
key options
how it composes with relevant Pydantic AI features
important safety or execution constraints

Keep examples runnable with the declared extras.

4.6 KiB Raw Permalink Blame History