furyhawk/pydantic-ai-harness
1
0
Fork 0
mirror of https://github.com/pydantic/pydantic-ai-harness.git synced 2026-07-21 02:45:34 +00:00
Code Issues Packages Projects Releases Wiki Activity

Guide agents building harness capabilities (#247)

Browse Source
This commit is contained in:
David SF
2026-05-20 18:01:53 -05:00
committed by GitHub
parent 11189a5951
commit d85b6b2031
6 changed files with 295 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+48 -21
View File
@@ -1,43 +1,63 @@
# Pydantic AI Harness
## Local setup
## Repository purpose
On first session, if `CLAUDE.local.md` does not exist, create it with the following content:
`pydantic-ai-harness` is the first-party capability library for Pydantic AI.
```markdown
This file is checked into the repository but ignored, any changes made stay local.
Pydantic AI core owns the primitive runtime: agent loop semantics, normalized
messages, model/provider/profile behavior, tool execution semantics, durable
execution primitives, and generic capability hooks.
Use this file only to persist information about the specific workstation you are used on.
Harness owns optional, batteries-included compositions built from those
primitives: coding-agent tools, guardrails, memory, context management, repo
tools, verification loops, skills, planning, sub-agents, and other reusable
agent behaviors.
Capabilities branch is checked out at: <absolute path to the capabilities branch checkout on this system>
```
Fill in the absolute path where the `capabilities` branch of `pydantic-ai` is checked out on your system. If unknown, ask the user.
When a change needs new core semantics, stop and propose the Pydantic AI core
change instead of reimplementing core behavior in harness.
## Vocabulary
- **Capability**: an `AbstractCapability` subclass that bundles tools, hooks, instructions, and model settings into a reusable unit. This is the core abstraction of pydantic-ai-harness
- **Capability**: an `AbstractCapability` subclass that bundles tools, hooks, instructions, and model settings into a reusable unit. This is the core abstraction of pydantic-ai-harness.
- **Hook**: a lifecycle method on `AbstractCapability` that intercepts agent graph execution (e.g. `before_model_request`, `wrap_run`, `after_tool_execute`)
- **Toolset**: a collection of tools that a capability can provide to the agent
- **Guard**: a type of capability that validates inputs/outputs or controls tool access (e.g. `InputGuardrail`, `CostGuard`)
- **Harness**: this package -- a collection of pre-made capabilities for pydantic-ai
- **Harness**: this package -- a collection of pre-made capabilities for Pydantic AI.
- **AICA**: AI Code Assistant -- the automated agent that implements issues, reviews plans, and handles PR feedback
- **Ralph loop**: the state-machine-based workflow that drives AICA through phases (TRIAGE -> GOALS -> PLAN -> CODE -> VERIFY -> REVIEW -> PUBLISH)
- **DDD+ protocol**: classification system for PR review comments (do, dismiss, discuss, waiting, done)
## AICA preflight
Before implementing or reviewing a capability change:
1. Read `agent_docs/index.md`.
2. Read the linked `agent_docs/` guide for the task.
3. Read the public Pydantic AI docs for every integration point you touch:
- capabilities: <https://pydantic.dev/docs/ai/core-concepts/capabilities/>
- hooks: <https://pydantic.dev/docs/ai/core-concepts/hooks/>
- toolsets: <https://pydantic.dev/docs/ai/tools-toolsets/toolsets/>
- advanced tools: <https://pydantic.dev/docs/ai/tools-toolsets/tools-advanced/>
- agents: <https://pydantic.dev/docs/ai/core-concepts/agent/>
- testing: <https://pydantic.dev/docs/ai/guides/testing/>
4. Inspect the installed `pydantic_ai` package source for exact hook/toolset
signatures when needed. Do not assume a contributor's local checkout layout.
5. Use `pydantic_ai_harness.code_mode` as the exemplar for capability shape,
docs, tests, and public exports until another capability becomes a better
example.
## Capabilities API reference
When implementing a new capability, reference these docs in the pydantic-ai repo (path in `CLAUDE.local.md`):
When implementing a new capability, reference these docs:
- `docs/capabilities.md` -- main capabilities documentation, usage patterns, built-in capabilities
- `docs/hooks.md` -- lifecycle hooks reference, hook ordering, all hook categories
- `docs/extensibility.md` -- publishing capabilities as packages, spec serialization
- `docs/toolsets.md` -- toolset abstraction, building tools for capabilities
- `docs/tools-advanced.md` -- tool hooks, prepare_tools, tool validation
- `docs/agent.md` -- agent configuration, instructions, model settings
- `pydantic_ai_slim/pydantic_ai/capabilities/abstract.py` -- the `AbstractCapability` base class (all hook methods)
- `pydantic_ai_slim/pydantic_ai/capabilities/hooks.py` -- decorator-based `Hooks` capability
- `pydantic_ai_slim/pydantic_ai/capabilities/combined.py` -- `CombinedCapability` for composition
- <https://pydantic.dev/docs/ai/core-concepts/capabilities/> -- main capabilities documentation, usage patterns, built-in capabilities
- <https://pydantic.dev/docs/ai/core-concepts/hooks/> -- lifecycle hooks reference, hook ordering, all hook categories
- <https://pydantic.dev/docs/ai/guides/extensibility/> -- publishing capabilities as packages, spec serialization
- <https://pydantic.dev/docs/ai/tools-toolsets/toolsets/> -- toolset abstraction, building tools for capabilities
- <https://pydantic.dev/docs/ai/tools-toolsets/tools-advanced/> -- tool hooks, prepare tools, tool validation
- <https://pydantic.dev/docs/ai/core-concepts/agent/> -- agent configuration, instructions, model settings
- Installed `pydantic_ai.capabilities` source -- `AbstractCapability`, hook signatures, and composition behavior
- Installed `pydantic_ai.toolsets` source -- `AbstractToolset`, `WrapperToolset`, and `ToolsetTool`
## Coding standards
@@ -84,12 +104,19 @@ tests/
test_<capability>.py
```
Do not add placeholder template files for new capabilities. Start from the
existing `CodeMode` package shape, then delete what the new capability does not
need.
## Testing patterns
- Use `pydantic_ai.models.TestModel` for all tests (no real API calls)
- `ALLOW_MODEL_REQUESTS = False` is set globally in `conftest.py`
- Tests use `pytest-anyio` for async support
- Each capability test class follows: `TestCapabilityName` with methods `test_<scenario>`
- Prefer tests through `Agent(..., capabilities=[...])` when that is the public
behavior. Use direct `Toolset`/`RunContext` tests for lower-level lifecycle,
schema, retry, or wrapper behavior that is hard to isolate through `Agent`.
## Contributing rules for AICAs
agent_docs/capability-authoring.md
+71
View File
@@ -0,0 +1,71 @@
# Capability Authoring
Harness capabilities should be small, composable batteries built on Pydantic AI
primitives.
## Choose The Abstraction
- Use `AbstractCapability` when the feature contributes instructions, model
settings, toolsets, native tools, or lifecycle hooks.
- Use a `WrapperToolset` when the feature changes how an existing toolset is
presented or called.
- Use a leaf `AbstractToolset` when the feature owns a new collection of tools.
- Use hooks when behavior belongs at a specific point in the agent lifecycle.
- Use capability ordering only when composition semantics require it. Keep the
reason visible in the code or docstring.
If the feature changes provider wire behavior, normalized message structure,
tool execution semantics, output selection, or durable execution primitives, it
probably belongs in Pydantic AI core first.
## Public Shape
Each capability package should normally have:
- `__init__.py` with public exports
- `_capability.py` for the public capability class
- `_toolset.py` only if the capability needs toolset behavior
- `README.md` with focused usage docs
- mirrored tests under `tests/<capability>/`
The root `pydantic_ai_harness/__init__.py` should re-export stable public
capabilities. Keep implementation helpers private unless users need them.
## API Design
- Prefer a small dataclass capability with typed fields.
- Name fields by the user concept, not the implementation mechanism.
- Accept the most generic useful input types.
- Avoid `Any` in new public signatures.
- Avoid casts. Fix the type shape instead.
- Keep defaults conservative and easy to explain.
- Do not add package dependencies without a clear issue and package-manager
command.
## Composition Checks
Before treating a capability as done, check how it composes with:
- other capabilities in the same `Agent(..., capabilities=[...])`
- toolsets and wrapper toolsets
- `ToolSearch`
- deferred tools and approval flows
- provider-native versus local fallback tools
- streaming/event behavior when the capability emits or wraps events
- durable execution when the capability affects tool calls, context,
serialization, retries, or lifecycle ordering
`CodeMode` is a useful reference for wrapper-toolset composition, tool
selection, `ToolSearch` interaction, public docs, and test depth.
## Docs
Each user-facing capability needs docs close to the code. Explain:
- what problem it solves
- minimal usage
- key options
- how it composes with relevant Pydantic AI features
- important safety or execution constraints
Keep examples runnable with the declared extras.
agent_docs/core-boundary.md
+43
View File
@@ -0,0 +1,43 @@
# Core Boundary
Harness should extend Pydantic AI through public primitives. It should not
quietly create a second runtime.
## Belongs In Pydantic AI Core
Start in Pydantic AI core when the change needs:
- new agent loop semantics
- new normalized message parts or message-history behavior
- provider API compatibility or provider wire mapping
- model/profile/provider capability facts
- generic tool execution semantics
- output-mode semantics
- durable execution primitives
- generic capability hooks or hook ordering changes
- MCP protocol behavior that should apply to all users
Harness can depend on these once core exposes the right primitive.
## Belongs In Harness
Harness is the right home for:
- reusable capability compositions
- coding-agent tools and repo workflows
- guardrails built on hooks
- memory and persistence policies
- context management policies
- tool-output handling policies
- planning, skills, task tracking, and sub-agent compositions
- opinionated defaults that are useful but not fundamental to every Pydantic AI
user
## Decision Rule
If a feature would be hard for a third-party capability package to implement
correctly through public Pydantic AI APIs, do not work around that in harness.
Identify the missing core primitive and propose that change first.
If a feature is mostly a policy decision over existing hooks/toolsets/messages,
build it in harness.
agent_docs/index.md
+45
View File
@@ -0,0 +1,45 @@
# Harness Agent Docs
Use these guides when building or reviewing `pydantic-ai-harness` changes.
## Required Reads
For any code change:
1. `AGENTS.md`
2. This file
3. The guide below that matches the task
4. The public Pydantic AI docs for the integration points you touch
## Task Routing
- New or changed capability API: `capability-authoring.md`
- New or changed tests: `testing-capabilities.md`
- Unsure whether behavior belongs in harness or Pydantic AI core: `core-boundary.md`
- Review, pre-PR check, or final self-check: `review-checklist.md`
## Exemplar
Use `pydantic_ai_harness.code_mode` as the current exemplar for capability
shape:
- public re-export from `pydantic_ai_harness/__init__.py`
- package-level re-export from `pydantic_ai_harness/code_mode/__init__.py`
- public capability class in `_capability.py`
- implementation toolset in `_toolset.py`
- capability README next to implementation
- mirrored tests under `tests/code_mode/`
Do not copy `CodeMode` mechanically. Use it to understand package shape,
testing depth, docs placement, and how a harness capability composes with
Pydantic AI toolsets.
## Pydantic AI References
- Capabilities: <https://pydantic.dev/docs/ai/core-concepts/capabilities/>
- Hooks: <https://pydantic.dev/docs/ai/core-concepts/hooks/>
- Toolsets: <https://pydantic.dev/docs/ai/tools-toolsets/toolsets/>
- Advanced tools: <https://pydantic.dev/docs/ai/tools-toolsets/tools-advanced/>
- Agents: <https://pydantic.dev/docs/ai/core-concepts/agent/>
- Testing: <https://pydantic.dev/docs/ai/guides/testing/>
- Extensibility: <https://pydantic.dev/docs/ai/guides/extensibility/>
agent_docs/review-checklist.md
+36
View File
@@ -0,0 +1,36 @@
# Review Checklist
Use this before opening a PR or reviewing a capability change.
## Product Fit
- The capability has a clear user or dogfooding need.
- The behavior belongs in harness, not Pydantic AI core.
- The public API is small and named around user concepts.
- The capability composes with relevant existing capabilities.
## Implementation
- Public exports are intentional.
- Private helpers stay private.
- Types are precise; new public signatures do not use `Any`.
- No casts are used to paper over type design.
- The implementation uses Pydantic AI hooks/toolsets instead of duplicating core
runtime behavior.
- Capability ordering is justified when present.
- Dependency changes were made through `uv` and have a clear reason.
## Tests
- Tests cover the public `Agent(..., capabilities=[...])` path where possible.
- Lower-level tests cover lifecycle, schemas, retries, and metadata when needed.
- Error paths and important option combinations are covered.
- Relevant protocol-shaped output is snapshotted.
- `make lint`, `make typecheck`, and `make test` pass before handoff.
## Docs
- Capability README or root README is updated for user-facing behavior.
- Examples match declared extras.
- Docs explain composition constraints and safety implications.
- The PR links an issue.
agent_docs/testing-capabilities.md
+52
View File
@@ -0,0 +1,52 @@
# Testing Capabilities
Harness tests should exercise the behavior users rely on, not only private
helpers.
## Default Shape
- Use `pydantic_ai.models.TestModel` for model behavior.
- Keep real provider calls out of tests.
- Prefer `Agent(..., capabilities=[...])` tests for public behavior.
- Mirror source packages under `tests/<capability>/`.
- Use `pytest-anyio` for async capability/toolset behavior.
## Lower-Level Tests
Direct toolset tests are appropriate when you need to inspect:
- listed tools and schemas
- wrapper-toolset lifecycle
- retry behavior
- metadata and synthetic tool-call records
- `RunContext` or `ToolManager` interactions
- edge cases that are awkward to force through a full agent run
Use the `CodeMode` tests as the current reference for direct `RunContext` and
`ToolManager` setup.
## Coverage
The project enforces 100% branch coverage with `make testcov`. Tests for a new
capability should cover:
- default configuration
- important option combinations
- failure and retry paths
- composition with relevant Pydantic AI features
- docs examples when examples are executable
Use snapshots when behavior is protocol-shaped: messages, event streams,
schemas, telemetry spans, or structured tool metadata.
## Commands
Run focused checks first, then broaden:
```bash
uv run pytest tests/<capability>
make lint
make typecheck
make test
make testcov
```