mirror of
https://github.com/pydantic/pydantic-ai-harness.git
synced 2026-07-21 02:45:34 +00:00
refactor(budget_disclosure): graduate to top-level layout, add docs-site page
Merge `main` and relocate the capability from `experimental/budget_disclosure` to the canonical top-level `pydantic_ai_harness/budget_disclosure/`, matching the layout from #347/#354. Drops the experimental import warning, fixes import paths, and moves tests. Adds the `docs/budget-disclosure.md` page (registered in nav.json, index.md, and the parity gate) and a README matrix row. No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
3fa45e5f9c
commit
166f8d7ca7
@@ -172,6 +172,7 @@ We studied leading coding agents, agent frameworks, and Claw-style assistants to
|
||||
| **Safety & guardrails** | **Input guardrails** | Validate user input before the agent run starts | :white_check_mark: [Docs](pydantic_ai_harness/guardrails/) | [pydantic-ai-shields](https://github.com/vstorm-co/pydantic-ai-shields) (vstorm‑co) |
|
||||
| | **Output guardrails** | Validate model output after the run completes | :white_check_mark: [Docs](pydantic_ai_harness/guardrails/) | [pydantic-ai-shields](https://github.com/vstorm-co/pydantic-ai-shields) (vstorm‑co) |
|
||||
| | **Cost/token budgets** | Enforce token and cost limits per run | :construction: [PR #182](https://github.com/pydantic/pydantic-ai-harness/pull/182) | [pydantic-ai-shields](https://github.com/vstorm-co/pydantic-ai-shields) (vstorm‑co) |
|
||||
| | **Budget disclosure** | Disclose the run's remaining token, cost, and tool-call budget to the model | :white_check_mark: [Docs](pydantic_ai_harness/budget_disclosure/) | |
|
||||
| | **Tool access control** | Block tools or require approval before execution | :construction: [PR #182](https://github.com/pydantic/pydantic-ai-harness/pull/182) | [pydantic-ai-shields](https://github.com/vstorm-co/pydantic-ai-shields) (vstorm‑co) |
|
||||
| | **Async guardrails** | Run validation concurrently with model requests | :construction: [PR #182](https://github.com/pydantic/pydantic-ai-harness/pull/182) | [pydantic-ai-shields](https://github.com/vstorm-co/pydantic-ai-shields) (vstorm‑co) |
|
||||
| | **Secret masking** | Detect and redact secrets in agent I/O | :construction: [PR #172](https://github.com/pydantic/pydantic-ai-harness/pull/172) | [pydantic-ai-shields](https://github.com/vstorm-co/pydantic-ai-shields) (vstorm‑co) |
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
---
|
||||
title: Budget Disclosure
|
||||
description: Tell the model how much of its usage budget is left, so it can pace itself.
|
||||
---
|
||||
|
||||
# Budget Disclosure
|
||||
|
||||
Tell the model how much of its usage budget is left, so it can pace itself.
|
||||
|
||||
> [!NOTE]
|
||||
> Import this capability from its submodule. It is not re-exported from `pydantic_ai_harness`:
|
||||
>
|
||||
> ```python
|
||||
> from pydantic_ai_harness.budget_disclosure import BudgetDisclosure
|
||||
> ```
|
||||
|
||||
Budget Disclosure is a released, non-experimental capability. Pydantic AI Harness is still on 0.x releases, so the API may change between minor releases. See the repository [version policy](https://github.com/pydantic/pydantic-ai-harness#version-policy).
|
||||
|
||||
`UsageLimits` enforces a run's budget, but the model never sees it. It gets stopped mid-step
|
||||
by a `UsageLimitExceeded` instead of adjusting as the budget runs down. `BudgetDisclosure`
|
||||
reads the run's accumulated usage on each request and contributes one short line of
|
||||
remaining-budget state, so the model can change strategy: with little left, land current
|
||||
results rather than start new work.
|
||||
|
||||
```
|
||||
Budget remaining: ~38k tokens, 7 requests. Pace your work; if nearly exhausted, prioritize
|
||||
delivering current results over starting new work.
|
||||
```
|
||||
|
||||
## Cache safety
|
||||
|
||||
The line is contributed through the capability `get_instructions` channel, so it is
|
||||
*ephemeral*: instructions are rebuilt for every request and are never stored as a message part
|
||||
in the run history. A remaining-budget number changes on every request by construction, so
|
||||
persisting it into the history would move the cacheable prefix each turn and re-charge the whole
|
||||
conversation. Instructions sit in the system-prompt region (for providers where instructions are
|
||||
the system-prompt tail, they sit after the cached tools + history prefix), so a changing budget
|
||||
line there does not disturb that prefix. The numbers are rounded and the line kept short
|
||||
regardless, so it costs little and reads as a stable-shaped status line rather than churning
|
||||
detail.
|
||||
|
||||
## Minimal usage
|
||||
|
||||
```python
|
||||
from pydantic_ai import Agent
|
||||
from pydantic_ai.usage import UsageLimits
|
||||
from pydantic_ai_harness.budget_disclosure import BudgetDisclosure
|
||||
|
||||
limits = UsageLimits(request_limit=20, total_tokens_limit=200_000)
|
||||
agent = Agent('anthropic:claude-sonnet-4-5', capabilities=[BudgetDisclosure(limits=limits)])
|
||||
await agent.run('...', usage_limits=limits)
|
||||
```
|
||||
|
||||
The run's `UsageLimits` is not reachable from a capability: it is passed to `Agent.run(...,
|
||||
usage_limits=...)` and lives on the agent graph's run deps, not on the `RunContext` a capability
|
||||
sees. So pass the same `UsageLimits` to the capability's `limits`. This mirrors
|
||||
`compaction.LimitWarner`, which is configured with its own `max_*` ceilings for the same reason.
|
||||
|
||||
When no `limits` are configured, or none of the disclosed dimensions has a limit set, the
|
||||
capability contributes nothing.
|
||||
|
||||
## Options
|
||||
|
||||
- `limits` (default `None`): the run's `UsageLimits`. `None` discloses nothing.
|
||||
- `disclose` (default `None`): which limited dimensions to disclose, from `requests`,
|
||||
`tool_calls`, `input_tokens`, `output_tokens`, `total_tokens`. `None` discloses every dimension
|
||||
whose limit is set on `limits`. An explicit collection must only name dimensions whose limit is
|
||||
set.
|
||||
- `start_at` (default `0.0`): fraction of a limit (0..1) that must be consumed before disclosure
|
||||
begins. `0.0` always discloses; `0.5` discloses only once any one disclosed dimension is at
|
||||
least half consumed, so short runs that never approach the budget stay silent.
|
||||
- `round_tokens_to` (default `1000`): granularity that remaining token counts are rounded to for
|
||||
display. Request and tool-call counts are small and shown exactly.
|
||||
- `format` (default `None`): override the line. Receives the run context and the remaining budget
|
||||
per active dimension (raw, unrounded); returns the line, or `None` to contribute nothing this
|
||||
request. When set, `round_tokens_to` and the default wording do not apply.
|
||||
|
||||
## Relationship to tool budgets
|
||||
|
||||
This capability *discloses* remaining budget; it does not enforce it. It complements
|
||||
enforcement capabilities (`ToolBudget`, #168) and the underlying `UsageLimits`: those stop the
|
||||
run when a ceiling is hit, this one lets the model see the ceiling coming and pace toward it.
|
||||
|
||||
## Scope
|
||||
|
||||
- **Stateless.** It reads `ctx.usage` and the static `limits` each request; there is no per-run
|
||||
state, so one instance can be reused across many runs.
|
||||
- **Disclosure only.** It never changes tool availability, model settings, or message history.
|
||||
|
||||
## Further reading
|
||||
|
||||
- [`pydantic_ai_harness.budget_disclosure` source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/budget_disclosure/)
|
||||
- [Pydantic AI capabilities](/ai/core-concepts/capabilities/)
|
||||
@@ -126,6 +126,7 @@ Each capability is a self-contained battery you drop into an agent's `capabiliti
|
||||
| [Memory](memory.md) | Gives an agent a persistent, namespaced notebook with bounded prompt injection, on-demand search, and concurrency-safe stores. | -- |
|
||||
| [Runtime Authoring](runtime-authoring.md) | Lets an agent author, validate, and load real capabilities at runtime. | -- |
|
||||
| [Guardrails](guardrails.md) | Validates user input before a run starts and model output after it completes -- block or redact, with structured results. | -- |
|
||||
| [Budget Disclosure](budget-disclosure.md) | Discloses the run's remaining token, cost, and tool-call budget to the model. | -- |
|
||||
| [Managed Prompt](managed-prompt.md) | Backs an agent's instructions with a [Logfire-managed prompt](https://logfire.pydantic.dev/docs/reference/advanced/prompt-management/), so you can version, label, and roll out prompt changes from the Logfire UI without redeploying -- with a code default that keeps the agent working when no remote value is available. | `logfire` |
|
||||
| [ACP](acp.md) *(experimental)* | Serves an agent to editors (Zed, etc.) over the [Agent Client Protocol](https://agentclientprotocol.com) -- streamed text, diff-rendered edits, and tool approval. | `acp` |
|
||||
|
||||
|
||||
@@ -18,6 +18,7 @@
|
||||
{ "label": "Memory", "slug": "memory" },
|
||||
{ "label": "Runtime Authoring", "slug": "runtime-authoring" },
|
||||
{ "label": "Guardrails", "slug": "guardrails" },
|
||||
{ "label": "Budget Disclosure", "slug": "budget-disclosure" },
|
||||
{ "label": "Managed Prompt", "slug": "managed-prompt" },
|
||||
{ "label": "ACP", "slug": "acp" }
|
||||
]
|
||||
|
||||
+16
-6
@@ -1,7 +1,16 @@
|
||||
# BudgetDisclosure
|
||||
# Budget Disclosure
|
||||
|
||||
Tell the model how much of its usage budget is left, so it can pace itself.
|
||||
|
||||
> [!NOTE]
|
||||
> Import this capability from its submodule. It is not re-exported from `pydantic_ai_harness`:
|
||||
>
|
||||
> ```python
|
||||
> from pydantic_ai_harness.budget_disclosure import BudgetDisclosure
|
||||
> ```
|
||||
|
||||
Budget Disclosure is a released, non-experimental capability. Pydantic AI Harness is still on 0.x releases, so the API may change between minor releases. See the repository [version policy](https://github.com/pydantic/pydantic-ai-harness#version-policy).
|
||||
|
||||
`UsageLimits` enforces a run's budget, but the model never sees it. It gets stopped mid-step
|
||||
by a `UsageLimitExceeded` instead of adjusting as the budget runs down. `BudgetDisclosure`
|
||||
reads the run's accumulated usage on each request and contributes one short line of
|
||||
@@ -13,10 +22,6 @@ Budget remaining: ~38k tokens, 7 requests. Pace your work; if nearly exhausted,
|
||||
delivering current results over starting new work.
|
||||
```
|
||||
|
||||
> This capability is experimental and private. It is not re-exported from
|
||||
> `pydantic_ai_harness`; import it from its own module. Its API may change or be removed in any
|
||||
> release.
|
||||
|
||||
## Cache safety
|
||||
|
||||
The line is contributed through the capability `get_instructions` channel, so it is
|
||||
@@ -34,7 +39,7 @@ detail.
|
||||
```python
|
||||
from pydantic_ai import Agent
|
||||
from pydantic_ai.usage import UsageLimits
|
||||
from pydantic_ai_harness.experimental.budget_disclosure import BudgetDisclosure
|
||||
from pydantic_ai_harness.budget_disclosure import BudgetDisclosure
|
||||
|
||||
limits = UsageLimits(request_limit=20, total_tokens_limit=200_000)
|
||||
agent = Agent('anthropic:claude-sonnet-4-5', capabilities=[BudgetDisclosure(limits=limits)])
|
||||
@@ -76,3 +81,8 @@ run when a ceiling is hit, this one lets the model see the ceiling coming and pa
|
||||
- **Stateless.** It reads `ctx.usage` and the static `limits` each request; there is no per-run
|
||||
state, so one instance can be reused across many runs.
|
||||
- **Disclosure only.** It never changes tool availability, model settings, or message history.
|
||||
|
||||
## Further reading
|
||||
|
||||
- [`pydantic_ai_harness.budget_disclosure` source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/budget_disclosure/)
|
||||
- [Pydantic AI capabilities](https://ai.pydantic.dev/capabilities/)
|
||||
+1
-4
@@ -1,14 +1,11 @@
|
||||
"""Disclose the run's remaining usage budget to the model (private, not re-exported at top level)."""
|
||||
|
||||
from pydantic_ai_harness.experimental._warn import warn_experimental
|
||||
from pydantic_ai_harness.experimental.budget_disclosure._capability import (
|
||||
from pydantic_ai_harness.budget_disclosure._capability import (
|
||||
BudgetDimension,
|
||||
BudgetDisclosure,
|
||||
BudgetFormatter,
|
||||
)
|
||||
|
||||
warn_experimental('budget_disclosure')
|
||||
|
||||
__all__ = [
|
||||
'BudgetDimension',
|
||||
'BudgetDisclosure',
|
||||
+1
-1
@@ -96,7 +96,7 @@ class BudgetDisclosure(AbstractCapability[AgentDepsT]):
|
||||
```python
|
||||
from pydantic_ai import Agent
|
||||
from pydantic_ai.usage import UsageLimits
|
||||
from pydantic_ai_harness.experimental.budget_disclosure import BudgetDisclosure
|
||||
from pydantic_ai_harness.budget_disclosure import BudgetDisclosure
|
||||
|
||||
limits = UsageLimits(request_limit=20, total_tokens_limit=200_000)
|
||||
agent = Agent('anthropic:claude-sonnet-4-5', capabilities=[BudgetDisclosure(limits=limits)])
|
||||
+1
-1
@@ -18,7 +18,7 @@ from pydantic_ai.messages import ModelMessage, ModelResponse, TextPart, ToolCall
|
||||
from pydantic_ai.models.function import AgentInfo, FunctionModel
|
||||
from pydantic_ai.usage import RequestUsage, RunUsage, UsageLimits
|
||||
|
||||
from pydantic_ai_harness.experimental.budget_disclosure import BudgetDimension, BudgetDisclosure
|
||||
from pydantic_ai_harness.budget_disclosure import BudgetDimension, BudgetDisclosure
|
||||
|
||||
pytestmark = pytest.mark.anyio
|
||||
|
||||
@@ -126,6 +126,7 @@ _CAPABILITY_PAGE_META = {
|
||||
'dynamic-workflow.md': ('dynamic_workflow', 'Dynamic Workflow'),
|
||||
'planning.md': ('planning', 'Planning'),
|
||||
'runtime-authoring.md': ('runtime_authoring', 'Runtime Authoring'),
|
||||
'budget-disclosure.md': ('budget_disclosure', 'Budget Disclosure'),
|
||||
'guardrails.md': ('guardrails', 'Input & Output Guardrails'),
|
||||
'acp.md': ('experimental/acp', 'ACP (Agent Client Protocol)'),
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user