diff --git a/README.md b/README.md index d395015..3934c46 100644 --- a/README.md +++ b/README.md @@ -172,6 +172,7 @@ We studied leading coding agents, agent frameworks, and Claw-style assistants to | **Safety & guardrails** | **Input guardrails** | Validate user input before the agent run starts | :white_check_mark: [Docs](pydantic_ai_harness/guardrails/) | [pydantic-ai-shields](https://github.com/vstorm-co/pydantic-ai-shields) (vstorm‑co) | | | **Output guardrails** | Validate model output after the run completes | :white_check_mark: [Docs](pydantic_ai_harness/guardrails/) | [pydantic-ai-shields](https://github.com/vstorm-co/pydantic-ai-shields) (vstorm‑co) | | | **Cost/token budgets** | Enforce token and cost limits per run | :construction: [PR #182](https://github.com/pydantic/pydantic-ai-harness/pull/182) | [pydantic-ai-shields](https://github.com/vstorm-co/pydantic-ai-shields) (vstorm‑co) | +| | **Budget disclosure** | Disclose the run's remaining token, cost, and tool-call budget to the model | :white_check_mark: [Docs](pydantic_ai_harness/budget_disclosure/) | | | | **Tool access control** | Block tools or require approval before execution | :construction: [PR #182](https://github.com/pydantic/pydantic-ai-harness/pull/182) | [pydantic-ai-shields](https://github.com/vstorm-co/pydantic-ai-shields) (vstorm‑co) | | | **Async guardrails** | Run validation concurrently with model requests | :construction: [PR #182](https://github.com/pydantic/pydantic-ai-harness/pull/182) | [pydantic-ai-shields](https://github.com/vstorm-co/pydantic-ai-shields) (vstorm‑co) | | | **Secret masking** | Detect and redact secrets in agent I/O | :construction: [PR #172](https://github.com/pydantic/pydantic-ai-harness/pull/172) | [pydantic-ai-shields](https://github.com/vstorm-co/pydantic-ai-shields) (vstorm‑co) | diff --git a/docs/budget-disclosure.md b/docs/budget-disclosure.md new file mode 100644 index 0000000..0cbc74a --- /dev/null +++ b/docs/budget-disclosure.md @@ -0,0 +1,93 @@ +--- +title: Budget Disclosure +description: Tell the model how much of its usage budget is left, so it can pace itself. +--- + +# Budget Disclosure + +Tell the model how much of its usage budget is left, so it can pace itself. + +> [!NOTE] +> Import this capability from its submodule. It is not re-exported from `pydantic_ai_harness`: +> +> ```python +> from pydantic_ai_harness.budget_disclosure import BudgetDisclosure +> ``` + +Budget Disclosure is a released, non-experimental capability. Pydantic AI Harness is still on 0.x releases, so the API may change between minor releases. See the repository [version policy](https://github.com/pydantic/pydantic-ai-harness#version-policy). + +`UsageLimits` enforces a run's budget, but the model never sees it. It gets stopped mid-step +by a `UsageLimitExceeded` instead of adjusting as the budget runs down. `BudgetDisclosure` +reads the run's accumulated usage on each request and contributes one short line of +remaining-budget state, so the model can change strategy: with little left, land current +results rather than start new work. + +``` +Budget remaining: ~38k tokens, 7 requests. Pace your work; if nearly exhausted, prioritize +delivering current results over starting new work. +``` + +## Cache safety + +The line is contributed through the capability `get_instructions` channel, so it is +*ephemeral*: instructions are rebuilt for every request and are never stored as a message part +in the run history. A remaining-budget number changes on every request by construction, so +persisting it into the history would move the cacheable prefix each turn and re-charge the whole +conversation. Instructions sit in the system-prompt region (for providers where instructions are +the system-prompt tail, they sit after the cached tools + history prefix), so a changing budget +line there does not disturb that prefix. The numbers are rounded and the line kept short +regardless, so it costs little and reads as a stable-shaped status line rather than churning +detail. + +## Minimal usage + +```python +from pydantic_ai import Agent +from pydantic_ai.usage import UsageLimits +from pydantic_ai_harness.budget_disclosure import BudgetDisclosure + +limits = UsageLimits(request_limit=20, total_tokens_limit=200_000) +agent = Agent('anthropic:claude-sonnet-4-5', capabilities=[BudgetDisclosure(limits=limits)]) +await agent.run('...', usage_limits=limits) +``` + +The run's `UsageLimits` is not reachable from a capability: it is passed to `Agent.run(..., +usage_limits=...)` and lives on the agent graph's run deps, not on the `RunContext` a capability +sees. So pass the same `UsageLimits` to the capability's `limits`. This mirrors +`compaction.LimitWarner`, which is configured with its own `max_*` ceilings for the same reason. + +When no `limits` are configured, or none of the disclosed dimensions has a limit set, the +capability contributes nothing. + +## Options + +- `limits` (default `None`): the run's `UsageLimits`. `None` discloses nothing. +- `disclose` (default `None`): which limited dimensions to disclose, from `requests`, + `tool_calls`, `input_tokens`, `output_tokens`, `total_tokens`. `None` discloses every dimension + whose limit is set on `limits`. An explicit collection must only name dimensions whose limit is + set. +- `start_at` (default `0.0`): fraction of a limit (0..1) that must be consumed before disclosure + begins. `0.0` always discloses; `0.5` discloses only once any one disclosed dimension is at + least half consumed, so short runs that never approach the budget stay silent. +- `round_tokens_to` (default `1000`): granularity that remaining token counts are rounded to for + display. Request and tool-call counts are small and shown exactly. +- `format` (default `None`): override the line. Receives the run context and the remaining budget + per active dimension (raw, unrounded); returns the line, or `None` to contribute nothing this + request. When set, `round_tokens_to` and the default wording do not apply. + +## Relationship to tool budgets + +This capability *discloses* remaining budget; it does not enforce it. It complements +enforcement capabilities (`ToolBudget`, #168) and the underlying `UsageLimits`: those stop the +run when a ceiling is hit, this one lets the model see the ceiling coming and pace toward it. + +## Scope + +- **Stateless.** It reads `ctx.usage` and the static `limits` each request; there is no per-run + state, so one instance can be reused across many runs. +- **Disclosure only.** It never changes tool availability, model settings, or message history. + +## Further reading + +- [`pydantic_ai_harness.budget_disclosure` source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/budget_disclosure/) +- [Pydantic AI capabilities](/ai/core-concepts/capabilities/) diff --git a/docs/index.md b/docs/index.md index bebcc40..436a046 100644 --- a/docs/index.md +++ b/docs/index.md @@ -126,6 +126,7 @@ Each capability is a self-contained battery you drop into an agent's `capabiliti | [Memory](memory.md) | Gives an agent a persistent, namespaced notebook with bounded prompt injection, on-demand search, and concurrency-safe stores. | -- | | [Runtime Authoring](runtime-authoring.md) | Lets an agent author, validate, and load real capabilities at runtime. | -- | | [Guardrails](guardrails.md) | Validates user input before a run starts and model output after it completes -- block or redact, with structured results. | -- | +| [Budget Disclosure](budget-disclosure.md) | Discloses the run's remaining token, cost, and tool-call budget to the model. | -- | | [Managed Prompt](managed-prompt.md) | Backs an agent's instructions with a [Logfire-managed prompt](https://logfire.pydantic.dev/docs/reference/advanced/prompt-management/), so you can version, label, and roll out prompt changes from the Logfire UI without redeploying -- with a code default that keeps the agent working when no remote value is available. | `logfire` | | [ACP](acp.md) *(experimental)* | Serves an agent to editors (Zed, etc.) over the [Agent Client Protocol](https://agentclientprotocol.com) -- streamed text, diff-rendered edits, and tool approval. | `acp` | diff --git a/docs/nav.json b/docs/nav.json index a2ca4c7..eef14c5 100644 --- a/docs/nav.json +++ b/docs/nav.json @@ -18,6 +18,7 @@ { "label": "Memory", "slug": "memory" }, { "label": "Runtime Authoring", "slug": "runtime-authoring" }, { "label": "Guardrails", "slug": "guardrails" }, + { "label": "Budget Disclosure", "slug": "budget-disclosure" }, { "label": "Managed Prompt", "slug": "managed-prompt" }, { "label": "ACP", "slug": "acp" } ] diff --git a/pydantic_ai_harness/experimental/budget_disclosure/README.md b/pydantic_ai_harness/budget_disclosure/README.md similarity index 83% rename from pydantic_ai_harness/experimental/budget_disclosure/README.md rename to pydantic_ai_harness/budget_disclosure/README.md index 9ca91ef..e8c6358 100644 --- a/pydantic_ai_harness/experimental/budget_disclosure/README.md +++ b/pydantic_ai_harness/budget_disclosure/README.md @@ -1,7 +1,16 @@ -# BudgetDisclosure +# Budget Disclosure Tell the model how much of its usage budget is left, so it can pace itself. +> [!NOTE] +> Import this capability from its submodule. It is not re-exported from `pydantic_ai_harness`: +> +> ```python +> from pydantic_ai_harness.budget_disclosure import BudgetDisclosure +> ``` + +Budget Disclosure is a released, non-experimental capability. Pydantic AI Harness is still on 0.x releases, so the API may change between minor releases. See the repository [version policy](https://github.com/pydantic/pydantic-ai-harness#version-policy). + `UsageLimits` enforces a run's budget, but the model never sees it. It gets stopped mid-step by a `UsageLimitExceeded` instead of adjusting as the budget runs down. `BudgetDisclosure` reads the run's accumulated usage on each request and contributes one short line of @@ -13,10 +22,6 @@ Budget remaining: ~38k tokens, 7 requests. Pace your work; if nearly exhausted, delivering current results over starting new work. ``` -> This capability is experimental and private. It is not re-exported from -> `pydantic_ai_harness`; import it from its own module. Its API may change or be removed in any -> release. - ## Cache safety The line is contributed through the capability `get_instructions` channel, so it is @@ -34,7 +39,7 @@ detail. ```python from pydantic_ai import Agent from pydantic_ai.usage import UsageLimits -from pydantic_ai_harness.experimental.budget_disclosure import BudgetDisclosure +from pydantic_ai_harness.budget_disclosure import BudgetDisclosure limits = UsageLimits(request_limit=20, total_tokens_limit=200_000) agent = Agent('anthropic:claude-sonnet-4-5', capabilities=[BudgetDisclosure(limits=limits)]) @@ -76,3 +81,8 @@ run when a ceiling is hit, this one lets the model see the ceiling coming and pa - **Stateless.** It reads `ctx.usage` and the static `limits` each request; there is no per-run state, so one instance can be reused across many runs. - **Disclosure only.** It never changes tool availability, model settings, or message history. + +## Further reading + +- [`pydantic_ai_harness.budget_disclosure` source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/budget_disclosure/) +- [Pydantic AI capabilities](https://ai.pydantic.dev/capabilities/) diff --git a/pydantic_ai_harness/experimental/budget_disclosure/__init__.py b/pydantic_ai_harness/budget_disclosure/__init__.py similarity index 57% rename from pydantic_ai_harness/experimental/budget_disclosure/__init__.py rename to pydantic_ai_harness/budget_disclosure/__init__.py index 23f296d..d0c2a57 100644 --- a/pydantic_ai_harness/experimental/budget_disclosure/__init__.py +++ b/pydantic_ai_harness/budget_disclosure/__init__.py @@ -1,14 +1,11 @@ """Disclose the run's remaining usage budget to the model (private, not re-exported at top level).""" -from pydantic_ai_harness.experimental._warn import warn_experimental -from pydantic_ai_harness.experimental.budget_disclosure._capability import ( +from pydantic_ai_harness.budget_disclosure._capability import ( BudgetDimension, BudgetDisclosure, BudgetFormatter, ) -warn_experimental('budget_disclosure') - __all__ = [ 'BudgetDimension', 'BudgetDisclosure', diff --git a/pydantic_ai_harness/experimental/budget_disclosure/_capability.py b/pydantic_ai_harness/budget_disclosure/_capability.py similarity index 99% rename from pydantic_ai_harness/experimental/budget_disclosure/_capability.py rename to pydantic_ai_harness/budget_disclosure/_capability.py index bbbb17a..336a5f5 100644 --- a/pydantic_ai_harness/experimental/budget_disclosure/_capability.py +++ b/pydantic_ai_harness/budget_disclosure/_capability.py @@ -96,7 +96,7 @@ class BudgetDisclosure(AbstractCapability[AgentDepsT]): ```python from pydantic_ai import Agent from pydantic_ai.usage import UsageLimits - from pydantic_ai_harness.experimental.budget_disclosure import BudgetDisclosure + from pydantic_ai_harness.budget_disclosure import BudgetDisclosure limits = UsageLimits(request_limit=20, total_tokens_limit=200_000) agent = Agent('anthropic:claude-sonnet-4-5', capabilities=[BudgetDisclosure(limits=limits)]) diff --git a/tests/experimental/budget_disclosure/__init__.py b/tests/budget_disclosure/__init__.py similarity index 100% rename from tests/experimental/budget_disclosure/__init__.py rename to tests/budget_disclosure/__init__.py diff --git a/tests/experimental/budget_disclosure/test_budget_disclosure.py b/tests/budget_disclosure/test_budget_disclosure.py similarity index 99% rename from tests/experimental/budget_disclosure/test_budget_disclosure.py rename to tests/budget_disclosure/test_budget_disclosure.py index 3ec0eb7..9dc5595 100644 --- a/tests/experimental/budget_disclosure/test_budget_disclosure.py +++ b/tests/budget_disclosure/test_budget_disclosure.py @@ -18,7 +18,7 @@ from pydantic_ai.messages import ModelMessage, ModelResponse, TextPart, ToolCall from pydantic_ai.models.function import AgentInfo, FunctionModel from pydantic_ai.usage import RequestUsage, RunUsage, UsageLimits -from pydantic_ai_harness.experimental.budget_disclosure import BudgetDimension, BudgetDisclosure +from pydantic_ai_harness.budget_disclosure import BudgetDimension, BudgetDisclosure pytestmark = pytest.mark.anyio diff --git a/tests/test_docs_parity.py b/tests/test_docs_parity.py index 6287ca7..2a0f4ee 100644 --- a/tests/test_docs_parity.py +++ b/tests/test_docs_parity.py @@ -126,6 +126,7 @@ _CAPABILITY_PAGE_META = { 'dynamic-workflow.md': ('dynamic_workflow', 'Dynamic Workflow'), 'planning.md': ('planning', 'Planning'), 'runtime-authoring.md': ('runtime_authoring', 'Runtime Authoring'), + 'budget-disclosure.md': ('budget_disclosure', 'Budget Disclosure'), 'guardrails.md': ('guardrails', 'Input & Output Guardrails'), 'acp.md': ('experimental/acp', 'ACP (Agent Client Protocol)'), }