refactor(permission_policy): graduate to top-level layout, add docs-site page

Merge `main` and relocate from `experimental/permission_policy` to top-level
`pydantic_ai_harness/permission_policy/`, matching #347/#354. Drops the
experimental import warning, fixes import paths, moves tests. Adds
`docs/permission-policy.md` (nav.json, index.md, parity gate) and flips the README
Tool access control row to Docs. No behavior change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Douwe Maan
2026-07-15 19:51:59 +00:00
co-authored by Claude Opus 4.8
parent 5715be9d24
commit 38162829f3
15 changed files with 197 additions and 28 deletions
+1 -1
View File
@@ -172,7 +172,7 @@ We studied leading coding agents, agent frameworks, and Claw-style assistants to
| **Safety &&nbsp;guardrails** | **Input guardrails** | Validate user input before the agent run starts | :white_check_mark: [Docs](pydantic_ai_harness/guardrails/) | [pydantic-ai-shields](https://github.com/vstorm-co/pydantic-ai-shields) (vstorm&#8209;co) |
| | **Output guardrails** | Validate model output after the run completes | :white_check_mark: [Docs](pydantic_ai_harness/guardrails/) | [pydantic-ai-shields](https://github.com/vstorm-co/pydantic-ai-shields) (vstorm&#8209;co) |
| | **Cost/token budgets** | Enforce token and cost limits per run | :construction: [PR&nbsp;#182](https://github.com/pydantic/pydantic-ai-harness/pull/182) | [pydantic-ai-shields](https://github.com/vstorm-co/pydantic-ai-shields) (vstorm&#8209;co) |
| | **Tool access control** | Block tools or require approval before execution | :construction: [PR&nbsp;#182](https://github.com/pydantic/pydantic-ai-harness/pull/182) | [pydantic-ai-shields](https://github.com/vstorm-co/pydantic-ai-shields) (vstorm&#8209;co) |
| | **Tool access control** | Block tools or require approval before execution | :white_check_mark: [Docs](pydantic_ai_harness/permission_policy/) | [pydantic-ai-shields](https://github.com/vstorm-co/pydantic-ai-shields) (vstorm&#8209;co) |
| | **Async guardrails** | Run validation concurrently with model requests | :construction: [PR&nbsp;#182](https://github.com/pydantic/pydantic-ai-harness/pull/182) | [pydantic-ai-shields](https://github.com/vstorm-co/pydantic-ai-shields) (vstorm&#8209;co) |
| | **Secret masking** | Detect and redact secrets in agent I/O | :construction: [PR&nbsp;#172](https://github.com/pydantic/pydantic-ai-harness/pull/172) | [pydantic-ai-shields](https://github.com/vstorm-co/pydantic-ai-shields) (vstorm&#8209;co) |
| | **Approval workflows** | Require human approval for sensitive operations | :construction: [PR&nbsp;#173](https://github.com/pydantic/pydantic-ai-harness/pull/173) | [Pydantic&nbsp;AI](https://ai.pydantic.dev/deferred-tools/#human-in-the-loop-tool-approval) (built&#8209;in) |
+1
View File
@@ -125,6 +125,7 @@ Each capability is a self-contained battery you drop into an agent's `capabiliti
| [Planning](planning.md) | Breaks a complex task into a structured plan before execution and tracks progress against it. | -- |
| [Memory](memory.md) | Gives an agent a persistent, namespaced notebook with bounded prompt injection, on-demand search, and concurrency-safe stores. | -- |
| [Runtime Authoring](runtime-authoring.md) | Lets an agent author, validate, and load real capabilities at runtime. | -- |
| [Permission Policy](permission-policy.md) | An allow/ask/deny rule engine over tool calls, with shell-command analysis. | -- |
| [Guardrails](guardrails.md) | Validates user input before a run starts and model output after it completes -- block or redact, with structured results. | -- |
| [Managed Prompt](managed-prompt.md) | Backs an agent's instructions with a [Logfire-managed prompt](https://logfire.pydantic.dev/docs/reference/advanced/prompt-management/), so you can version, label, and roll out prompt changes from the Logfire UI without redeploying -- with a code default that keeps the agent working when no remote value is available. | `logfire` |
| [ACP](acp.md) *(experimental)* | Serves an agent to editors (Zed, etc.) over the [Agent Client Protocol](https://agentclientprotocol.com) -- streamed text, diff-rendered edits, and tool approval. | `acp` |
+1
View File
@@ -17,6 +17,7 @@
{ "label": "Planning", "slug": "planning" },
{ "label": "Memory", "slug": "memory" },
{ "label": "Runtime Authoring", "slug": "runtime-authoring" },
{ "label": "Permission Policy", "slug": "permission-policy" },
{ "label": "Guardrails", "slug": "guardrails" },
{ "label": "Managed Prompt", "slug": "managed-prompt" },
{ "label": "ACP", "slug": "acp" }
+176
View File
@@ -0,0 +1,176 @@
---
title: Permission Policy
description: An allow/ask/deny rule engine over tool calls, evaluated as each tool runs, with the shell-command safety mechanics every coding harness has to re-derive.
---
# Permission Policy
An allow / ask / deny rule engine over tool calls, evaluated as each tool runs -- with the
hard-won shell-command mechanics (compound-command splitting, a conservative-parse gate,
wrapper stripping, a read-only safelist, and a flag denylist) that every coding harness has
had to re-derive.
> [!NOTE]
> Import this capability from its submodule. It is not re-exported from `pydantic_ai_harness`:
>
> ```python
> from pydantic_ai_harness.permission_policy import PermissionPolicy
> ```
Permission Policy is a released, non-experimental capability. Pydantic AI Harness is still on 0.x releases, so the API may change between minor releases. See the repository [version policy](https://github.com/pydantic/pydantic-ai-harness#version-policy).
## The problem
Pydantic AI core gives you `ApprovalRequired` and deferred tools -- the *mechanism* to pause a
tool call for human approval. What it does not give you is a *policy*: a way to say "let the
model run `git status` freely, ask me before `git push`, and never let it `rm -rf`". Building
that safely is deceptively hard, because the dangerous surface is the shell:
- `git status && rm -rf /` -- a compound command where one segment is fine and one is fatal.
- `echo $(rm -rf /)` -- the danger hides inside a command substitution.
- `timeout 5 env LD_PRELOAD=evil.so ls` -- wrappers and env assignments smuggle execution
past a naive `ls`-looks-safe check.
- `git status-evil` -- a prefix match that isn't a word-boundary match.
## The solution
`PermissionPolicy` is a capability that evaluates every tool call in the `wrap_tool_execute`
hook and resolves it to one of three verdicts:
| Verdict | Effect |
|---|---|
| **allow** | the tool runs |
| **deny** | the tool does not run; a message goes back to the model explaining whether re-requesting with justification can help, or the action is never allowed |
| **ask** | the call is routed through Pydantic AI's deferred-approval machinery (`ApprovalRequired`) for a human -- or an `on_ask` handler -- to resolve |
### Rules: ordered, last-match-wins
Rules are an **ordered list; the last matching rule wins** (the model
[opencode](https://github.com/sst/opencode) uses -- specificity comes from position, not an
action-precedence lattice). Each rule is a `(tool-name glob, optional argument matcher,
verdict)`:
```python
from pydantic_ai import Agent
from pydantic_ai_harness.permission_policy import PermissionPolicy, Rule
policy = PermissionPolicy(
rules=[
Rule('deny', tool='run_command', command='git'), # deny all git ...
Rule('allow', tool='run_command', command='git status'), # ... except git status
Rule('deny', tool='delete_file'), # bare-name deny (any args)
Rule('ask', tool='write_file', args=lambda a: a['path'].startswith('/etc')),
],
)
agent = Agent('anthropic:claude-sonnet-4-6', capabilities=[policy])
```
The argument matcher is, in precedence order: `command` (a shell prefix, see below), then
`args` (an arbitrary predicate over the argument dict), then neither (matches any arguments).
### Shell commands: the safe mechanics
For shell-class tools (by default `run_command`, `bash`, `shell`, ...; configurable via
`shell_tools`/`command_arg`), the `command` matcher and the built-in safety analysis operate
on the command with:
- **Compound-command splitting.** The command is split at `&&`, `||`, `;`, `|` and **every
segment must independently pass**; one bad segment poisons the whole call.
- **A conservative-parse gate.** If the command contains anything we can't prove is a plain
list of commands -- subshells `(...)`, command substitution `$(...)`/backticks, variable
expansion `$VAR`, redirection `>`/`<`, background `&`, globs, brace expansion, env-assignment
prefixes -- the whole call **degrades to `ask`. It never guess-allows.** (This is
[Codex CLI](https://github.com/openai/codex)'s reject-if-complex posture.)
- **Word-boundary prefix rules.** `git status` matches `git status -sb` but never
`git status-evil` or `git push`.
- **Wrapper stripping.** `timeout`, `nice`, `env`, `xargs`, `sudo`, ... are peeled to the inner
command before matching (`timeout 5 git status` is `git status`). Wrapper shapes that could
smuggle execution -- `env FOO=...` (LD_PRELOAD), `sudo -u`, options on `timeout` -- degrade
to `ask` instead of peeling.
- **A read-only safelist** auto-allows known-safe commands (`ls`, `cat`, `grep`, `git status`,
`find` without `-exec`, `rg` without `--pre`, `sed -n {N}p`, ...), and **a flag denylist**
overrides allows for their dangerous variants (`git -C`/`-p`, `find -delete`/`-exec`,
`rg --pre`, `rm -rf`, arbitrary interpreters like `bash -c`).
The safelist and denylists are adopted, with attribution, from Codex CLI's command-safety
module; see `_safelist.py`.
### How the two channels combine
A call has two verdict sources: your **rules** (last-match-wins) and the **built-in
command-safety analysis**. They are merged **most-restrictive-wins** (`deny > ask > allow`).
This is what makes the flag denylist *override* a broad allow: `Rule('allow', command='git')`
plus the command `git -C /etc status` resolves to `ask`, because the built-in analysis flags
`-C` and the more restrictive verdict wins. To opt out of the built-in analysis entirely and
govern shell tools by your rules alone, set `analyze_shell_commands=False`.
## `ask`: the deferred-approval integration
An `ask` verdict raises `ApprovalRequired`, so it flows through Pydantic AI's standard
deferred-tool machinery. You resolve it in one of two ways:
1. **Externally** (real human-in-the-loop): add `DeferredToolRequests` to the agent's
`output_type` (or use a `HandleDeferredToolCalls` capability). The run pauses and returns
the pending approvals; you approve/deny and resume with `DeferredToolResults`.
2. **Inline**, via an `on_ask` handler on the policy:
```python
def on_ask(ctx, request):
if request.command and request.command.startswith('git push'):
return True # approve
return 'pushing is disabled in CI' # deny with this message (or return False)
policy = PermissionPolicy(rules=[Rule('ask', tool='run_command')], on_ask=on_ask)
```
`on_ask` returns `True` to approve, `False` to deny with a default message, or a string to
deny with that message. It may be sync or async. The policy only resolves the asks **it**
raised, so it composes cleanly with other deferred-tool handlers.
If an `ask` verdict fires and neither of these is configured, the run raises the standard
"`DeferredToolRequests` is not among output types" `UserError` -- the same contract as core's
`requires_approval=True` tools.
## Escalation protocol
When `add_escalation_note=True` (the default), guarded tools' descriptions gain a short note
telling the model it may restate a denied call once with a brief justification, and not to
retry commands reported as never allowed. A denied call's result message says the same. This
is Codex CLI's prompt-taught escalation, minus the persistence channel.
## Options
| Option | Default | Purpose |
|---|---|---|
| `rules` | `[]` | Ordered allow/ask/deny rules; last match wins. |
| `default_verdict` | `'ask'` | Verdict when no rule matches and command-safety has no opinion. Set `'allow'` for allowlist-by-exception, `'deny'` to block-by-default. |
| `shell_tools` | `run_command`, `bash`, ... | Tool names whose command argument is analyzed. |
| `command_arg` | `'command'` | The argument holding the shell command. |
| `analyze_shell_commands` | `True` | Run the built-in command-safety analysis for shell tools. |
| `deny_removes_tool` | `False` | Remove a tool denied *regardless of arguments* from the model's toolset entirely (Claude Code bare-name-deny). |
| `add_escalation_note` | `True` | Append the escalation note to guarded tools' descriptions. |
| `on_ask` | `None` | Resolve `ask` verdicts inline instead of surfacing them as `DeferredToolRequests`. |
## Deliberate scope (so these don't read as oversights)
- **`ask` ships the real deferred-approval integration**, not a deny-fallback -- it raises
`ApprovalRequired` and resolves through core's machinery, with `on_ask` as an inline
convenience.
- **Rule persistence is out of scope for v1.** The model cannot propose "always allow this
prefix" rules yet. When that lands, proposals must be vetted against
`BANNED_PREFIX_SUGGESTIONS` (exported here, adopted from Codex) -- the rule-persistence
channel needs its own denylist, distinct from the execution channel. opencode's
doom-loop-to-session-rule pattern is the follow-up shape.
- **Standard-library parsing only.** The analyzer uses a quote-aware scanner plus `shlex`, no
new dependency. Because the conservative gate degrades anything it cannot prove plain, a
full bash AST parser would only *narrow* what degrades to `ask` -- it would never turn a
wrong-allow into a right-allow.
- **An optional LLM guardian** (auto-approver for `ask` verdicts) is a natural extension of
`on_ask` and is left for a follow-up.
## Further reading
- [`pydantic_ai_harness.permission_policy` source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/permission_policy/)
- [Pydantic AI capabilities](/ai/core-concepts/capabilities/)
- [Pydantic AI deferred tools](/ai/tools-toolsets/deferred-tools/) -- the `ApprovalRequired` /
`DeferredToolRequests` machinery an `ask` verdict resolves through
@@ -1,24 +1,14 @@
# Permission policy
# Permission Policy
> [!WARNING]
> **Experimental.** This capability lives under `pydantic_ai_harness.experimental` and may
> change or be removed in any release, without a deprecation period. Import it from the
> experimental path -- there is no top-level export:
> [!NOTE]
> Import this capability from its submodule. It is not re-exported from `pydantic_ai_harness`:
>
> ```python
> from pydantic_ai_harness.experimental.permission_policy import PermissionPolicy, Rule
> ```
>
> Importing any experimental capability emits a `HarnessExperimentalWarning`. Silence **all**
> harness experimental warnings with a single filter (no per-capability lines needed):
>
> ```python
> import warnings
> from pydantic_ai_harness.experimental import HarnessExperimentalWarning
>
> warnings.filterwarnings('ignore', category=HarnessExperimentalWarning)
> from pydantic_ai_harness.permission_policy import PermissionPolicy
> ```
Permission Policy is a released, non-experimental capability. Pydantic AI Harness is still on 0.x releases, so the API may change between minor releases. See the repository [version policy](https://github.com/pydantic/pydantic-ai-harness#version-policy).
An allow / ask / deny rule engine over tool calls, evaluated as each tool runs -- with the
hard-won shell-command mechanics (compound-command splitting, a conservative-parse gate,
wrapper stripping, a read-only safelist, and a flag denylist) that every coding harness has
@@ -57,7 +47,7 @@ verdict)`:
```python
from pydantic_ai import Agent
from pydantic_ai_harness.experimental.permission_policy import PermissionPolicy, Rule
from pydantic_ai_harness.permission_policy import PermissionPolicy, Rule
policy = PermissionPolicy(
rules=[
@@ -174,4 +164,8 @@ is Codex CLI's prompt-taught escalation, minus the persistence channel.
- **An optional LLM guardian** (auto-approver for `ask` verdicts) is a natural extension of
`on_ask` and is left for a follow-up.
## Further reading
- [`pydantic_ai_harness.permission_policy` source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/permission_policy/)
[mining]: https://github.com/pydantic/pydantic-ai-notes/blob/main/features/harness-comparison/2026-07-08%20oss%20implementation%20mining%20-%20oh-my-pi%2C%20codex%2C%20opencode.md
@@ -4,15 +4,11 @@ See `README.md` in this package for the full model, the command-analysis mechani
red-team edge cases the test suite locks down.
"""
from pydantic_ai_harness.experimental._warn import warn_experimental
from ._capability import DEFAULT_SHELL_TOOLS, PermissionPolicy, PermissionRequest
from ._command import PreparedCommand, analyze_command, prepare_command
from ._rules import Decision, Rule, resolve
from ._safelist import BANNED_PREFIX_SUGGESTIONS
warn_experimental('permission_policy')
__all__ = [
'BANNED_PREFIX_SUGGESTIONS',
'DEFAULT_SHELL_TOOLS',
@@ -108,7 +108,7 @@ class PermissionPolicy(AbstractCapability[AgentDepsT]):
```python
from pydantic_ai import Agent
from pydantic_ai_harness.experimental.permission_policy import PermissionPolicy, Rule
from pydantic_ai_harness.permission_policy import PermissionPolicy, Rule
policy = PermissionPolicy[None](
rules=[
@@ -149,7 +149,7 @@ class PermissionPolicy(AbstractCapability[AgentDepsT]):
on_ask: OnAsk[AgentDepsT] | None = None
"""Optional handler that resolves `ask` verdicts inline. Receives the `RunContext` and a
[`PermissionRequest`][pydantic_ai_harness.experimental.permission_policy.PermissionRequest];
[`PermissionRequest`][pydantic_ai_harness.permission_policy.PermissionRequest];
return `True` to approve, `False` to deny with the default message, or a string to deny
with that message. When `None`, `ask` calls surface as `DeferredToolRequests` instead."""
@@ -13,7 +13,7 @@ import pytest
with warnings.catch_warnings():
warnings.simplefilter('ignore')
from pydantic_ai_harness.experimental.permission_policy._command import (
from pydantic_ai_harness.permission_policy._command import (
analyze_command,
command_matches_prefix,
prepare_command,
@@ -21,7 +21,7 @@ from pydantic_ai.tools import DeferredToolRequests, RunContext
with warnings.catch_warnings():
warnings.simplefilter('ignore')
from pydantic_ai_harness.experimental.permission_policy import (
from pydantic_ai_harness.permission_policy import (
PermissionPolicy,
PermissionRequest,
Rule,
@@ -6,8 +6,8 @@ import warnings
with warnings.catch_warnings():
warnings.simplefilter('ignore')
from pydantic_ai_harness.experimental.permission_policy._command import prepare_command
from pydantic_ai_harness.experimental.permission_policy._rules import (
from pydantic_ai_harness.permission_policy._command import prepare_command
from pydantic_ai_harness.permission_policy._rules import (
Rule,
bare_name_verdict,
last_matching_rule,
+1
View File
@@ -126,6 +126,7 @@ _CAPABILITY_PAGE_META = {
'dynamic-workflow.md': ('dynamic_workflow', 'Dynamic Workflow'),
'planning.md': ('planning', 'Planning'),
'runtime-authoring.md': ('runtime_authoring', 'Runtime Authoring'),
'permission-policy.md': ('permission_policy', 'Permission Policy'),
'guardrails.md': ('guardrails', 'Input & Output Guardrails'),
'acp.md': ('experimental/acp', 'ACP (Agent Client Protocol)'),
}