refactor(loop_detection): graduate to top-level layout, add docs-site page

Merge `main` and relocate from `experimental/loop_detection` to top-level
`pydantic_ai_harness/loop_detection/`, matching #347/#354. Drops the experimental
import warning, fixes import paths, moves tests. Adds `docs/loop-detection.md`
(nav.json, index.md, parity gate) and flips the README Reliability row to Docs.
No behavior change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Douwe Maan
2026-07-15 19:50:28 +00:00
co-authored by Claude Opus 4.8
parent 6b4bee0f7a
commit 447e1f9155
10 changed files with 136 additions and 26 deletions
+1 -1
View File
@@ -177,7 +177,7 @@ We studied leading coding agents, agent frameworks, and Claw-style assistants to
| | **Secret masking** | Detect and redact secrets in agent I/O | :construction: [PR&nbsp;#172](https://github.com/pydantic/pydantic-ai-harness/pull/172) | [pydantic-ai-shields](https://github.com/vstorm-co/pydantic-ai-shields) (vstorm&#8209;co) |
| | **Approval workflows** | Require human approval for sensitive operations | :construction: [PR&nbsp;#173](https://github.com/pydantic/pydantic-ai-harness/pull/173) | [Pydantic&nbsp;AI](https://ai.pydantic.dev/deferred-tools/#human-in-the-loop-tool-approval) (built&#8209;in) |
| | **Tool budget** | Limit total tool calls or cost per run | :construction: [PR&nbsp;#168](https://github.com/pydantic/pydantic-ai-harness/pull/168) | |
| **Reliability** | **Stuck loop detection** | Detect and break out of repetitive agent loops | :construction: [PR&nbsp;#186](https://github.com/pydantic/pydantic-ai-harness/pull/186) | |
| **Reliability** | **Stuck loop detection** | Detect and break out of repetitive agent loops | :white_check_mark: [Docs](pydantic_ai_harness/loop_detection/) | |
| | **Tool error recovery** | Retry failed tool calls with backoff and budget | :construction: [PR&nbsp;#171](https://github.com/pydantic/pydantic-ai-harness/pull/171) | |
| | **Tool orphan repair** | Fix orphaned tool calls in conversation history | :construction: [PR&nbsp;#184](https://github.com/pydantic/pydantic-ai-harness/pull/184) | |
| **Reasoning** | **Adaptive reasoning** | Adjust thinking effort based on task complexity | :construction: [PR&nbsp;#174](https://github.com/pydantic/pydantic-ai-harness/pull/174) | |
+1
View File
@@ -125,6 +125,7 @@ Each capability is a self-contained battery you drop into an agent's `capabiliti
| [Planning](planning.md) | Breaks a complex task into a structured plan before execution and tracks progress against it. | -- |
| [Memory](memory.md) | Gives an agent a persistent, namespaced notebook with bounded prompt injection, on-demand search, and concurrency-safe stores. | -- |
| [Runtime Authoring](runtime-authoring.md) | Lets an agent author, validate, and load real capabilities at runtime. | -- |
| [Loop Detection](loop-detection.md) | Detects and interrupts repeated-action loops before they burn budget. | -- |
| [Guardrails](guardrails.md) | Validates user input before a run starts and model output after it completes -- block or redact, with structured results. | -- |
| [Managed Prompt](managed-prompt.md) | Backs an agent's instructions with a [Logfire-managed prompt](https://logfire.pydantic.dev/docs/reference/advanced/prompt-management/), so you can version, label, and roll out prompt changes from the Logfire UI without redeploying -- with a code default that keeps the agent working when no remote value is available. | `logfire` |
| [ACP](acp.md) *(experimental)* | Serves an agent to editors (Zed, etc.) over the [Agent Client Protocol](https://agentclientprotocol.com) -- streamed text, diff-rendered edits, and tool approval. | `acp` |
+116
View File
@@ -0,0 +1,116 @@
---
title: Loop Detection
description: Detect when an agent is stuck in a repeated-action loop and intervene.
---
# Loop Detection
Detect when an agent is stuck in a repeated-action loop and intervene.
> [!NOTE]
> Import this capability from its submodule. It is not re-exported from `pydantic_ai_harness`:
>
> ```python
> from pydantic_ai_harness.loop_detection import LoopDetection
> ```
Loop Detection is a released, non-experimental capability. Pydantic AI Harness is still on 0.x releases, so the API may change between minor releases. See the repository [version policy](https://github.com/pydantic/pydantic-ai-harness#version-policy).
## The problem
An autonomous agent that gets stuck does not stop -- it repeats the same action until it
exhausts its step or token budget: re-reading a missing file, re-running a command that keeps
failing, thrashing between two edits, or narrating what it will do without doing it. Five
major coding harnesses (Gemini CLI, OpenHands, Roo, Crush, goose) each grew some form of loop
detection to break out of this. `LoopDetection` is that guardrail as a composable capability.
## Minimal usage
```python
from pydantic_ai import Agent
from pydantic_ai_harness.loop_detection import LoopDetection
agent = Agent('anthropic:claude-sonnet-4-5', capabilities=[LoopDetection()])
await agent.run('...') # a stuck loop nudges the model to change approach
```
## Detection tiers
| Tier | Signal | Default |
|------|--------|---------|
| Exact repetition | The same `(tool_name, canonical_args)` fingerprint occurs `repeat_threshold` times within a sliding `window` of recent calls. Catches loops that interleave an occasional different call. | 5 within 10 |
| Error cycle | The same tool call returns a byte-identical result on `error_cycle_threshold` consecutive executions. Coding-agent tools usually report failure as an ordinary error-shaped result rather than by raising, so an identical repeated result is the signature of a call that is not making progress. | 3 |
| Alternation | Two distinct call fingerprints alternate A-B-A-B for `alternation_cycles` full cycles (a two-step thrash, e.g. edit then re-read). | 3 cycles |
| Monologue | `monologue_threshold` consecutive model responses carry no tool call and near-identical text (normalized prefix match): the model is narrating instead of acting. | 3 |
Arguments are canonicalized (JSON with sorted keys) before fingerprinting, so `{"a": 1, "b": 2}`
and `{"b": 2, "a": 1}` count as the same call. After a tier fires, its counters reset, so the
same loop has to rebuild before it fires again rather than triggering on every later step.
## Action on detection: `on_loop`
- `'nudge'` (default): enqueue a harness-marked message that the model sees on its next
request, e.g. *"You appear to be repeating the same action (`read_file` called 5 times with
identical arguments). Change approach, or state plainly what is blocking you."* The nudge
uses Pydantic AI's pending-message queue, so it is delivered as a real user turn on the next
model request (and redirects the run into one more request if the agent would otherwise
stop).
- `'error'`: raise `LoopDetectedError` to abort the run. The structured `LoopDetected` is on
`error.detected`.
- a callable `(LoopDetected) -> None | Awaitable[None]`: called with the structured detection.
It may be sync or async; raise from it to abort, or use it to log, record, or enqueue custom
steering.
```python
from pydantic_ai_harness.loop_detection import LoopDetected, LoopDetection
def on_loop(detected: LoopDetected) -> None:
print(detected.tier, detected.tool_name, detected.count)
agent = Agent('anthropic:claude-sonnet-4-5', capabilities=[LoopDetection(on_loop=on_loop)])
```
`LoopDetected` carries `tier`, `tool_name` (`None` for a monologue), `count`, `window`, the
canonical `fingerprints` involved, and the rendered `message`.
## Options
- `repeat_threshold` (default `5`), `window` (default `10`): tier 1 sliding-window counting.
- `error_cycle_threshold` (default `3`): tier 2a consecutive identical results.
- `alternation_cycles` (default `3`): tier 2b full A-B cycles.
- `monologue_threshold` (default `3`), `monologue_prefix_chars` (default `200`): tier 2c
consecutive near-identical text-only responses and how many leading normalized characters
two responses must share to count as near-identical.
- `on_loop` (default `'nudge'`): see above.
## Composition
- The capability only implements `for_run`, `after_model_request`, and `after_tool_execute`;
it adds no tools, instructions, or model settings, so it composes with any toolset,
`ToolSearch` setup, or other capability without interference.
- Per-run state (the sliding window and counters) is materialized in `for_run`, so one
`LoopDetection` instance can be reused across many `Agent.run` calls -- concurrent runs never
share counters.
## Observability
On detection the capability adds a `loop_detection.detected` event (with `loop.tier`,
`loop.tool_name`, `loop.count`, `loop.window` attributes) to the active OpenTelemetry span,
so a loop shows up in a Logfire/OTel trace whether or not `on_loop` aborts the run. When no
span is recording the event is a no-op.
## Scope
- **Signal-based, not semantic.** It catches structural loops (repetition, cycles,
monologues), not a model that is making slow but real progress. Keep the thresholds
conservative so a legitimately repeated action is not flagged.
- **No LLM-judge tier.** Detecting subtler "spinning" with a small model is intentionally out
of scope until harness settles a small-model-roles convention.
- **No pause/approval integration.** The actions are nudge, error, or callback; wiring a loop
into an approval/pause flow is left to the caller's callback.
## Further reading
- [`pydantic_ai_harness.loop_detection` source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/loop_detection/)
- [Pydantic AI capabilities](/ai/core-concepts/capabilities/)
- [Pydantic AI hooks](/ai/core-concepts/hooks/) -- `after_tool_execute` and `after_model_request` (observe) and `for_run` (per-run state) are the surfaces used here
+1
View File
@@ -17,6 +17,7 @@
{ "label": "Planning", "slug": "planning" },
{ "label": "Memory", "slug": "memory" },
{ "label": "Runtime Authoring", "slug": "runtime-authoring" },
{ "label": "Loop Detection", "slug": "loop-detection" },
{ "label": "Guardrails", "slug": "guardrails" },
{ "label": "Managed Prompt", "slug": "managed-prompt" },
{ "label": "ACP", "slug": "acp" }
@@ -1,24 +1,14 @@
# LoopDetection
# Loop Detection
> [!WARNING]
> **Experimental.** This capability lives under `pydantic_ai_harness.experimental` and may
> change or be removed in any release, without a deprecation period. Import it from the
> experimental path -- there is no top-level export:
> [!NOTE]
> Import this capability from its submodule. It is not re-exported from `pydantic_ai_harness`:
>
> ```python
> from pydantic_ai_harness.experimental.loop_detection import LoopDetection
> ```
>
> Importing any experimental capability emits a `HarnessExperimentalWarning`. Silence **all**
> harness experimental warnings with a single filter (no per-capability lines needed):
>
> ```python
> import warnings
> from pydantic_ai_harness.experimental import HarnessExperimentalWarning
>
> warnings.filterwarnings('ignore', category=HarnessExperimentalWarning)
> from pydantic_ai_harness.loop_detection import LoopDetection
> ```
Loop Detection is a released, non-experimental capability. Pydantic AI Harness is still on 0.x releases, so the API may change between minor releases. See the repository [version policy](https://github.com/pydantic/pydantic-ai-harness#version-policy).
Detect when an agent is stuck in a repeated-action loop and intervene.
## The problem
@@ -33,7 +23,7 @@ detection to break out of this. `LoopDetection` is that guardrail as a composabl
```python
from pydantic_ai import Agent
from pydantic_ai_harness.experimental.loop_detection import LoopDetection
from pydantic_ai_harness.loop_detection import LoopDetection
agent = Agent('anthropic:claude-sonnet-4-5', capabilities=[LoopDetection()])
await agent.run('...') # a stuck loop nudges the model to change approach
@@ -67,7 +57,7 @@ same loop has to rebuild before it fires again rather than triggering on every l
steering.
```python
from pydantic_ai_harness.experimental.loop_detection import LoopDetected, LoopDetection
from pydantic_ai_harness.loop_detection import LoopDetected, LoopDetection
def on_loop(detected: LoopDetected) -> None:
print(detected.tier, detected.tool_name, detected.count)
@@ -113,3 +103,7 @@ span is recording the event is a no-op.
of scope until harness settles a small-model-roles convention.
- **No pause/approval integration.** The actions are nudge, error, or callback; wiring a loop
into an approval/pause flow is left to the caller's callback.
## Further reading
- [`pydantic_ai_harness.loop_detection` source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/loop_detection/)
@@ -1,7 +1,6 @@
"""Detect and interrupt repeated-action loops (private, not re-exported at top level)."""
from pydantic_ai_harness.experimental._warn import warn_experimental
from pydantic_ai_harness.experimental.loop_detection._capability import (
from pydantic_ai_harness.loop_detection._capability import (
LoopDetected,
LoopDetectedError,
LoopDetection,
@@ -9,8 +8,6 @@ from pydantic_ai_harness.experimental.loop_detection._capability import (
OnLoop,
)
warn_experimental('loop_detection')
__all__ = [
'LoopDetected',
'LoopDetectedError',
@@ -186,7 +186,7 @@ class LoopDetection(AbstractCapability[AgentDepsT]):
```python
from pydantic_ai import Agent
from pydantic_ai_harness.experimental.loop_detection import LoopDetection
from pydantic_ai_harness.loop_detection import LoopDetection
agent = Agent('anthropic:claude-sonnet-4-5', capabilities=[LoopDetection()])
await agent.run('...') # a stuck loop nudges the model to change approach
@@ -1,4 +1,4 @@
"""Tests for the experimental `LoopDetection` capability.
"""Tests for the `LoopDetection` capability.
Behavior is driven through `Agent(..., capabilities=[LoopDetection()])` with a `FunctionModel`
that emits a scripted sequence of tool calls or text, exercising each detection tier the way a
@@ -18,7 +18,7 @@ from pydantic_ai import Agent
from pydantic_ai.messages import ModelMessage, ModelResponse, TextPart, ToolCallPart, UserPromptPart
from pydantic_ai.models.function import AgentInfo, FunctionModel
from pydantic_ai_harness.experimental.loop_detection import (
from pydantic_ai_harness.loop_detection import (
LoopDetected,
LoopDetectedError,
LoopDetection,
+1
View File
@@ -126,6 +126,7 @@ _CAPABILITY_PAGE_META = {
'dynamic-workflow.md': ('dynamic_workflow', 'Dynamic Workflow'),
'planning.md': ('planning', 'Planning'),
'runtime-authoring.md': ('runtime_authoring', 'Runtime Authoring'),
'loop-detection.md': ('loop_detection', 'Loop Detection'),
'guardrails.md': ('guardrails', 'Input & Output Guardrails'),
'acp.md': ('experimental/acp', 'ACP (Agent Client Protocol)'),
}