mirror of
https://github.com/pydantic/pydantic-ai-harness.git
synced 2026-07-21 02:45:34 +00:00
refactor(loop_detection): graduate to top-level layout, add docs-site page
Merge `main` and relocate from `experimental/loop_detection` to top-level `pydantic_ai_harness/loop_detection/`, matching #347/#354. Drops the experimental import warning, fixes import paths, moves tests. Adds `docs/loop-detection.md` (nav.json, index.md, parity gate) and flips the README Reliability row to Docs. No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
6b4bee0f7a
commit
447e1f9155
@@ -177,7 +177,7 @@ We studied leading coding agents, agent frameworks, and Claw-style assistants to
|
||||
| | **Secret masking** | Detect and redact secrets in agent I/O | :construction: [PR #172](https://github.com/pydantic/pydantic-ai-harness/pull/172) | [pydantic-ai-shields](https://github.com/vstorm-co/pydantic-ai-shields) (vstorm‑co) |
|
||||
| | **Approval workflows** | Require human approval for sensitive operations | :construction: [PR #173](https://github.com/pydantic/pydantic-ai-harness/pull/173) | [Pydantic AI](https://ai.pydantic.dev/deferred-tools/#human-in-the-loop-tool-approval) (built‑in) |
|
||||
| | **Tool budget** | Limit total tool calls or cost per run | :construction: [PR #168](https://github.com/pydantic/pydantic-ai-harness/pull/168) | |
|
||||
| **Reliability** | **Stuck loop detection** | Detect and break out of repetitive agent loops | :construction: [PR #186](https://github.com/pydantic/pydantic-ai-harness/pull/186) | |
|
||||
| **Reliability** | **Stuck loop detection** | Detect and break out of repetitive agent loops | :white_check_mark: [Docs](pydantic_ai_harness/loop_detection/) | |
|
||||
| | **Tool error recovery** | Retry failed tool calls with backoff and budget | :construction: [PR #171](https://github.com/pydantic/pydantic-ai-harness/pull/171) | |
|
||||
| | **Tool orphan repair** | Fix orphaned tool calls in conversation history | :construction: [PR #184](https://github.com/pydantic/pydantic-ai-harness/pull/184) | |
|
||||
| **Reasoning** | **Adaptive reasoning** | Adjust thinking effort based on task complexity | :construction: [PR #174](https://github.com/pydantic/pydantic-ai-harness/pull/174) | |
|
||||
|
||||
@@ -125,6 +125,7 @@ Each capability is a self-contained battery you drop into an agent's `capabiliti
|
||||
| [Planning](planning.md) | Breaks a complex task into a structured plan before execution and tracks progress against it. | -- |
|
||||
| [Memory](memory.md) | Gives an agent a persistent, namespaced notebook with bounded prompt injection, on-demand search, and concurrency-safe stores. | -- |
|
||||
| [Runtime Authoring](runtime-authoring.md) | Lets an agent author, validate, and load real capabilities at runtime. | -- |
|
||||
| [Loop Detection](loop-detection.md) | Detects and interrupts repeated-action loops before they burn budget. | -- |
|
||||
| [Guardrails](guardrails.md) | Validates user input before a run starts and model output after it completes -- block or redact, with structured results. | -- |
|
||||
| [Managed Prompt](managed-prompt.md) | Backs an agent's instructions with a [Logfire-managed prompt](https://logfire.pydantic.dev/docs/reference/advanced/prompt-management/), so you can version, label, and roll out prompt changes from the Logfire UI without redeploying -- with a code default that keeps the agent working when no remote value is available. | `logfire` |
|
||||
| [ACP](acp.md) *(experimental)* | Serves an agent to editors (Zed, etc.) over the [Agent Client Protocol](https://agentclientprotocol.com) -- streamed text, diff-rendered edits, and tool approval. | `acp` |
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
---
|
||||
title: Loop Detection
|
||||
description: Detect when an agent is stuck in a repeated-action loop and intervene.
|
||||
---
|
||||
|
||||
# Loop Detection
|
||||
|
||||
Detect when an agent is stuck in a repeated-action loop and intervene.
|
||||
|
||||
> [!NOTE]
|
||||
> Import this capability from its submodule. It is not re-exported from `pydantic_ai_harness`:
|
||||
>
|
||||
> ```python
|
||||
> from pydantic_ai_harness.loop_detection import LoopDetection
|
||||
> ```
|
||||
|
||||
Loop Detection is a released, non-experimental capability. Pydantic AI Harness is still on 0.x releases, so the API may change between minor releases. See the repository [version policy](https://github.com/pydantic/pydantic-ai-harness#version-policy).
|
||||
|
||||
## The problem
|
||||
|
||||
An autonomous agent that gets stuck does not stop -- it repeats the same action until it
|
||||
exhausts its step or token budget: re-reading a missing file, re-running a command that keeps
|
||||
failing, thrashing between two edits, or narrating what it will do without doing it. Five
|
||||
major coding harnesses (Gemini CLI, OpenHands, Roo, Crush, goose) each grew some form of loop
|
||||
detection to break out of this. `LoopDetection` is that guardrail as a composable capability.
|
||||
|
||||
## Minimal usage
|
||||
|
||||
```python
|
||||
from pydantic_ai import Agent
|
||||
from pydantic_ai_harness.loop_detection import LoopDetection
|
||||
|
||||
agent = Agent('anthropic:claude-sonnet-4-5', capabilities=[LoopDetection()])
|
||||
await agent.run('...') # a stuck loop nudges the model to change approach
|
||||
```
|
||||
|
||||
## Detection tiers
|
||||
|
||||
| Tier | Signal | Default |
|
||||
|------|--------|---------|
|
||||
| Exact repetition | The same `(tool_name, canonical_args)` fingerprint occurs `repeat_threshold` times within a sliding `window` of recent calls. Catches loops that interleave an occasional different call. | 5 within 10 |
|
||||
| Error cycle | The same tool call returns a byte-identical result on `error_cycle_threshold` consecutive executions. Coding-agent tools usually report failure as an ordinary error-shaped result rather than by raising, so an identical repeated result is the signature of a call that is not making progress. | 3 |
|
||||
| Alternation | Two distinct call fingerprints alternate A-B-A-B for `alternation_cycles` full cycles (a two-step thrash, e.g. edit then re-read). | 3 cycles |
|
||||
| Monologue | `monologue_threshold` consecutive model responses carry no tool call and near-identical text (normalized prefix match): the model is narrating instead of acting. | 3 |
|
||||
|
||||
Arguments are canonicalized (JSON with sorted keys) before fingerprinting, so `{"a": 1, "b": 2}`
|
||||
and `{"b": 2, "a": 1}` count as the same call. After a tier fires, its counters reset, so the
|
||||
same loop has to rebuild before it fires again rather than triggering on every later step.
|
||||
|
||||
## Action on detection: `on_loop`
|
||||
|
||||
- `'nudge'` (default): enqueue a harness-marked message that the model sees on its next
|
||||
request, e.g. *"You appear to be repeating the same action (`read_file` called 5 times with
|
||||
identical arguments). Change approach, or state plainly what is blocking you."* The nudge
|
||||
uses Pydantic AI's pending-message queue, so it is delivered as a real user turn on the next
|
||||
model request (and redirects the run into one more request if the agent would otherwise
|
||||
stop).
|
||||
- `'error'`: raise `LoopDetectedError` to abort the run. The structured `LoopDetected` is on
|
||||
`error.detected`.
|
||||
- a callable `(LoopDetected) -> None | Awaitable[None]`: called with the structured detection.
|
||||
It may be sync or async; raise from it to abort, or use it to log, record, or enqueue custom
|
||||
steering.
|
||||
|
||||
```python
|
||||
from pydantic_ai_harness.loop_detection import LoopDetected, LoopDetection
|
||||
|
||||
def on_loop(detected: LoopDetected) -> None:
|
||||
print(detected.tier, detected.tool_name, detected.count)
|
||||
|
||||
agent = Agent('anthropic:claude-sonnet-4-5', capabilities=[LoopDetection(on_loop=on_loop)])
|
||||
```
|
||||
|
||||
`LoopDetected` carries `tier`, `tool_name` (`None` for a monologue), `count`, `window`, the
|
||||
canonical `fingerprints` involved, and the rendered `message`.
|
||||
|
||||
## Options
|
||||
|
||||
- `repeat_threshold` (default `5`), `window` (default `10`): tier 1 sliding-window counting.
|
||||
- `error_cycle_threshold` (default `3`): tier 2a consecutive identical results.
|
||||
- `alternation_cycles` (default `3`): tier 2b full A-B cycles.
|
||||
- `monologue_threshold` (default `3`), `monologue_prefix_chars` (default `200`): tier 2c
|
||||
consecutive near-identical text-only responses and how many leading normalized characters
|
||||
two responses must share to count as near-identical.
|
||||
- `on_loop` (default `'nudge'`): see above.
|
||||
|
||||
## Composition
|
||||
|
||||
- The capability only implements `for_run`, `after_model_request`, and `after_tool_execute`;
|
||||
it adds no tools, instructions, or model settings, so it composes with any toolset,
|
||||
`ToolSearch` setup, or other capability without interference.
|
||||
- Per-run state (the sliding window and counters) is materialized in `for_run`, so one
|
||||
`LoopDetection` instance can be reused across many `Agent.run` calls -- concurrent runs never
|
||||
share counters.
|
||||
|
||||
## Observability
|
||||
|
||||
On detection the capability adds a `loop_detection.detected` event (with `loop.tier`,
|
||||
`loop.tool_name`, `loop.count`, `loop.window` attributes) to the active OpenTelemetry span,
|
||||
so a loop shows up in a Logfire/OTel trace whether or not `on_loop` aborts the run. When no
|
||||
span is recording the event is a no-op.
|
||||
|
||||
## Scope
|
||||
|
||||
- **Signal-based, not semantic.** It catches structural loops (repetition, cycles,
|
||||
monologues), not a model that is making slow but real progress. Keep the thresholds
|
||||
conservative so a legitimately repeated action is not flagged.
|
||||
- **No LLM-judge tier.** Detecting subtler "spinning" with a small model is intentionally out
|
||||
of scope until harness settles a small-model-roles convention.
|
||||
- **No pause/approval integration.** The actions are nudge, error, or callback; wiring a loop
|
||||
into an approval/pause flow is left to the caller's callback.
|
||||
|
||||
## Further reading
|
||||
|
||||
- [`pydantic_ai_harness.loop_detection` source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/loop_detection/)
|
||||
- [Pydantic AI capabilities](/ai/core-concepts/capabilities/)
|
||||
- [Pydantic AI hooks](/ai/core-concepts/hooks/) -- `after_tool_execute` and `after_model_request` (observe) and `for_run` (per-run state) are the surfaces used here
|
||||
@@ -17,6 +17,7 @@
|
||||
{ "label": "Planning", "slug": "planning" },
|
||||
{ "label": "Memory", "slug": "memory" },
|
||||
{ "label": "Runtime Authoring", "slug": "runtime-authoring" },
|
||||
{ "label": "Loop Detection", "slug": "loop-detection" },
|
||||
{ "label": "Guardrails", "slug": "guardrails" },
|
||||
{ "label": "Managed Prompt", "slug": "managed-prompt" },
|
||||
{ "label": "ACP", "slug": "acp" }
|
||||
|
||||
+12
-18
@@ -1,24 +1,14 @@
|
||||
# LoopDetection
|
||||
# Loop Detection
|
||||
|
||||
> [!WARNING]
|
||||
> **Experimental.** This capability lives under `pydantic_ai_harness.experimental` and may
|
||||
> change or be removed in any release, without a deprecation period. Import it from the
|
||||
> experimental path -- there is no top-level export:
|
||||
> [!NOTE]
|
||||
> Import this capability from its submodule. It is not re-exported from `pydantic_ai_harness`:
|
||||
>
|
||||
> ```python
|
||||
> from pydantic_ai_harness.experimental.loop_detection import LoopDetection
|
||||
> ```
|
||||
>
|
||||
> Importing any experimental capability emits a `HarnessExperimentalWarning`. Silence **all**
|
||||
> harness experimental warnings with a single filter (no per-capability lines needed):
|
||||
>
|
||||
> ```python
|
||||
> import warnings
|
||||
> from pydantic_ai_harness.experimental import HarnessExperimentalWarning
|
||||
>
|
||||
> warnings.filterwarnings('ignore', category=HarnessExperimentalWarning)
|
||||
> from pydantic_ai_harness.loop_detection import LoopDetection
|
||||
> ```
|
||||
|
||||
Loop Detection is a released, non-experimental capability. Pydantic AI Harness is still on 0.x releases, so the API may change between minor releases. See the repository [version policy](https://github.com/pydantic/pydantic-ai-harness#version-policy).
|
||||
|
||||
Detect when an agent is stuck in a repeated-action loop and intervene.
|
||||
|
||||
## The problem
|
||||
@@ -33,7 +23,7 @@ detection to break out of this. `LoopDetection` is that guardrail as a composabl
|
||||
|
||||
```python
|
||||
from pydantic_ai import Agent
|
||||
from pydantic_ai_harness.experimental.loop_detection import LoopDetection
|
||||
from pydantic_ai_harness.loop_detection import LoopDetection
|
||||
|
||||
agent = Agent('anthropic:claude-sonnet-4-5', capabilities=[LoopDetection()])
|
||||
await agent.run('...') # a stuck loop nudges the model to change approach
|
||||
@@ -67,7 +57,7 @@ same loop has to rebuild before it fires again rather than triggering on every l
|
||||
steering.
|
||||
|
||||
```python
|
||||
from pydantic_ai_harness.experimental.loop_detection import LoopDetected, LoopDetection
|
||||
from pydantic_ai_harness.loop_detection import LoopDetected, LoopDetection
|
||||
|
||||
def on_loop(detected: LoopDetected) -> None:
|
||||
print(detected.tier, detected.tool_name, detected.count)
|
||||
@@ -113,3 +103,7 @@ span is recording the event is a no-op.
|
||||
of scope until harness settles a small-model-roles convention.
|
||||
- **No pause/approval integration.** The actions are nudge, error, or callback; wiring a loop
|
||||
into an approval/pause flow is left to the caller's callback.
|
||||
|
||||
## Further reading
|
||||
|
||||
- [`pydantic_ai_harness.loop_detection` source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/loop_detection/)
|
||||
+1
-4
@@ -1,7 +1,6 @@
|
||||
"""Detect and interrupt repeated-action loops (private, not re-exported at top level)."""
|
||||
|
||||
from pydantic_ai_harness.experimental._warn import warn_experimental
|
||||
from pydantic_ai_harness.experimental.loop_detection._capability import (
|
||||
from pydantic_ai_harness.loop_detection._capability import (
|
||||
LoopDetected,
|
||||
LoopDetectedError,
|
||||
LoopDetection,
|
||||
@@ -9,8 +8,6 @@ from pydantic_ai_harness.experimental.loop_detection._capability import (
|
||||
OnLoop,
|
||||
)
|
||||
|
||||
warn_experimental('loop_detection')
|
||||
|
||||
__all__ = [
|
||||
'LoopDetected',
|
||||
'LoopDetectedError',
|
||||
+1
-1
@@ -186,7 +186,7 @@ class LoopDetection(AbstractCapability[AgentDepsT]):
|
||||
|
||||
```python
|
||||
from pydantic_ai import Agent
|
||||
from pydantic_ai_harness.experimental.loop_detection import LoopDetection
|
||||
from pydantic_ai_harness.loop_detection import LoopDetection
|
||||
|
||||
agent = Agent('anthropic:claude-sonnet-4-5', capabilities=[LoopDetection()])
|
||||
await agent.run('...') # a stuck loop nudges the model to change approach
|
||||
+2
-2
@@ -1,4 +1,4 @@
|
||||
"""Tests for the experimental `LoopDetection` capability.
|
||||
"""Tests for the `LoopDetection` capability.
|
||||
|
||||
Behavior is driven through `Agent(..., capabilities=[LoopDetection()])` with a `FunctionModel`
|
||||
that emits a scripted sequence of tool calls or text, exercising each detection tier the way a
|
||||
@@ -18,7 +18,7 @@ from pydantic_ai import Agent
|
||||
from pydantic_ai.messages import ModelMessage, ModelResponse, TextPart, ToolCallPart, UserPromptPart
|
||||
from pydantic_ai.models.function import AgentInfo, FunctionModel
|
||||
|
||||
from pydantic_ai_harness.experimental.loop_detection import (
|
||||
from pydantic_ai_harness.loop_detection import (
|
||||
LoopDetected,
|
||||
LoopDetectedError,
|
||||
LoopDetection,
|
||||
@@ -126,6 +126,7 @@ _CAPABILITY_PAGE_META = {
|
||||
'dynamic-workflow.md': ('dynamic_workflow', 'Dynamic Workflow'),
|
||||
'planning.md': ('planning', 'Planning'),
|
||||
'runtime-authoring.md': ('runtime_authoring', 'Runtime Authoring'),
|
||||
'loop-detection.md': ('loop_detection', 'Loop Detection'),
|
||||
'guardrails.md': ('guardrails', 'Input & Output Guardrails'),
|
||||
'acp.md': ('experimental/acp', 'ACP (Agent Client Protocol)'),
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user