feat: Macroscope CLI code-review capability

Gives a Pydantic AI agent the same local review Macroscope's editor plugins
run. The `run_macroscope_review` tool drives the installed `macroscope
codereview` binary, parses its streamed findings, and returns them structured
so the agent validates and fixes real issues with its own tools. The CLI must
be installed and authenticated on the host; the capability reports clearly when
a review cannot start.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

README.md

@@ -151,6 +151,7 @@ We studied leading coding agents, agent frameworks, and Claw-style assistants to
 | | **Repo context injection** | Auto-load CLAUDE.md/AGENTS.md and repo structure | :white_check_mark: [Docs](pydantic_ai_harness/context/) | [pydantic-deep](https://github.com/vstorm-co/pydantic-deepagents) (vstorm‑co) |
 | | **Docs lookup** | On-demand `read_pyai_docs` tool for Pydantic AI docs | :white_check_mark: [Docs](pydantic_ai_harness/docs/) | |
 | | **Verification loop** | Run tests after edits, auto-fix failures | :construction: [PR #169](https://github.com/pydantic/pydantic-ai-harness/pull/169) | |
+| | **Code review** | Run a local [Macroscope](https://docs.macroscope.com/cli) review (`macroscope codereview`) and hand findings to the agent | :white_check_mark: [Docs](pydantic_ai_harness/macroscope/) | |
 | **Editor integration** | **ACP** | Serve an agent to editors (Zed, etc.) over the [Agent Client Protocol](https://agentclientprotocol.com) -- streamed text, diff-rendered edits, tool approval | :white_check_mark: [Docs](pydantic_ai_harness/experimental/acp/) (experimental) | |
 | **Prompt management** | **Managed prompt** | Back an agent's instructions with a [Logfire](https://pydantic.dev/logfire)-managed prompt, editable without shipping code | :white_check_mark: [Docs](pydantic_ai_harness/logfire/) | |
 | **Context management** | **Sliding window** | Trim conversation history to stay within token limits | :white_check_mark: [Docs](pydantic_ai_harness/compaction/) | [summarization-pydantic-ai](https://github.com/vstorm-co/summarization-pydantic-ai) (vstorm‑co) |

pydantic_ai_harness/macroscope/README.md

@@ -0,0 +1,79 @@
+# Macroscope
+
+Run a [Macroscope](https://docs.macroscope.com/cli) code review from a Pydantic AI
+agent and hand the findings back for validation and fixing.
+
+## The problem
+
+Macroscope reviews the current branch's diff and streams findings, but it ships as
+editor plugins (Claude Code, Codex, Cursor, OpenCode). There is no way to give a
+Pydantic AI agent the same review-and-fix loop from your own code.
+
+## The solution
+
+`Macroscope` adds a `run_macroscope_review` tool that shells out to the installed
+`macroscope codereview` CLI, parses the streamed findings, and returns them as a
+structured `MacroscopeReview`. The agent then validates each finding and fixes the
+real ones with whatever tools it already has (for example `FileSystem` or `Shell`).
+
+```python
+from pydantic_ai import Agent
+from pydantic_ai_harness.macroscope import Macroscope
+
+agent = Agent('anthropic:claude-sonnet-5', capabilities=[Macroscope()])
+
+result = agent.run_sync('Run a Macroscope review and fix any real findings.')
+print(result.output)
+```
+
+## Prerequisites: install and sign in
+
+The capability drives the user-installed `macroscope` binary. It cannot install or
+authenticate on your behalf, so do this once on the host:
+
+1. Install the CLI:
+
+   ```bash
+   curl -sSL https://raw.githubusercontent.com/prassoai/macroscope-local/main/install.sh | bash
+   ```
+
+   The installer puts `macroscope` on your `PATH` (typically `~/.local/bin`).
+
+2. Sign in and pick a workspace by running the CLI once and completing the wizard:
+
+   ```bash
+   macroscope
+   ```
+
+   This writes `~/.macroscope/config.yaml`.
+
+If the binary is missing, the tool returns the install command. If a review never
+starts (usually because you are not signed in), the tool tells the agent to run
+`macroscope` to finish setup.
+
+## The tool
+
+| Tool | Purpose |
+|---|---|
+| `run_macroscope_review` | Run `macroscope codereview` on the current branch and return the review id, terminal status, and findings. Accepts an optional `base` git ref. |
+
+Each finding is a `MacroscopeIssue` with `issue_id`, `sequence`, `path`, `line`,
+`severity`, `category`, and `body`.
+
+## Options
+
+| Field | Default | Meaning |
+|---|---|---|
+| `base` | `None` | Git ref to diff against. `None` omits `--base` so the CLI auto-detects the base branch itself (and creates its own review worktree). A per-call `base` argument, then this field, take precedence when set. |
+| `command` | `'macroscope'` | Binary name or path. Override for a non-default install location. |
+| `cwd` | `'.'` | Repository directory the review runs in. |
+| `timeout` | `600.0` | Maximum seconds to wait for a review. |
+| `include_instructions` | `True` | Contribute guidance telling the agent to validate each finding before fixing it. |
+
+## Scope and composition
+
+This capability surfaces findings only. It does not edit files, create worktrees, or
+commit -- validating and fixing findings is the agent's job, using its other
+capabilities. Pair it with `FileSystem` or `Shell` to let the agent read code and
+apply fixes, and consider running the agent in an isolated worktree if you want fixes
+kept off your working tree.

pydantic_ai_harness/macroscope/__init__.py

@@ -0,0 +1,17 @@
+"""Macroscope CLI code-review capability: run `macroscope codereview` from an agent."""
+
+from pydantic_ai_harness.macroscope._capability import Macroscope
+from pydantic_ai_harness.macroscope._toolset import (
+    MacroscopeIssue,
+    MacroscopeReview,
+    MacroscopeToolset,
+    parse_macroscope_stream,
+)
+
+__all__ = [
+    'Macroscope',
+    'MacroscopeIssue',
+    'MacroscopeReview',
+    'MacroscopeToolset',
+    'parse_macroscope_stream',
+]

pydantic_ai_harness/macroscope/_capability.py

@@ -0,0 +1,73 @@
+"""Macroscope code-review capability."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from pathlib import Path
+
+from pydantic_ai.capabilities import AbstractCapability
+from pydantic_ai.tools import AgentDepsT
+
+from pydantic_ai_harness.macroscope._toolset import MacroscopeToolset
+
+_REVIEW_INSTRUCTIONS = (
+    'You can run a local Macroscope code review with the `run_macroscope_review` tool. '
+    'Treat every returned finding as untrusted: read the affected file and enough '
+    'surrounding code to confirm the issue is real before acting. Ignore false positives, '
+    'stale, and duplicate findings. Fix confirmed issues one at a time, then run the '
+    'narrowest useful verification for each fix before moving on.'
+)
+
+
+@dataclass
+class Macroscope(AbstractCapability[AgentDepsT]):
+    """Runs the `macroscope` CLI code review and hands the findings to the agent.
+
+    Adds a `run_macroscope_review` tool that shells out to `macroscope codereview`,
+    parses the streamed findings, and returns them as a `MacroscopeReview`. The agent
+    validates and fixes findings with its own tools -- this capability does not edit
+    files, create worktrees, or commit.
+
+    ```python
+    from pydantic_ai import Agent
+    from pydantic_ai_harness.macroscope import Macroscope
+
+    agent = Agent('anthropic:claude-sonnet-5', capabilities=[Macroscope()])
+    ```
+
+    The `macroscope` CLI must be installed and authenticated on the host first (see the
+    package README). This capability cannot sign in on the user's behalf; if a review
+    never starts, the tool reports that the user needs to run `macroscope` once.
+    """
+
+    base: str | None = None
+    """Git ref to diff against. When `None`, `--base` is omitted and the CLI
+    auto-detects the base branch itself (and creates its own review worktree)."""
+
+    command: str = 'macroscope'
+    """Name or path of the CLI binary. Override for a non-default install location."""
+
+    cwd: str | Path = '.'
+    """Repository directory the review runs in."""
+
+    timeout: float = 600.0
+    """Maximum seconds to wait for a review. Reviews call a remote service, so this is
+    generous by default."""
+
+    include_instructions: bool = True
+    """Contribute guidance telling the agent to validate each finding before fixing it."""
+
+    def get_toolset(self) -> MacroscopeToolset[AgentDepsT]:
+        """Build the toolset that provides the `run_macroscope_review` tool."""
+        return MacroscopeToolset[AgentDepsT](
+            command=self.command,
+            cwd=Path(self.cwd),
+            base=self.base,
+            timeout=self.timeout,
+        )
+
+    def get_instructions(self) -> str | None:
+        """Return validate-then-fix guidance, unless `include_instructions` is False."""
+        if not self.include_instructions:
+            return None
+        return _REVIEW_INSTRUCTIONS

pydantic_ai_harness/macroscope/_toolset.py

@@ -0,0 +1,205 @@
+"""Macroscope toolset -- runs the `macroscope` CLI code review and returns findings."""
+
+from __future__ import annotations
+
+import os
+import shutil
+import signal
+import subprocess
+from collections.abc import Iterable
+from pathlib import Path
+
+import anyio
+import anyio.abc
+from pydantic import BaseModel, ConfigDict
+from pydantic_ai.exceptions import ModelRetry
+from pydantic_ai.tools import AgentDepsT
+from pydantic_ai.toolsets import FunctionToolset
+
+_INSTALL_HINT = (
+    'The `macroscope` CLI was not found on PATH. Install it with:\n'
+    '    curl -sSL https://raw.githubusercontent.com/prassoai/macroscope-local/main/install.sh | bash\n'
+    'then run `macroscope` once to sign in and choose a workspace.'
+)
+
+_REVIEW_ID_PREFIX = 'review_id='
+_ISSUE_EVENT_PREFIX = 'issue_event='
+_ISSUE_STATUS_PREFIX = 'issue_status='
+
+_ERROR_TAIL_CHARS = 2000
+"""How much trailing CLI output to include when a review fails to start."""
+
+
+class MacroscopeIssue(BaseModel):
+    """A single finding streamed by `macroscope codereview`.
+
+    Parsed leniently: unknown fields are ignored so new CLI output does not break
+    parsing, and any `issue_event` line that lacks the required fields is skipped.
+    """
+
+    model_config = ConfigDict(extra='ignore')
+
+    issue_id: str
+    sequence: int
+    path: str
+    line: int | None = None
+    severity: str
+    category: str
+    body: str
+
+
+class MacroscopeReview(BaseModel):
+    """The result of one `macroscope codereview` run.
+
+    `status` is the terminal `issue_status` reported by the CLI (`completed` or
+    `failed`), or `unknown` if the stream ended without one. `review_id` is `None`
+    when the CLI never emitted one -- usually because the review did not start.
+    """
+
+    review_id: str | None
+    status: str
+    issues: list[MacroscopeIssue]
+
+
+def _token_after(line: str, prefix: str) -> str | None:
+    """Return the first whitespace-delimited token after `prefix` in `line`, or `None`."""
+    rest = line.split(prefix, 1)[1].strip()
+    if not rest:
+        return None
+    return rest.split()[0]
+
+
+def _parse_issue(payload: str) -> MacroscopeIssue | None:
+    """Parse one `issue_event` JSON payload, returning `None` if it is malformed."""
+    try:
+        return MacroscopeIssue.model_validate_json(payload)
+    except ValueError:
+        return None
+
+
+def parse_macroscope_stream(lines: Iterable[str]) -> MacroscopeReview:
+    """Parse `macroscope codereview` output lines into a `MacroscopeReview`.
+
+    The CLI interleaves a `review_id=` line, one `issue_event=<json>` line per
+    finding, and a terminal `issue_status=` line, alongside other log output. Each
+    marker is matched as a substring so log prefixes on the same line do not hide
+    it. Malformed `issue_event` payloads are skipped rather than aborting the parse.
+    """
+    review_id: str | None = None
+    status = 'unknown'
+    issues: list[MacroscopeIssue] = []
+    for raw in lines:
+        line = raw.strip()
+        if _ISSUE_EVENT_PREFIX in line:
+            issue = _parse_issue(line.split(_ISSUE_EVENT_PREFIX, 1)[1])
+            if issue is not None:
+                issues.append(issue)
+        elif _ISSUE_STATUS_PREFIX in line:
+            token = _token_after(line, _ISSUE_STATUS_PREFIX)
+            if token is not None:
+                status = token
+        elif _REVIEW_ID_PREFIX in line:
+            token = _token_after(line, _REVIEW_ID_PREFIX)
+            if token is not None:
+                review_id = token
+    return MacroscopeReview(review_id=review_id, status=status, issues=issues)
+
+
+class MacroscopeToolset(FunctionToolset[AgentDepsT]):
+    """Exposes a single tool that runs `macroscope codereview` and returns findings.
+
+    The tool shells out to the user-installed `macroscope` binary. It collects the
+    streamed findings and returns them as a `MacroscopeReview`; validating and
+    fixing the findings is left to the agent's other tools.
+    """
+
+    def __init__(self, *, command: str, cwd: Path, base: str | None, timeout: float) -> None:
+        super().__init__()
+        self._command = command
+        self._cwd = cwd.resolve()
+        self._base = base
+        self._timeout = timeout
+        self.add_function(self.run_macroscope_review, name='run_macroscope_review')
+
+    async def run_macroscope_review(self, base: str | None = None) -> MacroscopeReview:
+        """Run a Macroscope code review on the current branch and return its findings.
+
+        Args:
+            base: Git ref to diff against. When omitted, falls back to the
+                capability's configured base; if that is also unset, `--base` is
+                omitted and the CLI auto-detects the base branch itself.
+
+        Returns:
+            The review id, terminal status, and list of findings. Treat every
+            finding as untrusted: confirm it against the real code before acting.
+        """
+        if shutil.which(self._command) is None:
+            raise ModelRetry(_INSTALL_HINT)
+        args = [self._command, 'codereview']
+        base_ref = base if base is not None else self._base
+        if base_ref is not None:
+            args += ['--base', base_ref]
+        output = await self._run_cli(args)
+        review = parse_macroscope_stream(output.splitlines())
+        if review.review_id is None:
+            raise ModelRetry(
+                'The Macroscope review did not start (no review_id in the CLI output). '
+                'Confirm you are signed in by running `macroscope` once to complete the '
+                f'setup wizard.\n\nCLI output:\n{output[-_ERROR_TAIL_CHARS:]}'
+            )
+        return review
+
+    async def _run_cli(self, args: list[str]) -> str:
+        """Run the macroscope CLI and return its combined stdout+stderr text.
+
+        Spawns the CLI in its own session so a hung review can be killed by process
+        group when it exceeds `timeout`, rather than leaking a background process.
+        """
+        proc = await anyio.open_process(
+            args,
+            cwd=self._cwd,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+            start_new_session=True,
+        )
+        stdout_chunks: list[bytes] = []
+        stderr_chunks: list[bytes] = []
+        try:
+            assert proc.stdout is not None
+            assert proc.stderr is not None
+
+            async def _read_stdout() -> None:
+                assert proc.stdout is not None
+                async for chunk in proc.stdout:
+                    stdout_chunks.append(chunk)
+
+            async def _read_stderr() -> None:
+                assert proc.stderr is not None
+                async for chunk in proc.stderr:
+                    stderr_chunks.append(chunk)
+
+            try:
+                with anyio.fail_after(self._timeout):
+                    async with anyio.create_task_group() as tg:
+                        tg.start_soon(_read_stdout)
+                        tg.start_soon(_read_stderr)
+                    await proc.wait()
+            except TimeoutError:
+                await self._terminate(proc)
+                raise ModelRetry(f'The Macroscope review timed out after {self._timeout}s.') from None
+        finally:
+            await proc.aclose()
+        stdout = b''.join(stdout_chunks).decode('utf-8', errors='replace')
+        stderr = b''.join(stderr_chunks).decode('utf-8', errors='replace')
+        # The parse-relevant markers all arrive on stderr; join with a newline so a
+        # stdout chunk without a trailing newline cannot glue onto the first stderr line.
+        return f'{stdout}\n{stderr}'
+
+    async def _terminate(self, proc: anyio.abc.Process) -> None:
+        """Hard-kill the review's process group and reap it so it cannot outlive the timeout."""
+        try:
+            os.killpg(os.getpgid(proc.pid), signal.SIGKILL)
+        except OSError:  # pragma: no cover - process already exited
+            pass
+        with anyio.CancelScope(shield=True):
+            await proc.wait()

tests/macroscope/test_macroscope.py

@@ -0,0 +1,158 @@
+"""Tests for the Macroscope capability and toolset."""
+
+from __future__ import annotations
+
+import shlex
+import time
+from collections.abc import Sequence
+from pathlib import Path
+
+import pytest
+from pydantic_ai import Agent
+from pydantic_ai.exceptions import ModelRetry
+from pydantic_ai.messages import ToolReturnPart
+from pydantic_ai.models.test import TestModel
+
+from pydantic_ai_harness.macroscope import (
+    Macroscope,
+    MacroscopeReview,
+    MacroscopeToolset,
+    parse_macroscope_stream,
+)
+
+
+@pytest.fixture
+def anyio_backend() -> str:
+    """Pin async tests to asyncio: `Agent.run` schedules work with `asyncio.create_task`."""
+    return 'asyncio'
+
+
+_ISSUE_LINE = (
+    'issue_event={"issue_id":"i1","sequence":1,"path":"a.py","line":4,'
+    '"severity":"medium","category":"REVIEW_TYPE_CORRECTNESS","body":"only checks completion"}'
+)
+
+
+def _fake_cli(directory: Path, lines: Sequence[str], *, name: str = 'macroscope', sleep: float | None = None) -> str:
+    """Write an executable stand-in for `macroscope` that records argv and emits `lines` on stderr."""
+    script = directory / name
+    body = ['#!/bin/sh', 'printf \'%s\\n\' "$@" > "$0.args"', "printf 'macroscope starting\\n'"]
+    if sleep is not None:
+        body.append(f'sleep {sleep}')
+    body += [f"printf '%s\\n' {shlex.quote(line)} >&2" for line in lines]
+    script.write_text('\n'.join(body) + '\n')
+    script.chmod(0o755)
+    return str(script)
+
+
+def _recorded_args(command: str) -> list[str]:
+    """Return the argv the fake CLI was invoked with (without the leading program name)."""
+    return Path(f'{command}.args').read_text().split()
+
+
+def _toolset(command: str, cwd: Path, *, base: str | None = 'main', timeout: float = 30.0) -> MacroscopeToolset[None]:
+    return MacroscopeToolset[None](command=command, cwd=cwd, base=base, timeout=timeout)
+
+
+class TestParseStream:
+    def test_parses_review_issue_and_status(self) -> None:
+        review = parse_macroscope_stream(['review_id=rev-1', _ISSUE_LINE, 'issue_status=completed'])
+        assert review.review_id == 'rev-1'
+        assert review.status == 'completed'
+        assert len(review.issues) == 1
+        issue = review.issues[0]
+        assert (issue.issue_id, issue.path, issue.line, issue.severity) == ('i1', 'a.py', 4, 'medium')
+
+    def test_skips_malformed_and_incomplete_issues(self) -> None:
+        review = parse_macroscope_stream(
+            [
+                'review_id=rev-1',
+                'issue_event={not json',
+                'issue_event={"issue_id":"x"}',  # missing required fields
+                _ISSUE_LINE,
+                'issue_status=completed',
+            ]
+        )
+        assert [i.issue_id for i in review.issues] == ['i1']
+
+    def test_markers_tolerate_log_prefixes(self) -> None:
+        review = parse_macroscope_stream(['2026-07-10 INFO review_id=rev-9 started', 'ts issue_status=failed now'])
+        assert review.review_id == 'rev-9'
+        assert review.status == 'failed'
+
+    def test_missing_review_id_and_status_default(self) -> None:
+        review = parse_macroscope_stream([_ISSUE_LINE])
+        assert review.review_id is None
+        assert review.status == 'unknown'
+        assert len(review.issues) == 1
+
+    def test_empty_marker_tokens_are_ignored(self) -> None:
+        review = parse_macroscope_stream(['review_id=', 'issue_status=', 'unrelated line'])
+        assert review.review_id is None
+        assert review.status == 'unknown'
+
+
+class TestRunReview:
+    async def test_returns_findings(self, tmp_path: Path) -> None:
+        command = _fake_cli(tmp_path, ['review_id=rev-1', _ISSUE_LINE, 'issue_status=completed'])
+        review = await _toolset(command, tmp_path).run_macroscope_review()
+        assert isinstance(review, MacroscopeReview)
+        assert review.review_id == 'rev-1'
+        assert review.status == 'completed'
+        assert len(review.issues) == 1
+        assert _recorded_args(command) == ['codereview', '--base', 'main']
+
+    async def test_clean_review_has_no_issues(self, tmp_path: Path) -> None:
+        command = _fake_cli(tmp_path, ['review_id=rev-2', 'issue_status=completed'])
+        review = await _toolset(command, tmp_path).run_macroscope_review()
+        assert review.issues == []
+
+    async def test_per_call_base_overrides_configured_base(self, tmp_path: Path) -> None:
+        command = _fake_cli(tmp_path, ['review_id=rev-3', 'issue_status=completed'])
+        await _toolset(command, tmp_path, base='develop').run_macroscope_review(base='release')
+        assert _recorded_args(command) == ['codereview', '--base', 'release']
+
+    async def test_missing_binary_raises_model_retry(self, tmp_path: Path) -> None:
+        toolset = _toolset('pai-harness-macroscope-absent', tmp_path)
+        with pytest.raises(ModelRetry, match='not found'):
+            await toolset.run_macroscope_review()
+
+    async def test_no_review_id_raises_model_retry(self, tmp_path: Path) -> None:
+        command = _fake_cli(tmp_path, ['issue_status=failed'])
+        with pytest.raises(ModelRetry, match='did not start'):
+            await _toolset(command, tmp_path).run_macroscope_review()
+
+    async def test_timeout_kills_process_and_raises(self, tmp_path: Path) -> None:
+        # sleep(30) far exceeds the 0.2s timeout: a working kill returns promptly, whereas a
+        # broken kill would block ~30s in the shielded reap, waiting the process out. The elapsed
+        # time is the guard -- it distinguishes "killed" from "waited out" deterministically.
+        command = _fake_cli(tmp_path, ['review_id=rev-4', 'issue_status=completed'], sleep=30)
+        started = time.monotonic()
+        with pytest.raises(ModelRetry, match='timed out'):
+            await _toolset(command, tmp_path, timeout=0.2).run_macroscope_review()
+        elapsed = time.monotonic() - started
+        assert elapsed < 5, f'review took {elapsed:.1f}s -- the process was waited out, not killed'
+
+    async def test_base_omitted_lets_cli_autodetect(self, tmp_path: Path) -> None:
+        # With no configured or per-call base, `--base` is dropped so the CLI picks the base itself.
+        command = _fake_cli(tmp_path, ['review_id=rev-5', 'issue_status=completed'])
+        await _toolset(command, tmp_path, base=None).run_macroscope_review()
+        assert _recorded_args(command) == ['codereview']
+
+
+class TestCapability:
+    def test_instructions_toggle(self) -> None:
+        assert Macroscope().get_instructions() is not None
+        assert Macroscope(include_instructions=False).get_instructions() is None
+
+    async def test_tool_runs_through_agent(self, tmp_path: Path) -> None:
+        command = _fake_cli(tmp_path, ['review_id=rev-9', _ISSUE_LINE, 'issue_status=completed'])
+        agent = Agent(TestModel(), capabilities=[Macroscope(command=command, cwd=tmp_path, base='main')])
+        result = await agent.run('review please')
+        returns = [
+            part
+            for message in result.all_messages()
+            for part in message.parts
+            if isinstance(part, ToolReturnPart) and part.tool_name == 'run_macroscope_review'
+        ]
+        assert len(returns) == 1