Douwe MaanandGitHub bf692aff32 Rework README hero example and add ecosystem agent showcase (#226) ...

* Rework README hero example and add full ecosystem agent showcase

The previous hero example used GitHub's MCP server, which requires
authentication and ships tools without `outputSchema` -- triggering
CodeMode warnings and producing weakly-typed sandbox stubs. Replaced
with the open-source Hacker News MCP server (full output schemas, no
auth), and added `WebSearch` to make the prompt naturally exercise
parallel tool orchestration in a single `run_code` call.

`MCP(..., builtin=False)` is now explicit so the local toolset is used
and CodeMode actually wraps the tools (Anthropic supports MCP server
connectors as a builtin and would otherwise execute tools server-side,
bypassing the sandbox).

Added a "Full ecosystem agent" section above the capability matrix that
illustrates the breadth of the Pydantic AI ecosystem -- combining
capabilities from the harness, core, and the community packages we
endorse, with inline credits at each call site.

Made install requirements explicit above the hero block.

* Reword Quick start prompt to avoid datetime in generated code

Monty currently rejects `datetime` (pydantic/monty issue tracking),
so the previous prompt's "posted in the last 6 hours" filter caused
the agent to generate code that failed in the sandbox. Replaced with
a dedupe-by-id + score filter + parallel follow-up calls (thread,
user profile, web search) -- still exercises parallel orchestration,
dict manipulation, and multi-tool composition without any timestamps.

* Embed verbatim run output and soften 'single run_code' claim

Followed pydantic-ai docs convention of triple-quoted output after
print(). The included synthesis is the actual output from running
the example, not a fabrication.

Reworded the CodeMode explanation to match what Sonnet actually does:
two batched run_code calls each containing parallel tool calls, with
the dedupe/filter happening as plain Python in the sandbox -- not the
"all in a single run_code" claim from the previous draft.

* Surface monty timing restrictions in run_code description

Sonnet was occasionally generating `asyncio.sleep` between tool calls
(cargo-culted "polite rate-limiting" pattern) and `datetime.now()` for
recency filters -- both rejected by the monty sandbox. Adding an explicit
"no wall-clock or timing primitives" bullet to the `run_code` tool
description (and the matching code-mode README entry) keeps the prompt
free of sandbox-specific babysitting.

Switched the hero example to claude-opus-4-7 with a more natural prompt
("find the most-discussed story... summarize what you find in one
paragraph"), and refreshed the verbatim sample output to a clean run
that produced 3 run_code calls (3x parallel feed fetches, pure-Python
dedupe/filter, then 3x parallel follow-up calls) with no errors.

* Quick start: opt out of native WebSearch so CodeMode wraps it

`WebSearch()` defaults to `builtin=True`, and on providers like
Anthropic that natively support web search the local DDG fallback
is filtered out by `Model.prepare_request`. With `CodeMode` in the
mix that produced a split surface -- builtin web_search at top
level, duckduckgo_search inside `run_code` -- and the model
sometimes tried to call the builtin from inside the sandbox.

Passing `builtin=False` keeps every web call going through CodeMode
so it can be batched alongside the HN tools in a single `run_code`.
Same pattern we already use for `MCP(..., builtin=False)`.

The underlying flatten-prefer_builtin-into-run_code bug is filed
as #233 and is being fixed separately.

Refreshed the verbatim sample output to a clean run with this
configuration: 3 chats + 2 run_code calls, all parallel tool
batches, no errors.

* Quick start: link the public trace and refresh verbatim output

DouweM picked a run with substantively richer commentary (Simon
Willison's "Vibe coding and agentic engineering are getting closer
than I'd like" post on HN) and made the trace public, so we can
finally point readers at the actual run from the example.

* Add duckduckgo extra to install snippet for WebSearch(builtin=False)

* Move ecosystem example credits to dedicated comment lines

* Restructure README around the Quick start trace and an ecosystem agent

Quick start
- Move all per-capability commentary into inline comments at call sites,
  so links sit next to the code they document. Drops the trailing
  paragraph wall.
- Make the public Logfire trace the section's main visual: a screenshot
  linking to the trace, then a one-liner explainer.
- Trim the verbatim sample output to ~5 sentences. Keeps the Willison
  framing and the meta-substantive thread reference; drops the long
  commenter-by-commenter paraphrase and the news-coverage paragraph
  that previously took over half the section.

An ecosystem agent (formerly "Full ecosystem agent")
- Move below the capability matrix so readers meet the breadth before
  the worked example.
- Renamed and rewritten intro: shorter, voice closer to the rest of
  the README, no "upper bound of what's possible" phrasing.
- Long disclaimer paragraph moved below the code so readers see the
  example before the caveats.
- Imports split into official (`pydantic_ai*` + harness) vs community
  (alphabetical).
- Sections reorganized: Reasoning, Tools, Execution (CodeMode pulled
  out of Tools), Context management, Memory, Orchestration, Safety.
- Each capability gets its own one-line comment. AnthropicCompaction
  added as the official compaction option, with a comment pointing at
  vstorm-co's `summarization-pydantic-ai` as a provider-agnostic alt.
- Agent gets a name (`brian` -- Monty Python) which `MemoryCapability`
  picks up via its `agent_name` arg.
- Skills comment now flags @vstorm-co's pydantic-deep alongside Doug's,
  noting the shape difference.
- Switched to `claude-opus-4-7` and `Thinking(effort='high')` to match
  the Quick start.
- Drops `MCP(builtin=False)` here -- the ecosystem example isn't trying
  to demo CodeMode-wrapped MCP, it's demoing breadth.

Drops the no-wall-clock bullet from `_toolset.py` and the code-mode
README -- now covered by the standalone PR for that change.

Trace screenshot reference uses a relative path that resolves on GitHub
once the asset is committed; ready to swap in once the file is in place.

* Ecosystem agent: rename to shrubbery, bump thinking to xhigh

* Add Logfire trace screenshots and surface them prominently

`docs/images/quick-start-trace.png` is the trace tree: it's the visual
the main README's Quick start now leads with -- click-through to the
public trace.

`docs/images/code-mode-trace.png` is the wider shot showing the actual
Python the model wrote (parallel `asyncio.gather` over three HN feeds,
dedupe-by-id, score filter). It's the centrepiece of a new "In practice"
section in the code-mode README that points back to the harness Quick
start as the more representative example, replacing the toy weather
demo as the leading visual of what code mode does in real use.

* Address review on PR #226

- Reorder ecosystem agent capabilities: CodeMode first (the headline),
  then Thinking, Context management, Tools (MCP -> WebSearch -> Console),
  then everything else. First-party leads each tier; ConsoleCapability
  follows our two first-party tools.
- Drop the agent `name='shrubbery'` flourish.
- Use vstorm-co's `ContextManagerCapability` as the active compaction
  capability and mention pydantic-ai's `AnthropicCompaction` /
  `OpenAICompaction` in the comment instead of the other way around.
  Drops the "tied to Anthropic's prompt caching" phrasing.
- `MemoryCapability(agent_name='harness-example')`.
- Code-mode README: rewrite the "In practice" prose to be clear that
  CodeMode produces two `run_code` calls (parallel fetches + filter,
  then parallel follow-ups), not one. Fix the screenshot alt text to
  match.

* Add regression test for the README's Quick start example

The lead example in the main README is the most public surface of the
harness — if it stops working we need to know before the user tries it.
This test drives the example end-to-end with `FunctionModel` issuing the
two `run_code` calls the README's trace shows, and `CodeMode` actually
running the emitted Python through Monty.

What gets faked vs. what's real:
- The Hacker News MCP toolset is replaced with a `FunctionToolset` of
  fake functions returning canned data from the README's linked public
  trace. Avoids depending on an MCP package or hitting the network.
- `WebSearch(builtin=False, local=...)` skips the default DuckDuckGo
  fallback so the test doesn't pull `ddgs`.
- `FunctionModel` drives the conversation through two `run_code` calls
  and a final synthesis, mirroring the production trace.
- `CodeMode` itself is real; the `FunctionModel`'s emitted Python runs
  through the Monty sandbox and dispatches calls back to the fakes.

Asserts on the call shape (each feed fetched once in parallel, follow-up
calls target the expected winner id) and on the final synthesis content,
so any future change in pydantic-ai or the harness that breaks how these
capabilities compose makes this test fail.

* Address review on PR #226

- Quick start: list `CodeMode` first in `capabilities=[]` to match the
  ecosystem agent's ordering and to lead with the headline capability.
- Test: wire the fake HN tools through `MCP(local=hn_toolset, builtin=False)`
  instead of bypassing via `toolsets=[hn_toolset]`. Same composition path
  as production minus the network. Needs a narrow `# pyright: ignore`
  because MCP's `__init__` signature narrows `local` to MCP-shaped types
  but the parent `BuiltinOrLocalTool` accepts any `AbstractToolset` at
  runtime.
- Test: have the second `run_code` call `web_search` (the WebSearch
  capability's local fallback) instead of `hn_search_content`, so the
  fake function is actually exercised. This fixes the coverage failure
  on lines 231-232 and makes the test more representative of the README
  example, which uses WebSearch.

* Use snapshot()+dirty-equals for README quick-start test

* Re-export dirty-equals via tests/conftest, restructure tests/

Two related cleanups based on review:

- Re-export `IsDatetime` / `IsNow` / `IsStr` / `IsPartialDict` from
  `tests/conftest.py` with `TYPE_CHECKING` shims that pretend the
  matchers return the concrete type (`-> str`, `-> datetime`, etc.).
  Mirrors pydantic-ai's own conftest pattern and lets pyright strict
  accept `tool_call_id=IsStr()`, `timestamp=IsDatetime()`, etc. without
  the file-level `# pyright: reportArgumentType=false` we had before.
  Tests now `from .conftest import ...` instead of importing directly
  from `dirty_equals`.

- Rename `tests/_code_mode/` -> `tests/code_mode/` (drop the underscore
  prefix; matches the source package `pydantic_ai_harness/code_mode/`).
  Updates the convention note in AGENTS.md/CLAUDE.md.

- Move `test_readme_quick_start.py` up to `tests/` top level. It's an
  end-to-end check on the README's flagship example -- it composes
  `CodeMode` + `MCP` + `WebSearch` -- so it doesn't belong in any
  single capability's test directory.

2026-05-08 18:05:42 -06:00

0 lines

0 B

Python

Raw Permalink Blame History

The file is empty.