mirror of
https://github.com/pydantic/pydantic-ai-harness.git
synced 2026-07-21 02:45:34 +00:00
feat: excerpt-based web_search, retry seam, domain filters, spec support
Audit follow-up on the ExaSearch capability: - web_search now requests Exa highlights instead of full page text, per Exa's search guide for agents: excerpts make the survey step cheap and leave full-text reading to get_page, matching the capability's own survey-then-dig instructions. - Transient Exa failures (non-2xx ValueError, httpx errors) convert to ModelRetry via a _recoverable seam mirroring Shell/FileSystem; 401/403 auth failures propagate as configuration errors. - get_page requests one char of headroom above max_text_chars so the local truncation marker can actually fire (impossible at the 10,000 API ceiling, documented); deep_search answers are no longer capped. - Bounds follow the Exa API reference (num_results 1-100, max_text_chars 1-10,000) and validate in __post_init__ so the object the user wrote fails; include_domains/exclude_domains added as mutually exclusive filters; guidance field follows the Planning/Memory contract. - from_spec override (Memory precedent) keeps the protocol-typed client out of the spec schema, enabling Agent.from_file and schema generation. - Docs: MCP tool catalog corrected to the current live set with the create-then-poll vs single-call contrast, new core-WebSearch comparison (including the Anthropic web_search name duplication caveat), and an agent-spec section. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Bill Easton
co-authored by
Claude Fable 5
Claude Fable 5
parent
e72e11353d
commit
86cc4795b1
@@ -151,7 +151,7 @@ We studied leading coding agents, agent frameworks, and Claw-style assistants to
|
||||
| | **Shell** | Execute commands with allowlists, denylists, and timeouts | :white_check_mark: [Docs](pydantic_ai_harness/shell/) | [pydantic-ai-backend](https://github.com/vstorm-co/pydantic-ai-backend) (vstorm‑co) |
|
||||
| | **Repo context injection** | Auto-load CLAUDE.md/AGENTS.md and repo structure | :white_check_mark: [Docs](pydantic_ai_harness/context/) | [pydantic-deep](https://github.com/vstorm-co/pydantic-deepagents) (vstorm‑co) |
|
||||
| | **Docs lookup** | On-demand `read_pyai_docs` tool for Pydantic AI docs | :white_check_mark: [Docs](pydantic_ai_harness/docs/) | |
|
||||
| | **Web research** | Web search returning full page text, single-page retrieval, and opt-in deep search with cited answers, backed by [Exa](https://exa.ai) | :white_check_mark: [Docs](pydantic_ai_harness/exa/) | |
|
||||
| | **Web research** | Web search returning relevant page excerpts, full single-page retrieval, and opt-in deep search with cited answers, backed by [Exa](https://exa.ai) | :white_check_mark: [Docs](pydantic_ai_harness/exa/) | |
|
||||
| | **Verification loop** | Run tests after edits, auto-fix failures | :construction: [PR #169](https://github.com/pydantic/pydantic-ai-harness/pull/169) | |
|
||||
| **Editor integration** | **ACP** | Serve an agent to editors (Zed, etc.) over the [Agent Client Protocol](https://agentclientprotocol.com) -- streamed text, diff-rendered edits, tool approval | :white_check_mark: [Docs](pydantic_ai_harness/experimental/acp/) (experimental) | |
|
||||
| **Prompt management** | **Managed prompt** | Back an agent's instructions with a [Logfire](https://pydantic.dev/logfire)-managed prompt, editable without shipping code | :white_check_mark: [Docs](pydantic_ai_harness/logfire/) | |
|
||||
|
||||
+98
-35
@@ -1,29 +1,29 @@
|
||||
---
|
||||
title: Exa Search
|
||||
description: Give a Pydantic AI agent web research tools backed by the Exa search API -- search with page text, full-page retrieval, and opt-in deep search, with page text capped per result to bound tool output.
|
||||
description: Give a Pydantic AI agent web research tools backed by the Exa search API -- search with relevant excerpts, full-page retrieval, and opt-in deep search, with output budgeted per tool.
|
||||
---
|
||||
|
||||
# Exa Search
|
||||
|
||||
`ExaSearch` gives an agent web research tools backed by the
|
||||
[Exa](https://exa.ai) search API: search that returns page text alongside each
|
||||
hit, full-page retrieval for digging into a specific URL, and opt-in deep
|
||||
search that synthesizes a cited answer in one call. Page text is capped per
|
||||
result to bound tool output.
|
||||
[Exa](https://exa.ai) search API: search that returns the most relevant
|
||||
excerpts from each hit, full-page retrieval for digging into a specific URL,
|
||||
and opt-in deep search that synthesizes a cited answer in one call.
|
||||
|
||||
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/exa/)
|
||||
|
||||
## The problem
|
||||
|
||||
Search tools that return only titles and snippets force a second round of
|
||||
fetching before the agent can judge a source. Wiring a search API together with
|
||||
a page fetcher, capping page text so tool output doesn't overwhelm the model's
|
||||
context, and prompting the agent to research methodically is boilerplate every
|
||||
research agent reinvents.
|
||||
fetching before the agent can judge a source, while search tools that return
|
||||
full page text flood the context with pages the agent will discard. Wiring a
|
||||
search API together with a page fetcher, budgeting what each tool returns, and
|
||||
prompting the agent to research methodically is boilerplate every research
|
||||
agent reinvents.
|
||||
|
||||
`ExaSearch` bundles that plumbing into a single
|
||||
[capability](/ai/core-concepts/capabilities/): the research tools, a per-result
|
||||
page-text cap, and short research guidance in the system prompt.
|
||||
[capability](/ai/core-concepts/capabilities/): the research tools, per-tool
|
||||
output budgets, and short research guidance in the system prompt.
|
||||
|
||||
## Usage
|
||||
|
||||
@@ -52,20 +52,29 @@ print(result.output)
|
||||
|
||||
| Tool | Purpose |
|
||||
|---|---|
|
||||
| `web_search` | Search the web and return the top `num_results` pages, each with title, URL, and page text. |
|
||||
| `get_page` | Retrieve the text contents of one specific URL. |
|
||||
| `web_search` | Search the web and return the top `num_results` pages, each with title, URL, and its most relevant excerpts. |
|
||||
| `get_page` | Retrieve the full text of one specific URL -- a promising `web_search` hit, or a URL the user provided. |
|
||||
| `deep_search` | Run Exa's multi-step deep search and return a synthesized, cited answer. Opt-in via `include_deep_search=True`. |
|
||||
|
||||
Page text is capped at `max_text_chars` characters per result: the cap is sent
|
||||
to Exa as the contents limit and re-enforced when tool output is formatted, so
|
||||
text stays bounded even with a custom client. The result count is bounded the
|
||||
same way (`num_results` is requested from Exa and re-applied to the response).
|
||||
When text is cut, the **head** is kept (a page's lead carries the substance)
|
||||
and a `[... page text truncated at N characters]` marker is appended.
|
||||
`web_search` returns short excerpts (Exa highlights) rather than full page
|
||||
text, following [Exa's own guidance for agents](https://exa.ai/docs/reference/search-api-guide-for-coding-agents),
|
||||
so surveying several sources stays cheap; the agent reads a chosen page with
|
||||
`get_page`.
|
||||
|
||||
A URL or question that returns no content surfaces to the model as a
|
||||
`get_page` text is capped at `max_text_chars` characters, keeping the **head**
|
||||
(a page's lead carries the substance). One character of headroom above the cap
|
||||
is requested from Exa, so when a page exceeds the cap the output ends with a
|
||||
`[... page text truncated at N characters]` marker; at the API ceiling of
|
||||
10,000 characters no headroom exists, so the marker cannot appear there. The
|
||||
result count is bounded the same way: `num_results` is requested from Exa and
|
||||
re-applied to the response.
|
||||
|
||||
A URL or question that returns no content, a rate limit, or a transient API or
|
||||
network failure surfaces to the model as a
|
||||
[`ModelRetry`](/ai/tools-toolsets/tools-advanced/#tool-retries) rather than a
|
||||
hard error: the run continues and the model can correct the URL or rephrase.
|
||||
hard error: the run continues and the model can correct the URL, rephrase, or
|
||||
try again. Authentication failures (401/403) are configuration errors and
|
||||
propagate.
|
||||
|
||||
## Deep search
|
||||
|
||||
@@ -83,7 +92,9 @@ ExaSearch(include_deep_search=True)
|
||||
```
|
||||
|
||||
When enabled, the capability's instructions tell the model to treat it as an
|
||||
escalation from `web_search`, not a replacement.
|
||||
escalation from `web_search`, not a replacement. The synthesized answer is
|
||||
returned in full (it is Exa-generated and inherently bounded); `max_text_chars`
|
||||
only applies to `get_page`.
|
||||
|
||||
## Instructions
|
||||
|
||||
@@ -91,7 +102,8 @@ escalation from `web_search`, not a replacement.
|
||||
wide with `web_search` first, read the most promising pages in full with
|
||||
`get_page` before drawing conclusions, prefer primary sources, and cite the
|
||||
URLs relied on. With `include_deep_search=True`, the guidance also covers when
|
||||
to escalate to `deep_search`.
|
||||
to escalate to `deep_search`. Set `guidance` to replace the default text, or
|
||||
to `''` to contribute no instructions at all.
|
||||
|
||||
## Configuration
|
||||
|
||||
@@ -101,13 +113,20 @@ Every field of `ExaSearch` with its default:
|
||||
from pydantic_ai_harness.exa import ExaSearch
|
||||
|
||||
ExaSearch(
|
||||
num_results=5, # results per web_search call
|
||||
max_text_chars=10_000, # page-text cap per result, in characters
|
||||
num_results=5, # results per web_search call (1 to 100)
|
||||
max_text_chars=10_000, # get_page text cap, in characters (1 to 10,000)
|
||||
include_deep_search=False, # also expose the deep_search tool
|
||||
include_domains=[], # only search these domains (allowlist)
|
||||
exclude_domains=[], # never search these domains (denylist)
|
||||
guidance=None, # None = default instructions, '' = none, str = custom
|
||||
client=None, # ExaClient -- None builds exa_py.AsyncExa from EXA_API_KEY
|
||||
)
|
||||
```
|
||||
|
||||
`include_domains` and `exclude_domains` apply to `web_search` and
|
||||
`deep_search`, and are mutually exclusive -- set one, not both. Out-of-range
|
||||
limits and setting both domain lists raise a `ValueError` at construction.
|
||||
|
||||
## Custom client
|
||||
|
||||
The default client is `exa_py.AsyncExa`, configured from the `EXA_API_KEY`
|
||||
@@ -127,23 +146,41 @@ ExaSearch(client=AsyncExa(api_key='...'))
|
||||
The API may change between releases while the capability settles; breaking
|
||||
changes ship deprecation warnings where practical.
|
||||
|
||||
## ExaSearch vs core WebSearch
|
||||
|
||||
Pydantic AI core ships a provider-adaptive
|
||||
[`WebSearch`](/ai/core-concepts/capabilities/#provider-adaptive-tools)
|
||||
capability: on models with a native search tool it uses the provider's own
|
||||
search, executed server-side; elsewhere it falls back to a local DuckDuckGo
|
||||
tool. Reach for it when you want search that follows the model.
|
||||
|
||||
Reach for `ExaSearch` when you want the same search behavior on every model:
|
||||
one vendor, excerpts with every hit, explicit page retrieval, domain filters,
|
||||
and opt-in deep search.
|
||||
|
||||
One caveat when combining them: on Anthropic models the provider-native search
|
||||
tool is also named `web_search` on the wire, so
|
||||
`capabilities=[WebSearch(), ExaSearch()]` puts two tools with the same name in
|
||||
the request. Use one search capability per agent on native-search models, or
|
||||
force the local fallback with `WebSearch(native=False)` (its DuckDuckGo tool is
|
||||
named `duckduckgo_search`, which does not collide).
|
||||
|
||||
## ExaSearch vs Exa's MCP server
|
||||
|
||||
Exa also ships an official hosted MCP server at `https://mcp.exa.ai/mcp`
|
||||
([exa-labs/exa-mcp-server](https://github.com/exa-labs/exa-mcp-server)) with
|
||||
their full tool catalog: `web_search_exa`, `get_code_context_exa`,
|
||||
`crawling_exa`, `company_research_exa`, `linkedin_search_exa`, and a
|
||||
`deep_researcher_start` / `deep_researcher_check` pair.
|
||||
([exa-labs/exa-mcp-server](https://github.com/exa-labs/exa-mcp-server)). By
|
||||
default it exposes `web_search_exa` and `web_fetch_exa`; the full catalog adds
|
||||
`web_search_advanced_exa` and an agent-run set (`agent_create_run`,
|
||||
`agent_wait_for_run`, `agent_get_run_output`, `agent_cancel_run`).
|
||||
|
||||
`ExaSearch` is the curated, typed path: bounded output, a retry-on-empty
|
||||
contract, bundled research instructions, and a client seam that makes it
|
||||
testable offline. The MCP server is how you get Exa's full catalog with zero
|
||||
wrapper code, via Pydantic AI core's
|
||||
[MCP capability](/ai/core-concepts/capabilities/). Where the MCP deep
|
||||
researcher is a start/check tool pair the model must poll across calls,
|
||||
`deep_search` returns the answer in a single call. The two compose in one
|
||||
`capabilities` list, and the tool names don't collide (the MCP tools are
|
||||
`*_exa`-suffixed):
|
||||
wrapper code, via Pydantic AI core's MCP capability. Their agent runs are
|
||||
create-then-poll (`agent_create_run` returns an ID immediately;
|
||||
`agent_wait_for_run` polls it), where `deep_search` returns the answer in a
|
||||
single call. The two compose in one `capabilities` list, and none of the MCP
|
||||
tool names collide with `web_search`, `get_page`, or `deep_search`:
|
||||
|
||||
```python
|
||||
from pydantic_ai import Agent
|
||||
@@ -153,6 +190,32 @@ from pydantic_ai_harness.exa import ExaSearch
|
||||
agent = Agent('anthropic:claude-sonnet-4-6', capabilities=[ExaSearch(), MCP('https://mcp.exa.ai/mcp')])
|
||||
```
|
||||
|
||||
## Agent spec (YAML/JSON)
|
||||
|
||||
`ExaSearch` works with Pydantic AI's
|
||||
[agent spec](/ai/core-concepts/agent-spec/), so you can declare it in a config
|
||||
file instead of Python:
|
||||
|
||||
```yaml
|
||||
# agent.yaml
|
||||
model: anthropic:claude-sonnet-4-6
|
||||
capabilities:
|
||||
- ExaSearch:
|
||||
num_results: 3
|
||||
include_deep_search: true
|
||||
```
|
||||
|
||||
```python
|
||||
from pydantic_ai import Agent
|
||||
from pydantic_ai_harness.exa import ExaSearch
|
||||
|
||||
agent = Agent.from_file('agent.yaml', custom_capability_types=[ExaSearch])
|
||||
```
|
||||
|
||||
Pass `custom_capability_types` so the spec loader knows how to instantiate
|
||||
`ExaSearch`. The `client` field is not spec-serializable; spec-loaded instances
|
||||
always build the default client from `EXA_API_KEY`.
|
||||
|
||||
## Further reading
|
||||
|
||||
- [Pydantic AI capabilities](/ai/core-concepts/capabilities/)
|
||||
|
||||
+1
-1
@@ -117,7 +117,7 @@ Each capability is a self-contained battery you drop into an agent's `capabiliti
|
||||
| [Shell](shell.md) | Command execution in a subprocess rooted at a working directory, gated by allowlists, denylists, timeouts, and optional environment-variable stripping (including a preset for common LLM provider credentials). | -- |
|
||||
| [Context](context.md) | Auto-loads repo context -- `CLAUDE.md`/`AGENTS.md` and repository structure -- so the agent starts a run already oriented in the project. | -- |
|
||||
| [Pydantic AI Docs](pydantic-ai-docs.md) | An on-demand `read_pyai_docs` tool that pulls Pydantic AI documentation into the run when the agent needs it, instead of preloading it. | -- |
|
||||
| [Exa Search](exa-search.md) | Web research backed by the [Exa](https://exa.ai) search API: `web_search` returns results with page text, `get_page` reads a specific URL, and opt-in `deep_search` synthesizes a cited answer in one call. Page text is capped per result to bound tool output. | `exa` |
|
||||
| [Exa Search](exa-search.md) | Web research backed by the [Exa](https://exa.ai) search API: `web_search` returns results with their most relevant excerpts, `get_page` reads a specific URL in full, and opt-in `deep_search` synthesizes a cited answer in one call. Output is budgeted per tool. | `exa` |
|
||||
| [Compaction](compaction.md) | Keeps a run within token limits: sliding-window trimming, LLM-powered summarization of older messages, and warnings before the context or iteration ceiling is hit. | -- |
|
||||
| [Overflowing Tool Output](overflowing-tool-output.md) | Reduces an oversized tool return when it is produced -- truncate, spill to a queryable file, or summarize -- so a large payload does not persist in history and get re-sent every request. | -- |
|
||||
| [Step Persistence](step-persistence.md) | Saves and restores full conversation state; snapshot, resume (`continue_run`), and fork (`fork_run`) a run. | -- |
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
# Exa Search
|
||||
|
||||
Give an agent web research tools backed by the [Exa](https://exa.ai) search
|
||||
API: search that returns page text alongside each hit, full-page retrieval for
|
||||
digging into a specific URL, and opt-in deep search that synthesizes a cited
|
||||
answer in one call.
|
||||
API: search that returns the most relevant excerpts from each hit, full-page
|
||||
retrieval for digging into a specific URL, and opt-in deep search that
|
||||
synthesizes a cited answer in one call.
|
||||
|
||||
[Source](https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/exa/)
|
||||
|
||||
@@ -20,15 +20,17 @@ Set the `EXA_API_KEY` environment variable (create a key at
|
||||
## The problem
|
||||
|
||||
Search tools that return only titles and snippets force a second round of
|
||||
fetching before the agent can judge a source. Wiring a search API together with
|
||||
a page fetcher, capping page text so tool output doesn't overwhelm the model's
|
||||
context, and prompting the agent to research methodically is boilerplate every
|
||||
research agent reinvents.
|
||||
fetching before the agent can judge a source, while search tools that return
|
||||
full page text flood the context with pages the agent will discard. Wiring a
|
||||
search API together with a page fetcher, budgeting what each tool returns, and
|
||||
prompting the agent to research methodically is boilerplate every research
|
||||
agent reinvents.
|
||||
|
||||
## The solution
|
||||
|
||||
`ExaSearch` bundles the tools with output capping and short research guidance
|
||||
in the system prompt:
|
||||
`ExaSearch` bundles the tools with output budgeting and short research
|
||||
guidance in the system prompt: survey cheaply with excerpts, then read the
|
||||
pages that matter in full.
|
||||
|
||||
```python
|
||||
from pydantic_ai import Agent
|
||||
@@ -44,20 +46,27 @@ print(result.output)
|
||||
|
||||
| Tool | Purpose |
|
||||
|---|---|
|
||||
| `web_search` | Search the web and return the top `num_results` pages, each with title, URL, and page text. |
|
||||
| `get_page` | Retrieve the text contents of one specific URL. |
|
||||
| `web_search` | Search the web and return the top `num_results` pages, each with title, URL, and its most relevant excerpts. |
|
||||
| `get_page` | Retrieve the full text of one specific URL -- a promising `web_search` hit, or a URL the user provided. |
|
||||
| `deep_search` | Run Exa's multi-step deep search and return a synthesized, cited answer. Opt-in via `include_deep_search=True`. |
|
||||
|
||||
Page text is capped at `max_text_chars` characters per result: the cap is sent
|
||||
to Exa as the contents limit and re-enforced when tool output is formatted, so
|
||||
text stays bounded even with a custom client. The result count is bounded the
|
||||
same way (`num_results` is requested from Exa and re-applied to the response).
|
||||
When text is cut, the **head** is kept (a page's lead carries the substance)
|
||||
and a `[... page text truncated at N characters]` marker is appended.
|
||||
`web_search` returns short excerpts (Exa highlights) rather than full page
|
||||
text, following [Exa's own guidance for agents](https://exa.ai/docs/reference/search-api-guide-for-coding-agents),
|
||||
so surveying several sources stays cheap; the agent reads a chosen page with
|
||||
`get_page`.
|
||||
|
||||
A URL or question that returns no content surfaces to the model as a
|
||||
`ModelRetry` (the model can correct the URL or rephrase) rather than aborting
|
||||
the run.
|
||||
`get_page` text is capped at `max_text_chars` characters, keeping the **head**
|
||||
(a page's lead carries the substance). One character of headroom above the cap
|
||||
is requested from Exa, so when a page exceeds the cap the output ends with a
|
||||
`[... page text truncated at N characters]` marker; at the API ceiling of
|
||||
10,000 characters no headroom exists, so the marker cannot appear there. The
|
||||
result count is bounded the same way: `num_results` is requested from Exa and
|
||||
re-applied to the response.
|
||||
|
||||
A URL or question that returns no content, a rate limit, or a transient API or
|
||||
network failure surfaces to the model as a `ModelRetry` (the model can correct
|
||||
the URL, rephrase, or try again) rather than aborting the run. Authentication
|
||||
failures (401/403) are configuration errors and propagate.
|
||||
|
||||
## Deep search
|
||||
|
||||
@@ -75,7 +84,9 @@ ExaSearch(include_deep_search=True)
|
||||
```
|
||||
|
||||
When enabled, the capability's instructions tell the model to treat it as an
|
||||
escalation from `web_search`, not a replacement.
|
||||
escalation from `web_search`, not a replacement. The synthesized answer is
|
||||
returned in full (it is Exa-generated and inherently bounded); `max_text_chars`
|
||||
only applies to `get_page`.
|
||||
|
||||
## Instructions
|
||||
|
||||
@@ -83,7 +94,8 @@ escalation from `web_search`, not a replacement.
|
||||
wide with `web_search` first, read the most promising pages in full with
|
||||
`get_page` before drawing conclusions, prefer primary sources, and cite the
|
||||
URLs relied on. With `include_deep_search=True`, the guidance also covers when
|
||||
to escalate to `deep_search`.
|
||||
to escalate to `deep_search`. Set `guidance` to replace the default text, or
|
||||
to `''` to contribute no instructions at all.
|
||||
|
||||
## Configuration
|
||||
|
||||
@@ -93,13 +105,20 @@ Every field of `ExaSearch` with its default:
|
||||
from pydantic_ai_harness.exa import ExaSearch
|
||||
|
||||
ExaSearch(
|
||||
num_results=5, # results per web_search call
|
||||
max_text_chars=10_000, # page-text cap per result, in characters
|
||||
num_results=5, # results per web_search call (1 to 100)
|
||||
max_text_chars=10_000, # get_page text cap, in characters (1 to 10,000)
|
||||
include_deep_search=False, # also expose the deep_search tool
|
||||
include_domains=[], # only search these domains (allowlist)
|
||||
exclude_domains=[], # never search these domains (denylist)
|
||||
guidance=None, # None = default instructions, '' = none, str = custom
|
||||
client=None, # ExaClient -- None builds exa_py.AsyncExa from EXA_API_KEY
|
||||
)
|
||||
```
|
||||
|
||||
`include_domains` and `exclude_domains` apply to `web_search` and
|
||||
`deep_search`, and are mutually exclusive -- set one, not both. Out-of-range
|
||||
limits and setting both domain lists raise a `ValueError` at construction.
|
||||
|
||||
## Custom client
|
||||
|
||||
The default client is `exa_py.AsyncExa`, configured from the `EXA_API_KEY`
|
||||
@@ -119,22 +138,41 @@ ExaSearch(client=AsyncExa(api_key='...'))
|
||||
The API may change between releases while the capability settles; breaking
|
||||
changes ship deprecation warnings where practical.
|
||||
|
||||
## ExaSearch vs core `WebSearch`
|
||||
|
||||
Pydantic AI core ships a provider-adaptive
|
||||
[`WebSearch`](https://ai.pydantic.dev/capabilities/#provider-adaptive-tools)
|
||||
capability: on models with a native search tool it uses the provider's own
|
||||
search, executed server-side; elsewhere it falls back to a local DuckDuckGo
|
||||
tool. Reach for it when you want search that follows the model.
|
||||
|
||||
Reach for `ExaSearch` when you want the same search behavior on every model:
|
||||
one vendor, excerpts with every hit, explicit page retrieval, domain filters,
|
||||
and opt-in deep search.
|
||||
|
||||
One caveat when combining them: on Anthropic models the provider-native search
|
||||
tool is also named `web_search` on the wire, so
|
||||
`capabilities=[WebSearch(), ExaSearch()]` puts two tools with the same name in
|
||||
the request. Use one search capability per agent on native-search models, or
|
||||
force the local fallback with `WebSearch(native=False)` (its DuckDuckGo tool is
|
||||
named `duckduckgo_search`, which does not collide).
|
||||
|
||||
## ExaSearch vs Exa's MCP server
|
||||
|
||||
Exa also ships an official hosted MCP server at `https://mcp.exa.ai/mcp`
|
||||
([exa-labs/exa-mcp-server](https://github.com/exa-labs/exa-mcp-server)) with
|
||||
their full tool catalog: `web_search_exa`, `get_code_context_exa`,
|
||||
`crawling_exa`, `company_research_exa`, `linkedin_search_exa`, and a
|
||||
`deep_researcher_start` / `deep_researcher_check` pair.
|
||||
([exa-labs/exa-mcp-server](https://github.com/exa-labs/exa-mcp-server)). By
|
||||
default it exposes `web_search_exa` and `web_fetch_exa`; the full catalog adds
|
||||
`web_search_advanced_exa` and an agent-run set (`agent_create_run`,
|
||||
`agent_wait_for_run`, `agent_get_run_output`, `agent_cancel_run`).
|
||||
|
||||
`ExaSearch` is the curated, typed path: bounded output, a retry-on-empty
|
||||
contract, bundled research instructions, and a client seam that makes it
|
||||
testable offline. The MCP server is how you get Exa's full catalog with zero
|
||||
wrapper code, via Pydantic AI core's MCP capability. Where the MCP deep
|
||||
researcher is a start/check tool pair the model must poll across calls,
|
||||
`deep_search` returns the answer in a single call. The two compose in one
|
||||
`capabilities` list, and the tool names don't collide (the MCP tools are
|
||||
`*_exa`-suffixed):
|
||||
wrapper code, via Pydantic AI core's MCP capability. Their agent runs are
|
||||
create-then-poll (`agent_create_run` returns an ID immediately;
|
||||
`agent_wait_for_run` polls it), where `deep_search` returns the answer in a
|
||||
single call. The two compose in one `capabilities` list, and none of the MCP
|
||||
tool names collide with `web_search`, `get_page`, or `deep_search`:
|
||||
|
||||
```python
|
||||
from pydantic_ai import Agent
|
||||
@@ -144,6 +182,31 @@ from pydantic_ai_harness.exa import ExaSearch
|
||||
agent = Agent('anthropic:claude-sonnet-4-6', capabilities=[ExaSearch(), MCP('https://mcp.exa.ai/mcp')])
|
||||
```
|
||||
|
||||
## Agent spec (YAML/JSON)
|
||||
|
||||
`ExaSearch` works with Pydantic AI's
|
||||
[agent spec](https://ai.pydantic.dev/agent-spec/):
|
||||
|
||||
```yaml
|
||||
# agent.yaml
|
||||
model: anthropic:claude-sonnet-4-6
|
||||
capabilities:
|
||||
- ExaSearch:
|
||||
num_results: 3
|
||||
include_deep_search: true
|
||||
```
|
||||
|
||||
```python
|
||||
from pydantic_ai import Agent
|
||||
from pydantic_ai_harness.exa import ExaSearch
|
||||
|
||||
agent = Agent.from_file('agent.yaml', custom_capability_types=[ExaSearch])
|
||||
```
|
||||
|
||||
Pass `custom_capability_types` so the spec loader knows how to instantiate
|
||||
`ExaSearch`. The `client` field is not spec-serializable; spec-loaded instances
|
||||
always build the default client from `EXA_API_KEY`.
|
||||
|
||||
## Further reading
|
||||
|
||||
- [Pydantic AI capabilities](https://ai.pydantic.dev/capabilities/)
|
||||
|
||||
@@ -2,12 +2,22 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import dataclass, field
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from pydantic_ai.capabilities import AbstractCapability
|
||||
from pydantic_ai.tools import AgentDepsT
|
||||
|
||||
from pydantic_ai_harness.exa._toolset import ExaClient, ExaSearchToolset
|
||||
from pydantic_ai_harness.exa._toolset import (
|
||||
EXA_MAX_NUM_RESULTS,
|
||||
EXA_MAX_PAGE_TEXT_CHARS,
|
||||
ExaClient,
|
||||
ExaSearchToolset,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pydantic_ai._instructions import AgentInstructions
|
||||
|
||||
_INSTRUCTIONS = (
|
||||
'You have web research tools backed by the Exa search API. Start broad: use `web_search` '
|
||||
@@ -26,11 +36,11 @@ _DEEP_INSTRUCTIONS = _INSTRUCTIONS + (
|
||||
class ExaSearch(AbstractCapability[AgentDepsT]):
|
||||
"""Web research for agents, backed by the [Exa](https://exa.ai) search API.
|
||||
|
||||
Adds two tools: `web_search`, which returns search results together with
|
||||
page text, and `get_page`, which retrieves the text of a specific URL.
|
||||
Set `include_deep_search=True` to also expose `deep_search`, which runs
|
||||
Exa's multi-step deep search and returns a synthesized, cited answer in
|
||||
one tool call.
|
||||
Adds two tools: `web_search`, which returns search results with their most
|
||||
relevant excerpts, and `get_page`, which retrieves the full text of a
|
||||
specific URL. Set `include_deep_search=True` to also expose `deep_search`,
|
||||
which runs Exa's multi-step deep search and returns a synthesized, cited
|
||||
answer in one tool call.
|
||||
|
||||
```python
|
||||
from pydantic_ai import Agent
|
||||
@@ -44,13 +54,14 @@ class ExaSearch(AbstractCapability[AgentDepsT]):
|
||||
"""
|
||||
|
||||
num_results: int = 5
|
||||
"""Number of results `web_search` returns per query."""
|
||||
"""Number of results `web_search` returns per query (1 to 100, the Exa API range)."""
|
||||
|
||||
max_text_chars: int = 10_000
|
||||
"""Maximum characters of page text returned per result.
|
||||
"""Maximum characters of page text `get_page` returns (1 to 10,000, the Exa API range).
|
||||
|
||||
Sent to Exa as the contents cap and re-enforced when tool output is
|
||||
formatted, so page text stays bounded even with a custom `client`.
|
||||
One character of headroom above the cap is requested from Exa so local
|
||||
truncation can detect a longer page and append a truncation marker. At the
|
||||
API ceiling of 10,000 no headroom exists, so the marker cannot fire there.
|
||||
"""
|
||||
|
||||
include_deep_search: bool = False
|
||||
@@ -63,6 +74,27 @@ class ExaSearch(AbstractCapability[AgentDepsT]):
|
||||
than the default.
|
||||
"""
|
||||
|
||||
include_domains: Sequence[str] = field(default_factory=list[str])
|
||||
"""If non-empty, search results only come from these domains (allowlist).
|
||||
|
||||
Applies to `web_search` and `deep_search`. Mutually exclusive with
|
||||
`exclude_domains`.
|
||||
"""
|
||||
|
||||
exclude_domains: Sequence[str] = field(default_factory=list[str])
|
||||
"""Search results never come from these domains (denylist).
|
||||
|
||||
Applies to `web_search` and `deep_search`. Mutually exclusive with
|
||||
`include_domains`.
|
||||
"""
|
||||
|
||||
guidance: str | None = None
|
||||
"""Custom research guidance for the system prompt.
|
||||
|
||||
Leave as `None` for the default guidance (which adapts to
|
||||
`include_deep_search`), or set `''` to contribute no instructions at all.
|
||||
"""
|
||||
|
||||
client: ExaClient | None = None
|
||||
"""Exa client to use; when `None`, an `exa_py.AsyncExa` is built from `EXA_API_KEY`.
|
||||
|
||||
@@ -70,12 +102,26 @@ class ExaSearch(AbstractCapability[AgentDepsT]):
|
||||
key explicitly, point at a different base URL, or substitute a fake in tests.
|
||||
"""
|
||||
|
||||
def get_instructions(self) -> str:
|
||||
def __post_init__(self) -> None:
|
||||
"""Validate configuration against the Exa API's documented bounds."""
|
||||
if not 1 <= self.num_results <= EXA_MAX_NUM_RESULTS:
|
||||
raise ValueError(f'num_results must be between 1 and {EXA_MAX_NUM_RESULTS}, got {self.num_results}')
|
||||
if not 1 <= self.max_text_chars <= EXA_MAX_PAGE_TEXT_CHARS:
|
||||
raise ValueError(
|
||||
f'max_text_chars must be between 1 and {EXA_MAX_PAGE_TEXT_CHARS}, got {self.max_text_chars}'
|
||||
)
|
||||
if self.include_domains and self.exclude_domains:
|
||||
raise ValueError('Specify include_domains or exclude_domains, not both.')
|
||||
|
||||
def get_instructions(self) -> AgentInstructions[AgentDepsT] | None:
|
||||
"""Static research guidance: search wide, read the promising pages in full, cite URLs.
|
||||
|
||||
When `include_deep_search` is set, the guidance also covers when to
|
||||
escalate to `deep_search`.
|
||||
When `include_deep_search` is set, the default guidance also covers
|
||||
when to escalate to `deep_search`. A non-`None` `guidance` replaces the
|
||||
default; `''` disables instructions entirely.
|
||||
"""
|
||||
if self.guidance is not None:
|
||||
return self.guidance or None
|
||||
return _DEEP_INSTRUCTIONS if self.include_deep_search else _INSTRUCTIONS
|
||||
|
||||
def get_toolset(self) -> ExaSearchToolset[AgentDepsT]:
|
||||
@@ -85,4 +131,31 @@ class ExaSearch(AbstractCapability[AgentDepsT]):
|
||||
num_results=self.num_results,
|
||||
max_text_chars=self.max_text_chars,
|
||||
include_deep_search=self.include_deep_search,
|
||||
include_domains=self.include_domains,
|
||||
exclude_domains=self.exclude_domains,
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def from_spec(
|
||||
cls,
|
||||
*,
|
||||
num_results: int = 5,
|
||||
max_text_chars: int = 10_000,
|
||||
include_deep_search: bool = False,
|
||||
include_domains: Sequence[str] = (),
|
||||
exclude_domains: Sequence[str] = (),
|
||||
guidance: str | None = None,
|
||||
) -> ExaSearch[AgentDepsT]:
|
||||
"""Construct the capability from serializable spec options.
|
||||
|
||||
The `client` field is not spec-serializable, so spec-loaded instances
|
||||
always build the default `exa_py.AsyncExa` from `EXA_API_KEY`.
|
||||
"""
|
||||
return cls(
|
||||
num_results=num_results,
|
||||
max_text_chars=max_text_chars,
|
||||
include_deep_search=include_deep_search,
|
||||
include_domains=list(include_domains),
|
||||
exclude_domains=list(exclude_domains),
|
||||
guidance=guidance,
|
||||
)
|
||||
|
||||
@@ -2,9 +2,13 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import functools
|
||||
import json
|
||||
from typing import Literal, Protocol
|
||||
import re
|
||||
from collections.abc import Awaitable, Callable, Sequence
|
||||
from typing import Concatenate, Literal, ParamSpec, Protocol
|
||||
|
||||
import httpx
|
||||
from pydantic_ai.exceptions import ModelRetry, UserError
|
||||
from pydantic_ai.tools import AgentDepsT
|
||||
from pydantic_ai.toolsets import FunctionToolset
|
||||
@@ -24,6 +28,19 @@ except ImportError as _import_error: # pragma: no cover
|
||||
'exa-py is required for ExaSearch. Install it with: pip install "pydantic-ai-harness[exa]"'
|
||||
) from _import_error
|
||||
|
||||
EXA_MAX_NUM_RESULTS = 100
|
||||
"""Largest `num_results` the Exa search API accepts."""
|
||||
|
||||
EXA_MAX_PAGE_TEXT_CHARS = 10_000
|
||||
"""Largest per-page text budget the Exa contents API accepts."""
|
||||
|
||||
_P = ParamSpec('_P')
|
||||
|
||||
# exa-py raises a bare ValueError for any non-2xx response, embedding the HTTP
|
||||
# status in the message. 401/403 mean a bad or missing API key -- configuration
|
||||
# the model cannot correct, so those propagate instead of retrying.
|
||||
_AUTH_STATUS_RE = re.compile(r'status code (401|403)\b')
|
||||
|
||||
|
||||
class ExaClient(Protocol):
|
||||
"""The subset of the `exa_py.AsyncExa` API that `ExaSearchToolset` calls.
|
||||
@@ -45,6 +62,8 @@ class ExaClient(Protocol):
|
||||
num_results: int | None = None,
|
||||
type: SearchType | None = None,
|
||||
output_schema: DeepOutputSchema | None = None,
|
||||
include_domains: list[str] | None = None,
|
||||
exclude_domains: list[str] | None = None,
|
||||
) -> SearchResponse[Result]:
|
||||
"""Search the web and return results, optionally with page contents or a synthesized output."""
|
||||
... # pragma: no cover
|
||||
@@ -65,62 +84,104 @@ def _default_client() -> ExaClient:
|
||||
) from error
|
||||
|
||||
|
||||
def _recoverable(
|
||||
fn: Callable[Concatenate[ExaSearchToolset, _P], Awaitable[str]],
|
||||
) -> Callable[Concatenate[ExaSearchToolset, _P], Awaitable[str]]:
|
||||
"""Convert transient Exa API failures into `ModelRetry`.
|
||||
|
||||
pyai only feeds `ModelRetry` back to the model as a retry prompt; any other
|
||||
exception propagates and aborts the whole run. exa-py surfaces every non-2xx
|
||||
response as a bare `ValueError` (message embeds the status code) and network
|
||||
failures as `httpx.HTTPError`. Rate limits, transient 5xx, and rejected
|
||||
parameters are things a model can recover from (wait, rephrase, adjust), so
|
||||
they become retries; 401/403 auth failures are configuration errors and
|
||||
propagate.
|
||||
"""
|
||||
|
||||
@functools.wraps(fn)
|
||||
async def wrapper(self: ExaSearchToolset, *args: _P.args, **kwargs: _P.kwargs) -> str:
|
||||
try:
|
||||
return await fn(self, *args, **kwargs)
|
||||
except httpx.HTTPError as error:
|
||||
raise ModelRetry(f'Exa request failed: {error}') from error
|
||||
except ValueError as error:
|
||||
if _AUTH_STATUS_RE.search(str(error)):
|
||||
raise
|
||||
raise ModelRetry(f'Exa request failed: {error}') from error
|
||||
|
||||
return wrapper
|
||||
|
||||
|
||||
class ExaSearchToolset(FunctionToolset[AgentDepsT]):
|
||||
"""Gives an agent web research tools backed by the Exa search API.
|
||||
|
||||
`web_search` surveys the web and returns results together with page text,
|
||||
and `get_page` retrieves the text of one specific URL. With
|
||||
`web_search` surveys the web and returns results with their most relevant
|
||||
excerpts, and `get_page` retrieves the text of one specific URL. With
|
||||
`include_deep_search=True`, `deep_search` runs Exa's multi-step deep search
|
||||
and returns a synthesized, cited answer in one call.
|
||||
|
||||
Page text is capped at `max_text_chars` characters per result: the cap is
|
||||
sent to Exa as the contents limit and re-enforced when output is formatted,
|
||||
so text stays bounded even with a custom `client`. The result count is
|
||||
bounded the same way: `num_results` is requested from Exa and re-applied to
|
||||
the response.
|
||||
`get_page` text is capped at `max_text_chars` characters; one character of
|
||||
headroom above the cap is requested from Exa so local truncation can detect
|
||||
a longer page and append a marker (at the API ceiling of
|
||||
`EXA_MAX_PAGE_TEXT_CHARS` no headroom exists, so the marker cannot fire).
|
||||
The `web_search` result count is bounded the same way: `num_results` is
|
||||
requested from Exa and re-applied to the response. Bounds are validated by
|
||||
`ExaSearch` at construction.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self, *, client: ExaClient | None, num_results: int, max_text_chars: int, include_deep_search: bool
|
||||
self,
|
||||
*,
|
||||
client: ExaClient | None,
|
||||
num_results: int,
|
||||
max_text_chars: int,
|
||||
include_deep_search: bool,
|
||||
include_domains: Sequence[str] = (),
|
||||
exclude_domains: Sequence[str] = (),
|
||||
) -> None:
|
||||
if num_results < 1:
|
||||
raise ValueError(f'num_results must be at least 1, got {num_results}')
|
||||
if max_text_chars < 1:
|
||||
raise ValueError(f'max_text_chars must be at least 1, got {max_text_chars}')
|
||||
super().__init__()
|
||||
self._client = client if client is not None else _default_client()
|
||||
self._num_results = num_results
|
||||
self._max_text_chars = max_text_chars
|
||||
self._include_domains = list(include_domains) if include_domains else None
|
||||
self._exclude_domains = list(exclude_domains) if exclude_domains else None
|
||||
self.add_function(self.web_search, name='web_search')
|
||||
self.add_function(self.get_page, name='get_page')
|
||||
if include_deep_search:
|
||||
self.add_function(self.deep_search, name='deep_search')
|
||||
|
||||
@_recoverable
|
||||
async def web_search(self, query: str) -> str:
|
||||
"""Search the web and return matching pages, each with its text content.
|
||||
"""Search the web and return matching pages, each with its most relevant excerpts.
|
||||
|
||||
Args:
|
||||
query: The search query. Natural-language questions and keyword
|
||||
queries both work.
|
||||
|
||||
Returns:
|
||||
The matching pages, each with title, URL, and page text.
|
||||
The matching pages, each with title, URL, and excerpts.
|
||||
"""
|
||||
response = await self._client.search(
|
||||
query,
|
||||
contents={'text': {'max_characters': self._max_text_chars}},
|
||||
contents={'highlights': True},
|
||||
num_results=self._num_results,
|
||||
include_domains=self._include_domains,
|
||||
exclude_domains=self._exclude_domains,
|
||||
)
|
||||
if not response.results:
|
||||
return f'No results found for {query!r}.'
|
||||
results = response.results[: self._num_results]
|
||||
sections = [_format_result(result, self._max_text_chars) for result in results]
|
||||
sections = [_format_result(result, _highlights_body(result)) for result in results]
|
||||
plural = 's' if len(sections) != 1 else ''
|
||||
joined = '\n\n---\n\n'.join(sections)
|
||||
return f'Found {len(sections)} result{plural} for {query!r}:\n\n{joined}'
|
||||
|
||||
@_recoverable
|
||||
async def get_page(self, url: str) -> str:
|
||||
"""Retrieve the text contents of a specific URL.
|
||||
"""Retrieve the full text of a specific URL.
|
||||
|
||||
Use it to read a promising URL from `web_search` results in full, or a
|
||||
URL the user provided.
|
||||
|
||||
Args:
|
||||
url: The URL of the page to read.
|
||||
@@ -128,11 +189,14 @@ class ExaSearchToolset(FunctionToolset[AgentDepsT]):
|
||||
Returns:
|
||||
The page's title, URL, and text content.
|
||||
"""
|
||||
response = await self._client.get_contents(url, text={'max_characters': self._max_text_chars})
|
||||
if not response.results or not response.results[0].text:
|
||||
requested = min(self._max_text_chars + 1, EXA_MAX_PAGE_TEXT_CHARS)
|
||||
response = await self._client.get_contents(url, text={'max_characters': requested})
|
||||
first = response.results[0] if response.results else None
|
||||
if first is None or not first.text:
|
||||
raise ModelRetry(f'No content could be retrieved for {url!r}. Check the URL or try another page.')
|
||||
return _format_result(response.results[0], self._max_text_chars)
|
||||
return _format_result(first, _truncate(first.text, self._max_text_chars))
|
||||
|
||||
@_recoverable
|
||||
async def deep_search(self, question: str) -> str:
|
||||
"""Run Exa's multi-step deep search and return a synthesized answer with its sources.
|
||||
|
||||
@@ -150,6 +214,8 @@ class ExaSearchToolset(FunctionToolset[AgentDepsT]):
|
||||
contents=False,
|
||||
type='deep',
|
||||
output_schema={'type': 'text'},
|
||||
include_domains=self._include_domains,
|
||||
exclude_domains=self._exclude_domains,
|
||||
)
|
||||
output = response.output
|
||||
if output is None or not output.content:
|
||||
@@ -161,25 +227,32 @@ class ExaSearchToolset(FunctionToolset[AgentDepsT]):
|
||||
sources = {citation.url: citation.title for row in output.grounding for citation in row.citations}
|
||||
if not sources:
|
||||
sources = {result.url: result.title or '' for result in response.results}
|
||||
lines = [_truncate(answer, self._max_text_chars)]
|
||||
lines = [answer]
|
||||
if sources:
|
||||
lines.extend(['', 'Sources:'])
|
||||
lines.extend(f'- {title or "(untitled)"}: {url}' for url, title in sources.items())
|
||||
return '\n'.join(lines)
|
||||
|
||||
|
||||
def _format_result(result: Result, max_text_chars: int) -> str:
|
||||
"""Render one result as labelled metadata lines followed by the page text."""
|
||||
def _format_result(result: Result, body: str | None) -> str:
|
||||
"""Render one result as labelled metadata lines followed by its body text."""
|
||||
lines = [f'Title: {result.title or "(untitled)"}', f'URL: {result.url}']
|
||||
if result.published_date:
|
||||
lines.append(f'Published: {result.published_date}')
|
||||
if result.author:
|
||||
lines.append(f'Author: {result.author}')
|
||||
if result.text:
|
||||
lines.extend(['', _truncate(result.text, max_text_chars)])
|
||||
if body:
|
||||
lines.extend(['', body])
|
||||
return '\n'.join(lines)
|
||||
|
||||
|
||||
def _highlights_body(result: Result) -> str | None:
|
||||
"""Render a result's excerpts as a bullet list, or `None` when there are none."""
|
||||
if not result.highlights:
|
||||
return None
|
||||
return '\n'.join(f'- {highlight}' for highlight in result.highlights)
|
||||
|
||||
|
||||
def _truncate(text: str, max_chars: int) -> str:
|
||||
"""Cap page text at `max_chars`, keeping the head.
|
||||
|
||||
|
||||
+151
-40
@@ -2,10 +2,13 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Literal
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from exa_py.api import (
|
||||
ContentsOptions,
|
||||
@@ -19,6 +22,7 @@ from exa_py.api import (
|
||||
TextContentsOptions,
|
||||
)
|
||||
from pydantic_ai import Agent
|
||||
from pydantic_ai.agent.spec import AgentSpec
|
||||
from pydantic_ai.exceptions import ModelRetry, UserError
|
||||
from pydantic_ai.messages import ModelRequest, ModelResponse, ToolCallPart, ToolReturnPart
|
||||
from pydantic_ai.models.test import TestModel
|
||||
@@ -36,12 +40,21 @@ def _result(
|
||||
url: str = 'https://example.dev/page',
|
||||
*,
|
||||
title: str | None = 'Example page',
|
||||
text: str | None = 'Example page text.',
|
||||
text: str | None = None,
|
||||
highlights: list[str] | None = None,
|
||||
author: str | None = None,
|
||||
published_date: str | None = None,
|
||||
) -> Result:
|
||||
"""A real `exa_py` result with the fields the toolset reads."""
|
||||
return Result(url=url, id=url, title=title, text=text, author=author, published_date=published_date)
|
||||
return Result(
|
||||
url=url,
|
||||
id=url,
|
||||
title=title,
|
||||
text=text,
|
||||
highlights=highlights,
|
||||
author=author,
|
||||
published_date=published_date,
|
||||
)
|
||||
|
||||
|
||||
def _response(*results: Result) -> SearchResponse[Result]:
|
||||
@@ -82,6 +95,7 @@ class _FakeExaClient:
|
||||
search_response: SearchResponse[Result] = field(default_factory=_response)
|
||||
contents_response: SearchResponse[Result] = field(default_factory=_response)
|
||||
deep_response: SearchResponse[Result] = field(default_factory=_response)
|
||||
error: Exception | None = None
|
||||
search_calls: list[dict[str, object]] = field(default_factory=list[dict[str, object]])
|
||||
get_contents_calls: list[tuple[str, TextContentsOptions]] = field(
|
||||
default_factory=list[tuple[str, TextContentsOptions]]
|
||||
@@ -95,7 +109,11 @@ class _FakeExaClient:
|
||||
num_results: int | None = None,
|
||||
type: SearchType | None = None,
|
||||
output_schema: DeepOutputSchema | None = None,
|
||||
include_domains: list[str] | None = None,
|
||||
exclude_domains: list[str] | None = None,
|
||||
) -> SearchResponse[Result]:
|
||||
if self.error is not None:
|
||||
raise self.error
|
||||
self.search_calls.append(
|
||||
{
|
||||
'query': query,
|
||||
@@ -103,11 +121,15 @@ class _FakeExaClient:
|
||||
'num_results': num_results,
|
||||
'type': type,
|
||||
'output_schema': output_schema,
|
||||
'include_domains': include_domains,
|
||||
'exclude_domains': exclude_domains,
|
||||
}
|
||||
)
|
||||
return self.deep_response if type is not None else self.search_response
|
||||
|
||||
async def get_contents(self, urls: str, *, text: TextContentsOptions) -> SearchResponse[Result]:
|
||||
if self.error is not None:
|
||||
raise self.error
|
||||
self.get_contents_calls.append((urls, text))
|
||||
return self.contents_response
|
||||
|
||||
@@ -118,21 +140,31 @@ def _toolset(
|
||||
num_results: int = 5,
|
||||
max_text_chars: int = 10_000,
|
||||
include_deep_search: bool = False,
|
||||
include_domains: Sequence[str] = (),
|
||||
exclude_domains: Sequence[str] = (),
|
||||
) -> ExaSearchToolset[None]:
|
||||
return ExaSearch[None](
|
||||
num_results=num_results,
|
||||
max_text_chars=max_text_chars,
|
||||
include_deep_search=include_deep_search,
|
||||
include_domains=include_domains,
|
||||
exclude_domains=exclude_domains,
|
||||
client=client,
|
||||
).get_toolset()
|
||||
|
||||
|
||||
class TestWebSearch:
|
||||
async def test_formats_results_with_metadata(self) -> None:
|
||||
async def test_formats_results_with_excerpts_and_metadata(self) -> None:
|
||||
client = _FakeExaClient(
|
||||
search_response=_response(
|
||||
_result('https://a.dev', title='A', text='alpha text', author='Ada', published_date='2026-07-01'),
|
||||
_result('https://b.dev', title=None, text=None),
|
||||
_result(
|
||||
'https://a.dev',
|
||||
title='A',
|
||||
highlights=['alpha excerpt', 'another excerpt'],
|
||||
author='Ada',
|
||||
published_date='2026-07-01',
|
||||
),
|
||||
_result('https://b.dev', title=None),
|
||||
)
|
||||
)
|
||||
output = await _toolset(client).web_search('rust web frameworks')
|
||||
@@ -143,25 +175,35 @@ class TestWebSearch:
|
||||
'Published: 2026-07-01\n'
|
||||
'Author: Ada\n'
|
||||
'\n'
|
||||
'alpha text'
|
||||
'- alpha excerpt\n'
|
||||
'- another excerpt'
|
||||
'\n\n---\n\n'
|
||||
'Title: (untitled)\n'
|
||||
'URL: https://b.dev'
|
||||
)
|
||||
|
||||
async def test_passes_num_results_and_contents_cap_to_client(self) -> None:
|
||||
async def test_requests_highlights_num_results_and_domains(self) -> None:
|
||||
client = _FakeExaClient(search_response=_response(_result()))
|
||||
await _toolset(client, num_results=3, max_text_chars=42).web_search('q')
|
||||
toolset = _toolset(client, num_results=3, include_domains=['a.dev', 'b.dev'])
|
||||
await toolset.web_search('q')
|
||||
assert client.search_calls == [
|
||||
{
|
||||
'query': 'q',
|
||||
'contents': {'text': {'max_characters': 42}},
|
||||
'contents': {'highlights': True},
|
||||
'num_results': 3,
|
||||
'type': None,
|
||||
'output_schema': None,
|
||||
'include_domains': ['a.dev', 'b.dev'],
|
||||
'exclude_domains': None,
|
||||
}
|
||||
]
|
||||
|
||||
async def test_exclude_domains_plumbed(self) -> None:
|
||||
client = _FakeExaClient(search_response=_response(_result()))
|
||||
await _toolset(client, exclude_domains=['spam.dev']).web_search('q')
|
||||
assert client.search_calls[0]['include_domains'] is None
|
||||
assert client.search_calls[0]['exclude_domains'] == ['spam.dev']
|
||||
|
||||
async def test_no_results(self) -> None:
|
||||
client = _FakeExaClient()
|
||||
output = await _toolset(client).web_search('nothing to see')
|
||||
@@ -170,9 +212,9 @@ class TestWebSearch:
|
||||
async def test_num_results_enforced_on_oversized_response(self) -> None:
|
||||
client = _FakeExaClient(
|
||||
search_response=_response(
|
||||
_result('https://a.dev', title='A', text=None),
|
||||
_result('https://b.dev', title='B', text=None),
|
||||
_result('https://c.dev', title='C', text=None),
|
||||
_result('https://a.dev', title='A'),
|
||||
_result('https://b.dev', title='B'),
|
||||
_result('https://c.dev', title='C'),
|
||||
)
|
||||
)
|
||||
output = await _toolset(client, num_results=2).web_search('q')
|
||||
@@ -180,27 +222,31 @@ class TestWebSearch:
|
||||
"Found 2 results for 'q':\n\nTitle: A\nURL: https://a.dev\n\n---\n\nTitle: B\nURL: https://b.dev"
|
||||
)
|
||||
|
||||
async def test_text_at_cap_is_not_truncated(self) -> None:
|
||||
client = _FakeExaClient(search_response=_response(_result(title='T', text='x' * 10)))
|
||||
output = await _toolset(client, max_text_chars=10).web_search('q')
|
||||
assert output == f"Found 1 result for 'q':\n\nTitle: T\nURL: https://example.dev/page\n\n{'x' * 10}"
|
||||
|
||||
async def test_text_over_cap_is_truncated_keeping_the_head(self) -> None:
|
||||
client = _FakeExaClient(search_response=_response(_result(title='T', text='x' * 10 + 'TAIL')))
|
||||
output = await _toolset(client, max_text_chars=10).web_search('q')
|
||||
assert output == (
|
||||
f"Found 1 result for 'q':\n\nTitle: T\nURL: https://example.dev/page\n\n{'x' * 10}\n"
|
||||
'[... page text truncated at 10 characters]'
|
||||
)
|
||||
|
||||
|
||||
class TestGetPage:
|
||||
async def test_returns_page_text(self) -> None:
|
||||
async def test_returns_page_text_with_headroom_request(self) -> None:
|
||||
client = _FakeExaClient(contents_response=_response(_result('https://a.dev', title='A', text='alpha text')))
|
||||
output = await _toolset(client).get_page('https://a.dev')
|
||||
assert client.get_contents_calls == [('https://a.dev', {'max_characters': 10_000})]
|
||||
output = await _toolset(client, max_text_chars=100).get_page('https://a.dev')
|
||||
assert client.get_contents_calls == [('https://a.dev', {'max_characters': 101})]
|
||||
assert output == 'Title: A\nURL: https://a.dev\n\nalpha text'
|
||||
|
||||
async def test_headroom_request_stops_at_api_ceiling(self) -> None:
|
||||
client = _FakeExaClient(contents_response=_response(_result(text='alpha')))
|
||||
await _toolset(client, max_text_chars=10_000).get_page('https://a.dev')
|
||||
assert client.get_contents_calls == [('https://a.dev', {'max_characters': 10_000})]
|
||||
|
||||
async def test_text_at_cap_is_not_truncated(self) -> None:
|
||||
client = _FakeExaClient(contents_response=_response(_result(title='T', text='x' * 10)))
|
||||
output = await _toolset(client, max_text_chars=10).get_page('https://example.dev/page')
|
||||
assert output == f'Title: T\nURL: https://example.dev/page\n\n{"x" * 10}'
|
||||
|
||||
async def test_text_over_cap_is_truncated_keeping_the_head(self) -> None:
|
||||
client = _FakeExaClient(contents_response=_response(_result(title='T', text='x' * 10 + 'T')))
|
||||
output = await _toolset(client, max_text_chars=10).get_page('https://example.dev/page')
|
||||
assert output == (
|
||||
f'Title: T\nURL: https://example.dev/page\n\n{"x" * 10}\n[... page text truncated at 10 characters]'
|
||||
)
|
||||
|
||||
async def test_no_results_raises_model_retry(self) -> None:
|
||||
client = _FakeExaClient()
|
||||
with pytest.raises(ModelRetry, match='No content could be retrieved'):
|
||||
@@ -212,6 +258,23 @@ class TestGetPage:
|
||||
await _toolset(client).get_page('https://a.dev')
|
||||
|
||||
|
||||
class TestRecoverableErrors:
|
||||
async def test_non_2xx_becomes_model_retry(self) -> None:
|
||||
client = _FakeExaClient(error=ValueError('Request failed with status code 429: rate limited'))
|
||||
with pytest.raises(ModelRetry, match='Exa request failed: .*429'):
|
||||
await _toolset(client).web_search('q')
|
||||
|
||||
async def test_auth_failure_propagates(self) -> None:
|
||||
client = _FakeExaClient(error=ValueError('Request failed with status code 401: invalid API key'))
|
||||
with pytest.raises(ValueError, match='status code 401'):
|
||||
await _toolset(client).web_search('q')
|
||||
|
||||
async def test_network_failure_becomes_model_retry(self) -> None:
|
||||
client = _FakeExaClient(error=httpx.ConnectError('connection refused'))
|
||||
with pytest.raises(ModelRetry, match='Exa request failed: connection refused'):
|
||||
await _toolset(client).get_page('https://a.dev')
|
||||
|
||||
|
||||
class TestDeepSearch:
|
||||
async def test_returns_answer_with_deduplicated_sources(self) -> None:
|
||||
client = _FakeExaClient(
|
||||
@@ -228,19 +291,25 @@ class TestDeepSearch:
|
||||
'num_results': None,
|
||||
'type': 'deep',
|
||||
'output_schema': {'type': 'text'},
|
||||
'include_domains': None,
|
||||
'exclude_domains': None,
|
||||
}
|
||||
]
|
||||
assert output == 'Deep answer.\n\nSources:\n- A: https://a.dev\n- (untitled): https://b.dev'
|
||||
|
||||
async def test_domains_plumbed(self) -> None:
|
||||
client = _FakeExaClient(deep_response=_deep_response('Answer.'))
|
||||
toolset = _toolset(client, include_deep_search=True, include_domains=['a.dev'])
|
||||
await toolset.deep_search('q')
|
||||
assert client.search_calls[0]['include_domains'] == ['a.dev']
|
||||
|
||||
async def test_structured_content_is_rendered_as_json(self) -> None:
|
||||
client = _FakeExaClient(deep_response=_deep_response({'finding': 'x'}, citations=[('https://a.dev', 'A')]))
|
||||
output = await _toolset(client, include_deep_search=True).deep_search('q')
|
||||
assert output == '{"finding": "x"}\n\nSources:\n- A: https://a.dev'
|
||||
|
||||
async def test_without_grounding_falls_back_to_result_sources(self) -> None:
|
||||
client = _FakeExaClient(
|
||||
deep_response=_deep_response('Answer.', results=[_result('https://a.dev', title='A', text=None)])
|
||||
)
|
||||
client = _FakeExaClient(deep_response=_deep_response('Answer.', results=[_result('https://a.dev', title='A')]))
|
||||
output = await _toolset(client, include_deep_search=True).deep_search('q')
|
||||
assert output == 'Answer.\n\nSources:\n- A: https://a.dev'
|
||||
|
||||
@@ -249,6 +318,12 @@ class TestDeepSearch:
|
||||
output = await _toolset(client, include_deep_search=True).deep_search('q')
|
||||
assert output == 'Answer.'
|
||||
|
||||
async def test_answer_is_not_capped_by_max_text_chars(self) -> None:
|
||||
long_answer = 'a' * 50
|
||||
client = _FakeExaClient(deep_response=_deep_response(long_answer))
|
||||
output = await _toolset(client, include_deep_search=True, max_text_chars=10).deep_search('q')
|
||||
assert output == long_answer
|
||||
|
||||
async def test_missing_output_raises_model_retry(self) -> None:
|
||||
client = _FakeExaClient(deep_response=_response())
|
||||
with pytest.raises(ModelRetry, match='Deep search returned no answer'):
|
||||
@@ -276,29 +351,45 @@ class TestExaSearch:
|
||||
toolset = ExaSearch[None]().get_toolset()
|
||||
assert isinstance(toolset, ExaSearchToolset)
|
||||
|
||||
def test_non_positive_num_results_rejected(self) -> None:
|
||||
with pytest.raises(ValueError, match='num_results must be at least 1, got 0'):
|
||||
ExaSearch[None](num_results=0, client=_FakeExaClient()).get_toolset()
|
||||
@pytest.mark.parametrize('num_results', [0, 101])
|
||||
def test_num_results_out_of_bounds_rejected(self, num_results: int) -> None:
|
||||
with pytest.raises(ValueError, match=f'num_results must be between 1 and 100, got {num_results}'):
|
||||
ExaSearch[None](num_results=num_results)
|
||||
|
||||
def test_non_positive_max_text_chars_rejected(self) -> None:
|
||||
with pytest.raises(ValueError, match='max_text_chars must be at least 1, got -1'):
|
||||
ExaSearch[None](max_text_chars=-1, client=_FakeExaClient()).get_toolset()
|
||||
@pytest.mark.parametrize('max_text_chars', [0, 10_001])
|
||||
def test_max_text_chars_out_of_bounds_rejected(self, max_text_chars: int) -> None:
|
||||
with pytest.raises(ValueError, match=f'max_text_chars must be between 1 and 10000, got {max_text_chars}'):
|
||||
ExaSearch[None](max_text_chars=max_text_chars)
|
||||
|
||||
def test_include_and_exclude_domains_are_mutually_exclusive(self) -> None:
|
||||
with pytest.raises(ValueError, match='include_domains or exclude_domains, not both'):
|
||||
ExaSearch[None](include_domains=['a.dev'], exclude_domains=['b.dev'])
|
||||
|
||||
def test_instructions_reference_the_tools(self) -> None:
|
||||
instructions = ExaSearch[None]().get_instructions()
|
||||
assert isinstance(instructions, str)
|
||||
assert 'web_search' in instructions
|
||||
assert 'get_page' in instructions
|
||||
assert 'cite' in instructions
|
||||
assert 'deep_search' not in instructions
|
||||
|
||||
def test_instructions_cover_deep_search_when_enabled(self) -> None:
|
||||
base = ExaSearch[None]().get_instructions()
|
||||
instructions = ExaSearch[None](include_deep_search=True).get_instructions()
|
||||
assert instructions.startswith(ExaSearch[None]().get_instructions())
|
||||
assert isinstance(base, str) and isinstance(instructions, str)
|
||||
assert instructions.startswith(base)
|
||||
assert 'escalate to `deep_search`' in instructions
|
||||
|
||||
def test_custom_guidance_replaces_default(self) -> None:
|
||||
capability = ExaSearch[None](include_deep_search=True, guidance='Research with the Exa tools.')
|
||||
assert capability.get_instructions() == 'Research with the Exa tools.'
|
||||
|
||||
def test_empty_guidance_disables_instructions(self) -> None:
|
||||
assert ExaSearch[None](guidance='').get_instructions() is None
|
||||
|
||||
async def test_agent_run_uses_all_tools_and_instructions(self) -> None:
|
||||
client = _FakeExaClient(
|
||||
search_response=_response(_result('https://a.dev', title='A', text='alpha')),
|
||||
search_response=_response(_result('https://a.dev', title='A', highlights=['alpha'])),
|
||||
contents_response=_response(_result('https://b.dev', title='B', text='beta')),
|
||||
deep_response=_deep_response('Deep answer.', citations=[('https://c.dev', 'C')]),
|
||||
)
|
||||
@@ -329,6 +420,26 @@ class TestExaSearch:
|
||||
}
|
||||
assert set(returns) == {'web_search', 'get_page', 'deep_search'}
|
||||
query = calls['web_search']['query']
|
||||
assert returns['web_search'] == f'Found 1 result for {query!r}:\n\nTitle: A\nURL: https://a.dev\n\nalpha'
|
||||
assert returns['web_search'] == f'Found 1 result for {query!r}:\n\nTitle: A\nURL: https://a.dev\n\n- alpha'
|
||||
assert returns['get_page'] == 'Title: B\nURL: https://b.dev\n\nbeta'
|
||||
assert returns['deep_search'] == 'Deep answer.\n\nSources:\n- C: https://c.dev'
|
||||
|
||||
|
||||
class TestAgentSpec:
|
||||
def test_spec_schema_includes_exa_search(self) -> None:
|
||||
schema = AgentSpec.model_json_schema_with_capabilities([ExaSearch])
|
||||
assert 'ExaSearch' in json.dumps(schema)
|
||||
|
||||
def test_from_spec_builds_capability(self) -> None:
|
||||
capability = ExaSearch[None].from_spec(num_results=3, include_deep_search=True, include_domains=['a.dev'])
|
||||
assert capability.num_results == 3
|
||||
assert capability.include_deep_search is True
|
||||
assert capability.include_domains == ['a.dev']
|
||||
assert capability.client is None
|
||||
|
||||
def test_agent_loads_from_spec_file(self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
|
||||
monkeypatch.setenv('EXA_API_KEY', 'test-key')
|
||||
spec = tmp_path / 'agent.yaml'
|
||||
spec.write_text('model: test\ncapabilities:\n - ExaSearch:\n num_results: 3\n')
|
||||
agent = Agent.from_file(spec, custom_capability_types=[ExaSearch])
|
||||
assert isinstance(agent, Agent)
|
||||
|
||||
Reference in New Issue
Block a user