mirror of
https://github.com/pydantic/pydantic-ai-harness.git
synced 2026-07-20 10:25:35 +00:00
feat: add FileSystem and Shell capabilities (#260)
* feat: add FileSystem and Shell capabilities with exhaustive testing - FileSystemToolset: 8 tools (read, write, edit, list, search, find, mkdir, info) with path-traversal prevention, allow/deny patterns, optimistic concurrency - ShellToolset: 1 tool (run_command) with command validation, timeout handling, and async subprocess execution via anyio --- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: David Sanchez <64162682+dsfaccini@users.noreply.github.com>
This commit is contained in:
Bill Easton
co-authored by
Claude
David Sanchez
Claude
David Sanchez
parent
ad1a2014ef
commit
e1654ccfe2
@@ -25,3 +25,4 @@ wheels/
|
||||
# Hypothesis
|
||||
.hypothesis/
|
||||
.vscode/
|
||||
mutants/
|
||||
|
||||
@@ -103,8 +103,8 @@ We studied leading coding agents, agent frameworks, and Claw-style assistants to
|
||||
|---|---|---|---|---|
|
||||
| **Tools & execution** | **Code mode** | Sandboxed Python execution via [Monty](https://github.com/pydantic/monty) -- one `run_code` call replaces N tool calls | :white_check_mark: [Docs](pydantic_ai_harness/code_mode/) | |
|
||||
| | **Tool search** | Progressive tool discovery for large tool sets | :white_check_mark: [Pydantic AI](https://pydantic.dev/docs/ai/tools-toolsets/toolsets/#deferred-loading) | |
|
||||
| | **File system** | Read, write, edit, search files with path traversal prevention | :construction: [PR #177](https://github.com/pydantic/pydantic-ai-harness/pull/177) | [pydantic-ai-backend](https://github.com/vstorm-co/pydantic-ai-backend) (vstorm‑co) |
|
||||
| | **Shell** | Execute commands with allowlists, denylists, and timeouts | :construction: [PR #177](https://github.com/pydantic/pydantic-ai-harness/pull/177) | [pydantic-ai-backend](https://github.com/vstorm-co/pydantic-ai-backend) (vstorm‑co) |
|
||||
| | **File system** | Read, write, edit, search files with path traversal prevention | :white_check_mark: [Docs](pydantic_ai_harness/filesystem/) | [pydantic-ai-backend](https://github.com/vstorm-co/pydantic-ai-backend) (vstorm‑co) |
|
||||
| | **Shell** | Execute commands with allowlists, denylists, and timeouts | :white_check_mark: [Docs](pydantic_ai_harness/shell/) | [pydantic-ai-backend](https://github.com/vstorm-co/pydantic-ai-backend) (vstorm‑co) |
|
||||
| | **Repo context injection** | Auto-load CLAUDE.md/AGENTS.md and repo structure | :construction: [PR #175](https://github.com/pydantic/pydantic-ai-harness/pull/175) | [pydantic-deep](https://github.com/vstorm-co/pydantic-deepagents) (vstorm‑co) |
|
||||
| | **Verification loop** | Run tests after edits, auto-fix failures | :construction: [PR #169](https://github.com/pydantic/pydantic-ai-harness/pull/169) | |
|
||||
| **Context management** | **Sliding window** | Trim conversation history to stay within token limits | :construction: [PR #191](https://github.com/pydantic/pydantic-ai-harness/pull/191) | [summarization-pydantic-ai](https://github.com/vstorm-co/summarization-pydantic-ai) (vstorm‑co) |
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
# Mutation Testing
|
||||
|
||||
Mutation testing complements the 100% branch-coverage requirement: coverage
|
||||
proves every line and branch runs, mutation testing proves the assertions
|
||||
actually pin the behavior down.
|
||||
|
||||
Covers `pydantic_ai_harness/filesystem/_toolset.py` and
|
||||
`pydantic_ai_harness/shell/_toolset.py`.
|
||||
|
||||
Run with [mutmut](https://mutmut.readthedocs.io/) v3 via `scripts/run-mutmut.sh`,
|
||||
which installs mutmut ephemerally with `uv run --with` — no dev dependency
|
||||
required.
|
||||
|
||||
```bash
|
||||
scripts/run-mutmut.sh run --max-children 1
|
||||
scripts/run-mutmut.sh results
|
||||
scripts/run-mutmut.sh show <mutant-name>
|
||||
```
|
||||
|
||||
## Interpreting survivors
|
||||
|
||||
A surviving mutant is either a missing test or an equivalent mutant — a change
|
||||
that produces behavior no test could distinguish from the original. Triage each
|
||||
survivor; the recurring equivalent-mutant categories in this codebase are:
|
||||
|
||||
- **Trampoline default params** — mutmut v3 wraps functions, and the wrapper
|
||||
keeps the original defaults, so a mutated default is never observed.
|
||||
- **Omitted `name=` in `add_function()`** — pydantic-ai falls back to
|
||||
`method.__name__`, which equals the explicit name being mutated away.
|
||||
- **`'utf-8'` encoding mutations** — Python's codec lookup is case-insensitive
|
||||
and UTF-8 is the default text encoding, so case/omission changes are no-ops.
|
||||
- **`errors='replace'` mutations** — exercised only by invalid bytes; valid
|
||||
UTF-8 test data never invokes the error handler.
|
||||
- **Unreachable `except` blocks** (marked `pragma: no cover`) — paths that
|
||||
can't be triggered in the test environment.
|
||||
- **`CancelScope(shield=True)` flips** — require an outer cancellation during
|
||||
the near-instant cleanup window.
|
||||
|
||||
Anything outside these categories should be treated as a real gap and killed
|
||||
with a new test.
|
||||
|
||||
## Limitations
|
||||
|
||||
Trio-parametrized tests are excluded during mutation testing (`-k 'not trio'`
|
||||
in `pyproject.toml [tool.mutmut]`) because trio segfaults in mutmut's
|
||||
subprocess environment on Python 3.14 / macOS. The kill rate is unaffected —
|
||||
the trio tests exercise the same code paths as the asyncio tests.
|
||||
@@ -1,11 +1,13 @@
|
||||
"""The batteries for your Pydantic AI agent -- the official capability library."""
|
||||
"""Pydantic AI capability library."""
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from .code_mode import CodeMode
|
||||
from .filesystem import FileSystem
|
||||
from .shell import Shell
|
||||
|
||||
__all__ = ['CodeMode']
|
||||
__all__ = ['CodeMode', 'FileSystem', 'Shell']
|
||||
|
||||
|
||||
def __getattr__(name: str) -> object:
|
||||
@@ -13,4 +15,12 @@ def __getattr__(name: str) -> object:
|
||||
from .code_mode import CodeMode
|
||||
|
||||
return CodeMode
|
||||
elif name == 'FileSystem':
|
||||
from .filesystem import FileSystem
|
||||
|
||||
return FileSystem
|
||||
elif name == 'Shell':
|
||||
from .shell import Shell
|
||||
|
||||
return Shell
|
||||
raise AttributeError(f'module {__name__!r} has no attribute {name!r}')
|
||||
|
||||
@@ -0,0 +1,136 @@
|
||||
# FileSystem
|
||||
|
||||
Give an agent sandboxed, pattern-filtered access to a directory tree.
|
||||
|
||||
## The problem
|
||||
|
||||
Letting an agent touch the filesystem directly is risky: path traversal
|
||||
(`../../etc/passwd`), symlinks that escape the project, clobbering `.git`, or
|
||||
leaking `.env` secrets. Hand-rolling the guards around every tool call is
|
||||
repetitive and easy to get subtly wrong.
|
||||
|
||||
## The solution
|
||||
|
||||
`FileSystem` exposes a fixed set of file tools, all scoped to a single
|
||||
`root_dir`. Every path is resolved and containment-checked (symlinks included)
|
||||
before any I/O, and access is filtered through allow / deny / protected glob
|
||||
patterns.
|
||||
|
||||
```python
|
||||
from pydantic_ai import Agent
|
||||
from pydantic_ai_harness import FileSystem
|
||||
|
||||
agent = Agent(
|
||||
'anthropic:claude-sonnet-4-6',
|
||||
capabilities=[FileSystem(root_dir='./workspace')],
|
||||
)
|
||||
|
||||
result = agent.run_sync('Read config.toml and tell me the package name.')
|
||||
print(result.output)
|
||||
```
|
||||
|
||||
## Tools
|
||||
|
||||
| Tool | Purpose |
|
||||
|---|---|
|
||||
| `read_file` | Read a text file with line numbers and a content hash. Binary files are detected and not dumped. |
|
||||
| `write_file` | Create or overwrite a file. Optional `expected_hash` rejects stale writes (optimistic concurrency). |
|
||||
| `edit_file` | Exact-string replacement; `old_text` must match exactly once. Optional `expected_hash`. |
|
||||
| `list_directory` | List a directory's entries with type indicators and sizes. |
|
||||
| `search_files` | Regex search over file contents, optionally narrowed by an `include_glob`. |
|
||||
| `find_files` | Glob search over file names (e.g. `*.py`, `**/*.json`). |
|
||||
| `create_directory` | Create a directory and any missing parents. |
|
||||
| `file_info` | Metadata for a file or directory (size, type, line count, hash, symlink target). |
|
||||
|
||||
## Security model
|
||||
|
||||
- **Containment.** Paths resolve relative to `root_dir`; anything resolving
|
||||
outside — via `..`, an absolute path, or a symlink — is rejected. Symlinks
|
||||
are resolved with `os.path.realpath` *before* the containment check, closing
|
||||
the TOCTTOU window.
|
||||
- **Binary detection.** `read_file` returns a placeholder instead of dumping
|
||||
binary bytes into the model context.
|
||||
- **Optimistic concurrency.** `write_file`/`edit_file` accept an
|
||||
`expected_hash` so an agent operating on a stale read is told to re-read
|
||||
rather than silently overwriting newer content.
|
||||
|
||||
## Pattern filtering
|
||||
|
||||
Three independent glob lists control access. Patterns are matched with
|
||||
`fnmatch`, whose `*` spans `/`, so `*.py` matches `src/main.py` and you rarely
|
||||
need `**`.
|
||||
|
||||
| Field | Effect |
|
||||
|---|---|
|
||||
| `allowed_patterns` | If non-empty, only matching paths are accessible (allowlist). |
|
||||
| `denied_patterns` | Matching paths are always rejected (denylist). |
|
||||
| `protected_patterns` | Matching paths are read-only — reads succeed, writes are rejected. |
|
||||
|
||||
`protected_patterns` defaults to `.git/`, `.env`/`.env.*`, `*.pem`, `*.key`,
|
||||
and `**/secrets*`. Pass an empty list to disable protection.
|
||||
|
||||
### Direct access vs. walkers
|
||||
|
||||
The three rules apply at two different granularities:
|
||||
|
||||
- **Direct access** (`read_file`, `write_file`, `edit_file`, `file_info`,
|
||||
`create_directory`) gates the operation's target path. You must name a path
|
||||
that the patterns permit.
|
||||
- **Walkers** (`list_directory`, `search_files`, `find_files`) gate their root
|
||||
by deny/protected patterns, but **not** by `allowed_patterns` — a directory
|
||||
root like `.` never matches a file pattern such as `src/*.py`, so requiring
|
||||
it to would make every listing fail. Instead, the root is always walked and
|
||||
each **entry** is filtered against all three lists. A directory listing can
|
||||
never surface a path the agent couldn't otherwise read or write.
|
||||
|
||||
So with `allowed_patterns=['*.py']`, `list_directory('.')` succeeds and shows
|
||||
only the `.py` entries; `read_file('notes.md')` is rejected.
|
||||
|
||||
> Dotfiles and dot-directories (`.git`, `.env`, `.github`, …) are skipped by
|
||||
> all three walkers — `list_directory`, `search_files`, and `find_files` —
|
||||
> regardless of patterns.
|
||||
|
||||
## Configuration
|
||||
|
||||
```python
|
||||
FileSystem(
|
||||
root_dir='.', # str | Path — sandbox root
|
||||
allowed_patterns=[], # allowlist globs (empty = allow all)
|
||||
denied_patterns=[], # denylist globs
|
||||
protected_patterns=[...], # read-only globs (defaults to secrets/.git)
|
||||
max_read_lines=2000, # cap for a single read_file
|
||||
max_search_results=1000, # cap for search_files
|
||||
max_find_results=1000, # cap for find_files
|
||||
)
|
||||
```
|
||||
|
||||
The integer limits must be positive; they are validated at construction.
|
||||
|
||||
## Agent spec (YAML/JSON)
|
||||
|
||||
`FileSystem` works with Pydantic AI's
|
||||
[agent spec](https://ai.pydantic.dev/agent-spec/):
|
||||
|
||||
```yaml
|
||||
# agent.yaml
|
||||
model: anthropic:claude-sonnet-4-6
|
||||
capabilities:
|
||||
- FileSystem:
|
||||
root_dir: ./workspace
|
||||
allowed_patterns: ['*.py', '*.toml']
|
||||
```
|
||||
|
||||
```python
|
||||
from pydantic_ai import Agent
|
||||
from pydantic_ai_harness import FileSystem
|
||||
|
||||
agent = Agent.from_file('agent.yaml', custom_capability_types=[FileSystem])
|
||||
```
|
||||
|
||||
Pass `custom_capability_types` so the spec loader knows how to instantiate
|
||||
`FileSystem`.
|
||||
|
||||
## Further reading
|
||||
|
||||
- [Pydantic AI capabilities](https://ai.pydantic.dev/capabilities/)
|
||||
- [Toolsets](https://ai.pydantic.dev/toolsets/)
|
||||
@@ -0,0 +1,6 @@
|
||||
"""Filesystem capability: gives agents configurable, sandboxed file system access."""
|
||||
|
||||
from pydantic_ai_harness.filesystem._capability import FileSystem
|
||||
from pydantic_ai_harness.filesystem._toolset import FileSystemToolset
|
||||
|
||||
__all__ = ['FileSystem', 'FileSystemToolset']
|
||||
@@ -0,0 +1,81 @@
|
||||
"""Filesystem capability that provides sandboxed file system access."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from pydantic_ai.capabilities import AbstractCapability
|
||||
from pydantic_ai.tools import AgentDepsT
|
||||
from pydantic_ai.toolsets import AgentToolset
|
||||
|
||||
from pydantic_ai_harness.filesystem._toolset import FileSystemToolset
|
||||
|
||||
_DEFAULT_PROTECTED: list[str] = [
|
||||
'.git/*',
|
||||
'.env',
|
||||
'.env.*',
|
||||
'*.pem',
|
||||
'*.key',
|
||||
'**/secrets*',
|
||||
]
|
||||
|
||||
|
||||
@dataclass
|
||||
class FileSystem(AbstractCapability[AgentDepsT]):
|
||||
"""File system access scoped to a root directory.
|
||||
|
||||
All paths are resolved relative to `root_dir`. Traversal above the root
|
||||
is rejected. Symlinks are resolved before authorization.
|
||||
"""
|
||||
|
||||
root_dir: str | Path = '.'
|
||||
"""Root directory for all file operations. Defaults to the current directory."""
|
||||
|
||||
allowed_patterns: Sequence[str] = field(default_factory=list[str])
|
||||
"""If non-empty, only paths matching at least one glob pattern are accessible."""
|
||||
|
||||
denied_patterns: Sequence[str] = field(default_factory=list[str])
|
||||
"""Paths matching any of these glob patterns are rejected."""
|
||||
|
||||
protected_patterns: Sequence[str] = field(default_factory=lambda: list(_DEFAULT_PROTECTED))
|
||||
"""Paths matching these patterns are read-only (writes are rejected).
|
||||
|
||||
Defaults to protecting `.git/`, `.env`, key files, and secrets.
|
||||
Set to an empty list to disable protection.
|
||||
"""
|
||||
|
||||
max_read_lines: int = 2000
|
||||
"""Maximum number of lines returned by a single `read_file` call."""
|
||||
|
||||
max_search_results: int = 1000
|
||||
"""Maximum number of matches returned by `search_files`."""
|
||||
|
||||
max_find_results: int = 1000
|
||||
"""Maximum number of matches returned by `find_files`."""
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
# Runtime validation: dataclass field annotations are advisory, not enforced.
|
||||
# A config-driven caller could pass a string that would otherwise propagate.
|
||||
values: dict[str, Any] = {
|
||||
'max_read_lines': self.max_read_lines,
|
||||
'max_search_results': self.max_search_results,
|
||||
'max_find_results': self.max_find_results,
|
||||
}
|
||||
for name, value in values.items():
|
||||
if not isinstance(value, int) or value <= 0:
|
||||
raise ValueError(f'{name} must be a positive integer, got {value!r}')
|
||||
|
||||
def get_toolset(self) -> AgentToolset[AgentDepsT]:
|
||||
"""Build and return the filesystem toolset."""
|
||||
return FileSystemToolset[AgentDepsT](
|
||||
root_dir=Path(self.root_dir),
|
||||
allowed_patterns=self.allowed_patterns,
|
||||
denied_patterns=self.denied_patterns,
|
||||
protected_patterns=self.protected_patterns,
|
||||
max_read_lines=self.max_read_lines,
|
||||
max_search_results=self.max_search_results,
|
||||
max_find_results=self.max_find_results,
|
||||
)
|
||||
@@ -0,0 +1,507 @@
|
||||
"""Filesystem toolset providing sandboxed file operations."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import fnmatch
|
||||
import functools
|
||||
import hashlib
|
||||
import os
|
||||
import re
|
||||
from collections.abc import Awaitable, Callable, Sequence
|
||||
from pathlib import Path
|
||||
from typing import Concatenate, ParamSpec
|
||||
|
||||
from pydantic_ai.exceptions import ModelRetry
|
||||
from pydantic_ai.tools import AgentDepsT
|
||||
from pydantic_ai.toolsets import FunctionToolset
|
||||
|
||||
_P = ParamSpec('_P')
|
||||
|
||||
# Errors that mean "the model asked for something the tool couldn't do" — a
|
||||
# missing file, a denied path, a stale edit. pyai only feeds `ModelRetry` back
|
||||
# to the model; any other exception aborts the whole run. `_recoverable`
|
||||
# converts these so the agent can correct itself and continue.
|
||||
_RECOVERABLE_ERRORS = (PermissionError, FileNotFoundError, NotADirectoryError, IsADirectoryError, ValueError)
|
||||
|
||||
|
||||
def _recoverable(
|
||||
fn: Callable[Concatenate[FileSystemToolset, _P], Awaitable[str]],
|
||||
) -> Callable[Concatenate[FileSystemToolset, _P], Awaitable[str]]:
|
||||
"""Surface model-correctable tool errors as `ModelRetry`."""
|
||||
|
||||
@functools.wraps(fn)
|
||||
async def wrapper(self: FileSystemToolset, *args: _P.args, **kwargs: _P.kwargs) -> str:
|
||||
try:
|
||||
return await fn(self, *args, **kwargs)
|
||||
except _RECOVERABLE_ERRORS as e:
|
||||
raise ModelRetry(str(e)) from e
|
||||
|
||||
return wrapper
|
||||
|
||||
|
||||
def _format_lines(lines: Sequence[str], offset: int, limit: int) -> str:
|
||||
"""Format pre-split lines with line numbers and continuation hint."""
|
||||
total = len(lines)
|
||||
|
||||
if total == 0:
|
||||
return '(empty file)\n'
|
||||
|
||||
if offset >= total:
|
||||
raise ValueError(f'Offset {offset} exceeds file length ({total} lines).')
|
||||
|
||||
selected = lines[offset : offset + limit]
|
||||
numbered = [f'{i:>6}\t{line}' for i, line in enumerate(selected, start=offset + 1)]
|
||||
result = ''.join(numbered)
|
||||
if not result.endswith('\n'):
|
||||
result += '\n'
|
||||
|
||||
remaining = total - (offset + len(selected))
|
||||
if remaining > 0:
|
||||
next_offset = offset + len(selected)
|
||||
result += f'... ({remaining} more lines. Use offset={next_offset} to continue reading.)\n'
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def _is_binary(data: bytes, sample_size: int = 8192) -> bool:
|
||||
"""Detect binary content by checking for null bytes in the sample."""
|
||||
return b'\x00' in data[:sample_size]
|
||||
|
||||
|
||||
def _content_hash(content: str) -> str:
|
||||
"""Compute a short content hash for conflict detection."""
|
||||
return hashlib.sha256(content.encode('utf-8')).hexdigest()[:12]
|
||||
|
||||
|
||||
class FileSystemToolset(FunctionToolset[AgentDepsT]):
|
||||
"""Toolset providing filesystem operations scoped to a root directory.
|
||||
|
||||
Security model:
|
||||
- All paths resolved relative to root with canonical path checks
|
||||
- Symlinks resolved before authorization (prevents TOCTTOU)
|
||||
- Glob-based allow/deny filtering
|
||||
- Protected path patterns (e.g. `.git/`, `.env`)
|
||||
- Binary file detection blocks text operations
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
root_dir: Path,
|
||||
allowed_patterns: Sequence[str],
|
||||
denied_patterns: Sequence[str],
|
||||
protected_patterns: Sequence[str],
|
||||
max_read_lines: int,
|
||||
max_search_results: int,
|
||||
max_find_results: int,
|
||||
) -> None:
|
||||
super().__init__()
|
||||
self._root = root_dir.resolve()
|
||||
self._real_root = Path(os.path.realpath(self._root))
|
||||
self._allowed_patterns = list(allowed_patterns)
|
||||
self._denied_patterns = list(denied_patterns)
|
||||
self._protected_patterns = list(protected_patterns)
|
||||
self._max_read_lines = max_read_lines
|
||||
self._max_search_results = max_search_results
|
||||
self._max_find_results = max_find_results
|
||||
|
||||
self.add_function(self.read_file, name='read_file')
|
||||
self.add_function(self.write_file, name='write_file')
|
||||
self.add_function(self.edit_file, name='edit_file')
|
||||
self.add_function(self.list_directory, name='list_directory')
|
||||
self.add_function(self.search_files, name='search_files')
|
||||
self.add_function(self.find_files, name='find_files')
|
||||
self.add_function(self.create_directory, name='create_directory')
|
||||
self.add_function(self.file_info, name='file_info')
|
||||
|
||||
def _matches(self, path: str, pattern: str) -> bool:
|
||||
"""Glob-match a relative path, treating a leading `**/` as 'any directory, including the root'.
|
||||
|
||||
`fnmatch` has no recursive `**`, so a bare `**/secrets*` would miss a
|
||||
root-level `secrets.yaml` — there's no leading directory to match.
|
||||
Retrying with the `**/` prefix stripped covers the zero-directory case.
|
||||
"""
|
||||
if fnmatch.fnmatch(path, pattern):
|
||||
return True
|
||||
if pattern.startswith('**/'):
|
||||
return fnmatch.fnmatch(path, pattern[3:])
|
||||
return False
|
||||
|
||||
def _first_matching_pattern(self, path: str, patterns: list[str]) -> str | None:
|
||||
"""Return the first pattern that matches path, or None."""
|
||||
return next((p for p in patterns if self._matches(path, p)), None)
|
||||
|
||||
def _resolve_path(self, path: str) -> Path:
|
||||
"""Resolve path relative to root, rejecting traversal.
|
||||
|
||||
Uses os.path.realpath for symlink resolution before checking containment.
|
||||
"""
|
||||
candidate = (self._root / path).resolve()
|
||||
real = Path(os.path.realpath(candidate))
|
||||
if not real.is_relative_to(self._real_root):
|
||||
raise PermissionError(f'Path {path!r} resolves outside the root directory.')
|
||||
|
||||
return real
|
||||
|
||||
def _check_access(self, path: str, *, write: bool = False, check_allowed: bool = True) -> None:
|
||||
"""Validate path against allow/deny/protected patterns.
|
||||
|
||||
`check_allowed=False` skips the `allowed_patterns` gate. Walkers
|
||||
(`list_directory`, `search_files`, `find_files`) pass it so their root
|
||||
directory isn't required to match `allowed_patterns` itself — `.` or
|
||||
`src` would never match a file pattern like `src/*.py`. The walk's
|
||||
entries are still filtered against `allowed_patterns` per-entry via
|
||||
`_is_accessible`. Denied and protected patterns continue to gate the
|
||||
root.
|
||||
"""
|
||||
if write and self._protected_patterns:
|
||||
matched = self._first_matching_pattern(path, self._protected_patterns)
|
||||
if matched:
|
||||
raise PermissionError(f'Path {path!r} is protected (matches {matched!r}).')
|
||||
|
||||
if self._denied_patterns:
|
||||
matched = self._first_matching_pattern(path, self._denied_patterns)
|
||||
if matched:
|
||||
raise PermissionError(f'Path {path!r} is denied by pattern {matched!r}.')
|
||||
|
||||
if check_allowed and self._allowed_patterns:
|
||||
if not any(self._matches(path, p) for p in self._allowed_patterns):
|
||||
raise PermissionError(f'Path {path!r} does not match any allowed pattern.')
|
||||
|
||||
def _is_accessible(self, path: str, *, write: bool = False) -> bool:
|
||||
"""Predicate form of `_check_access` for filtering recursive walkers.
|
||||
|
||||
Used by `list_directory`, `search_files`, and `find_files` to skip
|
||||
children that would be rejected if accessed directly. Note this only
|
||||
checks the relative path against patterns; it does not resolve symlinks.
|
||||
"""
|
||||
if write and self._protected_patterns:
|
||||
if self._first_matching_pattern(path, self._protected_patterns) is not None:
|
||||
return False
|
||||
if self._denied_patterns:
|
||||
if self._first_matching_pattern(path, self._denied_patterns) is not None:
|
||||
return False
|
||||
if self._allowed_patterns and not any(self._matches(path, p) for p in self._allowed_patterns):
|
||||
return False
|
||||
return True
|
||||
|
||||
def _relative_to_root(self, resolved: Path) -> str:
|
||||
"""Canonical path of a resolved location relative to the real root."""
|
||||
return str(resolved.relative_to(self._real_root))
|
||||
|
||||
def _safe_resolve(self, path: str, *, write: bool = False, check_allowed: bool = True) -> Path:
|
||||
"""Resolve and access-check a path in one step.
|
||||
|
||||
Resolution happens first so the access check matches patterns against
|
||||
the canonical path relative to the root, collapsing `.`/`..`/`//`
|
||||
segments that would otherwise slip past a literal pattern (e.g.
|
||||
`config/./secret.txt` evading a `config/secret.txt` deny rule).
|
||||
"""
|
||||
resolved = self._resolve_path(path)
|
||||
self._check_access(self._relative_to_root(resolved), write=write, check_allowed=check_allowed)
|
||||
return resolved
|
||||
|
||||
@_recoverable
|
||||
async def read_file(self, path: str, *, offset: int = 0, limit: int | None = None) -> str:
|
||||
"""Read a text file with line numbers.
|
||||
|
||||
Args:
|
||||
path: File path relative to the root directory.
|
||||
offset: Zero-based line offset to start reading from.
|
||||
limit: Maximum number of lines to return (default: 2000).
|
||||
|
||||
Returns:
|
||||
File content with line numbers, plus metadata header.
|
||||
"""
|
||||
if limit is None:
|
||||
limit = self._max_read_lines
|
||||
resolved = self._safe_resolve(path)
|
||||
if not resolved.is_file():
|
||||
if resolved.is_dir():
|
||||
raise FileNotFoundError(f"'{path}' is a directory, not a file.")
|
||||
raise FileNotFoundError(f'File not found: {path}')
|
||||
|
||||
raw = resolved.read_bytes()
|
||||
if _is_binary(raw):
|
||||
size = len(raw)
|
||||
return f'[Binary file: {size} bytes. Use a binary-aware tool to inspect.]'
|
||||
|
||||
text = raw.decode('utf-8', errors='replace')
|
||||
lines = text.splitlines(keepends=True)
|
||||
content_hash = _content_hash(text)
|
||||
|
||||
header = f'[{path} | {len(lines)} lines | hash:{content_hash}]\n'
|
||||
return header + _format_lines(lines, offset, limit)
|
||||
|
||||
@_recoverable
|
||||
async def write_file(self, path: str, content: str, *, expected_hash: str | None = None) -> str:
|
||||
"""Create or overwrite a file with conflict detection.
|
||||
|
||||
Args:
|
||||
path: File path relative to the root directory.
|
||||
content: The text content to write.
|
||||
expected_hash: If provided, the write is rejected when the file exists
|
||||
and its current hash doesn't match (optimistic concurrency).
|
||||
|
||||
Returns:
|
||||
Confirmation message with new hash.
|
||||
"""
|
||||
resolved = self._safe_resolve(path, write=True)
|
||||
|
||||
# Optimistic concurrency: reject stale writes
|
||||
if expected_hash is not None and resolved.is_file():
|
||||
current = resolved.read_text(encoding='utf-8')
|
||||
current_hash = _content_hash(current)
|
||||
if current_hash != expected_hash:
|
||||
raise ValueError(
|
||||
f'Conflict: file {path!r} has changed (expected hash:{expected_hash}, '
|
||||
f'got hash:{current_hash}). Re-read the file and retry.'
|
||||
)
|
||||
|
||||
if not resolved.parent.exists():
|
||||
parent_rel = str(resolved.parent.relative_to(self._root))
|
||||
raise FileNotFoundError(f"Parent directory '{parent_rel}' does not exist. Use create_directory first.")
|
||||
resolved.write_text(content, encoding='utf-8')
|
||||
new_hash = _content_hash(content)
|
||||
lines = len(content.splitlines())
|
||||
return f'Wrote {len(content)} chars ({lines} lines) to {path}. [hash:{new_hash}]'
|
||||
|
||||
@_recoverable
|
||||
async def edit_file(self, path: str, old_text: str, new_text: str, *, expected_hash: str | None = None) -> str:
|
||||
"""Edit a file by exact string replacement with conflict detection.
|
||||
|
||||
The old_text must appear exactly once in the file. Include surrounding
|
||||
context lines to ensure uniqueness.
|
||||
|
||||
Args:
|
||||
path: File path relative to the root directory.
|
||||
old_text: The exact text to find (must appear exactly once).
|
||||
new_text: The replacement text.
|
||||
expected_hash: If provided, rejects the edit when the file's
|
||||
current hash doesn't match (optimistic concurrency).
|
||||
|
||||
Returns:
|
||||
Summary with new hash for subsequent operations.
|
||||
"""
|
||||
resolved = self._safe_resolve(path, write=True)
|
||||
if not resolved.is_file():
|
||||
raise FileNotFoundError(f'File not found: {path}')
|
||||
|
||||
text = resolved.read_text(encoding='utf-8')
|
||||
current_hash = _content_hash(text)
|
||||
|
||||
# Optimistic concurrency check
|
||||
if expected_hash is not None and current_hash != expected_hash:
|
||||
raise ValueError(
|
||||
f'Conflict: file {path!r} has changed (expected hash:{expected_hash}, '
|
||||
f'got hash:{current_hash}). Re-read the file and retry.'
|
||||
)
|
||||
|
||||
count = text.count(old_text)
|
||||
if count == 0:
|
||||
raise ValueError(f'old_text not found in {path}.')
|
||||
if count > 1:
|
||||
raise ValueError(
|
||||
f'old_text found {count} times in {path}. Include more surrounding context to make the match unique.'
|
||||
)
|
||||
|
||||
new_content = text.replace(old_text, new_text, 1)
|
||||
resolved.write_text(new_content, encoding='utf-8')
|
||||
new_hash = _content_hash(new_content)
|
||||
return f'Edited {path}. [hash:{new_hash}]'
|
||||
|
||||
@_recoverable
|
||||
async def list_directory(self, path: str = '.') -> str:
|
||||
"""List the contents of a directory.
|
||||
|
||||
Args:
|
||||
path: Directory path relative to the root directory.
|
||||
|
||||
Returns:
|
||||
A newline-separated listing with type indicators and sizes.
|
||||
"""
|
||||
# The listing root is gated by denied/protected patterns but not by
|
||||
# allowed_patterns: a directory like '.' never matches a file pattern.
|
||||
# Entries are filtered per-entry against allowed_patterns below.
|
||||
resolved = self._safe_resolve(path, check_allowed=False)
|
||||
if not resolved.is_dir():
|
||||
raise NotADirectoryError(f'Not a directory: {path}')
|
||||
|
||||
entries: list[str] = []
|
||||
for entry in sorted(resolved.iterdir()):
|
||||
try:
|
||||
rel_path = entry.relative_to(self._real_root)
|
||||
except ValueError: # pragma: no cover
|
||||
continue
|
||||
# Skip dotfiles and dot-directories, matching search_files and
|
||||
# find_files so the three walkers agree on what exists.
|
||||
if any(part.startswith('.') for part in rel_path.parts):
|
||||
continue
|
||||
rel = str(rel_path)
|
||||
# Apply the same allow/deny/protected filtering used for direct
|
||||
# access so a directory listing can't leak patterns the agent
|
||||
# couldn't otherwise read or write.
|
||||
if not self._is_accessible(rel, write=True):
|
||||
continue
|
||||
if entry.is_dir():
|
||||
entries.append(f'{rel}/')
|
||||
else:
|
||||
try:
|
||||
size = entry.stat().st_size
|
||||
except OSError: # pragma: no cover # file deleted between iterdir and stat
|
||||
size = 0
|
||||
entries.append(f'{rel} ({size} bytes)')
|
||||
return '\n'.join(entries) if entries else '(empty directory)'
|
||||
|
||||
@_recoverable
|
||||
async def search_files(self, pattern: str, *, path: str = '.', include_glob: str | None = None) -> str:
|
||||
"""Search file contents using a regular expression.
|
||||
|
||||
Args:
|
||||
pattern: Regex pattern to search for.
|
||||
path: Directory to search in, relative to the root directory.
|
||||
include_glob: If provided, only search files matching this glob (e.g. '*.py').
|
||||
|
||||
Returns:
|
||||
Matching lines formatted as file:line_number:text.
|
||||
"""
|
||||
# See list_directory: the search root isn't gated by allowed_patterns;
|
||||
# matched files are filtered per-entry below.
|
||||
resolved = self._safe_resolve(path, check_allowed=False)
|
||||
try:
|
||||
compiled = re.compile(pattern)
|
||||
except re.error as e:
|
||||
raise ValueError(f'Invalid regex pattern: {e}') from e
|
||||
|
||||
results: list[str] = []
|
||||
|
||||
if resolved.is_file():
|
||||
files = [resolved]
|
||||
else:
|
||||
files = sorted(resolved.rglob('*'))
|
||||
|
||||
real_root = Path(os.path.realpath(self._root))
|
||||
for file_path in files:
|
||||
if not file_path.is_file():
|
||||
continue
|
||||
try:
|
||||
rel_parts = file_path.relative_to(real_root).parts
|
||||
except ValueError: # pragma: no cover
|
||||
continue
|
||||
if any(part.startswith('.') for part in rel_parts):
|
||||
continue
|
||||
rel_str = str(file_path.relative_to(real_root))
|
||||
# Apply the same allow/deny/protected filtering used for direct
|
||||
# access so a recursive search can't read patterns the agent
|
||||
# couldn't otherwise read.
|
||||
if not self._is_accessible(rel_str, write=True):
|
||||
continue
|
||||
if include_glob and not fnmatch.fnmatch(rel_str, include_glob):
|
||||
continue
|
||||
try:
|
||||
raw = file_path.read_bytes()
|
||||
except OSError: # pragma: no cover
|
||||
continue
|
||||
if _is_binary(raw):
|
||||
continue
|
||||
text = raw.decode('utf-8', errors='replace')
|
||||
for line_num, line in enumerate(text.splitlines(), start=1):
|
||||
if compiled.search(line):
|
||||
results.append(f'{rel_str}:{line_num}:{line}')
|
||||
if len(results) >= self._max_search_results:
|
||||
results.append(f'[... truncated at {self._max_search_results} matches]')
|
||||
break
|
||||
|
||||
return '\n'.join(results) if results else 'No matches found.'
|
||||
|
||||
@_recoverable
|
||||
async def find_files(self, pattern: str, *, path: str = '.') -> str:
|
||||
"""Find files by glob pattern (name matching, not content search).
|
||||
|
||||
Args:
|
||||
pattern: Glob pattern to match (e.g. '*.py', '**/*.json').
|
||||
path: Directory to search in, relative to the root directory.
|
||||
|
||||
Returns:
|
||||
Newline-separated list of matching file paths relative to root.
|
||||
"""
|
||||
# See list_directory: the find root isn't gated by allowed_patterns;
|
||||
# matched entries are filtered per-entry below.
|
||||
resolved = self._safe_resolve(path, check_allowed=False)
|
||||
if not resolved.is_dir():
|
||||
raise NotADirectoryError(f'Not a directory: {path}')
|
||||
|
||||
matches: list[str] = []
|
||||
real_root = Path(os.path.realpath(self._root))
|
||||
for match in sorted(resolved.glob(pattern)):
|
||||
try:
|
||||
rel_parts = match.relative_to(real_root).parts
|
||||
except ValueError: # pragma: no cover
|
||||
continue
|
||||
if any(part.startswith('.') for part in rel_parts):
|
||||
continue
|
||||
rel = str(match.relative_to(real_root))
|
||||
# Apply the same allow/deny/protected filtering used for direct
|
||||
# access so a glob find can't surface patterns the agent
|
||||
# couldn't otherwise see.
|
||||
if not self._is_accessible(rel, write=True):
|
||||
continue
|
||||
suffix = '/' if match.is_dir() else ''
|
||||
matches.append(f'{rel}{suffix}')
|
||||
if len(matches) >= self._max_find_results:
|
||||
matches.append(f'[... truncated at {self._max_find_results} matches]')
|
||||
break
|
||||
|
||||
return '\n'.join(matches) if matches else 'No matches found.'
|
||||
|
||||
@_recoverable
|
||||
async def create_directory(self, path: str) -> str:
|
||||
"""Create a directory and any missing parents.
|
||||
|
||||
Args:
|
||||
path: Directory path relative to the root directory.
|
||||
|
||||
Returns:
|
||||
Confirmation message.
|
||||
"""
|
||||
resolved = self._safe_resolve(path, write=True)
|
||||
resolved.mkdir(parents=True, exist_ok=True)
|
||||
return f'Created directory: {path}'
|
||||
|
||||
@_recoverable
|
||||
async def file_info(self, path: str) -> str:
|
||||
"""Get metadata about a file or directory.
|
||||
|
||||
Args:
|
||||
path: File or directory path relative to the root directory.
|
||||
|
||||
Returns:
|
||||
Formatted metadata including size, type, and permissions.
|
||||
"""
|
||||
resolved = self._safe_resolve(path)
|
||||
if not resolved.exists():
|
||||
raise FileNotFoundError(f'Path not found: {path}')
|
||||
|
||||
# Check if the original (pre-resolve) path is a symlink
|
||||
original = self._root / path
|
||||
is_link = original.is_symlink()
|
||||
|
||||
stat = resolved.stat()
|
||||
kind = 'directory' if resolved.is_dir() else 'file'
|
||||
size = stat.st_size
|
||||
|
||||
parts = [f'path: {path}', f'type: {kind}', f'size: {size} bytes']
|
||||
|
||||
if resolved.is_file():
|
||||
raw = resolved.read_bytes()
|
||||
is_bin = _is_binary(raw)
|
||||
parts.append(f'binary: {is_bin}')
|
||||
if not is_bin:
|
||||
text = raw.decode('utf-8', errors='replace')
|
||||
parts.append(f'lines: {len(text.splitlines())}')
|
||||
parts.append(f'hash: {_content_hash(text)}')
|
||||
|
||||
if is_link:
|
||||
parts.append(f'symlink_target: {os.readlink(original)}')
|
||||
|
||||
return '\n'.join(parts)
|
||||
@@ -0,0 +1,129 @@
|
||||
# Shell
|
||||
|
||||
Give an agent the ability to run shell commands, with allow/deny controls and
|
||||
managed background processes.
|
||||
|
||||
## The problem
|
||||
|
||||
Agents frequently need to run a build, a test suite, a linter, or a quick
|
||||
`grep`. Wiring up subprocess handling — streaming output, timeouts, truncation,
|
||||
killing runaway processes, and cleaning up background jobs at the end of a run —
|
||||
is fiddly boilerplate that every agent reinvents.
|
||||
|
||||
## The solution
|
||||
|
||||
`Shell` exposes command-execution tools rooted at a working directory, with
|
||||
configurable allow/deny lists and automatic cleanup of background processes
|
||||
when the agent run ends.
|
||||
|
||||
```python
|
||||
from pydantic_ai import Agent
|
||||
from pydantic_ai_harness import Shell
|
||||
|
||||
agent = Agent(
|
||||
'anthropic:claude-sonnet-4-6',
|
||||
capabilities=[Shell(cwd='./workspace', allowed_commands=['ls', 'cat', 'rg'])],
|
||||
)
|
||||
|
||||
result = agent.run_sync('List the Python files and summarize the largest one.')
|
||||
print(result.output)
|
||||
```
|
||||
|
||||
## Tools
|
||||
|
||||
| Tool | Purpose |
|
||||
|---|---|
|
||||
| `run_command` | Run a command synchronously and return labelled stdout/stderr plus exit code. Honors a per-call or default timeout. |
|
||||
| `start_command` | Launch a long-running command (server, watcher) in the background; returns an ID. |
|
||||
| `check_command` | Report the status and accumulated output of a background command. |
|
||||
| `stop_command` | Terminate a background command and return its final output. |
|
||||
|
||||
Output is labelled with `[stdout]` / `[stderr]` markers and an `[exit code: N]`
|
||||
line on non-zero exit. When it exceeds `max_output_chars` the **tail** is kept
|
||||
(the head is dropped), so errors, stack traces, and the `[stderr]` section —
|
||||
which all land at the end — survive truncation.
|
||||
|
||||
## Command controls
|
||||
|
||||
| Field | Effect |
|
||||
|---|---|
|
||||
| `allowed_commands` | If non-empty, only these executables may run (allowlist). |
|
||||
| `denied_commands` | These executables are always rejected (denylist). |
|
||||
| `denied_operators` | Shell operators (e.g. `>`, `>>`, `|`) that are rejected when present. |
|
||||
| `allow_interactive` | If `False` (default), commands that expect a TTY (`vi`, `sudo`, `ssh`, …) are blocked. |
|
||||
|
||||
`allowed_commands` and `denied_commands` are mutually exclusive — set one, not
|
||||
both. `denied_commands` defaults to a list of destructive commands (`rm`,
|
||||
`rmdir`, `mkfs`, `dd`, `shutdown`, `reboot`, …); pass an empty list to disable.
|
||||
The executable name is extracted with `shlex`, so arguments don't bypass the
|
||||
check.
|
||||
|
||||
> **These checks are best-effort, not a security boundary.** A sufficiently
|
||||
> motivated agent can defeat them (e.g. `bash -c '...'`, env-var indirection).
|
||||
> For hard guarantees, run the agent inside OS-level isolation — a container or
|
||||
> sandbox.
|
||||
|
||||
## Background processes
|
||||
|
||||
`start_command` writes stdout/stderr to temp files and returns a short ID. Use
|
||||
`check_command(id)` to poll and `stop_command(id)` to terminate and collect
|
||||
final output. Processes are launched in their own session (`start_new_session`)
|
||||
so the whole process group can be signalled — `SIGTERM`, escalating to
|
||||
`SIGKILL` after a grace period.
|
||||
|
||||
On run end, the toolset's `__aexit__` terminates every still-running background
|
||||
process and deletes its temp files. The agent runtime enters toolsets via an
|
||||
`AsyncExitStack`, so this cleanup runs whether the run succeeds or raises — an
|
||||
agent that forgets to call `stop_command` won't leak processes.
|
||||
|
||||
## Working directory
|
||||
|
||||
By default each command runs in `cwd` and `cd` has no lasting effect. Set
|
||||
`persist_cwd=True` to make `cd` sticky: the toolset appends a `pwd` sentinel to
|
||||
successful commands, parses the result, and carries the new directory into
|
||||
subsequent calls. Commands containing `;` skip the sentinel injection so the
|
||||
`&&`-gated sentinel can't be bypassed.
|
||||
|
||||
## Configuration
|
||||
|
||||
```python
|
||||
Shell(
|
||||
cwd='.', # str | Path — working directory
|
||||
allowed_commands=[], # allowlist (mutually exclusive with denied)
|
||||
denied_commands=[...], # denylist (defaults to destructive commands)
|
||||
denied_operators=[], # blocked shell operators
|
||||
default_timeout=30.0, # seconds, per run_command
|
||||
max_output_chars=50_000, # output cap returned to the model
|
||||
persist_cwd=False, # make cd sticky across calls
|
||||
allow_interactive=False, # allow TTY-style commands
|
||||
)
|
||||
```
|
||||
|
||||
## Agent spec (YAML/JSON)
|
||||
|
||||
`Shell` works with Pydantic AI's
|
||||
[agent spec](https://ai.pydantic.dev/agent-spec/):
|
||||
|
||||
```yaml
|
||||
# agent.yaml
|
||||
model: anthropic:claude-sonnet-4-6
|
||||
capabilities:
|
||||
- Shell:
|
||||
cwd: ./workspace
|
||||
allowed_commands: ['ls', 'cat', 'rg', 'pytest']
|
||||
```
|
||||
|
||||
```python
|
||||
from pydantic_ai import Agent
|
||||
from pydantic_ai_harness import Shell
|
||||
|
||||
agent = Agent.from_file('agent.yaml', custom_capability_types=[Shell])
|
||||
```
|
||||
|
||||
Pass `custom_capability_types` so the spec loader knows how to instantiate
|
||||
`Shell`.
|
||||
|
||||
## Further reading
|
||||
|
||||
- [Pydantic AI capabilities](https://ai.pydantic.dev/capabilities/)
|
||||
- [Toolsets](https://ai.pydantic.dev/toolsets/)
|
||||
@@ -0,0 +1,6 @@
|
||||
"""Shell capability: gives agents configurable command execution."""
|
||||
|
||||
from pydantic_ai_harness.shell._capability import Shell
|
||||
from pydantic_ai_harness.shell._toolset import ShellToolset
|
||||
|
||||
__all__ = ['Shell', 'ShellToolset']
|
||||
@@ -0,0 +1,76 @@
|
||||
"""Shell capability that provides command execution for agents."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
|
||||
from pydantic_ai.capabilities import AbstractCapability
|
||||
from pydantic_ai.tools import AgentDepsT
|
||||
from pydantic_ai.toolsets import AgentToolset
|
||||
|
||||
from pydantic_ai_harness.shell._toolset import ShellToolset
|
||||
|
||||
_DEFAULT_DENIED_COMMANDS: list[str] = [
|
||||
'rm',
|
||||
'rmdir',
|
||||
'mkfs',
|
||||
'dd',
|
||||
'format',
|
||||
'shutdown',
|
||||
'reboot',
|
||||
'halt',
|
||||
'poweroff',
|
||||
'init',
|
||||
]
|
||||
|
||||
|
||||
@dataclass
|
||||
class Shell(AbstractCapability[AgentDepsT]):
|
||||
"""Shell command execution for agents.
|
||||
|
||||
Commands execute in a subprocess rooted at `cwd`. Use `allowed_commands`
|
||||
or `denied_commands` to control what the agent can invoke.
|
||||
"""
|
||||
|
||||
cwd: str | Path = '.'
|
||||
"""Working directory for command execution."""
|
||||
|
||||
allowed_commands: Sequence[str] = field(default_factory=list[str])
|
||||
"""If non-empty, only these command names may be executed (allowlist)."""
|
||||
|
||||
denied_commands: Sequence[str] = field(default_factory=lambda: list(_DEFAULT_DENIED_COMMANDS))
|
||||
"""These command names are always rejected (denylist).
|
||||
|
||||
Defaults to blocking destructive commands (rm, dd, shutdown, etc.).
|
||||
Set to an empty list to disable.
|
||||
"""
|
||||
|
||||
denied_operators: Sequence[str] = field(default_factory=list[str])
|
||||
"""Shell operators that are blocked (e.g. '>', '>>', '|' for restrictive mode)."""
|
||||
|
||||
default_timeout: float = 30.0
|
||||
"""Default timeout in seconds for command execution."""
|
||||
|
||||
max_output_chars: int = 50_000
|
||||
"""Maximum characters of output returned to the model."""
|
||||
|
||||
persist_cwd: bool = False
|
||||
"""If True, track cd commands and adjust the working directory for subsequent calls."""
|
||||
|
||||
allow_interactive: bool = False
|
||||
"""If True, allow interactive commands (vi, nano, ssh, etc.). Blocked by default."""
|
||||
|
||||
def get_toolset(self) -> AgentToolset[AgentDepsT]:
|
||||
"""Build and return the shell toolset."""
|
||||
return ShellToolset[AgentDepsT](
|
||||
cwd=Path(self.cwd),
|
||||
allowed_commands=self.allowed_commands,
|
||||
denied_commands=self.denied_commands,
|
||||
denied_operators=self.denied_operators,
|
||||
default_timeout=self.default_timeout,
|
||||
max_output_chars=self.max_output_chars,
|
||||
persist_cwd=self.persist_cwd,
|
||||
allow_interactive=self.allow_interactive,
|
||||
)
|
||||
@@ -0,0 +1,500 @@
|
||||
"""Shell toolset — gives agents the ability to run commands."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import functools
|
||||
import os
|
||||
import re
|
||||
import shlex
|
||||
import signal
|
||||
import subprocess
|
||||
import tempfile
|
||||
import uuid
|
||||
from collections.abc import Awaitable, Callable, Sequence
|
||||
from pathlib import Path
|
||||
from typing import Any, Concatenate, ParamSpec
|
||||
|
||||
import anyio
|
||||
import anyio.abc
|
||||
from pydantic_ai import RunContext
|
||||
from pydantic_ai.exceptions import ModelRetry
|
||||
from pydantic_ai.tools import AgentDepsT
|
||||
from pydantic_ai.toolsets import AbstractToolset, FunctionToolset
|
||||
|
||||
_IO_DRAIN_TIMEOUT: float = 2.0
|
||||
_KILL_GRACE_PERIOD: float = 2.0
|
||||
|
||||
_P = ParamSpec('_P')
|
||||
|
||||
|
||||
def _recoverable(
|
||||
fn: Callable[Concatenate[ShellToolset, _P], Awaitable[str]],
|
||||
) -> Callable[Concatenate[ShellToolset, _P], Awaitable[str]]:
|
||||
"""Convert model-correctable errors into `ModelRetry`.
|
||||
|
||||
pyai only feeds `ModelRetry` back to the model as a retry prompt; any other
|
||||
exception propagates and aborts the whole run. A denied command is something
|
||||
the model can recover from (pick an allowed one), so surface it as a retry
|
||||
instead of crashing the agent.
|
||||
"""
|
||||
|
||||
@functools.wraps(fn)
|
||||
async def wrapper(self: ShellToolset, *args: _P.args, **kwargs: _P.kwargs) -> str:
|
||||
try:
|
||||
return await fn(self, *args, **kwargs)
|
||||
except PermissionError as e:
|
||||
raise ModelRetry(str(e)) from e
|
||||
|
||||
return wrapper
|
||||
|
||||
|
||||
def _is_interactive_command(command: str) -> bool:
|
||||
"""Detect commands that typically require interactive input."""
|
||||
interactive_patterns = [
|
||||
r'^(vi|vim|nano|emacs|less|more|top|htop|man)\b',
|
||||
r'^sudo\s',
|
||||
r'^passwd\b',
|
||||
r'^ssh\b',
|
||||
r'^telnet\b',
|
||||
r'^ftp\b',
|
||||
]
|
||||
return any(re.match(p, command.strip()) for p in interactive_patterns)
|
||||
|
||||
|
||||
class _BackgroundProcess:
|
||||
"""State for a background command using temp files for output."""
|
||||
|
||||
__slots__ = ('proc', 'command', 'stdout_path', 'stderr_path', 'finished', 'exit_code')
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
proc: anyio.abc.Process,
|
||||
command: str,
|
||||
stdout_path: str,
|
||||
stderr_path: str,
|
||||
) -> None:
|
||||
self.proc = proc
|
||||
self.command = command
|
||||
self.stdout_path = stdout_path
|
||||
self.stderr_path = stderr_path
|
||||
self.finished = False
|
||||
self.exit_code: int | None = None
|
||||
|
||||
|
||||
class ShellToolset(FunctionToolset[AgentDepsT]):
|
||||
"""Gives an agent the ability to execute shell commands.
|
||||
|
||||
Supports synchronous execution (run_command) and background processes
|
||||
(start_command / check_command / stop_command). Output is streamed,
|
||||
truncated to fit model context, and labelled with stdout/stderr/exit code.
|
||||
|
||||
Optionally tracks the working directory across calls so ``cd`` persists.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
cwd: Path,
|
||||
allowed_commands: Sequence[str],
|
||||
denied_commands: Sequence[str],
|
||||
denied_operators: Sequence[str],
|
||||
default_timeout: float,
|
||||
max_output_chars: int,
|
||||
persist_cwd: bool,
|
||||
allow_interactive: bool,
|
||||
) -> None:
|
||||
super().__init__()
|
||||
self._cwd = cwd.resolve()
|
||||
# The configured starting directory, never mutated by persist_cwd, so
|
||||
# `for_run` can hand each run a fresh instance rooted back here.
|
||||
self._initial_cwd = self._cwd
|
||||
self._allowed_commands = list(allowed_commands)
|
||||
self._denied_commands = list(denied_commands)
|
||||
self._denied_operators = list(denied_operators)
|
||||
self._default_timeout = default_timeout
|
||||
self._max_output_chars = max_output_chars
|
||||
self._persist_cwd = persist_cwd
|
||||
self._allow_interactive = allow_interactive
|
||||
self._background: dict[str, _BackgroundProcess] = {}
|
||||
|
||||
if self._allowed_commands and self._denied_commands:
|
||||
raise ValueError('Specify allowed_commands or denied_commands, not both.')
|
||||
|
||||
self.add_function(self.run_command, name='run_command')
|
||||
self.add_function(self.start_command, name='start_command')
|
||||
self.add_function(self.check_command, name='check_command')
|
||||
self.add_function(self.stop_command, name='stop_command')
|
||||
|
||||
async def for_run(self, ctx: RunContext[AgentDepsT]) -> AbstractToolset[AgentDepsT]:
|
||||
"""Return a fresh instance per run so cwd and background processes are isolated.
|
||||
|
||||
`get_toolset` builds one shared instance at agent construction (see
|
||||
`AbstractToolset.for_run`, which defaults to returning `self`). This
|
||||
toolset holds mutable per-run state (`_cwd`, `_background`), so without
|
||||
an override two concurrent runs would corrupt each other's cwd and kill
|
||||
each other's background processes.
|
||||
"""
|
||||
return ShellToolset(
|
||||
cwd=self._initial_cwd,
|
||||
allowed_commands=self._allowed_commands,
|
||||
denied_commands=self._denied_commands,
|
||||
denied_operators=self._denied_operators,
|
||||
default_timeout=self._default_timeout,
|
||||
max_output_chars=self._max_output_chars,
|
||||
persist_cwd=self._persist_cwd,
|
||||
allow_interactive=self._allow_interactive,
|
||||
)
|
||||
|
||||
async def __aexit__(self, *args: Any) -> None:
|
||||
"""Terminate all remaining background processes and clean up temp files."""
|
||||
for bg in self._background.values():
|
||||
if not bg.finished:
|
||||
await self._kill_process_group(bg.proc)
|
||||
with anyio.CancelScope(shield=True):
|
||||
await bg.proc.wait()
|
||||
await bg.proc.aclose()
|
||||
self._cleanup_bg_files(bg)
|
||||
self._background.clear()
|
||||
|
||||
def _first_denied_operator(self, command: str) -> str | None:
|
||||
"""Return the first denied operator found in command, or None."""
|
||||
return next((op for op in self._denied_operators if op in command), None)
|
||||
|
||||
def _check_command(self, command: str) -> None:
|
||||
"""Validate command against allow/deny lists.
|
||||
|
||||
These checks are best-effort and are not a security boundary — a
|
||||
sufficiently motivated agent can bypass them. Use OS-level isolation
|
||||
(containers, sandboxes) for hard enforcement.
|
||||
"""
|
||||
if not self._allow_interactive and _is_interactive_command(command):
|
||||
raise PermissionError(f'Interactive commands are not allowed. Command: {command!r}')
|
||||
|
||||
matched_op = self._first_denied_operator(command)
|
||||
if matched_op:
|
||||
raise PermissionError(f'Shell operator {matched_op!r} is not allowed.')
|
||||
|
||||
try:
|
||||
tokens = shlex.split(command)
|
||||
except ValueError:
|
||||
return
|
||||
if not tokens:
|
||||
return
|
||||
executable = tokens[0]
|
||||
|
||||
if self._denied_commands and executable in self._denied_commands:
|
||||
raise PermissionError(f'Command {executable!r} is denied.')
|
||||
if self._allowed_commands and executable not in self._allowed_commands:
|
||||
raise PermissionError(f'Command {executable!r} is not in the allowed list.')
|
||||
|
||||
def _truncate(self, text: str) -> str:
|
||||
"""Truncate output to the configured cap, keeping the tail.
|
||||
|
||||
The most useful output — errors, stack traces, exit info, and the
|
||||
`[stderr]` section (which callers append last) — lands at the end, so
|
||||
the head is dropped and the final `max_output_chars` are kept.
|
||||
"""
|
||||
if len(text) <= self._max_output_chars:
|
||||
return text
|
||||
marker = f'[... output truncated, showing last {self._max_output_chars} chars]\n'
|
||||
return marker + text[-self._max_output_chars :]
|
||||
|
||||
def _build_cwd_capture(self, command: str) -> tuple[str, Path | None]:
|
||||
"""Wrap a command to record its final working directory out-of-band.
|
||||
|
||||
`pwd` is written to a private temp file whose random path the agent's
|
||||
command can't address, so command output can never spoof the tracked
|
||||
cwd — unlike parsing a sentinel out of stdout, where any command that
|
||||
prints the sentinel string (or one using `;` to skip success-gating)
|
||||
could redirect the cwd. Returns the wrapped command plus the temp-file
|
||||
path, or the command unchanged and `None` when cwd tracking is off.
|
||||
"""
|
||||
if not self._persist_cwd:
|
||||
return command, None
|
||||
fd, name = tempfile.mkstemp(prefix='harness_cwd_')
|
||||
os.close(fd)
|
||||
wrapped = f'{command}\n__harness_ec=$?\npwd > {shlex.quote(name)}\nexit $__harness_ec'
|
||||
return wrapped, Path(name)
|
||||
|
||||
def _apply_captured_cwd(self, cwd_file: Path) -> None:
|
||||
"""Update the persistent cwd from the capture file, ignoring junk."""
|
||||
try:
|
||||
recorded = cwd_file.read_text(encoding='utf-8').strip()
|
||||
except OSError: # pragma: no cover
|
||||
return
|
||||
if not recorded:
|
||||
return
|
||||
candidate = Path(recorded)
|
||||
if candidate.is_dir():
|
||||
self._cwd = candidate
|
||||
|
||||
async def _kill_process_group(self, proc: anyio.abc.Process) -> None:
|
||||
"""SIGTERM the process group, escalating to SIGKILL after the grace period."""
|
||||
pid = proc.pid
|
||||
try:
|
||||
os.killpg(os.getpgid(pid), signal.SIGTERM)
|
||||
except (ProcessLookupError, PermissionError, OSError):
|
||||
return
|
||||
|
||||
with anyio.move_on_after(_KILL_GRACE_PERIOD):
|
||||
await proc.wait()
|
||||
return
|
||||
|
||||
# Still alive after grace period — hard kill
|
||||
try:
|
||||
os.killpg(os.getpgid(pid), signal.SIGKILL)
|
||||
except (ProcessLookupError, PermissionError, OSError):
|
||||
pass
|
||||
|
||||
async def _drain_with_timeout(
|
||||
self,
|
||||
stdout_chunks: list[bytes],
|
||||
stderr_chunks: list[bytes],
|
||||
proc: anyio.abc.Process,
|
||||
) -> None:
|
||||
"""Drain remaining pipe data after kill (grandchildren may still hold the pipe)."""
|
||||
|
||||
async def _drain_stdout() -> None:
|
||||
if proc.stdout is None:
|
||||
return
|
||||
try:
|
||||
async for chunk in proc.stdout:
|
||||
stdout_chunks.append(chunk)
|
||||
except (anyio.ClosedResourceError, anyio.BrokenResourceError):
|
||||
pass
|
||||
|
||||
async def _drain_stderr() -> None:
|
||||
if proc.stderr is None:
|
||||
return
|
||||
try:
|
||||
async for chunk in proc.stderr:
|
||||
stderr_chunks.append(chunk)
|
||||
except (anyio.ClosedResourceError, anyio.BrokenResourceError):
|
||||
pass
|
||||
|
||||
with anyio.move_on_after(_IO_DRAIN_TIMEOUT):
|
||||
async with anyio.create_task_group() as tg:
|
||||
tg.start_soon(_drain_stdout)
|
||||
tg.start_soon(_drain_stderr)
|
||||
|
||||
@_recoverable
|
||||
async def run_command(self, command: str, *, timeout_seconds: float | None = None) -> str:
|
||||
"""Execute a shell command and return its output.
|
||||
|
||||
Args:
|
||||
command: The shell command to run.
|
||||
timeout_seconds: Maximum seconds to wait (default: 30).
|
||||
|
||||
Returns:
|
||||
Labeled stdout/stderr output with exit code on non-zero exit.
|
||||
"""
|
||||
self._check_command(command)
|
||||
timeout = timeout_seconds if timeout_seconds is not None else self._default_timeout
|
||||
|
||||
actual_command, cwd_file = self._build_cwd_capture(command)
|
||||
try:
|
||||
proc = await anyio.open_process(
|
||||
actual_command,
|
||||
cwd=self._cwd,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
start_new_session=True,
|
||||
)
|
||||
stdout_chunks: list[bytes] = []
|
||||
stderr_chunks: list[bytes] = []
|
||||
try:
|
||||
assert proc.stdout is not None
|
||||
assert proc.stderr is not None
|
||||
|
||||
async def _read_stdout() -> None:
|
||||
assert proc.stdout is not None
|
||||
async for chunk in proc.stdout:
|
||||
stdout_chunks.append(chunk)
|
||||
|
||||
async def _read_stderr() -> None:
|
||||
assert proc.stderr is not None
|
||||
async for chunk in proc.stderr:
|
||||
stderr_chunks.append(chunk)
|
||||
|
||||
with anyio.fail_after(timeout):
|
||||
async with anyio.create_task_group() as tg:
|
||||
tg.start_soon(_read_stdout)
|
||||
tg.start_soon(_read_stderr)
|
||||
await proc.wait()
|
||||
except TimeoutError:
|
||||
await self._kill_process_group(proc)
|
||||
with anyio.CancelScope(shield=True):
|
||||
await proc.wait()
|
||||
await self._drain_with_timeout(stdout_chunks, stderr_chunks, proc)
|
||||
return f'[Command timed out after {timeout}s]'
|
||||
finally:
|
||||
await proc.aclose()
|
||||
|
||||
stdout = b''.join(stdout_chunks).decode('utf-8', errors='replace')
|
||||
stderr = b''.join(stderr_chunks).decode('utf-8', errors='replace')
|
||||
|
||||
parts: list[str] = []
|
||||
if stdout:
|
||||
parts.append(f'[stdout]\n{stdout}')
|
||||
if stderr:
|
||||
parts.append(f'[stderr]\n{stderr}')
|
||||
output = '\n'.join(parts) if parts else '(no output)'
|
||||
|
||||
output = self._truncate(output)
|
||||
exit_code = proc.returncode if proc.returncode is not None else 0
|
||||
|
||||
if cwd_file is not None and exit_code == 0:
|
||||
self._apply_captured_cwd(cwd_file)
|
||||
|
||||
if exit_code != 0:
|
||||
return f'{output}\n[exit code: {exit_code}]'
|
||||
return output
|
||||
finally:
|
||||
if cwd_file is not None:
|
||||
cwd_file.unlink(missing_ok=True)
|
||||
|
||||
@_recoverable
|
||||
async def start_command(self, command: str) -> str:
|
||||
"""Start a long-running command in the background (e.g. a server or watcher).
|
||||
|
||||
Callers MUST call `stop_command(command_id)` when done to terminate the
|
||||
process and clean up temporary output files.
|
||||
|
||||
Args:
|
||||
command: The shell command to run in the background.
|
||||
|
||||
Returns:
|
||||
A message containing the unique command ID for later check/stop calls.
|
||||
"""
|
||||
self._check_command(command)
|
||||
command_id = uuid.uuid4().hex[:12]
|
||||
|
||||
stdout_file = tempfile.NamedTemporaryFile(mode='w+b', prefix=f'harness_{command_id}_out_', delete=False)
|
||||
stderr_file = tempfile.NamedTemporaryFile(mode='w+b', prefix=f'harness_{command_id}_err_', delete=False)
|
||||
|
||||
try:
|
||||
proc = await anyio.open_process(
|
||||
command,
|
||||
cwd=self._cwd,
|
||||
stdout=stdout_file,
|
||||
stderr=stderr_file,
|
||||
start_new_session=True,
|
||||
)
|
||||
except BaseException:
|
||||
stdout_file.close()
|
||||
stderr_file.close()
|
||||
os.unlink(stdout_file.name)
|
||||
os.unlink(stderr_file.name)
|
||||
raise
|
||||
|
||||
stdout_file.close()
|
||||
stderr_file.close()
|
||||
|
||||
bg = _BackgroundProcess(
|
||||
proc=proc,
|
||||
command=command,
|
||||
stdout_path=stdout_file.name,
|
||||
stderr_path=stderr_file.name,
|
||||
)
|
||||
self._background[command_id] = bg
|
||||
|
||||
return f'Started background command: {command!r}\nID: {command_id}'
|
||||
|
||||
def _read_bg_output(self, bg: _BackgroundProcess) -> tuple[str, str]:
|
||||
"""Read current output from background process temp files."""
|
||||
try:
|
||||
stdout = Path(bg.stdout_path).read_text(encoding='utf-8', errors='replace')
|
||||
except OSError:
|
||||
stdout = ''
|
||||
try:
|
||||
stderr = Path(bg.stderr_path).read_text(encoding='utf-8', errors='replace')
|
||||
except OSError:
|
||||
stderr = ''
|
||||
return stdout, stderr
|
||||
|
||||
def _cleanup_bg_files(self, bg: _BackgroundProcess) -> None:
|
||||
"""Remove temp files for a background process."""
|
||||
try:
|
||||
os.unlink(bg.stdout_path)
|
||||
except OSError:
|
||||
pass
|
||||
try:
|
||||
os.unlink(bg.stderr_path)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
async def check_command(self, command_id: str) -> str:
|
||||
"""Check the status and recent output of a background command.
|
||||
|
||||
Args:
|
||||
command_id: The ID returned by start_command.
|
||||
|
||||
Returns:
|
||||
Status and recent output of the background command.
|
||||
"""
|
||||
bg = self._background.get(command_id)
|
||||
if bg is None:
|
||||
return f'[Error: unknown command ID {command_id!r}]'
|
||||
|
||||
if not bg.finished and bg.proc.returncode is not None:
|
||||
bg.exit_code = bg.proc.returncode
|
||||
bg.finished = True
|
||||
|
||||
stdout, stderr = self._read_bg_output(bg)
|
||||
|
||||
status = 'finished' if bg.finished else 'running'
|
||||
parts = [f'[status: {status}]']
|
||||
if bg.finished and bg.exit_code is not None:
|
||||
parts.append(f'[exit code: {bg.exit_code}]')
|
||||
output_sections: list[str] = []
|
||||
if stdout:
|
||||
output_sections.append(f'[stdout]\n{stdout}')
|
||||
if stderr:
|
||||
output_sections.append(f'[stderr]\n{stderr}')
|
||||
if output_sections:
|
||||
parts.append(self._truncate('\n'.join(output_sections)))
|
||||
else:
|
||||
parts.append('(no output yet)')
|
||||
|
||||
return '\n'.join(parts)
|
||||
|
||||
async def stop_command(self, command_id: str) -> str:
|
||||
"""Stop a background command and return its final output.
|
||||
|
||||
Args:
|
||||
command_id: The ID returned by start_command.
|
||||
|
||||
Returns:
|
||||
Final output and exit status of the stopped command.
|
||||
"""
|
||||
bg = self._background.get(command_id)
|
||||
if bg is None:
|
||||
return f'[Error: unknown command ID {command_id!r}]'
|
||||
|
||||
if not bg.finished:
|
||||
await self._kill_process_group(bg.proc)
|
||||
with anyio.CancelScope(shield=True):
|
||||
await bg.proc.wait()
|
||||
bg.exit_code = bg.proc.returncode
|
||||
bg.finished = True
|
||||
|
||||
stdout, stderr = self._read_bg_output(bg)
|
||||
|
||||
self._cleanup_bg_files(bg)
|
||||
del self._background[command_id]
|
||||
await bg.proc.aclose()
|
||||
|
||||
parts = [f'[stopped: {bg.command!r}]']
|
||||
if bg.exit_code is not None:
|
||||
parts.append(f'[exit code: {bg.exit_code}]')
|
||||
output_sections: list[str] = []
|
||||
if stdout:
|
||||
output_sections.append(f'[stdout]\n{stdout}')
|
||||
if stderr:
|
||||
output_sections.append(f'[stderr]\n{stderr}')
|
||||
if output_sections:
|
||||
parts.append(self._truncate('\n'.join(output_sections)))
|
||||
else:
|
||||
parts.append('(no output)')
|
||||
|
||||
return '\n'.join(parts)
|
||||
+13
-1
@@ -112,7 +112,7 @@ quote-style = 'single'
|
||||
[tool.pyright]
|
||||
pythonVersion = '3.10'
|
||||
typeCheckingMode = 'strict'
|
||||
exclude = ['template', '.venv']
|
||||
exclude = ['template', '.venv', 'mutants']
|
||||
executionEnvironments = [
|
||||
{ root = 'tests', reportPrivateUsage = false },
|
||||
]
|
||||
@@ -143,3 +143,15 @@ exclude_lines = [
|
||||
'assert_never',
|
||||
'if TYPE_CHECKING:',
|
||||
]
|
||||
|
||||
[tool.mutmut]
|
||||
paths_to_mutate = [
|
||||
'pydantic_ai_harness/filesystem/_toolset.py',
|
||||
'pydantic_ai_harness/shell/_toolset.py',
|
||||
]
|
||||
tests_dir = ['tests/filesystem/', 'tests/shell/']
|
||||
also_copy = ['pydantic_ai_harness/', 'tests/']
|
||||
# Skip trio-parametrized tests during mutation testing — trio segfaults in
|
||||
# mutmut's subprocess environment on Python 3.14 (not a code bug).
|
||||
pytest_add_cli_args = ['-k', 'not trio']
|
||||
# See docs/mutation-testing.md for full results (89.7% kill rate, 60 equivalent mutants).
|
||||
|
||||
Executable
+24
@@ -0,0 +1,24 @@
|
||||
#!/usr/bin/env bash
|
||||
# One-off mutation testing runner.
|
||||
#
|
||||
# mutmut is intentionally not a project dev dependency: it pulls in a large
|
||||
# tree and is only needed when validating test quality. Install it ephemerally
|
||||
# via `uv run --with` and invoke it as a subcommand.
|
||||
#
|
||||
# Config (paths_to_mutate, tests_dir, also_copy, pytest_add_cli_args) lives in
|
||||
# [tool.mutmut] in pyproject.toml — mutmut v3 reads it from CWD by default.
|
||||
#
|
||||
# Usage:
|
||||
# scripts/run-mutmut.sh # run all mutants
|
||||
# scripts/run-mutmut.sh results # show pass/fail summary
|
||||
# scripts/run-mutmut.sh show <mutant> # inspect a specific mutant
|
||||
# scripts/run-mutmut.sh --max-children 4 run # any mutmut flag works
|
||||
#
|
||||
# Pair with `make testcov` to keep coverage at 100% — surviving mutants usually
|
||||
# indicate missing test cases for boundary conditions.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
uv run --with "mutmut>=3.5.0" -- mutmut "$@"
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,5 +1,7 @@
|
||||
import inspect
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from pydantic_ai import Agent
|
||||
from pydantic_ai.models.test import TestModel
|
||||
|
||||
@@ -11,6 +13,25 @@ def test_import():
|
||||
assert isinstance(pydantic_ai_harness.__all__, list)
|
||||
|
||||
|
||||
def test_lazy_import_filesystem():
|
||||
from pydantic_ai_harness import FileSystem
|
||||
|
||||
assert inspect.isclass(FileSystem)
|
||||
assert hasattr(FileSystem, 'get_toolset')
|
||||
|
||||
|
||||
def test_lazy_import_shell():
|
||||
from pydantic_ai_harness import Shell
|
||||
|
||||
assert inspect.isclass(Shell)
|
||||
assert hasattr(Shell, 'get_toolset')
|
||||
|
||||
|
||||
def test_lazy_import_unknown():
|
||||
with pytest.raises(AttributeError, match='has no attribute'):
|
||||
pydantic_ai_harness.__getattr__('Nonexistent')
|
||||
|
||||
|
||||
def test_test_model_fixture(test_model: TestModel):
|
||||
assert isinstance(test_model, TestModel)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user