diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..d2ce98d --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,125 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Commands + +```bash +# Install dependencies +uv sync + +# Install with PDF/HTML export support +uv sync --extra export + +# Run the server (opens on http://localhost:8080) +uv run deepresearch + +# Start Excalidraw canvas (separate terminal) +docker compose up -d excalidraw-canvas + +# Run without Excalidraw +EXCALIDRAW_ENABLED=0 uv run deepresearch +``` + +No tests, linters, or formatters are configured in this project. + +## Environment + +Copy `.env.example` to `.env` and configure: +- `OPENAI_API_KEY` or `ANTHROPIC_API_KEY` — LLM provider +- At least one search API key: `TAVILY_API_KEY`, `BRAVE_API_KEY`, or `JINA_API_KEY` +- Default model is `openai-responses:o4-mini` (override via `MODEL_NAME`) +- When `LLM_BASE_URL` is set, uses a local OpenAI-compatible endpoint instead + +## Architecture + +**Stack:** Python 3.12+ / FastAPI / WebSocket / pydantic-ai + pydantic-deep + +### Source Layout + +``` +src/deepresearch/ + app.py # FastAPI server, WebSocket /ws/chat, REST endpoints + agent.py # Agent factory — assembles the research agent with all toolsets + config.py # MCP server creation, model config, path constants + prompts.py # System prompt for the research agent (~300 lines of instructions) + middleware.py # Three capabilities: AuditCapability, PermissionCapability, RateLimitRetryCapability + todo_toolset.py # ForgiveWriteTodosCapability — handles malformed tool calls from local LLMs + types.py # Pydantic models: Source, Finding, ReportSection, ResearchReport +static/ + index.html # Single-page frontend + app.js # WebSocket client, tool rendering, file preview, Excalidraw iframe + styles.css # Dark theme UI +skills/ + research-methodology/SKILL.md # Loaded agent skill: source evaluation, search strategy + report-writing/SKILL.md # Loaded agent skill: report structure, citation format + diagram-design/SKILL.md # Loaded agent skill: Excalidraw diagram best practices +workspace/ + DEEP.md # Context file injected into every session (file organization conventions) + MEMORY.md # Persistent agent memory (written via `remember()` tool) +workspaces/ # Per-session data: history.json, meta.json, events.jsonl, canvas.json +``` + +### Runtime Flow + +1. **Lifespan startup** (`app.py:lifespan`): Creates MCP servers (Tavily, Brave, Jina, Excalidraw, Playwright, Firecrawl per API keys in env) and the research agent. Retry loop drops failing MCP servers and restarts. + +2. **WebSocket session** (`/ws/chat`): Each browser tab connects with a session UUID. A per-user Docker sandbox container is created (python-datascience runtime). Message history persists to `workspaces//history.json`. + +3. **Agent execution** (`run_agent_with_streaming`): Uses `agent.iter()` for streaming — processes nodes (ModelRequest → stream text deltas, CallTools → stream tool calls/results) and sends WebSocket JSON events to the frontend. + +4. **Approval flow**: Tools like `execute` trigger `DeferredToolRequests` — the frontend shows an approval dialog, and the agent resumes with `DeferredToolResults`. + +5. **Checkpointing**: Auto-saves after every turn to `InMemoryCheckpointStore`. Supports `rewind` and `fork` via REST endpoints. + +### Agent Architecture + +The agent (`create_research_agent` in `agent.py`) uses **pydantic-deep**'s `create_deep_agent` with: + +- **Toolsets**: MCP servers (search/browser/diagrams) + filesystem + execute + subagents + teams + TODO + skills + checkpoints + agent-factory + remember +- **Subagents**: `general-purpose` (research executor), `planner` (plan mode), `code-reviewer`, plus dynamic agents via factory +- **Hooks**: `audit_logger` (POST_TOOL_USE, background) + `safety_gate` (PRE_TOOL_USE on `execute`, blocks rm -rf /, mkfs, dd, etc.) +- **Middleware stack**: `ForgiveWriteTodosCapability` → `AuditCapability` → `PermissionCapability` → optionally `RateLimitRetryCapability` +- **Context management**: Sliding window (50 msg trigger, keep 30), token eviction (20k limit), context files from `/workspace/DEEP.md` and `/workspace/MEMORY.md` + +### MCP Servers (config.py) + +All optional, enabled by API key presence: +- **Tavily** — AI search (`npx tavily-mcp`) +- **Brave Search** — web search (`npx @anthropic-ai/brave-search-mcp`) +- **Jina** — URL→markdown reader (HTTP MCP at `https://mcp.jina.ai/v1`) +- **Excalidraw** — live diagrams (Podman container, requires `podman` binary) +- **Playwright** — browser automation (requires `PLAYWRIGHT_MCP=1`) +- **Firecrawl** — web scraping/crawling (`npx firecrawl-mcp`) + +### Research Workflow (prompts.py) + +The system prompt encodes a detailed 7-step research workflow: +1. **Plan** — dispatch planner subagent for complex topics +2. **Todo** — create progress tracking with `write_todos` +3. **Parallel subagents** — dispatch all research tasks async +4. **Wait** — `wait_tasks` for results +5. **Handle failures** — never stop, use own knowledge as fallback +6. **Write iteratively** — chapter by chapter via `edit_file` +7. **Present** — summarize and offer deeper analysis + +### REST Endpoints + +| Endpoint | Purpose | +|---|---| +| `GET /` | Serves `static/index.html` | +| `GET /health` | Health check with agent/session status | +| `GET /config` | Full feature configuration | +| `POST /upload` | File upload to sandbox | +| `GET /files?session_id=` | List files in sandbox | +| `GET /files/content/...` | Read file content | +| `GET /files/binary/...` | Read binary files (images) | +| `GET /todos?session_id=` | Get current TODOs | +| `GET /checkpoints?session_id=` | List checkpoints | +| `POST /checkpoints/:id/rewind` | Rewind to checkpoint | +| `POST /checkpoints/:id/fork` | Fork new session from checkpoint | +| `GET /history?session_id=` | Get conversation history | +| `GET /sessions` | List all sessions | +| `DELETE /sessions/:id` | Delete session | +| `GET /export/:fmt` | Export report (md/html/pdf) | +| `GET /preview/:session_id/...` | Serve raw files from container | diff --git a/workspaces/fa1132cd-34db-4576-aa0c-9ea7c6e8b923/events.jsonl b/workspaces/fa1132cd-34db-4576-aa0c-9ea7c6e8b923/events.jsonl index e6efe3e..51f9795 100644 --- a/workspaces/fa1132cd-34db-4576-aa0c-9ea7c6e8b923/events.jsonl +++ b/workspaces/fa1132cd-34db-4576-aa0c-9ea7c6e8b923/events.jsonl @@ -3773,3 +3773,7 @@ {"type": "checkpoint_saved", "checkpoint_id": "f31d3b56-1b3b-4e5d-a806-234f167e932e", "label": "turn-9", "turn": 9, "message_count": 33, "metadata": {}, "_ts": "2026-06-17T08:41:59.812102+00:00"} {"type": "response", "content": "I have completed the research on agentic AI breakthroughs for the past three months (March\u2013June 2026). The high-level overview highlights a significant shift toward **Coordination Engineering** and **Full-Horizon Planning**.\n\n### Key Breakthroughs Identified:\n* **Planning Horizons:** A transition from \"Single-Step\" to **Full-Horizon (FH) Planning**, which allows agents to generate complete multi-step plans upfront, significantly increasing token efficiency for data-centric tasks.\n* **Agentic Browsing:** The emergence of **agentic browsers** (e.g., Perplexity's Comet, OpenAI Operator) that interact with websites semantically and visually rather than relying on brittle scripts.\n* **Memory as Ontology:** A paradigm shift where agent memory is treated as a persistent digital identity governed by a **Constitutional Memory Architecture (CMA)**, featuring stability-based storage tiers.\n* **Swarm Intelligence:** The rise of **Portable Swarm Skills**, allowing complex multi-agent coordination protocols to be shared across different frameworks like AutoGen and LangGraph without custom adapters.\n* **Self-Evolution:** New methods like **Trajectory Distillation** allow agent teams to automatically improve their own coordination patterns by analyzing successful execution logs.\n\nThe full report, including specific model details (GPT-5.2-high, Kimi K2 Thinking) and academic citations, has been saved to:\n` /workspace/report.md`", "_ts": "2026-06-17T08:41:59.925824+00:00"} {"type": "done", "_ts": "2026-06-17T08:41:59.925901+00:00"} +{"type": "session_created", "session_id": "fa1132cd-34db-4576-aa0c-9ea7c6e8b923", "_ts": "2026-06-17T08:45:42.269188+00:00"} +{"type": "canvas_ready", "session_id": "fa1132cd-34db-4576-aa0c-9ea7c6e8b923", "_ts": "2026-06-17T08:45:42.471606+00:00"} +{"type": "session_created", "session_id": "fa1132cd-34db-4576-aa0c-9ea7c6e8b923", "_ts": "2026-06-17T08:45:42.934413+00:00"} +{"type": "canvas_ready", "session_id": "fa1132cd-34db-4576-aa0c-9ea7c6e8b923", "_ts": "2026-06-17T08:45:42.934787+00:00"}