Compare commits
5 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 copilot-swe-agent[bot]
|
036035dae0 | ||
|
copilot-swe-agent[bot]
|
cfad26b684 | ||
|
Willem Jiang
|
a8963f3ac7 | ||
|
Willem Jiang
|
24a8ea76ee | ||
|
Willem Jiang
|
34e3f5c9d4 |
@@ -1,181 +0,0 @@
|
||||
---
|
||||
name: smoke-test
|
||||
description: End-to-end smoke test skill for DeerFlow. Guides through: 1) Pulling latest code, 2) Docker OR Local installation and deployment (user preference, default to Local if Docker network issues), 3) Service availability verification, 4) Health check, 5) Final test report. Use when the user says "run smoke test", "smoke test deployment", "verify installation", "test service availability", "end-to-end test", or similar.
|
||||
---
|
||||
|
||||
# DeerFlow Smoke Test Skill
|
||||
|
||||
This skill guides the Agent through DeerFlow's full end-to-end smoke test workflow, including code updates, deployment (supporting both Docker and local installation modes), service availability verification, and health checks.
|
||||
|
||||
## Deployment Mode Selection
|
||||
|
||||
This skill supports two deployment modes:
|
||||
- **Local installation mode** (recommended, especially when network issues occur) - Run all services directly on the local machine
|
||||
- **Docker mode** - Run all services inside Docker containers
|
||||
|
||||
**Selection strategy**:
|
||||
- If the user explicitly asks for Docker mode, use Docker
|
||||
- If network issues occur (such as slow image pulls), automatically switch to local mode
|
||||
- Default to local mode whenever possible
|
||||
|
||||
## Structure
|
||||
|
||||
```
|
||||
smoke-test/
|
||||
├── SKILL.md ← You are here - core workflow and logic
|
||||
├── scripts/
|
||||
│ ├── check_docker.sh ← Check the Docker environment
|
||||
│ ├── check_local_env.sh ← Check local environment dependencies
|
||||
│ ├── frontend_check.sh ← Frontend page smoke check
|
||||
│ ├── pull_code.sh ← Pull the latest code
|
||||
│ ├── deploy_docker.sh ← Docker deployment
|
||||
│ ├── deploy_local.sh ← Local deployment
|
||||
│ └── health_check.sh ← Service health check
|
||||
├── references/
|
||||
│ ├── SOP.md ← Standard operating procedure
|
||||
│ └── troubleshooting.md ← Troubleshooting guide
|
||||
└── templates/
|
||||
├── report.local.template.md ← Local mode smoke test report template
|
||||
└── report.docker.template.md ← Docker mode smoke test report template
|
||||
```
|
||||
|
||||
## Standard Operating Procedure (SOP)
|
||||
|
||||
### Phase 1: Code Update Check
|
||||
|
||||
1. **Confirm current directory** - Verify that the current working directory is the DeerFlow project root
|
||||
2. **Check Git status** - See whether there are uncommitted changes
|
||||
3. **Pull the latest code** - Use `git pull origin main` to get the latest updates
|
||||
4. **Confirm code update** - Verify that the latest code was pulled successfully
|
||||
|
||||
### Phase 2: Deployment Mode Selection and Environment Check
|
||||
|
||||
**Choose deployment mode**:
|
||||
- Ask for user preference, or choose automatically based on network conditions
|
||||
- Default to local installation mode
|
||||
|
||||
**Local mode environment check**:
|
||||
1. **Check Node.js version** - Requires 22+
|
||||
2. **Check pnpm** - Package manager
|
||||
3. **Check uv** - Python package manager
|
||||
4. **Check nginx** - Reverse proxy
|
||||
5. **Check required ports** - Confirm that ports 2026, 3000, 8001, and 2024 are not occupied
|
||||
|
||||
**Docker mode environment check** (if Docker is selected):
|
||||
1. **Check whether Docker is installed** - Run `docker --version`
|
||||
2. **Check Docker daemon status** - Run `docker info`
|
||||
3. **Check Docker Compose availability** - Run `docker compose version`
|
||||
4. **Check required ports** - Confirm that port 2026 is not occupied
|
||||
|
||||
### Phase 3: Configuration Preparation
|
||||
|
||||
1. **Check whether config.yaml exists**
|
||||
- If it does not exist, run `make config` to generate it
|
||||
- If it already exists, check whether it needs an upgrade with `make config-upgrade`
|
||||
2. **Check the .env file**
|
||||
- Verify that required environment variables are configured
|
||||
- Especially model API keys such as `OPENAI_API_KEY`
|
||||
|
||||
### Phase 4: Deployment Execution
|
||||
|
||||
**Local mode deployment**:
|
||||
1. **Check dependencies** - Run `make check`
|
||||
2. **Install dependencies** - Run `make install`
|
||||
3. **(Optional) Pre-pull the sandbox image** - If needed, run `make setup-sandbox`
|
||||
4. **Start services** - Run `make dev-daemon` (background mode, recommended) or `make dev` (foreground mode)
|
||||
5. **Wait for startup** - Give all services enough time to start completely (90-120 seconds recommended)
|
||||
|
||||
**Docker mode deployment** (if Docker is selected):
|
||||
1. **Initialize Docker environment** - Run `make docker-init`
|
||||
2. **Start Docker services** - Run `make docker-start`
|
||||
3. **Wait for startup** - Give all containers enough time to start completely (60 seconds recommended)
|
||||
|
||||
### Phase 5: Service Health Check
|
||||
|
||||
**Local mode health check**:
|
||||
1. **Check process status** - Confirm that LangGraph, Gateway, Frontend, and Nginx processes are all running
|
||||
2. **Check frontend service** - Visit `http://localhost:2026` and verify that the page loads
|
||||
3. **Check API Gateway** - Verify the `http://localhost:2026/health` endpoint
|
||||
4. **Check LangGraph service** - Verify the availability of relevant endpoints
|
||||
5. **Frontend route smoke check** - Run `bash .agent/skills/smoke-test/scripts/frontend_check.sh` to verify key routes under `/workspace`
|
||||
|
||||
**Docker mode health check** (when using Docker):
|
||||
1. **Check container status** - Run `docker ps` and confirm that all containers are running
|
||||
2. **Check frontend service** - Visit `http://localhost:2026` and verify that the page loads
|
||||
3. **Check API Gateway** - Verify the `http://localhost:2026/health` endpoint
|
||||
4. **Check LangGraph service** - Verify the availability of relevant endpoints
|
||||
5. **Frontend route smoke check** - Run `bash .agent/skills/smoke-test/scripts/frontend_check.sh` to verify key routes under `/workspace`
|
||||
|
||||
### Optional Functional Verification
|
||||
|
||||
1. **List available models** - Verify that model configuration loads correctly
|
||||
2. **List available skills** - Verify that the skill directory is mounted correctly
|
||||
3. **Simple chat test** - Send a simple message to verify the end-to-end flow
|
||||
|
||||
### Phase 6: Generate Test Report
|
||||
|
||||
1. **Collect all test results** - Summarize execution status for each phase
|
||||
2. **Record encountered issues** - If anything fails, record the error details
|
||||
3. **Generate the final report** - Use the template that matches the selected deployment mode to create the complete test report, including overall conclusion, detailed key test cases, and explicit frontend page / route results
|
||||
4. **Provide follow-up recommendations** - Offer suggestions based on the test results
|
||||
|
||||
## Execution Rules
|
||||
|
||||
- **Follow the sequence** - Execute strictly in the order described above
|
||||
- **Idempotency** - Every step should be safe to repeat
|
||||
- **Error handling** - If a step fails, stop and report the issue, then provide troubleshooting suggestions
|
||||
- **Detailed logging** - Record the execution result and status of each step
|
||||
- **User confirmation** - Ask for confirmation before potentially risky operations such as overwriting config
|
||||
- **Mode preference** - Prefer local mode to avoid network-related issues
|
||||
- **Template requirement** - The final report must use the matching template under `templates/`; do not output a free-form summary instead of the template-based report
|
||||
- **Report clarity** - The execution summary must include the overall pass/fail conclusion plus per-case result explanations, and frontend smoke check results must be listed explicitly in the report
|
||||
- **Optional phase handling** - If functional verification is not executed, do not present it as a separate skipped phase in the final report
|
||||
|
||||
## Known Acceptable Warnings
|
||||
|
||||
The following warnings can appear during smoke testing and do not block a successful result:
|
||||
- Feishu/Lark SSL errors in Gateway logs (certificate verification failure) can be ignored if that channel is not enabled
|
||||
- Warnings in LangGraph logs about missing methods in the custom checkpointer, such as `adelete_for_runs` or `aprune`, do not affect the core functionality
|
||||
|
||||
## Key Tools
|
||||
|
||||
Use the following tools during execution:
|
||||
|
||||
1. **bash** - Run shell commands
|
||||
2. **present_file** - Show generated reports and important files
|
||||
3. **task_tool** - Organize complex steps with subtasks when needed
|
||||
|
||||
## Success Criteria
|
||||
|
||||
Smoke test pass criteria (local mode):
|
||||
- [x] Latest code is pulled successfully
|
||||
- [x] Local environment check passes (Node.js 22+, pnpm, uv, nginx)
|
||||
- [x] Configuration files are set up correctly
|
||||
- [x] `make check` passes
|
||||
- [x] `make install` completes successfully
|
||||
- [x] `make dev` starts successfully
|
||||
- [x] All service processes run normally
|
||||
- [x] Frontend page is accessible
|
||||
- [x] Frontend route smoke check passes (`/workspace` key routes)
|
||||
- [x] API Gateway health check passes
|
||||
- [x] Test report is generated completely
|
||||
|
||||
Smoke test pass criteria (Docker mode):
|
||||
- [x] Latest code is pulled successfully
|
||||
- [x] Docker environment check passes
|
||||
- [x] Configuration files are set up correctly
|
||||
- [x] `make docker-init` completes successfully
|
||||
- [x] `make docker-start` completes successfully
|
||||
- [x] All Docker containers run normally
|
||||
- [x] Frontend page is accessible
|
||||
- [x] Frontend route smoke check passes (`/workspace` key routes)
|
||||
- [x] API Gateway health check passes
|
||||
- [x] Test report is generated completely
|
||||
|
||||
## Read Reference Files
|
||||
|
||||
Before starting execution, read the following reference files:
|
||||
1. `references/SOP.md` - Detailed step-by-step operating instructions
|
||||
2. `references/troubleshooting.md` - Common issues and solutions
|
||||
3. `templates/report.local.template.md` - Local mode test report template
|
||||
4. `templates/report.docker.template.md` - Docker mode test report template
|
||||
@@ -1,452 +0,0 @@
|
||||
# DeerFlow Smoke Test Standard Operating Procedure (SOP)
|
||||
|
||||
This document describes the detailed operating steps for each phase of the DeerFlow smoke test.
|
||||
|
||||
## Phase 1: Code Update Check
|
||||
|
||||
### 1.1 Confirm Current Directory
|
||||
|
||||
**Objective**: Verify that the current working directory is the DeerFlow project root.
|
||||
|
||||
**Steps**:
|
||||
1. Run `pwd` to view the current working directory
|
||||
2. Check whether the directory contains the following files/directories:
|
||||
- `Makefile`
|
||||
- `backend/`
|
||||
- `frontend/`
|
||||
- `config.example.yaml`
|
||||
|
||||
**Success Criteria**: The current directory contains all of the files/directories listed above.
|
||||
|
||||
---
|
||||
|
||||
### 1.2 Check Git Status
|
||||
|
||||
**Objective**: Check whether there are uncommitted changes.
|
||||
|
||||
**Steps**:
|
||||
1. Run `git status`
|
||||
2. Check whether the output includes "Changes not staged for commit" or "Untracked files"
|
||||
|
||||
**Notes**:
|
||||
- If there are uncommitted changes, recommend that the user commit or stash them first to avoid conflicts while pulling
|
||||
- If the user confirms that they want to continue, this step can be skipped
|
||||
|
||||
---
|
||||
|
||||
### 1.3 Pull the Latest Code
|
||||
|
||||
**Objective**: Fetch the latest code updates.
|
||||
|
||||
**Steps**:
|
||||
1. Run `git fetch origin main`
|
||||
2. Run `git pull origin main`
|
||||
|
||||
**Success Criteria**:
|
||||
- The commands succeed without errors
|
||||
- The output shows "Already up to date" or indicates that new commits were pulled successfully
|
||||
|
||||
---
|
||||
|
||||
### 1.4 Confirm Code Update
|
||||
|
||||
**Objective**: Verify that the latest code was pulled successfully.
|
||||
|
||||
**Steps**:
|
||||
1. Run `git log -1 --oneline` to view the latest commit
|
||||
2. Record the commit hash and message
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Deployment Mode Selection and Environment Check
|
||||
|
||||
### 2.1 Choose Deployment Mode
|
||||
|
||||
**Objective**: Decide whether to use local mode or Docker mode.
|
||||
|
||||
**Decision Flow**:
|
||||
1. Prefer local mode first to avoid network-related issues
|
||||
2. If the user explicitly requests Docker, use Docker
|
||||
3. If Docker network issues occur, switch to local mode automatically
|
||||
|
||||
---
|
||||
|
||||
### 2.2 Local Mode Environment Check
|
||||
|
||||
**Objective**: Verify that local development environment dependencies are satisfied.
|
||||
|
||||
#### 2.2.1 Check Node.js Version
|
||||
|
||||
**Steps**:
|
||||
1. If nvm is used, run `nvm use 22` to switch to Node 22+
|
||||
2. Run `node --version`
|
||||
|
||||
**Success Criteria**: Version >= 22.x
|
||||
|
||||
**Failure Handling**:
|
||||
- If the version is too low, ask the user to install/switch Node.js with nvm:
|
||||
```bash
|
||||
nvm install 22
|
||||
nvm use 22
|
||||
```
|
||||
- Or install it from the official website: https://nodejs.org/
|
||||
|
||||
---
|
||||
|
||||
#### 2.2.2 Check pnpm
|
||||
|
||||
**Steps**:
|
||||
1. Run `pnpm --version`
|
||||
|
||||
**Success Criteria**: The command returns pnpm version information.
|
||||
|
||||
**Failure Handling**:
|
||||
- If pnpm is not installed, ask the user to install it with `npm install -g pnpm`
|
||||
|
||||
---
|
||||
|
||||
#### 2.2.3 Check uv
|
||||
|
||||
**Steps**:
|
||||
1. Run `uv --version`
|
||||
|
||||
**Success Criteria**: The command returns uv version information.
|
||||
|
||||
**Failure Handling**:
|
||||
- If uv is not installed, ask the user to install uv
|
||||
|
||||
---
|
||||
|
||||
#### 2.2.4 Check nginx
|
||||
|
||||
**Steps**:
|
||||
1. Run `nginx -v`
|
||||
|
||||
**Success Criteria**: The command returns nginx version information.
|
||||
|
||||
**Failure Handling**:
|
||||
- macOS: install with Homebrew using `brew install nginx`
|
||||
- Linux: install using the system package manager
|
||||
|
||||
---
|
||||
|
||||
#### 2.2.5 Check Required Ports
|
||||
|
||||
**Steps**:
|
||||
1. Run the following commands to check ports:
|
||||
```bash
|
||||
lsof -i :2026 # Main port
|
||||
lsof -i :3000 # Frontend
|
||||
lsof -i :8001 # Gateway
|
||||
lsof -i :2024 # LangGraph
|
||||
```
|
||||
|
||||
**Success Criteria**: All ports are free, or they are occupied only by DeerFlow-related processes.
|
||||
|
||||
**Failure Handling**:
|
||||
- If a port is occupied, ask the user to stop the related process
|
||||
|
||||
---
|
||||
|
||||
### 2.3 Docker Mode Environment Check (If Docker Is Selected)
|
||||
|
||||
#### 2.3.1 Check Whether Docker Is Installed
|
||||
|
||||
**Steps**:
|
||||
1. Run `docker --version`
|
||||
|
||||
**Success Criteria**: The command returns Docker version information, such as "Docker version 24.x.x".
|
||||
|
||||
---
|
||||
|
||||
#### 2.3.2 Check Docker Daemon Status
|
||||
|
||||
**Steps**:
|
||||
1. Run `docker info`
|
||||
|
||||
**Success Criteria**: The command runs successfully and shows Docker system information.
|
||||
|
||||
**Failure Handling**:
|
||||
- If it fails, ask the user to start Docker Desktop or the Docker service
|
||||
|
||||
---
|
||||
|
||||
#### 2.3.3 Check Docker Compose Availability
|
||||
|
||||
**Steps**:
|
||||
1. Run `docker compose version`
|
||||
|
||||
**Success Criteria**: The command returns Docker Compose version information.
|
||||
|
||||
---
|
||||
|
||||
#### 2.3.4 Check Required Ports
|
||||
|
||||
**Steps**:
|
||||
1. Run `lsof -i :2026` (macOS/Linux) or `netstat -ano | findstr :2026` (Windows)
|
||||
|
||||
**Success Criteria**: Port 2026 is free, or it is occupied only by a DeerFlow-related process.
|
||||
|
||||
**Failure Handling**:
|
||||
- If the port is occupied by another process, ask the user to stop that process or change the configuration
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Configuration Preparation
|
||||
|
||||
### 3.1 Check config.yaml
|
||||
|
||||
**Steps**:
|
||||
1. Check whether `config.yaml` exists
|
||||
2. If it does not exist, run `make config`
|
||||
3. If it already exists, consider running `make config-upgrade` to merge new fields
|
||||
|
||||
**Validation**:
|
||||
- Check whether at least one model is configured in config.yaml
|
||||
- Check whether the model configuration references the correct environment variables
|
||||
|
||||
---
|
||||
|
||||
### 3.2 Check the .env File
|
||||
|
||||
**Steps**:
|
||||
1. Check whether the `.env` file exists
|
||||
2. If it does not exist, copy it from `.env.example`
|
||||
3. Check whether the following environment variables are configured:
|
||||
- `OPENAI_API_KEY` (or other model API keys)
|
||||
- Other required settings
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Deployment Execution
|
||||
|
||||
### 4.1 Local Mode Deployment
|
||||
|
||||
#### 4.1.1 Check Dependencies
|
||||
|
||||
**Steps**:
|
||||
1. Run `make check`
|
||||
|
||||
**Description**: This command validates all required tools (Node.js 22+, pnpm, uv, nginx).
|
||||
|
||||
---
|
||||
|
||||
#### 4.1.2 Install Dependencies
|
||||
|
||||
**Steps**:
|
||||
1. Run `make install`
|
||||
|
||||
**Description**: This command installs both backend and frontend dependencies.
|
||||
|
||||
**Notes**:
|
||||
- This step may take some time
|
||||
- If network issues cause failures, try using a closer or mirrored package registry
|
||||
|
||||
---
|
||||
|
||||
#### 4.1.3 (Optional) Pre-pull the Sandbox Image
|
||||
|
||||
**Steps**:
|
||||
1. If Docker / Container sandbox is used, run `make setup-sandbox`
|
||||
|
||||
**Description**: This step is optional and not needed for local sandbox mode.
|
||||
|
||||
---
|
||||
|
||||
#### 4.1.4 Start Services
|
||||
|
||||
**Steps**:
|
||||
1. Run `make dev-daemon` (background mode)
|
||||
|
||||
**Description**: This command starts all services (LangGraph, Gateway, Frontend, Nginx).
|
||||
|
||||
**Notes**:
|
||||
- `make dev` runs in the foreground and stops with Ctrl+C
|
||||
- `make dev-daemon` runs in the background
|
||||
- Use `make stop` to stop services
|
||||
|
||||
---
|
||||
|
||||
#### 4.1.5 Wait for Services to Start
|
||||
|
||||
**Steps**:
|
||||
1. Wait 90-120 seconds for all services to start completely
|
||||
2. You can monitor startup progress by checking these log files:
|
||||
- `logs/langgraph.log`
|
||||
- `logs/gateway.log`
|
||||
- `logs/frontend.log`
|
||||
- `logs/nginx.log`
|
||||
|
||||
---
|
||||
|
||||
### 4.2 Docker Mode Deployment (If Docker Is Selected)
|
||||
|
||||
#### 4.2.1 Initialize the Docker Environment
|
||||
|
||||
**Steps**:
|
||||
1. Run `make docker-init`
|
||||
|
||||
**Description**: This command pulls the sandbox image if needed.
|
||||
|
||||
---
|
||||
|
||||
#### 4.2.2 Start Docker Services
|
||||
|
||||
**Steps**:
|
||||
1. Run `make docker-start`
|
||||
|
||||
**Description**: This command builds and starts all required Docker containers.
|
||||
|
||||
---
|
||||
|
||||
#### 4.2.3 Wait for Services to Start
|
||||
|
||||
**Steps**:
|
||||
1. Wait 60-90 seconds for all services to start completely
|
||||
2. You can run `make docker-logs` to monitor startup progress
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: Service Health Check
|
||||
|
||||
### 5.1 Local Mode Health Check
|
||||
|
||||
#### 5.1.1 Check Process Status
|
||||
|
||||
**Steps**:
|
||||
1. Run the following command to check processes:
|
||||
```bash
|
||||
ps aux | grep -E "(langgraph|uvicorn|next|nginx)" | grep -v grep
|
||||
```
|
||||
|
||||
**Success Criteria**: Confirm that the following processes are running:
|
||||
- LangGraph (`langgraph dev`)
|
||||
- Gateway (`uvicorn app.gateway.app:app`)
|
||||
- Frontend (`next dev` or `next start`)
|
||||
- Nginx (`nginx`)
|
||||
|
||||
---
|
||||
|
||||
#### 5.1.2 Check Frontend Service
|
||||
|
||||
**Steps**:
|
||||
1. Use curl or a browser to visit `http://localhost:2026`
|
||||
2. Verify that the page loads normally
|
||||
|
||||
**Example curl command**:
|
||||
```bash
|
||||
curl -I http://localhost:2026
|
||||
```
|
||||
|
||||
**Success Criteria**: Returns an HTTP 200 status code.
|
||||
|
||||
---
|
||||
|
||||
#### 5.1.3 Check API Gateway
|
||||
|
||||
**Steps**:
|
||||
1. Visit `http://localhost:2026/health`
|
||||
|
||||
**Example curl command**:
|
||||
```bash
|
||||
curl http://localhost:2026/health
|
||||
```
|
||||
|
||||
**Success Criteria**: Returns health status JSON.
|
||||
|
||||
---
|
||||
|
||||
#### 5.1.4 Check LangGraph Service
|
||||
|
||||
**Steps**:
|
||||
1. Visit relevant LangGraph endpoints to verify availability
|
||||
|
||||
---
|
||||
|
||||
### 5.2 Docker Mode Health Check (When Using Docker)
|
||||
|
||||
#### 5.2.1 Check Container Status
|
||||
|
||||
**Steps**:
|
||||
1. Run `docker ps`
|
||||
2. Confirm that the following containers are running:
|
||||
- `deer-flow-nginx`
|
||||
- `deer-flow-frontend`
|
||||
- `deer-flow-gateway`
|
||||
- `deer-flow-langgraph` (if not in gateway mode)
|
||||
|
||||
---
|
||||
|
||||
#### 5.2.2 Check Frontend Service
|
||||
|
||||
**Steps**:
|
||||
1. Use curl or a browser to visit `http://localhost:2026`
|
||||
2. Verify that the page loads normally
|
||||
|
||||
**Example curl command**:
|
||||
```bash
|
||||
curl -I http://localhost:2026
|
||||
```
|
||||
|
||||
**Success Criteria**: Returns an HTTP 200 status code.
|
||||
|
||||
---
|
||||
|
||||
#### 5.2.3 Check API Gateway
|
||||
|
||||
**Steps**:
|
||||
1. Visit `http://localhost:2026/health`
|
||||
|
||||
**Example curl command**:
|
||||
```bash
|
||||
curl http://localhost:2026/health
|
||||
```
|
||||
|
||||
**Success Criteria**: Returns health status JSON.
|
||||
|
||||
---
|
||||
|
||||
#### 5.2.4 Check LangGraph Service
|
||||
|
||||
**Steps**:
|
||||
1. Visit relevant LangGraph endpoints to verify availability
|
||||
|
||||
---
|
||||
|
||||
## Optional Functional Verification
|
||||
|
||||
### 6.1 List Available Models
|
||||
|
||||
**Steps**: Verify the model list through the API or UI.
|
||||
|
||||
---
|
||||
|
||||
### 6.2 List Available Skills
|
||||
|
||||
**Steps**: Verify the skill list through the API or UI.
|
||||
|
||||
---
|
||||
|
||||
### 6.3 Simple Chat Test
|
||||
|
||||
**Steps**: Send a simple message to test the complete workflow.
|
||||
|
||||
---
|
||||
|
||||
## Phase 6: Generate the Test Report
|
||||
|
||||
### 6.1 Collect Test Results
|
||||
|
||||
Summarize the execution status of each phase and record successful and failed items.
|
||||
|
||||
### 6.2 Record Issues
|
||||
|
||||
If anything fails, record detailed error information.
|
||||
|
||||
### 6.3 Generate the Report
|
||||
|
||||
Use the template to create a complete test report.
|
||||
|
||||
### 6.4 Provide Recommendations
|
||||
|
||||
Provide follow-up recommendations based on the test results.
|
||||
@@ -1,612 +0,0 @@
|
||||
# Troubleshooting Guide
|
||||
|
||||
This document lists common issues encountered during DeerFlow smoke testing and how to resolve them.
|
||||
|
||||
## Code Update Issues
|
||||
|
||||
### Issue: `git pull` Fails with a Merge Conflict Warning
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
error: Your local changes to the following files would be overwritten by merge
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
1. Option A: Commit local changes first
|
||||
```bash
|
||||
git add .
|
||||
git commit -m "Save local changes"
|
||||
git pull origin main
|
||||
```
|
||||
|
||||
2. Option B: Stash local changes
|
||||
```bash
|
||||
git stash
|
||||
git pull origin main
|
||||
git stash pop # Restore changes later if needed
|
||||
```
|
||||
|
||||
3. Option C: Discard local changes (use with caution)
|
||||
```bash
|
||||
git reset --hard HEAD
|
||||
git pull origin main
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Local Mode Environment Issues
|
||||
|
||||
### Issue: Node.js Version Is Too Old
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
Node.js version is too old. Requires 22+, got x.x.x
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
1. Install or upgrade Node.js with nvm:
|
||||
```bash
|
||||
nvm install 22
|
||||
nvm use 22
|
||||
```
|
||||
|
||||
2. Or download and install it from the official website: https://nodejs.org/
|
||||
|
||||
3. Verify the version:
|
||||
```bash
|
||||
node --version
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: pnpm Is Not Installed
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
command not found: pnpm
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
1. Install pnpm with npm:
|
||||
```bash
|
||||
npm install -g pnpm
|
||||
```
|
||||
|
||||
2. Or use the official installation script:
|
||||
```bash
|
||||
curl -fsSL https://get.pnpm.io/install.sh | sh -
|
||||
```
|
||||
|
||||
3. Verify the installation:
|
||||
```bash
|
||||
pnpm --version
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: uv Is Not Installed
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
command not found: uv
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
1. Use the official installation script:
|
||||
```bash
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
```
|
||||
|
||||
2. macOS users can also install it with Homebrew:
|
||||
```bash
|
||||
brew install uv
|
||||
```
|
||||
|
||||
3. Verify the installation:
|
||||
```bash
|
||||
uv --version
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: nginx Is Not Installed
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
command not found: nginx
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
1. macOS (Homebrew):
|
||||
```bash
|
||||
brew install nginx
|
||||
```
|
||||
|
||||
2. Ubuntu/Debian:
|
||||
```bash
|
||||
sudo apt update
|
||||
sudo apt install nginx
|
||||
```
|
||||
|
||||
3. CentOS/RHEL:
|
||||
```bash
|
||||
sudo yum install nginx
|
||||
```
|
||||
|
||||
4. Verify the installation:
|
||||
```bash
|
||||
nginx -v
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: Port Is Already in Use
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
Error: listen EADDRINUSE: address already in use :::2026
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
1. Find the process using the port:
|
||||
```bash
|
||||
lsof -i :2026 # macOS/Linux
|
||||
netstat -ano | findstr :2026 # Windows
|
||||
```
|
||||
|
||||
2. Stop that process:
|
||||
```bash
|
||||
kill -9 <PID> # macOS/Linux
|
||||
taskkill /PID <PID> /F # Windows
|
||||
```
|
||||
|
||||
3. Or stop DeerFlow services first:
|
||||
```bash
|
||||
make stop
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Local Mode Dependency Installation Issues
|
||||
|
||||
### Issue: `make install` Fails Due to Network Timeout
|
||||
|
||||
**Symptoms**:
|
||||
Network timeouts or connection failures occur during dependency installation.
|
||||
|
||||
**Solutions**:
|
||||
1. Configure pnpm to use a mirror registry:
|
||||
```bash
|
||||
pnpm config set registry https://registry.npmmirror.com
|
||||
```
|
||||
|
||||
2. Configure uv to use a mirror registry:
|
||||
```bash
|
||||
uv pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
|
||||
```
|
||||
|
||||
3. Retry the installation:
|
||||
```bash
|
||||
make install
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: Python Dependency Installation Fails
|
||||
|
||||
**Symptoms**:
|
||||
Errors occur during `uv sync`.
|
||||
|
||||
**Solutions**:
|
||||
1. Clean the uv cache:
|
||||
```bash
|
||||
cd backend
|
||||
uv cache clean
|
||||
```
|
||||
|
||||
2. Resync dependencies:
|
||||
```bash
|
||||
cd backend
|
||||
uv sync
|
||||
```
|
||||
|
||||
3. View detailed error logs:
|
||||
```bash
|
||||
cd backend
|
||||
uv sync --verbose
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: Frontend Dependency Installation Fails
|
||||
|
||||
**Symptoms**:
|
||||
Errors occur during `pnpm install`.
|
||||
|
||||
**Solutions**:
|
||||
1. Clean the pnpm cache:
|
||||
```bash
|
||||
cd frontend
|
||||
pnpm store prune
|
||||
```
|
||||
|
||||
2. Remove node_modules and the lock file:
|
||||
```bash
|
||||
cd frontend
|
||||
rm -rf node_modules pnpm-lock.yaml
|
||||
```
|
||||
|
||||
3. Reinstall:
|
||||
```bash
|
||||
cd frontend
|
||||
pnpm install
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Local Mode Service Startup Issues
|
||||
|
||||
### Issue: Services Exit Immediately After Startup
|
||||
|
||||
**Symptoms**:
|
||||
Processes exit quickly after running `make dev-daemon`.
|
||||
|
||||
**Solutions**:
|
||||
1. Check log files:
|
||||
```bash
|
||||
tail -f logs/langgraph.log
|
||||
tail -f logs/gateway.log
|
||||
tail -f logs/frontend.log
|
||||
tail -f logs/nginx.log
|
||||
```
|
||||
|
||||
2. Check whether config.yaml is configured correctly
|
||||
3. Check environment variables in the .env file
|
||||
4. Confirm that required ports are not occupied
|
||||
5. Stop all services and restart:
|
||||
```bash
|
||||
make stop
|
||||
make dev-daemon
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: Nginx Fails to Start Because Temp Directories Do Not Exist
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
nginx: [emerg] mkdir() "/opt/homebrew/var/run/nginx/client_body_temp" failed (2: No such file or directory)
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
Add local temp directory configuration to `docker/nginx/nginx.local.conf` so nginx uses the repository's temp directory.
|
||||
|
||||
Add the following at the beginning of the `http` block:
|
||||
```nginx
|
||||
client_body_temp_path temp/client_body_temp;
|
||||
proxy_temp_path temp/proxy_temp;
|
||||
fastcgi_temp_path temp/fastcgi_temp;
|
||||
uwsgi_temp_path temp/uwsgi_temp;
|
||||
scgi_temp_path temp/scgi_temp;
|
||||
```
|
||||
|
||||
Note: The `temp/` directory under the repository root is created automatically by `make dev` or `make dev-daemon`.
|
||||
|
||||
---
|
||||
|
||||
### Issue: Nginx Fails to Start (General)
|
||||
|
||||
**Symptoms**:
|
||||
The nginx process fails to start or reports an error.
|
||||
|
||||
**Solutions**:
|
||||
1. Check the nginx configuration:
|
||||
```bash
|
||||
nginx -t -c docker/nginx/nginx.local.conf -p .
|
||||
```
|
||||
|
||||
2. Check nginx logs:
|
||||
```bash
|
||||
tail -f logs/nginx.log
|
||||
```
|
||||
|
||||
3. Ensure no other nginx process is running:
|
||||
```bash
|
||||
ps aux | grep nginx
|
||||
```
|
||||
|
||||
4. If needed, stop existing nginx processes:
|
||||
```bash
|
||||
pkill -9 nginx
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: Frontend Compilation Fails
|
||||
|
||||
**Symptoms**:
|
||||
Compilation errors appear in `frontend.log`.
|
||||
|
||||
**Solutions**:
|
||||
1. Check frontend logs:
|
||||
```bash
|
||||
tail -f logs/frontend.log
|
||||
```
|
||||
|
||||
2. Check whether Node.js version is 22+
|
||||
3. Reinstall frontend dependencies:
|
||||
```bash
|
||||
cd frontend
|
||||
rm -rf node_modules .next
|
||||
pnpm install
|
||||
```
|
||||
|
||||
4. Restart services:
|
||||
```bash
|
||||
make stop
|
||||
make dev-daemon
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: Gateway Fails to Start
|
||||
|
||||
**Symptoms**:
|
||||
Errors appear in `gateway.log`.
|
||||
|
||||
**Solutions**:
|
||||
1. Check gateway logs:
|
||||
```bash
|
||||
tail -f logs/gateway.log
|
||||
```
|
||||
|
||||
2. Check whether config.yaml exists and has valid formatting
|
||||
3. Check whether Python dependencies are complete:
|
||||
```bash
|
||||
cd backend
|
||||
uv sync
|
||||
```
|
||||
|
||||
4. Confirm that the LangGraph service is running normally (if not in gateway mode)
|
||||
|
||||
---
|
||||
|
||||
### Issue: LangGraph Fails to Start
|
||||
|
||||
**Symptoms**:
|
||||
Errors appear in `langgraph.log`.
|
||||
|
||||
**Solutions**:
|
||||
1. Check LangGraph logs:
|
||||
```bash
|
||||
tail -f logs/langgraph.log
|
||||
```
|
||||
|
||||
2. Check config.yaml
|
||||
3. Check whether Python dependencies are complete
|
||||
4. Confirm that port 2024 is not occupied
|
||||
|
||||
---
|
||||
|
||||
## Docker-Related Issues
|
||||
|
||||
### Issue: Docker Commands Cannot Run
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
Cannot connect to the Docker daemon
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
1. Confirm that Docker Desktop is running
|
||||
2. macOS: check whether the Docker icon appears in the top menu bar
|
||||
3. Linux: run `sudo systemctl start docker`
|
||||
4. Run `docker info` again to verify
|
||||
|
||||
---
|
||||
|
||||
### Issue: `make docker-init` Fails to Pull the Image
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
Error pulling image: connection refused
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
1. Check network connectivity
|
||||
2. Configure a Docker image mirror if needed
|
||||
3. Check whether a proxy is required
|
||||
4. Switch to local installation mode if necessary (recommended)
|
||||
|
||||
---
|
||||
|
||||
## Configuration File Issues
|
||||
|
||||
### Issue: config.yaml Is Missing or Invalid
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
Error: could not read config.yaml
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
1. Regenerate the configuration file:
|
||||
```bash
|
||||
make config
|
||||
```
|
||||
|
||||
2. Check YAML syntax:
|
||||
- Make sure indentation is correct (use 2 spaces)
|
||||
- Make sure there are no tab characters
|
||||
- Check that there is a space after each colon
|
||||
|
||||
3. Use a YAML validation tool to check the format
|
||||
|
||||
---
|
||||
|
||||
### Issue: Model API Key Is Not Configured
|
||||
|
||||
**Symptoms**:
|
||||
After services start, API requests fail with authentication errors.
|
||||
|
||||
**Solutions**:
|
||||
1. Edit the .env file and add the API key:
|
||||
```bash
|
||||
OPENAI_API_KEY=your-actual-api-key-here
|
||||
```
|
||||
|
||||
2. Restart services (local mode):
|
||||
```bash
|
||||
make stop
|
||||
make dev-daemon
|
||||
```
|
||||
|
||||
3. Restart services (Docker mode):
|
||||
```bash
|
||||
make docker-stop
|
||||
make docker-start
|
||||
```
|
||||
|
||||
4. Confirm that the model configuration in config.yaml references the environment variable correctly
|
||||
|
||||
---
|
||||
|
||||
## Service Health Check Issues
|
||||
|
||||
### Issue: Frontend Page Is Not Accessible
|
||||
|
||||
**Symptoms**:
|
||||
The browser shows a connection failure when visiting http://localhost:2026.
|
||||
|
||||
**Solutions** (local mode):
|
||||
1. Confirm that the nginx process is running:
|
||||
```bash
|
||||
ps aux | grep nginx
|
||||
```
|
||||
|
||||
2. Check nginx logs:
|
||||
```bash
|
||||
tail -f logs/nginx.log
|
||||
```
|
||||
|
||||
3. Check firewall settings
|
||||
|
||||
**Solutions** (Docker mode):
|
||||
1. Confirm that the nginx container is running:
|
||||
```bash
|
||||
docker ps | grep nginx
|
||||
```
|
||||
|
||||
2. Check nginx logs:
|
||||
```bash
|
||||
cd docker && docker compose -p deer-flow-dev -f docker-compose-dev.yaml logs nginx
|
||||
```
|
||||
|
||||
3. Check firewall settings
|
||||
|
||||
---
|
||||
|
||||
### Issue: API Gateway Health Check Fails
|
||||
|
||||
**Symptoms**:
|
||||
Accessing `/health` returns an error or times out.
|
||||
|
||||
**Solutions** (local mode):
|
||||
1. Check gateway logs:
|
||||
```bash
|
||||
tail -f logs/gateway.log
|
||||
```
|
||||
|
||||
2. Confirm that config.yaml exists and has valid formatting
|
||||
3. Check whether Python dependencies are complete
|
||||
4. Confirm that the LangGraph service is running normally
|
||||
|
||||
**Solutions** (Docker mode):
|
||||
1. Check gateway container logs:
|
||||
```bash
|
||||
make docker-logs-gateway
|
||||
```
|
||||
|
||||
2. Confirm that config.yaml is mounted correctly
|
||||
3. Check whether Python dependencies are complete
|
||||
4. Confirm that the LangGraph service is running normally
|
||||
|
||||
---
|
||||
|
||||
## Common Diagnostic Commands
|
||||
|
||||
### Local Mode Diagnostics
|
||||
|
||||
#### View All Service Processes
|
||||
```bash
|
||||
ps aux | grep -E "(langgraph|uvicorn|next|nginx)" | grep -v grep
|
||||
```
|
||||
|
||||
#### View Service Logs
|
||||
```bash
|
||||
# View all logs
|
||||
tail -f logs/*.log
|
||||
|
||||
# View specific service logs
|
||||
tail -f logs/langgraph.log
|
||||
tail -f logs/gateway.log
|
||||
tail -f logs/frontend.log
|
||||
tail -f logs/nginx.log
|
||||
```
|
||||
|
||||
#### Stop All Services
|
||||
```bash
|
||||
make stop
|
||||
```
|
||||
|
||||
#### Fully Reset the Local Environment
|
||||
```bash
|
||||
make stop
|
||||
make clean
|
||||
make config
|
||||
make install
|
||||
make dev-daemon
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Docker Mode Diagnostics
|
||||
|
||||
#### View All Container Status
|
||||
```bash
|
||||
docker ps -a
|
||||
```
|
||||
|
||||
#### View Container Resource Usage
|
||||
```bash
|
||||
docker stats
|
||||
```
|
||||
|
||||
#### Enter a Container for Debugging
|
||||
```bash
|
||||
docker exec -it deer-flow-gateway sh
|
||||
```
|
||||
|
||||
#### Clean Up All DeerFlow-Related Containers and Images
|
||||
```bash
|
||||
make docker-stop
|
||||
cd docker && docker compose -p deer-flow-dev -f docker-compose-dev.yaml down -v
|
||||
```
|
||||
|
||||
#### Fully Reset the Docker Environment
|
||||
```bash
|
||||
make docker-stop
|
||||
make clean
|
||||
make config
|
||||
make docker-init
|
||||
make docker-start
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Get More Help
|
||||
|
||||
If the solutions above do not resolve the issue:
|
||||
1. Check the GitHub issues for the project: https://github.com/bytedance/deer-flow/issues
|
||||
2. Review the project documentation: README.md and the `backend/docs/` directory
|
||||
3. Open a new issue and include detailed error logs
|
||||
@@ -1,80 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
set -e
|
||||
|
||||
echo "=========================================="
|
||||
echo " Checking Docker Environment"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
|
||||
# Check whether Docker is installed
|
||||
if command -v docker >/dev/null 2>&1; then
|
||||
echo "✓ Docker is installed"
|
||||
docker --version
|
||||
else
|
||||
echo "✗ Docker is not installed"
|
||||
exit 1
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check the Docker daemon
|
||||
if docker info >/dev/null 2>&1; then
|
||||
echo "✓ Docker daemon is running normally"
|
||||
else
|
||||
echo "✗ Docker daemon is not running"
|
||||
echo " Please start Docker Desktop or the Docker service"
|
||||
exit 1
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check Docker Compose
|
||||
if docker compose version >/dev/null 2>&1; then
|
||||
echo "✓ Docker Compose is available"
|
||||
docker compose version
|
||||
else
|
||||
echo "✗ Docker Compose is not available"
|
||||
exit 1
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check port 2026
|
||||
if ! command -v lsof >/dev/null 2>&1; then
|
||||
echo "✗ lsof is required to check whether port 2026 is available"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
port_2026_usage="$(lsof -nP -iTCP:2026 -sTCP:LISTEN 2>/dev/null || true)"
|
||||
if [ -n "$port_2026_usage" ]; then
|
||||
echo "⚠ Port 2026 is already in use"
|
||||
echo " Occupying process:"
|
||||
echo "$port_2026_usage"
|
||||
|
||||
deerflow_process_found=0
|
||||
while IFS= read -r pid; do
|
||||
if [ -z "$pid" ]; then
|
||||
continue
|
||||
fi
|
||||
|
||||
process_command="$(ps -p "$pid" -o command= 2>/dev/null || true)"
|
||||
case "$process_command" in
|
||||
*[Dd]eer[Ff]low*|*[Dd]eerflow*|*[Nn]ginx*deerflow*|*deerflow/*[Nn]ginx*)
|
||||
deerflow_process_found=1
|
||||
;;
|
||||
esac
|
||||
done <<EOF
|
||||
$(printf '%s\n' "$port_2026_usage" | awk 'NR > 1 {print $2}')
|
||||
EOF
|
||||
|
||||
if [ "$deerflow_process_found" -eq 1 ]; then
|
||||
echo "✓ Port 2026 is occupied by DeerFlow"
|
||||
else
|
||||
echo "✗ Port 2026 must be free before starting DeerFlow"
|
||||
exit 1
|
||||
fi
|
||||
else
|
||||
echo "✓ Port 2026 is available"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
echo "=========================================="
|
||||
echo " Docker Environment Check Complete"
|
||||
echo "=========================================="
|
||||
@@ -1,93 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
set -e
|
||||
|
||||
echo "=========================================="
|
||||
echo " Checking Local Development Environment"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
|
||||
all_passed=true
|
||||
|
||||
# Check Node.js
|
||||
echo "1. Checking Node.js..."
|
||||
if command -v node >/dev/null 2>&1; then
|
||||
NODE_VERSION=$(node --version | sed 's/v//')
|
||||
NODE_MAJOR=$(echo "$NODE_VERSION" | cut -d. -f1)
|
||||
if [ "$NODE_MAJOR" -ge 22 ]; then
|
||||
echo "✓ Node.js is installed (version: $NODE_VERSION)"
|
||||
else
|
||||
echo "✗ Node.js version is too old (current: $NODE_VERSION, required: 22+)"
|
||||
all_passed=false
|
||||
fi
|
||||
else
|
||||
echo "✗ Node.js is not installed"
|
||||
all_passed=false
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check pnpm
|
||||
echo "2. Checking pnpm..."
|
||||
if command -v pnpm >/dev/null 2>&1; then
|
||||
echo "✓ pnpm is installed (version: $(pnpm --version))"
|
||||
else
|
||||
echo "✗ pnpm is not installed"
|
||||
echo " Install command: npm install -g pnpm"
|
||||
all_passed=false
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check uv
|
||||
echo "3. Checking uv..."
|
||||
if command -v uv >/dev/null 2>&1; then
|
||||
echo "✓ uv is installed (version: $(uv --version))"
|
||||
else
|
||||
echo "✗ uv is not installed"
|
||||
all_passed=false
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check nginx
|
||||
echo "4. Checking nginx..."
|
||||
if command -v nginx >/dev/null 2>&1; then
|
||||
echo "✓ nginx is installed (version: $(nginx -v 2>&1))"
|
||||
else
|
||||
echo "✗ nginx is not installed"
|
||||
echo " macOS: brew install nginx"
|
||||
echo " Linux: install it with the system package manager"
|
||||
all_passed=false
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check ports
|
||||
echo "5. Checking ports..."
|
||||
if ! command -v lsof >/dev/null 2>&1; then
|
||||
echo "✗ lsof is not installed, so port availability cannot be verified"
|
||||
echo " Install lsof and rerun this check"
|
||||
all_passed=false
|
||||
else
|
||||
for port in 2026 3000 8001 2024; do
|
||||
if lsof -i :$port >/dev/null 2>&1; then
|
||||
echo "⚠ Port $port is already in use:"
|
||||
lsof -i :$port | head -2
|
||||
all_passed=false
|
||||
else
|
||||
echo "✓ Port $port is available"
|
||||
fi
|
||||
done
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Summary
|
||||
echo "=========================================="
|
||||
echo " Environment Check Summary"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
if [ "$all_passed" = true ]; then
|
||||
echo "✅ All environment checks passed!"
|
||||
echo ""
|
||||
echo "Next step: run make install to install dependencies"
|
||||
exit 0
|
||||
else
|
||||
echo "❌ Some checks failed. Please fix the issues above first"
|
||||
exit 1
|
||||
fi
|
||||
@@ -1,65 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
set -e
|
||||
|
||||
echo "=========================================="
|
||||
echo " Docker Deployment"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
|
||||
# Check config.yaml
|
||||
if [ ! -f "config.yaml" ]; then
|
||||
echo "config.yaml does not exist. Generating it..."
|
||||
make config
|
||||
echo ""
|
||||
echo "⚠ Please edit config.yaml to configure your models and API keys"
|
||||
echo " Then run this script again"
|
||||
exit 1
|
||||
else
|
||||
echo "✓ config.yaml exists"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check the .env file
|
||||
if [ ! -f ".env" ]; then
|
||||
echo ".env does not exist. Copying it from the example..."
|
||||
if [ -f ".env.example" ]; then
|
||||
cp .env.example .env
|
||||
echo "✓ Created the .env file"
|
||||
else
|
||||
echo "⚠ .env.example does not exist. Please create the .env file manually"
|
||||
fi
|
||||
else
|
||||
echo "✓ .env file exists"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check the frontend .env file
|
||||
if [ ! -f "frontend/.env" ]; then
|
||||
echo "frontend/.env does not exist. Copying it from the example..."
|
||||
if [ -f "frontend/.env.example" ]; then
|
||||
cp frontend/.env.example frontend/.env
|
||||
echo "✓ Created the frontend/.env file"
|
||||
else
|
||||
echo "⚠ frontend/.env.example does not exist. Please create frontend/.env manually"
|
||||
fi
|
||||
else
|
||||
echo "✓ frontend/.env file exists"
|
||||
fi
|
||||
echo ""
|
||||
# Initialize the Docker environment
|
||||
echo "Initializing the Docker environment..."
|
||||
make docker-init
|
||||
echo ""
|
||||
|
||||
# Start Docker services
|
||||
echo "Starting Docker services..."
|
||||
make docker-start
|
||||
echo ""
|
||||
|
||||
echo "=========================================="
|
||||
echo " Deployment Complete"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
echo "🌐 Access URL: http://localhost:2026"
|
||||
echo "📋 View logs: make docker-logs"
|
||||
echo "🛑 Stop services: make docker-stop"
|
||||
@@ -1,63 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
set -e
|
||||
|
||||
echo "=========================================="
|
||||
echo " Local Mode Deployment"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
|
||||
# Check config.yaml
|
||||
if [ ! -f "config.yaml" ]; then
|
||||
echo "config.yaml does not exist. Generating it..."
|
||||
make config
|
||||
echo ""
|
||||
echo "⚠ Please edit config.yaml to configure your models and API keys"
|
||||
echo " Then run this script again"
|
||||
exit 1
|
||||
else
|
||||
echo "✓ config.yaml exists"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check the .env file
|
||||
if [ ! -f ".env" ]; then
|
||||
echo ".env does not exist. Copying it from the example..."
|
||||
if [ -f ".env.example" ]; then
|
||||
cp .env.example .env
|
||||
echo "✓ Created the .env file"
|
||||
else
|
||||
echo "⚠ .env.example does not exist. Please create the .env file manually"
|
||||
fi
|
||||
else
|
||||
echo "✓ .env file exists"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check dependencies
|
||||
echo "Checking dependencies..."
|
||||
make check
|
||||
echo ""
|
||||
|
||||
# Install dependencies
|
||||
echo "Installing dependencies..."
|
||||
make install
|
||||
echo ""
|
||||
|
||||
# Start services
|
||||
echo "Starting services (background mode)..."
|
||||
make dev-daemon
|
||||
echo ""
|
||||
|
||||
echo "=========================================="
|
||||
echo " Deployment Complete"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
echo "🌐 Access URL: http://localhost:2026"
|
||||
echo "📋 View logs:"
|
||||
echo " - logs/langgraph.log"
|
||||
echo " - logs/gateway.log"
|
||||
echo " - logs/frontend.log"
|
||||
echo " - logs/nginx.log"
|
||||
echo "🛑 Stop services: make stop"
|
||||
echo ""
|
||||
echo "Please wait 90-120 seconds for all services to start completely, then run the health check"
|
||||
@@ -1,70 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
set +e
|
||||
|
||||
echo "=========================================="
|
||||
echo " Frontend Page Smoke Check"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
|
||||
BASE_URL="${BASE_URL:-http://localhost:2026}"
|
||||
DOC_PATH="${DOC_PATH:-/en/docs}"
|
||||
|
||||
all_passed=true
|
||||
|
||||
check_status() {
|
||||
local name="$1"
|
||||
local url="$2"
|
||||
local expected_re="$3"
|
||||
|
||||
local status
|
||||
status="$(curl -s -o /dev/null -w "%{http_code}" -L "$url")"
|
||||
if echo "$status" | grep -Eq "$expected_re"; then
|
||||
echo "✓ $name ($url) -> $status"
|
||||
else
|
||||
echo "✗ $name ($url) -> $status (expected: $expected_re)"
|
||||
all_passed=false
|
||||
fi
|
||||
}
|
||||
|
||||
check_final_url() {
|
||||
local name="$1"
|
||||
local url="$2"
|
||||
local expected_path_re="$3"
|
||||
|
||||
local effective
|
||||
effective="$(curl -s -o /dev/null -w "%{url_effective}" -L "$url")"
|
||||
if echo "$effective" | grep -Eq "$expected_path_re"; then
|
||||
echo "✓ $name redirect target -> $effective"
|
||||
else
|
||||
echo "✗ $name redirect target -> $effective (expected path: $expected_path_re)"
|
||||
all_passed=false
|
||||
fi
|
||||
}
|
||||
|
||||
echo "1. Checking entry pages..."
|
||||
check_status "Landing page" "${BASE_URL}/" "200"
|
||||
check_status "Workspace redirect" "${BASE_URL}/workspace" "200|301|302|307|308"
|
||||
check_final_url "Workspace redirect" "${BASE_URL}/workspace" "/workspace/chats/"
|
||||
echo ""
|
||||
|
||||
echo "2. Checking key workspace routes..."
|
||||
check_status "New chat page" "${BASE_URL}/workspace/chats/new" "200"
|
||||
check_status "Chats list page" "${BASE_URL}/workspace/chats" "200"
|
||||
check_status "Agents gallery page" "${BASE_URL}/workspace/agents" "200"
|
||||
echo ""
|
||||
|
||||
echo "3. Checking docs route (optional)..."
|
||||
check_status "Docs page" "${BASE_URL}${DOC_PATH}" "200|404"
|
||||
echo ""
|
||||
|
||||
echo "=========================================="
|
||||
echo " Frontend Smoke Check Summary"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
if [ "$all_passed" = true ]; then
|
||||
echo "✅ Frontend smoke checks passed!"
|
||||
exit 0
|
||||
else
|
||||
echo "❌ Frontend smoke checks failed"
|
||||
exit 1
|
||||
fi
|
||||
@@ -1,125 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
set +e
|
||||
|
||||
echo "=========================================="
|
||||
echo " Service Health Check"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
|
||||
all_passed=true
|
||||
mode="${SMOKE_TEST_MODE:-auto}"
|
||||
summary_hint="make logs"
|
||||
|
||||
print_step() {
|
||||
echo "$1"
|
||||
}
|
||||
|
||||
check_http_status() {
|
||||
local name="$1"
|
||||
local url="$2"
|
||||
local expected_re="$3"
|
||||
local status
|
||||
|
||||
status="$(curl -s -o /dev/null -w "%{http_code}" "$url" 2>/dev/null)"
|
||||
if echo "$status" | grep -Eq "$expected_re"; then
|
||||
echo "✓ $name is accessible ($url -> $status)"
|
||||
else
|
||||
echo "✗ $name is not accessible ($url -> ${status:-000})"
|
||||
all_passed=false
|
||||
fi
|
||||
}
|
||||
|
||||
check_listen_port() {
|
||||
local name="$1"
|
||||
local port="$2"
|
||||
|
||||
if lsof -nP -iTCP:"$port" -sTCP:LISTEN >/dev/null 2>&1; then
|
||||
echo "✓ $name is listening on port $port"
|
||||
else
|
||||
echo "✗ $name is not listening on port $port"
|
||||
all_passed=false
|
||||
fi
|
||||
}
|
||||
|
||||
docker_available() {
|
||||
command -v docker >/dev/null 2>&1 && docker info >/dev/null 2>&1
|
||||
}
|
||||
|
||||
detect_mode() {
|
||||
case "$mode" in
|
||||
local|docker)
|
||||
echo "$mode"
|
||||
return
|
||||
;;
|
||||
esac
|
||||
|
||||
if docker_available && docker ps --format "{{.Names}}" | grep -q "deer-flow"; then
|
||||
echo "docker"
|
||||
else
|
||||
echo "local"
|
||||
fi
|
||||
}
|
||||
|
||||
mode="$(detect_mode)"
|
||||
|
||||
echo "Deployment mode: $mode"
|
||||
echo ""
|
||||
|
||||
if [ "$mode" = "docker" ]; then
|
||||
summary_hint="make docker-logs"
|
||||
print_step "1. Checking container status..."
|
||||
if docker ps --format "{{.Names}}" | grep -q "deer-flow"; then
|
||||
echo "✓ Containers are running:"
|
||||
docker ps --format " - {{.Names}} ({{.Status}})"
|
||||
else
|
||||
echo "✗ No DeerFlow-related containers are running"
|
||||
all_passed=false
|
||||
fi
|
||||
else
|
||||
summary_hint="logs/{langgraph,gateway,frontend,nginx}.log"
|
||||
print_step "1. Checking local service ports..."
|
||||
check_listen_port "Nginx" 2026
|
||||
check_listen_port "Frontend" 3000
|
||||
check_listen_port "Gateway" 8001
|
||||
check_listen_port "LangGraph" 2024
|
||||
fi
|
||||
echo ""
|
||||
|
||||
echo "2. Waiting for services to fully start (30 seconds)..."
|
||||
sleep 30
|
||||
echo ""
|
||||
|
||||
echo "3. Checking frontend service..."
|
||||
check_http_status "Frontend service" "http://localhost:2026" "200|301|302|307|308"
|
||||
echo ""
|
||||
|
||||
echo "4. Checking API Gateway..."
|
||||
health_response=$(curl -s http://localhost:2026/health 2>/dev/null)
|
||||
if [ $? -eq 0 ] && [ -n "$health_response" ]; then
|
||||
echo "✓ API Gateway health check passed"
|
||||
echo " Response: $health_response"
|
||||
else
|
||||
echo "✗ API Gateway health check failed"
|
||||
all_passed=false
|
||||
fi
|
||||
echo ""
|
||||
|
||||
echo "5. Checking LangGraph service..."
|
||||
check_http_status "LangGraph service" "http://localhost:2024/" "200|301|302|307|308|404"
|
||||
echo ""
|
||||
|
||||
echo "=========================================="
|
||||
echo " Health Check Summary"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
if [ "$all_passed" = true ]; then
|
||||
echo "✅ All checks passed!"
|
||||
echo ""
|
||||
echo "🌐 Application URL: http://localhost:2026"
|
||||
exit 0
|
||||
else
|
||||
echo "❌ Some checks failed"
|
||||
echo ""
|
||||
echo "Please review: $summary_hint"
|
||||
exit 1
|
||||
fi
|
||||
@@ -1,49 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
set -e
|
||||
|
||||
echo "=========================================="
|
||||
echo " Pulling the Latest Code"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
|
||||
# Check whether the current directory is a Git repository
|
||||
if [ ! -d ".git" ]; then
|
||||
echo "✗ The current directory is not a Git repository"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check Git status
|
||||
echo "Checking Git status..."
|
||||
if git status --porcelain | grep -q .; then
|
||||
echo "⚠ Uncommitted changes detected:"
|
||||
git status --short
|
||||
echo ""
|
||||
echo "Please commit or stash your changes before continuing"
|
||||
echo "Options:"
|
||||
echo " 1. git add . && git commit -m 'Save changes'"
|
||||
echo " 2. git stash (stash changes and restore them later)"
|
||||
echo " 3. git reset --hard HEAD (discard local changes - use with caution)"
|
||||
exit 1
|
||||
else
|
||||
echo "✓ Working tree is clean"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Fetch remote updates
|
||||
echo "Fetching remote updates..."
|
||||
git fetch origin main
|
||||
echo ""
|
||||
|
||||
# Pull the latest code
|
||||
echo "Pulling the latest code..."
|
||||
git pull origin main
|
||||
echo ""
|
||||
|
||||
# Show the latest commit
|
||||
echo "Latest commit:"
|
||||
git log -1 --oneline
|
||||
echo ""
|
||||
|
||||
echo "=========================================="
|
||||
echo " Code Update Complete"
|
||||
echo "=========================================="
|
||||
@@ -1,180 +0,0 @@
|
||||
# DeerFlow Smoke Test Report
|
||||
|
||||
**Test Date**: {{test_date}}
|
||||
**Test Environment**: {{test_environment}}
|
||||
**Deployment Mode**: Docker
|
||||
**Test Version**: {{git_commit}}
|
||||
|
||||
---
|
||||
|
||||
## Execution Summary
|
||||
|
||||
| Metric | Status |
|
||||
|------|------|
|
||||
| Total Test Phases | 6 |
|
||||
| Passed Phases | {{passed_stages}} |
|
||||
| Failed Phases | {{failed_stages}} |
|
||||
| Overall Conclusion | **{{overall_status}}** |
|
||||
|
||||
### Key Test Cases
|
||||
|
||||
| Case | Result | Details |
|
||||
|------|--------|---------|
|
||||
| Code update check | {{case_code_update}} | {{case_code_update_details}} |
|
||||
| Environment check | {{case_env_check}} | {{case_env_check_details}} |
|
||||
| Configuration preparation | {{case_config_prep}} | {{case_config_prep_details}} |
|
||||
| Deployment | {{case_deploy}} | {{case_deploy_details}} |
|
||||
| Health check | {{case_health_check}} | {{case_health_check_details}} |
|
||||
| Frontend routes | {{case_frontend_routes_overall}} | {{case_frontend_routes_details}} |
|
||||
|
||||
---
|
||||
|
||||
## Detailed Test Results
|
||||
|
||||
### Phase 1: Code Update Check
|
||||
|
||||
- [x] Confirm current directory - {{status_dir_check}}
|
||||
- [x] Check Git status - {{status_git_status}}
|
||||
- [x] Pull latest code - {{status_git_pull}}
|
||||
- [x] Confirm code update - {{status_git_verify}}
|
||||
|
||||
**Phase Status**: {{stage1_status}}
|
||||
|
||||
---
|
||||
|
||||
### Phase 2: Docker Environment Check
|
||||
|
||||
- [x] Docker version - {{status_docker_version}}
|
||||
- [x] Docker daemon - {{status_docker_daemon}}
|
||||
- [x] Docker Compose - {{status_docker_compose}}
|
||||
- [x] Port check - {{status_port_check}}
|
||||
|
||||
**Phase Status**: {{stage2_status}}
|
||||
|
||||
---
|
||||
|
||||
### Phase 3: Configuration Preparation
|
||||
|
||||
- [x] config.yaml - {{status_config_yaml}}
|
||||
- [x] .env file - {{status_env_file}}
|
||||
- [x] Model configuration - {{status_model_config}}
|
||||
|
||||
**Phase Status**: {{stage3_status}}
|
||||
|
||||
---
|
||||
|
||||
### Phase 4: Docker Deployment
|
||||
|
||||
- [x] docker-init - {{status_docker_init}}
|
||||
- [x] docker-start - {{status_docker_start}}
|
||||
- [x] Service startup wait - {{status_wait_startup}}
|
||||
|
||||
**Phase Status**: {{stage4_status}}
|
||||
|
||||
---
|
||||
|
||||
### Phase 5: Service Health Check
|
||||
|
||||
- [x] Container status - {{status_containers}}
|
||||
- [x] Frontend service - {{status_frontend}}
|
||||
- [x] API Gateway - {{status_api_gateway}}
|
||||
- [x] LangGraph service - {{status_langgraph}}
|
||||
|
||||
**Phase Status**: {{stage5_status}}
|
||||
|
||||
---
|
||||
|
||||
### Frontend Routes Smoke Results
|
||||
|
||||
| Route | Status | Details |
|
||||
|-------|--------|---------|
|
||||
| Landing `/` | {{landing_status}} | {{landing_details}} |
|
||||
| Workspace redirect `/workspace` | {{workspace_redirect_status}} | target {{workspace_redirect_target}} |
|
||||
| New chat `/workspace/chats/new` | {{new_chat_status}} | {{new_chat_details}} |
|
||||
| Chats list `/workspace/chats` | {{chats_list_status}} | {{chats_list_details}} |
|
||||
| Agents gallery `/workspace/agents` | {{agents_gallery_status}} | {{agents_gallery_details}} |
|
||||
| Docs `{{docs_path}}` | {{docs_status}} | {{docs_details}} |
|
||||
|
||||
**Summary**: {{frontend_routes_summary}}
|
||||
|
||||
---
|
||||
|
||||
### Phase 6: Test Report Generation
|
||||
|
||||
- [x] Result summary - {{status_summary}}
|
||||
- [x] Issue log - {{status_issues}}
|
||||
- [x] Report generation - {{status_report}}
|
||||
|
||||
**Phase Status**: {{stage6_status}}
|
||||
|
||||
---
|
||||
|
||||
## Issue Log
|
||||
|
||||
### Issue 1
|
||||
**Description**: {{issue1_description}}
|
||||
**Severity**: {{issue1_severity}}
|
||||
**Solution**: {{issue1_solution}}
|
||||
|
||||
---
|
||||
|
||||
## Environment Information
|
||||
|
||||
### Docker Version
|
||||
```text
|
||||
{{docker_version_output}}
|
||||
```
|
||||
|
||||
### Git Information
|
||||
```text
|
||||
Repository: {{git_repo}}
|
||||
Branch: {{git_branch}}
|
||||
Commit: {{git_commit}}
|
||||
Commit Message: {{git_commit_message}}
|
||||
```
|
||||
|
||||
### Configuration Summary
|
||||
- config.yaml exists: {{config_exists}}
|
||||
- .env file exists: {{env_exists}}
|
||||
- Number of configured models: {{model_count}}
|
||||
|
||||
---
|
||||
|
||||
## Container Status
|
||||
|
||||
| Container Name | Status | Uptime |
|
||||
|----------|------|----------|
|
||||
| deer-flow-nginx | {{nginx_status}} | {{nginx_uptime}} |
|
||||
| deer-flow-frontend | {{frontend_status}} | {{frontend_uptime}} |
|
||||
| deer-flow-gateway | {{gateway_status}} | {{gateway_uptime}} |
|
||||
| deer-flow-langgraph | {{langgraph_status}} | {{langgraph_uptime}} |
|
||||
|
||||
---
|
||||
|
||||
## Recommendations and Next Steps
|
||||
|
||||
### If the Test Passes
|
||||
1. [ ] Visit http://localhost:2026 to start using DeerFlow
|
||||
2. [ ] Configure your preferred model if it is not configured yet
|
||||
3. [ ] Explore available skills
|
||||
4. [ ] Refer to the documentation to learn more features
|
||||
|
||||
### If the Test Fails
|
||||
1. [ ] Review references/troubleshooting.md for common solutions
|
||||
2. [ ] Check Docker logs: `make docker-logs`
|
||||
3. [ ] Verify configuration file format and content
|
||||
4. [ ] If needed, fully reset the environment: `make clean && make config && make docker-init && make docker-start`
|
||||
|
||||
---
|
||||
|
||||
## Appendix
|
||||
|
||||
### Full Logs
|
||||
{{full_logs}}
|
||||
|
||||
### Tester
|
||||
{{tester_name}}
|
||||
|
||||
---
|
||||
|
||||
*Report generated at: {{report_time}}*
|
||||
@@ -1,185 +0,0 @@
|
||||
# DeerFlow Smoke Test Report
|
||||
|
||||
**Test Date**: {{test_date}}
|
||||
**Test Environment**: {{test_environment}}
|
||||
**Deployment Mode**: Local
|
||||
**Test Version**: {{git_commit}}
|
||||
|
||||
---
|
||||
|
||||
## Execution Summary
|
||||
|
||||
| Metric | Status |
|
||||
|------|------|
|
||||
| Total Test Phases | 6 |
|
||||
| Passed Phases | {{passed_stages}} |
|
||||
| Failed Phases | {{failed_stages}} |
|
||||
| Overall Conclusion | **{{overall_status}}** |
|
||||
|
||||
### Key Test Cases
|
||||
|
||||
| Case | Result | Details |
|
||||
|------|--------|---------|
|
||||
| Code update check | {{case_code_update}} | {{case_code_update_details}} |
|
||||
| Environment check | {{case_env_check}} | {{case_env_check_details}} |
|
||||
| Configuration preparation | {{case_config_prep}} | {{case_config_prep_details}} |
|
||||
| Deployment | {{case_deploy}} | {{case_deploy_details}} |
|
||||
| Health check | {{case_health_check}} | {{case_health_check_details}} |
|
||||
| Frontend routes | {{case_frontend_routes_overall}} | {{case_frontend_routes_details}} |
|
||||
|
||||
---
|
||||
|
||||
## Detailed Test Results
|
||||
|
||||
### Phase 1: Code Update Check
|
||||
|
||||
- [x] Confirm current directory - {{status_dir_check}}
|
||||
- [x] Check Git status - {{status_git_status}}
|
||||
- [x] Pull latest code - {{status_git_pull}}
|
||||
- [x] Confirm code update - {{status_git_verify}}
|
||||
|
||||
**Phase Status**: {{stage1_status}}
|
||||
|
||||
---
|
||||
|
||||
### Phase 2: Local Environment Check
|
||||
|
||||
- [x] Node.js version - {{status_node_version}}
|
||||
- [x] pnpm - {{status_pnpm}}
|
||||
- [x] uv - {{status_uv}}
|
||||
- [x] nginx - {{status_nginx}}
|
||||
- [x] Port check - {{status_port_check}}
|
||||
|
||||
**Phase Status**: {{stage2_status}}
|
||||
|
||||
---
|
||||
|
||||
### Phase 3: Configuration Preparation
|
||||
|
||||
- [x] config.yaml - {{status_config_yaml}}
|
||||
- [x] .env file - {{status_env_file}}
|
||||
- [x] Model configuration - {{status_model_config}}
|
||||
|
||||
**Phase Status**: {{stage3_status}}
|
||||
|
||||
---
|
||||
|
||||
### Phase 4: Local Deployment
|
||||
|
||||
- [x] make check - {{status_make_check}}
|
||||
- [x] make install - {{status_make_install}}
|
||||
- [x] make dev-daemon / make dev - {{status_local_start}}
|
||||
- [x] Service startup wait - {{status_wait_startup}}
|
||||
|
||||
**Phase Status**: {{stage4_status}}
|
||||
|
||||
---
|
||||
|
||||
### Phase 5: Service Health Check
|
||||
|
||||
- [x] Process status - {{status_processes}}
|
||||
- [x] Frontend service - {{status_frontend}}
|
||||
- [x] API Gateway - {{status_api_gateway}}
|
||||
- [x] LangGraph service - {{status_langgraph}}
|
||||
|
||||
**Phase Status**: {{stage5_status}}
|
||||
|
||||
---
|
||||
|
||||
### Frontend Routes Smoke Results
|
||||
|
||||
| Route | Status | Details |
|
||||
|-------|--------|---------|
|
||||
| Landing `/` | {{landing_status}} | {{landing_details}} |
|
||||
| Workspace redirect `/workspace` | {{workspace_redirect_status}} | target {{workspace_redirect_target}} |
|
||||
| New chat `/workspace/chats/new` | {{new_chat_status}} | {{new_chat_details}} |
|
||||
| Chats list `/workspace/chats` | {{chats_list_status}} | {{chats_list_details}} |
|
||||
| Agents gallery `/workspace/agents` | {{agents_gallery_status}} | {{agents_gallery_details}} |
|
||||
| Docs `{{docs_path}}` | {{docs_status}} | {{docs_details}} |
|
||||
|
||||
**Summary**: {{frontend_routes_summary}}
|
||||
|
||||
---
|
||||
|
||||
### Phase 6: Test Report Generation
|
||||
|
||||
- [x] Result summary - {{status_summary}}
|
||||
- [x] Issue log - {{status_issues}}
|
||||
- [x] Report generation - {{status_report}}
|
||||
|
||||
**Phase Status**: {{stage6_status}}
|
||||
|
||||
---
|
||||
|
||||
## Issue Log
|
||||
|
||||
### Issue 1
|
||||
**Description**: {{issue1_description}}
|
||||
**Severity**: {{issue1_severity}}
|
||||
**Solution**: {{issue1_solution}}
|
||||
|
||||
---
|
||||
|
||||
## Environment Information
|
||||
|
||||
### Local Dependency Versions
|
||||
```text
|
||||
Node.js: {{node_version_output}}
|
||||
pnpm: {{pnpm_version_output}}
|
||||
uv: {{uv_version_output}}
|
||||
nginx: {{nginx_version_output}}
|
||||
```
|
||||
|
||||
### Git Information
|
||||
```text
|
||||
Repository: {{git_repo}}
|
||||
Branch: {{git_branch}}
|
||||
Commit: {{git_commit}}
|
||||
Commit Message: {{git_commit_message}}
|
||||
```
|
||||
|
||||
### Configuration Summary
|
||||
- config.yaml exists: {{config_exists}}
|
||||
- .env file exists: {{env_exists}}
|
||||
- Number of configured models: {{model_count}}
|
||||
|
||||
---
|
||||
|
||||
## Local Service Status
|
||||
|
||||
| Service | Status | Endpoint |
|
||||
|---------|--------|----------|
|
||||
| Nginx | {{nginx_status}} | {{nginx_endpoint}} |
|
||||
| Frontend | {{frontend_status}} | {{frontend_endpoint}} |
|
||||
| Gateway | {{gateway_status}} | {{gateway_endpoint}} |
|
||||
| LangGraph | {{langgraph_status}} | {{langgraph_endpoint}} |
|
||||
|
||||
---
|
||||
|
||||
## Recommendations and Next Steps
|
||||
|
||||
### If the Test Passes
|
||||
1. [ ] Visit http://localhost:2026 to start using DeerFlow
|
||||
2. [ ] Configure your preferred model if it is not configured yet
|
||||
3. [ ] Explore available skills
|
||||
4. [ ] Refer to the documentation to learn more features
|
||||
|
||||
### If the Test Fails
|
||||
1. [ ] Review references/troubleshooting.md for common solutions
|
||||
2. [ ] Check local logs: `logs/{langgraph,gateway,frontend,nginx}.log`
|
||||
3. [ ] Verify configuration file format and content
|
||||
4. [ ] If needed, fully reset the environment: `make stop && make clean && make install && make dev-daemon`
|
||||
|
||||
---
|
||||
|
||||
## Appendix
|
||||
|
||||
### Full Logs
|
||||
{{full_logs}}
|
||||
|
||||
### Tester
|
||||
{{tester_name}}
|
||||
|
||||
---
|
||||
|
||||
*Report generated at: {{report_time}}*
|
||||
+1
-26
@@ -4,8 +4,6 @@ TAVILY_API_KEY=your-tavily-api-key
|
||||
# Jina API Key
|
||||
JINA_API_KEY=your-jina-api-key
|
||||
|
||||
# InfoQuest API Key
|
||||
INFOQUEST_API_KEY=your-infoquest-api-key
|
||||
# CORS Origins (comma-separated) - e.g., http://localhost:3000,http://localhost:3001
|
||||
# CORS_ORIGINS=http://localhost:3000
|
||||
|
||||
@@ -15,27 +13,4 @@ INFOQUEST_API_KEY=your-infoquest-api-key
|
||||
# OPENAI_API_KEY=your-openai-api-key
|
||||
# GEMINI_API_KEY=your-gemini-api-key
|
||||
# DEEPSEEK_API_KEY=your-deepseek-api-key
|
||||
# NOVITA_API_KEY=your-novita-api-key # OpenAI-compatible, see https://novita.ai
|
||||
# MINIMAX_API_KEY=your-minimax-api-key # OpenAI-compatible, see https://platform.minimax.io
|
||||
# VLLM_API_KEY=your-vllm-api-key # OpenAI-compatible
|
||||
# FEISHU_APP_ID=your-feishu-app-id
|
||||
# FEISHU_APP_SECRET=your-feishu-app-secret
|
||||
|
||||
# SLACK_BOT_TOKEN=your-slack-bot-token
|
||||
# SLACK_APP_TOKEN=your-slack-app-token
|
||||
# TELEGRAM_BOT_TOKEN=your-telegram-bot-token
|
||||
|
||||
# Enable LangSmith to monitor and debug your LLM calls, agent runs, and tool executions.
|
||||
# LANGSMITH_TRACING=true
|
||||
# LANGSMITH_ENDPOINT=https://api.smith.langchain.com
|
||||
# LANGSMITH_API_KEY=your-langsmith-api-key
|
||||
# LANGSMITH_PROJECT=your-langsmith-project
|
||||
|
||||
# GitHub API Token
|
||||
# GITHUB_TOKEN=your-github-token
|
||||
|
||||
# Database (only needed when config.yaml has database.backend: postgres)
|
||||
# DATABASE_URL=postgresql://deerflow:password@localhost:5432/deerflow
|
||||
#
|
||||
# WECOM_BOT_ID=your-wecom-bot-id
|
||||
# WECOM_BOT_SECRET=your-wecom-bot-secret
|
||||
# NOVITA_API_KEY=your-novita-api-key # OpenAI-compatible, see https://novita.ai
|
||||
@@ -1,128 +0,0 @@
|
||||
name: Runtime Information
|
||||
description: Report runtime/environment details to help reproduce an issue.
|
||||
title: "[runtime] "
|
||||
labels:
|
||||
- needs-triage
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
Thanks for sharing runtime details.
|
||||
Complete this form so maintainers can quickly reproduce and diagnose the problem.
|
||||
|
||||
- type: input
|
||||
id: summary
|
||||
attributes:
|
||||
label: Problem summary
|
||||
description: Short summary of the issue.
|
||||
placeholder: e.g. make dev fails to start gateway service
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: expected
|
||||
attributes:
|
||||
label: Expected behavior
|
||||
placeholder: What did you expect to happen?
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: actual
|
||||
attributes:
|
||||
label: Actual behavior
|
||||
placeholder: What happened instead? Include key error lines.
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: dropdown
|
||||
id: os
|
||||
attributes:
|
||||
label: Operating system
|
||||
options:
|
||||
- macOS
|
||||
- Linux
|
||||
- Windows
|
||||
- Other
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: input
|
||||
id: platform_details
|
||||
attributes:
|
||||
label: Platform details
|
||||
description: Add architecture and shell if relevant.
|
||||
placeholder: e.g. arm64, zsh
|
||||
|
||||
- type: input
|
||||
id: python_version
|
||||
attributes:
|
||||
label: Python version
|
||||
placeholder: e.g. Python 3.12.9
|
||||
|
||||
- type: input
|
||||
id: node_version
|
||||
attributes:
|
||||
label: Node.js version
|
||||
placeholder: e.g. v23.11.0
|
||||
|
||||
- type: input
|
||||
id: pnpm_version
|
||||
attributes:
|
||||
label: pnpm version
|
||||
placeholder: e.g. 10.26.2
|
||||
|
||||
- type: input
|
||||
id: uv_version
|
||||
attributes:
|
||||
label: uv version
|
||||
placeholder: e.g. 0.7.20
|
||||
|
||||
- type: dropdown
|
||||
id: run_mode
|
||||
attributes:
|
||||
label: How are you running DeerFlow?
|
||||
options:
|
||||
- Local (make dev)
|
||||
- Docker (make docker-dev)
|
||||
- CI
|
||||
- Other
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: reproduce
|
||||
attributes:
|
||||
label: Reproduction steps
|
||||
description: Provide exact commands and sequence.
|
||||
placeholder: |
|
||||
1. make check
|
||||
2. make install
|
||||
3. make dev
|
||||
4. ...
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: logs
|
||||
attributes:
|
||||
label: Relevant logs
|
||||
description: Paste key lines from logs (for example logs/gateway.log, logs/frontend.log).
|
||||
render: shell
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: git_info
|
||||
attributes:
|
||||
label: Git state
|
||||
description: Share output of git branch and latest commit SHA.
|
||||
placeholder: |
|
||||
branch: feature/my-branch
|
||||
commit: abcdef1
|
||||
|
||||
- type: textarea
|
||||
id: additional
|
||||
attributes:
|
||||
label: Additional context
|
||||
description: Add anything else that might help triage.
|
||||
@@ -1,213 +0,0 @@
|
||||
# Copilot Onboarding Instructions for DeerFlow
|
||||
|
||||
Use this file as the default operating guide for this repository. Follow it first, and only search the codebase when this file is incomplete or incorrect.
|
||||
|
||||
## 1) Repository Summary
|
||||
|
||||
DeerFlow is a full-stack "super agent harness".
|
||||
|
||||
- Backend: Python 3.12, LangGraph + FastAPI gateway, sandbox/tool system, memory, MCP integration.
|
||||
- Frontend: Next.js 16 + React 19 + TypeScript + pnpm.
|
||||
- Local dev entrypoint: root `Makefile` starts backend + frontend + nginx on `http://localhost:2026`.
|
||||
- Docker dev entrypoint: `make docker-*` (mode-aware provisioner startup from `config.yaml`).
|
||||
|
||||
Current repo footprint is medium-large (backend service, frontend app, docker stack, skills library, docs).
|
||||
|
||||
## 2) Runtime and Toolchain Requirements
|
||||
|
||||
Validated in this repo on macOS:
|
||||
|
||||
- Node.js `>=22` (validated with Node `23.11.0`)
|
||||
- pnpm (repo expects lockfile generated by pnpm 10; validated with pnpm `10.26.2` and `10.15.0`)
|
||||
- Python `>=3.12` (CI uses `3.12`)
|
||||
- `uv` (validated with `0.7.20`)
|
||||
- `nginx` (required for `make dev` unified local endpoint)
|
||||
|
||||
Always run from repo root unless a command explicitly says otherwise.
|
||||
|
||||
## 3) Build/Test/Lint/Run - Verified Command Sequences
|
||||
|
||||
These were executed and validated in this repository.
|
||||
|
||||
### A. Bootstrap and install
|
||||
|
||||
1. Check prerequisites:
|
||||
|
||||
```bash
|
||||
make check
|
||||
```
|
||||
|
||||
Observed: passes when required tools are installed.
|
||||
|
||||
2. Install dependencies (recommended order: backend then frontend, as implemented by `make install`):
|
||||
|
||||
```bash
|
||||
make install
|
||||
```
|
||||
|
||||
### B. Backend CI-equivalent validation
|
||||
|
||||
Run from `backend/`:
|
||||
|
||||
```bash
|
||||
make lint
|
||||
make test
|
||||
```
|
||||
|
||||
Validated results:
|
||||
|
||||
- `make lint`: pass (`ruff check .`)
|
||||
- `make test`: pass (`277 passed, 15 warnings in ~76.6s`)
|
||||
|
||||
CI parity:
|
||||
|
||||
- `.github/workflows/backend-unit-tests.yml` runs on pull requests.
|
||||
- CI executes `uv sync --group dev`, then `make lint`, then `make test` in `backend/`.
|
||||
|
||||
### C. Frontend validation
|
||||
|
||||
Run from `frontend/`.
|
||||
|
||||
Recommended reliable sequence:
|
||||
|
||||
```bash
|
||||
pnpm lint
|
||||
pnpm typecheck
|
||||
BETTER_AUTH_SECRET=local-dev-secret pnpm build
|
||||
```
|
||||
|
||||
Observed failure modes and workarounds:
|
||||
|
||||
- `pnpm build` fails without `BETTER_AUTH_SECRET` in production-mode env validation.
|
||||
- Workaround: set `BETTER_AUTH_SECRET` (best) or set `SKIP_ENV_VALIDATION=1`.
|
||||
- Even with `SKIP_ENV_VALIDATION=1`, Better Auth can still warn/error in logs about default secret; prefer setting a real non-default secret.
|
||||
- `pnpm check` currently fails (`next lint` invocation is incompatible here and resolves to an invalid directory). Do not rely on `pnpm check`; run `pnpm lint` and `pnpm typecheck` explicitly.
|
||||
|
||||
### D. Run locally (all services)
|
||||
|
||||
From root:
|
||||
|
||||
```bash
|
||||
make dev
|
||||
```
|
||||
|
||||
Behavior:
|
||||
|
||||
- Stops existing local services first.
|
||||
- Starts LangGraph (`2024`), Gateway (`8001`), Frontend (`3000`), nginx (`2026`).
|
||||
- Unified app endpoint: `http://localhost:2026`.
|
||||
- Logs: `logs/langgraph.log`, `logs/gateway.log`, `logs/frontend.log`, `logs/nginx.log`.
|
||||
|
||||
Stop services:
|
||||
|
||||
```bash
|
||||
make stop
|
||||
```
|
||||
|
||||
If tool sessions/timeouts interrupt `make dev`, run `make stop` again to ensure cleanup.
|
||||
|
||||
### E. Config bootstrap
|
||||
|
||||
From root:
|
||||
|
||||
```bash
|
||||
make config
|
||||
```
|
||||
|
||||
Important behavior:
|
||||
|
||||
- This intentionally aborts if `config.yaml` (or `config.yml`/`configure.yml`) already exists.
|
||||
- Use `make config` only for first-time setup in a clean clone.
|
||||
|
||||
## 4) Command Order That Minimizes Failures
|
||||
|
||||
Use this exact order for local code changes:
|
||||
|
||||
1. `make check`
|
||||
2. `make install` (if frontend fails with proxy errors, rerun frontend install with proxy vars unset)
|
||||
3. Backend checks: `cd backend && make lint && make test`
|
||||
4. Frontend checks: `cd frontend && pnpm lint && pnpm typecheck`
|
||||
5. Frontend build (if UI changes or release-sensitive changes): `BETTER_AUTH_SECRET=... pnpm build`
|
||||
|
||||
Always run backend lint/tests before opening PRs because that is what CI enforces.
|
||||
|
||||
## 5) Project Layout and Architecture (High-Value Paths)
|
||||
|
||||
Root-level orchestration and config:
|
||||
|
||||
- `Makefile` - main local/dev/docker command entrypoints
|
||||
- `config.example.yaml` - primary app config template
|
||||
- `config.yaml` - local active config (gitignored)
|
||||
- `docker/docker-compose-dev.yaml` - Docker dev topology
|
||||
- `.github/workflows/backend-unit-tests.yml` - PR validation workflow
|
||||
|
||||
Backend core:
|
||||
|
||||
- `backend/packages/harness/deerflow/agents/` - lead agent, middleware chain, memory
|
||||
- `backend/app/gateway/` - FastAPI gateway API
|
||||
- `backend/packages/harness/deerflow/sandbox/` - sandbox provider + tool wrappers
|
||||
- `backend/packages/harness/deerflow/subagents/` - subagent registry/execution
|
||||
- `backend/packages/harness/deerflow/mcp/` - MCP integration
|
||||
- `backend/langgraph.json` - graph entrypoint (`deerflow.agents:make_lead_agent`)
|
||||
- `backend/pyproject.toml` - Python deps and `requires-python`
|
||||
- `backend/ruff.toml` - lint/format policy
|
||||
- `backend/tests/` - backend unit and integration-like tests
|
||||
|
||||
Frontend core:
|
||||
|
||||
- `frontend/src/app/` - Next.js routes/pages
|
||||
- `frontend/src/components/` - UI components
|
||||
- `frontend/src/core/` - app logic (threads, tools, API, models)
|
||||
- `frontend/src/env.js` - env schema/validation (critical for build behavior)
|
||||
- `frontend/package.json` - scripts/deps
|
||||
- `frontend/eslint.config.js` - lint rules
|
||||
- `frontend/tsconfig.json` - TS config
|
||||
|
||||
Skills and assets:
|
||||
|
||||
- `skills/public/` - built-in skill packs loaded by agent runtime
|
||||
|
||||
## 6) Pre-Checkin / Validation Expectations
|
||||
|
||||
Before submitting changes, run at minimum:
|
||||
|
||||
- Backend: `cd backend && make lint && make test`
|
||||
- Frontend (if touched): `cd frontend && pnpm lint && pnpm typecheck`
|
||||
- Frontend build when changing env/auth/routing/build-sensitive files: `BETTER_AUTH_SECRET=... pnpm build`
|
||||
|
||||
If touching orchestration/config (`Makefile`, `docker/*`, `config*.yaml`), also run `make dev` and verify the four services start.
|
||||
|
||||
## 7) Non-Obvious Dependencies and Gotchas
|
||||
|
||||
- Proxy env vars can silently break frontend network operations (`pnpm install`/registry access).
|
||||
- `BETTER_AUTH_SECRET` is effectively required for reliable frontend production build validation.
|
||||
- Next.js may warn about multiple lockfiles and workspace root inference; this is currently a warning, not a build blocker.
|
||||
- `make config` is non-idempotent by design when config already exists.
|
||||
- `make dev` includes process cleanup and can emit shutdown logs/noise if interrupted; this is expected.
|
||||
|
||||
## 8) Root Inventory (quick reference)
|
||||
|
||||
Important root entries:
|
||||
|
||||
- `.github/`
|
||||
- `backend/`
|
||||
- `frontend/`
|
||||
- `docker/`
|
||||
- `skills/`
|
||||
- `scripts/`
|
||||
- `docs/`
|
||||
- `README.md`
|
||||
- `CONTRIBUTING.md`
|
||||
- `Makefile`
|
||||
- `config.example.yaml`
|
||||
- `extensions_config.example.json`
|
||||
|
||||
## 9) Instruction Priority
|
||||
|
||||
Trust this onboarding guide first.
|
||||
|
||||
Only do broad repo searches (`grep/find/code search`) when:
|
||||
|
||||
- you need file-level implementation details not listed here,
|
||||
- a command here fails and you need updated replacement behavior,
|
||||
- or CI/workflow definitions have changed since this file was written.
|
||||
@@ -1,8 +1,6 @@
|
||||
name: Unit Tests
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [ 'main' ]
|
||||
pull_request:
|
||||
types: [opened, synchronize, reopened, ready_for_review]
|
||||
|
||||
@@ -10,9 +8,6 @@ concurrency:
|
||||
group: unit-tests-${{ github.event.pull_request.number || github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
backend-unit-tests:
|
||||
if: github.event.pull_request.draft == false
|
||||
@@ -35,6 +30,10 @@ jobs:
|
||||
working-directory: backend
|
||||
run: uv sync --group dev
|
||||
|
||||
- name: Lint backend
|
||||
working-directory: backend
|
||||
run: make lint
|
||||
|
||||
- name: Run unit tests of backend
|
||||
working-directory: backend
|
||||
run: make test
|
||||
|
||||
@@ -1,74 +0,0 @@
|
||||
name: Lint Check
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [ 'main' ]
|
||||
pull_request:
|
||||
branches: [ '*' ]
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
lint:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v7
|
||||
|
||||
- name: Install dependencies
|
||||
working-directory: backend
|
||||
run: |
|
||||
uv sync --group dev
|
||||
|
||||
- name: Lint backend
|
||||
working-directory: backend
|
||||
run: make lint
|
||||
|
||||
lint-frontend:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '22'
|
||||
|
||||
- name: Enable Corepack
|
||||
run: corepack enable
|
||||
|
||||
- name: Use pinned pnpm version
|
||||
run: corepack prepare pnpm@10.26.2 --activate
|
||||
|
||||
- name: Install frontend dependencies
|
||||
run: |
|
||||
cd frontend
|
||||
pnpm install --frozen-lockfile
|
||||
|
||||
- name: Check frontend formatting
|
||||
run: |
|
||||
cd frontend
|
||||
pnpm format
|
||||
|
||||
- name: Run frontend linting
|
||||
run: |
|
||||
cd frontend
|
||||
pnpm lint
|
||||
|
||||
- name: Check TypeScript types
|
||||
run: |
|
||||
cd frontend
|
||||
pnpm typecheck
|
||||
|
||||
- name: Build frontend
|
||||
run: |
|
||||
cd frontend
|
||||
BETTER_AUTH_SECRET=local-dev-secret pnpm build
|
||||
@@ -1,7 +1,5 @@
|
||||
# DeerFlow docker image cache
|
||||
docker/.cache/
|
||||
# oh-my-claudecode state
|
||||
.omc/
|
||||
# OS generated files
|
||||
.DS_Store
|
||||
*.local
|
||||
@@ -50,10 +48,3 @@ sandbox_image_cache.tar
|
||||
|
||||
# ignore the legacy `web` folder
|
||||
web/
|
||||
|
||||
# Deployment artifacts
|
||||
backend/Dockerfile.langgraph
|
||||
config.yaml.bak
|
||||
.playwright-mcp
|
||||
.gstack/
|
||||
.worktrees
|
||||
|
||||
@@ -1,128 +0,0 @@
|
||||
# Contributor Covenant Code of Conduct
|
||||
|
||||
## Our Pledge
|
||||
|
||||
We as members, contributors, and leaders pledge to make participation in our
|
||||
community a harassment-free experience for everyone, regardless of age, body
|
||||
size, visible or invisible disability, ethnicity, sex characteristics, gender
|
||||
identity and expression, level of experience, education, socio-economic status,
|
||||
nationality, personal appearance, race, religion, or sexual identity
|
||||
and orientation.
|
||||
|
||||
We pledge to act and interact in ways that contribute to an open, welcoming,
|
||||
diverse, inclusive, and healthy community.
|
||||
|
||||
## Our Standards
|
||||
|
||||
Examples of behavior that contributes to a positive environment for our
|
||||
community include:
|
||||
|
||||
* Demonstrating empathy and kindness toward other people
|
||||
* Being respectful of differing opinions, viewpoints, and experiences
|
||||
* Giving and gracefully accepting constructive feedback
|
||||
* Accepting responsibility and apologizing to those affected by our mistakes,
|
||||
and learning from the experience
|
||||
* Focusing on what is best not just for us as individuals, but for the
|
||||
overall community
|
||||
|
||||
Examples of unacceptable behavior include:
|
||||
|
||||
* The use of sexualized language or imagery, and sexual attention or
|
||||
advances of any kind
|
||||
* Trolling, insulting or derogatory comments, and personal or political attacks
|
||||
* Public or private harassment
|
||||
* Publishing others' private information, such as a physical or email
|
||||
address, without their explicit permission
|
||||
* Other conduct which could reasonably be considered inappropriate in a
|
||||
professional setting
|
||||
|
||||
## Enforcement Responsibilities
|
||||
|
||||
Community leaders are responsible for clarifying and enforcing our standards of
|
||||
acceptable behavior and will take appropriate and fair corrective action in
|
||||
response to any behavior that they deem inappropriate, threatening, offensive,
|
||||
or harmful.
|
||||
|
||||
Community leaders have the right and responsibility to remove, edit, or reject
|
||||
comments, commits, code, wiki edits, issues, and other contributions that are
|
||||
not aligned to this Code of Conduct, and will communicate reasons for moderation
|
||||
decisions when appropriate.
|
||||
|
||||
## Scope
|
||||
|
||||
This Code of Conduct applies within all community spaces, and also applies when
|
||||
an individual is officially representing the community in public spaces.
|
||||
Examples of representing our community include using an official e-mail address,
|
||||
posting via an official social media account, or acting as an appointed
|
||||
representative at an online or offline event.
|
||||
|
||||
## Enforcement
|
||||
|
||||
Instances of abusive, harassing, or otherwise unacceptable behavior may be
|
||||
reported to the community leaders responsible for enforcement at
|
||||
willem.jiang@gmail.com.
|
||||
All complaints will be reviewed and investigated promptly and fairly.
|
||||
|
||||
All community leaders are obligated to respect the privacy and security of the
|
||||
reporter of any incident.
|
||||
|
||||
## Enforcement Guidelines
|
||||
|
||||
Community leaders will follow these Community Impact Guidelines in determining
|
||||
the consequences for any action they deem in violation of this Code of Conduct:
|
||||
|
||||
### 1. Correction
|
||||
|
||||
**Community Impact**: Use of inappropriate language or other behavior deemed
|
||||
unprofessional or unwelcome in the community.
|
||||
|
||||
**Consequence**: A private, written warning from community leaders, providing
|
||||
clarity around the nature of the violation and an explanation of why the
|
||||
behavior was inappropriate. A public apology may be requested.
|
||||
|
||||
### 2. Warning
|
||||
|
||||
**Community Impact**: A violation through a single incident or series
|
||||
of actions.
|
||||
|
||||
**Consequence**: A warning with consequences for continued behavior. No
|
||||
interaction with the people involved, including unsolicited interaction with
|
||||
those enforcing the Code of Conduct, for a specified period of time. This
|
||||
includes avoiding interactions in community spaces as well as external channels
|
||||
like social media. Violating these terms may lead to a temporary or
|
||||
permanent ban.
|
||||
|
||||
### 3. Temporary Ban
|
||||
|
||||
**Community Impact**: A serious violation of community standards, including
|
||||
sustained inappropriate behavior.
|
||||
|
||||
**Consequence**: A temporary ban from any sort of interaction or public
|
||||
communication with the community for a specified period of time. No public or
|
||||
private interaction with the people involved, including unsolicited interaction
|
||||
with those enforcing the Code of Conduct, is allowed during this period.
|
||||
Violating these terms may lead to a permanent ban.
|
||||
|
||||
### 4. Permanent Ban
|
||||
|
||||
**Community Impact**: Demonstrating a pattern of violation of community
|
||||
standards, including sustained inappropriate behavior, harassment of an
|
||||
individual, or aggression toward or disparagement of classes of individuals.
|
||||
|
||||
**Consequence**: A permanent ban from any sort of public interaction within
|
||||
the community.
|
||||
|
||||
## Attribution
|
||||
|
||||
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
|
||||
version 2.0, available at
|
||||
https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
|
||||
|
||||
Community Impact Guidelines were inspired by [Mozilla's code of conduct
|
||||
enforcement ladder](https://github.com/mozilla/diversity).
|
||||
|
||||
[homepage]: https://www.contributor-covenant.org
|
||||
|
||||
For answers to common questions about this code of conduct, see the FAQ at
|
||||
https://www.contributor-covenant.org/faq. Translations are available at
|
||||
https://www.contributor-covenant.org/translations.
|
||||
+8
-73
@@ -70,59 +70,6 @@ make docker-logs-frontend
|
||||
make docker-logs-gateway
|
||||
```
|
||||
|
||||
If Docker builds are slow in your network, you can override the default package registries before running `make docker-init` or `make docker-start`:
|
||||
|
||||
```bash
|
||||
export UV_INDEX_URL=https://pypi.org/simple
|
||||
export NPM_REGISTRY=https://registry.npmjs.org
|
||||
```
|
||||
|
||||
#### Recommended host resources
|
||||
|
||||
Use these as practical starting points for development and review environments:
|
||||
|
||||
| Scenario | Starting point | Recommended | Notes |
|
||||
|---------|-----------|------------|-------|
|
||||
| `make dev` on one machine | 4 vCPU, 8 GB RAM | 8 vCPU, 16 GB RAM | Best when DeerFlow uses hosted model APIs. |
|
||||
| `make docker-start` review environment | 4 vCPU, 8 GB RAM | 8 vCPU, 16 GB RAM | Docker image builds and sandbox containers need extra headroom. |
|
||||
| Shared Linux test server | 8 vCPU, 16 GB RAM | 16 vCPU, 32 GB RAM | Prefer this for heavier multi-agent runs or multiple reviewers. |
|
||||
|
||||
`2 vCPU / 4 GB` environments often fail to start reliably or become unresponsive under normal DeerFlow workloads.
|
||||
|
||||
#### Linux: Docker daemon permission denied
|
||||
|
||||
If `make docker-init`, `make docker-start`, or `make docker-stop` fails on Linux with an error like below, your current user likely does not have permission to access the Docker daemon socket:
|
||||
|
||||
```text
|
||||
unable to get image 'deer-flow-dev-langgraph': permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock
|
||||
```
|
||||
|
||||
Recommended fix: add your current user to the `docker` group so Docker commands work without `sudo`.
|
||||
|
||||
1. Confirm the `docker` group exists:
|
||||
```bash
|
||||
getent group docker
|
||||
```
|
||||
2. Add your current user to the `docker` group:
|
||||
```bash
|
||||
sudo usermod -aG docker $USER
|
||||
```
|
||||
3. Apply the new group membership. The most reliable option is to log out completely and then log back in. If you want to refresh the current shell session instead, run:
|
||||
```bash
|
||||
newgrp docker
|
||||
```
|
||||
4. Verify Docker access:
|
||||
```bash
|
||||
docker ps
|
||||
```
|
||||
5. Retry the DeerFlow command:
|
||||
```bash
|
||||
make docker-stop
|
||||
make docker-start
|
||||
```
|
||||
|
||||
If `docker ps` still reports a permission error after `usermod`, fully log out and log back in before retrying.
|
||||
|
||||
#### Docker Architecture
|
||||
|
||||
```
|
||||
@@ -269,26 +216,15 @@ Nginx (port 2026) ← Unified entry point
|
||||
|
||||
2. **Make your changes** with hot-reload enabled
|
||||
|
||||
3. **Format and lint your code** (CI will reject unformatted code):
|
||||
```bash
|
||||
# Backend
|
||||
cd backend
|
||||
make format # ruff check --fix + ruff format
|
||||
3. **Test your changes** thoroughly
|
||||
|
||||
# Frontend
|
||||
cd frontend
|
||||
pnpm format:write # Prettier
|
||||
```
|
||||
|
||||
4. **Test your changes** thoroughly
|
||||
|
||||
5. **Commit your changes**:
|
||||
4. **Commit your changes**:
|
||||
```bash
|
||||
git add .
|
||||
git commit -m "feat: description of your changes"
|
||||
```
|
||||
|
||||
6. **Push and create a Pull Request**:
|
||||
5. **Push and create a Pull Request**:
|
||||
```bash
|
||||
git push origin feature/your-feature-name
|
||||
```
|
||||
@@ -300,9 +236,9 @@ Nginx (port 2026) ← Unified entry point
|
||||
cd backend
|
||||
uv run pytest
|
||||
|
||||
# Frontend checks
|
||||
# Frontend tests
|
||||
cd frontend
|
||||
pnpm check
|
||||
pnpm test
|
||||
```
|
||||
|
||||
### PR Regression Checks
|
||||
@@ -314,15 +250,14 @@ Every pull request runs the backend regression workflow at [.github/workflows/ba
|
||||
|
||||
## Code Style
|
||||
|
||||
- **Backend (Python)**: We use `ruff` for linting and formatting. Run `make format` before committing.
|
||||
- **Frontend (TypeScript)**: We use ESLint and Prettier. Run `pnpm format:write` before committing.
|
||||
- CI enforces formatting — PRs with unformatted code will fail the lint check.
|
||||
- **Backend (Python)**: We use `ruff` for linting and formatting
|
||||
- **Frontend (TypeScript)**: We use ESLint and Prettier
|
||||
|
||||
## Documentation
|
||||
|
||||
- [Configuration Guide](backend/docs/CONFIGURATION.md) - Setup and configuration
|
||||
- [Architecture Overview](backend/CLAUDE.md) - Technical architecture
|
||||
- [MCP Setup Guide](backend/docs/MCP_SERVER.md) - Model Context Protocol configuration
|
||||
- [MCP Setup Guide](MCP_SETUP.md) - Model Context Protocol configuration
|
||||
|
||||
## Need Help?
|
||||
|
||||
|
||||
-87
@@ -1,87 +0,0 @@
|
||||
# DeerFlow Install
|
||||
|
||||
This file is for coding agents. If the DeerFlow repository is not already cloned and open, clone `https://github.com/bytedance/deer-flow.git` first, then continue from the repository root.
|
||||
|
||||
## Goal
|
||||
|
||||
Bootstrap a DeerFlow local development workspace on the user's machine with the least risky path available.
|
||||
|
||||
Default preference:
|
||||
|
||||
1. Docker development environment
|
||||
2. Local development environment
|
||||
|
||||
Do not assume API keys or model credentials exist. Set up everything that can be prepared safely, then stop with a concise summary of what the user still needs to provide.
|
||||
|
||||
## Operating Rules
|
||||
|
||||
- Be idempotent. Re-running this document should not damage an existing setup.
|
||||
- Prefer existing repo commands over ad hoc shell commands.
|
||||
- Do not use `sudo` or install system packages without explicit user approval.
|
||||
- Do not overwrite existing user config values unless the user asks.
|
||||
- If a step fails, stop, explain the blocker, and provide the smallest next action.
|
||||
- If multiple setup paths are possible, prefer Docker when Docker is already available.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
Consider the setup successful when all of the following are true:
|
||||
|
||||
- The DeerFlow repository is cloned and the current working directory is the repo root.
|
||||
- `config.yaml` exists.
|
||||
- For Docker setup, `make docker-init` completed successfully and Docker prerequisites are prepared, but services are not assumed to be running yet.
|
||||
- For local setup, `make check` passed or reported no missing prerequisites, and `make install` completed successfully.
|
||||
- The user receives the exact next command to launch DeerFlow.
|
||||
- The user also receives any missing model configuration or referenced environment variable names from `config.yaml`, without inspecting secret-bearing files for actual values.
|
||||
|
||||
## Steps
|
||||
|
||||
- If the current directory is not the DeerFlow repository root, clone `https://github.com/bytedance/deer-flow.git` if needed, then change into the repository root.
|
||||
- Confirm the current directory is the DeerFlow repository root by checking that `Makefile`, `backend/`, `frontend/`, and `config.example.yaml` exist.
|
||||
- Detect whether `config.yaml` already exists.
|
||||
- If `config.yaml` does not exist, run `make config`.
|
||||
- Detect whether Docker is available and the daemon is reachable with `docker info`.
|
||||
- If Docker is available:
|
||||
- Run `make docker-init`.
|
||||
- Treat this as Docker prerequisite preparation only. Do not claim that app services, compose validation, or image builds have already succeeded.
|
||||
- Do not start long-running services unless the user explicitly asks or this setup request clearly includes launch verification.
|
||||
- Tell the user the recommended next command is `make docker-start`.
|
||||
- If Docker is not available:
|
||||
- Run `make check`.
|
||||
- If `make check` reports missing system dependencies such as `node`, `pnpm`, `uv`, or `nginx`, stop and report the missing tools instead of attempting privileged installs.
|
||||
- If prerequisites are satisfied, run `make install`.
|
||||
- Tell the user the recommended next command is `make dev`.
|
||||
- Inspect `config.yaml` only for missing model entries or referenced environment variable placeholders. Do not read `.env`, `frontend/.env`, or other secret-bearing files.
|
||||
- If no model is configured, tell the user they must add at least one entry under `models` in `config.yaml`.
|
||||
- If `config.yaml` references variables such as `$OPENAI_API_KEY`, tell the user which variable names still need real values, but do not verify them by opening secret-bearing files.
|
||||
- If the repository already appears configured, avoid repeating expensive work unless it is necessary to verify the environment.
|
||||
|
||||
## Verification
|
||||
|
||||
Use the lightest verification that matches the chosen setup path.
|
||||
|
||||
For Docker setup:
|
||||
|
||||
- Confirm `make docker-init` completed successfully.
|
||||
- Confirm `config.yaml` exists.
|
||||
- State explicitly that Docker services were not started and `make docker-start` is still the first real launch step.
|
||||
- Do not leave background services running unless the user asked for that.
|
||||
|
||||
For local setup:
|
||||
|
||||
- Confirm `make install` completed successfully.
|
||||
- Confirm `config.yaml` exists.
|
||||
- Do not leave background services running unless the user asked for that.
|
||||
|
||||
## Final Response Format
|
||||
|
||||
Return a short status report with:
|
||||
|
||||
1. Setup path used: Docker or local
|
||||
2. Setup level reached: Docker prerequisites prepared or local dependencies installed
|
||||
3. Files created or detected: for example `config.yaml`
|
||||
4. Remaining user action: model config, env var values, auth files, or nothing
|
||||
5. Exact next command to start DeerFlow
|
||||
|
||||
## EXECUTE NOW
|
||||
|
||||
Complete the steps above. When finished, stop after the setup boundary and report status instead of continuing into unrelated project work.
|
||||
@@ -1,71 +1,108 @@
|
||||
# DeerFlow - Unified Development Environment
|
||||
|
||||
.PHONY: help config config-upgrade check install setup doctor dev dev-pro dev-daemon dev-daemon-pro start start-pro start-daemon start-daemon-pro stop up up-pro down clean docker-init docker-start docker-start-pro docker-stop docker-logs docker-logs-frontend docker-logs-gateway
|
||||
|
||||
BASH ?= bash
|
||||
BACKEND_UV_RUN = cd backend && uv run
|
||||
|
||||
# Detect OS for Windows compatibility
|
||||
ifeq ($(OS),Windows_NT)
|
||||
SHELL := cmd.exe
|
||||
PYTHON ?= python
|
||||
# Run repo shell scripts through Git Bash when Make is launched from cmd.exe / PowerShell.
|
||||
RUN_WITH_GIT_BASH = call scripts\run-with-git-bash.cmd
|
||||
else
|
||||
PYTHON ?= python3
|
||||
RUN_WITH_GIT_BASH =
|
||||
endif
|
||||
.PHONY: help config check install dev stop clean docker-init docker-start docker-stop docker-logs docker-logs-frontend docker-logs-gateway
|
||||
|
||||
help:
|
||||
@echo "DeerFlow Development Commands:"
|
||||
@echo " make setup - Interactive setup wizard (recommended for new users)"
|
||||
@echo " make doctor - Check configuration and system requirements"
|
||||
@echo " make config - Generate local config files (aborts if config already exists)"
|
||||
@echo " make config-upgrade - Merge new fields from config.example.yaml into config.yaml"
|
||||
@echo " make check - Check if all required tools are installed"
|
||||
@echo " make install - Install all dependencies (frontend + backend)"
|
||||
@echo " make setup-sandbox - Pre-pull sandbox container image (recommended)"
|
||||
@echo " make dev - Start all services in development mode (with hot-reloading)"
|
||||
@echo " make dev-pro - Start in dev + Gateway mode (experimental, no LangGraph server)"
|
||||
@echo " make dev-daemon - Start dev services in background (daemon mode)"
|
||||
@echo " make dev-daemon-pro - Start dev daemon + Gateway mode (experimental)"
|
||||
@echo " make start - Start all services in production mode (optimized, no hot-reloading)"
|
||||
@echo " make start-pro - Start in prod + Gateway mode (experimental)"
|
||||
@echo " make start-daemon - Start prod services in background (daemon mode)"
|
||||
@echo " make start-daemon-pro - Start prod daemon + Gateway mode (experimental)"
|
||||
@echo " make dev - Start all services (frontend + backend + nginx on localhost:2026)"
|
||||
@echo " make stop - Stop all running services"
|
||||
@echo " make clean - Clean up processes and temporary files"
|
||||
@echo ""
|
||||
@echo "Docker Production Commands:"
|
||||
@echo " make up - Build and start production Docker services (localhost:2026)"
|
||||
@echo " make up-pro - Build and start production Docker in Gateway mode (experimental)"
|
||||
@echo " make down - Stop and remove production Docker containers"
|
||||
@echo ""
|
||||
@echo "Docker Development Commands:"
|
||||
@echo " make docker-init - Pull the sandbox image"
|
||||
@echo " make docker-init - Build the custom k3s image (with pre-cached sandbox image)"
|
||||
@echo " make docker-start - Start Docker services (mode-aware from config.yaml, localhost:2026)"
|
||||
@echo " make docker-start-pro - Start Docker in Gateway mode (experimental, no LangGraph container)"
|
||||
@echo " make docker-stop - Stop Docker development services"
|
||||
@echo " make docker-logs - View Docker development logs"
|
||||
@echo " make docker-logs-frontend - View Docker frontend logs"
|
||||
@echo " make docker-logs-gateway - View Docker gateway logs"
|
||||
|
||||
## Setup & Diagnosis
|
||||
setup:
|
||||
@$(BACKEND_UV_RUN) python ../scripts/setup_wizard.py
|
||||
|
||||
doctor:
|
||||
@$(BACKEND_UV_RUN) python ../scripts/doctor.py
|
||||
|
||||
config:
|
||||
@$(PYTHON) ./scripts/configure.py
|
||||
|
||||
config-upgrade:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/config-upgrade.sh
|
||||
@if [ -f config.yaml ] || [ -f config.yml ] || [ -f configure.yml ]; then \
|
||||
echo "Error: configuration file already exists (config.yaml/config.yml/configure.yml). Aborting."; \
|
||||
exit 1; \
|
||||
fi
|
||||
@cp config.example.yaml config.yaml
|
||||
@test -f .env || cp .env.example .env
|
||||
@test -f frontend/.env || cp frontend/.env.example frontend/.env
|
||||
|
||||
# Check required tools
|
||||
check:
|
||||
@$(PYTHON) ./scripts/check.py
|
||||
@echo "=========================================="
|
||||
@echo " Checking Required Dependencies"
|
||||
@echo "=========================================="
|
||||
@echo ""
|
||||
@FAILED=0; \
|
||||
echo "Checking Node.js..."; \
|
||||
if command -v node >/dev/null 2>&1; then \
|
||||
NODE_VERSION=$$(node -v | sed 's/v//'); \
|
||||
NODE_MAJOR=$$(echo $$NODE_VERSION | cut -d. -f1); \
|
||||
if [ $$NODE_MAJOR -ge 22 ]; then \
|
||||
echo " ✓ Node.js $$NODE_VERSION (>= 22 required)"; \
|
||||
else \
|
||||
echo " ✗ Node.js $$NODE_VERSION found, but version 22+ is required"; \
|
||||
echo " Install from: https://nodejs.org/"; \
|
||||
FAILED=1; \
|
||||
fi; \
|
||||
else \
|
||||
echo " ✗ Node.js not found (version 22+ required)"; \
|
||||
echo " Install from: https://nodejs.org/"; \
|
||||
FAILED=1; \
|
||||
fi; \
|
||||
echo ""; \
|
||||
echo "Checking pnpm..."; \
|
||||
if command -v pnpm >/dev/null 2>&1; then \
|
||||
PNPM_VERSION=$$(pnpm -v); \
|
||||
echo " ✓ pnpm $$PNPM_VERSION"; \
|
||||
else \
|
||||
echo " ✗ pnpm not found"; \
|
||||
echo " Install: npm install -g pnpm"; \
|
||||
echo " Or visit: https://pnpm.io/installation"; \
|
||||
FAILED=1; \
|
||||
fi; \
|
||||
echo ""; \
|
||||
echo "Checking uv..."; \
|
||||
if command -v uv >/dev/null 2>&1; then \
|
||||
UV_VERSION=$$(uv --version | awk '{print $$2}'); \
|
||||
echo " ✓ uv $$UV_VERSION"; \
|
||||
else \
|
||||
echo " ✗ uv not found"; \
|
||||
echo " Install: curl -LsSf https://astral.sh/uv/install.sh | sh"; \
|
||||
echo " Or visit: https://docs.astral.sh/uv/getting-started/installation/"; \
|
||||
FAILED=1; \
|
||||
fi; \
|
||||
echo ""; \
|
||||
echo "Checking nginx..."; \
|
||||
if command -v nginx >/dev/null 2>&1; then \
|
||||
NGINX_VERSION=$$(nginx -v 2>&1 | awk -F'/' '{print $$2}'); \
|
||||
echo " ✓ nginx $$NGINX_VERSION"; \
|
||||
else \
|
||||
echo " ✗ nginx not found"; \
|
||||
echo " macOS: brew install nginx"; \
|
||||
echo " Ubuntu: sudo apt install nginx"; \
|
||||
echo " Or visit: https://nginx.org/en/download.html"; \
|
||||
FAILED=1; \
|
||||
fi; \
|
||||
echo ""; \
|
||||
if [ $$FAILED -eq 0 ]; then \
|
||||
echo "=========================================="; \
|
||||
echo " ✓ All dependencies are installed!"; \
|
||||
echo "=========================================="; \
|
||||
echo ""; \
|
||||
echo "You can now run:"; \
|
||||
echo " make install - Install project dependencies"; \
|
||||
echo " make dev - Start development server"; \
|
||||
else \
|
||||
echo "=========================================="; \
|
||||
echo " ✗ Some dependencies are missing"; \
|
||||
echo "=========================================="; \
|
||||
echo ""; \
|
||||
echo "Please install the missing tools and run 'make check' again."; \
|
||||
exit 1; \
|
||||
fi
|
||||
|
||||
# Install all dependencies
|
||||
install:
|
||||
@@ -103,68 +140,109 @@ setup-sandbox:
|
||||
fi; \
|
||||
if command -v docker >/dev/null 2>&1; then \
|
||||
echo "Pulling image using Docker..."; \
|
||||
if docker pull "$$IMAGE"; then \
|
||||
echo ""; \
|
||||
echo "✓ Sandbox image pulled successfully"; \
|
||||
else \
|
||||
echo ""; \
|
||||
echo "⚠ Failed to pull sandbox image (this is OK for local sandbox mode)"; \
|
||||
fi; \
|
||||
docker pull "$$IMAGE"; \
|
||||
echo ""; \
|
||||
echo "✓ Sandbox image pulled successfully"; \
|
||||
else \
|
||||
echo "✗ Neither Docker nor Apple Container is available"; \
|
||||
echo " Please install Docker: https://docs.docker.com/get-docker/"; \
|
||||
exit 1; \
|
||||
fi
|
||||
|
||||
# Start all services in development mode (with hot-reloading)
|
||||
# Start all services
|
||||
dev:
|
||||
@$(PYTHON) ./scripts/check.py
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --dev
|
||||
|
||||
# Start all services in dev + Gateway mode (experimental: agent runtime embedded in Gateway)
|
||||
dev-pro:
|
||||
@$(PYTHON) ./scripts/check.py
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --dev --gateway
|
||||
|
||||
# Start all services in production mode (with optimizations)
|
||||
start:
|
||||
@$(PYTHON) ./scripts/check.py
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --prod
|
||||
|
||||
# Start all services in prod + Gateway mode (experimental)
|
||||
start-pro:
|
||||
@$(PYTHON) ./scripts/check.py
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --prod --gateway
|
||||
|
||||
# Start all services in daemon mode (background)
|
||||
dev-daemon:
|
||||
@$(PYTHON) ./scripts/check.py
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --dev --daemon
|
||||
|
||||
# Start daemon + Gateway mode (experimental)
|
||||
dev-daemon-pro:
|
||||
@$(PYTHON) ./scripts/check.py
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --dev --gateway --daemon
|
||||
|
||||
# Start prod services in daemon mode (background)
|
||||
start-daemon:
|
||||
@$(PYTHON) ./scripts/check.py
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --prod --daemon
|
||||
|
||||
# Start prod daemon + Gateway mode (experimental)
|
||||
start-daemon-pro:
|
||||
@$(PYTHON) ./scripts/check.py
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --prod --gateway --daemon
|
||||
@echo "Stopping existing services if any..."
|
||||
@-pkill -f "langgraph dev" 2>/dev/null || true
|
||||
@-pkill -f "uvicorn src.gateway.app:app" 2>/dev/null || true
|
||||
@-pkill -f "next dev" 2>/dev/null || true
|
||||
@-nginx -c $(PWD)/docker/nginx/nginx.local.conf -p $(PWD) -s quit 2>/dev/null || true
|
||||
@sleep 1
|
||||
@-pkill -9 nginx 2>/dev/null || true
|
||||
@-./scripts/cleanup-containers.sh deer-flow-sandbox 2>/dev/null || true
|
||||
@sleep 1
|
||||
@echo ""
|
||||
@echo "=========================================="
|
||||
@echo " Starting DeerFlow Development Server"
|
||||
@echo "=========================================="
|
||||
@echo ""
|
||||
@echo "Services starting up..."
|
||||
@echo " → Backend: LangGraph + Gateway"
|
||||
@echo " → Frontend: Next.js"
|
||||
@echo " → Nginx: Reverse Proxy"
|
||||
@echo ""
|
||||
@cleanup() { \
|
||||
trap - INT TERM; \
|
||||
echo ""; \
|
||||
echo "Shutting down services..."; \
|
||||
pkill -f "langgraph dev" 2>/dev/null || true; \
|
||||
pkill -f "uvicorn src.gateway.app:app" 2>/dev/null || true; \
|
||||
pkill -f "next dev" 2>/dev/null || true; \
|
||||
nginx -c $(PWD)/docker/nginx/nginx.local.conf -p $(PWD) -s quit 2>/dev/null || true; \
|
||||
sleep 1; \
|
||||
pkill -9 nginx 2>/dev/null || true; \
|
||||
echo "Cleaning up sandbox containers..."; \
|
||||
./scripts/cleanup-containers.sh deer-flow-sandbox 2>/dev/null || true; \
|
||||
echo "✓ All services stopped"; \
|
||||
exit 0; \
|
||||
}; \
|
||||
trap cleanup INT TERM; \
|
||||
mkdir -p logs; \
|
||||
echo "Starting LangGraph server..."; \
|
||||
cd backend && NO_COLOR=1 uv run langgraph dev --no-browser --allow-blocking --no-reload > ../logs/langgraph.log 2>&1 & \
|
||||
sleep 3; \
|
||||
echo "✓ LangGraph server started on localhost:2024"; \
|
||||
echo "Starting Gateway API..."; \
|
||||
cd backend && uv run uvicorn src.gateway.app:app --host 0.0.0.0 --port 8001 > ../logs/gateway.log 2>&1 & \
|
||||
sleep 3; \
|
||||
if ! lsof -i :8001 -sTCP:LISTEN -t >/dev/null 2>&1; then \
|
||||
echo "✗ Gateway API failed to start. Last log output:"; \
|
||||
tail -30 logs/gateway.log; \
|
||||
cleanup; \
|
||||
fi; \
|
||||
echo "✓ Gateway API started on localhost:8001"; \
|
||||
echo "Starting Frontend..."; \
|
||||
cd frontend && pnpm run dev > ../logs/frontend.log 2>&1 & \
|
||||
sleep 3; \
|
||||
echo "✓ Frontend started on localhost:3000"; \
|
||||
echo "Starting Nginx reverse proxy..."; \
|
||||
mkdir -p logs && nginx -g 'daemon off;' -c $(PWD)/docker/nginx/nginx.local.conf -p $(PWD) > logs/nginx.log 2>&1 & \
|
||||
sleep 2; \
|
||||
echo "✓ Nginx started on localhost:2026"; \
|
||||
echo ""; \
|
||||
echo "=========================================="; \
|
||||
echo " DeerFlow is ready!"; \
|
||||
echo "=========================================="; \
|
||||
echo ""; \
|
||||
echo " 🌐 Application: http://localhost:2026"; \
|
||||
echo " 📡 API Gateway: http://localhost:2026/api/*"; \
|
||||
echo " 🤖 LangGraph: http://localhost:2026/api/langgraph/*"; \
|
||||
echo ""; \
|
||||
echo " 📋 Logs:"; \
|
||||
echo " - LangGraph: logs/langgraph.log"; \
|
||||
echo " - Gateway: logs/gateway.log"; \
|
||||
echo " - Frontend: logs/frontend.log"; \
|
||||
echo " - Nginx: logs/nginx.log"; \
|
||||
echo ""; \
|
||||
echo "Press Ctrl+C to stop all services"; \
|
||||
echo ""; \
|
||||
wait
|
||||
|
||||
# Stop all services
|
||||
stop:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --stop
|
||||
@echo "Stopping all services..."
|
||||
@-pkill -f "langgraph dev" 2>/dev/null || true
|
||||
@-pkill -f "uvicorn src.gateway.app:app" 2>/dev/null || true
|
||||
@-pkill -f "next dev" 2>/dev/null || true
|
||||
@-nginx -c $(PWD)/docker/nginx/nginx.local.conf -p $(PWD) -s quit 2>/dev/null || true
|
||||
@sleep 1
|
||||
@-pkill -9 nginx 2>/dev/null || true
|
||||
@echo "Cleaning up sandbox containers..."
|
||||
@-./scripts/cleanup-containers.sh deer-flow-sandbox 2>/dev/null || true
|
||||
@echo "✓ All services stopped"
|
||||
|
||||
# Clean up
|
||||
clean: stop
|
||||
@echo "Cleaning up..."
|
||||
@-rm -rf backend/.deer-flow 2>/dev/null || true
|
||||
@-rm -rf backend/.langgraph_api 2>/dev/null || true
|
||||
@-rm -rf logs/*.log 2>/dev/null || true
|
||||
@echo "✓ Cleanup complete"
|
||||
|
||||
@@ -174,42 +252,22 @@ clean: stop
|
||||
|
||||
# Initialize Docker containers and install dependencies
|
||||
docker-init:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/docker.sh init
|
||||
@./scripts/docker.sh init
|
||||
|
||||
# Start Docker development environment
|
||||
docker-start:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/docker.sh start
|
||||
|
||||
# Start Docker in Gateway mode (experimental)
|
||||
docker-start-pro:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/docker.sh start --gateway
|
||||
@./scripts/docker.sh start
|
||||
|
||||
# Stop Docker development environment
|
||||
docker-stop:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/docker.sh stop
|
||||
@./scripts/docker.sh stop
|
||||
|
||||
# View Docker development logs
|
||||
docker-logs:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/docker.sh logs
|
||||
@./scripts/docker.sh logs
|
||||
|
||||
# View Docker development logs
|
||||
docker-logs-frontend:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/docker.sh logs --frontend
|
||||
@./scripts/docker.sh logs --frontend
|
||||
docker-logs-gateway:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/docker.sh logs --gateway
|
||||
|
||||
# ==========================================
|
||||
# Production Docker Commands
|
||||
# ==========================================
|
||||
|
||||
# Build and start production services
|
||||
up:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/deploy.sh
|
||||
|
||||
# Build and start production services in Gateway mode
|
||||
up-pro:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/deploy.sh --gateway
|
||||
|
||||
# Stop and remove production containers
|
||||
down:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/deploy.sh down
|
||||
@./scripts/docker.sh logs --gateway
|
||||
@@ -1,11 +1,5 @@
|
||||
# 🦌 DeerFlow - 2.0
|
||||
|
||||
English | [中文](./README_zh.md) | [日本語](./README_ja.md) | [Français](./README_fr.md) | [Русский](./README_ru.md)
|
||||
|
||||
[](./backend/pyproject.toml)
|
||||
[](./Makefile)
|
||||
[](./LICENSE)
|
||||
|
||||
<a href="https://trendshift.io/repositories/14699" target="_blank"><img src="https://trendshift.io/api/badge/repositories/14699" alt="bytedance%2Fdeer-flow | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
|
||||
> On February 28th, 2026, DeerFlow claimed the 🏆 #1 spot on GitHub Trending following the launch of version 2. Thanks a million to our incredible community — you made this happen! 💪🔥
|
||||
|
||||
@@ -18,27 +12,9 @@ https://github.com/user-attachments/assets/a8bcadc4-e040-4cf2-8fda-dd768b999c18
|
||||
|
||||
## Official Website
|
||||
|
||||
[<img width="2880" height="1600" alt="image" src="https://github.com/user-attachments/assets/a598c49f-3b2f-41ea-a052-05e21349188a" />](https://deerflow.tech)
|
||||
Learn more and see **real demos** on our official website.
|
||||
|
||||
Learn more and see **real demos** on our [**official website**](https://deerflow.tech).
|
||||
|
||||
## Coding Plan from ByteDance Volcengine
|
||||
|
||||
<img width="4808" height="2400" alt="英文方舟" src="https://github.com/user-attachments/assets/2ecc7b9d-50be-4185-b1f7-5542d222fb2d" />
|
||||
|
||||
- We strongly recommend using Doubao-Seed-2.0-Code, DeepSeek v3.2 and Kimi 2.5 to run DeerFlow
|
||||
- [Learn more](https://www.byteplus.com/en/activity/codingplan?utm_campaign=deer_flow&utm_content=deer_flow&utm_medium=devrel&utm_source=OWO&utm_term=deer_flow)
|
||||
- [中国大陆地区的开发者请点击这里](https://www.volcengine.com/activity/codingplan?utm_campaign=deer_flow&utm_content=deer_flow&utm_medium=devrel&utm_source=OWO&utm_term=deer_flow)
|
||||
|
||||
## InfoQuest
|
||||
|
||||
DeerFlow has newly integrated the intelligent search and crawling toolset independently developed by BytePlus--[InfoQuest (supports free online experience)](https://docs.byteplus.com/en/docs/InfoQuest/What_is_Info_Quest)
|
||||
|
||||
<a href="https://docs.byteplus.com/en/docs/InfoQuest/What_is_Info_Quest" target="_blank">
|
||||
<img
|
||||
src="https://sf16-sg.tiktokcdn.com/obj/eden-sg/hubseh7bsbps/20251208-160108.png" alt="InfoQuest_banner"
|
||||
/>
|
||||
</a>
|
||||
**[deerflow.tech](https://deerflow.tech/)**
|
||||
|
||||
---
|
||||
|
||||
@@ -46,53 +22,30 @@ DeerFlow has newly integrated the intelligent search and crawling toolset indepe
|
||||
|
||||
- [🦌 DeerFlow - 2.0](#-deerflow---20)
|
||||
- [Official Website](#official-website)
|
||||
- [Coding Plan from ByteDance Volcengine](#coding-plan-from-bytedance-volcengine)
|
||||
- [InfoQuest](#infoquest)
|
||||
- [Table of Contents](#table-of-contents)
|
||||
- [One-Line Agent Setup](#one-line-agent-setup)
|
||||
- [Quick Start](#quick-start)
|
||||
- [Configuration](#configuration)
|
||||
- [Running the Application](#running-the-application)
|
||||
- [Deployment Sizing](#deployment-sizing)
|
||||
- [Option 1: Docker (Recommended)](#option-1-docker-recommended)
|
||||
- [Option 2: Local Development](#option-2-local-development)
|
||||
- [Advanced](#advanced)
|
||||
- [Sandbox Mode](#sandbox-mode)
|
||||
- [MCP Server](#mcp-server)
|
||||
- [IM Channels](#im-channels)
|
||||
- [LangSmith Tracing](#langsmith-tracing)
|
||||
- [Langfuse Tracing](#langfuse-tracing)
|
||||
- [Using Both Providers](#using-both-providers)
|
||||
- [From Deep Research to Super Agent Harness](#from-deep-research-to-super-agent-harness)
|
||||
- [Core Features](#core-features)
|
||||
- [Skills \& Tools](#skills--tools)
|
||||
- [Claude Code Integration](#claude-code-integration)
|
||||
- [Sub-Agents](#sub-agents)
|
||||
- [Sandbox \& File System](#sandbox--file-system)
|
||||
- [Context Engineering](#context-engineering)
|
||||
- [Long-Term Memory](#long-term-memory)
|
||||
- [Recommended Models](#recommended-models)
|
||||
- [Embedded Python Client](#embedded-python-client)
|
||||
- [Documentation](#documentation)
|
||||
- [⚠️ Security Notice](#️-security-notice)
|
||||
- [Improper Deployment May Introduce Security Risks](#improper-deployment-may-introduce-security-risks)
|
||||
- [Security Recommendations](#security-recommendations)
|
||||
- [Contributing](#contributing)
|
||||
- [License](#license)
|
||||
- [Acknowledgments](#acknowledgments)
|
||||
- [Key Contributors](#key-contributors)
|
||||
- [Star History](#star-history)
|
||||
|
||||
## One-Line Agent Setup
|
||||
|
||||
If you use Claude Code, Codex, Cursor, Windsurf, or another coding agent, you can hand it the setup instructions in one sentence:
|
||||
|
||||
```text
|
||||
Help me clone DeerFlow if needed, then bootstrap it for local development by following https://raw.githubusercontent.com/bytedance/deer-flow/main/Install.md
|
||||
```
|
||||
|
||||
That prompt is intended for coding agents. It tells the agent to clone the repo if needed, choose Docker when available, and stop with the exact next command plus any missing config the user still needs to provide.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Configuration
|
||||
@@ -104,149 +57,74 @@ That prompt is intended for coding agents. It tells the agent to clone the repo
|
||||
cd deer-flow
|
||||
```
|
||||
|
||||
2. **Run the setup wizard**
|
||||
2. **Generate local configuration files**
|
||||
|
||||
From the project root directory (`deer-flow/`), run:
|
||||
|
||||
```bash
|
||||
make setup
|
||||
make config
|
||||
```
|
||||
|
||||
This launches an interactive wizard that guides you through choosing an LLM provider, optional web search, and execution/safety preferences such as sandbox mode, bash access, and file-write tools. It generates a minimal `config.yaml` and writes your keys to `.env`. Takes about 2 minutes.
|
||||
This command creates local configuration files based on the provided example templates.
|
||||
|
||||
The wizard also lets you configure an optional web search provider, or skip it for now.
|
||||
3. **Configure your preferred model(s)**
|
||||
|
||||
Run `make doctor` at any time to verify your setup and get actionable fix hints.
|
||||
|
||||
> **Advanced / manual configuration**: If you prefer to edit `config.yaml` directly, run `make config` instead to copy the full template. See `config.example.yaml` for the complete reference including CLI-backed providers (Codex CLI, Claude Code OAuth), OpenRouter, Responses API, and more.
|
||||
|
||||
<details>
|
||||
<summary>Manual model configuration examples</summary>
|
||||
Edit `config.yaml` and define at least one model:
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- name: gpt-4o
|
||||
display_name: GPT-4o
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: gpt-4o
|
||||
api_key: $OPENAI_API_KEY
|
||||
|
||||
- name: openrouter-gemini-2.5-flash
|
||||
display_name: Gemini 2.5 Flash (OpenRouter)
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: google/gemini-2.5-flash-preview
|
||||
api_key: $OPENROUTER_API_KEY
|
||||
base_url: https://openrouter.ai/api/v1
|
||||
|
||||
- name: gpt-5-responses
|
||||
display_name: GPT-5 (Responses API)
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: gpt-5
|
||||
api_key: $OPENAI_API_KEY
|
||||
use_responses_api: true
|
||||
output_version: responses/v1
|
||||
|
||||
- name: qwen3-32b-vllm
|
||||
display_name: Qwen3 32B (vLLM)
|
||||
use: deerflow.models.vllm_provider:VllmChatModel
|
||||
model: Qwen/Qwen3-32B
|
||||
api_key: $VLLM_API_KEY
|
||||
base_url: http://localhost:8000/v1
|
||||
supports_thinking: true
|
||||
when_thinking_enabled:
|
||||
extra_body:
|
||||
chat_template_kwargs:
|
||||
enable_thinking: true
|
||||
- name: gpt-4 # Internal identifier
|
||||
display_name: GPT-4 # Human-readable name
|
||||
use: langchain_openai:ChatOpenAI # LangChain class path
|
||||
model: gpt-4 # Model identifier for API
|
||||
api_key: $OPENAI_API_KEY # API key (recommended: use env var)
|
||||
max_tokens: 4096 # Maximum tokens per request
|
||||
temperature: 0.7 # Sampling temperature
|
||||
```
|
||||
|
||||
OpenRouter and similar OpenAI-compatible gateways should be configured with `langchain_openai:ChatOpenAI` plus `base_url`. If you prefer a provider-specific environment variable name, point `api_key` at that variable explicitly (for example `api_key: $OPENROUTER_API_KEY`).
|
||||
|
||||
4. **Set API keys for your configured model(s)**
|
||||
|
||||
To route OpenAI models through `/v1/responses`, keep using `langchain_openai:ChatOpenAI` and set `use_responses_api: true` with `output_version: responses/v1`.
|
||||
Choose one of the following methods:
|
||||
|
||||
For vLLM 0.19.0, use `deerflow.models.vllm_provider:VllmChatModel`. For Qwen-style reasoning models, DeerFlow toggles reasoning with `extra_body.chat_template_kwargs.enable_thinking` and preserves vLLM's non-standard `reasoning` field across multi-turn tool-call conversations. Legacy `thinking` configs are normalized automatically for backward compatibility. Reasoning models may also require the server to be started with `--reasoning-parser ...`. If your local vLLM deployment accepts any non-empty API key, you can still set `VLLM_API_KEY` to a placeholder value.
|
||||
- Option A: Edit the `.env` file in the project root (Recommended)
|
||||
|
||||
CLI-backed provider examples:
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- name: gpt-5.4
|
||||
display_name: GPT-5.4 (Codex CLI)
|
||||
use: deerflow.models.openai_codex_provider:CodexChatModel
|
||||
model: gpt-5.4
|
||||
supports_thinking: true
|
||||
supports_reasoning_effort: true
|
||||
|
||||
- name: claude-sonnet-4.6
|
||||
display_name: Claude Sonnet 4.6 (Claude Code OAuth)
|
||||
use: deerflow.models.claude_provider:ClaudeChatModel
|
||||
model: claude-sonnet-4-6
|
||||
max_tokens: 4096
|
||||
supports_thinking: true
|
||||
```
|
||||
|
||||
- Codex CLI reads `~/.codex/auth.json`
|
||||
- Claude Code accepts `CLAUDE_CODE_OAUTH_TOKEN`, `ANTHROPIC_AUTH_TOKEN`, `CLAUDE_CODE_CREDENTIALS_PATH`, or `~/.claude/.credentials.json`
|
||||
- ACP agent entries are separate from model providers — if you configure `acp_agents.codex`, point it at a Codex ACP adapter such as `npx -y @zed-industries/codex-acp`
|
||||
- On macOS, export Claude Code auth explicitly if needed:
|
||||
|
||||
```bash
|
||||
eval "$(python3 scripts/export_claude_code_oauth.py --print-export)"
|
||||
```
|
||||
|
||||
API keys can also be set manually in `.env` (recommended) or exported in your shell:
|
||||
|
||||
```bash
|
||||
OPENAI_API_KEY=your-openai-api-key
|
||||
TAVILY_API_KEY=your-tavily-api-key
|
||||
OPENAI_API_KEY=your-openai-api-key
|
||||
# Add other provider keys as needed
|
||||
```
|
||||
|
||||
</details>
|
||||
- Option B: Export environment variables in your shell
|
||||
|
||||
```bash
|
||||
export OPENAI_API_KEY=your-openai-api-key
|
||||
```
|
||||
|
||||
- Option C: Edit `config.yaml` directly (Not recommended for production)
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- name: gpt-4
|
||||
api_key: your-actual-api-key-here # Replace placeholder
|
||||
```
|
||||
|
||||
### Running the Application
|
||||
|
||||
#### Deployment Sizing
|
||||
|
||||
Use the table below as a practical starting point when choosing how to run DeerFlow:
|
||||
|
||||
| Deployment target | Starting point | Recommended | Notes |
|
||||
|---------|-----------|------------|-------|
|
||||
| Local evaluation / `make dev` | 4 vCPU, 8 GB RAM, 20 GB free SSD | 8 vCPU, 16 GB RAM | Good for one developer or one light session with hosted model APIs. `2 vCPU / 4 GB` is usually not enough. |
|
||||
| Docker development / `make docker-start` | 4 vCPU, 8 GB RAM, 25 GB free SSD | 8 vCPU, 16 GB RAM | Image builds, bind mounts, and sandbox containers need more headroom than pure local dev. |
|
||||
| Long-running server / `make up` | 8 vCPU, 16 GB RAM, 40 GB free SSD | 16 vCPU, 32 GB RAM | Preferred for shared use, multi-agent runs, report generation, or heavier sandbox workloads. |
|
||||
|
||||
- These numbers cover DeerFlow itself. If you also host a local LLM, size that service separately.
|
||||
- Linux plus Docker is the recommended deployment target for a persistent server. macOS and Windows are best treated as development or evaluation environments.
|
||||
- If CPU or memory usage stays pinned, reduce concurrent runs first, then move to the next sizing tier.
|
||||
|
||||
#### Option 1: Docker (Recommended)
|
||||
|
||||
**Development** (hot-reload, source mounts):
|
||||
The fastest way to get started with a consistent environment:
|
||||
|
||||
```bash
|
||||
make docker-init # Pull sandbox image (only once or when image updates)
|
||||
make docker-start # Start services (auto-detects sandbox mode from config.yaml)
|
||||
```
|
||||
1. **Initialize and start**:
|
||||
```bash
|
||||
make docker-init # Pull sandbox image (Only once or when image updates)
|
||||
make docker-start # Start services (auto-detects sandbox mode from config.yaml)
|
||||
```
|
||||
|
||||
`make docker-start` starts `provisioner` only when `config.yaml` uses provisioner mode (`sandbox.use: deerflow.community.aio_sandbox:AioSandboxProvider` with `provisioner_url`).
|
||||
`make docker-start` now starts `provisioner` only when `config.yaml` uses provisioner mode (`sandbox.use: src.community.aio_sandbox:AioSandboxProvider` with `provisioner_url`).
|
||||
|
||||
Docker builds use the upstream `uv` registry by default. If you need faster mirrors in restricted networks, export `UV_INDEX_URL=https://pypi.tuna.tsinghua.edu.cn/simple` and `NPM_REGISTRY=https://registry.npmmirror.com` before running `make docker-init` or `make docker-start`.
|
||||
|
||||
Backend processes automatically pick up `config.yaml` changes on the next config access, so model metadata updates do not require a manual restart during development.
|
||||
|
||||
> [!TIP]
|
||||
> On Linux, if Docker-based commands fail with `permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock`, add your user to the `docker` group and re-login before retrying. See [CONTRIBUTING.md](CONTRIBUTING.md#linux-docker-daemon-permission-denied) for the full fix.
|
||||
|
||||
**Production** (builds images locally, mounts runtime config and data):
|
||||
|
||||
```bash
|
||||
make up # Build images and start all production services
|
||||
make down # Stop and remove containers
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> The LangGraph agent server currently runs via `langgraph dev` (the open-source CLI server).
|
||||
|
||||
Access: http://localhost:2026
|
||||
2. **Access**: http://localhost:2026
|
||||
|
||||
See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed Docker development guide.
|
||||
|
||||
@@ -254,9 +132,6 @@ See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed Docker development guide.
|
||||
|
||||
If you prefer running services locally:
|
||||
|
||||
Prerequisite: complete the "Configuration" steps above first (`make setup`). `make dev` requires a valid `config.yaml` in the project root (can be overridden via `DEER_FLOW_CONFIG_PATH`). Run `make doctor` to verify your setup before starting.
|
||||
On Windows, run the local development flow from Git Bash. Native `cmd.exe` and PowerShell shells are not supported for the bash-based service scripts, and WSL is not guaranteed because some scripts rely on Git for Windows utilities such as `cygpath`.
|
||||
|
||||
1. **Check prerequisites**:
|
||||
```bash
|
||||
make check # Verifies Node.js 22+, pnpm, uv, nginx
|
||||
@@ -273,73 +148,12 @@ On Windows, run the local development flow from Git Bash. Native `cmd.exe` and P
|
||||
make setup-sandbox
|
||||
```
|
||||
|
||||
4. **(Optional) Load sample memory data for local review**:
|
||||
```bash
|
||||
python scripts/load_memory_sample.py
|
||||
```
|
||||
This copies the sample fixture into the default local runtime memory file so reviewers can immediately test `Settings > Memory`.
|
||||
See [backend/docs/MEMORY_SETTINGS_REVIEW.md](backend/docs/MEMORY_SETTINGS_REVIEW.md) for the shortest review flow.
|
||||
|
||||
5. **Start services**:
|
||||
4. **Start services**:
|
||||
```bash
|
||||
make dev
|
||||
```
|
||||
|
||||
6. **Access**: http://localhost:2026
|
||||
|
||||
#### Startup Modes
|
||||
|
||||
DeerFlow supports multiple startup modes across two dimensions:
|
||||
|
||||
- **Dev / Prod** — dev enables hot-reload; prod uses pre-built frontend
|
||||
- **Standard / Gateway** — standard uses a separate LangGraph server (4 processes); Gateway mode (experimental) embeds the agent runtime in the Gateway API (3 processes)
|
||||
|
||||
| | **Local Foreground** | **Local Daemon** | **Docker Dev** | **Docker Prod** |
|
||||
|---|---|---|---|---|
|
||||
| **Dev** | `./scripts/serve.sh --dev`<br/>`make dev` | `./scripts/serve.sh --dev --daemon`<br/>`make dev-daemon` | `./scripts/docker.sh start`<br/>`make docker-start` | — |
|
||||
| **Dev + Gateway** | `./scripts/serve.sh --dev --gateway`<br/>`make dev-pro` | `./scripts/serve.sh --dev --gateway --daemon`<br/>`make dev-daemon-pro` | `./scripts/docker.sh start --gateway`<br/>`make docker-start-pro` | — |
|
||||
| **Prod** | `./scripts/serve.sh --prod`<br/>`make start` | `./scripts/serve.sh --prod --daemon`<br/>`make start-daemon` | — | `./scripts/deploy.sh`<br/>`make up` |
|
||||
| **Prod + Gateway** | `./scripts/serve.sh --prod --gateway`<br/>`make start-pro` | `./scripts/serve.sh --prod --gateway --daemon`<br/>`make start-daemon-pro` | — | `./scripts/deploy.sh --gateway`<br/>`make up-pro` |
|
||||
|
||||
| Action | Local | Docker Dev | Docker Prod |
|
||||
|---|---|---|---|
|
||||
| **Stop** | `./scripts/serve.sh --stop`<br/>`make stop` | `./scripts/docker.sh stop`<br/>`make docker-stop` | `./scripts/deploy.sh down`<br/>`make down` |
|
||||
| **Restart** | `./scripts/serve.sh --restart [flags]` | `./scripts/docker.sh restart` | — |
|
||||
|
||||
> **Gateway mode** eliminates the LangGraph server process — the Gateway API handles agent execution directly via async tasks, managing its own concurrency.
|
||||
|
||||
#### Why Gateway Mode?
|
||||
|
||||
In standard mode, DeerFlow runs a dedicated [LangGraph Platform](https://langchain-ai.github.io/langgraph/) server alongside the Gateway API. This architecture works well but has trade-offs:
|
||||
|
||||
| | Standard Mode | Gateway Mode |
|
||||
|---|---|---|
|
||||
| **Architecture** | Gateway (REST API) + LangGraph (agent runtime) | Gateway embeds agent runtime |
|
||||
| **Concurrency** | `--n-jobs-per-worker` per worker (requires license) | `--workers` × async tasks (no per-worker cap) |
|
||||
| **Containers / Processes** | 4 (frontend, gateway, langgraph, nginx) | 3 (frontend, gateway, nginx) |
|
||||
| **Resource usage** | Higher (two Python runtimes) | Lower (single Python runtime) |
|
||||
| **LangGraph Platform license** | Required for production images | Not required |
|
||||
| **Cold start** | Slower (two services to initialize) | Faster |
|
||||
|
||||
Both modes are functionally equivalent — the same agents, tools, and skills work in either mode.
|
||||
|
||||
#### Docker Production Deployment
|
||||
|
||||
`deploy.sh` supports building and starting separately. Images are mode-agnostic — runtime mode is selected at start time:
|
||||
|
||||
```bash
|
||||
# One-step (build + start)
|
||||
deploy.sh # standard mode (default)
|
||||
deploy.sh --gateway # gateway mode
|
||||
|
||||
# Two-step (build once, start with any mode)
|
||||
deploy.sh build # build all images
|
||||
deploy.sh start # start in standard mode
|
||||
deploy.sh start --gateway # start in gateway mode
|
||||
|
||||
# Stop
|
||||
deploy.sh down
|
||||
```
|
||||
5. **Access**: http://localhost:2026
|
||||
|
||||
### Advanced
|
||||
#### Sandbox Mode
|
||||
@@ -359,203 +173,6 @@ DeerFlow supports configurable MCP servers and skills to extend its capabilities
|
||||
For HTTP/SSE MCP servers, OAuth token flows are supported (`client_credentials`, `refresh_token`).
|
||||
See the [MCP Server Guide](backend/docs/MCP_SERVER.md) for detailed instructions.
|
||||
|
||||
#### IM Channels
|
||||
|
||||
DeerFlow supports receiving tasks from messaging apps. Channels auto-start when configured — no public IP required for any of them.
|
||||
|
||||
| Channel | Transport | Difficulty |
|
||||
|---------|-----------|------------|
|
||||
| Telegram | Bot API (long-polling) | Easy |
|
||||
| Slack | Socket Mode | Moderate |
|
||||
| Feishu / Lark | WebSocket | Moderate |
|
||||
| WeChat | Tencent iLink (long-polling) | Moderate |
|
||||
| WeCom | WebSocket | Moderate |
|
||||
|
||||
**Configuration in `config.yaml`:**
|
||||
|
||||
```yaml
|
||||
channels:
|
||||
# LangGraph Server URL (default: http://localhost:2024)
|
||||
langgraph_url: http://localhost:2024
|
||||
# Gateway API URL (default: http://localhost:8001)
|
||||
gateway_url: http://localhost:8001
|
||||
|
||||
# Optional: global session defaults for all mobile channels
|
||||
session:
|
||||
assistant_id: lead_agent # or a custom agent name; custom agents are routed via lead_agent + agent_name
|
||||
config:
|
||||
recursion_limit: 100
|
||||
context:
|
||||
thinking_enabled: true
|
||||
is_plan_mode: false
|
||||
subagent_enabled: false
|
||||
|
||||
feishu:
|
||||
enabled: true
|
||||
app_id: $FEISHU_APP_ID
|
||||
app_secret: $FEISHU_APP_SECRET
|
||||
# domain: https://open.feishu.cn # China (default)
|
||||
# domain: https://open.larksuite.com # International
|
||||
|
||||
wecom:
|
||||
enabled: true
|
||||
bot_id: $WECOM_BOT_ID
|
||||
bot_secret: $WECOM_BOT_SECRET
|
||||
|
||||
slack:
|
||||
enabled: true
|
||||
bot_token: $SLACK_BOT_TOKEN # xoxb-...
|
||||
app_token: $SLACK_APP_TOKEN # xapp-... (Socket Mode)
|
||||
allowed_users: [] # empty = allow all
|
||||
|
||||
telegram:
|
||||
enabled: true
|
||||
bot_token: $TELEGRAM_BOT_TOKEN
|
||||
allowed_users: [] # empty = allow all
|
||||
|
||||
wechat:
|
||||
enabled: false
|
||||
bot_token: $WECHAT_BOT_TOKEN
|
||||
ilink_bot_id: $WECHAT_ILINK_BOT_ID
|
||||
qrcode_login_enabled: true # optional: allow first-time QR bootstrap when bot_token is absent
|
||||
allowed_users: [] # empty = allow all
|
||||
polling_timeout: 35
|
||||
state_dir: ./.deer-flow/wechat/state
|
||||
max_inbound_image_bytes: 20971520
|
||||
max_outbound_image_bytes: 20971520
|
||||
max_inbound_file_bytes: 52428800
|
||||
max_outbound_file_bytes: 52428800
|
||||
|
||||
# Optional: per-channel / per-user session settings
|
||||
session:
|
||||
assistant_id: mobile-agent # custom agent names are also supported here
|
||||
context:
|
||||
thinking_enabled: false
|
||||
users:
|
||||
"123456789":
|
||||
assistant_id: vip-agent
|
||||
config:
|
||||
recursion_limit: 150
|
||||
context:
|
||||
thinking_enabled: true
|
||||
subagent_enabled: true
|
||||
```
|
||||
|
||||
Notes:
|
||||
- `assistant_id: lead_agent` calls the default LangGraph assistant directly.
|
||||
- If `assistant_id` is set to a custom agent name, DeerFlow still routes through `lead_agent` and injects that value as `agent_name`, so the custom agent's SOUL/config takes effect for IM channels.
|
||||
|
||||
Set the corresponding API keys in your `.env` file:
|
||||
|
||||
```bash
|
||||
# Telegram
|
||||
TELEGRAM_BOT_TOKEN=123456789:ABCdefGHIjklMNOpqrSTUvwxYZ
|
||||
|
||||
# Slack
|
||||
SLACK_BOT_TOKEN=xoxb-...
|
||||
SLACK_APP_TOKEN=xapp-...
|
||||
|
||||
# Feishu / Lark
|
||||
FEISHU_APP_ID=cli_xxxx
|
||||
FEISHU_APP_SECRET=your_app_secret
|
||||
|
||||
# WeChat iLink
|
||||
WECHAT_BOT_TOKEN=your_ilink_bot_token
|
||||
WECHAT_ILINK_BOT_ID=your_ilink_bot_id
|
||||
|
||||
# WeCom
|
||||
WECOM_BOT_ID=your_bot_id
|
||||
WECOM_BOT_SECRET=your_bot_secret
|
||||
```
|
||||
|
||||
**Telegram Setup**
|
||||
|
||||
1. Chat with [@BotFather](https://t.me/BotFather), send `/newbot`, and copy the HTTP API token.
|
||||
2. Set `TELEGRAM_BOT_TOKEN` in `.env` and enable the channel in `config.yaml`.
|
||||
|
||||
**Slack Setup**
|
||||
|
||||
1. Create a Slack App at [api.slack.com/apps](https://api.slack.com/apps) → Create New App → From scratch.
|
||||
2. Under **OAuth & Permissions**, add Bot Token Scopes: `app_mentions:read`, `chat:write`, `im:history`, `im:read`, `im:write`, `files:write`.
|
||||
3. Enable **Socket Mode** → generate an App-Level Token (`xapp-…`) with `connections:write` scope.
|
||||
4. Under **Event Subscriptions**, subscribe to bot events: `app_mention`, `message.im`.
|
||||
5. Set `SLACK_BOT_TOKEN` and `SLACK_APP_TOKEN` in `.env` and enable the channel in `config.yaml`.
|
||||
|
||||
**Feishu / Lark Setup**
|
||||
|
||||
1. Create an app on [Feishu Open Platform](https://open.feishu.cn/) → enable **Bot** capability.
|
||||
2. Add permissions: `im:message`, `im:message.p2p_msg:readonly`, `im:resource`.
|
||||
3. Under **Events**, subscribe to `im.message.receive_v1` and select **Long Connection** mode.
|
||||
4. Copy the App ID and App Secret. Set `FEISHU_APP_ID` and `FEISHU_APP_SECRET` in `.env` and enable the channel in `config.yaml`.
|
||||
|
||||
**WeChat Setup**
|
||||
|
||||
1. Enable the `wechat` channel in `config.yaml`.
|
||||
2. Either set `WECHAT_BOT_TOKEN` in `.env`, or set `qrcode_login_enabled: true` for first-time QR bootstrap.
|
||||
3. When `bot_token` is absent and QR bootstrap is enabled, watch backend logs for the QR content returned by iLink and complete the binding flow.
|
||||
4. After the QR flow succeeds, DeerFlow persists the acquired token under `state_dir` for later restarts.
|
||||
5. For Docker Compose deployments, keep `state_dir` on a persistent volume so the `get_updates_buf` cursor and saved auth state survive restarts.
|
||||
|
||||
**WeCom Setup**
|
||||
|
||||
1. Create a bot on the WeCom AI Bot platform and obtain the `bot_id` and `bot_secret`.
|
||||
2. Enable `channels.wecom` in `config.yaml` and fill in `bot_id` / `bot_secret`.
|
||||
3. Set `WECOM_BOT_ID` and `WECOM_BOT_SECRET` in `.env`.
|
||||
4. Make sure backend dependencies include `wecom-aibot-python-sdk`. The channel uses a WebSocket long connection and does not require a public callback URL.
|
||||
5. The current integration supports inbound text, image, and file messages. Final images/files generated by the agent are also sent back to the WeCom conversation.
|
||||
|
||||
When DeerFlow runs in Docker Compose, IM channels execute inside the `gateway` container. In that case, do not point `channels.langgraph_url` or `channels.gateway_url` at `localhost`; use container service names such as `http://langgraph:2024` and `http://gateway:8001`, or set `DEER_FLOW_CHANNELS_LANGGRAPH_URL` and `DEER_FLOW_CHANNELS_GATEWAY_URL`.
|
||||
|
||||
**Commands**
|
||||
|
||||
Once a channel is connected, you can interact with DeerFlow directly from the chat:
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `/new` | Start a new conversation |
|
||||
| `/status` | Show current thread info |
|
||||
| `/models` | List available models |
|
||||
| `/memory` | View memory |
|
||||
| `/help` | Show help |
|
||||
|
||||
> Messages without a command prefix are treated as regular chat — DeerFlow creates a thread and responds conversationally.
|
||||
|
||||
#### LangSmith Tracing
|
||||
|
||||
DeerFlow has built-in [LangSmith](https://smith.langchain.com) integration for observability. When enabled, all LLM calls, agent runs, and tool executions are traced and visible in the LangSmith dashboard.
|
||||
|
||||
Add the following to your `.env` file:
|
||||
|
||||
```bash
|
||||
LANGSMITH_TRACING=true
|
||||
LANGSMITH_ENDPOINT=https://api.smith.langchain.com
|
||||
LANGSMITH_API_KEY=lsv2_pt_xxxxxxxxxxxxxxxx
|
||||
LANGSMITH_PROJECT=xxx
|
||||
```
|
||||
|
||||
#### Langfuse Tracing
|
||||
|
||||
DeerFlow also supports [Langfuse](https://langfuse.com) observability for LangChain-compatible runs.
|
||||
|
||||
Add the following to your `.env` file:
|
||||
|
||||
```bash
|
||||
LANGFUSE_TRACING=true
|
||||
LANGFUSE_PUBLIC_KEY=pk-lf-xxxxxxxxxxxxxxxx
|
||||
LANGFUSE_SECRET_KEY=sk-lf-xxxxxxxxxxxxxxxx
|
||||
LANGFUSE_BASE_URL=https://cloud.langfuse.com
|
||||
```
|
||||
|
||||
If you are using a self-hosted Langfuse instance, set `LANGFUSE_BASE_URL` to your deployment URL.
|
||||
|
||||
#### Using Both Providers
|
||||
|
||||
If both LangSmith and Langfuse are enabled, DeerFlow attaches both tracing callbacks and reports the same model activity to both systems.
|
||||
|
||||
If a provider is explicitly enabled but missing required credentials, or if its callback fails to initialize, DeerFlow fails fast when tracing is initialized during model creation and the error message names the provider that caused the failure.
|
||||
|
||||
For Docker deployments, tracing is disabled by default. Set `LANGSMITH_TRACING=true` and `LANGSMITH_API_KEY` in your `.env` to enable it.
|
||||
|
||||
## From Deep Research to Super Agent Harness
|
||||
|
||||
DeerFlow started as a Deep Research framework — and the community ran with it. Since launch, developers have pushed it far beyond research: building data pipelines, generating slide decks, spinning up dashboards, automating content workflows. Things we never anticipated.
|
||||
@@ -564,7 +181,7 @@ That told us something important: DeerFlow wasn't just a research tool. It was a
|
||||
|
||||
So we rebuilt it from scratch.
|
||||
|
||||
DeerFlow 2.0 is no longer a framework you wire together. It's a super agent harness — batteries included, fully extensible. Built on LangGraph and LangChain, it ships with everything an agent needs out of the box: a filesystem, memory, skills, sandbox-aware execution, and the ability to plan and spawn sub-agents for complex, multi-step tasks.
|
||||
DeerFlow 2.0 is no longer a framework you wire together. It's a super agent harness — batteries included, fully extensible. Built on LangGraph and LangChain, it ships with everything an agent needs out of the box: a filesystem, memory, skills, sandboxed execution, and the ability to plan and spawn sub-agents for complex, multi-step tasks.
|
||||
|
||||
Use it as-is. Or tear it apart and make it yours.
|
||||
|
||||
@@ -578,12 +195,8 @@ A standard Agent Skill is a structured capability module — a Markdown file tha
|
||||
|
||||
Skills are loaded progressively — only when the task needs them, not all at once. This keeps the context window lean and makes DeerFlow work well even with token-sensitive models.
|
||||
|
||||
When you install `.skill` archives through the Gateway, DeerFlow accepts standard optional frontmatter metadata such as `version`, `author`, and `compatibility` instead of rejecting otherwise valid external skills.
|
||||
|
||||
Tools follow the same philosophy. DeerFlow comes with a core toolset — web search, web fetch, file operations, bash execution — and supports custom tools via MCP servers and Python functions. Swap anything. Add anything.
|
||||
|
||||
Gateway-generated follow-up suggestions now normalize both plain-string model output and block/list-style rich content before parsing the JSON array response, so provider-specific content wrappers do not silently drop suggestions.
|
||||
|
||||
```
|
||||
# Paths inside the sandbox container
|
||||
/mnt/skills/public
|
||||
@@ -597,35 +210,6 @@ Gateway-generated follow-up suggestions now normalize both plain-string model ou
|
||||
└── your-custom-skill/SKILL.md ← yours
|
||||
```
|
||||
|
||||
#### Claude Code Integration
|
||||
|
||||
The `claude-to-deerflow` skill lets you interact with a running DeerFlow instance directly from [Claude Code](https://docs.anthropic.com/en/docs/claude-code). Send research tasks, check status, manage threads — all without leaving the terminal.
|
||||
|
||||
**Install the skill**:
|
||||
|
||||
```bash
|
||||
npx skills add https://github.com/bytedance/deer-flow --skill claude-to-deerflow
|
||||
```
|
||||
|
||||
Then make sure DeerFlow is running (default at `http://localhost:2026`) and use the `/claude-to-deerflow` command in Claude Code.
|
||||
|
||||
**What you can do**:
|
||||
- Send messages to DeerFlow and get streaming responses
|
||||
- Choose execution modes: flash (fast), standard, pro (planning), ultra (sub-agents)
|
||||
- Check DeerFlow health, list models/skills/agents
|
||||
- Manage threads and conversation history
|
||||
- Upload files for analysis
|
||||
|
||||
**Environment variables** (optional, for custom endpoints):
|
||||
|
||||
```bash
|
||||
DEERFLOW_URL=http://localhost:2026 # Unified proxy base URL
|
||||
DEERFLOW_GATEWAY_URL=http://localhost:2026 # Gateway API
|
||||
DEERFLOW_LANGGRAPH_URL=http://localhost:2026/api/langgraph # LangGraph API
|
||||
```
|
||||
|
||||
See [`skills/public/claude-to-deerflow/SKILL.md`](skills/public/claude-to-deerflow/SKILL.md) for the full API reference.
|
||||
|
||||
### Sub-Agents
|
||||
|
||||
Complex tasks rarely fit in a single pass. DeerFlow decomposes them.
|
||||
@@ -638,9 +222,7 @@ This is how DeerFlow handles tasks that take minutes to hours: a research task m
|
||||
|
||||
DeerFlow doesn't just *talk* about doing things. It has its own computer.
|
||||
|
||||
Each task gets its own execution environment with a full filesystem view — skills, workspace, uploads, outputs. The agent reads, writes, and edits files. It can view images and, when configured safely, execute shell commands.
|
||||
|
||||
With `AioSandboxProvider`, shell execution runs inside isolated containers. With `LocalSandboxProvider`, file tools still map to per-thread directories on the host, but host `bash` is disabled by default because it is not a secure isolation boundary. Re-enable host bash only for fully trusted local workflows.
|
||||
Each task runs inside an isolated Docker container with a full filesystem — skills, workspace, uploads, outputs. The agent reads, writes, and edits files. It executes bash commands and codes. It views images. All sandboxed, all auditable, zero contamination between sessions.
|
||||
|
||||
This is the difference between a chatbot with tool access and an agent with an actual execution environment.
|
||||
|
||||
@@ -664,8 +246,6 @@ Most agents forget everything the moment a conversation ends. DeerFlow remembers
|
||||
|
||||
Across sessions, DeerFlow builds a persistent memory of your profile, preferences, and accumulated knowledge. The more you use it, the better it knows you — your writing style, your technical stack, your recurring workflows. Memory is stored locally and stays under your control.
|
||||
|
||||
Memory updates now skip duplicate fact entries at apply time, so repeated preferences and context do not accumulate endlessly across sessions.
|
||||
|
||||
## Recommended Models
|
||||
|
||||
DeerFlow is model-agnostic — it works with any LLM that implements the OpenAI-compatible API. That said, it performs best with models that support:
|
||||
@@ -677,10 +257,10 @@ DeerFlow is model-agnostic — it works with any LLM that implements the OpenAI-
|
||||
|
||||
## Embedded Python Client
|
||||
|
||||
DeerFlow can be used as an embedded Python library without running the full HTTP services. The `DeerFlowClient` provides direct in-process access to all agent and Gateway capabilities, returning the same response schemas as the HTTP Gateway API. The HTTP Gateway also exposes `DELETE /api/threads/{thread_id}` to remove DeerFlow-managed local thread data after the LangGraph thread itself has been deleted:
|
||||
DeerFlow can be used as an embedded Python library without running the full HTTP services. The `DeerFlowClient` provides direct in-process access to all agent and Gateway capabilities, returning the same response schemas as the HTTP Gateway API:
|
||||
|
||||
```python
|
||||
from deerflow.client import DeerFlowClient
|
||||
from src.client import DeerFlowClient
|
||||
|
||||
client = DeerFlowClient()
|
||||
|
||||
@@ -699,7 +279,7 @@ client.update_skill("web-search", enabled=True)
|
||||
client.upload_files("thread-1", ["./report.pdf"]) # {"success": True, "files": [...]}
|
||||
```
|
||||
|
||||
All dict-returning methods are validated against Gateway Pydantic response models in CI (`TestGatewayConformance`), ensuring the embedded client stays in sync with the HTTP API schemas. See `backend/packages/harness/deerflow/client.py` for full API documentation.
|
||||
All dict-returning methods are validated against Gateway Pydantic response models in CI (`TestGatewayConformance`), ensuring the embedded client stays in sync with the HTTP API schemas. See `backend/src/client.py` for full API documentation.
|
||||
|
||||
## Documentation
|
||||
|
||||
@@ -708,30 +288,11 @@ All dict-returning methods are validated against Gateway Pydantic response model
|
||||
- [Architecture Overview](backend/CLAUDE.md) - Technical architecture details
|
||||
- [Backend Architecture](backend/README.md) - Backend architecture and API reference
|
||||
|
||||
## ⚠️ Security Notice
|
||||
|
||||
### Improper Deployment May Introduce Security Risks
|
||||
|
||||
DeerFlow has key high-privilege capabilities including **system command execution, resource operations, and business logic invocation**, and is designed by default to be **deployed in a local trusted environment (accessible only via the 127.0.0.1 loopback interface)**. If you deploy the agent in untrusted environments — such as LAN networks, public cloud servers, or other multi-endpoint accessible environments — without strict security measures, it may introduce security risks, including:
|
||||
|
||||
- **Unauthorized illegal invocation**: Agent functionality could be discovered by unauthorized third parties or malicious internet scanners, triggering bulk unauthorized requests that execute high-risk operations such as system commands and file read/write, potentially causing serious security consequences.
|
||||
- **Compliance and legal risks**: If the agent is illegally invoked to conduct cyberattacks, data theft, or other illegal activities, it may result in legal liability and compliance risks.
|
||||
|
||||
### Security Recommendations
|
||||
|
||||
**Note: We strongly recommend deploying DeerFlow in a local trusted network environment.** If you need cross-device or cross-network deployment, you must implement strict security measures, such as:
|
||||
|
||||
- **IP allowlist**: Use `iptables`, or deploy hardware firewalls / switches with Access Control Lists (ACL), to **configure IP allowlist rules** and deny access from all other IP addresses.
|
||||
- **Authentication gateway**: Configure a reverse proxy (e.g., nginx) and **enable strong pre-authentication**, blocking any unauthenticated access.
|
||||
- **Network isolation**: Where possible, place the agent and trusted devices in the **same dedicated VLAN**, isolated from other network devices.
|
||||
- **Stay updated**: Continue to follow DeerFlow's security feature updates.
|
||||
|
||||
## Contributing
|
||||
|
||||
We welcome contributions! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, workflow, and guidelines.
|
||||
|
||||
Regression coverage includes Docker sandbox mode detection and provisioner kubeconfig-path handling tests in `backend/tests/`.
|
||||
Gateway artifact serving now forces active web content types (`text/html`, `application/xhtml+xml`, `image/svg+xml`) to download as attachments instead of inline rendering, reducing XSS risk for generated artifacts.
|
||||
|
||||
## License
|
||||
|
||||
|
||||
-610
@@ -1,610 +0,0 @@
|
||||
# 🦌 DeerFlow - 2.0
|
||||
|
||||
[English](./README.md) | [中文](./README_zh.md) | [日本語](./README_ja.md) | Français | [Русский](./README_ru.md)
|
||||
|
||||
[](./backend/pyproject.toml)
|
||||
[](./Makefile)
|
||||
[](./LICENSE)
|
||||
|
||||
<a href="https://trendshift.io/repositories/14699" target="_blank"><img src="https://trendshift.io/api/badge/repositories/14699" alt="bytedance%2Fdeer-flow | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
|
||||
> Le 28 février 2026, DeerFlow a décroché la 🏆 1re place sur GitHub Trending suite au lancement de la version 2. Un immense merci à notre incroyable communauté — c'est grâce à vous ! 💪🔥
|
||||
|
||||
DeerFlow (**D**eep **E**xploration and **E**fficient **R**esearch **Flow**) est un **super agent harness** open source qui orchestre des **sub-agents**, de la **mémoire** et des **sandboxes** pour accomplir pratiquement n'importe quelle tâche — le tout propulsé par des **skills extensibles**.
|
||||
|
||||
https://github.com/user-attachments/assets/a8bcadc4-e040-4cf2-8fda-dd768b999c18
|
||||
|
||||
> [!NOTE]
|
||||
> **DeerFlow 2.0 est une réécriture complète.** Il ne partage aucun code avec la v1. Si vous cherchez le framework Deep Research original, il est maintenu sur la [branche `1.x`](https://github.com/bytedance/deer-flow/tree/main-1.x) — les contributions y sont toujours les bienvenues. Le développement actif a migré vers la 2.0.
|
||||
|
||||
## Site officiel
|
||||
|
||||
[<img width="2880" height="1600" alt="image" src="https://github.com/user-attachments/assets/a598c49f-3b2f-41ea-a052-05e21349188a" />](https://deerflow.tech)
|
||||
|
||||
Découvrez-en plus et regardez des **démos réelles** sur notre [**site officiel**](https://deerflow.tech).
|
||||
|
||||
## Coding Plan de ByteDance Volcengine
|
||||
|
||||
<img width="4808" height="2400" alt="英文方舟" src="https://github.com/user-attachments/assets/2ecc7b9d-50be-4185-b1f7-5542d222fb2d" />
|
||||
|
||||
- Nous recommandons fortement d'utiliser Doubao-Seed-2.0-Code, DeepSeek v3.2 et Kimi 2.5 pour exécuter DeerFlow
|
||||
- [En savoir plus](https://www.byteplus.com/en/activity/codingplan?utm_campaign=deer_flow&utm_content=deer_flow&utm_medium=devrel&utm_source=OWO&utm_term=deer_flow)
|
||||
- [Développeurs en Chine continentale, cliquez ici](https://www.volcengine.com/activity/codingplan?utm_campaign=deer_flow&utm_content=deer_flow&utm_medium=devrel&utm_source=OWO&utm_term=deer_flow)
|
||||
|
||||
## InfoQuest
|
||||
|
||||
DeerFlow intègre désormais le toolkit de recherche et de crawling intelligent développé par BytePlus — [InfoQuest (essai gratuit en ligne)](https://docs.byteplus.com/en/docs/InfoQuest/What_is_Info_Quest)
|
||||
|
||||
<a href="https://docs.byteplus.com/en/docs/InfoQuest/What_is_Info_Quest" target="_blank">
|
||||
<img
|
||||
src="https://sf16-sg.tiktokcdn.com/obj/eden-sg/hubseh7bsbps/20251208-160108.png" alt="InfoQuest_banner"
|
||||
/>
|
||||
</a>
|
||||
|
||||
---
|
||||
|
||||
## Table des matières
|
||||
|
||||
- [🦌 DeerFlow - 2.0](#-deerflow---20)
|
||||
- [Site officiel](#site-officiel)
|
||||
- [InfoQuest](#infoquest)
|
||||
- [Table des matières](#table-des-matières)
|
||||
- [Installation en une phrase pour un coding agent](#installation-en-une-phrase-pour-un-coding-agent)
|
||||
- [Démarrage rapide](#démarrage-rapide)
|
||||
- [Configuration](#configuration)
|
||||
- [Lancer l'application](#lancer-lapplication)
|
||||
- [Option 1 : Docker (recommandé)](#option-1--docker-recommandé)
|
||||
- [Option 2 : Développement local](#option-2--développement-local)
|
||||
- [Avancé](#avancé)
|
||||
- [Mode Sandbox](#mode-sandbox)
|
||||
- [Serveur MCP](#serveur-mcp)
|
||||
- [Canaux de messagerie](#canaux-de-messagerie)
|
||||
- [Traçage LangSmith](#traçage-langsmith)
|
||||
- [Du Deep Research au Super Agent Harness](#du-deep-research-au-super-agent-harness)
|
||||
- [Fonctionnalités principales](#fonctionnalités-principales)
|
||||
- [Skills et outils](#skills-et-outils)
|
||||
- [Intégration Claude Code](#intégration-claude-code)
|
||||
- [Sub-Agents](#sub-agents)
|
||||
- [Sandbox et système de fichiers](#sandbox-et-système-de-fichiers)
|
||||
- [Context Engineering](#context-engineering)
|
||||
- [Mémoire à long terme](#mémoire-à-long-terme)
|
||||
- [Modèles recommandés](#modèles-recommandés)
|
||||
- [Client Python intégré](#client-python-intégré)
|
||||
- [Documentation](#documentation)
|
||||
- [⚠️ Avertissement de sécurité](#️-avertissement-de-sécurité)
|
||||
- [Contribuer](#contribuer)
|
||||
- [Licence](#licence)
|
||||
- [Remerciements](#remerciements)
|
||||
- [Contributeurs principaux](#contributeurs-principaux)
|
||||
- [Star History](#star-history)
|
||||
|
||||
## Installation en une phrase pour un coding agent
|
||||
|
||||
Si vous utilisez Claude Code, Codex, Cursor, Windsurf ou un autre coding agent, vous pouvez simplement lui envoyer cette phrase :
|
||||
|
||||
```text
|
||||
Aide-moi à cloner DeerFlow si nécessaire, puis à initialiser son environnement de développement local en suivant https://raw.githubusercontent.com/bytedance/deer-flow/main/Install.md
|
||||
```
|
||||
|
||||
Ce prompt est destiné aux coding agents. Il leur demande de cloner le dépôt si nécessaire, de privilégier Docker quand il est disponible, puis de s'arrêter avec la commande exacte pour lancer DeerFlow et la liste des configurations encore manquantes.
|
||||
|
||||
## Démarrage rapide
|
||||
|
||||
### Configuration
|
||||
|
||||
1. **Cloner le dépôt DeerFlow**
|
||||
|
||||
```bash
|
||||
git clone https://github.com/bytedance/deer-flow.git
|
||||
cd deer-flow
|
||||
```
|
||||
|
||||
2. **Générer les fichiers de configuration locaux**
|
||||
|
||||
Depuis le répertoire racine du projet (`deer-flow/`), exécutez :
|
||||
|
||||
```bash
|
||||
make config
|
||||
```
|
||||
|
||||
Cette commande crée les fichiers de configuration locaux à partir des templates fournis.
|
||||
|
||||
3. **Configurer le(s) modèle(s) de votre choix**
|
||||
|
||||
Éditez `config.yaml` et définissez au moins un modèle :
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- name: gpt-4 # Internal identifier
|
||||
display_name: GPT-4 # Human-readable name
|
||||
use: langchain_openai:ChatOpenAI # LangChain class path
|
||||
model: gpt-4 # Model identifier for API
|
||||
api_key: $OPENAI_API_KEY # API key (recommended: use env var)
|
||||
max_tokens: 4096 # Maximum tokens per request
|
||||
temperature: 0.7 # Sampling temperature
|
||||
|
||||
- name: openrouter-gemini-2.5-flash
|
||||
display_name: Gemini 2.5 Flash (OpenRouter)
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: google/gemini-2.5-flash-preview
|
||||
api_key: $OPENAI_API_KEY # OpenRouter still uses the OpenAI-compatible field name here
|
||||
base_url: https://openrouter.ai/api/v1
|
||||
|
||||
- name: gpt-5-responses
|
||||
display_name: GPT-5 (Responses API)
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: gpt-5
|
||||
api_key: $OPENAI_API_KEY
|
||||
use_responses_api: true
|
||||
output_version: responses/v1
|
||||
```
|
||||
|
||||
OpenRouter et les passerelles compatibles OpenAI similaires doivent être configurés avec `langchain_openai:ChatOpenAI` et `base_url`. Si vous préférez utiliser un nom de variable d'environnement propre au fournisseur, pointez `api_key` vers cette variable explicitement (par exemple `api_key: $OPENROUTER_API_KEY`).
|
||||
|
||||
Pour router les modèles OpenAI via `/v1/responses`, continuez d'utiliser `langchain_openai:ChatOpenAI` et définissez `use_responses_api: true` avec `output_version: responses/v1`.
|
||||
|
||||
Exemples de providers basés sur un CLI :
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- name: gpt-5.4
|
||||
display_name: GPT-5.4 (Codex CLI)
|
||||
use: deerflow.models.openai_codex_provider:CodexChatModel
|
||||
model: gpt-5.4
|
||||
supports_thinking: true
|
||||
supports_reasoning_effort: true
|
||||
|
||||
- name: claude-sonnet-4.6
|
||||
display_name: Claude Sonnet 4.6 (Claude Code OAuth)
|
||||
use: deerflow.models.claude_provider:ClaudeChatModel
|
||||
model: claude-sonnet-4-6
|
||||
max_tokens: 4096
|
||||
supports_thinking: true
|
||||
```
|
||||
|
||||
- Codex CLI lit `~/.codex/auth.json`
|
||||
- L'endpoint Responses de Codex rejette actuellement `max_tokens` et `max_output_tokens`, donc `CodexChatModel` n'expose pas de limite de tokens par requête
|
||||
- Claude Code accepte `CLAUDE_CODE_OAUTH_TOKEN`, `ANTHROPIC_AUTH_TOKEN`, `CLAUDE_CODE_OAUTH_TOKEN_FILE_DESCRIPTOR`, `CLAUDE_CODE_CREDENTIALS_PATH`, ou en clair `~/.claude/.credentials.json`
|
||||
- Sur macOS, DeerFlow ne sonde pas le Keychain automatiquement. Exportez l'auth Claude Code explicitement si nécessaire :
|
||||
|
||||
```bash
|
||||
eval "$(python3 scripts/export_claude_code_oauth.py --print-export)"
|
||||
```
|
||||
|
||||
4. **Définir les clés API pour le(s) modèle(s) configuré(s)**
|
||||
|
||||
Choisissez l'une des méthodes suivantes :
|
||||
|
||||
- Option A : Éditer le fichier `.env` à la racine du projet (recommandé)
|
||||
|
||||
|
||||
```bash
|
||||
TAVILY_API_KEY=your-tavily-api-key
|
||||
OPENAI_API_KEY=your-openai-api-key
|
||||
# OpenRouter also uses OPENAI_API_KEY when your config uses langchain_openai:ChatOpenAI + base_url.
|
||||
# Add other provider keys as needed
|
||||
INFOQUEST_API_KEY=your-infoquest-api-key
|
||||
```
|
||||
|
||||
- Option B : Exporter les variables d'environnement dans votre shell
|
||||
|
||||
```bash
|
||||
export OPENAI_API_KEY=your-openai-api-key
|
||||
```
|
||||
|
||||
Pour les providers basés sur un CLI :
|
||||
- Codex CLI : `~/.codex/auth.json`
|
||||
- Claude Code OAuth : handoff explicite via env/fichier ou `~/.claude/.credentials.json`
|
||||
|
||||
- Option C : Éditer `config.yaml` directement (non recommandé en production)
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- name: gpt-4
|
||||
api_key: your-actual-api-key-here # Replace placeholder
|
||||
```
|
||||
|
||||
### Lancer l'application
|
||||
|
||||
#### Option 1 : Docker (recommandé)
|
||||
|
||||
**Développement** (hot-reload, montage des sources) :
|
||||
|
||||
```bash
|
||||
make docker-init # Pull sandbox image (only once or when image updates)
|
||||
make docker-start # Start services (auto-detects sandbox mode from config.yaml)
|
||||
```
|
||||
|
||||
`make docker-start` ne lance `provisioner` que si `config.yaml` utilise le mode provisioner (`sandbox.use: deerflow.community.aio_sandbox:AioSandboxProvider` avec `provisioner_url`).
|
||||
Les processus backend récupèrent automatiquement les changements dans `config.yaml` au prochain accès à la configuration, donc les mises à jour de métadonnées des modèles ne nécessitent pas de redémarrage manuel en développement.
|
||||
|
||||
> [!TIP]
|
||||
> Sous Linux, si les commandes Docker échouent avec `permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock`, ajoutez votre utilisateur au groupe `docker` et reconnectez-vous avant de réessayer. Voir [CONTRIBUTING.md](CONTRIBUTING.md#linux-docker-daemon-permission-denied) pour la solution complète.
|
||||
|
||||
**Production** (build des images en local, montage de la config et des données) :
|
||||
|
||||
```bash
|
||||
make up # Build images and start all production services
|
||||
make down # Stop and remove containers
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> Le serveur d'agents LangGraph fonctionne actuellement via `langgraph dev` (le serveur CLI open source).
|
||||
|
||||
Accès : http://localhost:2026
|
||||
|
||||
Voir [CONTRIBUTING.md](CONTRIBUTING.md) pour le guide complet de développement avec Docker.
|
||||
|
||||
#### Option 2 : Développement local
|
||||
|
||||
Si vous préférez lancer les services en local :
|
||||
|
||||
Prérequis : complétez d'abord les étapes de « Configuration » ci-dessus (`make config` et clés API des modèles). `make dev` nécessite un fichier de configuration valide (par défaut `config.yaml` à la racine du projet ; modifiable via `DEER_FLOW_CONFIG_PATH`).
|
||||
|
||||
1. **Vérifier les prérequis** :
|
||||
```bash
|
||||
make check # Verifies Node.js 22+, pnpm, uv, nginx
|
||||
```
|
||||
|
||||
2. **Installer les dépendances** :
|
||||
```bash
|
||||
make install # Install backend + frontend dependencies
|
||||
```
|
||||
|
||||
3. **(Optionnel) Pré-télécharger l'image sandbox** :
|
||||
```bash
|
||||
# Recommended if using Docker/Container-based sandbox
|
||||
make setup-sandbox
|
||||
```
|
||||
|
||||
4. **Démarrer les services** :
|
||||
```bash
|
||||
make dev
|
||||
```
|
||||
|
||||
5. **Accès** : http://localhost:2026
|
||||
|
||||
### Avancé
|
||||
#### Mode Sandbox
|
||||
|
||||
DeerFlow supporte plusieurs modes d'exécution sandbox :
|
||||
- **Exécution locale** (exécute le code sandbox directement sur la machine hôte)
|
||||
- **Exécution Docker** (exécute le code sandbox dans des conteneurs Docker isolés)
|
||||
- **Exécution Docker avec Kubernetes** (exécute le code sandbox dans des pods Kubernetes via le service provisioner)
|
||||
|
||||
En développement Docker, le démarrage des services suit le mode sandbox défini dans `config.yaml`. En mode Local/Docker, `provisioner` n'est pas démarré.
|
||||
|
||||
Voir le [Guide de configuration Sandbox](backend/docs/CONFIGURATION.md#sandbox) pour configurer le mode de votre choix.
|
||||
|
||||
#### Serveur MCP
|
||||
|
||||
DeerFlow supporte des serveurs MCP et des skills configurables pour étendre ses capacités.
|
||||
Pour les serveurs MCP HTTP/SSE, les flux de tokens OAuth sont supportés (`client_credentials`, `refresh_token`).
|
||||
Voir le [Guide MCP Server](backend/docs/MCP_SERVER.md) pour les instructions détaillées.
|
||||
|
||||
#### Canaux de messagerie
|
||||
|
||||
DeerFlow peut recevoir des tâches depuis des applications de messagerie. Les canaux démarrent automatiquement une fois configurés — aucune IP publique n'est requise.
|
||||
|
||||
| Canal | Transport | Difficulté |
|
||||
|---------|-----------|------------|
|
||||
| Telegram | Bot API (long-polling) | Facile |
|
||||
| Slack | Socket Mode | Modérée |
|
||||
| Feishu / Lark | WebSocket | Modérée |
|
||||
|
||||
**Configuration dans `config.yaml` :**
|
||||
|
||||
```yaml
|
||||
channels:
|
||||
# LangGraph Server URL (default: http://localhost:2024)
|
||||
langgraph_url: http://localhost:2024
|
||||
# Gateway API URL (default: http://localhost:8001)
|
||||
gateway_url: http://localhost:8001
|
||||
|
||||
# Optional: global session defaults for all mobile channels
|
||||
session:
|
||||
assistant_id: lead_agent
|
||||
config:
|
||||
recursion_limit: 100
|
||||
context:
|
||||
thinking_enabled: true
|
||||
is_plan_mode: false
|
||||
subagent_enabled: false
|
||||
|
||||
feishu:
|
||||
enabled: true
|
||||
app_id: $FEISHU_APP_ID
|
||||
app_secret: $FEISHU_APP_SECRET
|
||||
# domain: https://open.feishu.cn # China (default)
|
||||
# domain: https://open.larksuite.com # International
|
||||
|
||||
slack:
|
||||
enabled: true
|
||||
bot_token: $SLACK_BOT_TOKEN # xoxb-...
|
||||
app_token: $SLACK_APP_TOKEN # xapp-... (Socket Mode)
|
||||
allowed_users: [] # empty = allow all
|
||||
|
||||
telegram:
|
||||
enabled: true
|
||||
bot_token: $TELEGRAM_BOT_TOKEN
|
||||
allowed_users: [] # empty = allow all
|
||||
|
||||
# Optional: per-channel / per-user session settings
|
||||
session:
|
||||
assistant_id: mobile_agent
|
||||
context:
|
||||
thinking_enabled: false
|
||||
users:
|
||||
"123456789":
|
||||
assistant_id: vip_agent
|
||||
config:
|
||||
recursion_limit: 150
|
||||
context:
|
||||
thinking_enabled: true
|
||||
subagent_enabled: true
|
||||
```
|
||||
|
||||
Définissez les clés API correspondantes dans votre fichier `.env` :
|
||||
|
||||
```bash
|
||||
# Telegram
|
||||
TELEGRAM_BOT_TOKEN=123456789:ABCdefGHIjklMNOpqrSTUvwxYZ
|
||||
|
||||
# Slack
|
||||
SLACK_BOT_TOKEN=xoxb-...
|
||||
SLACK_APP_TOKEN=xapp-...
|
||||
|
||||
# Feishu / Lark
|
||||
FEISHU_APP_ID=cli_xxxx
|
||||
FEISHU_APP_SECRET=your_app_secret
|
||||
```
|
||||
|
||||
**Configuration Telegram**
|
||||
|
||||
1. Ouvrez une conversation avec [@BotFather](https://t.me/BotFather), envoyez `/newbot`, et copiez le token HTTP API.
|
||||
2. Définissez `TELEGRAM_BOT_TOKEN` dans `.env` et activez le canal dans `config.yaml`.
|
||||
|
||||
**Configuration Slack**
|
||||
|
||||
1. Créez une Slack App sur [api.slack.com/apps](https://api.slack.com/apps) → Create New App → From scratch.
|
||||
2. Dans **OAuth & Permissions**, ajoutez les Bot Token Scopes : `app_mentions:read`, `chat:write`, `im:history`, `im:read`, `im:write`, `files:write`.
|
||||
3. Activez le **Socket Mode** → générez un App-Level Token (`xapp-…`) avec le scope `connections:write`.
|
||||
4. Dans **Event Subscriptions**, abonnez-vous aux bot events : `app_mention`, `message.im`.
|
||||
5. Définissez `SLACK_BOT_TOKEN` et `SLACK_APP_TOKEN` dans `.env` et activez le canal dans `config.yaml`.
|
||||
|
||||
**Configuration Feishu / Lark**
|
||||
|
||||
1. Créez une application sur [Feishu Open Platform](https://open.feishu.cn/) → activez la capacité **Bot**.
|
||||
2. Ajoutez les permissions : `im:message`, `im:message.p2p_msg:readonly`, `im:resource`.
|
||||
3. Dans **Events**, abonnez-vous à `im.message.receive_v1` et sélectionnez le mode **Long Connection**.
|
||||
4. Copiez l'App ID et l'App Secret. Définissez `FEISHU_APP_ID` et `FEISHU_APP_SECRET` dans `.env` et activez le canal dans `config.yaml`.
|
||||
|
||||
**Commandes**
|
||||
|
||||
Une fois un canal connecté, vous pouvez interagir avec DeerFlow directement depuis le chat :
|
||||
|
||||
| Commande | Description |
|
||||
|---------|-------------|
|
||||
| `/new` | Démarrer une nouvelle conversation |
|
||||
| `/status` | Afficher les infos du thread en cours |
|
||||
| `/models` | Lister les modèles disponibles |
|
||||
| `/memory` | Consulter la mémoire |
|
||||
| `/help` | Afficher l'aide |
|
||||
|
||||
> Les messages sans préfixe de commande sont traités comme du chat classique — DeerFlow crée un thread et répond de manière conversationnelle.
|
||||
|
||||
#### Traçage LangSmith
|
||||
|
||||
DeerFlow intègre nativement [LangSmith](https://smith.langchain.com) pour l'observabilité. Une fois activé, tous les appels LLM, les exécutions d'agents et les exécutions d'outils sont tracés et visibles dans le tableau de bord LangSmith.
|
||||
|
||||
Ajoutez les lignes suivantes à votre fichier `.env` :
|
||||
|
||||
```bash
|
||||
LANGSMITH_TRACING=true
|
||||
LANGSMITH_ENDPOINT=https://api.smith.langchain.com
|
||||
LANGSMITH_API_KEY=lsv2_pt_xxxxxxxxxxxxxxxx
|
||||
LANGSMITH_PROJECT=xxx
|
||||
```
|
||||
|
||||
Pour les déploiements Docker, le traçage est désactivé par défaut. Définissez `LANGSMITH_TRACING=true` et `LANGSMITH_API_KEY` dans votre `.env` pour l'activer.
|
||||
|
||||
## Du Deep Research au Super Agent Harness
|
||||
|
||||
DeerFlow a démarré comme un framework de Deep Research — et la communauté s'en est emparée. Depuis le lancement, les développeurs l'ont poussé bien au-delà de la recherche : construction de pipelines de données, génération de présentations, mise en place de dashboards, automatisation de workflows de contenu. Des usages qu'on n'avait jamais anticipés.
|
||||
|
||||
Ça nous a révélé quelque chose d'important : DeerFlow n'était pas qu'un simple outil de recherche. C'était un **harness** — un runtime qui donne aux agents l'infrastructure nécessaire pour vraiment accomplir du travail.
|
||||
|
||||
On l'a donc reconstruit de zéro.
|
||||
|
||||
DeerFlow 2.0 n'est plus un framework à assembler soi-même. C'est un super agent harness — clé en main et entièrement extensible. Construit sur LangGraph et LangChain, il embarque tout ce dont un agent a besoin out of the box : un système de fichiers, de la mémoire, des skills, une exécution sandboxée, et la capacité de planifier et de lancer des sub-agents pour les tâches complexes et multi-étapes.
|
||||
|
||||
Utilisez-le tel quel. Ou démontez-le et faites-en le vôtre.
|
||||
|
||||
## Fonctionnalités principales
|
||||
|
||||
### Skills et outils
|
||||
|
||||
Les skills sont ce qui permet à DeerFlow de faire *pratiquement n'importe quoi*.
|
||||
|
||||
Un Agent Skill standard est un module de capacité structuré — un fichier Markdown qui définit un workflow, des bonnes pratiques et des références vers des ressources associées. DeerFlow est livré avec des skills intégrés pour la recherche, la génération de rapports, la création de présentations, les pages web, la génération d'images et de vidéos, et bien plus. Mais la vraie force réside dans l'extensibilité : ajoutez vos propres skills, remplacez ceux fournis, ou combinez-les en workflows composites.
|
||||
|
||||
Les skills sont chargés progressivement — uniquement quand la tâche le nécessite, pas tous en même temps. Ça permet de garder la fenêtre de contexte légère et de bien fonctionner même avec des modèles sensibles au nombre de tokens.
|
||||
|
||||
Quand vous installez des archives `.skill` via le Gateway, DeerFlow accepte les métadonnées frontmatter optionnelles standard comme `version`, `author` et `compatibility`, plutôt que de rejeter des skills externes par ailleurs valides.
|
||||
|
||||
Les outils suivent la même philosophie. DeerFlow est livré avec un ensemble d'outils de base — recherche web, fetch de pages web, opérations sur les fichiers, exécution bash — et supporte les outils custom via des serveurs MCP et des fonctions Python. Remplacez n'importe quoi. Ajoutez n'importe quoi.
|
||||
|
||||
Les suggestions de suivi générées par le Gateway normalisent désormais aussi bien la sortie texte brut du modèle que le contenu riche au format bloc/liste avant de parser la réponse en tableau JSON, de sorte que les wrappers de contenu propres à chaque provider ne suppriment plus silencieusement les suggestions.
|
||||
|
||||
```
|
||||
# Paths inside the sandbox container
|
||||
/mnt/skills/public
|
||||
├── research/SKILL.md
|
||||
├── report-generation/SKILL.md
|
||||
├── slide-creation/SKILL.md
|
||||
├── web-page/SKILL.md
|
||||
└── image-generation/SKILL.md
|
||||
|
||||
/mnt/skills/custom
|
||||
└── your-custom-skill/SKILL.md ← yours
|
||||
```
|
||||
|
||||
#### Intégration Claude Code
|
||||
|
||||
Le skill `claude-to-deerflow` vous permet d'interagir avec une instance DeerFlow en cours d'exécution directement depuis [Claude Code](https://docs.anthropic.com/en/docs/claude-code). Envoyez des tâches de recherche, vérifiez le statut, gérez les threads — le tout sans quitter le terminal.
|
||||
|
||||
**Installer le skill** :
|
||||
|
||||
```bash
|
||||
npx skills add https://github.com/bytedance/deer-flow --skill claude-to-deerflow
|
||||
```
|
||||
|
||||
Assurez-vous ensuite que DeerFlow tourne (par défaut sur `http://localhost:2026`) et utilisez la commande `/claude-to-deerflow` dans Claude Code.
|
||||
|
||||
**Ce que vous pouvez faire** :
|
||||
- Envoyer des messages à DeerFlow et recevoir des réponses en streaming
|
||||
- Choisir le mode d'exécution : flash (rapide), standard, pro (planification), ultra (sub-agents)
|
||||
- Vérifier la santé de DeerFlow, lister les modèles/skills/agents
|
||||
- Gérer les threads et l'historique des conversations
|
||||
- Upload des fichiers pour analyse
|
||||
|
||||
**Variables d'environnement** (optionnel, pour des endpoints custom) :
|
||||
|
||||
```bash
|
||||
DEERFLOW_URL=http://localhost:2026 # Unified proxy base URL
|
||||
DEERFLOW_GATEWAY_URL=http://localhost:2026 # Gateway API
|
||||
DEERFLOW_LANGGRAPH_URL=http://localhost:2026/api/langgraph # LangGraph API
|
||||
```
|
||||
|
||||
Voir [`skills/public/claude-to-deerflow/SKILL.md`](skills/public/claude-to-deerflow/SKILL.md) pour la référence API complète.
|
||||
|
||||
### Sub-Agents
|
||||
|
||||
Les tâches complexes tiennent rarement en une seule passe. DeerFlow les décompose.
|
||||
|
||||
L'agent principal peut lancer des sub-agents à la volée — chacun avec son propre contexte délimité, ses outils et ses conditions d'arrêt. Les sub-agents s'exécutent en parallèle quand c'est possible, remontent des résultats structurés, et l'agent principal synthétise le tout en une sortie cohérente.
|
||||
|
||||
C'est comme ça que DeerFlow gère les tâches qui prennent de quelques minutes à plusieurs heures : une tâche de recherche peut se déployer en une dizaine de sub-agents, chacun explorant un angle différent, puis converger vers un seul rapport — ou un site web — ou un jeu de slides avec des visuels générés. Un seul harness, de nombreuses mains.
|
||||
|
||||
### Sandbox et système de fichiers
|
||||
|
||||
DeerFlow ne se contente pas de *parler* de faire les choses. Il dispose de son propre ordinateur.
|
||||
|
||||
Chaque tâche s'exécute dans un conteneur Docker isolé avec un système de fichiers complet — skills, workspace, uploads, outputs. L'agent lit, écrit et édite des fichiers. Il exécute des commandes bash et du code. Il visualise des images. Le tout sandboxé, le tout auditable, zéro contamination entre les sessions.
|
||||
|
||||
C'est la différence entre un chatbot avec accès à des outils et un agent doté d'un véritable environnement d'exécution.
|
||||
|
||||
```
|
||||
# Paths inside the sandbox container
|
||||
/mnt/user-data/
|
||||
├── uploads/ ← your files
|
||||
├── workspace/ ← agents' working directory
|
||||
└── outputs/ ← final deliverables
|
||||
```
|
||||
|
||||
### Context Engineering
|
||||
|
||||
**Contexte isolé des Sub-Agents** : chaque sub-agent s'exécute dans son propre contexte isolé. Il ne peut voir ni le contexte de l'agent principal, ni celui des autres sub-agents. L'objectif est de garantir que chaque sub-agent reste concentré sur sa tâche sans être parasité par des informations non pertinentes.
|
||||
|
||||
**Résumé** : au sein d'une session, DeerFlow gère le contexte de manière agressive — en résumant les sous-tâches terminées, en déchargeant les résultats intermédiaires vers le système de fichiers, en compressant ce qui n'est plus immédiatement pertinent. Ça lui permet de rester efficace sur des tâches longues et multi-étapes sans faire exploser la fenêtre de contexte.
|
||||
|
||||
### Mémoire à long terme
|
||||
|
||||
La plupart des agents oublient tout dès qu'une conversation se termine. DeerFlow, lui, se souvient.
|
||||
|
||||
D'une session à l'autre, DeerFlow construit une mémoire persistante de votre profil, de vos préférences et de vos connaissances accumulées. Plus vous l'utilisez, mieux il vous connaît — votre style d'écriture, votre stack technique, vos workflows récurrents. La mémoire est stockée localement et reste sous votre contrôle.
|
||||
|
||||
Les mises à jour de la mémoire ignorent désormais les entrées de faits en double au moment de l'application, de sorte que les préférences et le contexte répétés ne s'accumulent plus indéfiniment entre les sessions.
|
||||
|
||||
## Modèles recommandés
|
||||
|
||||
DeerFlow est agnostique en termes de modèle — il fonctionne avec n'importe quel LLM implémentant l'API compatible OpenAI. Cela dit, il offre de meilleures performances avec des modèles qui supportent :
|
||||
|
||||
- **De longues fenêtres de contexte** (100k+ tokens) pour la recherche approfondie et les tâches multi-étapes
|
||||
- **Des capacités de raisonnement** pour la planification adaptative et la décomposition de tâches complexes
|
||||
- **Des entrées multimodales** pour la compréhension d'images et de vidéos
|
||||
- **Un usage fiable des outils (tool use)** pour des appels de fonctions et des sorties structurées fiables
|
||||
|
||||
## Client Python intégré
|
||||
|
||||
DeerFlow peut être utilisé comme bibliothèque Python intégrée sans lancer l'ensemble des services HTTP. Le `DeerFlowClient` fournit un accès direct in-process à toutes les capacités d'agent et de Gateway, en retournant les mêmes schémas de réponse que l'API HTTP Gateway. Le HTTP Gateway expose également `DELETE /api/threads/{thread_id}` pour supprimer les données de thread locales gérées par DeerFlow après la suppression du thread LangGraph :
|
||||
|
||||
```python
|
||||
from deerflow.client import DeerFlowClient
|
||||
|
||||
client = DeerFlowClient()
|
||||
|
||||
# Chat
|
||||
response = client.chat("Analyze this paper for me", thread_id="my-thread")
|
||||
|
||||
# Streaming (LangGraph SSE protocol: values, messages-tuple, end)
|
||||
for event in client.stream("hello"):
|
||||
if event.type == "messages-tuple" and event.data.get("type") == "ai":
|
||||
print(event.data["content"])
|
||||
|
||||
# Configuration & management — returns Gateway-aligned dicts
|
||||
models = client.list_models() # {"models": [...]}
|
||||
skills = client.list_skills() # {"skills": [...]}
|
||||
client.update_skill("web-search", enabled=True)
|
||||
client.upload_files("thread-1", ["./report.pdf"]) # {"success": True, "files": [...]}
|
||||
```
|
||||
|
||||
Toutes les méthodes retournant des dicts sont validées en CI contre les modèles de réponse Pydantic du Gateway (`TestGatewayConformance`), garantissant que le client intégré reste synchronisé avec les schémas de l'API HTTP. Voir `backend/packages/harness/deerflow/client.py` pour la documentation API complète.
|
||||
|
||||
## Documentation
|
||||
|
||||
- [Guide de contribution](CONTRIBUTING.md) - Mise en place de l'environnement de développement et workflow
|
||||
- [Guide de configuration](backend/docs/CONFIGURATION.md) - Instructions d'installation et de configuration
|
||||
- [Vue d'ensemble de l'architecture](backend/CLAUDE.md) - Détails de l'architecture technique
|
||||
- [Architecture backend](backend/README.md) - Architecture backend et référence API
|
||||
|
||||
## ⚠️ Avertissement de sécurité
|
||||
|
||||
### Un déploiement inapproprié peut introduire des risques de sécurité
|
||||
|
||||
DeerFlow dispose de capacités clés à hauts privilèges, notamment **l'exécution de commandes système, les opérations sur les ressources et l'invocation de logique métier**. Il est conçu par défaut pour être **déployé dans un environnement local de confiance (accessible uniquement via l'interface de loopback 127.0.0.1)**. Si vous déployez l'agent dans des environnements non fiables — tels que des réseaux LAN, des serveurs cloud publics ou d'autres environnements accessibles depuis plusieurs terminaux — sans mesures de sécurité strictes, cela peut introduire des risques, notamment :
|
||||
|
||||
- **Invocation non autorisée** : les fonctionnalités de l'agent pourraient être découvertes par des tiers non autorisés ou des scanners malveillants, déclenchant des requêtes non autorisées en masse qui exécutent des opérations à haut risque (commandes système, lecture/écriture de fichiers), pouvant causer de graves conséquences.
|
||||
- **Risques juridiques et de conformité** : si l'agent est utilisé illégalement pour mener des cyberattaques, du vol de données ou d'autres activités illicites, cela peut entraîner des responsabilités juridiques et des risques de conformité.
|
||||
|
||||
### Recommandations de sécurité
|
||||
|
||||
**Note : nous recommandons fortement de déployer DeerFlow dans un environnement réseau local de confiance.** Si vous avez besoin d'un déploiement multi-appareils ou multi-réseaux, vous devez mettre en place des mesures de sécurité strictes, par exemple :
|
||||
|
||||
- **Liste blanche d'IP** : utilisez `iptables`, ou déployez des pare-feux matériels / commutateurs avec ACL, pour **configurer des règles de liste blanche d'IP** et refuser l'accès à toutes les autres adresses IP.
|
||||
- **Passerelle d'authentification** : configurez un proxy inverse (ex. nginx) et **activez une authentification forte en amont**, bloquant tout accès non authentifié.
|
||||
- **Isolation réseau** : si possible, placez l'agent et les appareils de confiance dans le **même VLAN dédié**, isolé des autres équipements réseau.
|
||||
- **Restez informé** : continuez à suivre les mises à jour de sécurité du projet DeerFlow.
|
||||
|
||||
## Contribuer
|
||||
|
||||
Les contributions sont les bienvenues ! Consultez [CONTRIBUTING.md](CONTRIBUTING.md) pour la mise en place de l'environnement de développement, le workflow et les conventions.
|
||||
|
||||
La couverture de tests de régression inclut la détection du mode sandbox Docker et les tests de gestion du kubeconfig-path du provisioner dans `backend/tests/`.
|
||||
|
||||
## Licence
|
||||
|
||||
Ce projet est open source et disponible sous la [Licence MIT](./LICENSE).
|
||||
|
||||
## Remerciements
|
||||
|
||||
DeerFlow est construit sur le travail remarquable de la communauté open source. Nous sommes profondément reconnaissants envers tous les projets et contributeurs dont les efforts ont rendu DeerFlow possible. Nous nous tenons véritablement sur les épaules de géants.
|
||||
|
||||
Nous tenons à exprimer notre sincère gratitude aux projets suivants pour leurs contributions inestimables :
|
||||
|
||||
- **[LangChain](https://github.com/langchain-ai/langchain)** : leur excellent framework propulse nos interactions LLM et nos chaînes, permettant une intégration et des fonctionnalités fluides.
|
||||
- **[LangGraph](https://github.com/langchain-ai/langgraph)** : leur approche innovante de l'orchestration multi-agents a été déterminante pour les workflows sophistiqués de DeerFlow.
|
||||
|
||||
Ces projets illustrent le pouvoir transformateur de la collaboration open source, et nous sommes fiers de bâtir sur leurs fondations.
|
||||
|
||||
### Contributeurs principaux
|
||||
|
||||
Un grand merci aux auteurs principaux de `DeerFlow`, dont la vision, la passion et le dévouement ont donné vie à ce projet :
|
||||
|
||||
- **[Daniel Walnut](https://github.com/hetaoBackend/)**
|
||||
- **[Henry Li](https://github.com/magiccube/)**
|
||||
|
||||
Votre engagement sans faille et votre expertise sont le moteur du succès de DeerFlow. Nous sommes honorés de vous avoir à la barre de cette aventure.
|
||||
|
||||
## Star History
|
||||
|
||||
[](https://star-history.com/#bytedance/deer-flow&Date)
|
||||
-563
@@ -1,563 +0,0 @@
|
||||
# 🦌 DeerFlow - 2.0
|
||||
|
||||
[English](./README.md) | [中文](./README_zh.md) | 日本語 | [Français](./README_fr.md) | [Русский](./README_ru.md)
|
||||
|
||||
[](./backend/pyproject.toml)
|
||||
[](./Makefile)
|
||||
[](./LICENSE)
|
||||
|
||||
<a href="https://trendshift.io/repositories/14699" target="_blank"><img src="https://trendshift.io/api/badge/repositories/14699" alt="bytedance%2Fdeer-flow | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
|
||||
> 2026年2月28日、バージョン2のリリースに伴い、DeerFlowはGitHub Trendingで🏆 第1位を獲得しました。素晴らしいコミュニティの皆さん、ありがとうございます！💪🔥
|
||||
|
||||
DeerFlow（**D**eep **E**xploration and **E**fficient **R**esearch **Flow**）は、**サブエージェント**、**メモリ**、**サンドボックス**を統合し、**拡張可能なスキル**によってあらゆるタスクを実行できるオープンソースの**スーパーエージェントハーネス**です。
|
||||
|
||||
https://github.com/user-attachments/assets/a8bcadc4-e040-4cf2-8fda-dd768b999c18
|
||||
|
||||
> [!NOTE]
|
||||
> **DeerFlow 2.0はゼロからの完全な書き直しです。** v1とコードを共有していません。オリジナルのDeep Researchフレームワークをお探しの場合は、[`1.x`ブランチ](https://github.com/bytedance/deer-flow/tree/main-1.x)で引き続きメンテナンスされています。現在の開発は2.0に移行しています。
|
||||
|
||||
## 公式ウェブサイト
|
||||
|
||||
[<img width="2880" height="1600" alt="image" src="https://github.com/user-attachments/assets/a598c49f-3b2f-41ea-a052-05e21349188a" />](https://deerflow.tech)
|
||||
|
||||
**実際のデモ**は[**公式ウェブサイト**](https://deerflow.tech)でご覧いただけます。
|
||||
|
||||
## ByteDance Volcengine のコーディングプラン
|
||||
|
||||
<img width="4808" height="2400" alt="英文方舟" src="https://github.com/user-attachments/assets/2ecc7b9d-50be-4185-b1f7-5542d222fb2d" />
|
||||
|
||||
- DeerFlowの実行には、Doubao-Seed-2.0-Code、DeepSeek v3.2、Kimi 2.5の使用を強く推奨します
|
||||
- [詳細はこちら](https://www.byteplus.com/en/activity/codingplan?utm_campaign=deer_flow&utm_content=deer_flow&utm_medium=devrel&utm_source=OWO&utm_term=deer_flow)
|
||||
- [中国大陸の開発者はこちらをクリック](https://www.volcengine.com/activity/codingplan?utm_campaign=deer_flow&utm_content=deer_flow&utm_medium=devrel&utm_source=OWO&utm_term=deer_flow)
|
||||
|
||||
## InfoQuest
|
||||
|
||||
DeerFlowは、BytePlusが独自に開発したインテリジェント検索・クローリングツールセット「[InfoQuest（無料オンライン体験対応）](https://docs.byteplus.com/en/docs/InfoQuest/What_is_Info_Quest)」を新たに統合しました。
|
||||
|
||||
<a href="https://docs.byteplus.com/en/docs/InfoQuest/What_is_Info_Quest" target="_blank">
|
||||
<img
|
||||
src="https://sf16-sg.tiktokcdn.com/obj/eden-sg/hubseh7bsbps/20251208-160108.png" alt="InfoQuest_banner"
|
||||
/>
|
||||
</a>
|
||||
|
||||
---
|
||||
|
||||
## 目次
|
||||
|
||||
- [🦌 DeerFlow - 2.0](#-deerflow---20)
|
||||
- [公式ウェブサイト](#公式ウェブサイト)
|
||||
- [InfoQuest](#infoquest)
|
||||
- [目次](#目次)
|
||||
- [Coding Agent に一文でセットアップを依頼](#coding-agent-に一文でセットアップを依頼)
|
||||
- [クイックスタート](#クイックスタート)
|
||||
- [設定](#設定)
|
||||
- [アプリケーションの実行](#アプリケーションの実行)
|
||||
- [オプション1: Docker（推奨）](#オプション1-docker推奨)
|
||||
- [オプション2: ローカル開発](#オプション2-ローカル開発)
|
||||
- [詳細設定](#詳細設定)
|
||||
- [サンドボックスモード](#サンドボックスモード)
|
||||
- [MCPサーバー](#mcpサーバー)
|
||||
- [IMチャネル](#imチャネル)
|
||||
- [LangSmithトレーシング](#langsmithトレーシング)
|
||||
- [Deep Researchからスーパーエージェントハーネスへ](#deep-researchからスーパーエージェントハーネスへ)
|
||||
- [コア機能](#コア機能)
|
||||
- [スキルとツール](#スキルとツール)
|
||||
- [Claude Code連携](#claude-code連携)
|
||||
- [サブエージェント](#サブエージェント)
|
||||
- [サンドボックスとファイルシステム](#サンドボックスとファイルシステム)
|
||||
- [コンテキストエンジニアリング](#コンテキストエンジニアリング)
|
||||
- [長期メモリ](#長期メモリ)
|
||||
- [推奨モデル](#推奨モデル)
|
||||
- [組み込みPythonクライアント](#組み込みpythonクライアント)
|
||||
- [ドキュメント](#ドキュメント)
|
||||
- [⚠️ セキュリティに関する注意](#️-セキュリティに関する注意)
|
||||
- [コントリビュート](#コントリビュート)
|
||||
- [ライセンス](#ライセンス)
|
||||
- [謝辞](#謝辞)
|
||||
- [主要コントリビューター](#主要コントリビューター)
|
||||
- [Star History](#star-history)
|
||||
|
||||
## Coding Agent に一文でセットアップを依頼
|
||||
|
||||
Claude Code、Codex、Cursor、Windsurf などの coding agent を使っているなら、次の一文をそのまま渡せます。
|
||||
|
||||
```text
|
||||
DeerFlow がまだ clone されていなければ先に clone してから、https://raw.githubusercontent.com/bytedance/deer-flow/main/Install.md に従ってローカル開発環境を初期化してください
|
||||
```
|
||||
|
||||
このプロンプトは coding agent 向けです。必要なら先にリポジトリを clone し、Docker が使える場合は Docker を優先して初期セットアップを行い、最後に次の起動コマンドと不足している設定項目だけを返します。
|
||||
|
||||
## クイックスタート
|
||||
|
||||
### 設定
|
||||
|
||||
1. **DeerFlowリポジトリをクローン**
|
||||
|
||||
```bash
|
||||
git clone https://github.com/bytedance/deer-flow.git
|
||||
cd deer-flow
|
||||
```
|
||||
|
||||
2. **ローカル設定ファイルの生成**
|
||||
|
||||
プロジェクトルートディレクトリ（`deer-flow/`）から以下を実行します：
|
||||
|
||||
```bash
|
||||
make config
|
||||
```
|
||||
|
||||
このコマンドは、提供されたテンプレートに基づいてローカル設定ファイルを作成します。
|
||||
|
||||
3. **使用するモデルの設定**
|
||||
|
||||
`config.yaml`を編集し、少なくとも1つのモデルを定義します：
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- name: gpt-4 # 内部識別子
|
||||
display_name: GPT-4 # 表示名
|
||||
use: langchain_openai:ChatOpenAI # LangChainクラスパス
|
||||
model: gpt-4 # API用モデル識別子
|
||||
api_key: $OPENAI_API_KEY # APIキー（推奨：環境変数を使用）
|
||||
max_tokens: 4096 # リクエストあたりの最大トークン数
|
||||
temperature: 0.7 # サンプリング温度
|
||||
|
||||
- name: openrouter-gemini-2.5-flash
|
||||
display_name: Gemini 2.5 Flash (OpenRouter)
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: google/gemini-2.5-flash-preview
|
||||
api_key: $OPENAI_API_KEY # OpenRouterもここではOpenAI互換のフィールド名を使用
|
||||
base_url: https://openrouter.ai/api/v1
|
||||
```
|
||||
|
||||
OpenRouterやOpenAI互換のゲートウェイは、`langchain_openai:ChatOpenAI`と`base_url`で設定します。プロバイダー固有の環境変数名を使用したい場合は、`api_key`でその変数を明示的に指定してください（例：`api_key: $OPENROUTER_API_KEY`）。
|
||||
|
||||
4. **設定したモデルのAPIキーを設定**
|
||||
|
||||
以下のいずれかの方法を選択してください：
|
||||
|
||||
- オプションA：プロジェクトルートの`.env`ファイルを編集（推奨）
|
||||
|
||||
```bash
|
||||
TAVILY_API_KEY=your-tavily-api-key
|
||||
OPENAI_API_KEY=your-openai-api-key
|
||||
# OpenRouterもlangchain_openai:ChatOpenAI + base_url使用時はOPENAI_API_KEYを使用します。
|
||||
# 必要に応じて他のプロバイダーキーを追加
|
||||
INFOQUEST_API_KEY=your-infoquest-api-key
|
||||
```
|
||||
|
||||
- オプションB：シェルで環境変数をエクスポート
|
||||
|
||||
```bash
|
||||
export OPENAI_API_KEY=your-openai-api-key
|
||||
```
|
||||
|
||||
- オプションC：`config.yaml`を直接編集（本番環境には非推奨）
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- name: gpt-4
|
||||
api_key: your-actual-api-key-here # プレースホルダーを置換
|
||||
```
|
||||
|
||||
### アプリケーションの実行
|
||||
|
||||
#### オプション1: Docker（推奨）
|
||||
|
||||
**開発環境**（ホットリロード、ソースマウント）：
|
||||
|
||||
```bash
|
||||
make docker-init # サンドボックスイメージをプル（初回またはイメージ更新時のみ）
|
||||
make docker-start # サービスを開始（config.yamlからサンドボックスモードを自動検出）
|
||||
```
|
||||
|
||||
`make docker-start`は、`config.yaml`がプロビジョナーモード（`sandbox.use: deerflow.community.aio_sandbox:AioSandboxProvider`と`provisioner_url`）を使用している場合にのみ`provisioner`を起動します。
|
||||
|
||||
**本番環境**（ローカルでイメージをビルドし、ランタイム設定とデータをマウント）：
|
||||
|
||||
```bash
|
||||
make up # イメージをビルドして全本番サービスを開始
|
||||
make down # コンテナを停止して削除
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> LangGraphエージェントサーバーは現在`langgraph dev`（オープンソースCLIサーバー）経由で実行されます。
|
||||
|
||||
アクセス: http://localhost:2026
|
||||
|
||||
詳細なDocker開発ガイドは[CONTRIBUTING.md](CONTRIBUTING.md)をご覧ください。
|
||||
|
||||
#### オプション2: ローカル開発
|
||||
|
||||
サービスをローカルで実行する場合：
|
||||
|
||||
前提条件：上記の「設定」手順を先に完了してください（`make config`とモデルAPIキー）。`make dev`には有効な設定ファイルが必要です（デフォルトはプロジェクトルートの`config.yaml`。`DEER_FLOW_CONFIG_PATH`で上書き可能）。
|
||||
|
||||
1. **前提条件の確認**：
|
||||
```bash
|
||||
make check # Node.js 22+、pnpm、uv、nginxを検証
|
||||
```
|
||||
|
||||
2. **依存関係のインストール**：
|
||||
```bash
|
||||
make install # バックエンド＋フロントエンドの依存関係をインストール
|
||||
```
|
||||
|
||||
3. **（オプション）サンドボックスイメージの事前プル**：
|
||||
```bash
|
||||
# Docker/コンテナベースのサンドボックス使用時に推奨
|
||||
make setup-sandbox
|
||||
```
|
||||
|
||||
4. **サービスの開始**：
|
||||
```bash
|
||||
make dev
|
||||
```
|
||||
|
||||
5. **アクセス**: http://localhost:2026
|
||||
|
||||
### 詳細設定
|
||||
#### サンドボックスモード
|
||||
|
||||
DeerFlowは複数のサンドボックス実行モードをサポートしています：
|
||||
- **ローカル実行**（ホストマシン上で直接サンドボックスコードを実行）
|
||||
- **Docker実行**（分離されたDockerコンテナ内でサンドボックスコードを実行）
|
||||
- **KubernetesによるDocker実行**（プロビジョナーサービス経由でKubernetesポッドでサンドボックスコードを実行）
|
||||
|
||||
Docker開発では、サービスの起動は`config.yaml`のサンドボックスモードに従います。ローカル/Dockerモードでは`provisioner`は起動されません。
|
||||
|
||||
お好みのモードの設定については[サンドボックス設定ガイド](backend/docs/CONFIGURATION.md#sandbox)をご覧ください。
|
||||
|
||||
#### MCPサーバー
|
||||
|
||||
DeerFlowは、機能を拡張するための設定可能なMCPサーバーとスキルをサポートしています。
|
||||
HTTP/SSE MCPサーバーでは、OAuthトークンフロー（`client_credentials`、`refresh_token`）がサポートされています。
|
||||
詳細な手順は[MCPサーバーガイド](backend/docs/MCP_SERVER.md)をご覧ください。
|
||||
|
||||
#### IMチャネル
|
||||
|
||||
DeerFlowはメッセージングアプリからのタスク受信をサポートしています。チャネルは設定時に自動的に開始されます。いずれもパブリックIPは不要です。
|
||||
|
||||
| チャネル | トランスポート | 難易度 |
|
||||
|---------|-----------|------------|
|
||||
| Telegram | Bot API（ロングポーリング） | 簡単 |
|
||||
| Slack | Socket Mode | 中程度 |
|
||||
| Feishu / Lark | WebSocket | 中程度 |
|
||||
|
||||
**`config.yaml`での設定：**
|
||||
|
||||
```yaml
|
||||
channels:
|
||||
# LangGraphサーバーURL（デフォルト: http://localhost:2024）
|
||||
langgraph_url: http://localhost:2024
|
||||
# Gateway API URL（デフォルト: http://localhost:8001）
|
||||
gateway_url: http://localhost:8001
|
||||
|
||||
# オプション: 全モバイルチャネルのグローバルセッションデフォルト
|
||||
session:
|
||||
assistant_id: lead_agent
|
||||
config:
|
||||
recursion_limit: 100
|
||||
context:
|
||||
thinking_enabled: true
|
||||
is_plan_mode: false
|
||||
subagent_enabled: false
|
||||
|
||||
feishu:
|
||||
enabled: true
|
||||
app_id: $FEISHU_APP_ID
|
||||
app_secret: $FEISHU_APP_SECRET
|
||||
# domain: https://open.feishu.cn # China (default)
|
||||
# domain: https://open.larksuite.com # International
|
||||
|
||||
slack:
|
||||
enabled: true
|
||||
bot_token: $SLACK_BOT_TOKEN # xoxb-...
|
||||
app_token: $SLACK_APP_TOKEN # xapp-...（Socket Mode）
|
||||
allowed_users: [] # 空 = 全員許可
|
||||
|
||||
telegram:
|
||||
enabled: true
|
||||
bot_token: $TELEGRAM_BOT_TOKEN
|
||||
allowed_users: [] # 空 = 全員許可
|
||||
|
||||
# オプション: チャネル/ユーザーごとのセッション設定
|
||||
session:
|
||||
assistant_id: mobile_agent
|
||||
context:
|
||||
thinking_enabled: false
|
||||
users:
|
||||
"123456789":
|
||||
assistant_id: vip_agent
|
||||
config:
|
||||
recursion_limit: 150
|
||||
context:
|
||||
thinking_enabled: true
|
||||
subagent_enabled: true
|
||||
```
|
||||
|
||||
対応するAPIキーを`.env`ファイルに設定します：
|
||||
|
||||
```bash
|
||||
# Telegram
|
||||
TELEGRAM_BOT_TOKEN=123456789:ABCdefGHIjklMNOpqrSTUvwxYZ
|
||||
|
||||
# Slack
|
||||
SLACK_BOT_TOKEN=xoxb-...
|
||||
SLACK_APP_TOKEN=xapp-...
|
||||
|
||||
# Feishu / Lark
|
||||
FEISHU_APP_ID=cli_xxxx
|
||||
FEISHU_APP_SECRET=your_app_secret
|
||||
```
|
||||
|
||||
**Telegramのセットアップ**
|
||||
|
||||
1. [@BotFather](https://t.me/BotFather)とチャットし、`/newbot`を送信してHTTP APIトークンをコピーします。
|
||||
2. `.env`に`TELEGRAM_BOT_TOKEN`を設定し、`config.yaml`でチャネルを有効にします。
|
||||
|
||||
**Slackのセットアップ**
|
||||
|
||||
1. [api.slack.com/apps](https://api.slack.com/apps)でSlackアプリを作成 → 新規アプリ作成 → 最初から作成。
|
||||
2. **OAuth & Permissions**で、Botトークンスコープを追加：`app_mentions:read`、`chat:write`、`im:history`、`im:read`、`im:write`、`files:write`。
|
||||
3. **Socket Mode**を有効化 → `connections:write`スコープのApp-Levelトークン（`xapp-…`）を生成。
|
||||
4. **Event Subscriptions**で、ボットイベントを購読：`app_mention`、`message.im`。
|
||||
5. `.env`に`SLACK_BOT_TOKEN`と`SLACK_APP_TOKEN`を設定し、`config.yaml`でチャネルを有効にします。
|
||||
|
||||
**Feishu / Larkのセットアップ**
|
||||
|
||||
1. [Feishu Open Platform](https://open.feishu.cn/)でアプリを作成 → **ボット**機能を有効化。
|
||||
2. 権限を追加：`im:message`、`im:message.p2p_msg:readonly`、`im:resource`。
|
||||
3. **イベント**で`im.message.receive_v1`を購読し、**ロングコネクション**モードを選択。
|
||||
4. App IDとApp Secretをコピー。`.env`に`FEISHU_APP_ID`と`FEISHU_APP_SECRET`を設定し、`config.yaml`でチャネルを有効にします。
|
||||
|
||||
**コマンド**
|
||||
|
||||
チャネル接続後、チャットから直接DeerFlowと対話できます：
|
||||
|
||||
| コマンド | 説明 |
|
||||
|---------|-------------|
|
||||
| `/new` | 新しい会話を開始 |
|
||||
| `/status` | 現在のスレッド情報を表示 |
|
||||
| `/models` | 利用可能なモデルを一覧表示 |
|
||||
| `/memory` | メモリを表示 |
|
||||
| `/help` | ヘルプを表示 |
|
||||
|
||||
> コマンドプレフィックスのないメッセージは通常のチャットとして扱われ、DeerFlowがスレッドを作成して会話形式で応答します。
|
||||
|
||||
#### LangSmithトレーシング
|
||||
|
||||
DeerFlowには[LangSmith](https://smith.langchain.com)による可観測性が組み込まれています。有効にすると、すべてのLLM呼び出し、エージェント実行、ツール実行がトレースされ、LangSmithダッシュボードで確認できます。
|
||||
|
||||
`.env`ファイルに以下を追加します：
|
||||
|
||||
```bash
|
||||
LANGSMITH_TRACING=true
|
||||
LANGSMITH_ENDPOINT=https://api.smith.langchain.com
|
||||
LANGSMITH_API_KEY=lsv2_pt_xxxxxxxxxxxxxxxx
|
||||
LANGSMITH_PROJECT=xxx
|
||||
```
|
||||
|
||||
Dockerデプロイでは、トレーシングはデフォルトで無効です。`.env`で`LANGSMITH_TRACING=true`と`LANGSMITH_API_KEY`を設定して有効にします。
|
||||
|
||||
## Deep Researchからスーパーエージェントハーネスへ
|
||||
|
||||
DeerFlowはDeep Researchフレームワークとして始まり、コミュニティがそれを大きく発展させました。リリース以来、開発者たちはリサーチを超えて活用してきました：データパイプラインの構築、スライドデッキの生成、ダッシュボードの立ち上げ、コンテンツワークフローの自動化。私たちが予想もしなかったことです。
|
||||
|
||||
これは重要なことを示していました：DeerFlowは単なるリサーチツールではなかったのです。それは**ハーネス**——エージェントが実際に仕事をこなすためのインフラを提供するランタイムでした。
|
||||
|
||||
そこで、ゼロから再構築しました。
|
||||
|
||||
DeerFlow 2.0は、もはやつなぎ合わせるフレームワークではありません。バッテリー同梱、完全に拡張可能なスーパーエージェントハーネスです。LangGraphとLangChainの上に構築され、エージェントが必要とするすべてを標準搭載しています：ファイルシステム、メモリ、スキル、サンドボックス実行、そして複雑なマルチステップタスクのためのプランニングとサブエージェントの生成機能。
|
||||
|
||||
そのまま使うもよし。分解して自分のものにするもよし。
|
||||
|
||||
## コア機能
|
||||
|
||||
### スキルとツール
|
||||
|
||||
スキルこそが、DeerFlowを*ほぼ何でもできる*ものにしています。
|
||||
|
||||
標準的なエージェントスキルは構造化された機能モジュールです——ワークフロー、ベストプラクティス、サポートリソースへの参照を定義するMarkdownファイルです。DeerFlowにはリサーチ、レポート生成、スライド作成、Webページ、画像・動画生成などの組み込みスキルが付属しています。しかし、真の力は拡張性にあります：独自のスキルを追加し、組み込みスキルを置き換え、複合ワークフローに組み合わせることができます。
|
||||
|
||||
スキルはプログレッシブに読み込まれます——タスクが必要とする時にのみ、一度にすべてではありません。これによりコンテキストウィンドウを軽量に保ち、トークンに敏感なモデルでもDeerFlowがうまく動作します。
|
||||
|
||||
Gateway経由で`.skill`アーカイブをインストールする際、DeerFlowは`version`、`author`、`compatibility`などの標準的なオプショナルフロントマターメタデータを受け入れ、有効な外部スキルを拒否しません。
|
||||
|
||||
ツールも同じ哲学に従います。DeerFlowにはコアツールセット——Web検索、Webフェッチ、ファイル操作、bash実行——が付属し、MCPサーバーやPython関数によるカスタムツールをサポートしています。何でも入れ替え可能、何でも追加可能です。
|
||||
|
||||
Gatewayが生成するフォローアップ提案は、プレーン文字列のモデル出力とブロック/リスト形式のリッチコンテンツの両方をJSON配列レスポンスの解析前に正規化するため、プロバイダー固有のコンテンツラッパーが提案をサイレントにドロップすることはありません。
|
||||
|
||||
```
|
||||
# サンドボックスコンテナ内のパス
|
||||
/mnt/skills/public
|
||||
├── research/SKILL.md
|
||||
├── report-generation/SKILL.md
|
||||
├── slide-creation/SKILL.md
|
||||
├── web-page/SKILL.md
|
||||
└── image-generation/SKILL.md
|
||||
|
||||
/mnt/skills/custom
|
||||
└── your-custom-skill/SKILL.md ← あなたのカスタムスキル
|
||||
```
|
||||
|
||||
#### Claude Code連携
|
||||
|
||||
`claude-to-deerflow`スキルを使えば、[Claude Code](https://docs.anthropic.com/en/docs/claude-code)から直接、実行中のDeerFlowインスタンスと対話できます。リサーチタスクの送信、ステータスの確認、スレッドの管理——すべてターミナルから離れずに実行できます。
|
||||
|
||||
**スキルのインストール**：
|
||||
|
||||
```bash
|
||||
npx skills add https://github.com/bytedance/deer-flow --skill claude-to-deerflow
|
||||
```
|
||||
|
||||
DeerFlowが実行中であることを確認し（デフォルトは`http://localhost:2026`）、Claude Codeで`/claude-to-deerflow`コマンドを使用します。
|
||||
|
||||
**できること**：
|
||||
- DeerFlowにメッセージを送信してストリーミングレスポンスを取得
|
||||
- 実行モードの選択：flash（高速）、standard、pro（プランニング）、ultra（サブエージェント）
|
||||
- DeerFlowのヘルスチェック、モデル/スキル/エージェントの一覧表示
|
||||
- スレッドと会話履歴の管理
|
||||
- 分析用ファイルのアップロード
|
||||
|
||||
**環境変数**（オプション、カスタムエンドポイント用）：
|
||||
|
||||
```bash
|
||||
DEERFLOW_URL=http://localhost:2026 # 統合プロキシベースURL
|
||||
DEERFLOW_GATEWAY_URL=http://localhost:2026 # Gateway API
|
||||
DEERFLOW_LANGGRAPH_URL=http://localhost:2026/api/langgraph # LangGraph API
|
||||
```
|
||||
|
||||
完全なAPIリファレンスは[`skills/public/claude-to-deerflow/SKILL.md`](skills/public/claude-to-deerflow/SKILL.md)をご覧ください。
|
||||
|
||||
### サブエージェント
|
||||
|
||||
複雑なタスクは単一のパスに収まりません。DeerFlowはそれを分解します。
|
||||
|
||||
リードエージェントはオンザフライでサブエージェントを生成できます——それぞれ独自のスコープ付きコンテキスト、ツール、終了条件を持ちます。サブエージェントは可能な限り並列で実行され、構造化された結果を報告し、リードエージェントがすべてを一貫した出力に統合します。
|
||||
|
||||
これがDeerFlowが数分から数時間かかるタスクを処理する方法です：リサーチタスクが十数のサブエージェントに展開され、それぞれが異なる角度を探索し、1つのレポート——またはWebサイト——または生成されたビジュアル付きのスライドデッキに収束します。1つのハーネス、多くの手。
|
||||
|
||||
### サンドボックスとファイルシステム
|
||||
|
||||
DeerFlowは物事を*語る*だけではありません。自分のコンピューターを持っています。
|
||||
|
||||
各タスクは、完全なファイルシステムを持つ分離されたDockerコンテナ内で実行されます——スキル、ワークスペース、アップロード、出力。エージェントはファイルの読み書き・編集を行います。bashコマンドを実行し、コーディングを行います。画像を表示します。すべてサンドボックス化され、すべて監査可能で、セッション間の汚染はゼロです。
|
||||
|
||||
これが、ツールアクセスのあるチャットボットと、実際の実行環境を持つエージェントの違いです。
|
||||
|
||||
```
|
||||
# サンドボックスコンテナ内のパス
|
||||
/mnt/user-data/
|
||||
├── uploads/ ← あなたのファイル
|
||||
├── workspace/ ← エージェントの作業ディレクトリ
|
||||
└── outputs/ ← 最終成果物
|
||||
```
|
||||
|
||||
### コンテキストエンジニアリング
|
||||
|
||||
**分離されたサブエージェントコンテキスト**：各サブエージェントは独自の分離されたコンテキストで実行されます。これにより、サブエージェントはメインエージェントや他のサブエージェントのコンテキストを見ることができません。これは、サブエージェントが目の前のタスクに集中し、メインエージェントや他のサブエージェントのコンテキストに気を取られないようにするために重要です。
|
||||
|
||||
**要約化**：セッション内で、DeerFlowはコンテキストを積極的に管理します——完了したサブタスクの要約、中間結果のファイルシステムへのオフロード、もはや直接関係のないものの圧縮。これにより、コンテキストウィンドウを超えることなく、長いマルチステップタスク全体を通じてシャープさを維持します。
|
||||
|
||||
### 長期メモリ
|
||||
|
||||
ほとんどのエージェントは、会話が終わるとすべてを忘れます。DeerFlowは記憶します。
|
||||
|
||||
セッションをまたいで、DeerFlowはあなたのプロフィール、好み、蓄積された知識の永続的なメモリを構築します。使えば使うほど、あなたのことをよく知るようになります——あなたの文体、技術スタック、繰り返されるワークフロー。メモリはローカルに保存され、あなたの管理下にあります。
|
||||
|
||||
メモリ更新は適用時に重複するファクトエントリをスキップするようになり、繰り返される好みやコンテキストがセッションをまたいで際限なく蓄積されることはありません。
|
||||
|
||||
## 推奨モデル
|
||||
|
||||
DeerFlowはモデルに依存しません——OpenAI互換APIを実装する任意のLLMで動作します。とはいえ、以下をサポートするモデルで最高のパフォーマンスを発揮します：
|
||||
|
||||
- **長いコンテキストウィンドウ**（10万トークン以上）：深いリサーチとマルチステップタスク向け
|
||||
- **推論能力**：適応的なプランニングと複雑な分解向け
|
||||
- **マルチモーダル入力**：画像理解と動画理解向け
|
||||
- **強力なツール使用**：信頼性の高いファンクションコーリングと構造化された出力向け
|
||||
|
||||
## 組み込みPythonクライアント
|
||||
|
||||
DeerFlowは、完全なHTTPサービスを実行せずに組み込みPythonライブラリとして使用できます。`DeerFlowClient`は、すべてのエージェントとGateway機能へのプロセス内直接アクセスを提供し、HTTP Gateway APIと同じレスポンススキーマを返します：
|
||||
|
||||
```python
|
||||
from deerflow.client import DeerFlowClient
|
||||
|
||||
client = DeerFlowClient()
|
||||
|
||||
# チャット
|
||||
response = client.chat("Analyze this paper for me", thread_id="my-thread")
|
||||
|
||||
# ストリーミング（LangGraph SSEプロトコル：values、messages-tuple、end）
|
||||
for event in client.stream("hello"):
|
||||
if event.type == "messages-tuple" and event.data.get("type") == "ai":
|
||||
print(event.data["content"])
|
||||
|
||||
# 設定＆管理 — Gateway準拠のdictを返す
|
||||
models = client.list_models() # {"models": [...]}
|
||||
skills = client.list_skills() # {"skills": [...]}
|
||||
client.update_skill("web-search", enabled=True)
|
||||
client.upload_files("thread-1", ["./report.pdf"]) # {"success": True, "files": [...]}
|
||||
```
|
||||
|
||||
すべてのdict返却メソッドはCIでGateway Pydanticレスポンスモデルに対して検証されており（`TestGatewayConformance`）、組み込みクライアントがHTTP APIスキーマと同期していることを保証します。完全なAPIドキュメントは`backend/packages/harness/deerflow/client.py`をご覧ください。
|
||||
|
||||
## ドキュメント
|
||||
|
||||
- [コントリビュートガイド](CONTRIBUTING.md) - 開発環境のセットアップとワークフロー
|
||||
- [設定ガイド](backend/docs/CONFIGURATION.md) - セットアップと設定の手順
|
||||
- [アーキテクチャ概要](backend/CLAUDE.md) - 技術的なアーキテクチャの詳細
|
||||
- [バックエンドアーキテクチャ](backend/README.md) - バックエンドアーキテクチャとAPIリファレンス
|
||||
|
||||
## ⚠️ セキュリティに関する注意
|
||||
|
||||
### 不適切なデプロイはセキュリティリスクを引き起こす可能性があります
|
||||
|
||||
DeerFlowは**システムコマンドの実行、リソース操作、ビジネスロジックの呼び出し**などの重要な高権限機能を備えており、デフォルトでは**ローカルの信頼できる環境（127.0.0.1のループバックアクセスのみ）にデプロイされる設計**になっています。信頼できないLAN、公開クラウドサーバー、または複数のエンドポイントからアクセス可能なネットワーク環境にエージェントをデプロイし、厳格なセキュリティ対策を講じない場合、以下のようなセキュリティリスクが生じる可能性があります：
|
||||
|
||||
- **不正な違法呼び出し**：エージェントの機能が権限のない第三者や悪意のあるインターネットスキャナーに発見され、システムコマンドやファイル読み書きなどの高リスク操作を実行する不正な一括リクエストが引き起こされ、重大なセキュリティ上の問題が発生する可能性があります。
|
||||
- **コンプライアンスおよび法的リスク**：エージェントがサイバー攻撃やデータ窃取などの違法行為に不正使用された場合、法的責任やコンプライアンス上のリスクが生じる可能性があります。
|
||||
|
||||
### セキュリティ推奨事項
|
||||
|
||||
**注意：DeerFlowはローカルの信頼できるネットワーク環境にデプロイすることを強く推奨します。** クロスデバイス・クロスネットワークのデプロイが必要な場合は、以下のような厳格なセキュリティ対策を実装する必要があります：
|
||||
|
||||
- **IPホワイトリストの設定**：`iptables`を使用するか、ハードウェアファイアウォール / ACL機能付きスイッチをデプロイして**IPホワイトリストルールを設定**し、他のすべてのIPアドレスからのアクセスを拒否します。
|
||||
- **前置認証**：リバースプロキシ（nginxなど）を設定し、**強力な前置認証を有効化**して、認証なしのアクセスをブロックします。
|
||||
- **ネットワーク分離**：可能であれば、エージェントと信頼できるデバイスを**同一の専用VLAN**に配置し、他のネットワークデバイスから隔離します。
|
||||
- **アップデートを継続的に確認**：DeerFlowのセキュリティ機能のアップデートを継続的にフォローしてください。
|
||||
|
||||
## コントリビュート
|
||||
|
||||
コントリビューションを歓迎します！開発環境のセットアップ、ワークフロー、ガイドラインについては[CONTRIBUTING.md](CONTRIBUTING.md)をご覧ください。
|
||||
|
||||
回帰テストのカバレッジには、`backend/tests/`でのDockerサンドボックスモード検出とプロビジョナーkubeconfig-pathハンドリングテストが含まれます。
|
||||
|
||||
## ライセンス
|
||||
|
||||
このプロジェクトはオープンソースであり、[MITライセンス](./LICENSE)の下で提供されています。
|
||||
|
||||
## 謝辞
|
||||
|
||||
DeerFlowはオープンソースコミュニティの素晴らしい成果の上に構築されています。DeerFlowを可能にしてくれたすべてのプロジェクトとコントリビューターに深く感謝いたします。まさに、巨人の肩の上に立っています。
|
||||
|
||||
以下のプロジェクトの貴重な貢献に心からの感謝を申し上げます：
|
||||
|
||||
- **[LangChain](https://github.com/langchain-ai/langchain)**：その優れたフレームワークがLLMのインタラクションとチェーンを支え、シームレスな統合と機能を実現しています。
|
||||
- **[LangGraph](https://github.com/langchain-ai/langgraph)**：マルチエージェントオーケストレーションへの革新的なアプローチが、DeerFlowの洗練されたワークフローの実現に大きく貢献しています。
|
||||
|
||||
これらのプロジェクトはオープンソースコラボレーションの変革的な力を体現しており、その基盤の上に構築できることを誇りに思います。
|
||||
|
||||
### 主要コントリビューター
|
||||
|
||||
`DeerFlow`のコア著者に心からの感謝を捧げます。そのビジョン、情熱、献身がこのプロジェクトに命を吹き込みました：
|
||||
|
||||
- **[Daniel Walnut](https://github.com/hetaoBackend/)**
|
||||
- **[Henry Li](https://github.com/magiccube/)**
|
||||
|
||||
揺るぎないコミットメントと専門知識が、DeerFlowの成功の原動力です。この旅の先頭に立ってくださっていることを光栄に思います。
|
||||
|
||||
## Star History
|
||||
|
||||
[](https://star-history.com/#bytedance/deer-flow&Date)
|
||||
-490
@@ -1,490 +0,0 @@
|
||||
# 🦌 DeerFlow - 2.0
|
||||
|
||||
[English](./README.md) | [中文](./README_zh.md) | [日本語](./README_ja.md) | [Français](./README_fr.md) | Русский
|
||||
|
||||
[](./backend/pyproject.toml)
|
||||
[](./Makefile)
|
||||
[](./LICENSE)
|
||||
|
||||
<a href="https://trendshift.io/repositories/14699" target="_blank"><img src="https://trendshift.io/api/badge/repositories/14699" alt="bytedance%2Fdeer-flow | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
|
||||
|
||||
> 28 февраля 2026 года DeerFlow занял 🏆 #1 в GitHub Trending после релиза версии 2. Спасибо огромное нашему сообществу — всё благодаря вам! 💪🔥
|
||||
|
||||
DeerFlow (**D**eep **E**xploration and **E**fficient **R**esearch **Flow**) — open-source **Super Agent Harness**, который управляет **Sub-Agents**, **Memory** и **Sandbox** для решения почти любой задачи. Всё на основе расширяемых **Skills**.
|
||||
|
||||
https://github.com/user-attachments/assets/a8bcadc4-e040-4cf2-8fda-dd768b999c18
|
||||
|
||||
> [!NOTE]
|
||||
> **DeerFlow 2.0 — проект переписан с нуля.** Общего кода с v1 нет. Если нужен оригинальный Deep Research фреймворк — он живёт в ветке [`1.x`](https://github.com/bytedance/deer-flow/tree/main-1.x), туда тоже принимают контрибьюты. Активная разработка идёт в 2.0.
|
||||
|
||||
## Официальный сайт
|
||||
|
||||
[<img width="2880" height="1600" alt="image" src="https://github.com/user-attachments/assets/a598c49f-3b2f-41ea-a052-05e21349188a" />](https://deerflow.tech)
|
||||
|
||||
Больше информации и живые демо на [**официальном сайте**](https://deerflow.tech).
|
||||
|
||||
## Coding Plan от ByteDance Volcengine
|
||||
|
||||
<img width="4808" height="2400" alt="英文方舟" src="https://github.com/user-attachments/assets/2ecc7b9d-50be-4185-b1f7-5542d222fb2d" />
|
||||
|
||||
- Рекомендуем Doubao-Seed-2.0-Code, DeepSeek v3.2 и Kimi 2.5 для запуска DeerFlow
|
||||
- [Подробнее](https://www.byteplus.com/en/activity/codingplan?utm_campaign=deer_flow&utm_content=deer_flow&utm_medium=devrel&utm_source=OWO&utm_term=deer_flow)
|
||||
- [Для разработчиков из материкового Китая](https://www.volcengine.com/activity/codingplan?utm_campaign=deer_flow&utm_content=deer_flow&utm_medium=devrel&utm_source=OWO&utm_term=deer_flow)
|
||||
|
||||
## InfoQuest
|
||||
|
||||
DeerFlow интегрирован с инструментарием для умного поиска и краулинга от BytePlus — [InfoQuest (есть бесплатный онлайн-доступ)](https://docs.byteplus.com/en/docs/InfoQuest/What_is_Info_Quest)
|
||||
|
||||
<a href="https://docs.byteplus.com/en/docs/InfoQuest/What_is_Info_Quest" target="_blank">
|
||||
<img
|
||||
src="https://sf16-sg.tiktokcdn.com/obj/eden-sg/hubseh7bsbps/20251208-160108.png"
|
||||
alt="InfoQuest_banner"
|
||||
/>
|
||||
</a>
|
||||
|
||||
---
|
||||
|
||||
## Содержание
|
||||
|
||||
- [🦌 DeerFlow - 2.0](#-deerflow---20)
|
||||
- [Официальный сайт](#официальный-сайт)
|
||||
- [InfoQuest](#infoquest)
|
||||
- [Содержание](#содержание)
|
||||
- [Установка одной фразой для coding agent](#установка-одной-фразой-для-coding-agent)
|
||||
- [Быстрый старт](#быстрый-старт)
|
||||
- [Конфигурация](#конфигурация)
|
||||
- [Запуск](#запуск)
|
||||
- [Вариант 1: Docker (рекомендуется)](#вариант-1-docker-рекомендуется)
|
||||
- [Вариант 2: Локальная разработка](#вариант-2-локальная-разработка)
|
||||
- [Дополнительно](#дополнительно)
|
||||
- [Режим Sandbox](#режим-sandbox)
|
||||
- [MCP-сервер](#mcp-сервер)
|
||||
- [Мессенджеры](#мессенджеры)
|
||||
- [Трассировка LangSmith](#трассировка-langsmith)
|
||||
- [От Deep Research к Super Agent Harness](#от-deep-research-к-super-agent-harness)
|
||||
- [Core Features](#core-features)
|
||||
- [Skills & Tools](#skills--tools)
|
||||
- [Интеграция с Claude Code](#интеграция-с-claude-code)
|
||||
- [Sub-Agents](#sub-agents)
|
||||
- [Sandbox & файловая система](#sandbox--файловая-система)
|
||||
- [Context Engineering](#context-engineering)
|
||||
- [Long-Term Memory](#long-term-memory)
|
||||
- [Рекомендуемые модели](#рекомендуемые-модели)
|
||||
- [Встроенный Python-клиент](#встроенный-python-клиент)
|
||||
- [Документация](#документация)
|
||||
- [⚠️ Безопасность](#️-безопасность)
|
||||
- [Участие в разработке](#участие-в-разработке)
|
||||
- [Лицензия](#лицензия)
|
||||
- [Благодарности](#благодарности)
|
||||
- [Ключевые контрибьюторы](#ключевые-контрибьюторы)
|
||||
- [История звёзд](#история-звёзд)
|
||||
|
||||
## Установка одной фразой для coding agent
|
||||
|
||||
Если вы используете Claude Code, Codex, Cursor, Windsurf или другой coding agent, просто отправьте ему эту фразу:
|
||||
|
||||
```text
|
||||
Если DeerFlow еще не клонирован, сначала клонируй его, а затем подготовь локальное окружение разработки по инструкции https://raw.githubusercontent.com/bytedance/deer-flow/main/Install.md
|
||||
```
|
||||
|
||||
Этот prompt предназначен для coding agent. Он просит агента при необходимости сначала клонировать репозиторий, предпочесть Docker, если он доступен, и в конце вернуть точную команду запуска и список недостающих настроек.
|
||||
|
||||
## Быстрый старт
|
||||
|
||||
### Конфигурация
|
||||
|
||||
1. **Склонировать репозиторий DeerFlow**
|
||||
|
||||
```bash
|
||||
git clone https://github.com/bytedance/deer-flow.git
|
||||
cd deer-flow
|
||||
```
|
||||
|
||||
2. **Сгенерировать локальные конфиги**
|
||||
|
||||
Из корня проекта (`deer-flow/`) запустите:
|
||||
|
||||
```bash
|
||||
make config
|
||||
```
|
||||
|
||||
Команда создаёт локальные конфиги на основе шаблонов.
|
||||
|
||||
3. **Настроить модель**
|
||||
|
||||
Отредактируйте `config.yaml` и задайте хотя бы одну модель:
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- name: gpt-4 # Внутренний идентификатор
|
||||
display_name: GPT-4 # Отображаемое имя
|
||||
use: langchain_openai:ChatOpenAI # Путь к классу LangChain
|
||||
model: gpt-4 # Идентификатор модели для API
|
||||
api_key: $OPENAI_API_KEY # API-ключ (рекомендуется: переменная окружения)
|
||||
max_tokens: 4096 # Максимальное количество токенов на запрос
|
||||
temperature: 0.7 # Температура сэмплирования
|
||||
|
||||
- name: openrouter-gemini-2.5-flash
|
||||
display_name: Gemini 2.5 Flash (OpenRouter)
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: google/gemini-2.5-flash-preview
|
||||
api_key: $OPENAI_API_KEY
|
||||
base_url: https://openrouter.ai/api/v1
|
||||
|
||||
- name: gpt-5-responses
|
||||
display_name: GPT-5 (Responses API)
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: gpt-5
|
||||
api_key: $OPENAI_API_KEY
|
||||
use_responses_api: true
|
||||
output_version: responses/v1
|
||||
```
|
||||
|
||||
OpenRouter и аналогичные OpenAI-совместимые шлюзы настраиваются через `langchain_openai:ChatOpenAI` с параметром `base_url`. Для CLI-провайдеров:
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- name: gpt-5.4
|
||||
display_name: GPT-5.4 (Codex CLI)
|
||||
use: deerflow.models.openai_codex_provider:CodexChatModel
|
||||
model: gpt-5.4
|
||||
supports_thinking: true
|
||||
supports_reasoning_effort: true
|
||||
|
||||
- name: claude-sonnet-4.6
|
||||
display_name: Claude Sonnet 4.6 (Claude Code OAuth)
|
||||
use: deerflow.models.claude_provider:ClaudeChatModel
|
||||
model: claude-sonnet-4-6
|
||||
max_tokens: 4096
|
||||
supports_thinking: true
|
||||
```
|
||||
|
||||
- Codex CLI читает `~/.codex/auth.json`
|
||||
- Claude Code принимает `CLAUDE_CODE_OAUTH_TOKEN`, `ANTHROPIC_AUTH_TOKEN` или `~/.claude/.credentials.json`
|
||||
- На macOS при необходимости экспортируйте аутентификацию Claude Code явно:
|
||||
|
||||
```bash
|
||||
eval "$(python3 scripts/export_claude_code_oauth.py --print-export)"
|
||||
```
|
||||
|
||||
4. **Указать API-ключи**
|
||||
|
||||
- **Вариант А**: файл `.env` в корне проекта (рекомендуется)
|
||||
|
||||
```bash
|
||||
TAVILY_API_KEY=your-tavily-api-key
|
||||
OPENAI_API_KEY=your-openai-api-key
|
||||
INFOQUEST_API_KEY=your-infoquest-api-key
|
||||
```
|
||||
|
||||
- **Вариант Б**: переменные окружения в терминале
|
||||
|
||||
```bash
|
||||
export OPENAI_API_KEY=your-openai-api-key
|
||||
```
|
||||
|
||||
- **Вариант В**: напрямую в `config.yaml` (не рекомендуется для продакшена)
|
||||
|
||||
### Запуск
|
||||
|
||||
#### Вариант 1: Docker (рекомендуется)
|
||||
|
||||
**Разработка** (hot-reload, монтирование исходников):
|
||||
|
||||
```bash
|
||||
make docker-init # Загрузить образ Sandbox (один раз или при обновлении)
|
||||
make docker-start # Запустить сервисы
|
||||
```
|
||||
|
||||
**Продакшен** (собирает образы локально):
|
||||
|
||||
```bash
|
||||
make up # Собрать образы и запустить все сервисы
|
||||
make down # Остановить и удалить контейнеры
|
||||
```
|
||||
|
||||
> [!TIP]
|
||||
> На Linux при ошибке `permission denied` для Docker daemon добавьте пользователя в группу `docker` и перелогиньтесь. Подробнее в [CONTRIBUTING.md](CONTRIBUTING.md#linux-docker-daemon-permission-denied).
|
||||
|
||||
Адрес: http://localhost:2026
|
||||
|
||||
#### Вариант 2: Локальная разработка
|
||||
|
||||
1. **Проверить зависимости**:
|
||||
```bash
|
||||
make check # Проверяет Node.js 22+, pnpm, uv, nginx
|
||||
```
|
||||
|
||||
2. **Установить зависимости**:
|
||||
```bash
|
||||
make install
|
||||
```
|
||||
|
||||
3. **(Опционально) Загрузить образ Sandbox заранее**:
|
||||
```bash
|
||||
make setup-sandbox
|
||||
```
|
||||
|
||||
4. **Запустить сервисы**:
|
||||
```bash
|
||||
make dev
|
||||
```
|
||||
|
||||
5. **Адрес**: http://localhost:2026
|
||||
|
||||
### Дополнительно
|
||||
|
||||
#### Режим Sandbox
|
||||
|
||||
DeerFlow поддерживает несколько режимов выполнения:
|
||||
- **Локальное выполнение** — код запускается прямо на хосте
|
||||
- **Docker** — код выполняется в изолированных Docker-контейнерах
|
||||
- **Docker + Kubernetes** — выполнение в Kubernetes-подах через provisioner
|
||||
|
||||
Подробнее в [руководстве по конфигурации Sandbox](backend/docs/CONFIGURATION.md#sandbox).
|
||||
|
||||
#### MCP-сервер
|
||||
|
||||
DeerFlow поддерживает настраиваемые MCP-серверы для расширения возможностей. Для HTTP/SSE MCP-серверов поддерживаются OAuth-токены (`client_credentials`, `refresh_token`). Подробнее в [руководстве по MCP-серверу](backend/docs/MCP_SERVER.md).
|
||||
|
||||
#### Мессенджеры
|
||||
|
||||
DeerFlow принимает задачи прямо из мессенджеров. Каналы запускаются автоматически при настройке, публичный IP не нужен.
|
||||
|
||||
| Канал | Транспорт | Сложность |
|
||||
|-------|-----------|-----------|
|
||||
| Telegram | Bot API (long-polling) | Просто |
|
||||
| Slack | Socket Mode | Средне |
|
||||
| Feishu / Lark | WebSocket | Средне |
|
||||
|
||||
**Конфигурация в `config.yaml`:**
|
||||
|
||||
```yaml
|
||||
channels:
|
||||
feishu:
|
||||
enabled: true
|
||||
app_id: $FEISHU_APP_ID
|
||||
app_secret: $FEISHU_APP_SECRET
|
||||
# domain: https://open.feishu.cn # China (default)
|
||||
# domain: https://open.larksuite.com # International
|
||||
|
||||
slack:
|
||||
enabled: true
|
||||
bot_token: $SLACK_BOT_TOKEN
|
||||
app_token: $SLACK_APP_TOKEN
|
||||
allowed_users: []
|
||||
|
||||
telegram:
|
||||
enabled: true
|
||||
bot_token: $TELEGRAM_BOT_TOKEN
|
||||
allowed_users: []
|
||||
```
|
||||
|
||||
**Настройка Telegram**
|
||||
|
||||
1. Напишите [@BotFather](https://t.me/BotFather), отправьте `/newbot` и скопируйте HTTP API-токен.
|
||||
2. Укажите `TELEGRAM_BOT_TOKEN` в `.env` и включите канал в `config.yaml`.
|
||||
|
||||
**Доступные команды**
|
||||
|
||||
| Команда | Описание |
|
||||
|---------|----------|
|
||||
| `/new` | Начать новый диалог |
|
||||
| `/status` | Показать информацию о текущем треде |
|
||||
| `/models` | Список доступных моделей |
|
||||
| `/memory` | Просмотреть память |
|
||||
| `/help` | Показать справку |
|
||||
|
||||
> Сообщения без команды воспринимаются как обычный чат — DeerFlow создаёт тред и отвечает.
|
||||
|
||||
#### Трассировка LangSmith
|
||||
|
||||
DeerFlow имеет встроенную интеграцию с [LangSmith](https://smith.langchain.com) для наблюдаемости. При включении все вызовы LLM, запуски агентов и выполнения инструментов отслеживаются и отображаются в дашборде LangSmith.
|
||||
|
||||
Добавьте в файл `.env` в корне проекта:
|
||||
|
||||
```bash
|
||||
LANGSMITH_TRACING=true
|
||||
LANGSMITH_API_KEY=lsv2_pt_xxxxxxxxxxxxxxxx
|
||||
LANGSMITH_PROJECT=deer-flow
|
||||
```
|
||||
|
||||
`LANGSMITH_ENDPOINT` по умолчанию `https://api.smith.langchain.com` и может быть переопределён при необходимости. Устаревшие переменные `LANGCHAIN_*` (`LANGCHAIN_TRACING_V2`, `LANGCHAIN_API_KEY` и т.д.) также поддерживаются для обратной совместимости; `LANGSMITH_*` имеет приоритет, когда заданы обе.
|
||||
|
||||
В Docker-развёртываниях трассировка отключена по умолчанию. Установите `LANGSMITH_TRACING=true` и `LANGSMITH_API_KEY` в `.env` для включения.
|
||||
|
||||
## От Deep Research к Super Agent Harness
|
||||
|
||||
DeerFlow начинался как фреймворк для Deep Research, и сообщество вышло далеко за эти рамки. После запуска разработчики строили пайплайны, генерировали презентации, поднимали дашборды, автоматизировали контент. То, чего мы не ожидали.
|
||||
|
||||
Стало понятно: DeerFlow не просто research-инструмент. Это **harness**: runtime, который даёт агентам необходимую инфраструктуру.
|
||||
|
||||
Поэтому мы переписали всё с нуля.
|
||||
|
||||
DeerFlow 2.0 — это Super Agent Harness «из коробки». Batteries included, полностью расширяемый. Построен на LangGraph и LangChain. По умолчанию есть всё, что нужно агенту: файловая система, memory, skills, sandbox-выполнение и возможность планировать и запускать sub-agents для сложных многошаговых задач.
|
||||
|
||||
Используйте как есть. Или разберите и переделайте под себя.
|
||||
|
||||
## Core Features
|
||||
|
||||
### Skills & Tools
|
||||
|
||||
Skills — это то, что позволяет DeerFlow делать почти что угодно.
|
||||
|
||||
Agent Skill — это структурированный модуль: Markdown-файл с описанием воркфлоу, лучших практик и ссылок на ресурсы. DeerFlow поставляется со встроенными skills для ресёрча, генерации отчётов, слайдов, веб-страниц, изображений и видео. Но главное — расширяемость: добавляйте свои skills, заменяйте встроенные или собирайте из них составные воркфлоу.
|
||||
|
||||
Skills загружаются по мере необходимости, только когда задача их требует. Это держит контекстное окно чистым.
|
||||
|
||||
```
|
||||
# Пути внутри контейнера sandbox
|
||||
/mnt/skills/public
|
||||
├── research/SKILL.md
|
||||
├── report-generation/SKILL.md
|
||||
├── slide-creation/SKILL.md
|
||||
├── web-page/SKILL.md
|
||||
└── image-generation/SKILL.md
|
||||
|
||||
/mnt/skills/custom
|
||||
└── your-custom-skill/SKILL.md ← ваш skill
|
||||
```
|
||||
|
||||
#### Интеграция с Claude Code
|
||||
|
||||
Skill `claude-to-deerflow` позволяет работать с DeerFlow прямо из [Claude Code](https://docs.anthropic.com/en/docs/claude-code). Отправляйте задачи, проверяйте статус, управляйте тредами, не выходя из терминала.
|
||||
|
||||
**Установка скилла**:
|
||||
|
||||
```bash
|
||||
npx skills add https://github.com/bytedance/deer-flow --skill claude-to-deerflow
|
||||
```
|
||||
|
||||
**Что можно делать**:
|
||||
- Отправлять сообщения в DeerFlow и получать потоковые ответы
|
||||
- Выбирать режимы выполнения: flash (быстро), standard, pro (planning), ultra (sub-agents)
|
||||
- Проверять статус DeerFlow, просматривать модели, скиллы, агентов
|
||||
- Управлять тредами и историей диалога
|
||||
- Загружать файлы для анализа
|
||||
|
||||
Полный справочник API в [`skills/public/claude-to-deerflow/SKILL.md`](skills/public/claude-to-deerflow/SKILL.md).
|
||||
|
||||
### Sub-Agents
|
||||
|
||||
Сложные задачи редко решаются за один проход. DeerFlow их декомпозирует.
|
||||
|
||||
Lead agent запускает sub-agents на лету, каждый со своим изолированным контекстом, инструментами и условиями завершения. Sub-agents работают параллельно, возвращают структурированные результаты, а lead agent собирает всё в единый итог.
|
||||
|
||||
Вот как DeerFlow справляется с задачами на минуты и часы: research-задача разветвляется в дюжину sub-agents, каждый копает свой угол, потом всё сходится в один отчёт, или сайт, или слайддек со сгенерированными визуалами. Один harness, много рук.
|
||||
|
||||
### Sandbox & файловая система
|
||||
|
||||
DeerFlow не просто *говорит* о том, что умеет что-то делать. У него есть собственный компьютер.
|
||||
|
||||
Каждая задача выполняется внутри изолированного Docker-контейнера с полной файловой системой: skills, workspace, uploads, outputs. Агент читает, пишет и редактирует файлы. Выполняет bash-команды и пишет код. Смотрит на изображения. Всё изолировано, всё прозрачно, никакого пересечения между сессиями.
|
||||
|
||||
Это разница между чатботом с доступом к инструментам и агентом с реальной средой выполнения.
|
||||
|
||||
```
|
||||
# Пути внутри контейнера sandbox
|
||||
/mnt/user-data/
|
||||
├── uploads/ ← ваши файлы
|
||||
├── workspace/ ← рабочая директория агентов
|
||||
└── outputs/ ← результаты
|
||||
```
|
||||
|
||||
### Context Engineering
|
||||
|
||||
**Изолированный контекст**: каждый sub-agent работает в своём контексте и не видит контекст главного агента или других sub-agents. Агент фокусируется на своей задаче.
|
||||
|
||||
**Управление контекстом**: внутри сессии DeerFlow агрессивно сжимает контекст и суммирует завершённые подзадачи, выгружает промежуточные результаты в файловую систему, сжимает то, что уже не актуально. На длинных многошаговых задачах контекстное окно не переполняется.
|
||||
|
||||
### Long-Term Memory
|
||||
|
||||
Большинство агентов забывают всё, когда диалог заканчивается. DeerFlow помнит.
|
||||
|
||||
DeerFlow сохраняет ваш профиль, предпочтения и накопленные знания между сессиями. Чем больше используете, тем лучше он вас знает: стиль, технологический стек, повторяющиеся воркфлоу. Всё хранится локально и остаётся под вашим контролем.
|
||||
|
||||
## Рекомендуемые модели
|
||||
|
||||
DeerFlow работает с любым LLM через OpenAI-совместимый API. Лучше всего — с моделями, которые поддерживают:
|
||||
|
||||
- **Большое контекстное окно** (100k+ токенов) — для deep research и многошаговых задач
|
||||
- **Reasoning capabilities** — для адаптивного планирования и сложной декомпозиции
|
||||
- **Multimodal inputs** — для работы с изображениями и видео
|
||||
- **Strong tool-use** — для надёжного вызова функций и структурированных ответов
|
||||
|
||||
## Встроенный Python-клиент
|
||||
|
||||
DeerFlow можно использовать как Python-библиотеку прямо в коде — без запуска HTTP-сервисов. `DeerFlowClient` даёт доступ ко всем возможностям агента и Gateway, возвращает те же схемы ответов, что и HTTP Gateway API:
|
||||
|
||||
```python
|
||||
from deerflow.client import DeerFlowClient
|
||||
|
||||
client = DeerFlowClient()
|
||||
|
||||
# Chat
|
||||
response = client.chat("Analyze this paper for me", thread_id="my-thread")
|
||||
|
||||
# Streaming (LangGraph SSE protocol: values, messages-tuple, end)
|
||||
for event in client.stream("hello"):
|
||||
if event.type == "messages-tuple" and event.data.get("type") == "ai":
|
||||
print(event.data["content"])
|
||||
|
||||
# Configuration & management — returns Gateway-aligned dicts
|
||||
models = client.list_models() # {"models": [...]}
|
||||
skills = client.list_skills() # {"skills": [...]}
|
||||
client.update_skill("web-search", enabled=True)
|
||||
client.upload_files("thread-1", ["./report.pdf"]) # {"success": True, "files": [...]}
|
||||
```
|
||||
|
||||
## Документация
|
||||
|
||||
- [Руководство по участию](CONTRIBUTING.md) — настройка среды разработки, воркфлоу и гайдлайны
|
||||
- [Руководство по конфигурации](backend/docs/CONFIGURATION.md) — инструкции по настройке
|
||||
- [Обзор архитектуры](backend/CLAUDE.md) — технические детали
|
||||
- [Архитектура бэкенда](backend/README.md) — бэкенд и справочник API
|
||||
|
||||
## ⚠️ Безопасность
|
||||
|
||||
### Неправильное развёртывание может привести к угрозам безопасности
|
||||
|
||||
DeerFlow обладает ключевыми высокопривилегированными возможностями, включая **выполнение системных команд, операции с ресурсами и вызов бизнес-логики**. По умолчанию он рассчитан на **развёртывание в локальной доверенной среде (доступ только через loopback-адрес 127.0.0.1)**. Если вы разворачиваете агент в недоверенных средах — локальных сетях, публичных облачных серверах или других окружениях, доступных с нескольких устройств — без строгих мер безопасности, это может привести к следующим угрозам:
|
||||
|
||||
- **Несанкционированные вызовы**: функциональность агента может быть обнаружена неавторизованными третьими лицами или вредоносными сканерами, что приведёт к массовым несанкционированным запросам с выполнением высокорисковых операций (системные команды, чтение/запись файлов) и серьёзным последствиям для безопасности.
|
||||
- **Юридические и compliance-риски**: если агент будет незаконно использован для кибератак, кражи данных или других противоправных действий, это может повлечь юридическую ответственность и compliance-риски.
|
||||
|
||||
### Рекомендации по безопасности
|
||||
|
||||
**Примечание: настоятельно рекомендуем развёртывать DeerFlow только в локальной доверенной сети.** Если вам необходимо развёртывание через несколько устройств или сетей, обязательно реализуйте строгие меры безопасности, например:
|
||||
|
||||
- **Белый список IP-адресов**: используйте `iptables` или аппаратные межсетевые экраны / коммутаторы с ACL, чтобы **настроить правила белого списка IP** и заблокировать доступ со всех остальных адресов.
|
||||
- **Шлюз аутентификации**: настройте обратный прокси (nginx и др.) и **включите строгую предварительную аутентификацию**, запрещающую любой доступ без авторизации.
|
||||
- **Сетевая изоляция**: по возможности разместите агент и доверенные устройства в **одном выделенном VLAN**, изолированном от остальной сети.
|
||||
- **Следите за обновлениями**: регулярно отслеживайте обновления безопасности проекта DeerFlow.
|
||||
|
||||
## Участие в разработке
|
||||
|
||||
Приветствуем контрибьюторов! Настройка среды разработки, воркфлоу и гайдлайны — в [CONTRIBUTING.md](CONTRIBUTING.md).
|
||||
|
||||
## Лицензия
|
||||
|
||||
Проект распространяется под [лицензией MIT](./LICENSE).
|
||||
|
||||
## Благодарности
|
||||
|
||||
DeerFlow стоит на плечах open-source сообщества. Спасибо всем проектам и разработчикам, чья работа сделала его возможным.
|
||||
|
||||
Отдельная благодарность:
|
||||
|
||||
- **[LangChain](https://github.com/langchain-ai/langchain)** — фреймворк для взаимодействия с LLM и построения цепочек.
|
||||
- **[LangGraph](https://github.com/langchain-ai/langgraph)** — многоагентная оркестрация, на которой держатся сложные воркфлоу DeerFlow.
|
||||
|
||||
### Ключевые контрибьюторы
|
||||
|
||||
Авторы DeerFlow, без которых проекта бы не было:
|
||||
|
||||
- **[Daniel Walnut](https://github.com/hetaoBackend/)**
|
||||
- **[Henry Li](https://github.com/magiccube/)**
|
||||
|
||||
## История звёзд
|
||||
|
||||
[](https://star-history.com/#bytedance/deer-flow&Date)
|
||||
-585
@@ -1,585 +0,0 @@
|
||||
# 🦌 DeerFlow - 2.0
|
||||
|
||||
[English](./README.md) | 中文 | [日本語](./README_ja.md) | [Français](./README_fr.md) | [Русский](./README_ru.md)
|
||||
|
||||
[](./backend/pyproject.toml)
|
||||
[](./Makefile)
|
||||
[](./LICENSE)
|
||||
|
||||
<a href="https://trendshift.io/repositories/14699" target="_blank"><img src="https://trendshift.io/api/badge/repositories/14699" alt="bytedance%2Fdeer-flow | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
|
||||
> 2026 年 2 月 28 日，DeerFlow 2 发布后登上 GitHub Trending 第 1 名。非常感谢社区的支持，这是大家一起做到的。
|
||||
|
||||
DeerFlow（**D**eep **E**xploration and **E**fficient **R**esearch **Flow**）是一个开源的 **super agent harness**。它把 **sub-agents**、**memory** 和 **sandbox** 组织在一起，再配合可扩展的 **skills**，让 agent 可以完成几乎任何事情。
|
||||
|
||||
https://github.com/user-attachments/assets/a8bcadc4-e040-4cf2-8fda-dd768b999c18
|
||||
|
||||
> [!NOTE]
|
||||
> **DeerFlow 2.0 是一次彻底重写。** 它和 v1 没有共用代码。如果你要找的是最初的 Deep Research 框架，可以前往 [`1.x` 分支](https://github.com/bytedance/deer-flow/tree/main-1.x)。那里仍然欢迎贡献；当前的主要开发已经转向 2.0。
|
||||
|
||||
## 官网
|
||||
|
||||
[<img width="2880" height="1600" alt="image" src="https://github.com/user-attachments/assets/a598c49f-3b2f-41ea-a052-05e21349188a" />](https://deerflow.tech)
|
||||
|
||||
想了解更多，或者直接看**真实演示**，可以访问[**官网**](https://deerflow.tech)。
|
||||
|
||||
## 字节跳动火山引擎方舟 Coding Plan
|
||||
|
||||
[<img width="4808" height="2400" alt="codingplan -banner 素材" src="https://github.com/user-attachments/assets/d30dae52-84f2-4021-b32f-6d281252b9ea" />](https://www.volcengine.com/activity/codingplan?utm_campaign=deer_flow&utm_content=deer_flow&utm_medium=devrel&utm_source=OWO&utm_term=deer_flow)
|
||||
|
||||
- 我们推荐使用 Doubao-Seed-2.0-Code、DeepSeek v3.2 和 Kimi 2.5 运行 DeerFlow
|
||||
- [现在就加入 Coding Plan](https://www.volcengine.com/activity/codingplan?utm_campaign=deer_flow&utm_content=deer_flow&utm_medium=devrel&utm_source=OWO&utm_term=deer_flow)
|
||||
- [海外地区的开发者请点击这里](https://www.byteplus.com/en/activity/codingplan?utm_campaign=deer_flow&utm_content=deer_flow&utm_medium=devrel&utm_source=OWO&utm_term=deer_flow)
|
||||
|
||||
## 目录
|
||||
|
||||
- [🦌 DeerFlow - 2.0](#-deerflow---20)
|
||||
- [官网](#官网)
|
||||
- [InfoQuest](#infoquest)
|
||||
- [目录](#目录)
|
||||
- [一句话交给 Coding Agent 安装](#一句话交给-coding-agent-安装)
|
||||
- [快速开始](#快速开始)
|
||||
- [配置](#配置)
|
||||
- [运行应用](#运行应用)
|
||||
- [部署建议与资源规划](#部署建议与资源规划)
|
||||
- [方式一：Docker（推荐）](#方式一docker推荐)
|
||||
- [方式二：本地开发](#方式二本地开发)
|
||||
- [进阶配置](#进阶配置)
|
||||
- [Sandbox 模式](#sandbox-模式)
|
||||
- [MCP Server](#mcp-server)
|
||||
- [IM 渠道](#im-渠道)
|
||||
- [LangSmith 链路追踪](#langsmith-链路追踪)
|
||||
- [从 Deep Research 到 Super Agent Harness](#从-deep-research-到-super-agent-harness)
|
||||
- [核心特性](#核心特性)
|
||||
- [Skills 与 Tools](#skills-与-tools)
|
||||
- [Claude Code 集成](#claude-code-集成)
|
||||
- [Sub-Agents](#sub-agents)
|
||||
- [Sandbox 与文件系统](#sandbox-与文件系统)
|
||||
- [Context Engineering](#context-engineering)
|
||||
- [长期记忆](#长期记忆)
|
||||
- [推荐模型](#推荐模型)
|
||||
- [内嵌 Python Client](#内嵌-python-client)
|
||||
- [文档](#文档)
|
||||
- [⚠️ 安全使用](#️-安全使用)
|
||||
- [参与贡献](#参与贡献)
|
||||
- [许可证](#许可证)
|
||||
- [致谢](#致谢)
|
||||
- [核心贡献者](#核心贡献者)
|
||||
- [Star History](#star-history)
|
||||
|
||||
## 一句话交给 Coding Agent 安装
|
||||
|
||||
如果你在用 Claude Code、Codex、Cursor、Windsurf 或其他 coding agent，可以直接把下面这句话发给它：
|
||||
|
||||
```text
|
||||
如果还没 clone DeerFlow，就先 clone，然后按照 https://raw.githubusercontent.com/bytedance/deer-flow/main/Install.md 把它的本地开发环境初始化好
|
||||
```
|
||||
|
||||
这条提示词是给 coding agent 用的。它会在需要时先 clone 仓库，优先选择 Docker，完成初始化，并在结束时告诉你下一条启动命令，以及还缺哪些配置需要你补充。
|
||||
|
||||
## 快速开始
|
||||
|
||||
### 配置
|
||||
|
||||
1. **克隆 DeerFlow 仓库**
|
||||
|
||||
```bash
|
||||
git clone https://github.com/bytedance/deer-flow.git
|
||||
cd deer-flow
|
||||
```
|
||||
|
||||
2. **生成本地配置文件**
|
||||
|
||||
在项目根目录（`deer-flow/`）执行：
|
||||
|
||||
```bash
|
||||
make config
|
||||
```
|
||||
|
||||
这个命令会基于示例模板生成本地配置文件。
|
||||
|
||||
3. **配置你要使用的模型**
|
||||
|
||||
编辑 `config.yaml`，至少定义一个模型：
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- name: gpt-4 # 内部标识
|
||||
display_name: GPT-4 # 展示名称
|
||||
use: langchain_openai:ChatOpenAI # LangChain 类路径
|
||||
model: gpt-4 # API 使用的模型标识
|
||||
api_key: $OPENAI_API_KEY # API key（推荐使用环境变量）
|
||||
max_tokens: 4096 # 单次请求最大 tokens
|
||||
temperature: 0.7 # 采样温度
|
||||
|
||||
- name: openrouter-gemini-2.5-flash
|
||||
display_name: Gemini 2.5 Flash (OpenRouter)
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: google/gemini-2.5-flash-preview
|
||||
api_key: $OPENAI_API_KEY # 这里 OpenRouter 依然沿用 OpenAI 兼容字段名
|
||||
base_url: https://openrouter.ai/api/v1
|
||||
```
|
||||
|
||||
OpenRouter 以及类似的 OpenAI 兼容网关，建议通过 `langchain_openai:ChatOpenAI` 配合 `base_url` 来配置。如果你更想用 provider 自己的环境变量名，也可以直接把 `api_key` 指向对应变量，例如 `api_key: $OPENROUTER_API_KEY`。
|
||||
|
||||
4. **为已配置的模型设置 API key**
|
||||
|
||||
可任选以下一种方式：
|
||||
|
||||
- 方式 A：编辑项目根目录下的 `.env` 文件（推荐）
|
||||
|
||||
```bash
|
||||
TAVILY_API_KEY=your-tavily-api-key
|
||||
OPENAI_API_KEY=your-openai-api-key
|
||||
# 如果配置使用的是 langchain_openai:ChatOpenAI + base_url，OpenRouter 也会读取 OPENAI_API_KEY
|
||||
# 其他 provider 的 key 按需补充
|
||||
INFOQUEST_API_KEY=your-infoquest-api-key
|
||||
```
|
||||
|
||||
- 方式 B：在 shell 中导出环境变量
|
||||
|
||||
```bash
|
||||
export OPENAI_API_KEY=your-openai-api-key
|
||||
```
|
||||
|
||||
- 方式 C：直接编辑 `config.yaml`（不建议用于生产环境）
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- name: gpt-4
|
||||
api_key: your-actual-api-key-here # 替换为真实 key
|
||||
```
|
||||
|
||||
### 运行应用
|
||||
|
||||
#### 部署建议与资源规划
|
||||
|
||||
可以先按下面的资源档位来选择 DeerFlow 的运行方式：
|
||||
|
||||
| 部署场景 | 起步配置 | 推荐配置 | 说明 |
|
||||
|---------|-----------|------------|-------|
|
||||
| 本地体验 / `make dev` | 4 vCPU、8 GB 内存、20 GB SSD 可用空间 | 8 vCPU、16 GB 内存 | 适合单个开发者或单个轻量会话，且模型走外部 API。`2 核 / 4 GB` 通常跑不稳。 |
|
||||
| Docker 开发 / `make docker-start` | 4 vCPU、8 GB 内存、25 GB SSD 可用空间 | 8 vCPU、16 GB 内存 | 镜像构建、源码挂载和 sandbox 容器都会比纯本地模式更吃资源。 |
|
||||
| 长期运行服务 / `make up` | 8 vCPU、16 GB 内存、40 GB SSD 可用空间 | 16 vCPU、32 GB 内存 | 更适合共享环境、多 agent 任务、报告生成或更重的 sandbox 负载。 |
|
||||
|
||||
- 上面的配置只覆盖 DeerFlow 本身；如果你还要本机部署本地大模型，请单独为模型服务预留资源。
|
||||
- 持续运行的服务更推荐使用 Linux + Docker。macOS 和 Windows 更适合作为开发机或体验环境。
|
||||
- 如果 CPU 或内存长期打满，先降低并发会话或重任务数量，再考虑升级到更高一档配置。
|
||||
|
||||
#### 方式一：Docker（推荐）
|
||||
|
||||
**开发模式**（支持热更新，挂载源码）：
|
||||
|
||||
```bash
|
||||
make docker-init # 拉取 sandbox 镜像（首次运行或镜像更新时执行）
|
||||
make docker-start # 启动服务（会根据 config.yaml 自动判断 sandbox 模式）
|
||||
```
|
||||
|
||||
如果 `config.yaml` 使用的是 provisioner 模式（`sandbox.use: deerflow.community.aio_sandbox:AioSandboxProvider` 且配置了 `provisioner_url`），`make docker-start` 才会启动 `provisioner`。
|
||||
|
||||
**生产模式**（本地构建镜像，并挂载运行期配置与数据）：
|
||||
|
||||
```bash
|
||||
make up # 构建镜像并启动全部生产服务
|
||||
make down # 停止并移除容器
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> 当前 LangGraph agent server 通过开源 CLI 服务 `langgraph dev` 运行。
|
||||
|
||||
访问地址：http://localhost:2026
|
||||
|
||||
更完整的 Docker 开发说明见 [CONTRIBUTING.md](CONTRIBUTING.md)。
|
||||
|
||||
#### 方式二：本地开发
|
||||
|
||||
如果你更希望直接在本地启动各个服务：
|
||||
|
||||
前提：先完成上面的“配置”步骤（`make config` 和模型 API key 配置）。`make dev` 需要有效配置文件，默认读取项目根目录下的 `config.yaml`，也可以通过 `DEER_FLOW_CONFIG_PATH` 覆盖。
|
||||
在 Windows 上，请使用 Git Bash 运行本地开发流程。基于 bash 的服务脚本不支持直接在原生 `cmd.exe` 或 PowerShell 中执行，且 WSL 也不保证可用，因为部分脚本依赖 Git for Windows 的 `cygpath` 等工具。
|
||||
|
||||
1. **检查依赖环境**：
|
||||
```bash
|
||||
make check # 校验 Node.js 22+、pnpm、uv、nginx
|
||||
```
|
||||
|
||||
2. **安装依赖**：
|
||||
```bash
|
||||
make install # 安装 backend + frontend 依赖
|
||||
```
|
||||
|
||||
3. **（可选）预拉取 sandbox 镜像**：
|
||||
```bash
|
||||
# 如果使用 Docker / Container sandbox，建议先执行
|
||||
make setup-sandbox
|
||||
```
|
||||
|
||||
4. **启动服务**：
|
||||
```bash
|
||||
make dev
|
||||
```
|
||||
|
||||
5. **访问地址**：http://localhost:2026
|
||||
|
||||
### 进阶配置
|
||||
#### Sandbox 模式
|
||||
|
||||
DeerFlow 支持多种 sandbox 执行方式：
|
||||
- **本地执行**（直接在宿主机上运行 sandbox 代码）
|
||||
- **Docker 执行**（在隔离的 Docker 容器里运行 sandbox 代码）
|
||||
- **Docker + Kubernetes 执行**（通过 provisioner 服务在 Kubernetes Pod 中运行 sandbox 代码）
|
||||
|
||||
Docker 开发时，服务启动行为会遵循 `config.yaml` 里的 sandbox 模式。在 Local / Docker 模式下，不会启动 `provisioner`。
|
||||
|
||||
如果要配置你自己的模式，参见 [Sandbox 配置指南](backend/docs/CONFIGURATION.md#sandbox)。
|
||||
|
||||
#### MCP Server
|
||||
|
||||
DeerFlow 支持可配置的 MCP Server 和 skills，用来扩展能力。
|
||||
对于 HTTP/SSE MCP Server，还支持 OAuth token 流程（`client_credentials`、`refresh_token`）。
|
||||
详细说明见 [MCP Server 指南](backend/docs/MCP_SERVER.md)。
|
||||
|
||||
#### IM 渠道
|
||||
|
||||
DeerFlow 支持从即时通讯应用接收任务。只要配置完成，对应渠道会自动启动，而且都不需要公网 IP。
|
||||
|
||||
| 渠道 | 传输方式 | 上手难度 |
|
||||
|---------|-----------|------------|
|
||||
| Telegram | Bot API（long-polling） | 简单 |
|
||||
| Slack | Socket Mode | 中等 |
|
||||
| Feishu / Lark | WebSocket | 中等 |
|
||||
| 企业微信智能机器人 | WebSocket | 中等 |
|
||||
|
||||
**`config.yaml` 中的配置示例：**
|
||||
|
||||
```yaml
|
||||
channels:
|
||||
# LangGraph Server URL（默认：http://localhost:2024）
|
||||
langgraph_url: http://localhost:2024
|
||||
# Gateway API URL（默认：http://localhost:8001）
|
||||
gateway_url: http://localhost:8001
|
||||
|
||||
# 可选：所有移动端渠道共用的全局 session 默认值
|
||||
session:
|
||||
assistant_id: lead_agent # 也可以填自定义 agent 名；渠道层会自动转换为 lead_agent + agent_name
|
||||
config:
|
||||
recursion_limit: 100
|
||||
context:
|
||||
thinking_enabled: true
|
||||
is_plan_mode: false
|
||||
subagent_enabled: false
|
||||
|
||||
feishu:
|
||||
enabled: true
|
||||
app_id: $FEISHU_APP_ID
|
||||
app_secret: $FEISHU_APP_SECRET
|
||||
# domain: https://open.feishu.cn # 国内版（默认）
|
||||
# domain: https://open.larksuite.com # 国际版
|
||||
|
||||
wecom:
|
||||
enabled: true
|
||||
bot_id: $WECOM_BOT_ID
|
||||
bot_secret: $WECOM_BOT_SECRET
|
||||
|
||||
slack:
|
||||
enabled: true
|
||||
bot_token: $SLACK_BOT_TOKEN # xoxb-...
|
||||
app_token: $SLACK_APP_TOKEN # xapp-...（Socket Mode）
|
||||
allowed_users: [] # 留空表示允许所有人
|
||||
|
||||
telegram:
|
||||
enabled: true
|
||||
bot_token: $TELEGRAM_BOT_TOKEN
|
||||
allowed_users: [] # 留空表示允许所有人
|
||||
|
||||
# 可选：按渠道 / 按用户单独覆盖 session 配置
|
||||
session:
|
||||
assistant_id: mobile-agent # 这里同样支持自定义 agent 名
|
||||
context:
|
||||
thinking_enabled: false
|
||||
users:
|
||||
"123456789":
|
||||
assistant_id: vip-agent
|
||||
config:
|
||||
recursion_limit: 150
|
||||
context:
|
||||
thinking_enabled: true
|
||||
subagent_enabled: true
|
||||
```
|
||||
|
||||
说明：
|
||||
- `assistant_id: lead_agent` 会直接调用默认的 LangGraph assistant。
|
||||
- 如果 `assistant_id` 填的是自定义 agent 名，DeerFlow 仍然会走 `lead_agent`，同时把该值注入为 `agent_name`，这样 IM 渠道也会生效对应 agent 的 SOUL 和配置。
|
||||
|
||||
在 `.env` 里设置对应的 API key：
|
||||
|
||||
```bash
|
||||
# Telegram
|
||||
TELEGRAM_BOT_TOKEN=123456789:ABCdefGHIjklMNOpqrSTUvwxYZ
|
||||
|
||||
# Slack
|
||||
SLACK_BOT_TOKEN=xoxb-...
|
||||
SLACK_APP_TOKEN=xapp-...
|
||||
|
||||
# Feishu / Lark
|
||||
FEISHU_APP_ID=cli_xxxx
|
||||
FEISHU_APP_SECRET=your_app_secret
|
||||
|
||||
# 企业微信智能机器人
|
||||
WECOM_BOT_ID=your_bot_id
|
||||
WECOM_BOT_SECRET=your_bot_secret
|
||||
```
|
||||
|
||||
**Telegram 配置**
|
||||
|
||||
1. 打开 [@BotFather](https://t.me/BotFather)，发送 `/newbot`，复制生成的 HTTP API token。
|
||||
2. 在 `.env` 中设置 `TELEGRAM_BOT_TOKEN`，并在 `config.yaml` 里启用该渠道。
|
||||
|
||||
**Slack 配置**
|
||||
|
||||
1. 前往 [api.slack.com/apps](https://api.slack.com/apps) 创建 Slack App：Create New App → From scratch。
|
||||
2. 在 **OAuth & Permissions** 中添加 Bot Token Scopes：`app_mentions:read`、`chat:write`、`im:history`、`im:read`、`im:write`、`files:write`。
|
||||
3. 启用 **Socket Mode**，生成带 `connections:write` 权限的 App-Level Token（`xapp-...`）。
|
||||
4. 在 **Event Subscriptions** 中订阅 bot events：`app_mention`、`message.im`。
|
||||
5. 在 `.env` 中设置 `SLACK_BOT_TOKEN` 和 `SLACK_APP_TOKEN`，并在 `config.yaml` 中启用该渠道。
|
||||
|
||||
**Feishu / Lark 配置**
|
||||
|
||||
1. 在 [飞书开放平台](https://open.feishu.cn/) 创建应用，并启用 **Bot** 能力。
|
||||
2. 添加权限：`im:message`、`im:message.p2p_msg:readonly`、`im:resource`。
|
||||
3. 在 **事件订阅** 中订阅 `im.message.receive_v1`，连接方式选择 **长连接**。
|
||||
4. 复制 App ID 和 App Secret，在 `.env` 中设置 `FEISHU_APP_ID` 和 `FEISHU_APP_SECRET`，并在 `config.yaml` 中启用该渠道。
|
||||
|
||||
**企业微信智能机器人配置**
|
||||
|
||||
1. 在企业微信智能机器人平台创建机器人，获取 `bot_id` 和 `bot_secret`。
|
||||
2. 在 `config.yaml` 中启用 `channels.wecom`，并填入 `bot_id` / `bot_secret`。
|
||||
3. 在 `.env` 中设置 `WECOM_BOT_ID` 和 `WECOM_BOT_SECRET`。
|
||||
4. 安装后端依赖时确保包含 `wecom-aibot-python-sdk`，渠道会通过 WebSocket 长连接接收消息，无需公网回调地址。
|
||||
5. 当前支持文本、图片和文件入站消息；agent 生成的最终图片/文件也会回传到企业微信会话中。
|
||||
|
||||
**命令**
|
||||
|
||||
渠道连接完成后，你可以直接在聊天窗口里和 DeerFlow 交互：
|
||||
|
||||
| 命令 | 说明 |
|
||||
|---------|-------------|
|
||||
| `/new` | 开启新对话 |
|
||||
| `/status` | 查看当前 thread 信息 |
|
||||
| `/models` | 列出可用模型 |
|
||||
| `/memory` | 查看 memory |
|
||||
| `/help` | 查看帮助 |
|
||||
|
||||
> 没有命令前缀的消息会被当作普通聊天处理。DeerFlow 会自动创建 thread，并以对话方式回复。
|
||||
|
||||
#### LangSmith 链路追踪
|
||||
|
||||
DeerFlow 内置了 [LangSmith](https://smith.langchain.com) 集成，用于可观测性。启用后，所有 LLM 调用、agent 运行和工具执行都会被追踪，并在 LangSmith 仪表盘中展示。
|
||||
|
||||
在 `.env` 文件中添加以下配置：
|
||||
|
||||
```bash
|
||||
LANGSMITH_TRACING=true
|
||||
LANGSMITH_ENDPOINT=https://api.smith.langchain.com
|
||||
LANGSMITH_API_KEY=lsv2_pt_xxxxxxxxxxxxxxxx
|
||||
LANGSMITH_PROJECT=xxx
|
||||
```
|
||||
|
||||
Docker 部署时，追踪默认关闭。在 `.env` 中设置 `LANGSMITH_TRACING=true` 和 `LANGSMITH_API_KEY` 即可启用。
|
||||
|
||||
## 从 Deep Research 到 Super Agent Harness
|
||||
|
||||
DeerFlow 最初是一个 Deep Research 框架，后来社区把它一路推到了更远的地方。上线之后，开发者拿它去做的事情早就不止研究：搭数据流水线、生成演示文稿、快速起 dashboard、自动化内容流程，很多方向一开始连我们自己都没想到。
|
||||
|
||||
这让我们意识到一件事：DeerFlow 不只是一个研究工具。它更像一个 **harness**，一个真正让 agents 把事情做完的运行时基础设施。
|
||||
|
||||
所以我们把它从头重做了一遍。
|
||||
|
||||
DeerFlow 2.0 不再是一个需要你自己拼装的 framework。它是一个开箱即用、同时又足够可扩展的 super agent harness。基于 LangGraph 和 LangChain 构建，默认就带上了 agent 真正会用到的关键能力：文件系统、memory、skills、sandbox 执行环境，以及为复杂多步骤任务做规划、拉起 sub-agents 的能力。
|
||||
|
||||
你可以直接拿来用，也可以拆开重组，改成你自己的样子。
|
||||
|
||||
## 核心特性
|
||||
|
||||
### Skills 与 Tools
|
||||
|
||||
Skills 是 DeerFlow 能做“几乎任何事”的关键。
|
||||
|
||||
标准的 Agent Skill 是一种结构化能力模块，通常就是一个 Markdown 文件，里面定义了工作流、最佳实践，以及相关的参考资源。DeerFlow 自带一批内置 skills，覆盖研究、报告生成、演示文稿制作、网页生成、图像和视频生成等场景。真正有意思的地方在于它的扩展性：你可以加自己的 skills，替换内置 skills，或者把多个 skills 组合成复合工作流。
|
||||
|
||||
Skills 采用按需渐进加载，不会一次性把所有内容都塞进上下文。只有任务确实需要时才加载，这样能把上下文窗口控制得更干净，也更适合对 token 比较敏感的模型。
|
||||
|
||||
通过 Gateway 安装 `.skill` 压缩包时，DeerFlow 会接受标准的可选 frontmatter 元数据，比如 `version`、`author`、`compatibility`，不会把本来合法的外部 skill 拒之门外。
|
||||
|
||||
Tools 也是同样的思路。DeerFlow 自带一组核心工具：网页搜索、网页抓取、文件操作、bash 执行；同时也支持通过 MCP Server 和 Python 函数扩展自定义工具。你可以替换任何一项，也可以继续往里加。
|
||||
|
||||
Gateway 生成后续建议时，现在会先把普通字符串输出和 block/list 风格的富文本内容统一归一化，再去解析 JSON 数组响应，因此不同 provider 的内容包装方式不会再悄悄把建议吞掉。
|
||||
|
||||
```text
|
||||
# sandbox 容器内的路径
|
||||
/mnt/skills/public
|
||||
├── research/SKILL.md
|
||||
├── report-generation/SKILL.md
|
||||
├── slide-creation/SKILL.md
|
||||
├── web-page/SKILL.md
|
||||
└── image-generation/SKILL.md
|
||||
|
||||
/mnt/skills/custom
|
||||
└── your-custom-skill/SKILL.md ← 你的 skill
|
||||
```
|
||||
|
||||
#### Claude Code 集成
|
||||
|
||||
借助 `claude-to-deerflow` skill，你可以直接在 [Claude Code](https://docs.anthropic.com/en/docs/claude-code) 里和正在运行的 DeerFlow 实例交互。不用离开终端，就能下发研究任务、查看状态、管理 threads。
|
||||
|
||||
**安装这个 skill：**
|
||||
|
||||
```bash
|
||||
npx skills add https://github.com/bytedance/deer-flow --skill claude-to-deerflow
|
||||
```
|
||||
|
||||
然后确认 DeerFlow 已经启动（默认地址是 `http://localhost:2026`），在 Claude Code 里使用 `/claude-to-deerflow` 命令即可。
|
||||
|
||||
**你可以做的事情包括：**
|
||||
- 给 DeerFlow 发送消息，并接收流式响应
|
||||
- 选择执行模式：flash（更快）、standard、pro（规划模式）、ultra（sub-agents 模式）
|
||||
- 检查 DeerFlow 健康状态，列出 models / skills / agents
|
||||
- 管理 threads 和会话历史
|
||||
- 上传文件做分析
|
||||
|
||||
**环境变量**（可选，用于自定义端点）：
|
||||
|
||||
```bash
|
||||
DEERFLOW_URL=http://localhost:2026 # 统一代理基地址
|
||||
DEERFLOW_GATEWAY_URL=http://localhost:2026 # Gateway API
|
||||
DEERFLOW_LANGGRAPH_URL=http://localhost:2026/api/langgraph # LangGraph API
|
||||
```
|
||||
|
||||
完整 API 说明见 [`skills/public/claude-to-deerflow/SKILL.md`](skills/public/claude-to-deerflow/SKILL.md)。
|
||||
|
||||
### Sub-Agents
|
||||
|
||||
复杂任务通常不可能一次完成，DeerFlow 会先拆解，再执行。
|
||||
|
||||
lead agent 可以按需动态拉起 sub-agents。每个 sub-agent 都有自己独立的上下文、工具和终止条件。只要条件允许，它们就会并行运行，返回结构化结果，最后再由 lead agent 汇总成一份完整输出。
|
||||
|
||||
这也是 DeerFlow 能处理从几分钟到几小时任务的原因。比如一个研究任务，可以拆成十几个 sub-agents，分别探索不同方向，最后合并成一份报告，或者一个网站，或者一套带生成视觉内容的演示文稿。一个 harness，多路并行。
|
||||
|
||||
### Sandbox 与文件系统
|
||||
|
||||
DeerFlow 不只是“会说它能做”，它是真的有一台自己的“电脑”。
|
||||
|
||||
每个任务都运行在隔离的 Docker 容器里，里面有完整的文件系统，包括 skills、workspace、uploads、outputs。agent 可以读写和编辑文件，可以执行 bash 命令和代码，也可以查看图片。整个过程都在 sandbox 内完成，可审计、会隔离，不会在不同 session 之间互相污染。
|
||||
|
||||
这就是“带工具的聊天机器人”和“真正有执行环境的 agent”之间的差别。
|
||||
|
||||
```text
|
||||
# sandbox 容器内的路径
|
||||
/mnt/user-data/
|
||||
├── uploads/ ← 你的文件
|
||||
├── workspace/ ← agents 的工作目录
|
||||
└── outputs/ ← 最终交付物
|
||||
```
|
||||
|
||||
### Context Engineering
|
||||
|
||||
**隔离的 Sub-Agent Context**：每个 sub-agent 都在自己独立的上下文里运行。它看不到主 agent 的上下文，也看不到其他 sub-agents 的上下文。这样做的目的很直接，就是让它只聚焦当前任务，不被无关信息干扰。
|
||||
|
||||
**摘要压缩**：在单个 session 内，DeerFlow 会比较积极地管理上下文，包括总结已完成的子任务、把中间结果转存到文件系统、压缩暂时不重要的信息。这样在长链路、多步骤任务里，它也能保持聚焦，而不会轻易把上下文窗口打爆。
|
||||
|
||||
### 长期记忆
|
||||
|
||||
大多数 agents 会在对话结束后把一切都忘掉，DeerFlow 不一样。
|
||||
|
||||
跨 session 使用时，DeerFlow 会逐步积累关于你的持久 memory，包括你的个人偏好、知识背景，以及长期沉淀下来的工作习惯。你用得越多，它越了解你的写作风格、技术栈和重复出现的工作流。memory 保存在本地，控制权也始终在你手里。
|
||||
|
||||
## 推荐模型
|
||||
|
||||
DeerFlow 对模型没有强绑定，只要实现了 OpenAI 兼容 API 的 LLM，理论上都可以接入。不过在下面这些能力上表现更强的模型，通常会更适合 DeerFlow：
|
||||
|
||||
- **长上下文窗口**（100k+ tokens），适合深度研究和多步骤任务
|
||||
- **推理能力**，适合自适应规划和复杂拆解
|
||||
- **多模态输入**，适合理解图片和视频
|
||||
- **稳定的 tool use 能力**，适合可靠的函数调用和结构化输出
|
||||
|
||||
## 内嵌 Python Client
|
||||
|
||||
DeerFlow 也可以作为内嵌的 Python 库使用，不必启动完整的 HTTP 服务。`DeerFlowClient` 提供了进程内的直接访问方式，覆盖所有 agent 和 Gateway 能力，返回的数据结构与 HTTP Gateway API 保持一致：
|
||||
|
||||
```python
|
||||
from deerflow.client import DeerFlowClient
|
||||
|
||||
client = DeerFlowClient()
|
||||
|
||||
# Chat
|
||||
response = client.chat("Analyze this paper for me", thread_id="my-thread")
|
||||
|
||||
# Streaming（LangGraph SSE 协议：values、messages-tuple、end）
|
||||
for event in client.stream("hello"):
|
||||
if event.type == "messages-tuple" and event.data.get("type") == "ai":
|
||||
print(event.data["content"])
|
||||
|
||||
# 配置与管理：返回值与 Gateway 对齐的 dict
|
||||
models = client.list_models() # {"models": [...]}
|
||||
skills = client.list_skills() # {"skills": [...]}
|
||||
client.update_skill("web-search", enabled=True)
|
||||
client.upload_files("thread-1", ["./report.pdf"]) # {"success": True, "files": [...]}
|
||||
```
|
||||
|
||||
所有返回 dict 的方法都会在 CI 中通过 Gateway 的 Pydantic 响应模型校验（`TestGatewayConformance`），以确保内嵌 client 始终和 HTTP API schema 保持同步。完整 API 说明见 `backend/packages/harness/deerflow/client.py`。
|
||||
|
||||
## 文档
|
||||
|
||||
- [贡献指南](CONTRIBUTING.md) - 开发环境搭建与协作流程
|
||||
- [配置指南](backend/docs/CONFIGURATION.md) - 安装与配置说明
|
||||
- [架构概览](backend/CLAUDE.md) - 技术架构说明
|
||||
- [后端架构](backend/README.md) - 后端架构与 API 参考
|
||||
|
||||
## ⚠️ 安全使用
|
||||
|
||||
### 不恰当的部署可能导致安全风险
|
||||
|
||||
DeerFlow 具备**系统指令执行、资源操作、业务逻辑调用**等关键高权限能力，默认设计为**部署在本地可信环境（仅本机 127.0.0.1 回环访问）**。若您将 agent 部署至不可信局域网、公网云服务器等可被多终端访问的网络环境，且未采取严格的安全防护措施，可能导致安全风险，例如：
|
||||
|
||||
- **未授权的非法调用**：agent 功能被未授权的第三方、公网恶意扫描程序探测到，进而发起批量非法调用请求，执行系统命令、文件读写等高危操作，可能导致安全后果。
|
||||
- **合规与法律风险**：若 agent 被非法调用用于实施网络攻击、信息窃取等违法违规行为，可能产生法律责任与合规风险。
|
||||
|
||||
### 安全使用建议
|
||||
|
||||
**注意：建议您将 DeerFlow 部署在本地可信的网络环境下。** 若您有跨设备、跨网络的部署需求，必须加入严格的安全措施。例如，采取如下手段：
|
||||
|
||||
- **设置访问 IP 白名单**：使用 `iptables`，或部署硬件防火墙 / 带访问控制（ACL）功能的交换机等，**配置规则设置 IP 白名单**，拒绝其他所有 IP 进行访问。
|
||||
- **前置身份验证**：配置反向代理（nginx 等），并**开启高强度的前置身份验证功能**，禁止无任何身份验证的访问。
|
||||
- **网络隔离**：若有可能，建议将 agent 和可信设备划分到**同一个专用 VLAN**，与其他网络设备做隔离。
|
||||
- **持续关注项目更新**：请持续关注 DeerFlow 项目的安全功能更新。
|
||||
|
||||
## 参与贡献
|
||||
|
||||
欢迎参与贡献。开发环境、工作流和相关规范见 [CONTRIBUTING.md](CONTRIBUTING.md)。
|
||||
|
||||
目前回归测试已经覆盖 Docker sandbox 模式识别，以及 `backend/tests/` 中 provisioner kubeconfig-path 处理相关测试。
|
||||
|
||||
## 许可证
|
||||
|
||||
本项目采用 [MIT License](./LICENSE) 开源发布。
|
||||
|
||||
## 致谢
|
||||
|
||||
DeerFlow 建立在开源社区大量优秀工作的基础上。所有让 DeerFlow 成为可能的项目和贡献者，我们都心怀感谢。毫不夸张地说，我们是站在巨人的肩膀上继续往前走。
|
||||
|
||||
特别感谢以下项目带来的关键支持：
|
||||
|
||||
- **[LangChain](https://github.com/langchain-ai/langchain)**：它们提供的优秀框架支撑了我们的 LLM 交互与 chains，让整体集成和能力编排顺畅可用。
|
||||
- **[LangGraph](https://github.com/langchain-ai/langgraph)**：它们在多 agent 编排上的创新方式，是 DeerFlow 复杂工作流得以成立的重要基础。
|
||||
|
||||
这些项目体现了开源协作真正的力量，我们也很高兴能继续建立在这些基础之上。
|
||||
|
||||
### 核心贡献者
|
||||
|
||||
感谢 `DeerFlow` 的核心作者，是他们的判断、投入和持续推进，才让这个项目真正落地：
|
||||
|
||||
- **[Daniel Walnut](https://github.com/hetaoBackend/)**
|
||||
- **[Henry Li](https://github.com/magiccube/)**
|
||||
|
||||
## Star History
|
||||
|
||||
[](https://star-history.com/#bytedance/deer-flow&Date)
|
||||
+2
-2
@@ -2,8 +2,8 @@
|
||||
|
||||
## Supported Versions
|
||||
|
||||
As deer-flow doesn't provide an official release yet, please use the latest version for the security updates.
|
||||
Currently, we have two branches to maintain:
|
||||
As deer-flow doesn't provide an offical release yet, please use the latest version for the security updates.
|
||||
Current we have two branches to maintain:
|
||||
* main branch for deer-flow 2.x
|
||||
* main-1.x branch for deer-flow 1.x
|
||||
|
||||
|
||||
+1
-1
@@ -1,2 +1,2 @@
|
||||
For the backend architecture and design patterns:
|
||||
For the backend architeture and design patterns:
|
||||
@./CLAUDE.md
|
||||
+71
-198
@@ -8,15 +8,11 @@ DeerFlow is a LangGraph-based AI super agent system with a full-stack architectu
|
||||
|
||||
**Architecture**:
|
||||
- **LangGraph Server** (port 2024): Agent runtime and workflow execution
|
||||
- **Gateway API** (port 8001): REST API for models, MCP, skills, memory, artifacts, uploads, and local thread cleanup
|
||||
- **Gateway API** (port 8001): REST API for models, MCP, skills, memory, artifacts, and uploads
|
||||
- **Frontend** (port 3000): Next.js web interface
|
||||
- **Nginx** (port 2026): Unified reverse proxy entry point
|
||||
- **Provisioner** (port 8002, optional in Docker dev): Started only when sandbox is configured for provisioner/Kubernetes mode
|
||||
|
||||
**Runtime Modes**:
|
||||
- **Standard mode** (`make dev`): LangGraph Server handles agent execution as a separate process. 4 processes total.
|
||||
- **Gateway mode** (`make dev-pro`, experimental): Agent runtime embedded in Gateway via `RunManager` + `run_agent()` + `StreamBridge` (`packages/harness/deerflow/runtime/`). Service manages its own concurrency via async tasks. 3 processes total, no LangGraph Server.
|
||||
|
||||
**Project Structure**:
|
||||
```
|
||||
deer-flow/
|
||||
@@ -26,38 +22,33 @@ deer-flow/
|
||||
├── backend/ # Backend application (this directory)
|
||||
│ ├── Makefile # Backend-only commands (dev, gateway, lint)
|
||||
│ ├── langgraph.json # LangGraph server configuration
|
||||
│ ├── packages/
|
||||
│ │ └── harness/ # deerflow-harness package (import: deerflow.*)
|
||||
│ │ ├── pyproject.toml
|
||||
│ │ └── deerflow/
|
||||
│ │ ├── agents/ # LangGraph agent system
|
||||
│ │ │ ├── lead_agent/ # Main agent (factory + system prompt)
|
||||
│ │ │ ├── middlewares/ # 10 middleware components
|
||||
│ │ │ ├── memory/ # Memory extraction, queue, prompts
|
||||
│ │ │ └── thread_state.py # ThreadState schema
|
||||
│ │ ├── sandbox/ # Sandbox execution system
|
||||
│ │ │ ├── local/ # Local filesystem provider
|
||||
│ │ │ ├── sandbox.py # Abstract Sandbox interface
|
||||
│ │ │ ├── tools.py # bash, ls, read/write/str_replace
|
||||
│ │ │ └── middleware.py # Sandbox lifecycle management
|
||||
│ │ ├── subagents/ # Subagent delegation system
|
||||
│ │ │ ├── builtins/ # general-purpose, bash agents
|
||||
│ │ │ ├── executor.py # Background execution engine
|
||||
│ │ │ └── registry.py # Agent registry
|
||||
│ │ ├── tools/builtins/ # Built-in tools (present_files, ask_clarification, view_image)
|
||||
│ │ ├── mcp/ # MCP integration (tools, cache, client)
|
||||
│ │ ├── models/ # Model factory with thinking/vision support
|
||||
│ │ ├── skills/ # Skills discovery, loading, parsing
|
||||
│ │ ├── config/ # Configuration system (app, model, sandbox, tool, etc.)
|
||||
│ │ ├── community/ # Community tools (tavily, jina_ai, firecrawl, image_search, aio_sandbox)
|
||||
│ │ ├── reflection/ # Dynamic module loading (resolve_variable, resolve_class)
|
||||
│ │ ├── utils/ # Utilities (network, readability)
|
||||
│ │ └── client.py # Embedded Python client (DeerFlowClient)
|
||||
│ ├── app/ # Application layer (import: app.*)
|
||||
│ ├── src/
|
||||
│ │ ├── agents/ # LangGraph agent system
|
||||
│ │ │ ├── lead_agent/ # Main agent (factory + system prompt)
|
||||
│ │ │ ├── middlewares/ # 10 middleware components
|
||||
│ │ │ ├── memory/ # Memory extraction, queue, prompts
|
||||
│ │ │ └── thread_state.py # ThreadState schema
|
||||
│ │ ├── gateway/ # FastAPI Gateway API
|
||||
│ │ │ ├── app.py # FastAPI application
|
||||
│ │ │ └── routers/ # FastAPI route modules (models, mcp, memory, skills, uploads, threads, artifacts, agents, suggestions, channels)
|
||||
│ │ └── channels/ # IM platform integrations
|
||||
│ │ │ └── routers/ # 6 route modules
|
||||
│ │ ├── sandbox/ # Sandbox execution system
|
||||
│ │ │ ├── local/ # Local filesystem provider
|
||||
│ │ │ ├── sandbox.py # Abstract Sandbox interface
|
||||
│ │ │ ├── tools.py # bash, ls, read/write/str_replace
|
||||
│ │ │ └── middleware.py # Sandbox lifecycle management
|
||||
│ │ ├── subagents/ # Subagent delegation system
|
||||
│ │ │ ├── builtins/ # general-purpose, bash agents
|
||||
│ │ │ ├── executor.py # Background execution engine
|
||||
│ │ │ └── registry.py # Agent registry
|
||||
│ │ ├── tools/builtins/ # Built-in tools (present_files, ask_clarification, view_image)
|
||||
│ │ ├── mcp/ # MCP integration (tools, cache, client)
|
||||
│ │ ├── models/ # Model factory with thinking/vision support
|
||||
│ │ ├── skills/ # Skills discovery, loading, parsing
|
||||
│ │ ├── config/ # Configuration system (app, model, sandbox, tool, etc.)
|
||||
│ │ ├── community/ # Community tools (tavily, jina_ai, firecrawl, image_search, aio_sandbox)
|
||||
│ │ ├── reflection/ # Dynamic module loading (resolve_variable, resolve_class)
|
||||
│ │ ├── utils/ # Utilities (network, readability)
|
||||
│ │ └── client.py # Embedded Python client (DeerFlowClient)
|
||||
│ ├── tests/ # Test suite
|
||||
│ └── docs/ # Documentation
|
||||
├── frontend/ # Next.js frontend application
|
||||
@@ -83,9 +74,7 @@ When making code changes, you MUST update the relevant documentation:
|
||||
```bash
|
||||
make check # Check system requirements
|
||||
make install # Install all dependencies (frontend + backend)
|
||||
make dev # Start all services (LangGraph + Gateway + Frontend + Nginx), with config.yaml preflight
|
||||
make dev-pro # Gateway mode (experimental): skip LangGraph, agent runtime embedded in Gateway
|
||||
make start-pro # Production + Gateway mode (experimental)
|
||||
make dev # Start all services (LangGraph + Gateway + Frontend + Nginx)
|
||||
make stop # Stop all services
|
||||
```
|
||||
|
||||
@@ -103,48 +92,19 @@ Regression tests related to Docker/provisioner behavior:
|
||||
- `tests/test_docker_sandbox_mode_detection.py` (mode detection from `config.yaml`)
|
||||
- `tests/test_provisioner_kubeconfig.py` (kubeconfig file/directory handling)
|
||||
|
||||
Boundary check (harness → app import firewall):
|
||||
- `tests/test_harness_boundary.py` — ensures `packages/harness/deerflow/` never imports from `app.*`
|
||||
|
||||
CI runs these regression tests for every pull request via [.github/workflows/backend-unit-tests.yml](../.github/workflows/backend-unit-tests.yml).
|
||||
|
||||
## Architecture
|
||||
|
||||
### Harness / App Split
|
||||
|
||||
The backend is split into two layers with a strict dependency direction:
|
||||
|
||||
- **Harness** (`packages/harness/deerflow/`): Publishable agent framework package (`deerflow-harness`). Import prefix: `deerflow.*`. Contains agent orchestration, tools, sandbox, models, MCP, skills, config — everything needed to build and run agents.
|
||||
- **App** (`app/`): Unpublished application code. Import prefix: `app.*`. Contains the FastAPI Gateway API and IM channel integrations (Feishu, Slack, Telegram).
|
||||
|
||||
**Dependency rule**: App imports deerflow, but deerflow never imports app. This boundary is enforced by `tests/test_harness_boundary.py` which runs in CI.
|
||||
|
||||
**Import conventions**:
|
||||
```python
|
||||
# Harness internal
|
||||
from deerflow.agents import make_lead_agent
|
||||
from deerflow.models import create_chat_model
|
||||
|
||||
# App internal
|
||||
from app.gateway.app import app
|
||||
from app.channels.service import start_channel_service
|
||||
|
||||
# App → Harness (allowed)
|
||||
from deerflow.config import get_app_config
|
||||
|
||||
# Harness → App (FORBIDDEN — enforced by test_harness_boundary.py)
|
||||
# from app.gateway.routers.uploads import ... # ← will fail CI
|
||||
```
|
||||
|
||||
### Agent System
|
||||
|
||||
**Lead Agent** (`packages/harness/deerflow/agents/lead_agent/agent.py`):
|
||||
**Lead Agent** (`src/agents/lead_agent/agent.py`):
|
||||
- Entry point: `make_lead_agent(config: RunnableConfig)` registered in `langgraph.json`
|
||||
- Dynamic model selection via `create_chat_model()` with thinking/vision support
|
||||
- Tools loaded via `get_available_tools()` - combines sandbox, built-in, MCP, community, and subagent tools
|
||||
- System prompt generated by `apply_prompt_template()` with skills, memory, and subagent instructions
|
||||
|
||||
**ThreadState** (`packages/harness/deerflow/agents/thread_state.py`):
|
||||
**ThreadState** (`src/agents/thread_state.py`):
|
||||
- Extends `AgentState` with: `sandbox`, `thread_data`, `title`, `artifacts`, `todos`, `uploaded_files`, `viewed_images`
|
||||
- Uses custom reducers: `merge_artifacts` (deduplicate), `merge_viewed_images` (merge/clear)
|
||||
|
||||
@@ -156,20 +116,19 @@ from deerflow.config import get_app_config
|
||||
|
||||
### Middleware Chain
|
||||
|
||||
Middlewares execute in strict order in `packages/harness/deerflow/agents/lead_agent/agent.py`:
|
||||
Middlewares execute in strict order in `src/agents/lead_agent/agent.py`:
|
||||
|
||||
1. **ThreadDataMiddleware** - Creates per-thread directories under the user's isolation scope (`backend/.deer-flow/users/{user_id}/threads/{thread_id}/user-data/{workspace,uploads,outputs}`); resolves `user_id` via `get_effective_user_id()` (falls back to `"default"` in no-auth mode); Web UI thread deletion now follows LangGraph thread removal with Gateway cleanup of the local thread directory
|
||||
1. **ThreadDataMiddleware** - Creates per-thread directories (`backend/.deer-flow/threads/{thread_id}/user-data/{workspace,uploads,outputs}`)
|
||||
2. **UploadsMiddleware** - Tracks and injects newly uploaded files into conversation
|
||||
3. **SandboxMiddleware** - Acquires sandbox, stores `sandbox_id` in state
|
||||
4. **DanglingToolCallMiddleware** - Injects placeholder ToolMessages for AIMessage tool_calls that lack responses (e.g., due to user interruption)
|
||||
5. **GuardrailMiddleware** - Pre-tool-call authorization via pluggable `GuardrailProvider` protocol (optional, if `guardrails.enabled` in config). Evaluates each tool call and returns error ToolMessage on deny. Three provider options: built-in `AllowlistProvider` (zero deps), OAP policy providers (e.g. `aport-agent-guardrails`), or custom providers. See [docs/GUARDRAILS.md](docs/GUARDRAILS.md) for setup, usage, and how to implement a provider.
|
||||
6. **SummarizationMiddleware** - Context reduction when approaching token limits (optional, if enabled)
|
||||
7. **TodoListMiddleware** - Task tracking with `write_todos` tool (optional, if plan_mode)
|
||||
8. **TitleMiddleware** - Auto-generates thread title after first complete exchange and normalizes structured message content before prompting the title model
|
||||
9. **MemoryMiddleware** - Queues conversations for async memory update (filters to user + final AI responses)
|
||||
10. **ViewImageMiddleware** - Injects base64 image data before LLM call (conditional on vision support)
|
||||
11. **SubagentLimitMiddleware** - Truncates excess `task` tool calls from model response to enforce `MAX_CONCURRENT_SUBAGENTS` limit (optional, if subagent_enabled)
|
||||
12. **ClarificationMiddleware** - Intercepts `ask_clarification` tool calls, interrupts via `Command(goto=END)` (must be last)
|
||||
5. **SummarizationMiddleware** - Context reduction when approaching token limits (optional, if enabled)
|
||||
6. **TodoListMiddleware** - Task tracking with `write_todos` tool (optional, if plan_mode)
|
||||
7. **TitleMiddleware** - Auto-generates thread title after first complete exchange
|
||||
8. **MemoryMiddleware** - Queues conversations for async memory update (filters to user + final AI responses)
|
||||
9. **ViewImageMiddleware** - Injects base64 image data before LLM call (conditional on vision support)
|
||||
10. **SubagentLimitMiddleware** - Truncates excess `task` tool calls from model response to enforce `MAX_CONCURRENT_SUBAGENTS` limit (optional, if subagent_enabled)
|
||||
11. **ClarificationMiddleware** - Intercepts `ask_clarification` tool calls, interrupts via `Command(goto=END)` (must be last)
|
||||
|
||||
### Configuration System
|
||||
|
||||
@@ -177,10 +136,6 @@ Middlewares execute in strict order in `packages/harness/deerflow/agents/lead_ag
|
||||
|
||||
Setup: Copy `config.example.yaml` to `config.yaml` in the **project root** directory.
|
||||
|
||||
**Config Versioning**: `config.example.yaml` has a `config_version` field. On startup, `AppConfig.from_file()` compares user version vs example version and emits a warning if outdated. Missing `config_version` = version 0. Run `make config-upgrade` to auto-merge missing fields. When changing the config schema, bump `config_version` in `config.example.yaml`.
|
||||
|
||||
**Config Caching**: `get_app_config()` caches the parsed config, but automatically reloads it when the resolved config path changes or the file's mtime increases. This keeps Gateway and LangGraph reads aligned with `config.yaml` edits without requiring a manual process restart.
|
||||
|
||||
Configuration priority:
|
||||
1. Explicit `config_path` argument
|
||||
2. `DEER_FLOW_CONFIG_PATH` environment variable
|
||||
@@ -188,7 +143,6 @@ Configuration priority:
|
||||
4. `config.yaml` in parent directory (project root - **recommended location**)
|
||||
|
||||
Config values starting with `$` are resolved as environment variables (e.g., `$OPENAI_API_KEY`).
|
||||
`ModelConfig` also declares `use_responses_api` and `output_version` so OpenAI `/v1/responses` can be enabled explicitly while still using `langchain_openai:ChatOpenAI`.
|
||||
|
||||
**Extensions Configuration** (`extensions_config.json`):
|
||||
|
||||
@@ -200,7 +154,7 @@ Configuration priority:
|
||||
3. `extensions_config.json` in current directory (backend/)
|
||||
4. `extensions_config.json` in parent directory (project root - **recommended location**)
|
||||
|
||||
### Gateway API (`app/gateway/`)
|
||||
### Gateway API (`src/gateway/`)
|
||||
|
||||
FastAPI application on port 8001 with health check at `GET /health`.
|
||||
|
||||
@@ -210,40 +164,35 @@ FastAPI application on port 8001 with health check at `GET /health`.
|
||||
|--------|-----------|
|
||||
| **Models** (`/api/models`) | `GET /` - list models; `GET /{name}` - model details |
|
||||
| **MCP** (`/api/mcp`) | `GET /config` - get config; `PUT /config` - update config (saves to extensions_config.json) |
|
||||
| **Skills** (`/api/skills`) | `GET /` - list skills; `GET /{name}` - details; `PUT /{name}` - update enabled; `POST /install` - install from .skill archive (accepts standard optional frontmatter like `version`, `author`, `compatibility`) |
|
||||
| **Skills** (`/api/skills`) | `GET /` - list skills; `GET /{name}` - details; `PUT /{name}` - update enabled; `POST /install` - install from .skill archive |
|
||||
| **Memory** (`/api/memory`) | `GET /` - memory data; `POST /reload` - force reload; `GET /config` - config; `GET /status` - config + data |
|
||||
| **Uploads** (`/api/threads/{id}/uploads`) | `POST /` - upload files (auto-converts PDF/PPT/Excel/Word); `GET /list` - list; `DELETE /{filename}` - delete |
|
||||
| **Threads** (`/api/threads/{id}`) | `DELETE /` - remove DeerFlow-managed local thread data after LangGraph thread deletion; unexpected failures are logged server-side and return a generic 500 detail |
|
||||
| **Artifacts** (`/api/threads/{id}/artifacts`) | `GET /{path}` - serve artifacts; active content types (`text/html`, `application/xhtml+xml`, `image/svg+xml`) are always forced as download attachments to reduce XSS risk; `?download=true` still forces download for other file types |
|
||||
| **Suggestions** (`/api/threads/{id}/suggestions`) | `POST /` - generate follow-up questions; rich list/block model content is normalized before JSON parsing |
|
||||
| **Thread Runs** (`/api/threads/{id}/runs`) | `POST /` - create background run; `POST /stream` - create + SSE stream; `POST /wait` - create + block; `GET /` - list runs; `GET /{rid}` - run details; `POST /{rid}/cancel` - cancel; `GET /{rid}/join` - join SSE; `GET /{rid}/messages` - paginated messages `{data, has_more}`; `GET /{rid}/events` - full event stream; `GET /../messages` - thread messages with feedback; `GET /../token-usage` - aggregate tokens |
|
||||
| **Feedback** (`/api/threads/{id}/runs/{rid}/feedback`) | `PUT /` - upsert feedback; `DELETE /` - delete user feedback; `POST /` - create feedback; `GET /` - list feedback; `GET /stats` - aggregate stats; `DELETE /{fid}` - delete specific |
|
||||
| **Runs** (`/api/runs`) | `POST /stream` - stateless run + SSE; `POST /wait` - stateless run + block; `GET /{rid}/messages` - paginated messages by run_id `{data, has_more}` (cursor: `after_seq`/`before_seq`); `GET /{rid}/feedback` - list feedback by run_id |
|
||||
| **Artifacts** (`/api/threads/{id}/artifacts`) | `GET /{path}` - serve artifacts; `?download=true` for file download |
|
||||
|
||||
Proxied through nginx: `/api/langgraph/*` → LangGraph, all other `/api/*` → Gateway.
|
||||
|
||||
### Sandbox System (`packages/harness/deerflow/sandbox/`)
|
||||
### Sandbox System (`src/sandbox/`)
|
||||
|
||||
**Interface**: Abstract `Sandbox` with `execute_command`, `read_file`, `write_file`, `list_dir`
|
||||
**Provider Pattern**: `SandboxProvider` with `acquire`, `get`, `release` lifecycle
|
||||
**Implementations**:
|
||||
- `LocalSandboxProvider` - Singleton local filesystem execution with path mappings
|
||||
- `AioSandboxProvider` (`packages/harness/deerflow/community/`) - Docker-based isolation
|
||||
- `AioSandboxProvider` (`src/community/`) - Docker-based isolation
|
||||
|
||||
**Virtual Path System**:
|
||||
- Agent sees: `/mnt/user-data/{workspace,uploads,outputs}`, `/mnt/skills`
|
||||
- Physical: `backend/.deer-flow/users/{user_id}/threads/{thread_id}/user-data/...`, `deer-flow/skills/`
|
||||
- Physical: `backend/.deer-flow/threads/{thread_id}/user-data/...`, `deer-flow/skills/`
|
||||
- Translation: `replace_virtual_path()` / `replace_virtual_paths_in_command()`
|
||||
- Detection: `is_local_sandbox()` checks `sandbox_id == "local"`
|
||||
|
||||
**Sandbox Tools** (in `packages/harness/deerflow/sandbox/tools.py`):
|
||||
**Sandbox Tools** (in `src/sandbox/tools.py`):
|
||||
- `bash` - Execute commands with path translation and error handling
|
||||
- `ls` - Directory listing (tree format, max 2 levels)
|
||||
- `read_file` - Read file contents with optional line range
|
||||
- `write_file` - Write/append to files, creates directories
|
||||
- `str_replace` - Substring replacement (single or all occurrences); same-path serialization is scoped to `(sandbox.id, path)` so isolated sandboxes do not contend on identical virtual paths inside one process
|
||||
- `str_replace` - Substring replacement (single or all occurrences)
|
||||
|
||||
### Subagent System (`packages/harness/deerflow/subagents/`)
|
||||
### Subagent System (`src/subagents/`)
|
||||
|
||||
**Built-in Agents**: `general-purpose` (all tools except `task`) and `bash` (command specialist)
|
||||
**Execution**: Dual thread pool - `_scheduler_pool` (3 workers) + `_execution_pool` (3 workers)
|
||||
@@ -251,7 +200,7 @@ Proxied through nginx: `/api/langgraph/*` → LangGraph, all other `/api/*` →
|
||||
**Flow**: `task()` tool → `SubagentExecutor` → background thread → poll 5s → SSE events → result
|
||||
**Events**: `task_started`, `task_running`, `task_completed`/`task_failed`/`task_timed_out`
|
||||
|
||||
### Tool System (`packages/harness/deerflow/tools/`)
|
||||
### Tool System (`src/tools/`)
|
||||
|
||||
`get_available_tools(groups, include_mcp, model_name, subagent_enabled)` assembles:
|
||||
1. **Config-defined tools** - Resolved from `config.yaml` via `resolve_variable()`
|
||||
@@ -263,19 +212,13 @@ Proxied through nginx: `/api/langgraph/*` → LangGraph, all other `/api/*` →
|
||||
4. **Subagent tool** (if enabled):
|
||||
- `task` - Delegate to subagent (description, prompt, subagent_type, max_turns)
|
||||
|
||||
**Community tools** (`packages/harness/deerflow/community/`):
|
||||
**Community tools** (`src/community/`):
|
||||
- `tavily/` - Web search (5 results default) and web fetch (4KB limit)
|
||||
- `jina_ai/` - Web fetch via Jina reader API with readability extraction
|
||||
- `firecrawl/` - Web scraping via Firecrawl API
|
||||
|
||||
**ACP agent tools**:
|
||||
- `invoke_acp_agent` - Invokes external ACP-compatible agents from `config.yaml`
|
||||
- ACP launchers must be real ACP adapters. The standard `codex` CLI is not ACP-compatible by itself; configure a wrapper such as `npx -y @zed-industries/codex-acp` or an installed `codex-acp` binary
|
||||
- Missing ACP executables now return an actionable error message instead of a raw `[Errno 2]`
|
||||
- Each ACP agent uses a per-thread workspace at `{base_dir}/users/{user_id}/threads/{thread_id}/acp-workspace/`. The workspace is accessible to the lead agent via the virtual path `/mnt/acp-workspace/` (read-only). In docker sandbox mode, the directory is volume-mounted into the container at `/mnt/acp-workspace` (read-only); in local sandbox mode, path translation is handled by `tools.py`
|
||||
- `image_search/` - Image search via DuckDuckGo
|
||||
|
||||
### MCP System (`packages/harness/deerflow/mcp/`)
|
||||
### MCP System (`src/mcp/`)
|
||||
|
||||
- Uses `langchain-mcp-adapters` `MultiServerMCPClient` for multi-server management
|
||||
- **Lazy initialization**: Tools loaded on first use via `get_cached_mcp_tools()`
|
||||
@@ -284,7 +227,7 @@ Proxied through nginx: `/api/langgraph/*` → LangGraph, all other `/api/*` →
|
||||
- **OAuth (HTTP/SSE)**: Supports token endpoint flows (`client_credentials`, `refresh_token`) with automatic token refresh + Authorization header injection
|
||||
- **Runtime updates**: Gateway API saves to extensions_config.json; LangGraph detects via mtime
|
||||
|
||||
### Skills System (`packages/harness/deerflow/skills/`)
|
||||
### Skills System (`src/skills/`)
|
||||
|
||||
- **Location**: `deer-flow/skills/{public,custom}/`
|
||||
- **Format**: Directory with `SKILL.md` (YAML frontmatter: name, description, license, allowed-tools)
|
||||
@@ -292,90 +235,42 @@ Proxied through nginx: `/api/langgraph/*` → LangGraph, all other `/api/*` →
|
||||
- **Injection**: Enabled skills listed in agent system prompt with container paths
|
||||
- **Installation**: `POST /api/skills/install` extracts .skill ZIP archive to custom/ directory
|
||||
|
||||
### Model Factory (`packages/harness/deerflow/models/factory.py`)
|
||||
### Model Factory (`src/models/factory.py`)
|
||||
|
||||
- `create_chat_model(name, thinking_enabled)` instantiates LLM from config via reflection
|
||||
- Supports `thinking_enabled` flag with per-model `when_thinking_enabled` overrides
|
||||
- Supports vLLM-style thinking toggles via `when_thinking_enabled.extra_body.chat_template_kwargs.enable_thinking` for Qwen reasoning models, while normalizing legacy `thinking` configs for backward compatibility
|
||||
- Supports `supports_vision` flag for image understanding models
|
||||
- Config values starting with `$` resolved as environment variables
|
||||
- Missing provider modules surface actionable install hints from reflection resolvers (for example `uv add langchain-google-genai`)
|
||||
|
||||
### vLLM Provider (`packages/harness/deerflow/models/vllm_provider.py`)
|
||||
|
||||
- `VllmChatModel` subclasses `langchain_openai:ChatOpenAI` for vLLM 0.19.0 OpenAI-compatible endpoints
|
||||
- Preserves vLLM's non-standard assistant `reasoning` field on full responses, streaming deltas, and follow-up tool-call turns
|
||||
- Designed for configs that enable thinking through `extra_body.chat_template_kwargs.enable_thinking` on vLLM 0.19.0 Qwen reasoning models, while accepting the older `thinking` alias
|
||||
|
||||
### IM Channels System (`app/channels/`)
|
||||
|
||||
Bridges external messaging platforms (Feishu, Slack, Telegram) to the DeerFlow agent via the LangGraph Server.
|
||||
|
||||
**Architecture**: Channels communicate with the LangGraph Server through `langgraph-sdk` HTTP client (same as the frontend), ensuring threads are created and managed server-side.
|
||||
### Memory System (`src/agents/memory/`)
|
||||
|
||||
**Components**:
|
||||
- `message_bus.py` - Async pub/sub hub (`InboundMessage` → queue → dispatcher; `OutboundMessage` → callbacks → channels)
|
||||
- `store.py` - JSON-file persistence mapping `channel_name:chat_id[:topic_id]` → `thread_id` (keys are `channel:chat` for root conversations and `channel:chat:topic` for threaded conversations)
|
||||
- `manager.py` - Core dispatcher: creates threads via `client.threads.create()`, routes commands, keeps Slack/Telegram on `client.runs.wait()`, and uses `client.runs.stream(["messages-tuple", "values"])` for Feishu incremental outbound updates
|
||||
- `base.py` - Abstract `Channel` base class (start/stop/send lifecycle)
|
||||
- `service.py` - Manages lifecycle of all configured channels from `config.yaml`
|
||||
- `slack.py` / `feishu.py` / `telegram.py` - Platform-specific implementations (`feishu.py` tracks the running card `message_id` in memory and patches the same card in place)
|
||||
|
||||
**Message Flow**:
|
||||
1. External platform -> Channel impl -> `MessageBus.publish_inbound()`
|
||||
2. `ChannelManager._dispatch_loop()` consumes from queue
|
||||
3. For chat: look up/create thread on LangGraph Server
|
||||
4. Feishu chat: `runs.stream()` → accumulate AI text → publish multiple outbound updates (`is_final=False`) → publish final outbound (`is_final=True`)
|
||||
5. Slack/Telegram chat: `runs.wait()` → extract final response → publish outbound
|
||||
6. Feishu channel sends one running reply card up front, then patches the same card for each outbound update (card JSON sets `config.update_multi=true` for Feishu's patch API requirement)
|
||||
7. For commands (`/new`, `/status`, `/models`, `/memory`, `/help`): handle locally or query Gateway API
|
||||
8. Outbound → channel callbacks → platform reply
|
||||
|
||||
**Configuration** (`config.yaml` -> `channels`):
|
||||
- `langgraph_url` - LangGraph Server URL (default: `http://localhost:2024`)
|
||||
- `gateway_url` - Gateway API URL for auxiliary commands (default: `http://localhost:8001`)
|
||||
- In Docker Compose, IM channels run inside the `gateway` container, so `localhost` points back to that container. Use `http://langgraph:2024` / `http://gateway:8001`, or set `DEER_FLOW_CHANNELS_LANGGRAPH_URL` / `DEER_FLOW_CHANNELS_GATEWAY_URL`.
|
||||
- Per-channel configs: `feishu` (app_id, app_secret), `slack` (bot_token, app_token), `telegram` (bot_token)
|
||||
|
||||
### Memory System (`packages/harness/deerflow/agents/memory/`)
|
||||
|
||||
**Components**:
|
||||
- `updater.py` - LLM-based memory updates with fact extraction, whitespace-normalized fact deduplication (trims leading/trailing whitespace before comparing), and atomic file I/O
|
||||
- `queue.py` - Debounced update queue (per-thread deduplication, configurable wait time); captures `user_id` at enqueue time so it survives the `threading.Timer` boundary
|
||||
- `updater.py` - LLM-based memory updates with fact extraction and atomic file I/O
|
||||
- `queue.py` - Debounced update queue (per-thread deduplication, configurable wait time)
|
||||
- `prompt.py` - Prompt templates for memory updates
|
||||
- `storage.py` - File-based storage with per-user isolation; cache keyed by `(user_id, agent_name)` tuple
|
||||
|
||||
**Per-User Isolation**:
|
||||
- Memory is stored per-user at `{base_dir}/users/{user_id}/memory.json`
|
||||
- Per-agent per-user memory at `{base_dir}/users/{user_id}/agents/{agent_name}/memory.json`
|
||||
- `user_id` is resolved via `get_effective_user_id()` from `deerflow.runtime.user_context`
|
||||
- In no-auth mode, `user_id` defaults to `"default"` (constant `DEFAULT_USER_ID`)
|
||||
- Absolute `storage_path` in config opts out of per-user isolation
|
||||
- **Migration**: Run `PYTHONPATH=. python scripts/migrate_user_isolation.py` to move legacy `memory.json` and `threads/` into per-user layout; supports `--dry-run`
|
||||
|
||||
**Data Structure** (stored in `{base_dir}/users/{user_id}/memory.json`):
|
||||
**Data Structure** (stored in `backend/.deer-flow/memory.json`):
|
||||
- **User Context**: `workContext`, `personalContext`, `topOfMind` (1-3 sentence summaries)
|
||||
- **History**: `recentMonths`, `earlierContext`, `longTermBackground`
|
||||
- **Facts**: Discrete facts with `id`, `content`, `category` (preference/knowledge/context/behavior/goal), `confidence` (0-1), `createdAt`, `source`
|
||||
|
||||
**Workflow**:
|
||||
1. `MemoryMiddleware` filters messages (user inputs + final AI responses), captures `user_id` via `get_effective_user_id()`, and queues conversation with the captured `user_id`
|
||||
1. `MemoryMiddleware` filters messages (user inputs + final AI responses) and queues conversation
|
||||
2. Queue debounces (30s default), batches updates, deduplicates per-thread
|
||||
3. Background thread invokes LLM to extract context updates and facts, using the stored `user_id` (not the contextvar, which is unavailable on timer threads)
|
||||
4. Applies updates atomically (temp file + rename) with cache invalidation, skipping duplicate fact content before append
|
||||
3. Background thread invokes LLM to extract context updates and facts
|
||||
4. Applies updates atomically (temp file + rename) with cache invalidation
|
||||
5. Next interaction injects top 15 facts + context into `<memory>` tags in system prompt
|
||||
|
||||
Focused regression coverage for the updater lives in `backend/tests/test_memory_updater.py`.
|
||||
|
||||
**Configuration** (`config.yaml` → `memory`):
|
||||
- `enabled` / `injection_enabled` - Master switches
|
||||
- `storage_path` - Path to memory.json (absolute path opts out of per-user isolation)
|
||||
- `storage_path` - Path to memory.json
|
||||
- `debounce_seconds` - Wait time before processing (default: 30)
|
||||
- `model_name` - LLM for updates (null = default model)
|
||||
- `max_facts` / `fact_confidence_threshold` - Fact storage limits (100 / 0.7)
|
||||
- `max_injection_tokens` - Token limit for prompt injection (2000)
|
||||
|
||||
### Reflection System (`packages/harness/deerflow/reflection/`)
|
||||
### Reflection System (`src/reflection/`)
|
||||
|
||||
- `resolve_variable(path)` - Import module and return variable (e.g., `module.path:variable_name`)
|
||||
- `resolve_class(path, base_class)` - Import and validate class against base class
|
||||
@@ -384,7 +279,6 @@ Focused regression coverage for the updater lives in `backend/tests/test_memory_
|
||||
|
||||
**`config.yaml`** key sections:
|
||||
- `models[]` - LLM configs with `use` class path, `supports_thinking`, `supports_vision`, provider-specific fields
|
||||
- vLLM reasoning models should use `deerflow.models.vllm_provider:VllmChatModel`; for Qwen-style parsers prefer `when_thinking_enabled.extra_body.chat_template_kwargs.enable_thinking`, and DeerFlow will also normalize the older `thinking` alias
|
||||
- `tools[]` - Tool configs with `use` variable path and `group`
|
||||
- `tool_groups[]` - Logical groupings for tools
|
||||
- `sandbox.use` - Sandbox provider class path
|
||||
@@ -400,23 +294,21 @@ Focused regression coverage for the updater lives in `backend/tests/test_memory_
|
||||
|
||||
Both can be modified at runtime via Gateway API endpoints or `DeerFlowClient` methods.
|
||||
|
||||
### Embedded Client (`packages/harness/deerflow/client.py`)
|
||||
### Embedded Client (`src/client.py`)
|
||||
|
||||
`DeerFlowClient` provides direct in-process access to all DeerFlow capabilities without HTTP services. All return types align with the Gateway API response schemas, so consumer code works identically in HTTP and embedded modes.
|
||||
|
||||
**Architecture**: Imports the same `deerflow` modules that LangGraph Server and Gateway API use. Shares the same config files and data directories. No FastAPI dependency.
|
||||
**Architecture**: Imports the same `src/` modules that LangGraph Server and Gateway API use. Shares the same config files and data directories. No FastAPI dependency.
|
||||
|
||||
**Agent Conversation** (replaces LangGraph Server):
|
||||
- `chat(message, thread_id)` — synchronous, accumulates streaming deltas per message-id and returns the final AI text
|
||||
- `stream(message, thread_id)` — subscribes to LangGraph `stream_mode=["values", "messages", "custom"]` and yields `StreamEvent`:
|
||||
- `"values"` — full state snapshot (title, messages, artifacts); AI text already delivered via `messages` mode is **not** re-synthesized here to avoid duplicate deliveries
|
||||
- `"messages-tuple"` — per-chunk update: for AI text this is a **delta** (concat per `id` to rebuild the full message); tool calls and tool results are emitted once each
|
||||
- `"custom"` — forwarded from `StreamWriter`
|
||||
- `"end"` — stream finished (carries cumulative `usage` counted once per message id)
|
||||
- `chat(message, thread_id)` — synchronous, returns final text
|
||||
- `stream(message, thread_id)` — yields `StreamEvent` aligned with LangGraph SSE protocol:
|
||||
- `"values"` — full state snapshot (title, messages, artifacts)
|
||||
- `"messages-tuple"` — per-message update (AI text, tool calls, tool results)
|
||||
- `"end"` — stream finished
|
||||
- Agent created lazily via `create_agent()` + `_build_middlewares()`, same as `make_lead_agent`
|
||||
- Supports `checkpointer` parameter for state persistence across turns
|
||||
- `reset_agent()` forces agent recreation (e.g. after memory or skill changes)
|
||||
- See [docs/STREAMING.md](docs/STREAMING.md) for the full design: why Gateway and DeerFlowClient are parallel paths, LangGraph's `stream_mode` semantics, the per-id dedup invariants, and regression testing strategy
|
||||
|
||||
**Gateway Equivalent Methods** (replaces Gateway API):
|
||||
|
||||
@@ -429,7 +321,7 @@ Both can be modified at runtime via Gateway API endpoints or `DeerFlowClient` me
|
||||
| Uploads | `upload_files(thread_id, files)`, `list_uploads(thread_id)`, `delete_upload(thread_id, filename)` | `{"success": true, "files": [...]}`, `{"files": [...], "count": N}` |
|
||||
| Artifacts | `get_artifact(thread_id, path)` → `(bytes, mime_type)` | tuple |
|
||||
|
||||
**Key difference from Gateway**: Upload accepts local `Path` objects instead of HTTP `UploadFile`, rejects directory paths before copying, and reuses a single worker when document conversion must run inside an active event loop. Artifact returns `(bytes, mime_type)` instead of HTTP Response. The new Gateway-only thread cleanup route deletes `.deer-flow/threads/{thread_id}` after LangGraph thread deletion; there is no matching `DeerFlowClient` method yet. `update_mcp_config()` and `update_skill()` automatically invalidate the cached agent.
|
||||
**Key difference from Gateway**: Upload accepts local `Path` objects instead of HTTP `UploadFile`. Artifact returns `(bytes, mime_type)` instead of HTTP Response. `update_mcp_config()` and `update_skill()` automatically invalidate the cached agent.
|
||||
|
||||
**Tests**: `tests/test_client.py` (77 unit tests including `TestGatewayConformance`), `tests/test_client_live.py` (live integration tests, requires config.yaml)
|
||||
|
||||
@@ -445,7 +337,7 @@ Both can be modified at runtime via Gateway API endpoints or `DeerFlowClient` me
|
||||
- Run the full suite before and after your change: `make test`
|
||||
- Tests must pass before a feature is considered complete
|
||||
- For lightweight config/utility modules, prefer pure unit tests with no external dependencies
|
||||
- If a module causes circular import issues in tests, add a `sys.modules` mock in `tests/conftest.py` (see existing example for `deerflow.subagents.executor`)
|
||||
- If a module causes circular import issues in tests, add a `sys.modules` mock in `tests/conftest.py` (see existing example for `src.subagents.executor`)
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
@@ -464,25 +356,8 @@ make dev
|
||||
|
||||
This starts all services and makes the application available at `http://localhost:2026`.
|
||||
|
||||
**All startup modes:**
|
||||
|
||||
| | **Local Foreground** | **Local Daemon** | **Docker Dev** | **Docker Prod** |
|
||||
|---|---|---|---|---|
|
||||
| **Dev** | `./scripts/serve.sh --dev`<br/>`make dev` | `./scripts/serve.sh --dev --daemon`<br/>`make dev-daemon` | `./scripts/docker.sh start`<br/>`make docker-start` | — |
|
||||
| **Dev + Gateway** | `./scripts/serve.sh --dev --gateway`<br/>`make dev-pro` | `./scripts/serve.sh --dev --gateway --daemon`<br/>`make dev-daemon-pro` | `./scripts/docker.sh start --gateway`<br/>`make docker-start-pro` | — |
|
||||
| **Prod** | `./scripts/serve.sh --prod`<br/>`make start` | `./scripts/serve.sh --prod --daemon`<br/>`make start-daemon` | — | `./scripts/deploy.sh`<br/>`make up` |
|
||||
| **Prod + Gateway** | `./scripts/serve.sh --prod --gateway`<br/>`make start-pro` | `./scripts/serve.sh --prod --gateway --daemon`<br/>`make start-daemon-pro` | — | `./scripts/deploy.sh --gateway`<br/>`make up-pro` |
|
||||
|
||||
| Action | Local | Docker Dev | Docker Prod |
|
||||
|---|---|---|---|
|
||||
| **Stop** | `./scripts/serve.sh --stop`<br/>`make stop` | `./scripts/docker.sh stop`<br/>`make docker-stop` | `./scripts/deploy.sh down`<br/>`make down` |
|
||||
| **Restart** | `./scripts/serve.sh --restart [flags]` | `./scripts/docker.sh restart` | — |
|
||||
|
||||
Gateway mode embeds the agent runtime in Gateway, no LangGraph server.
|
||||
|
||||
**Nginx routing**:
|
||||
- Standard mode: `/api/langgraph/*` → LangGraph Server (2024)
|
||||
- Gateway mode: `/api/langgraph/*` → Gateway embedded runtime (8001) (via envsubst)
|
||||
- `/api/langgraph/*` → LangGraph Server (2024)
|
||||
- `/api/*` (other) → Gateway API (8001)
|
||||
- `/` (non-API) → Frontend (3000)
|
||||
|
||||
@@ -517,8 +392,6 @@ When using `make dev` from root, the frontend automatically connects through ngi
|
||||
Multi-file upload with automatic document conversion:
|
||||
- Endpoint: `POST /api/threads/{thread_id}/uploads`
|
||||
- Supports: PDF, PPT, Excel, Word documents (converted via `markitdown`)
|
||||
- Rejects directory inputs before copying so uploads stay all-or-nothing
|
||||
- Reuses one conversion worker per request when called from an active event loop
|
||||
- Files stored in thread-isolated directories
|
||||
- Agent receives uploaded file list via `UploadsMiddleware`
|
||||
|
||||
|
||||
+12
-12
@@ -227,7 +227,7 @@ Example test:
|
||||
|
||||
```python
|
||||
import pytest
|
||||
from deerflow.models.factory import create_chat_model
|
||||
from src.models.factory import create_chat_model
|
||||
|
||||
def test_create_chat_model_with_valid_name():
|
||||
"""Test that a valid model name creates a model instance."""
|
||||
@@ -269,10 +269,10 @@ Include in your PR description:
|
||||
|
||||
### Adding New Tools
|
||||
|
||||
1. Create tool in `packages/harness/deerflow/tools/builtins/` or `packages/harness/deerflow/community/`:
|
||||
1. Create tool in `src/tools/builtins/` or `src/community/`:
|
||||
|
||||
```python
|
||||
# packages/harness/deerflow/tools/builtins/my_tool.py
|
||||
# src/tools/builtins/my_tool.py
|
||||
from langchain_core.tools import tool
|
||||
|
||||
@tool
|
||||
@@ -294,15 +294,15 @@ def my_tool(param: str) -> str:
|
||||
tools:
|
||||
- name: my_tool
|
||||
group: my_group
|
||||
use: deerflow.tools.builtins.my_tool:my_tool
|
||||
use: src.tools.builtins.my_tool:my_tool
|
||||
```
|
||||
|
||||
### Adding New Middleware
|
||||
|
||||
1. Create middleware in `packages/harness/deerflow/agents/middlewares/`:
|
||||
1. Create middleware in `src/agents/middlewares/`:
|
||||
|
||||
```python
|
||||
# packages/harness/deerflow/agents/middlewares/my_middleware.py
|
||||
# src/agents/middlewares/my_middleware.py
|
||||
from langchain.agents.middleware import BaseMiddleware
|
||||
from langchain_core.runnables import RunnableConfig
|
||||
|
||||
@@ -315,7 +315,7 @@ class MyMiddleware(BaseMiddleware):
|
||||
return state
|
||||
```
|
||||
|
||||
2. Register in `packages/harness/deerflow/agents/lead_agent/agent.py`:
|
||||
2. Register in `src/agents/lead_agent/agent.py`:
|
||||
|
||||
```python
|
||||
middlewares = [
|
||||
@@ -329,10 +329,10 @@ middlewares = [
|
||||
|
||||
### Adding New API Endpoints
|
||||
|
||||
1. Create router in `app/gateway/routers/`:
|
||||
1. Create router in `src/gateway/routers/`:
|
||||
|
||||
```python
|
||||
# app/gateway/routers/my_router.py
|
||||
# src/gateway/routers/my_router.py
|
||||
from fastapi import APIRouter
|
||||
|
||||
router = APIRouter(prefix="/my-endpoint", tags=["my-endpoint"])
|
||||
@@ -348,10 +348,10 @@ async def create_item(data: dict):
|
||||
return {"created": data}
|
||||
```
|
||||
|
||||
2. Register in `app/gateway/app.py`:
|
||||
2. Register in `src/gateway/app.py`:
|
||||
|
||||
```python
|
||||
from app.gateway.routers import my_router
|
||||
from src.gateway.routers import my_router
|
||||
|
||||
app.include_router(my_router.router)
|
||||
```
|
||||
@@ -360,7 +360,7 @@ app.include_router(my_router.router)
|
||||
|
||||
When adding new configuration options:
|
||||
|
||||
1. Update `packages/harness/deerflow/config/app_config.py` with new fields
|
||||
1. Update `src/config/app_config.py` with new fields
|
||||
2. Add default values in `config.example.yaml`
|
||||
3. Document in `docs/CONFIGURATION.md`
|
||||
|
||||
|
||||
+9
-72
@@ -1,91 +1,28 @@
|
||||
# Backend Dockerfile — multi-stage build
|
||||
# Stage 1 (builder): compiles native Python extensions with build-essential
|
||||
# Stage 2 (dev): retains toolchain for dev containers (uv sync at startup)
|
||||
# Stage 3 (runtime): clean image without compiler toolchain for production
|
||||
# Backend Development Dockerfile
|
||||
FROM python:3.12-slim
|
||||
|
||||
# UV source image (override for restricted networks that cannot reach ghcr.io)
|
||||
ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.7.20
|
||||
FROM ${UV_IMAGE} AS uv-source
|
||||
|
||||
# ── Stage 1: Builder ──────────────────────────────────────────────────────────
|
||||
FROM python:3.12-slim-bookworm AS builder
|
||||
|
||||
ARG NODE_MAJOR=22
|
||||
ARG APT_MIRROR
|
||||
ARG UV_INDEX_URL
|
||||
# Optional extras to install (e.g. "postgres" for PostgreSQL support)
|
||||
# Usage: docker build --build-arg UV_EXTRAS=postgres ...
|
||||
ARG UV_EXTRAS
|
||||
|
||||
# Optionally override apt mirror for restricted networks (e.g. APT_MIRROR=mirrors.aliyun.com)
|
||||
RUN if [ -n "${APT_MIRROR}" ]; then \
|
||||
sed -i "s|deb.debian.org|${APT_MIRROR}|g" /etc/apt/sources.list.d/debian.sources 2>/dev/null || true; \
|
||||
sed -i "s|deb.debian.org|${APT_MIRROR}|g" /etc/apt/sources.list 2>/dev/null || true; \
|
||||
fi
|
||||
|
||||
# Install build tools + Node.js (build-essential needed for native Python extensions)
|
||||
# Install system dependencies
|
||||
RUN apt-get update && apt-get install -y \
|
||||
curl \
|
||||
build-essential \
|
||||
gnupg \
|
||||
ca-certificates \
|
||||
&& mkdir -p /etc/apt/keyrings \
|
||||
&& curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg \
|
||||
&& echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_${NODE_MAJOR}.x nodistro main" > /etc/apt/sources.list.d/nodesource.list \
|
||||
&& apt-get update \
|
||||
&& apt-get install -y nodejs \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install uv (source image overridable via UV_IMAGE build arg)
|
||||
COPY --from=uv-source /uv /uvx /usr/local/bin/
|
||||
# Install uv
|
||||
RUN curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
ENV PATH="/root/.local/bin:$PATH"
|
||||
|
||||
# Set working directory
|
||||
WORKDIR /app
|
||||
|
||||
# Copy backend source code
|
||||
# Copy frontend source code
|
||||
COPY backend ./backend
|
||||
|
||||
# Install dependencies with cache mount
|
||||
# When UV_EXTRAS is set (e.g. "postgres"), installs optional dependencies.
|
||||
RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
sh -c "cd backend && UV_INDEX_URL=${UV_INDEX_URL:-https://pypi.org/simple} uv sync ${UV_EXTRAS:+--extra $UV_EXTRAS}"
|
||||
|
||||
# ── Stage 2: Dev ──────────────────────────────────────────────────────────────
|
||||
# Retains compiler toolchain from builder so startup-time `uv sync` can build
|
||||
# source distributions in development containers.
|
||||
FROM builder AS dev
|
||||
|
||||
# Install Docker CLI (for DooD: allows starting sandbox containers via host Docker socket)
|
||||
COPY --from=docker:cli /usr/local/bin/docker /usr/local/bin/docker
|
||||
|
||||
EXPOSE 8001 2024
|
||||
|
||||
CMD ["sh", "-c", "cd backend && PYTHONPATH=. uv run uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001"]
|
||||
|
||||
# ── Stage 3: Runtime ──────────────────────────────────────────────────────────
|
||||
# Clean image without build-essential — reduces size (~200 MB) and attack surface.
|
||||
FROM python:3.12-slim-bookworm
|
||||
|
||||
# Copy Node.js runtime from builder (provides npx for MCP servers)
|
||||
COPY --from=builder /usr/bin/node /usr/bin/node
|
||||
COPY --from=builder /usr/lib/node_modules /usr/lib/node_modules
|
||||
RUN ln -s ../lib/node_modules/npm/bin/npm-cli.js /usr/bin/npm \
|
||||
&& ln -s ../lib/node_modules/npm/bin/npx-cli.js /usr/bin/npx
|
||||
|
||||
# Install Docker CLI (for DooD: allows starting sandbox containers via host Docker socket)
|
||||
COPY --from=docker:cli /usr/local/bin/docker /usr/local/bin/docker
|
||||
|
||||
# Install uv (source image overridable via UV_IMAGE build arg)
|
||||
COPY --from=uv-source /uv /uvx /usr/local/bin/
|
||||
|
||||
# Set working directory
|
||||
WORKDIR /app
|
||||
|
||||
# Copy backend with pre-built virtualenv from builder
|
||||
COPY --from=builder /app/backend ./backend
|
||||
sh -c "cd backend && uv sync"
|
||||
|
||||
# Expose ports (gateway: 8001, langgraph: 2024)
|
||||
EXPOSE 8001 2024
|
||||
|
||||
# Default command (can be overridden in docker-compose)
|
||||
CMD ["sh", "-c", "cd backend && PYTHONPATH=. uv run --no-sync uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001"]
|
||||
CMD ["sh", "-c", "uv run uvicorn src.gateway.app:app --host 0.0.0.0 --port 8001"]
|
||||
|
||||
+3
-4
@@ -2,17 +2,16 @@ install:
|
||||
uv sync
|
||||
|
||||
dev:
|
||||
uv run langgraph dev --no-browser --no-reload --n-jobs-per-worker 10
|
||||
uv run langgraph dev --no-browser --allow-blocking --no-reload
|
||||
|
||||
gateway:
|
||||
PYTHONPATH=. uv run uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001
|
||||
uv run uvicorn src.gateway.app:app --host 0.0.0.0 --port 8001
|
||||
|
||||
test:
|
||||
PYTHONPATH=. uv run pytest tests/unittest -v
|
||||
PYTHONPATH=. uv run pytest tests/ -v
|
||||
|
||||
lint:
|
||||
uvx ruff check .
|
||||
uvx ruff format --check .
|
||||
|
||||
format:
|
||||
uvx ruff check . --fix && uvx ruff format .
|
||||
|
||||
+4
-62
@@ -36,7 +36,7 @@ DeerFlow is a LangGraph-based AI super agent with sandbox execution, persistent
|
||||
|
||||
**Request Routing** (via Nginx):
|
||||
- `/api/langgraph/*` → LangGraph Server - agent interactions, threads, streaming
|
||||
- `/api/*` (other) → Gateway API - models, MCP, skills, memory, artifacts, uploads, thread-local cleanup
|
||||
- `/api/*` (other) → Gateway API - models, MCP, skills, memory, artifacts, uploads
|
||||
- `/` (non-API) → Frontend - Next.js web interface
|
||||
|
||||
---
|
||||
@@ -78,14 +78,13 @@ Per-thread isolated execution with virtual path translation:
|
||||
- **Virtual paths**: `/mnt/user-data/{workspace,uploads,outputs}` → thread-specific physical directories
|
||||
- **Skills path**: `/mnt/skills` → `deer-flow/skills/` directory
|
||||
- **Skills loading**: Recursively discovers nested `SKILL.md` files under `skills/{public,custom}` and preserves nested container paths
|
||||
- **File-write safety**: `str_replace` serializes read-modify-write per `(sandbox.id, path)` so isolated sandboxes keep concurrency even when virtual paths match
|
||||
- **Tools**: `bash`, `ls`, `read_file`, `write_file`, `str_replace` (`bash` is disabled by default when using `LocalSandboxProvider`; use `AioSandboxProvider` for isolated shell access)
|
||||
- **Tools**: `bash`, `ls`, `read_file`, `write_file`, `str_replace`
|
||||
|
||||
### Subagent System
|
||||
|
||||
Async task delegation with concurrent execution:
|
||||
|
||||
- **Built-in agents**: `general-purpose` (full toolset) and `bash` (command specialist, exposed only when shell access is available)
|
||||
- **Built-in agents**: `general-purpose` (full toolset) and `bash` (command specialist)
|
||||
- **Concurrency**: Max 3 subagents per turn, 15-minute timeout
|
||||
- **Execution**: Background thread pools with status tracking and SSE events
|
||||
- **Flow**: Agent calls `task()` tool → executor runs subagent in background → polls for completion → returns result
|
||||
@@ -124,17 +123,10 @@ FastAPI application providing REST endpoints for frontend integration:
|
||||
| `POST /api/memory/reload` | Force memory reload |
|
||||
| `GET /api/memory/config` | Memory configuration |
|
||||
| `GET /api/memory/status` | Combined config + data |
|
||||
| `POST /api/threads/{id}/uploads` | Upload files (auto-converts PDF/PPT/Excel/Word to Markdown, rejects directory paths) |
|
||||
| `POST /api/threads/{id}/uploads` | Upload files (auto-converts PDF/PPT/Excel/Word to Markdown) |
|
||||
| `GET /api/threads/{id}/uploads/list` | List uploaded files |
|
||||
| `DELETE /api/threads/{id}` | Delete DeerFlow-managed local thread data after LangGraph thread deletion; unexpected failures are logged server-side and return a generic 500 detail |
|
||||
| `GET /api/threads/{id}/artifacts/{path}` | Serve generated artifacts |
|
||||
|
||||
### IM Channels
|
||||
|
||||
The IM bridge supports Feishu, Slack, and Telegram. Slack and Telegram still use the final `runs.wait()` response path, while Feishu now streams through `runs.stream(["messages-tuple", "values"])` and updates a single in-thread card in place.
|
||||
|
||||
For Feishu card updates, DeerFlow stores the running card's `message_id` per inbound message and patches that same card until the run finishes, preserving the existing `OK` / `DONE` reaction flow.
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
@@ -171,15 +163,6 @@ models:
|
||||
api_key: $OPENAI_API_KEY
|
||||
supports_thinking: false
|
||||
supports_vision: true
|
||||
|
||||
- name: gpt-5-responses
|
||||
display_name: GPT-5 (Responses API)
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: gpt-5
|
||||
api_key: $OPENAI_API_KEY
|
||||
use_responses_api: true
|
||||
output_version: responses/v1
|
||||
supports_vision: true
|
||||
```
|
||||
|
||||
Set your API keys:
|
||||
@@ -313,47 +296,6 @@ MCP servers and skill states in a single file:
|
||||
- Model API keys: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `DEEPSEEK_API_KEY`, etc.
|
||||
- Tool API keys: `TAVILY_API_KEY`, `GITHUB_TOKEN`, etc.
|
||||
|
||||
### LangSmith Tracing
|
||||
|
||||
DeerFlow has built-in [LangSmith](https://smith.langchain.com) integration for observability. When enabled, all LLM calls, agent runs, tool executions, and middleware processing are traced and visible in the LangSmith dashboard.
|
||||
|
||||
**Setup:**
|
||||
|
||||
1. Sign up at [smith.langchain.com](https://smith.langchain.com) and create a project.
|
||||
2. Add the following to your `.env` file in the project root:
|
||||
|
||||
```bash
|
||||
LANGSMITH_TRACING=true
|
||||
LANGSMITH_ENDPOINT=https://api.smith.langchain.com
|
||||
LANGSMITH_API_KEY=lsv2_pt_xxxxxxxxxxxxxxxx
|
||||
LANGSMITH_PROJECT=xxx
|
||||
```
|
||||
|
||||
**Legacy variables:** The `LANGCHAIN_TRACING_V2`, `LANGCHAIN_API_KEY`, `LANGCHAIN_PROJECT`, and `LANGCHAIN_ENDPOINT` variables are also supported for backward compatibility. `LANGSMITH_*` variables take precedence when both are set.
|
||||
|
||||
### Langfuse Tracing
|
||||
|
||||
DeerFlow also supports [Langfuse](https://langfuse.com) observability for LangChain-compatible runs.
|
||||
|
||||
Add the following to your `.env` file:
|
||||
|
||||
```bash
|
||||
LANGFUSE_TRACING=true
|
||||
LANGFUSE_PUBLIC_KEY=pk-lf-xxxxxxxxxxxxxxxx
|
||||
LANGFUSE_SECRET_KEY=sk-lf-xxxxxxxxxxxxxxxx
|
||||
LANGFUSE_BASE_URL=https://cloud.langfuse.com
|
||||
```
|
||||
|
||||
If you are using a self-hosted Langfuse deployment, set `LANGFUSE_BASE_URL` to your Langfuse host.
|
||||
|
||||
### Dual Provider Behavior
|
||||
|
||||
If both LangSmith and Langfuse are enabled, DeerFlow initializes and attaches both callbacks so the same run data is reported to both systems.
|
||||
|
||||
If a provider is explicitly enabled but required credentials are missing, or the provider callback cannot be initialized, DeerFlow raises an error when tracing is initialized during model creation instead of silently disabling tracing.
|
||||
|
||||
**Docker:** In `docker-compose.yaml`, tracing is disabled by default (`LANGSMITH_TRACING=false`). Set `LANGSMITH_TRACING=true` and/or `LANGFUSE_TRACING=true` in your `.env`, together with the required credentials, to enable tracing in containerized deployments.
|
||||
|
||||
---
|
||||
|
||||
## Development
|
||||
|
||||
@@ -1,16 +0,0 @@
|
||||
"""IM Channel integration for DeerFlow.
|
||||
|
||||
Provides a pluggable channel system that connects external messaging platforms
|
||||
(Feishu/Lark, Slack, Telegram) to the DeerFlow agent via the ChannelManager,
|
||||
which uses ``langgraph-sdk`` to communicate with the underlying LangGraph Server.
|
||||
"""
|
||||
|
||||
from app.channels.base import Channel
|
||||
from app.channels.message_bus import InboundMessage, MessageBus, OutboundMessage
|
||||
|
||||
__all__ = [
|
||||
"Channel",
|
||||
"InboundMessage",
|
||||
"MessageBus",
|
||||
"OutboundMessage",
|
||||
]
|
||||
@@ -1,126 +0,0 @@
|
||||
"""Abstract base class for IM channels."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import Any
|
||||
|
||||
from app.channels.message_bus import InboundMessage, InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class Channel(ABC):
|
||||
"""Base class for all IM channel implementations.
|
||||
|
||||
Each channel connects to an external messaging platform and:
|
||||
1. Receives messages, wraps them as InboundMessage, publishes to the bus.
|
||||
2. Subscribes to outbound messages and sends replies back to the platform.
|
||||
|
||||
Subclasses must implement ``start``, ``stop``, and ``send``.
|
||||
"""
|
||||
|
||||
def __init__(self, name: str, bus: MessageBus, config: dict[str, Any]) -> None:
|
||||
self.name = name
|
||||
self.bus = bus
|
||||
self.config = config
|
||||
self._running = False
|
||||
|
||||
@property
|
||||
def is_running(self) -> bool:
|
||||
return self._running
|
||||
|
||||
# -- lifecycle ---------------------------------------------------------
|
||||
|
||||
@abstractmethod
|
||||
async def start(self) -> None:
|
||||
"""Start listening for messages from the external platform."""
|
||||
|
||||
@abstractmethod
|
||||
async def stop(self) -> None:
|
||||
"""Gracefully stop the channel."""
|
||||
|
||||
# -- outbound ----------------------------------------------------------
|
||||
|
||||
@abstractmethod
|
||||
async def send(self, msg: OutboundMessage) -> None:
|
||||
"""Send a message back to the external platform.
|
||||
|
||||
The implementation should use ``msg.chat_id`` and ``msg.thread_ts``
|
||||
to route the reply to the correct conversation/thread.
|
||||
"""
|
||||
|
||||
async def send_file(self, msg: OutboundMessage, attachment: ResolvedAttachment) -> bool:
|
||||
"""Upload a single file attachment to the platform.
|
||||
|
||||
Returns True if the upload succeeded, False otherwise.
|
||||
Default implementation returns False (no file upload support).
|
||||
"""
|
||||
return False
|
||||
|
||||
# -- helpers -----------------------------------------------------------
|
||||
|
||||
def _make_inbound(
|
||||
self,
|
||||
chat_id: str,
|
||||
user_id: str,
|
||||
text: str,
|
||||
*,
|
||||
msg_type: InboundMessageType = InboundMessageType.CHAT,
|
||||
thread_ts: str | None = None,
|
||||
files: list[dict[str, Any]] | None = None,
|
||||
metadata: dict[str, Any] | None = None,
|
||||
) -> InboundMessage:
|
||||
"""Convenience factory for creating InboundMessage instances."""
|
||||
return InboundMessage(
|
||||
channel_name=self.name,
|
||||
chat_id=chat_id,
|
||||
user_id=user_id,
|
||||
text=text,
|
||||
msg_type=msg_type,
|
||||
thread_ts=thread_ts,
|
||||
files=files or [],
|
||||
metadata=metadata or {},
|
||||
)
|
||||
|
||||
async def _on_outbound(self, msg: OutboundMessage) -> None:
|
||||
"""Outbound callback registered with the bus.
|
||||
|
||||
Only forwards messages targeted at this channel.
|
||||
Sends the text message first, then uploads any file attachments.
|
||||
File uploads are skipped entirely when the text send fails to avoid
|
||||
partial deliveries (files without accompanying text).
|
||||
"""
|
||||
if msg.channel_name == self.name:
|
||||
try:
|
||||
await self.send(msg)
|
||||
except Exception:
|
||||
logger.exception("Failed to send outbound message on channel %s", self.name)
|
||||
return # Do not attempt file uploads when the text message failed
|
||||
|
||||
for attachment in msg.attachments:
|
||||
try:
|
||||
success = await self.send_file(msg, attachment)
|
||||
if not success:
|
||||
logger.warning("[%s] file upload skipped for %s", self.name, attachment.filename)
|
||||
except Exception:
|
||||
logger.exception("[%s] failed to upload file %s", self.name, attachment.filename)
|
||||
|
||||
async def receive_file(self, msg: InboundMessage, thread_id: str) -> InboundMessage:
|
||||
"""
|
||||
Optionally process and materialize inbound file attachments for this channel.
|
||||
|
||||
By default, this method does nothing and simply returns the original message.
|
||||
Subclasses (e.g. FeishuChannel) may override this to download files (images, documents, etc)
|
||||
referenced in msg.files, save them to the sandbox, and update msg.text to include
|
||||
the sandbox file paths for downstream model consumption.
|
||||
|
||||
Args:
|
||||
msg: The inbound message, possibly containing file metadata in msg.files.
|
||||
thread_id: The resolved DeerFlow thread ID for sandbox path context.
|
||||
|
||||
Returns:
|
||||
The (possibly modified) InboundMessage, with text and/or files updated as needed.
|
||||
"""
|
||||
return msg
|
||||
@@ -1,20 +0,0 @@
|
||||
"""Shared command definitions used by all channel implementations.
|
||||
|
||||
Keeping the authoritative command set in one place ensures that channel
|
||||
parsers (e.g. Feishu) and the ChannelManager dispatcher stay in sync
|
||||
automatically — adding or removing a command here is the single edit
|
||||
required.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
KNOWN_CHANNEL_COMMANDS: frozenset[str] = frozenset(
|
||||
{
|
||||
"/bootstrap",
|
||||
"/new",
|
||||
"/status",
|
||||
"/models",
|
||||
"/memory",
|
||||
"/help",
|
||||
}
|
||||
)
|
||||
@@ -1,716 +0,0 @@
|
||||
"""Feishu/Lark channel — connects to Feishu via WebSocket (no public IP needed)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import threading
|
||||
from typing import Any, Literal
|
||||
|
||||
from app.plugins.auth.security.actor_context import bind_user_actor_context
|
||||
from app.channels.base import Channel
|
||||
from app.channels.commands import KNOWN_CHANNEL_COMMANDS
|
||||
from app.channels.message_bus import InboundMessage, InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
|
||||
from deerflow.config.paths import VIRTUAL_PATH_PREFIX, get_paths
|
||||
from deerflow.runtime.actor_context import get_effective_user_id
|
||||
from deerflow.sandbox.sandbox_provider import get_sandbox_provider
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _is_feishu_command(text: str) -> bool:
|
||||
if not text.startswith("/"):
|
||||
return False
|
||||
return text.split(maxsplit=1)[0].lower() in KNOWN_CHANNEL_COMMANDS
|
||||
|
||||
|
||||
class FeishuChannel(Channel):
|
||||
"""Feishu/Lark IM channel using the ``lark-oapi`` WebSocket client.
|
||||
|
||||
Configuration keys (in ``config.yaml`` under ``channels.feishu``):
|
||||
- ``app_id``: Feishu app ID.
|
||||
- ``app_secret``: Feishu app secret.
|
||||
- ``verification_token``: (optional) Event verification token.
|
||||
|
||||
The channel uses WebSocket long-connection mode so no public IP is required.
|
||||
|
||||
Message flow:
|
||||
1. User sends a message → bot adds "OK" emoji reaction
|
||||
2. Bot replies in thread: "Working on it......"
|
||||
3. Agent processes the message and returns a result
|
||||
4. Bot replies in thread with the result
|
||||
5. Bot adds "DONE" emoji reaction to the original message
|
||||
"""
|
||||
|
||||
def __init__(self, bus: MessageBus, config: dict[str, Any]) -> None:
|
||||
super().__init__(name="feishu", bus=bus, config=config)
|
||||
self._thread: threading.Thread | None = None
|
||||
self._main_loop: asyncio.AbstractEventLoop | None = None
|
||||
self._api_client = None
|
||||
self._CreateMessageReactionRequest = None
|
||||
self._CreateMessageReactionRequestBody = None
|
||||
self._Emoji = None
|
||||
self._PatchMessageRequest = None
|
||||
self._PatchMessageRequestBody = None
|
||||
self._background_tasks: set[asyncio.Task] = set()
|
||||
self._running_card_ids: dict[str, str] = {}
|
||||
self._running_card_tasks: dict[str, asyncio.Task] = {}
|
||||
self._CreateFileRequest = None
|
||||
self._CreateFileRequestBody = None
|
||||
self._CreateImageRequest = None
|
||||
self._CreateImageRequestBody = None
|
||||
self._GetMessageResourceRequest = None
|
||||
self._thread_lock = threading.Lock()
|
||||
|
||||
async def start(self) -> None:
|
||||
if self._running:
|
||||
return
|
||||
|
||||
try:
|
||||
import lark_oapi as lark
|
||||
from lark_oapi.api.im.v1 import (
|
||||
CreateFileRequest,
|
||||
CreateFileRequestBody,
|
||||
CreateImageRequest,
|
||||
CreateImageRequestBody,
|
||||
CreateMessageReactionRequest,
|
||||
CreateMessageReactionRequestBody,
|
||||
CreateMessageRequest,
|
||||
CreateMessageRequestBody,
|
||||
Emoji,
|
||||
GetMessageResourceRequest,
|
||||
PatchMessageRequest,
|
||||
PatchMessageRequestBody,
|
||||
ReplyMessageRequest,
|
||||
ReplyMessageRequestBody,
|
||||
)
|
||||
except ImportError:
|
||||
logger.error("lark-oapi is not installed. Install it with: uv add lark-oapi")
|
||||
return
|
||||
|
||||
self._lark = lark
|
||||
self._CreateMessageRequest = CreateMessageRequest
|
||||
self._CreateMessageRequestBody = CreateMessageRequestBody
|
||||
self._ReplyMessageRequest = ReplyMessageRequest
|
||||
self._ReplyMessageRequestBody = ReplyMessageRequestBody
|
||||
self._CreateMessageReactionRequest = CreateMessageReactionRequest
|
||||
self._CreateMessageReactionRequestBody = CreateMessageReactionRequestBody
|
||||
self._Emoji = Emoji
|
||||
self._PatchMessageRequest = PatchMessageRequest
|
||||
self._PatchMessageRequestBody = PatchMessageRequestBody
|
||||
self._CreateFileRequest = CreateFileRequest
|
||||
self._CreateFileRequestBody = CreateFileRequestBody
|
||||
self._CreateImageRequest = CreateImageRequest
|
||||
self._CreateImageRequestBody = CreateImageRequestBody
|
||||
self._GetMessageResourceRequest = GetMessageResourceRequest
|
||||
|
||||
app_id = self.config.get("app_id", "")
|
||||
app_secret = self.config.get("app_secret", "")
|
||||
domain = self.config.get("domain", "https://open.feishu.cn")
|
||||
|
||||
if not app_id or not app_secret:
|
||||
logger.error("Feishu channel requires app_id and app_secret")
|
||||
return
|
||||
|
||||
self._api_client = lark.Client.builder().app_id(app_id).app_secret(app_secret).domain(domain).build()
|
||||
logger.info("[Feishu] using domain: %s", domain)
|
||||
self._main_loop = asyncio.get_event_loop()
|
||||
|
||||
self._running = True
|
||||
self.bus.subscribe_outbound(self._on_outbound)
|
||||
|
||||
# Both ws.Client construction and start() must happen in a dedicated
|
||||
# thread with its own event loop. lark-oapi caches the running loop
|
||||
# at construction time and later calls loop.run_until_complete(),
|
||||
# which conflicts with an already-running uvloop.
|
||||
self._thread = threading.Thread(
|
||||
target=self._run_ws,
|
||||
args=(app_id, app_secret, domain),
|
||||
daemon=True,
|
||||
)
|
||||
self._thread.start()
|
||||
logger.info("Feishu channel started")
|
||||
|
||||
def _run_ws(self, app_id: str, app_secret: str, domain: str) -> None:
|
||||
"""Construct and run the lark WS client in a thread with a fresh event loop.
|
||||
|
||||
The lark-oapi SDK captures a module-level event loop at import time
|
||||
(``lark_oapi.ws.client.loop``). When uvicorn uses uvloop, that
|
||||
captured loop is the *main* thread's uvloop — which is already
|
||||
running, so ``loop.run_until_complete()`` inside ``Client.start()``
|
||||
raises ``RuntimeError``.
|
||||
|
||||
We work around this by creating a plain asyncio event loop for this
|
||||
thread and patching the SDK's module-level reference before calling
|
||||
``start()``.
|
||||
"""
|
||||
loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(loop)
|
||||
try:
|
||||
import lark_oapi as lark
|
||||
import lark_oapi.ws.client as _ws_client_mod
|
||||
|
||||
# Replace the SDK's module-level loop so Client.start() uses
|
||||
# this thread's (non-running) event loop instead of the main
|
||||
# thread's uvloop.
|
||||
_ws_client_mod.loop = loop
|
||||
|
||||
event_handler = lark.EventDispatcherHandler.builder("", "").register_p2_im_message_receive_v1(self._on_message).build()
|
||||
ws_client = lark.ws.Client(
|
||||
app_id=app_id,
|
||||
app_secret=app_secret,
|
||||
event_handler=event_handler,
|
||||
log_level=lark.LogLevel.INFO,
|
||||
domain=domain,
|
||||
)
|
||||
ws_client.start()
|
||||
except Exception:
|
||||
if self._running:
|
||||
logger.exception("Feishu WebSocket error")
|
||||
|
||||
async def stop(self) -> None:
|
||||
self._running = False
|
||||
self.bus.unsubscribe_outbound(self._on_outbound)
|
||||
for task in list(self._background_tasks):
|
||||
task.cancel()
|
||||
self._background_tasks.clear()
|
||||
for task in list(self._running_card_tasks.values()):
|
||||
task.cancel()
|
||||
self._running_card_tasks.clear()
|
||||
if self._thread:
|
||||
self._thread.join(timeout=5)
|
||||
self._thread = None
|
||||
logger.info("Feishu channel stopped")
|
||||
|
||||
async def send(self, msg: OutboundMessage, *, _max_retries: int = 3) -> None:
|
||||
if not self._api_client:
|
||||
logger.warning("[Feishu] send called but no api_client available")
|
||||
return
|
||||
|
||||
logger.info(
|
||||
"[Feishu] sending reply: chat_id=%s, thread_ts=%s, text_len=%d",
|
||||
msg.chat_id,
|
||||
msg.thread_ts,
|
||||
len(msg.text),
|
||||
)
|
||||
|
||||
last_exc: Exception | None = None
|
||||
for attempt in range(_max_retries):
|
||||
try:
|
||||
await self._send_card_message(msg)
|
||||
return # success
|
||||
except Exception as exc:
|
||||
last_exc = exc
|
||||
if attempt < _max_retries - 1:
|
||||
delay = 2**attempt # 1s, 2s
|
||||
logger.warning(
|
||||
"[Feishu] send failed (attempt %d/%d), retrying in %ds: %s",
|
||||
attempt + 1,
|
||||
_max_retries,
|
||||
delay,
|
||||
exc,
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
|
||||
logger.error("[Feishu] send failed after %d attempts: %s", _max_retries, last_exc)
|
||||
if last_exc is None:
|
||||
raise RuntimeError("Feishu send failed without an exception from any attempt")
|
||||
raise last_exc
|
||||
|
||||
async def send_file(self, msg: OutboundMessage, attachment: ResolvedAttachment) -> bool:
|
||||
if not self._api_client:
|
||||
return False
|
||||
|
||||
# Check size limits (image: 10MB, file: 30MB)
|
||||
if attachment.is_image and attachment.size > 10 * 1024 * 1024:
|
||||
logger.warning("[Feishu] image too large (%d bytes), skipping: %s", attachment.size, attachment.filename)
|
||||
return False
|
||||
if not attachment.is_image and attachment.size > 30 * 1024 * 1024:
|
||||
logger.warning("[Feishu] file too large (%d bytes), skipping: %s", attachment.size, attachment.filename)
|
||||
return False
|
||||
|
||||
try:
|
||||
if attachment.is_image:
|
||||
file_key = await self._upload_image(attachment.actual_path)
|
||||
msg_type = "image"
|
||||
content = json.dumps({"image_key": file_key})
|
||||
else:
|
||||
file_key = await self._upload_file(attachment.actual_path, attachment.filename)
|
||||
msg_type = "file"
|
||||
content = json.dumps({"file_key": file_key})
|
||||
|
||||
if msg.thread_ts:
|
||||
request = self._ReplyMessageRequest.builder().message_id(msg.thread_ts).request_body(self._ReplyMessageRequestBody.builder().msg_type(msg_type).content(content).reply_in_thread(True).build()).build()
|
||||
await asyncio.to_thread(self._api_client.im.v1.message.reply, request)
|
||||
else:
|
||||
request = self._CreateMessageRequest.builder().receive_id_type("chat_id").request_body(self._CreateMessageRequestBody.builder().receive_id(msg.chat_id).msg_type(msg_type).content(content).build()).build()
|
||||
await asyncio.to_thread(self._api_client.im.v1.message.create, request)
|
||||
|
||||
logger.info("[Feishu] file sent: %s (type=%s)", attachment.filename, msg_type)
|
||||
return True
|
||||
except Exception:
|
||||
logger.exception("[Feishu] failed to upload/send file: %s", attachment.filename)
|
||||
return False
|
||||
|
||||
async def _upload_image(self, path) -> str:
|
||||
"""Upload an image to Feishu and return the image_key."""
|
||||
with open(str(path), "rb") as f:
|
||||
request = self._CreateImageRequest.builder().request_body(self._CreateImageRequestBody.builder().image_type("message").image(f).build()).build()
|
||||
response = await asyncio.to_thread(self._api_client.im.v1.image.create, request)
|
||||
if not response.success():
|
||||
raise RuntimeError(f"Feishu image upload failed: code={response.code}, msg={response.msg}")
|
||||
return response.data.image_key
|
||||
|
||||
async def _upload_file(self, path, filename: str) -> str:
|
||||
"""Upload a file to Feishu and return the file_key."""
|
||||
suffix = path.suffix.lower() if hasattr(path, "suffix") else ""
|
||||
if suffix in (".xls", ".xlsx", ".csv"):
|
||||
file_type = "xls"
|
||||
elif suffix in (".ppt", ".pptx"):
|
||||
file_type = "ppt"
|
||||
elif suffix == ".pdf":
|
||||
file_type = "pdf"
|
||||
elif suffix in (".doc", ".docx"):
|
||||
file_type = "doc"
|
||||
else:
|
||||
file_type = "stream"
|
||||
|
||||
with open(str(path), "rb") as f:
|
||||
request = self._CreateFileRequest.builder().request_body(self._CreateFileRequestBody.builder().file_type(file_type).file_name(filename).file(f).build()).build()
|
||||
response = await asyncio.to_thread(self._api_client.im.v1.file.create, request)
|
||||
if not response.success():
|
||||
raise RuntimeError(f"Feishu file upload failed: code={response.code}, msg={response.msg}")
|
||||
return response.data.file_key
|
||||
|
||||
async def receive_file(self, msg: InboundMessage, thread_id: str) -> InboundMessage:
|
||||
"""Download a Feishu file into the thread uploads directory.
|
||||
|
||||
Returns the sandbox virtual path when the image is persisted successfully.
|
||||
"""
|
||||
if not msg.thread_ts:
|
||||
logger.warning("[Feishu] received file message without thread_ts, cannot associate with conversation: %s", msg)
|
||||
return msg
|
||||
files = msg.files
|
||||
if not files:
|
||||
logger.warning("[Feishu] received message with no files: %s", msg)
|
||||
return msg
|
||||
text = msg.text
|
||||
for file in files:
|
||||
if file.get("image_key"):
|
||||
virtual_path = await self._receive_single_file(
|
||||
msg.thread_ts,
|
||||
file["image_key"],
|
||||
"image",
|
||||
thread_id,
|
||||
user_id=msg.user_id,
|
||||
)
|
||||
text = text.replace("[image]", virtual_path, 1)
|
||||
elif file.get("file_key"):
|
||||
virtual_path = await self._receive_single_file(
|
||||
msg.thread_ts,
|
||||
file["file_key"],
|
||||
"file",
|
||||
thread_id,
|
||||
user_id=msg.user_id,
|
||||
)
|
||||
text = text.replace("[file]", virtual_path, 1)
|
||||
msg.text = text
|
||||
return msg
|
||||
|
||||
async def _receive_single_file(
|
||||
self,
|
||||
message_id: str,
|
||||
file_key: str,
|
||||
type: Literal["image", "file"],
|
||||
thread_id: str,
|
||||
*,
|
||||
user_id: str | None = None,
|
||||
) -> str:
|
||||
request = self._GetMessageResourceRequest.builder().message_id(message_id).file_key(file_key).type(type).build()
|
||||
|
||||
def inner():
|
||||
return self._api_client.im.v1.message_resource.get(request)
|
||||
|
||||
try:
|
||||
response = await asyncio.to_thread(inner)
|
||||
except Exception:
|
||||
logger.exception("[Feishu] resource get request failed for resource_key=%s type=%s", file_key, type)
|
||||
return f"Failed to obtain the [{type}]"
|
||||
|
||||
if not response.success():
|
||||
logger.warning(
|
||||
"[Feishu] resource get failed: resource_key=%s, type=%s, code=%s, msg=%s, log_id=%s ",
|
||||
file_key,
|
||||
type,
|
||||
response.code,
|
||||
response.msg,
|
||||
response.get_log_id(),
|
||||
)
|
||||
return f"Failed to obtain the [{type}]"
|
||||
|
||||
image_stream = getattr(response, "file", None)
|
||||
if image_stream is None:
|
||||
logger.warning("[Feishu] resource get returned no file stream: resource_key=%s, type=%s", file_key, type)
|
||||
return f"Failed to obtain the [{type}]"
|
||||
|
||||
try:
|
||||
content: bytes = await asyncio.to_thread(image_stream.read)
|
||||
except Exception:
|
||||
logger.exception("[Feishu] failed to read resource stream: resource_key=%s, type=%s", file_key, type)
|
||||
return f"Failed to obtain the [{type}]"
|
||||
|
||||
if not content:
|
||||
logger.warning("[Feishu] empty resource content: resource_key=%s, type=%s", file_key, type)
|
||||
return f"Failed to obtain the [{type}]"
|
||||
|
||||
paths = get_paths()
|
||||
with bind_user_actor_context(user_id):
|
||||
effective_user_id = get_effective_user_id()
|
||||
paths.ensure_thread_dirs(thread_id, user_id=effective_user_id)
|
||||
uploads_dir = paths.sandbox_uploads_dir(thread_id, user_id=effective_user_id).resolve()
|
||||
|
||||
ext = "png" if type == "image" else "bin"
|
||||
raw_filename = getattr(response, "file_name", "") or f"feishu_{file_key[-12:]}.{ext}"
|
||||
|
||||
# Sanitize filename: preserve extension, replace path chars in name part
|
||||
if "." in raw_filename:
|
||||
name_part, ext = raw_filename.rsplit(".", 1)
|
||||
name_part = re.sub(r"[./\\]", "_", name_part)
|
||||
filename = f"{name_part}.{ext}"
|
||||
else:
|
||||
filename = re.sub(r"[./\\]", "_", raw_filename)
|
||||
resolved_target = uploads_dir / filename
|
||||
|
||||
def down_load():
|
||||
# use thread_lock to avoid filename conflicts when writing
|
||||
with self._thread_lock:
|
||||
resolved_target.write_bytes(content)
|
||||
|
||||
try:
|
||||
await asyncio.to_thread(down_load)
|
||||
except Exception:
|
||||
logger.exception("[Feishu] failed to persist downloaded resource: %s, type=%s", resolved_target, type)
|
||||
return f"Failed to obtain the [{type}]"
|
||||
|
||||
virtual_path = f"{VIRTUAL_PATH_PREFIX}/uploads/{resolved_target.name}"
|
||||
|
||||
try:
|
||||
sandbox_provider = get_sandbox_provider()
|
||||
sandbox_id = sandbox_provider.acquire(thread_id)
|
||||
if sandbox_id != "local":
|
||||
sandbox = sandbox_provider.get(sandbox_id)
|
||||
if sandbox is None:
|
||||
logger.warning("[Feishu] sandbox not found for thread_id=%s", thread_id)
|
||||
return f"Failed to obtain the [{type}]"
|
||||
sandbox.update_file(virtual_path, content)
|
||||
except Exception:
|
||||
logger.exception("[Feishu] failed to sync resource into non-local sandbox: %s", virtual_path)
|
||||
return f"Failed to obtain the [{type}]"
|
||||
|
||||
logger.info("[Feishu] downloaded resource mapped: file_key=%s -> %s", file_key, virtual_path)
|
||||
return virtual_path
|
||||
|
||||
# -- message formatting ------------------------------------------------
|
||||
|
||||
@staticmethod
|
||||
def _build_card_content(text: str) -> str:
|
||||
"""Build a Feishu interactive card with markdown content.
|
||||
|
||||
Feishu's interactive card format natively renders markdown, including
|
||||
headers, bold/italic, code blocks, lists, and links.
|
||||
"""
|
||||
card = {
|
||||
"config": {"wide_screen_mode": True, "update_multi": True},
|
||||
"elements": [{"tag": "markdown", "content": text}],
|
||||
}
|
||||
return json.dumps(card)
|
||||
|
||||
# -- reaction helpers --------------------------------------------------
|
||||
|
||||
async def _add_reaction(self, message_id: str, emoji_type: str = "THUMBSUP") -> None:
|
||||
"""Add an emoji reaction to a message."""
|
||||
if not self._api_client or not self._CreateMessageReactionRequest:
|
||||
return
|
||||
try:
|
||||
request = self._CreateMessageReactionRequest.builder().message_id(message_id).request_body(self._CreateMessageReactionRequestBody.builder().reaction_type(self._Emoji.builder().emoji_type(emoji_type).build()).build()).build()
|
||||
await asyncio.to_thread(self._api_client.im.v1.message_reaction.create, request)
|
||||
logger.info("[Feishu] reaction '%s' added to message %s", emoji_type, message_id)
|
||||
except Exception:
|
||||
logger.exception("[Feishu] failed to add reaction '%s' to message %s", emoji_type, message_id)
|
||||
|
||||
async def _reply_card(self, message_id: str, text: str) -> str | None:
|
||||
"""Reply with an interactive card and return the created card message ID."""
|
||||
if not self._api_client:
|
||||
return None
|
||||
|
||||
content = self._build_card_content(text)
|
||||
request = self._ReplyMessageRequest.builder().message_id(message_id).request_body(self._ReplyMessageRequestBody.builder().msg_type("interactive").content(content).reply_in_thread(True).build()).build()
|
||||
response = await asyncio.to_thread(self._api_client.im.v1.message.reply, request)
|
||||
response_data = getattr(response, "data", None)
|
||||
return getattr(response_data, "message_id", None)
|
||||
|
||||
async def _create_card(self, chat_id: str, text: str) -> None:
|
||||
"""Create a new card message in the target chat."""
|
||||
if not self._api_client:
|
||||
return
|
||||
|
||||
content = self._build_card_content(text)
|
||||
request = self._CreateMessageRequest.builder().receive_id_type("chat_id").request_body(self._CreateMessageRequestBody.builder().receive_id(chat_id).msg_type("interactive").content(content).build()).build()
|
||||
await asyncio.to_thread(self._api_client.im.v1.message.create, request)
|
||||
|
||||
async def _update_card(self, message_id: str, text: str) -> None:
|
||||
"""Patch an existing card message in place."""
|
||||
if not self._api_client or not self._PatchMessageRequest:
|
||||
return
|
||||
|
||||
content = self._build_card_content(text)
|
||||
request = self._PatchMessageRequest.builder().message_id(message_id).request_body(self._PatchMessageRequestBody.builder().content(content).build()).build()
|
||||
await asyncio.to_thread(self._api_client.im.v1.message.patch, request)
|
||||
|
||||
def _track_background_task(self, task: asyncio.Task, *, name: str, msg_id: str) -> None:
|
||||
"""Keep a strong reference to fire-and-forget tasks and surface errors."""
|
||||
self._background_tasks.add(task)
|
||||
task.add_done_callback(lambda done_task, task_name=name, mid=msg_id: self._finalize_background_task(done_task, task_name, mid))
|
||||
|
||||
def _finalize_background_task(self, task: asyncio.Task, name: str, msg_id: str) -> None:
|
||||
self._background_tasks.discard(task)
|
||||
self._log_task_error(task, name, msg_id)
|
||||
|
||||
async def _create_running_card(self, source_message_id: str, text: str) -> str | None:
|
||||
"""Create the running card and cache its message ID when available."""
|
||||
running_card_id = await self._reply_card(source_message_id, text)
|
||||
if running_card_id:
|
||||
self._running_card_ids[source_message_id] = running_card_id
|
||||
logger.info("[Feishu] running card created: source=%s card=%s", source_message_id, running_card_id)
|
||||
else:
|
||||
logger.warning("[Feishu] running card creation returned no message_id for source=%s, subsequent updates will fall back to new replies", source_message_id)
|
||||
return running_card_id
|
||||
|
||||
def _ensure_running_card_started(self, source_message_id: str, text: str = "Working on it...") -> asyncio.Task | None:
|
||||
"""Start running-card creation once per source message."""
|
||||
running_card_id = self._running_card_ids.get(source_message_id)
|
||||
if running_card_id:
|
||||
return None
|
||||
|
||||
running_card_task = self._running_card_tasks.get(source_message_id)
|
||||
if running_card_task:
|
||||
return running_card_task
|
||||
|
||||
running_card_task = asyncio.create_task(self._create_running_card(source_message_id, text))
|
||||
self._running_card_tasks[source_message_id] = running_card_task
|
||||
running_card_task.add_done_callback(lambda done_task, mid=source_message_id: self._finalize_running_card_task(mid, done_task))
|
||||
return running_card_task
|
||||
|
||||
def _finalize_running_card_task(self, source_message_id: str, task: asyncio.Task) -> None:
|
||||
if self._running_card_tasks.get(source_message_id) is task:
|
||||
self._running_card_tasks.pop(source_message_id, None)
|
||||
self._log_task_error(task, "create_running_card", source_message_id)
|
||||
|
||||
async def _ensure_running_card(self, source_message_id: str, text: str = "Working on it...") -> str | None:
|
||||
"""Ensure the in-thread running card exists and track its message ID."""
|
||||
running_card_id = self._running_card_ids.get(source_message_id)
|
||||
if running_card_id:
|
||||
return running_card_id
|
||||
|
||||
running_card_task = self._ensure_running_card_started(source_message_id, text)
|
||||
if running_card_task is None:
|
||||
return self._running_card_ids.get(source_message_id)
|
||||
return await running_card_task
|
||||
|
||||
async def _send_running_reply(self, message_id: str) -> None:
|
||||
"""Reply to a message in-thread with a running card."""
|
||||
try:
|
||||
await self._ensure_running_card(message_id)
|
||||
except Exception:
|
||||
logger.exception("[Feishu] failed to send running reply for message %s", message_id)
|
||||
|
||||
async def _send_card_message(self, msg: OutboundMessage) -> None:
|
||||
"""Send or update the Feishu card tied to the current request."""
|
||||
source_message_id = msg.thread_ts
|
||||
if source_message_id:
|
||||
running_card_id = self._running_card_ids.get(source_message_id)
|
||||
awaited_running_card_task = False
|
||||
|
||||
if not running_card_id:
|
||||
running_card_task = self._running_card_tasks.get(source_message_id)
|
||||
if running_card_task:
|
||||
awaited_running_card_task = True
|
||||
running_card_id = await running_card_task
|
||||
|
||||
if running_card_id:
|
||||
try:
|
||||
await self._update_card(running_card_id, msg.text)
|
||||
except Exception:
|
||||
if not msg.is_final:
|
||||
raise
|
||||
logger.exception(
|
||||
"[Feishu] failed to patch running card %s, falling back to final reply",
|
||||
running_card_id,
|
||||
)
|
||||
await self._reply_card(source_message_id, msg.text)
|
||||
else:
|
||||
logger.info("[Feishu] running card updated: source=%s card=%s", source_message_id, running_card_id)
|
||||
elif msg.is_final:
|
||||
await self._reply_card(source_message_id, msg.text)
|
||||
elif awaited_running_card_task:
|
||||
logger.warning(
|
||||
"[Feishu] running card task finished without message_id for source=%s, skipping duplicate non-final creation",
|
||||
source_message_id,
|
||||
)
|
||||
else:
|
||||
await self._ensure_running_card(source_message_id, msg.text)
|
||||
|
||||
if msg.is_final:
|
||||
self._running_card_ids.pop(source_message_id, None)
|
||||
await self._add_reaction(source_message_id, "DONE")
|
||||
return
|
||||
|
||||
await self._create_card(msg.chat_id, msg.text)
|
||||
|
||||
# -- internal ----------------------------------------------------------
|
||||
|
||||
@staticmethod
|
||||
def _log_future_error(fut, name: str, msg_id: str) -> None:
|
||||
"""Callback for run_coroutine_threadsafe futures to surface errors."""
|
||||
try:
|
||||
exc = fut.exception()
|
||||
if exc:
|
||||
logger.error("[Feishu] %s failed for msg_id=%s: %s", name, msg_id, exc)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
@staticmethod
|
||||
def _log_task_error(task: asyncio.Task, name: str, msg_id: str) -> None:
|
||||
"""Callback for background asyncio tasks to surface errors."""
|
||||
try:
|
||||
exc = task.exception()
|
||||
if exc:
|
||||
logger.error("[Feishu] %s failed for msg_id=%s: %s", name, msg_id, exc)
|
||||
except asyncio.CancelledError:
|
||||
logger.info("[Feishu] %s cancelled for msg_id=%s", name, msg_id)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
async def _prepare_inbound(self, msg_id: str, inbound) -> None:
|
||||
"""Kick off Feishu side effects without delaying inbound dispatch."""
|
||||
reaction_task = asyncio.create_task(self._add_reaction(msg_id, "OK"))
|
||||
self._track_background_task(reaction_task, name="add_reaction", msg_id=msg_id)
|
||||
self._ensure_running_card_started(msg_id)
|
||||
await self.bus.publish_inbound(inbound)
|
||||
|
||||
def _on_message(self, event) -> None:
|
||||
"""Called by lark-oapi when a message is received (runs in lark thread)."""
|
||||
try:
|
||||
logger.info("[Feishu] raw event received: type=%s", type(event).__name__)
|
||||
message = event.event.message
|
||||
chat_id = message.chat_id
|
||||
msg_id = message.message_id
|
||||
sender_id = event.event.sender.sender_id.open_id
|
||||
|
||||
# root_id is set when the message is a reply within a Feishu thread.
|
||||
# Use it as topic_id so all replies share the same DeerFlow thread.
|
||||
root_id = getattr(message, "root_id", None) or None
|
||||
|
||||
# Parse message content
|
||||
content = json.loads(message.content)
|
||||
|
||||
# files_list store the any-file-key in feishu messages, which can be used to download the file content later
|
||||
# In Feishu channel, image_keys are independent of file_keys.
|
||||
# The file_key includes files, videos, and audio, but does not include stickers.
|
||||
files_list = []
|
||||
|
||||
if "text" in content:
|
||||
# Handle plain text messages
|
||||
text = content["text"]
|
||||
elif "file_key" in content:
|
||||
file_key = content.get("file_key")
|
||||
if isinstance(file_key, str) and file_key:
|
||||
files_list.append({"file_key": file_key})
|
||||
text = "[file]"
|
||||
else:
|
||||
text = ""
|
||||
elif "image_key" in content:
|
||||
image_key = content.get("image_key")
|
||||
if isinstance(image_key, str) and image_key:
|
||||
files_list.append({"image_key": image_key})
|
||||
text = "[image]"
|
||||
else:
|
||||
text = ""
|
||||
elif "content" in content and isinstance(content["content"], list):
|
||||
# Handle rich-text messages with a top-level "content" list (e.g., topic groups/posts)
|
||||
text_paragraphs: list[str] = []
|
||||
for paragraph in content["content"]:
|
||||
if isinstance(paragraph, list):
|
||||
paragraph_text_parts: list[str] = []
|
||||
for element in paragraph:
|
||||
if isinstance(element, dict):
|
||||
# Include both normal text and @ mentions
|
||||
if element.get("tag") in ("text", "at"):
|
||||
text_value = element.get("text", "")
|
||||
if text_value:
|
||||
paragraph_text_parts.append(text_value)
|
||||
elif element.get("tag") == "img":
|
||||
image_key = element.get("image_key")
|
||||
if isinstance(image_key, str) and image_key:
|
||||
files_list.append({"image_key": image_key})
|
||||
paragraph_text_parts.append("[image]")
|
||||
elif element.get("tag") in ("file", "media"):
|
||||
file_key = element.get("file_key")
|
||||
if isinstance(file_key, str) and file_key:
|
||||
files_list.append({"file_key": file_key})
|
||||
paragraph_text_parts.append("[file]")
|
||||
if paragraph_text_parts:
|
||||
# Join text segments within a paragraph with spaces to avoid "helloworld"
|
||||
text_paragraphs.append(" ".join(paragraph_text_parts))
|
||||
|
||||
# Join paragraphs with blank lines to preserve paragraph boundaries
|
||||
text = "\n\n".join(text_paragraphs)
|
||||
else:
|
||||
text = ""
|
||||
text = text.strip()
|
||||
|
||||
logger.info(
|
||||
"[Feishu] parsed message: chat_id=%s, msg_id=%s, root_id=%s, sender=%s, text=%r",
|
||||
chat_id,
|
||||
msg_id,
|
||||
root_id,
|
||||
sender_id,
|
||||
text[:100] if text else "",
|
||||
)
|
||||
|
||||
if not (text or files_list):
|
||||
logger.info("[Feishu] empty text, ignoring message")
|
||||
return
|
||||
|
||||
# Only treat known slash commands as commands; absolute paths and
|
||||
# other slash-prefixed text should be handled as normal chat.
|
||||
if _is_feishu_command(text):
|
||||
msg_type = InboundMessageType.COMMAND
|
||||
else:
|
||||
msg_type = InboundMessageType.CHAT
|
||||
|
||||
# topic_id: use root_id for replies (same topic), msg_id for new messages (new topic)
|
||||
topic_id = root_id or msg_id
|
||||
|
||||
inbound = self._make_inbound(
|
||||
chat_id=chat_id,
|
||||
user_id=sender_id,
|
||||
text=text,
|
||||
msg_type=msg_type,
|
||||
thread_ts=msg_id,
|
||||
files=files_list,
|
||||
metadata={"message_id": msg_id, "root_id": root_id},
|
||||
)
|
||||
inbound.topic_id = topic_id
|
||||
|
||||
# Schedule on the async event loop
|
||||
if self._main_loop and self._main_loop.is_running():
|
||||
logger.info("[Feishu] publishing inbound message to bus (type=%s, msg_id=%s)", msg_type.value, msg_id)
|
||||
fut = asyncio.run_coroutine_threadsafe(self._prepare_inbound(msg_id, inbound), self._main_loop)
|
||||
fut.add_done_callback(lambda f, mid=msg_id: self._log_future_error(f, "prepare_inbound", mid))
|
||||
else:
|
||||
logger.warning("[Feishu] main loop not running, cannot publish inbound message")
|
||||
except Exception:
|
||||
logger.exception("[Feishu] error processing message")
|
||||
@@ -1,976 +0,0 @@
|
||||
"""ChannelManager — consumes inbound messages and dispatches them to the DeerFlow agent via LangGraph Server."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import mimetypes
|
||||
import re
|
||||
import time
|
||||
from collections.abc import Awaitable, Callable, Mapping
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
from langgraph_sdk.errors import ConflictError
|
||||
|
||||
from app.plugins.auth.security.actor_context import bind_user_actor_context
|
||||
from app.channels.commands import KNOWN_CHANNEL_COMMANDS
|
||||
from app.channels.message_bus import InboundMessage, InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
|
||||
from app.channels.store import ChannelStore
|
||||
from deerflow.runtime.actor_context import get_effective_user_id
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
DEFAULT_LANGGRAPH_URL = "http://localhost:2024"
|
||||
DEFAULT_GATEWAY_URL = "http://localhost:8001"
|
||||
DEFAULT_ASSISTANT_ID = "lead_agent"
|
||||
CUSTOM_AGENT_NAME_PATTERN = re.compile(r"^[A-Za-z0-9-]+$")
|
||||
|
||||
DEFAULT_RUN_CONFIG: dict[str, Any] = {"recursion_limit": 100}
|
||||
DEFAULT_RUN_CONTEXT: dict[str, Any] = {
|
||||
"thinking_enabled": True,
|
||||
"is_plan_mode": False,
|
||||
"subagent_enabled": False,
|
||||
}
|
||||
STREAM_UPDATE_MIN_INTERVAL_SECONDS = 0.35
|
||||
THREAD_BUSY_MESSAGE = "This conversation is already processing another request. Please wait for it to finish and try again."
|
||||
|
||||
CHANNEL_CAPABILITIES = {
|
||||
"feishu": {"supports_streaming": True},
|
||||
"slack": {"supports_streaming": False},
|
||||
"telegram": {"supports_streaming": False},
|
||||
"wechat": {"supports_streaming": False},
|
||||
"wecom": {"supports_streaming": True},
|
||||
}
|
||||
|
||||
InboundFileReader = Callable[[dict[str, Any], httpx.AsyncClient], Awaitable[bytes | None]]
|
||||
|
||||
|
||||
INBOUND_FILE_READERS: dict[str, InboundFileReader] = {}
|
||||
|
||||
|
||||
def register_inbound_file_reader(channel_name: str, reader: InboundFileReader) -> None:
|
||||
INBOUND_FILE_READERS[channel_name] = reader
|
||||
|
||||
|
||||
async def _read_http_inbound_file(file_info: dict[str, Any], client: httpx.AsyncClient) -> bytes | None:
|
||||
url = file_info.get("url")
|
||||
if not isinstance(url, str) or not url:
|
||||
return None
|
||||
|
||||
resp = await client.get(url)
|
||||
resp.raise_for_status()
|
||||
return resp.content
|
||||
|
||||
|
||||
async def _read_wecom_inbound_file(file_info: dict[str, Any], client: httpx.AsyncClient) -> bytes | None:
|
||||
data = await _read_http_inbound_file(file_info, client)
|
||||
if data is None:
|
||||
return None
|
||||
|
||||
aeskey = file_info.get("aeskey") if isinstance(file_info.get("aeskey"), str) else None
|
||||
if not aeskey:
|
||||
return data
|
||||
|
||||
try:
|
||||
from aibot.crypto_utils import decrypt_file
|
||||
except Exception:
|
||||
logger.exception("[Manager] failed to import WeCom decrypt_file")
|
||||
return None
|
||||
|
||||
return decrypt_file(data, aeskey)
|
||||
|
||||
|
||||
async def _read_wechat_inbound_file(file_info: dict[str, Any], client: httpx.AsyncClient) -> bytes | None:
|
||||
raw_path = file_info.get("path")
|
||||
if isinstance(raw_path, str) and raw_path.strip():
|
||||
try:
|
||||
return await asyncio.to_thread(Path(raw_path).read_bytes)
|
||||
except OSError:
|
||||
logger.exception("[Manager] failed to read WeChat inbound file from local path: %s", raw_path)
|
||||
return None
|
||||
|
||||
full_url = file_info.get("full_url")
|
||||
if isinstance(full_url, str) and full_url.strip():
|
||||
return await _read_http_inbound_file({"url": full_url}, client)
|
||||
|
||||
return None
|
||||
|
||||
|
||||
register_inbound_file_reader("wecom", _read_wecom_inbound_file)
|
||||
register_inbound_file_reader("wechat", _read_wechat_inbound_file)
|
||||
|
||||
|
||||
class InvalidChannelSessionConfigError(ValueError):
|
||||
"""Raised when IM channel session overrides contain invalid agent config."""
|
||||
|
||||
|
||||
def _is_thread_busy_error(exc: BaseException | None) -> bool:
|
||||
if exc is None:
|
||||
return False
|
||||
if isinstance(exc, ConflictError):
|
||||
return True
|
||||
return "already running a task" in str(exc)
|
||||
|
||||
|
||||
def _as_dict(value: Any) -> dict[str, Any]:
|
||||
return dict(value) if isinstance(value, Mapping) else {}
|
||||
|
||||
|
||||
def _merge_dicts(*layers: Any) -> dict[str, Any]:
|
||||
merged: dict[str, Any] = {}
|
||||
for layer in layers:
|
||||
if isinstance(layer, Mapping):
|
||||
merged.update(layer)
|
||||
return merged
|
||||
|
||||
|
||||
def _normalize_custom_agent_name(raw_value: str) -> str:
|
||||
"""Normalize legacy channel assistant IDs into valid custom agent names."""
|
||||
normalized = raw_value.strip().lower().replace("_", "-")
|
||||
if not normalized:
|
||||
raise InvalidChannelSessionConfigError("Channel session assistant_id is empty. Use 'lead_agent' or a valid custom agent name.")
|
||||
if not CUSTOM_AGENT_NAME_PATTERN.fullmatch(normalized):
|
||||
raise InvalidChannelSessionConfigError(f"Invalid channel session assistant_id {raw_value!r}. Use 'lead_agent' or a custom agent name containing only letters, digits, and hyphens.")
|
||||
return normalized
|
||||
|
||||
|
||||
def _extract_response_text(result: dict | list) -> str:
|
||||
"""Extract the last AI message text from a LangGraph runs.wait result.
|
||||
|
||||
``runs.wait`` returns the final state dict which contains a ``messages``
|
||||
list. Each message is a dict with at least ``type`` and ``content``.
|
||||
|
||||
Handles special cases:
|
||||
- Regular AI text responses
|
||||
- Clarification interrupts (``ask_clarification`` tool messages)
|
||||
- AI messages with tool_calls but no text content
|
||||
"""
|
||||
if isinstance(result, list):
|
||||
messages = result
|
||||
elif isinstance(result, dict):
|
||||
messages = result.get("messages", [])
|
||||
else:
|
||||
return ""
|
||||
|
||||
# Walk backwards to find usable response text, but stop at the last
|
||||
# human message to avoid returning text from a previous turn.
|
||||
for msg in reversed(messages):
|
||||
if not isinstance(msg, dict):
|
||||
continue
|
||||
|
||||
msg_type = msg.get("type")
|
||||
|
||||
# Stop at the last human message — anything before it is a previous turn
|
||||
if msg_type == "human":
|
||||
break
|
||||
|
||||
# Check for tool messages from ask_clarification (interrupt case)
|
||||
if msg_type == "tool" and msg.get("name") == "ask_clarification":
|
||||
content = msg.get("content", "")
|
||||
if isinstance(content, str) and content:
|
||||
return content
|
||||
|
||||
# Regular AI message with text content
|
||||
if msg_type == "ai":
|
||||
content = msg.get("content", "")
|
||||
if isinstance(content, str) and content:
|
||||
return content
|
||||
# content can be a list of content blocks
|
||||
if isinstance(content, list):
|
||||
parts = []
|
||||
for block in content:
|
||||
if isinstance(block, dict) and block.get("type") == "text":
|
||||
parts.append(block.get("text", ""))
|
||||
elif isinstance(block, str):
|
||||
parts.append(block)
|
||||
text = "".join(parts)
|
||||
if text:
|
||||
return text
|
||||
return ""
|
||||
|
||||
|
||||
def _extract_text_content(content: Any) -> str:
|
||||
"""Extract text from a streaming payload content field."""
|
||||
if isinstance(content, str):
|
||||
return content
|
||||
if isinstance(content, list):
|
||||
parts: list[str] = []
|
||||
for block in content:
|
||||
if isinstance(block, str):
|
||||
parts.append(block)
|
||||
elif isinstance(block, Mapping):
|
||||
text = block.get("text")
|
||||
if isinstance(text, str):
|
||||
parts.append(text)
|
||||
else:
|
||||
nested = block.get("content")
|
||||
if isinstance(nested, str):
|
||||
parts.append(nested)
|
||||
return "".join(parts)
|
||||
if isinstance(content, Mapping):
|
||||
for key in ("text", "content"):
|
||||
value = content.get(key)
|
||||
if isinstance(value, str):
|
||||
return value
|
||||
return ""
|
||||
|
||||
|
||||
def _merge_stream_text(existing: str, chunk: str) -> str:
|
||||
"""Merge either delta text or cumulative text into a single snapshot."""
|
||||
if not chunk:
|
||||
return existing
|
||||
if not existing or chunk == existing:
|
||||
return chunk or existing
|
||||
if chunk.startswith(existing):
|
||||
return chunk
|
||||
if existing.endswith(chunk):
|
||||
return existing
|
||||
return existing + chunk
|
||||
|
||||
|
||||
def _extract_stream_message_id(payload: Any, metadata: Any) -> str | None:
|
||||
"""Best-effort extraction of the streamed AI message identifier."""
|
||||
candidates = [payload, metadata]
|
||||
if isinstance(payload, Mapping):
|
||||
candidates.append(payload.get("kwargs"))
|
||||
|
||||
for candidate in candidates:
|
||||
if not isinstance(candidate, Mapping):
|
||||
continue
|
||||
for key in ("id", "message_id"):
|
||||
value = candidate.get(key)
|
||||
if isinstance(value, str) and value:
|
||||
return value
|
||||
return None
|
||||
|
||||
|
||||
def _accumulate_stream_text(
|
||||
buffers: dict[str, str],
|
||||
current_message_id: str | None,
|
||||
event_data: Any,
|
||||
) -> tuple[str | None, str | None]:
|
||||
"""Convert a ``messages-tuple`` event into the latest displayable AI text."""
|
||||
payload = event_data
|
||||
metadata: Any = None
|
||||
if isinstance(event_data, (list, tuple)):
|
||||
if event_data:
|
||||
payload = event_data[0]
|
||||
if len(event_data) > 1:
|
||||
metadata = event_data[1]
|
||||
|
||||
if isinstance(payload, str):
|
||||
message_id = current_message_id or "__default__"
|
||||
buffers[message_id] = _merge_stream_text(buffers.get(message_id, ""), payload)
|
||||
return buffers[message_id], message_id
|
||||
|
||||
if not isinstance(payload, Mapping):
|
||||
return None, current_message_id
|
||||
|
||||
payload_type = str(payload.get("type", "")).lower()
|
||||
if "tool" in payload_type:
|
||||
return None, current_message_id
|
||||
|
||||
text = _extract_text_content(payload.get("content"))
|
||||
if not text and isinstance(payload.get("kwargs"), Mapping):
|
||||
text = _extract_text_content(payload["kwargs"].get("content"))
|
||||
if not text:
|
||||
return None, current_message_id
|
||||
|
||||
message_id = _extract_stream_message_id(payload, metadata) or current_message_id or "__default__"
|
||||
buffers[message_id] = _merge_stream_text(buffers.get(message_id, ""), text)
|
||||
return buffers[message_id], message_id
|
||||
|
||||
|
||||
def _extract_artifacts(result: dict | list) -> list[str]:
|
||||
"""Extract artifact paths from the last AI response cycle only.
|
||||
|
||||
Instead of reading the full accumulated ``artifacts`` state (which contains
|
||||
all artifacts ever produced in the thread), this inspects the messages after
|
||||
the last human message and collects file paths from ``present_files`` tool
|
||||
calls. This ensures only newly-produced artifacts are returned.
|
||||
"""
|
||||
if isinstance(result, list):
|
||||
messages = result
|
||||
elif isinstance(result, dict):
|
||||
messages = result.get("messages", [])
|
||||
else:
|
||||
return []
|
||||
|
||||
artifacts: list[str] = []
|
||||
for msg in reversed(messages):
|
||||
if not isinstance(msg, dict):
|
||||
continue
|
||||
# Stop at the last human message — anything before it is a previous turn
|
||||
if msg.get("type") == "human":
|
||||
break
|
||||
# Look for AI messages with present_files tool calls
|
||||
if msg.get("type") == "ai":
|
||||
for tc in msg.get("tool_calls", []):
|
||||
if isinstance(tc, dict) and tc.get("name") == "present_files":
|
||||
args = tc.get("args", {})
|
||||
paths = args.get("filepaths", [])
|
||||
if isinstance(paths, list):
|
||||
artifacts.extend(p for p in paths if isinstance(p, str))
|
||||
return artifacts
|
||||
|
||||
|
||||
def _format_artifact_text(artifacts: list[str]) -> str:
|
||||
"""Format artifact paths into a human-readable text block listing filenames."""
|
||||
import posixpath
|
||||
|
||||
filenames = [posixpath.basename(p) for p in artifacts]
|
||||
if len(filenames) == 1:
|
||||
return f"Created File: 📎 {filenames[0]}"
|
||||
return "Created Files: 📎 " + "、".join(filenames)
|
||||
|
||||
|
||||
_OUTPUTS_VIRTUAL_PREFIX = "/mnt/user-data/outputs/"
|
||||
|
||||
|
||||
def _resolve_attachments(thread_id: str, artifacts: list[str], *, user_id: str | None = None) -> list[ResolvedAttachment]:
|
||||
"""Resolve virtual artifact paths to host filesystem paths with metadata.
|
||||
|
||||
Only paths under ``/mnt/user-data/outputs/`` are accepted; any other
|
||||
virtual path is rejected with a warning to prevent exfiltrating uploads
|
||||
or workspace files via IM channels.
|
||||
|
||||
Skips artifacts that cannot be resolved (missing files, invalid paths)
|
||||
and logs warnings for them.
|
||||
"""
|
||||
from deerflow.config.paths import get_paths
|
||||
|
||||
attachments: list[ResolvedAttachment] = []
|
||||
paths = get_paths()
|
||||
with bind_user_actor_context(user_id):
|
||||
effective_user_id = get_effective_user_id()
|
||||
outputs_dir = paths.sandbox_outputs_dir(thread_id, user_id=effective_user_id).resolve()
|
||||
for virtual_path in artifacts:
|
||||
# Security: only allow files from the agent outputs directory
|
||||
if not virtual_path.startswith(_OUTPUTS_VIRTUAL_PREFIX):
|
||||
logger.warning("[Manager] rejected non-outputs artifact path: %s", virtual_path)
|
||||
continue
|
||||
try:
|
||||
actual = paths.resolve_virtual_path(thread_id, virtual_path, user_id=effective_user_id)
|
||||
# Verify the resolved path is actually under the outputs directory
|
||||
# (guards against path-traversal even after prefix check)
|
||||
try:
|
||||
actual.resolve().relative_to(outputs_dir)
|
||||
except ValueError:
|
||||
logger.warning("[Manager] artifact path escapes outputs dir: %s -> %s", virtual_path, actual)
|
||||
continue
|
||||
if not actual.is_file():
|
||||
logger.warning("[Manager] artifact not found on disk: %s -> %s", virtual_path, actual)
|
||||
continue
|
||||
mime, _ = mimetypes.guess_type(str(actual))
|
||||
mime = mime or "application/octet-stream"
|
||||
attachments.append(
|
||||
ResolvedAttachment(
|
||||
virtual_path=virtual_path,
|
||||
actual_path=actual,
|
||||
filename=actual.name,
|
||||
mime_type=mime,
|
||||
size=actual.stat().st_size,
|
||||
is_image=mime.startswith("image/"),
|
||||
)
|
||||
)
|
||||
except (ValueError, OSError) as exc:
|
||||
logger.warning("[Manager] failed to resolve artifact %s: %s", virtual_path, exc)
|
||||
return attachments
|
||||
|
||||
|
||||
def _prepare_artifact_delivery(
|
||||
thread_id: str,
|
||||
response_text: str,
|
||||
artifacts: list[str],
|
||||
*,
|
||||
user_id: str | None = None,
|
||||
) -> tuple[str, list[ResolvedAttachment]]:
|
||||
"""Resolve attachments and append filename fallbacks to the text response."""
|
||||
attachments: list[ResolvedAttachment] = []
|
||||
if not artifacts:
|
||||
return response_text, attachments
|
||||
|
||||
attachments = _resolve_attachments(thread_id, artifacts, user_id=user_id)
|
||||
resolved_virtuals = {attachment.virtual_path for attachment in attachments}
|
||||
unresolved = [path for path in artifacts if path not in resolved_virtuals]
|
||||
|
||||
if unresolved:
|
||||
artifact_text = _format_artifact_text(unresolved)
|
||||
response_text = (response_text + "\n\n" + artifact_text) if response_text else artifact_text
|
||||
|
||||
# Always include resolved attachment filenames as a text fallback so files
|
||||
# remain discoverable even when the upload is skipped or fails.
|
||||
if attachments:
|
||||
resolved_text = _format_artifact_text([attachment.virtual_path for attachment in attachments])
|
||||
response_text = (response_text + "\n\n" + resolved_text) if response_text else resolved_text
|
||||
|
||||
return response_text, attachments
|
||||
|
||||
|
||||
async def _ingest_inbound_files(thread_id: str, msg: InboundMessage) -> list[dict[str, Any]]:
|
||||
if not msg.files:
|
||||
return []
|
||||
|
||||
from deerflow.uploads.manager import claim_unique_filename, ensure_uploads_dir, normalize_filename
|
||||
|
||||
with bind_user_actor_context(msg.user_id):
|
||||
uploads_dir = ensure_uploads_dir(thread_id)
|
||||
seen_names = {entry.name for entry in uploads_dir.iterdir() if entry.is_file()}
|
||||
|
||||
created: list[dict[str, Any]] = []
|
||||
file_reader = INBOUND_FILE_READERS.get(msg.channel_name, _read_http_inbound_file)
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(20.0)) as client:
|
||||
for idx, f in enumerate(msg.files):
|
||||
if not isinstance(f, dict):
|
||||
continue
|
||||
|
||||
ftype = f.get("type") if isinstance(f.get("type"), str) else "file"
|
||||
filename = f.get("filename") if isinstance(f.get("filename"), str) else ""
|
||||
|
||||
try:
|
||||
data = await file_reader(f, client)
|
||||
except Exception:
|
||||
logger.exception(
|
||||
"[Manager] failed to read inbound file: channel=%s, file=%s",
|
||||
msg.channel_name,
|
||||
f.get("url") or filename or idx,
|
||||
)
|
||||
continue
|
||||
|
||||
if data is None:
|
||||
logger.warning(
|
||||
"[Manager] inbound file reader returned no data: channel=%s, file=%s",
|
||||
msg.channel_name,
|
||||
f.get("url") or filename or idx,
|
||||
)
|
||||
continue
|
||||
|
||||
if not filename:
|
||||
ext = ".bin"
|
||||
if ftype == "image":
|
||||
ext = ".png"
|
||||
filename = f"{msg.thread_ts or 'msg'}_{idx}{ext}"
|
||||
|
||||
try:
|
||||
safe_name = claim_unique_filename(normalize_filename(filename), seen_names)
|
||||
except ValueError:
|
||||
logger.warning(
|
||||
"[Manager] skipping inbound file with unsafe filename: channel=%s, file=%r",
|
||||
msg.channel_name,
|
||||
filename,
|
||||
)
|
||||
continue
|
||||
|
||||
dest = uploads_dir / safe_name
|
||||
try:
|
||||
dest.write_bytes(data)
|
||||
except Exception:
|
||||
logger.exception("[Manager] failed to write inbound file: %s", dest)
|
||||
continue
|
||||
|
||||
created.append(
|
||||
{
|
||||
"filename": safe_name,
|
||||
"size": len(data),
|
||||
"path": f"/mnt/user-data/uploads/{safe_name}",
|
||||
"is_image": ftype == "image",
|
||||
}
|
||||
)
|
||||
|
||||
return created
|
||||
|
||||
|
||||
def _format_uploaded_files_block(files: list[dict[str, Any]]) -> str:
|
||||
lines = [
|
||||
"<uploaded_files>",
|
||||
"The following files were uploaded in this message:",
|
||||
"",
|
||||
]
|
||||
if not files:
|
||||
lines.append("(empty)")
|
||||
else:
|
||||
for f in files:
|
||||
filename = f.get("filename", "")
|
||||
size = int(f.get("size") or 0)
|
||||
size_kb = size / 1024 if size else 0
|
||||
size_str = f"{size_kb:.1f} KB" if size_kb < 1024 else f"{size_kb / 1024:.1f} MB"
|
||||
path = f.get("path", "")
|
||||
is_image = bool(f.get("is_image"))
|
||||
file_kind = "image" if is_image else "file"
|
||||
lines.append(f"- {filename} ({size_str})")
|
||||
lines.append(f" Type: {file_kind}")
|
||||
lines.append(f" Path: {path}")
|
||||
lines.append("")
|
||||
lines.append("Use `read_file` for text-based files and documents.")
|
||||
lines.append("Use `view_image` for image files (jpg, jpeg, png, webp) so the model can inspect the image content.")
|
||||
lines.append("</uploaded_files>")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
class ChannelManager:
|
||||
"""Core dispatcher that bridges IM channels to the DeerFlow agent.
|
||||
|
||||
It reads from the MessageBus inbound queue, creates/reuses threads on
|
||||
the LangGraph Server, sends messages via ``runs.wait``, and publishes
|
||||
outbound responses back through the bus.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
bus: MessageBus,
|
||||
store: ChannelStore,
|
||||
*,
|
||||
max_concurrency: int = 5,
|
||||
langgraph_url: str = DEFAULT_LANGGRAPH_URL,
|
||||
gateway_url: str = DEFAULT_GATEWAY_URL,
|
||||
assistant_id: str = DEFAULT_ASSISTANT_ID,
|
||||
default_session: dict[str, Any] | None = None,
|
||||
channel_sessions: dict[str, Any] | None = None,
|
||||
) -> None:
|
||||
self.bus = bus
|
||||
self.store = store
|
||||
self._max_concurrency = max_concurrency
|
||||
self._langgraph_url = langgraph_url
|
||||
self._gateway_url = gateway_url
|
||||
self._assistant_id = assistant_id
|
||||
self._default_session = _as_dict(default_session)
|
||||
self._channel_sessions = dict(channel_sessions or {})
|
||||
self._client = None # lazy init — langgraph_sdk async client
|
||||
self._semaphore: asyncio.Semaphore | None = None
|
||||
self._running = False
|
||||
self._task: asyncio.Task | None = None
|
||||
|
||||
@staticmethod
|
||||
def _channel_supports_streaming(channel_name: str) -> bool:
|
||||
return CHANNEL_CAPABILITIES.get(channel_name, {}).get("supports_streaming", False)
|
||||
|
||||
def _resolve_session_layer(self, msg: InboundMessage) -> tuple[dict[str, Any], dict[str, Any]]:
|
||||
channel_layer = _as_dict(self._channel_sessions.get(msg.channel_name))
|
||||
users_layer = _as_dict(channel_layer.get("users"))
|
||||
user_layer = _as_dict(users_layer.get(msg.user_id))
|
||||
return channel_layer, user_layer
|
||||
|
||||
def _resolve_run_params(self, msg: InboundMessage, thread_id: str) -> tuple[str, dict[str, Any], dict[str, Any]]:
|
||||
channel_layer, user_layer = self._resolve_session_layer(msg)
|
||||
|
||||
assistant_id = user_layer.get("assistant_id") or channel_layer.get("assistant_id") or self._default_session.get("assistant_id") or self._assistant_id
|
||||
if not isinstance(assistant_id, str) or not assistant_id.strip():
|
||||
assistant_id = self._assistant_id
|
||||
|
||||
run_config = _merge_dicts(
|
||||
DEFAULT_RUN_CONFIG,
|
||||
self._default_session.get("config"),
|
||||
channel_layer.get("config"),
|
||||
user_layer.get("config"),
|
||||
)
|
||||
|
||||
run_context = _merge_dicts(
|
||||
DEFAULT_RUN_CONTEXT,
|
||||
self._default_session.get("context"),
|
||||
channel_layer.get("context"),
|
||||
user_layer.get("context"),
|
||||
{"thread_id": thread_id},
|
||||
)
|
||||
|
||||
# Custom agents are implemented as lead_agent + agent_name context.
|
||||
# Keep backward compatibility for channel configs that set
|
||||
# assistant_id: <custom-agent-name> by routing through lead_agent.
|
||||
if assistant_id != DEFAULT_ASSISTANT_ID:
|
||||
run_context.setdefault("agent_name", _normalize_custom_agent_name(assistant_id))
|
||||
assistant_id = DEFAULT_ASSISTANT_ID
|
||||
|
||||
return assistant_id, run_config, run_context
|
||||
|
||||
# -- LangGraph SDK client (lazy) ----------------------------------------
|
||||
|
||||
def _get_client(self):
|
||||
"""Return the ``langgraph_sdk`` async client, creating it on first use."""
|
||||
if self._client is None:
|
||||
from langgraph_sdk import get_client
|
||||
|
||||
self._client = get_client(url=self._langgraph_url)
|
||||
return self._client
|
||||
|
||||
# -- lifecycle ---------------------------------------------------------
|
||||
|
||||
async def start(self) -> None:
|
||||
"""Start the dispatch loop."""
|
||||
if self._running:
|
||||
return
|
||||
self._running = True
|
||||
self._semaphore = asyncio.Semaphore(self._max_concurrency)
|
||||
self._task = asyncio.create_task(self._dispatch_loop())
|
||||
logger.info("ChannelManager started (max_concurrency=%d)", self._max_concurrency)
|
||||
|
||||
async def stop(self) -> None:
|
||||
"""Stop the dispatch loop."""
|
||||
self._running = False
|
||||
if self._task:
|
||||
self._task.cancel()
|
||||
try:
|
||||
await self._task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
self._task = None
|
||||
logger.info("ChannelManager stopped")
|
||||
|
||||
# -- dispatch loop -----------------------------------------------------
|
||||
|
||||
async def _dispatch_loop(self) -> None:
|
||||
logger.info("[Manager] dispatch loop started, waiting for inbound messages")
|
||||
while self._running:
|
||||
try:
|
||||
msg = await asyncio.wait_for(self.bus.get_inbound(), timeout=1.0)
|
||||
except TimeoutError:
|
||||
continue
|
||||
except asyncio.CancelledError:
|
||||
break
|
||||
|
||||
logger.info(
|
||||
"[Manager] received inbound: channel=%s, chat_id=%s, type=%s, text=%r",
|
||||
msg.channel_name,
|
||||
msg.chat_id,
|
||||
msg.msg_type.value,
|
||||
msg.text[:100] if msg.text else "",
|
||||
)
|
||||
task = asyncio.create_task(self._handle_message(msg))
|
||||
task.add_done_callback(self._log_task_error)
|
||||
|
||||
@staticmethod
|
||||
def _log_task_error(task: asyncio.Task) -> None:
|
||||
"""Surface unhandled exceptions from background tasks."""
|
||||
if task.cancelled():
|
||||
return
|
||||
exc = task.exception()
|
||||
if exc:
|
||||
logger.error("[Manager] unhandled error in message task: %s", exc, exc_info=exc)
|
||||
|
||||
async def _handle_message(self, msg: InboundMessage) -> None:
|
||||
async with self._semaphore:
|
||||
try:
|
||||
if msg.msg_type == InboundMessageType.COMMAND:
|
||||
await self._handle_command(msg)
|
||||
else:
|
||||
await self._handle_chat(msg)
|
||||
except InvalidChannelSessionConfigError as exc:
|
||||
logger.warning(
|
||||
"Invalid channel session config for %s (chat=%s): %s",
|
||||
msg.channel_name,
|
||||
msg.chat_id,
|
||||
exc,
|
||||
)
|
||||
await self._send_error(msg, str(exc))
|
||||
except Exception:
|
||||
logger.exception(
|
||||
"Error handling message from %s (chat=%s)",
|
||||
msg.channel_name,
|
||||
msg.chat_id,
|
||||
)
|
||||
await self._send_error(msg, "An internal error occurred. Please try again.")
|
||||
|
||||
# -- chat handling -----------------------------------------------------
|
||||
|
||||
async def _create_thread(self, client, msg: InboundMessage) -> str:
|
||||
"""Create a new thread on the LangGraph Server and store the mapping."""
|
||||
thread = await client.threads.create()
|
||||
thread_id = thread["thread_id"]
|
||||
self.store.set_thread_id(
|
||||
msg.channel_name,
|
||||
msg.chat_id,
|
||||
thread_id,
|
||||
topic_id=msg.topic_id,
|
||||
user_id=msg.user_id,
|
||||
)
|
||||
logger.info("[Manager] new thread created on LangGraph Server: thread_id=%s for chat_id=%s topic_id=%s", thread_id, msg.chat_id, msg.topic_id)
|
||||
return thread_id
|
||||
|
||||
async def _handle_chat(self, msg: InboundMessage, extra_context: dict[str, Any] | None = None) -> None:
|
||||
client = self._get_client()
|
||||
|
||||
# Look up existing DeerFlow thread.
|
||||
# topic_id may be None (e.g. Telegram private chats) — the store
|
||||
# handles this by using the "channel:chat_id" key without a topic suffix.
|
||||
thread_id = self.store.get_thread_id(msg.channel_name, msg.chat_id, topic_id=msg.topic_id)
|
||||
if thread_id:
|
||||
logger.info("[Manager] reusing thread: thread_id=%s for topic_id=%s", thread_id, msg.topic_id)
|
||||
|
||||
# No existing thread found — create a new one
|
||||
if thread_id is None:
|
||||
thread_id = await self._create_thread(client, msg)
|
||||
|
||||
assistant_id, run_config, run_context = self._resolve_run_params(msg, thread_id)
|
||||
|
||||
# If the inbound message contains file attachments, let the channel
|
||||
# materialize (download) them and update msg.text to include sandbox file paths.
|
||||
# This enables downstream models to access user-uploaded files by path.
|
||||
# Channels that do not support file download will simply return the original message.
|
||||
if msg.files:
|
||||
from .service import get_channel_service
|
||||
|
||||
service = get_channel_service()
|
||||
channel = service.get_channel(msg.channel_name) if service else None
|
||||
logger.info("[Manager] preparing receive file context for %d attachments", len(msg.files))
|
||||
msg = await channel.receive_file(msg, thread_id) if channel else msg
|
||||
if extra_context:
|
||||
run_context.update(extra_context)
|
||||
|
||||
uploaded = await _ingest_inbound_files(thread_id, msg)
|
||||
if uploaded:
|
||||
msg.text = f"{_format_uploaded_files_block(uploaded)}\n\n{msg.text}".strip()
|
||||
|
||||
if self._channel_supports_streaming(msg.channel_name):
|
||||
await self._handle_streaming_chat(
|
||||
client,
|
||||
msg,
|
||||
thread_id,
|
||||
assistant_id,
|
||||
run_config,
|
||||
run_context,
|
||||
)
|
||||
return
|
||||
|
||||
logger.info("[Manager] invoking runs.wait(thread_id=%s, text=%r)", thread_id, msg.text[:100])
|
||||
result = await client.runs.wait(
|
||||
thread_id,
|
||||
assistant_id,
|
||||
input={"messages": [{"role": "human", "content": msg.text}]},
|
||||
config=run_config,
|
||||
context=run_context,
|
||||
)
|
||||
|
||||
response_text = _extract_response_text(result)
|
||||
artifacts = _extract_artifacts(result)
|
||||
|
||||
logger.info(
|
||||
"[Manager] agent response received: thread_id=%s, response_len=%d, artifacts=%d",
|
||||
thread_id,
|
||||
len(response_text) if response_text else 0,
|
||||
len(artifacts),
|
||||
)
|
||||
|
||||
response_text, attachments = _prepare_artifact_delivery(
|
||||
thread_id,
|
||||
response_text,
|
||||
artifacts,
|
||||
user_id=msg.user_id,
|
||||
)
|
||||
|
||||
if not response_text:
|
||||
if attachments:
|
||||
response_text = _format_artifact_text([a.virtual_path for a in attachments])
|
||||
else:
|
||||
response_text = "(No response from agent)"
|
||||
|
||||
outbound = OutboundMessage(
|
||||
channel_name=msg.channel_name,
|
||||
chat_id=msg.chat_id,
|
||||
thread_id=thread_id,
|
||||
text=response_text,
|
||||
artifacts=artifacts,
|
||||
attachments=attachments,
|
||||
thread_ts=msg.thread_ts,
|
||||
)
|
||||
logger.info("[Manager] publishing outbound message to bus: channel=%s, chat_id=%s", msg.channel_name, msg.chat_id)
|
||||
await self.bus.publish_outbound(outbound)
|
||||
|
||||
async def _handle_streaming_chat(
|
||||
self,
|
||||
client,
|
||||
msg: InboundMessage,
|
||||
thread_id: str,
|
||||
assistant_id: str,
|
||||
run_config: dict[str, Any],
|
||||
run_context: dict[str, Any],
|
||||
) -> None:
|
||||
logger.info("[Manager] invoking runs.stream(thread_id=%s, text=%r)", thread_id, msg.text[:100])
|
||||
|
||||
last_values: dict[str, Any] | list | None = None
|
||||
streamed_buffers: dict[str, str] = {}
|
||||
current_message_id: str | None = None
|
||||
latest_text = ""
|
||||
last_published_text = ""
|
||||
last_publish_at = 0.0
|
||||
stream_error: BaseException | None = None
|
||||
|
||||
try:
|
||||
async for chunk in client.runs.stream(
|
||||
thread_id,
|
||||
assistant_id,
|
||||
input={"messages": [{"role": "human", "content": msg.text}]},
|
||||
config=run_config,
|
||||
context=run_context,
|
||||
stream_mode=["messages-tuple", "values"],
|
||||
multitask_strategy="reject",
|
||||
):
|
||||
event = getattr(chunk, "event", "")
|
||||
data = getattr(chunk, "data", None)
|
||||
|
||||
if event == "messages-tuple":
|
||||
accumulated_text, current_message_id = _accumulate_stream_text(streamed_buffers, current_message_id, data)
|
||||
if accumulated_text:
|
||||
latest_text = accumulated_text
|
||||
elif event == "values" and isinstance(data, (dict, list)):
|
||||
last_values = data
|
||||
snapshot_text = _extract_response_text(data)
|
||||
if snapshot_text:
|
||||
latest_text = snapshot_text
|
||||
|
||||
if not latest_text or latest_text == last_published_text:
|
||||
continue
|
||||
|
||||
now = time.monotonic()
|
||||
if last_published_text and now - last_publish_at < STREAM_UPDATE_MIN_INTERVAL_SECONDS:
|
||||
continue
|
||||
|
||||
await self.bus.publish_outbound(
|
||||
OutboundMessage(
|
||||
channel_name=msg.channel_name,
|
||||
chat_id=msg.chat_id,
|
||||
thread_id=thread_id,
|
||||
text=latest_text,
|
||||
is_final=False,
|
||||
thread_ts=msg.thread_ts,
|
||||
)
|
||||
)
|
||||
last_published_text = latest_text
|
||||
last_publish_at = now
|
||||
except Exception as exc:
|
||||
stream_error = exc
|
||||
if _is_thread_busy_error(exc):
|
||||
logger.warning("[Manager] thread busy (concurrent run rejected): thread_id=%s", thread_id)
|
||||
else:
|
||||
logger.exception("[Manager] streaming error: thread_id=%s", thread_id)
|
||||
finally:
|
||||
result = last_values if last_values is not None else {"messages": [{"type": "ai", "content": latest_text}]}
|
||||
response_text = _extract_response_text(result)
|
||||
artifacts = _extract_artifacts(result)
|
||||
response_text, attachments = _prepare_artifact_delivery(
|
||||
thread_id,
|
||||
response_text,
|
||||
artifacts,
|
||||
user_id=msg.user_id,
|
||||
)
|
||||
|
||||
if not response_text:
|
||||
if attachments:
|
||||
response_text = _format_artifact_text([attachment.virtual_path for attachment in attachments])
|
||||
elif stream_error:
|
||||
if _is_thread_busy_error(stream_error):
|
||||
response_text = THREAD_BUSY_MESSAGE
|
||||
else:
|
||||
response_text = "An error occurred while processing your request. Please try again."
|
||||
else:
|
||||
response_text = latest_text or "(No response from agent)"
|
||||
|
||||
logger.info(
|
||||
"[Manager] streaming response completed: thread_id=%s, response_len=%d, artifacts=%d, error=%s",
|
||||
thread_id,
|
||||
len(response_text),
|
||||
len(artifacts),
|
||||
stream_error,
|
||||
)
|
||||
await self.bus.publish_outbound(
|
||||
OutboundMessage(
|
||||
channel_name=msg.channel_name,
|
||||
chat_id=msg.chat_id,
|
||||
thread_id=thread_id,
|
||||
text=response_text,
|
||||
artifacts=artifacts,
|
||||
attachments=attachments,
|
||||
is_final=True,
|
||||
thread_ts=msg.thread_ts,
|
||||
)
|
||||
)
|
||||
|
||||
# -- command handling --------------------------------------------------
|
||||
|
||||
async def _handle_command(self, msg: InboundMessage) -> None:
|
||||
text = msg.text.strip()
|
||||
parts = text.split(maxsplit=1)
|
||||
command = parts[0].lower().lstrip("/")
|
||||
|
||||
if command == "bootstrap":
|
||||
from dataclasses import replace as _dc_replace
|
||||
|
||||
chat_text = parts[1] if len(parts) > 1 else "Initialize workspace"
|
||||
chat_msg = _dc_replace(msg, text=chat_text, msg_type=InboundMessageType.CHAT)
|
||||
await self._handle_chat(chat_msg, extra_context={"is_bootstrap": True})
|
||||
return
|
||||
|
||||
if command == "new":
|
||||
# Create a new thread on the LangGraph Server
|
||||
client = self._get_client()
|
||||
thread = await client.threads.create()
|
||||
new_thread_id = thread["thread_id"]
|
||||
self.store.set_thread_id(
|
||||
msg.channel_name,
|
||||
msg.chat_id,
|
||||
new_thread_id,
|
||||
topic_id=msg.topic_id,
|
||||
user_id=msg.user_id,
|
||||
)
|
||||
reply = "New conversation started."
|
||||
elif command == "status":
|
||||
thread_id = self.store.get_thread_id(msg.channel_name, msg.chat_id, topic_id=msg.topic_id)
|
||||
reply = f"Active thread: {thread_id}" if thread_id else "No active conversation."
|
||||
elif command == "models":
|
||||
reply = await self._fetch_gateway("/api/models", "models")
|
||||
elif command == "memory":
|
||||
reply = await self._fetch_gateway("/api/memory", "memory")
|
||||
elif command == "help":
|
||||
reply = (
|
||||
"Available commands:\n"
|
||||
"/bootstrap — Start a bootstrap session (enables agent setup)\n"
|
||||
"/new — Start a new conversation\n"
|
||||
"/status — Show current thread info\n"
|
||||
"/models — List available models\n"
|
||||
"/memory — Show memory status\n"
|
||||
"/help — Show this help"
|
||||
)
|
||||
else:
|
||||
available = " | ".join(sorted(KNOWN_CHANNEL_COMMANDS))
|
||||
reply = f"Unknown command: /{command}. Available commands: {available}"
|
||||
|
||||
outbound = OutboundMessage(
|
||||
channel_name=msg.channel_name,
|
||||
chat_id=msg.chat_id,
|
||||
thread_id=self.store.get_thread_id(msg.channel_name, msg.chat_id) or "",
|
||||
text=reply,
|
||||
thread_ts=msg.thread_ts,
|
||||
)
|
||||
await self.bus.publish_outbound(outbound)
|
||||
|
||||
async def _fetch_gateway(self, path: str, kind: str) -> str:
|
||||
"""Fetch data from the Gateway API for command responses."""
|
||||
import httpx
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient() as http:
|
||||
resp = await http.get(f"{self._gateway_url}{path}", timeout=10)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
except Exception:
|
||||
logger.exception("Failed to fetch %s from gateway", kind)
|
||||
return f"Failed to fetch {kind} information."
|
||||
|
||||
if kind == "models":
|
||||
names = [m["name"] for m in data.get("models", [])]
|
||||
return ("Available models:\n" + "\n".join(f"• {n}" for n in names)) if names else "No models configured."
|
||||
elif kind == "memory":
|
||||
facts = data.get("facts", [])
|
||||
return f"Memory contains {len(facts)} fact(s)."
|
||||
return str(data)
|
||||
|
||||
# -- error helper ------------------------------------------------------
|
||||
|
||||
async def _send_error(self, msg: InboundMessage, error_text: str) -> None:
|
||||
outbound = OutboundMessage(
|
||||
channel_name=msg.channel_name,
|
||||
chat_id=msg.chat_id,
|
||||
thread_id=self.store.get_thread_id(msg.channel_name, msg.chat_id) or "",
|
||||
text=error_text,
|
||||
thread_ts=msg.thread_ts,
|
||||
)
|
||||
await self.bus.publish_outbound(outbound)
|
||||
@@ -1,173 +0,0 @@
|
||||
"""MessageBus — async pub/sub hub that decouples channels from the agent dispatcher."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import time
|
||||
from collections.abc import Callable, Coroutine
|
||||
from dataclasses import dataclass, field
|
||||
from enum import StrEnum
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Message types
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class InboundMessageType(StrEnum):
|
||||
"""Types of messages arriving from IM channels."""
|
||||
|
||||
CHAT = "chat"
|
||||
COMMAND = "command"
|
||||
|
||||
|
||||
@dataclass
|
||||
class InboundMessage:
|
||||
"""A message arriving from an IM channel toward the agent dispatcher.
|
||||
|
||||
Attributes:
|
||||
channel_name: Name of the source channel (e.g. "feishu", "slack").
|
||||
chat_id: Platform-specific chat/conversation identifier.
|
||||
user_id: Platform-specific user identifier.
|
||||
text: The message text.
|
||||
msg_type: Whether this is a regular chat message or a command.
|
||||
thread_ts: Optional platform thread identifier (for threaded replies).
|
||||
topic_id: Conversation topic identifier used to map to a DeerFlow thread.
|
||||
Messages sharing the same ``topic_id`` within a ``chat_id`` will
|
||||
reuse the same DeerFlow thread. When ``None``, each message
|
||||
creates a new thread (one-shot Q&A).
|
||||
files: Optional list of file attachments (platform-specific dicts).
|
||||
metadata: Arbitrary extra data from the channel.
|
||||
created_at: Unix timestamp when the message was created.
|
||||
"""
|
||||
|
||||
channel_name: str
|
||||
chat_id: str
|
||||
user_id: str
|
||||
text: str
|
||||
msg_type: InboundMessageType = InboundMessageType.CHAT
|
||||
thread_ts: str | None = None
|
||||
topic_id: str | None = None
|
||||
files: list[dict[str, Any]] = field(default_factory=list)
|
||||
metadata: dict[str, Any] = field(default_factory=dict)
|
||||
created_at: float = field(default_factory=time.time)
|
||||
|
||||
|
||||
@dataclass
|
||||
class ResolvedAttachment:
|
||||
"""A file attachment resolved to a host filesystem path, ready for upload.
|
||||
|
||||
Attributes:
|
||||
virtual_path: Original virtual path (e.g. /mnt/user-data/outputs/report.pdf).
|
||||
actual_path: Resolved host filesystem path.
|
||||
filename: Basename of the file.
|
||||
mime_type: MIME type (e.g. "application/pdf").
|
||||
size: File size in bytes.
|
||||
is_image: True for image/* MIME types (platforms may handle images differently).
|
||||
"""
|
||||
|
||||
virtual_path: str
|
||||
actual_path: Path
|
||||
filename: str
|
||||
mime_type: str
|
||||
size: int
|
||||
is_image: bool
|
||||
|
||||
|
||||
@dataclass
|
||||
class OutboundMessage:
|
||||
"""A message from the agent dispatcher back to a channel.
|
||||
|
||||
Attributes:
|
||||
channel_name: Target channel name (used for routing).
|
||||
chat_id: Target chat/conversation identifier.
|
||||
thread_id: DeerFlow thread ID that produced this response.
|
||||
text: The response text.
|
||||
artifacts: List of artifact paths produced by the agent.
|
||||
is_final: Whether this is the final message in the response stream.
|
||||
thread_ts: Optional platform thread identifier for threaded replies.
|
||||
metadata: Arbitrary extra data.
|
||||
created_at: Unix timestamp.
|
||||
"""
|
||||
|
||||
channel_name: str
|
||||
chat_id: str
|
||||
thread_id: str
|
||||
text: str
|
||||
artifacts: list[str] = field(default_factory=list)
|
||||
attachments: list[ResolvedAttachment] = field(default_factory=list)
|
||||
is_final: bool = True
|
||||
thread_ts: str | None = None
|
||||
metadata: dict[str, Any] = field(default_factory=dict)
|
||||
created_at: float = field(default_factory=time.time)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# MessageBus
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
OutboundCallback = Callable[[OutboundMessage], Coroutine[Any, Any, None]]
|
||||
|
||||
|
||||
class MessageBus:
|
||||
"""Async pub/sub hub connecting channels and the agent dispatcher.
|
||||
|
||||
Channels publish inbound messages; the dispatcher consumes them.
|
||||
The dispatcher publishes outbound messages; channels receive them
|
||||
via registered callbacks.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._inbound_queue: asyncio.Queue[InboundMessage] = asyncio.Queue()
|
||||
self._outbound_listeners: list[OutboundCallback] = []
|
||||
|
||||
# -- inbound -----------------------------------------------------------
|
||||
|
||||
async def publish_inbound(self, msg: InboundMessage) -> None:
|
||||
"""Enqueue an inbound message from a channel."""
|
||||
await self._inbound_queue.put(msg)
|
||||
logger.info(
|
||||
"[Bus] inbound enqueued: channel=%s, chat_id=%s, type=%s, queue_size=%d",
|
||||
msg.channel_name,
|
||||
msg.chat_id,
|
||||
msg.msg_type.value,
|
||||
self._inbound_queue.qsize(),
|
||||
)
|
||||
|
||||
async def get_inbound(self) -> InboundMessage:
|
||||
"""Block until the next inbound message is available."""
|
||||
return await self._inbound_queue.get()
|
||||
|
||||
@property
|
||||
def inbound_queue(self) -> asyncio.Queue[InboundMessage]:
|
||||
return self._inbound_queue
|
||||
|
||||
# -- outbound ----------------------------------------------------------
|
||||
|
||||
def subscribe_outbound(self, callback: OutboundCallback) -> None:
|
||||
"""Register an async callback for outbound messages."""
|
||||
self._outbound_listeners.append(callback)
|
||||
|
||||
def unsubscribe_outbound(self, callback: OutboundCallback) -> None:
|
||||
"""Remove a previously registered outbound callback."""
|
||||
self._outbound_listeners = [cb for cb in self._outbound_listeners if cb is not callback]
|
||||
|
||||
async def publish_outbound(self, msg: OutboundMessage) -> None:
|
||||
"""Dispatch an outbound message to all registered listeners."""
|
||||
logger.info(
|
||||
"[Bus] outbound dispatching: channel=%s, chat_id=%s, listeners=%d, text_len=%d",
|
||||
msg.channel_name,
|
||||
msg.chat_id,
|
||||
len(self._outbound_listeners),
|
||||
len(msg.text),
|
||||
)
|
||||
for callback in self._outbound_listeners:
|
||||
try:
|
||||
await callback(msg)
|
||||
except Exception:
|
||||
logger.exception("Error in outbound callback for channel=%s", msg.channel_name)
|
||||
@@ -1,199 +0,0 @@
|
||||
"""ChannelService — manages the lifecycle of all IM channels."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
from typing import Any
|
||||
|
||||
from app.channels.base import Channel
|
||||
from app.channels.manager import DEFAULT_GATEWAY_URL, DEFAULT_LANGGRAPH_URL, ChannelManager
|
||||
from app.channels.message_bus import MessageBus
|
||||
from app.channels.store import ChannelStore
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Channel name → import path for lazy loading
|
||||
_CHANNEL_REGISTRY: dict[str, str] = {
|
||||
"feishu": "app.channels.feishu:FeishuChannel",
|
||||
"slack": "app.channels.slack:SlackChannel",
|
||||
"telegram": "app.channels.telegram:TelegramChannel",
|
||||
"wechat": "app.channels.wechat:WechatChannel",
|
||||
"wecom": "app.channels.wecom:WeComChannel",
|
||||
}
|
||||
|
||||
_CHANNELS_LANGGRAPH_URL_ENV = "DEER_FLOW_CHANNELS_LANGGRAPH_URL"
|
||||
_CHANNELS_GATEWAY_URL_ENV = "DEER_FLOW_CHANNELS_GATEWAY_URL"
|
||||
|
||||
|
||||
def _resolve_service_url(config: dict[str, Any], config_key: str, env_key: str, default: str) -> str:
|
||||
value = config.pop(config_key, None)
|
||||
if isinstance(value, str) and value.strip():
|
||||
return value
|
||||
env_value = os.getenv(env_key, "").strip()
|
||||
if env_value:
|
||||
return env_value
|
||||
return default
|
||||
|
||||
|
||||
class ChannelService:
|
||||
"""Manages the lifecycle of all configured IM channels.
|
||||
|
||||
Reads configuration from ``config.yaml`` under the ``channels`` key,
|
||||
instantiates enabled channels, and starts the ChannelManager dispatcher.
|
||||
"""
|
||||
|
||||
def __init__(self, channels_config: dict[str, Any] | None = None) -> None:
|
||||
self.bus = MessageBus()
|
||||
self.store = ChannelStore()
|
||||
config = dict(channels_config or {})
|
||||
langgraph_url = _resolve_service_url(config, "langgraph_url", _CHANNELS_LANGGRAPH_URL_ENV, DEFAULT_LANGGRAPH_URL)
|
||||
gateway_url = _resolve_service_url(config, "gateway_url", _CHANNELS_GATEWAY_URL_ENV, DEFAULT_GATEWAY_URL)
|
||||
default_session = config.pop("session", None)
|
||||
channel_sessions = {name: channel_config.get("session") for name, channel_config in config.items() if isinstance(channel_config, dict)}
|
||||
self.manager = ChannelManager(
|
||||
bus=self.bus,
|
||||
store=self.store,
|
||||
langgraph_url=langgraph_url,
|
||||
gateway_url=gateway_url,
|
||||
default_session=default_session if isinstance(default_session, dict) else None,
|
||||
channel_sessions=channel_sessions,
|
||||
)
|
||||
self._channels: dict[str, Any] = {} # name -> Channel instance
|
||||
self._config = config
|
||||
self._running = False
|
||||
|
||||
@classmethod
|
||||
def from_app_config(cls) -> ChannelService:
|
||||
"""Create a ChannelService from the application config."""
|
||||
from deerflow.config.app_config import get_app_config
|
||||
|
||||
config = get_app_config()
|
||||
channels_config = {}
|
||||
# extra fields are allowed by AppConfig (extra="allow")
|
||||
extra = config.model_extra or {}
|
||||
if "channels" in extra:
|
||||
channels_config = extra["channels"]
|
||||
return cls(channels_config=channels_config)
|
||||
|
||||
async def start(self) -> None:
|
||||
"""Start the manager and all enabled channels."""
|
||||
if self._running:
|
||||
return
|
||||
|
||||
await self.manager.start()
|
||||
|
||||
for name, channel_config in self._config.items():
|
||||
if not isinstance(channel_config, dict):
|
||||
continue
|
||||
if not channel_config.get("enabled", False):
|
||||
logger.info("Channel %s is disabled, skipping", name)
|
||||
continue
|
||||
|
||||
await self._start_channel(name, channel_config)
|
||||
|
||||
self._running = True
|
||||
logger.info("ChannelService started with channels: %s", list(self._channels.keys()))
|
||||
|
||||
async def stop(self) -> None:
|
||||
"""Stop all channels and the manager."""
|
||||
for name, channel in list(self._channels.items()):
|
||||
try:
|
||||
await channel.stop()
|
||||
logger.info("Channel %s stopped", name)
|
||||
except Exception:
|
||||
logger.exception("Error stopping channel %s", name)
|
||||
self._channels.clear()
|
||||
|
||||
await self.manager.stop()
|
||||
self._running = False
|
||||
logger.info("ChannelService stopped")
|
||||
|
||||
async def restart_channel(self, name: str) -> bool:
|
||||
"""Restart a specific channel. Returns True if successful."""
|
||||
if name in self._channels:
|
||||
try:
|
||||
await self._channels[name].stop()
|
||||
except Exception:
|
||||
logger.exception("Error stopping channel %s for restart", name)
|
||||
del self._channels[name]
|
||||
|
||||
config = self._config.get(name)
|
||||
if not config or not isinstance(config, dict):
|
||||
logger.warning("No config for channel %s", name)
|
||||
return False
|
||||
|
||||
return await self._start_channel(name, config)
|
||||
|
||||
async def _start_channel(self, name: str, config: dict[str, Any]) -> bool:
|
||||
"""Instantiate and start a single channel."""
|
||||
import_path = _CHANNEL_REGISTRY.get(name)
|
||||
if not import_path:
|
||||
logger.warning("Unknown channel type: %s", name)
|
||||
return False
|
||||
|
||||
try:
|
||||
from deerflow.reflection import resolve_class
|
||||
|
||||
channel_cls = resolve_class(import_path, base_class=None)
|
||||
except Exception:
|
||||
logger.exception("Failed to import channel class for %s", name)
|
||||
return False
|
||||
|
||||
try:
|
||||
channel = channel_cls(bus=self.bus, config=config)
|
||||
await channel.start()
|
||||
self._channels[name] = channel
|
||||
logger.info("Channel %s started", name)
|
||||
return True
|
||||
except Exception:
|
||||
logger.exception("Failed to start channel %s", name)
|
||||
return False
|
||||
|
||||
def get_status(self) -> dict[str, Any]:
|
||||
"""Return status information for all channels."""
|
||||
channels_status = {}
|
||||
for name in _CHANNEL_REGISTRY:
|
||||
config = self._config.get(name, {})
|
||||
enabled = isinstance(config, dict) and config.get("enabled", False)
|
||||
running = name in self._channels and self._channels[name].is_running
|
||||
channels_status[name] = {
|
||||
"enabled": enabled,
|
||||
"running": running,
|
||||
}
|
||||
return {
|
||||
"service_running": self._running,
|
||||
"channels": channels_status,
|
||||
}
|
||||
|
||||
def get_channel(self, name: str) -> Channel | None:
|
||||
"""Return a running channel instance by name when available."""
|
||||
return self._channels.get(name)
|
||||
|
||||
|
||||
# -- singleton access -------------------------------------------------------
|
||||
|
||||
_channel_service: ChannelService | None = None
|
||||
|
||||
|
||||
def get_channel_service() -> ChannelService | None:
|
||||
"""Get the singleton ChannelService instance (if started)."""
|
||||
return _channel_service
|
||||
|
||||
|
||||
async def start_channel_service() -> ChannelService:
|
||||
"""Create and start the global ChannelService from app config."""
|
||||
global _channel_service
|
||||
if _channel_service is not None:
|
||||
return _channel_service
|
||||
_channel_service = ChannelService.from_app_config()
|
||||
await _channel_service.start()
|
||||
return _channel_service
|
||||
|
||||
|
||||
async def stop_channel_service() -> None:
|
||||
"""Stop the global ChannelService."""
|
||||
global _channel_service
|
||||
if _channel_service is not None:
|
||||
await _channel_service.stop()
|
||||
_channel_service = None
|
||||
@@ -1,246 +0,0 @@
|
||||
"""Slack channel — connects via Socket Mode (no public IP needed)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from markdown_to_mrkdwn import SlackMarkdownConverter
|
||||
|
||||
from app.channels.base import Channel
|
||||
from app.channels.message_bus import InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_slack_md_converter = SlackMarkdownConverter()
|
||||
|
||||
|
||||
class SlackChannel(Channel):
|
||||
"""Slack IM channel using Socket Mode (WebSocket, no public IP).
|
||||
|
||||
Configuration keys (in ``config.yaml`` under ``channels.slack``):
|
||||
- ``bot_token``: Slack Bot User OAuth Token (xoxb-...).
|
||||
- ``app_token``: Slack App-Level Token (xapp-...) for Socket Mode.
|
||||
- ``allowed_users``: (optional) List of allowed Slack user IDs. Empty = allow all.
|
||||
"""
|
||||
|
||||
def __init__(self, bus: MessageBus, config: dict[str, Any]) -> None:
|
||||
super().__init__(name="slack", bus=bus, config=config)
|
||||
self._socket_client = None
|
||||
self._web_client = None
|
||||
self._loop: asyncio.AbstractEventLoop | None = None
|
||||
self._allowed_users: set[str] = {str(user_id) for user_id in config.get("allowed_users", [])}
|
||||
|
||||
async def start(self) -> None:
|
||||
if self._running:
|
||||
return
|
||||
|
||||
try:
|
||||
from slack_sdk import WebClient
|
||||
from slack_sdk.socket_mode import SocketModeClient
|
||||
from slack_sdk.socket_mode.response import SocketModeResponse
|
||||
except ImportError:
|
||||
logger.error("slack-sdk is not installed. Install it with: uv add slack-sdk")
|
||||
return
|
||||
|
||||
self._SocketModeResponse = SocketModeResponse
|
||||
|
||||
bot_token = self.config.get("bot_token", "")
|
||||
app_token = self.config.get("app_token", "")
|
||||
|
||||
if not bot_token or not app_token:
|
||||
logger.error("Slack channel requires bot_token and app_token")
|
||||
return
|
||||
|
||||
self._web_client = WebClient(token=bot_token)
|
||||
self._socket_client = SocketModeClient(
|
||||
app_token=app_token,
|
||||
web_client=self._web_client,
|
||||
)
|
||||
self._loop = asyncio.get_event_loop()
|
||||
|
||||
self._socket_client.socket_mode_request_listeners.append(self._on_socket_event)
|
||||
|
||||
self._running = True
|
||||
self.bus.subscribe_outbound(self._on_outbound)
|
||||
|
||||
# Start socket mode in background thread
|
||||
asyncio.get_event_loop().run_in_executor(None, self._socket_client.connect)
|
||||
logger.info("Slack channel started")
|
||||
|
||||
async def stop(self) -> None:
|
||||
self._running = False
|
||||
self.bus.unsubscribe_outbound(self._on_outbound)
|
||||
if self._socket_client:
|
||||
self._socket_client.close()
|
||||
self._socket_client = None
|
||||
logger.info("Slack channel stopped")
|
||||
|
||||
async def send(self, msg: OutboundMessage, *, _max_retries: int = 3) -> None:
|
||||
if not self._web_client:
|
||||
return
|
||||
|
||||
kwargs: dict[str, Any] = {
|
||||
"channel": msg.chat_id,
|
||||
"text": _slack_md_converter.convert(msg.text),
|
||||
}
|
||||
if msg.thread_ts:
|
||||
kwargs["thread_ts"] = msg.thread_ts
|
||||
|
||||
last_exc: Exception | None = None
|
||||
for attempt in range(_max_retries):
|
||||
try:
|
||||
await asyncio.to_thread(self._web_client.chat_postMessage, **kwargs)
|
||||
# Add a completion reaction to the thread root
|
||||
if msg.thread_ts:
|
||||
await asyncio.to_thread(
|
||||
self._add_reaction,
|
||||
msg.chat_id,
|
||||
msg.thread_ts,
|
||||
"white_check_mark",
|
||||
)
|
||||
return
|
||||
except Exception as exc:
|
||||
last_exc = exc
|
||||
if attempt < _max_retries - 1:
|
||||
delay = 2**attempt # 1s, 2s
|
||||
logger.warning(
|
||||
"[Slack] send failed (attempt %d/%d), retrying in %ds: %s",
|
||||
attempt + 1,
|
||||
_max_retries,
|
||||
delay,
|
||||
exc,
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
|
||||
logger.error("[Slack] send failed after %d attempts: %s", _max_retries, last_exc)
|
||||
# Add failure reaction on error
|
||||
if msg.thread_ts:
|
||||
try:
|
||||
await asyncio.to_thread(
|
||||
self._add_reaction,
|
||||
msg.chat_id,
|
||||
msg.thread_ts,
|
||||
"x",
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
if last_exc is None:
|
||||
raise RuntimeError("Slack send failed without an exception from any attempt")
|
||||
raise last_exc
|
||||
|
||||
async def send_file(self, msg: OutboundMessage, attachment: ResolvedAttachment) -> bool:
|
||||
if not self._web_client:
|
||||
return False
|
||||
|
||||
try:
|
||||
kwargs: dict[str, Any] = {
|
||||
"channel": msg.chat_id,
|
||||
"file": str(attachment.actual_path),
|
||||
"filename": attachment.filename,
|
||||
"title": attachment.filename,
|
||||
}
|
||||
if msg.thread_ts:
|
||||
kwargs["thread_ts"] = msg.thread_ts
|
||||
|
||||
await asyncio.to_thread(self._web_client.files_upload_v2, **kwargs)
|
||||
logger.info("[Slack] file uploaded: %s to channel=%s", attachment.filename, msg.chat_id)
|
||||
return True
|
||||
except Exception:
|
||||
logger.exception("[Slack] failed to upload file: %s", attachment.filename)
|
||||
return False
|
||||
|
||||
# -- internal ----------------------------------------------------------
|
||||
|
||||
def _add_reaction(self, channel_id: str, timestamp: str, emoji: str) -> None:
|
||||
"""Add an emoji reaction to a message (best-effort, non-blocking)."""
|
||||
if not self._web_client:
|
||||
return
|
||||
try:
|
||||
self._web_client.reactions_add(
|
||||
channel=channel_id,
|
||||
timestamp=timestamp,
|
||||
name=emoji,
|
||||
)
|
||||
except Exception as exc:
|
||||
if "already_reacted" not in str(exc):
|
||||
logger.warning("[Slack] failed to add reaction %s: %s", emoji, exc)
|
||||
|
||||
def _send_running_reply(self, channel_id: str, thread_ts: str) -> None:
|
||||
"""Send a 'Working on it......' reply in the thread (called from SDK thread)."""
|
||||
if not self._web_client:
|
||||
return
|
||||
try:
|
||||
self._web_client.chat_postMessage(
|
||||
channel=channel_id,
|
||||
text=":hourglass_flowing_sand: Working on it...",
|
||||
thread_ts=thread_ts,
|
||||
)
|
||||
logger.info("[Slack] 'Working on it...' reply sent in channel=%s, thread_ts=%s", channel_id, thread_ts)
|
||||
except Exception:
|
||||
logger.exception("[Slack] failed to send running reply in channel=%s", channel_id)
|
||||
|
||||
def _on_socket_event(self, client, req) -> None:
|
||||
"""Called by slack-sdk for each Socket Mode event."""
|
||||
try:
|
||||
# Acknowledge the event
|
||||
response = self._SocketModeResponse(envelope_id=req.envelope_id)
|
||||
client.send_socket_mode_response(response)
|
||||
|
||||
event_type = req.type
|
||||
if event_type != "events_api":
|
||||
return
|
||||
|
||||
event = req.payload.get("event", {})
|
||||
etype = event.get("type", "")
|
||||
|
||||
# Handle message events (DM or @mention)
|
||||
if etype in ("message", "app_mention"):
|
||||
self._handle_message_event(event)
|
||||
|
||||
except Exception:
|
||||
logger.exception("Error processing Slack event")
|
||||
|
||||
def _handle_message_event(self, event: dict) -> None:
|
||||
# Ignore bot messages
|
||||
if event.get("bot_id") or event.get("subtype"):
|
||||
return
|
||||
|
||||
user_id = event.get("user", "")
|
||||
|
||||
# Check allowed users
|
||||
if self._allowed_users and user_id not in self._allowed_users:
|
||||
logger.debug("Ignoring message from non-allowed user: %s", user_id)
|
||||
return
|
||||
|
||||
text = event.get("text", "").strip()
|
||||
if not text:
|
||||
return
|
||||
|
||||
channel_id = event.get("channel", "")
|
||||
thread_ts = event.get("thread_ts") or event.get("ts", "")
|
||||
|
||||
if text.startswith("/"):
|
||||
msg_type = InboundMessageType.COMMAND
|
||||
else:
|
||||
msg_type = InboundMessageType.CHAT
|
||||
|
||||
# topic_id: use thread_ts as the topic identifier.
|
||||
# For threaded messages, thread_ts is the root message ts (shared topic).
|
||||
# For non-threaded messages, thread_ts is the message's own ts (new topic).
|
||||
inbound = self._make_inbound(
|
||||
chat_id=channel_id,
|
||||
user_id=user_id,
|
||||
text=text,
|
||||
msg_type=msg_type,
|
||||
thread_ts=thread_ts,
|
||||
)
|
||||
inbound.topic_id = thread_ts
|
||||
|
||||
if self._loop and self._loop.is_running():
|
||||
# Acknowledge with an eyes reaction
|
||||
self._add_reaction(channel_id, event.get("ts", thread_ts), "eyes")
|
||||
# Send "running" reply first (fire-and-forget from SDK thread)
|
||||
self._send_running_reply(channel_id, thread_ts)
|
||||
asyncio.run_coroutine_threadsafe(self.bus.publish_inbound(inbound), self._loop)
|
||||
@@ -1,153 +0,0 @@
|
||||
"""ChannelStore — persists IM chat-to-DeerFlow thread mappings."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class ChannelStore:
|
||||
"""JSON-file-backed store that maps IM conversations to DeerFlow threads.
|
||||
|
||||
Data layout (on disk)::
|
||||
|
||||
{
|
||||
"<channel_name>:<chat_id>": {
|
||||
"thread_id": "<uuid>",
|
||||
"user_id": "<platform_user>",
|
||||
"created_at": 1700000000.0,
|
||||
"updated_at": 1700000000.0
|
||||
},
|
||||
...
|
||||
}
|
||||
|
||||
The store is intentionally simple — a single JSON file that is atomically
|
||||
rewritten on every mutation. For production workloads with high concurrency,
|
||||
this can be swapped for a proper database backend.
|
||||
"""
|
||||
|
||||
def __init__(self, path: str | Path | None = None) -> None:
|
||||
if path is None:
|
||||
from deerflow.config.paths import get_paths
|
||||
|
||||
path = Path(get_paths().base_dir) / "channels" / "store.json"
|
||||
self._path = Path(path)
|
||||
self._path.parent.mkdir(parents=True, exist_ok=True)
|
||||
self._data: dict[str, dict[str, Any]] = self._load()
|
||||
self._lock = threading.Lock()
|
||||
|
||||
# -- persistence -------------------------------------------------------
|
||||
|
||||
def _load(self) -> dict[str, dict[str, Any]]:
|
||||
if self._path.exists():
|
||||
try:
|
||||
return json.loads(self._path.read_text(encoding="utf-8"))
|
||||
except (json.JSONDecodeError, OSError):
|
||||
logger.warning("Corrupt channel store at %s, starting fresh", self._path)
|
||||
return {}
|
||||
|
||||
def _save(self) -> None:
|
||||
fd = tempfile.NamedTemporaryFile(
|
||||
mode="w",
|
||||
dir=self._path.parent,
|
||||
suffix=".tmp",
|
||||
delete=False,
|
||||
)
|
||||
try:
|
||||
json.dump(self._data, fd, indent=2)
|
||||
fd.close()
|
||||
Path(fd.name).replace(self._path)
|
||||
except BaseException:
|
||||
fd.close()
|
||||
Path(fd.name).unlink(missing_ok=True)
|
||||
raise
|
||||
|
||||
# -- key helpers -------------------------------------------------------
|
||||
|
||||
@staticmethod
|
||||
def _key(channel_name: str, chat_id: str, topic_id: str | None = None) -> str:
|
||||
if topic_id:
|
||||
return f"{channel_name}:{chat_id}:{topic_id}"
|
||||
return f"{channel_name}:{chat_id}"
|
||||
|
||||
# -- public API --------------------------------------------------------
|
||||
|
||||
def get_thread_id(self, channel_name: str, chat_id: str, topic_id: str | None = None) -> str | None:
|
||||
"""Look up the DeerFlow thread_id for a given IM conversation/topic."""
|
||||
entry = self._data.get(self._key(channel_name, chat_id, topic_id))
|
||||
return entry["thread_id"] if entry else None
|
||||
|
||||
def set_thread_id(
|
||||
self,
|
||||
channel_name: str,
|
||||
chat_id: str,
|
||||
thread_id: str,
|
||||
*,
|
||||
topic_id: str | None = None,
|
||||
user_id: str = "",
|
||||
) -> None:
|
||||
"""Create or update the mapping for an IM conversation/topic."""
|
||||
with self._lock:
|
||||
key = self._key(channel_name, chat_id, topic_id)
|
||||
now = time.time()
|
||||
existing = self._data.get(key)
|
||||
self._data[key] = {
|
||||
"thread_id": thread_id,
|
||||
"user_id": user_id,
|
||||
"created_at": existing["created_at"] if existing else now,
|
||||
"updated_at": now,
|
||||
}
|
||||
self._save()
|
||||
|
||||
def remove(self, channel_name: str, chat_id: str, topic_id: str | None = None) -> bool:
|
||||
"""Remove a mapping.
|
||||
|
||||
If ``topic_id`` is provided, only that specific conversation/topic mapping is removed.
|
||||
If ``topic_id`` is omitted, all mappings whose key starts with
|
||||
``"<channel_name>:<chat_id>"`` (including topic-specific ones) are removed.
|
||||
|
||||
Returns True if at least one mapping was removed.
|
||||
"""
|
||||
with self._lock:
|
||||
# Remove a specific conversation/topic mapping.
|
||||
if topic_id is not None:
|
||||
key = self._key(channel_name, chat_id, topic_id)
|
||||
if key in self._data:
|
||||
del self._data[key]
|
||||
self._save()
|
||||
return True
|
||||
return False
|
||||
|
||||
# Remove all mappings for this channel/chat_id (base and any topic-specific keys).
|
||||
prefix = self._key(channel_name, chat_id)
|
||||
keys_to_delete = [k for k in self._data if k == prefix or k.startswith(prefix + ":")]
|
||||
if not keys_to_delete:
|
||||
return False
|
||||
|
||||
for k in keys_to_delete:
|
||||
del self._data[k]
|
||||
self._save()
|
||||
return True
|
||||
|
||||
def list_entries(self, channel_name: str | None = None) -> list[dict[str, Any]]:
|
||||
"""List all stored mappings, optionally filtered by channel."""
|
||||
results = []
|
||||
for key, entry in self._data.items():
|
||||
parts = key.split(":", 2)
|
||||
ch = parts[0]
|
||||
chat = parts[1] if len(parts) > 1 else ""
|
||||
topic = parts[2] if len(parts) > 2 else None
|
||||
if channel_name and ch != channel_name:
|
||||
continue
|
||||
item: dict[str, Any] = {"channel_name": ch, "chat_id": chat, **entry}
|
||||
if topic is not None:
|
||||
item["topic_id"] = topic
|
||||
results.append(item)
|
||||
return results
|
||||
@@ -1,317 +0,0 @@
|
||||
"""Telegram channel — connects via long-polling (no public IP needed)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import threading
|
||||
from typing import Any
|
||||
|
||||
from app.channels.base import Channel
|
||||
from app.channels.message_bus import InboundMessage, InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class TelegramChannel(Channel):
|
||||
"""Telegram bot channel using long-polling.
|
||||
|
||||
Configuration keys (in ``config.yaml`` under ``channels.telegram``):
|
||||
- ``bot_token``: Telegram Bot API token (from @BotFather).
|
||||
- ``allowed_users``: (optional) List of allowed Telegram user IDs. Empty = allow all.
|
||||
"""
|
||||
|
||||
def __init__(self, bus: MessageBus, config: dict[str, Any]) -> None:
|
||||
super().__init__(name="telegram", bus=bus, config=config)
|
||||
self._application = None
|
||||
self._thread: threading.Thread | None = None
|
||||
self._tg_loop: asyncio.AbstractEventLoop | None = None
|
||||
self._main_loop: asyncio.AbstractEventLoop | None = None
|
||||
self._allowed_users: set[int] = set()
|
||||
for uid in config.get("allowed_users", []):
|
||||
try:
|
||||
self._allowed_users.add(int(uid))
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
# chat_id -> last sent message_id for threaded replies
|
||||
self._last_bot_message: dict[str, int] = {}
|
||||
|
||||
async def start(self) -> None:
|
||||
if self._running:
|
||||
return
|
||||
|
||||
try:
|
||||
from telegram.ext import ApplicationBuilder, CommandHandler, MessageHandler, filters
|
||||
except ImportError:
|
||||
logger.error("python-telegram-bot is not installed. Install it with: uv add python-telegram-bot")
|
||||
return
|
||||
|
||||
bot_token = self.config.get("bot_token", "")
|
||||
if not bot_token:
|
||||
logger.error("Telegram channel requires bot_token")
|
||||
return
|
||||
|
||||
self._main_loop = asyncio.get_event_loop()
|
||||
self._running = True
|
||||
self.bus.subscribe_outbound(self._on_outbound)
|
||||
|
||||
# Build the application
|
||||
app = ApplicationBuilder().token(bot_token).build()
|
||||
|
||||
# Command handlers
|
||||
app.add_handler(CommandHandler("start", self._cmd_start))
|
||||
app.add_handler(CommandHandler("new", self._cmd_generic))
|
||||
app.add_handler(CommandHandler("status", self._cmd_generic))
|
||||
app.add_handler(CommandHandler("models", self._cmd_generic))
|
||||
app.add_handler(CommandHandler("memory", self._cmd_generic))
|
||||
app.add_handler(CommandHandler("help", self._cmd_generic))
|
||||
|
||||
# General message handler
|
||||
app.add_handler(MessageHandler(filters.TEXT & ~filters.COMMAND, self._on_text))
|
||||
|
||||
self._application = app
|
||||
|
||||
# Run polling in a dedicated thread with its own event loop
|
||||
self._thread = threading.Thread(target=self._run_polling, daemon=True)
|
||||
self._thread.start()
|
||||
logger.info("Telegram channel started")
|
||||
|
||||
async def stop(self) -> None:
|
||||
self._running = False
|
||||
self.bus.unsubscribe_outbound(self._on_outbound)
|
||||
if self._tg_loop and self._tg_loop.is_running():
|
||||
self._tg_loop.call_soon_threadsafe(self._tg_loop.stop)
|
||||
if self._thread:
|
||||
self._thread.join(timeout=10)
|
||||
self._thread = None
|
||||
self._application = None
|
||||
logger.info("Telegram channel stopped")
|
||||
|
||||
async def send(self, msg: OutboundMessage, *, _max_retries: int = 3) -> None:
|
||||
if not self._application:
|
||||
return
|
||||
|
||||
try:
|
||||
chat_id = int(msg.chat_id)
|
||||
except (ValueError, TypeError):
|
||||
logger.error("Invalid Telegram chat_id: %s", msg.chat_id)
|
||||
return
|
||||
|
||||
kwargs: dict[str, Any] = {"chat_id": chat_id, "text": msg.text}
|
||||
|
||||
# Reply to the last bot message in this chat for threading
|
||||
reply_to = self._last_bot_message.get(msg.chat_id)
|
||||
if reply_to:
|
||||
kwargs["reply_to_message_id"] = reply_to
|
||||
|
||||
bot = self._application.bot
|
||||
last_exc: Exception | None = None
|
||||
for attempt in range(_max_retries):
|
||||
try:
|
||||
sent = await bot.send_message(**kwargs)
|
||||
self._last_bot_message[msg.chat_id] = sent.message_id
|
||||
return
|
||||
except Exception as exc:
|
||||
last_exc = exc
|
||||
if attempt < _max_retries - 1:
|
||||
delay = 2**attempt # 1s, 2s
|
||||
logger.warning(
|
||||
"[Telegram] send failed (attempt %d/%d), retrying in %ds: %s",
|
||||
attempt + 1,
|
||||
_max_retries,
|
||||
delay,
|
||||
exc,
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
|
||||
logger.error("[Telegram] send failed after %d attempts: %s", _max_retries, last_exc)
|
||||
if last_exc is None:
|
||||
raise RuntimeError("Telegram send failed without an exception from any attempt")
|
||||
raise last_exc
|
||||
|
||||
async def send_file(self, msg: OutboundMessage, attachment: ResolvedAttachment) -> bool:
|
||||
if not self._application:
|
||||
return False
|
||||
|
||||
try:
|
||||
chat_id = int(msg.chat_id)
|
||||
except (ValueError, TypeError):
|
||||
logger.error("[Telegram] Invalid chat_id: %s", msg.chat_id)
|
||||
return False
|
||||
|
||||
# Telegram limits: 10MB for photos, 50MB for documents
|
||||
if attachment.size > 50 * 1024 * 1024:
|
||||
logger.warning("[Telegram] file too large (%d bytes), skipping: %s", attachment.size, attachment.filename)
|
||||
return False
|
||||
|
||||
bot = self._application.bot
|
||||
reply_to = self._last_bot_message.get(msg.chat_id)
|
||||
|
||||
try:
|
||||
if attachment.is_image and attachment.size <= 10 * 1024 * 1024:
|
||||
with open(attachment.actual_path, "rb") as f:
|
||||
kwargs: dict[str, Any] = {"chat_id": chat_id, "photo": f}
|
||||
if reply_to:
|
||||
kwargs["reply_to_message_id"] = reply_to
|
||||
sent = await bot.send_photo(**kwargs)
|
||||
else:
|
||||
from telegram import InputFile
|
||||
|
||||
with open(attachment.actual_path, "rb") as f:
|
||||
input_file = InputFile(f, filename=attachment.filename)
|
||||
kwargs = {"chat_id": chat_id, "document": input_file}
|
||||
if reply_to:
|
||||
kwargs["reply_to_message_id"] = reply_to
|
||||
sent = await bot.send_document(**kwargs)
|
||||
|
||||
self._last_bot_message[msg.chat_id] = sent.message_id
|
||||
logger.info("[Telegram] file sent: %s to chat=%s", attachment.filename, msg.chat_id)
|
||||
return True
|
||||
except Exception:
|
||||
logger.exception("[Telegram] failed to send file: %s", attachment.filename)
|
||||
return False
|
||||
|
||||
# -- helpers -----------------------------------------------------------
|
||||
|
||||
async def _send_running_reply(self, chat_id: str, reply_to_message_id: int) -> None:
|
||||
"""Send a 'Working on it...' reply to the user's message."""
|
||||
if not self._application:
|
||||
return
|
||||
try:
|
||||
bot = self._application.bot
|
||||
await bot.send_message(
|
||||
chat_id=int(chat_id),
|
||||
text="Working on it...",
|
||||
reply_to_message_id=reply_to_message_id,
|
||||
)
|
||||
logger.info("[Telegram] 'Working on it...' reply sent in chat=%s", chat_id)
|
||||
except Exception:
|
||||
logger.exception("[Telegram] failed to send running reply in chat=%s", chat_id)
|
||||
|
||||
# -- internal ----------------------------------------------------------
|
||||
@staticmethod
|
||||
def _log_future_error(fut, name: str, msg_id: str):
|
||||
try:
|
||||
exc = fut.exception()
|
||||
if exc:
|
||||
logger.error("[Telegram] %s failed for msg_id=%s: %s", name, msg_id, exc)
|
||||
except Exception:
|
||||
logger.exception("[Telegram] Failed to inspect future for %s (msg_id=%s)", name, msg_id)
|
||||
|
||||
def _run_polling(self) -> None:
|
||||
"""Run telegram polling in a dedicated thread."""
|
||||
self._tg_loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(self._tg_loop)
|
||||
try:
|
||||
# Cannot use run_polling() because it calls add_signal_handler(),
|
||||
# which only works in the main thread. Instead, manually
|
||||
# initialize the application and start the updater.
|
||||
self._tg_loop.run_until_complete(self._application.initialize())
|
||||
self._tg_loop.run_until_complete(self._application.start())
|
||||
self._tg_loop.run_until_complete(self._application.updater.start_polling())
|
||||
self._tg_loop.run_forever()
|
||||
except Exception:
|
||||
if self._running:
|
||||
logger.exception("Telegram polling error")
|
||||
finally:
|
||||
# Graceful shutdown
|
||||
try:
|
||||
if self._application.updater.running:
|
||||
self._tg_loop.run_until_complete(self._application.updater.stop())
|
||||
self._tg_loop.run_until_complete(self._application.stop())
|
||||
self._tg_loop.run_until_complete(self._application.shutdown())
|
||||
except Exception:
|
||||
logger.exception("Error during Telegram shutdown")
|
||||
|
||||
def _check_user(self, user_id: int) -> bool:
|
||||
if not self._allowed_users:
|
||||
return True
|
||||
return user_id in self._allowed_users
|
||||
|
||||
async def _cmd_start(self, update, context) -> None:
|
||||
"""Handle /start command."""
|
||||
if not self._check_user(update.effective_user.id):
|
||||
return
|
||||
await update.message.reply_text("Welcome to DeerFlow! Send me a message to start a conversation.\nType /help for available commands.")
|
||||
|
||||
async def _process_incoming_with_reply(self, chat_id: str, msg_id: int, inbound: InboundMessage) -> None:
|
||||
await self._send_running_reply(chat_id, msg_id)
|
||||
await self.bus.publish_inbound(inbound)
|
||||
|
||||
async def _cmd_generic(self, update, context) -> None:
|
||||
"""Forward slash commands to the channel manager."""
|
||||
if not self._check_user(update.effective_user.id):
|
||||
return
|
||||
|
||||
text = update.message.text
|
||||
chat_id = str(update.effective_chat.id)
|
||||
user_id = str(update.effective_user.id)
|
||||
msg_id = str(update.message.message_id)
|
||||
|
||||
# Use the same topic_id logic as _on_text so that commands
|
||||
# like /new target the correct thread mapping.
|
||||
if update.effective_chat.type == "private":
|
||||
topic_id = None
|
||||
else:
|
||||
reply_to = update.message.reply_to_message
|
||||
if reply_to:
|
||||
topic_id = str(reply_to.message_id)
|
||||
else:
|
||||
topic_id = msg_id
|
||||
|
||||
inbound = self._make_inbound(
|
||||
chat_id=chat_id,
|
||||
user_id=user_id,
|
||||
text=text,
|
||||
msg_type=InboundMessageType.COMMAND,
|
||||
thread_ts=msg_id,
|
||||
)
|
||||
inbound.topic_id = topic_id
|
||||
|
||||
if self._main_loop and self._main_loop.is_running():
|
||||
fut = asyncio.run_coroutine_threadsafe(self._process_incoming_with_reply(chat_id, update.message.message_id, inbound), self._main_loop)
|
||||
fut.add_done_callback(lambda f: self._log_future_error(f, "process_incoming_with_reply", update.message.message_id))
|
||||
else:
|
||||
logger.warning("[Telegram] Main loop not running. Cannot publish inbound message.")
|
||||
|
||||
async def _on_text(self, update, context) -> None:
|
||||
"""Handle regular text messages."""
|
||||
if not self._check_user(update.effective_user.id):
|
||||
return
|
||||
|
||||
text = update.message.text.strip()
|
||||
if not text:
|
||||
return
|
||||
|
||||
chat_id = str(update.effective_chat.id)
|
||||
user_id = str(update.effective_user.id)
|
||||
msg_id = str(update.message.message_id)
|
||||
|
||||
# topic_id determines which DeerFlow thread the message maps to.
|
||||
# In private chats, use None so that all messages share a single
|
||||
# thread (the store key becomes "channel:chat_id").
|
||||
# In group chats, use the reply-to message id or the current
|
||||
# message id to keep separate conversation threads.
|
||||
if update.effective_chat.type == "private":
|
||||
topic_id = None
|
||||
else:
|
||||
reply_to = update.message.reply_to_message
|
||||
if reply_to:
|
||||
topic_id = str(reply_to.message_id)
|
||||
else:
|
||||
topic_id = msg_id
|
||||
|
||||
inbound = self._make_inbound(
|
||||
chat_id=chat_id,
|
||||
user_id=user_id,
|
||||
text=text,
|
||||
msg_type=InboundMessageType.CHAT,
|
||||
thread_ts=msg_id,
|
||||
)
|
||||
inbound.topic_id = topic_id
|
||||
|
||||
if self._main_loop and self._main_loop.is_running():
|
||||
fut = asyncio.run_coroutine_threadsafe(self._process_incoming_with_reply(chat_id, update.message.message_id, inbound), self._main_loop)
|
||||
fut.add_done_callback(lambda f: self._log_future_error(f, "process_incoming_with_reply", update.message.message_id))
|
||||
else:
|
||||
logger.warning("[Telegram] Main loop not running. Cannot publish inbound message.")
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,394 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import hashlib
|
||||
import logging
|
||||
from collections.abc import Awaitable, Callable
|
||||
from typing import Any, cast
|
||||
|
||||
from app.channels.base import Channel
|
||||
from app.channels.message_bus import (
|
||||
InboundMessageType,
|
||||
MessageBus,
|
||||
OutboundMessage,
|
||||
ResolvedAttachment,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class WeComChannel(Channel):
|
||||
def __init__(self, bus: MessageBus, config: dict[str, Any]) -> None:
|
||||
super().__init__(name="wecom", bus=bus, config=config)
|
||||
self._bot_id: str | None = None
|
||||
self._bot_secret: str | None = None
|
||||
self._ws_client = None
|
||||
self._ws_task: asyncio.Task | None = None
|
||||
self._ws_frames: dict[str, dict[str, Any]] = {}
|
||||
self._ws_stream_ids: dict[str, str] = {}
|
||||
self._working_message = "Working on it..."
|
||||
|
||||
def _clear_ws_context(self, thread_ts: str | None) -> None:
|
||||
if not thread_ts:
|
||||
return
|
||||
self._ws_frames.pop(thread_ts, None)
|
||||
self._ws_stream_ids.pop(thread_ts, None)
|
||||
|
||||
async def _send_ws_upload_command(self, req_id: str, body: dict[str, Any], cmd: str) -> dict[str, Any]:
|
||||
if not self._ws_client:
|
||||
raise RuntimeError("WeCom WebSocket client is not available")
|
||||
|
||||
ws_manager = getattr(self._ws_client, "_ws_manager", None)
|
||||
send_reply = getattr(ws_manager, "send_reply", None)
|
||||
if not callable(send_reply):
|
||||
raise RuntimeError("Installed wecom-aibot-python-sdk does not expose the WebSocket media upload API expected by DeerFlow. Use wecom-aibot-python-sdk==0.1.6 or update the adapter.")
|
||||
|
||||
send_reply_async = cast(Callable[[str, dict[str, Any], str], Awaitable[dict[str, Any]]], send_reply)
|
||||
return await send_reply_async(req_id, body, cmd)
|
||||
|
||||
async def start(self) -> None:
|
||||
if self._running:
|
||||
return
|
||||
|
||||
bot_id = self.config.get("bot_id")
|
||||
bot_secret = self.config.get("bot_secret")
|
||||
working_message = self.config.get("working_message")
|
||||
|
||||
self._bot_id = bot_id if isinstance(bot_id, str) and bot_id else None
|
||||
self._bot_secret = bot_secret if isinstance(bot_secret, str) and bot_secret else None
|
||||
self._working_message = working_message if isinstance(working_message, str) and working_message else "Working on it..."
|
||||
|
||||
if not self._bot_id or not self._bot_secret:
|
||||
logger.error("WeCom channel requires bot_id and bot_secret")
|
||||
return
|
||||
|
||||
try:
|
||||
from aibot import WSClient, WSClientOptions
|
||||
except ImportError:
|
||||
logger.error("wecom-aibot-python-sdk is not installed. Install it with: uv add wecom-aibot-python-sdk")
|
||||
return
|
||||
else:
|
||||
self._ws_client = WSClient(WSClientOptions(bot_id=self._bot_id, secret=self._bot_secret, logger=logger))
|
||||
self._ws_client.on("message.text", self._on_ws_text)
|
||||
self._ws_client.on("message.mixed", self._on_ws_mixed)
|
||||
self._ws_client.on("message.image", self._on_ws_image)
|
||||
self._ws_client.on("message.file", self._on_ws_file)
|
||||
self._ws_task = asyncio.create_task(self._ws_client.connect())
|
||||
|
||||
self._running = True
|
||||
self.bus.subscribe_outbound(self._on_outbound)
|
||||
logger.info("WeCom channel started")
|
||||
|
||||
async def stop(self) -> None:
|
||||
self._running = False
|
||||
self.bus.unsubscribe_outbound(self._on_outbound)
|
||||
if self._ws_task:
|
||||
try:
|
||||
self._ws_task.cancel()
|
||||
except Exception:
|
||||
pass
|
||||
self._ws_task = None
|
||||
if self._ws_client:
|
||||
try:
|
||||
self._ws_client.disconnect()
|
||||
except Exception:
|
||||
pass
|
||||
self._ws_client = None
|
||||
self._ws_frames.clear()
|
||||
self._ws_stream_ids.clear()
|
||||
logger.info("WeCom channel stopped")
|
||||
|
||||
async def send(self, msg: OutboundMessage, *, _max_retries: int = 3) -> None:
|
||||
if self._ws_client:
|
||||
await self._send_ws(msg, _max_retries=_max_retries)
|
||||
return
|
||||
logger.warning("[WeCom] send called but WebSocket client is not available")
|
||||
|
||||
async def _on_outbound(self, msg: OutboundMessage) -> None:
|
||||
if msg.channel_name != self.name:
|
||||
return
|
||||
|
||||
try:
|
||||
await self.send(msg)
|
||||
except Exception:
|
||||
logger.exception("Failed to send outbound message on channel %s", self.name)
|
||||
if msg.is_final:
|
||||
self._clear_ws_context(msg.thread_ts)
|
||||
return
|
||||
|
||||
for attachment in msg.attachments:
|
||||
try:
|
||||
success = await self.send_file(msg, attachment)
|
||||
if not success:
|
||||
logger.warning("[%s] file upload skipped for %s", self.name, attachment.filename)
|
||||
except Exception:
|
||||
logger.exception("[%s] failed to upload file %s", self.name, attachment.filename)
|
||||
|
||||
if msg.is_final:
|
||||
self._clear_ws_context(msg.thread_ts)
|
||||
|
||||
async def send_file(self, msg: OutboundMessage, attachment: ResolvedAttachment) -> bool:
|
||||
if not msg.is_final:
|
||||
return True
|
||||
if not self._ws_client:
|
||||
return False
|
||||
if not msg.thread_ts:
|
||||
return False
|
||||
frame = self._ws_frames.get(msg.thread_ts)
|
||||
if not frame:
|
||||
return False
|
||||
|
||||
media_type = "image" if attachment.is_image else "file"
|
||||
size_limit = 2 * 1024 * 1024 if attachment.is_image else 20 * 1024 * 1024
|
||||
if attachment.size > size_limit:
|
||||
logger.warning(
|
||||
"[WeCom] %s too large (%d bytes), skipping: %s",
|
||||
media_type,
|
||||
attachment.size,
|
||||
attachment.filename,
|
||||
)
|
||||
return False
|
||||
|
||||
try:
|
||||
media_id = await self._upload_media_ws(
|
||||
media_type=media_type,
|
||||
filename=attachment.filename,
|
||||
path=str(attachment.actual_path),
|
||||
size=attachment.size,
|
||||
)
|
||||
if not media_id:
|
||||
return False
|
||||
|
||||
body = {media_type: {"media_id": media_id}, "msgtype": media_type}
|
||||
await self._ws_client.reply(frame, body)
|
||||
logger.debug("[WeCom] %s sent via ws: %s", media_type, attachment.filename)
|
||||
return True
|
||||
except Exception:
|
||||
logger.exception("[WeCom] failed to upload/send file via ws: %s", attachment.filename)
|
||||
return False
|
||||
|
||||
async def _on_ws_text(self, frame: dict[str, Any]) -> None:
|
||||
body = frame.get("body", {}) or {}
|
||||
text = ((body.get("text") or {}).get("content") or "").strip()
|
||||
quote = body.get("quote", {}).get("text", {}).get("content", "").strip()
|
||||
if not text and not quote:
|
||||
return
|
||||
await self._publish_ws_inbound(frame, text + (f"\nQuote message: {quote}" if quote else ""))
|
||||
|
||||
async def _on_ws_mixed(self, frame: dict[str, Any]) -> None:
|
||||
body = frame.get("body", {}) or {}
|
||||
mixed = body.get("mixed") or {}
|
||||
items = mixed.get("msg_item") or []
|
||||
parts: list[str] = []
|
||||
files: list[dict[str, Any]] = []
|
||||
for item in items:
|
||||
item_type = (item or {}).get("msgtype")
|
||||
if item_type == "text":
|
||||
content = (((item or {}).get("text") or {}).get("content") or "").strip()
|
||||
if content:
|
||||
parts.append(content)
|
||||
elif item_type in ("image", "file"):
|
||||
payload = (item or {}).get(item_type) or {}
|
||||
url = payload.get("url")
|
||||
aeskey = payload.get("aeskey")
|
||||
if isinstance(url, str) and url:
|
||||
files.append(
|
||||
{
|
||||
"type": item_type,
|
||||
"url": url,
|
||||
"aeskey": (aeskey if isinstance(aeskey, str) and aeskey else None),
|
||||
}
|
||||
)
|
||||
text = "\n\n".join(parts).strip()
|
||||
if not text and not files:
|
||||
return
|
||||
if not text:
|
||||
text = "（receive image/file）"
|
||||
await self._publish_ws_inbound(frame, text, files=files)
|
||||
|
||||
async def _on_ws_image(self, frame: dict[str, Any]) -> None:
|
||||
body = frame.get("body", {}) or {}
|
||||
image = body.get("image") or {}
|
||||
url = image.get("url")
|
||||
aeskey = image.get("aeskey")
|
||||
if not isinstance(url, str) or not url:
|
||||
return
|
||||
await self._publish_ws_inbound(
|
||||
frame,
|
||||
"（receive image ）",
|
||||
files=[
|
||||
{
|
||||
"type": "image",
|
||||
"url": url,
|
||||
"aeskey": aeskey if isinstance(aeskey, str) and aeskey else None,
|
||||
}
|
||||
],
|
||||
)
|
||||
|
||||
async def _on_ws_file(self, frame: dict[str, Any]) -> None:
|
||||
body = frame.get("body", {}) or {}
|
||||
file_obj = body.get("file") or {}
|
||||
url = file_obj.get("url")
|
||||
aeskey = file_obj.get("aeskey")
|
||||
if not isinstance(url, str) or not url:
|
||||
return
|
||||
await self._publish_ws_inbound(
|
||||
frame,
|
||||
"（receive file）",
|
||||
files=[
|
||||
{
|
||||
"type": "file",
|
||||
"url": url,
|
||||
"aeskey": aeskey if isinstance(aeskey, str) and aeskey else None,
|
||||
}
|
||||
],
|
||||
)
|
||||
|
||||
async def _publish_ws_inbound(
|
||||
self,
|
||||
frame: dict[str, Any],
|
||||
text: str,
|
||||
*,
|
||||
files: list[dict[str, Any]] | None = None,
|
||||
) -> None:
|
||||
if not self._ws_client:
|
||||
return
|
||||
try:
|
||||
from aibot import generate_req_id
|
||||
except Exception:
|
||||
return
|
||||
|
||||
body = frame.get("body", {}) or {}
|
||||
msg_id = body.get("msgid")
|
||||
if not msg_id:
|
||||
return
|
||||
|
||||
user_id = (body.get("from") or {}).get("userid")
|
||||
|
||||
inbound_type = InboundMessageType.COMMAND if text.startswith("/") else InboundMessageType.CHAT
|
||||
inbound = self._make_inbound(
|
||||
chat_id=user_id, # keep user's conversation in memory
|
||||
user_id=user_id,
|
||||
text=text,
|
||||
msg_type=inbound_type,
|
||||
thread_ts=msg_id,
|
||||
files=files or [],
|
||||
metadata={"aibotid": body.get("aibotid"), "chattype": body.get("chattype")},
|
||||
)
|
||||
inbound.topic_id = user_id # keep the same thread
|
||||
|
||||
stream_id = generate_req_id("stream")
|
||||
self._ws_frames[msg_id] = frame
|
||||
self._ws_stream_ids[msg_id] = stream_id
|
||||
|
||||
try:
|
||||
await self._ws_client.reply_stream(frame, stream_id, self._working_message, False)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
await self.bus.publish_inbound(inbound)
|
||||
|
||||
async def _send_ws(self, msg: OutboundMessage, *, _max_retries: int = 3) -> None:
|
||||
if not self._ws_client:
|
||||
return
|
||||
try:
|
||||
from aibot import generate_req_id
|
||||
except Exception:
|
||||
generate_req_id = None
|
||||
|
||||
if msg.thread_ts and msg.thread_ts in self._ws_frames:
|
||||
frame = self._ws_frames[msg.thread_ts]
|
||||
stream_id = self._ws_stream_ids.get(msg.thread_ts)
|
||||
if not stream_id and generate_req_id:
|
||||
stream_id = generate_req_id("stream")
|
||||
self._ws_stream_ids[msg.thread_ts] = stream_id
|
||||
if not stream_id:
|
||||
return
|
||||
|
||||
last_exc: Exception | None = None
|
||||
for attempt in range(_max_retries):
|
||||
try:
|
||||
await self._ws_client.reply_stream(frame, stream_id, msg.text, bool(msg.is_final))
|
||||
return
|
||||
except Exception as exc:
|
||||
last_exc = exc
|
||||
if attempt < _max_retries - 1:
|
||||
await asyncio.sleep(2**attempt)
|
||||
if last_exc:
|
||||
raise last_exc
|
||||
|
||||
body = {"msgtype": "markdown", "markdown": {"content": msg.text}}
|
||||
last_exc = None
|
||||
for attempt in range(_max_retries):
|
||||
try:
|
||||
await self._ws_client.send_message(msg.chat_id, body)
|
||||
return
|
||||
except Exception as exc:
|
||||
last_exc = exc
|
||||
if attempt < _max_retries - 1:
|
||||
await asyncio.sleep(2**attempt)
|
||||
if last_exc:
|
||||
raise last_exc
|
||||
|
||||
async def _upload_media_ws(
|
||||
self,
|
||||
*,
|
||||
media_type: str,
|
||||
filename: str,
|
||||
path: str,
|
||||
size: int,
|
||||
) -> str | None:
|
||||
if not self._ws_client:
|
||||
return None
|
||||
try:
|
||||
from aibot import generate_req_id
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
chunk_size = 512 * 1024
|
||||
total_chunks = (size + chunk_size - 1) // chunk_size
|
||||
if total_chunks < 1 or total_chunks > 100:
|
||||
logger.warning("[WeCom] invalid total_chunks=%d for %s", total_chunks, filename)
|
||||
return None
|
||||
|
||||
md5_hasher = hashlib.md5()
|
||||
with open(path, "rb") as f:
|
||||
for chunk in iter(lambda: f.read(1024 * 1024), b""):
|
||||
md5_hasher.update(chunk)
|
||||
md5 = md5_hasher.hexdigest()
|
||||
|
||||
init_req_id = generate_req_id("aibot_upload_media_init")
|
||||
init_body = {
|
||||
"type": media_type,
|
||||
"filename": filename,
|
||||
"total_size": int(size),
|
||||
"total_chunks": int(total_chunks),
|
||||
"md5": md5,
|
||||
}
|
||||
init_ack = await self._send_ws_upload_command(init_req_id, init_body, "aibot_upload_media_init")
|
||||
upload_id = (init_ack.get("body") or {}).get("upload_id")
|
||||
if not upload_id:
|
||||
logger.warning("[WeCom] upload init returned no upload_id: %s", init_ack)
|
||||
return None
|
||||
|
||||
with open(path, "rb") as f:
|
||||
for idx in range(total_chunks):
|
||||
data = f.read(chunk_size)
|
||||
if not data:
|
||||
break
|
||||
chunk_req_id = generate_req_id("aibot_upload_media_chunk")
|
||||
chunk_body = {
|
||||
"upload_id": upload_id,
|
||||
"chunk_index": int(idx),
|
||||
"base64_data": base64.b64encode(data).decode("utf-8"),
|
||||
}
|
||||
await self._send_ws_upload_command(chunk_req_id, chunk_body, "aibot_upload_media_chunk")
|
||||
|
||||
finish_req_id = generate_req_id("aibot_upload_media_finish")
|
||||
finish_ack = await self._send_ws_upload_command(finish_req_id, {"upload_id": upload_id}, "aibot_upload_media_finish")
|
||||
media_id = (finish_ack.get("body") or {}).get("media_id")
|
||||
if not media_id:
|
||||
logger.warning("[WeCom] upload finish returned no media_id: %s", finish_ack)
|
||||
return None
|
||||
return media_id
|
||||
@@ -1,23 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
__all__ = ["GatewayConfig", "app", "get_gateway_config", "register_app"]
|
||||
|
||||
|
||||
def __getattr__(name: str):
|
||||
if name == "app":
|
||||
from .app import app
|
||||
|
||||
return app
|
||||
if name == "GatewayConfig":
|
||||
from .config import GatewayConfig
|
||||
|
||||
return GatewayConfig
|
||||
if name == "get_gateway_config":
|
||||
from .config import get_gateway_config
|
||||
|
||||
return get_gateway_config
|
||||
if name == "register_app":
|
||||
from .registrar import register_app
|
||||
|
||||
return register_app
|
||||
raise AttributeError(name)
|
||||
@@ -1,8 +0,0 @@
|
||||
from app.gateway.registrar import register_app
|
||||
|
||||
|
||||
def create_app():
|
||||
return register_app()
|
||||
|
||||
|
||||
app = register_app()
|
||||
@@ -1,3 +0,0 @@
|
||||
from .lifespan import lifespan_manager
|
||||
|
||||
__all__ = ["lifespan_manager"]
|
||||
@@ -1,52 +0,0 @@
|
||||
from collections.abc import Callable
|
||||
from contextlib import AbstractAsyncContextManager, AsyncExitStack, asynccontextmanager
|
||||
from typing import Any
|
||||
|
||||
from fastapi import FastAPI
|
||||
|
||||
LifespanFunc = Callable[[FastAPI], AbstractAsyncContextManager[dict[str, Any] | None]]
|
||||
|
||||
|
||||
class LifespanManager:
|
||||
"""FastAPI lifespan manager"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._lifespans: list[LifespanFunc] = []
|
||||
|
||||
def register(self, func: LifespanFunc) -> LifespanFunc:
|
||||
"""
|
||||
Register a lifespan hook.
|
||||
|
||||
:param func: lifespan hook
|
||||
:return:
|
||||
"""
|
||||
if func not in self._lifespans:
|
||||
self._lifespans.append(func)
|
||||
return func
|
||||
|
||||
def build(self) -> LifespanFunc:
|
||||
"""
|
||||
Build the combined lifespan hook.
|
||||
|
||||
:return:
|
||||
"""
|
||||
|
||||
@asynccontextmanager
|
||||
async def combined_lifespan(app: FastAPI): # noqa: ANN202
|
||||
state: dict[str, Any] = {}
|
||||
async with AsyncExitStack() as exit_stack:
|
||||
for lifespan_fn in self._lifespans:
|
||||
result = await exit_stack.enter_async_context(lifespan_fn(app))
|
||||
if isinstance(result, dict):
|
||||
state.update(result)
|
||||
|
||||
for key, value in state.items():
|
||||
setattr(app.state, key, value)
|
||||
|
||||
yield state or None
|
||||
|
||||
return combined_lifespan
|
||||
|
||||
|
||||
# Singleton lifespan_manager instance
|
||||
lifespan_manager = LifespanManager()
|
||||
@@ -1,59 +0,0 @@
|
||||
from app.gateway.dependencies.checkpointer import (
|
||||
CurrentCheckpointer,
|
||||
get_checkpointer,
|
||||
)
|
||||
from app.plugins.auth.security.dependencies import (
|
||||
CurrentAuthService,
|
||||
CurrentUserRepository,
|
||||
get_auth_service,
|
||||
get_current_user_from_request,
|
||||
get_current_user_id,
|
||||
get_optional_user_from_request,
|
||||
get_user_repository,
|
||||
)
|
||||
from app.gateway.dependencies.db import (
|
||||
CurrentSession,
|
||||
CurrentSessionTransaction,
|
||||
get_db_session,
|
||||
get_db_session_transaction,
|
||||
)
|
||||
from app.gateway.dependencies.repositories import (
|
||||
CurrentFeedbackRepository,
|
||||
CurrentRunRepository,
|
||||
CurrentThreadMetaRepository,
|
||||
CurrentThreadMetaStorage,
|
||||
get_feedback_repository,
|
||||
get_run_repository,
|
||||
get_thread_meta_repository,
|
||||
get_thread_meta_storage,
|
||||
)
|
||||
from app.gateway.dependencies.stream_bridge import (
|
||||
CurrentStreamBridge,
|
||||
get_stream_bridge,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"CurrentCheckpointer",
|
||||
"CurrentAuthService",
|
||||
"CurrentFeedbackRepository",
|
||||
"CurrentRunRepository",
|
||||
"CurrentSession",
|
||||
"CurrentSessionTransaction",
|
||||
"CurrentStreamBridge",
|
||||
"CurrentThreadMetaRepository",
|
||||
"CurrentThreadMetaStorage",
|
||||
"CurrentUserRepository",
|
||||
"get_auth_service",
|
||||
"get_checkpointer",
|
||||
"get_current_user_from_request",
|
||||
"get_current_user_id",
|
||||
"get_db_session",
|
||||
"get_db_session_transaction",
|
||||
"get_feedback_repository",
|
||||
"get_optional_user_from_request",
|
||||
"get_run_repository",
|
||||
"get_stream_bridge",
|
||||
"get_thread_meta_repository",
|
||||
"get_thread_meta_storage",
|
||||
"get_user_repository",
|
||||
]
|
||||
@@ -1,20 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Annotated
|
||||
|
||||
from fastapi import Depends, HTTPException, Request
|
||||
from langgraph.types import Checkpointer
|
||||
|
||||
|
||||
def get_checkpointer(request: Request) -> Checkpointer:
|
||||
"""Get checkpointer from app.state.persistence."""
|
||||
persistence = getattr(request.app.state, "persistence", None)
|
||||
if persistence is None:
|
||||
raise HTTPException(status_code=503, detail="Persistence not available")
|
||||
checkpointer = getattr(persistence, "checkpointer", None)
|
||||
if checkpointer is None:
|
||||
raise HTTPException(status_code=503, detail="Checkpointer not available")
|
||||
return checkpointer
|
||||
|
||||
|
||||
CurrentCheckpointer = Annotated[Checkpointer, Depends(get_checkpointer)]
|
||||
@@ -1,37 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import AsyncIterator
|
||||
from typing import Annotated
|
||||
|
||||
from fastapi import Depends, HTTPException, Request
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
|
||||
|
||||
def _get_session_factory(request: Request) -> async_sessionmaker[AsyncSession]:
|
||||
factory = getattr(request.app.state.persistence, "session_factory", None)
|
||||
if factory is None:
|
||||
raise HTTPException(status_code=503, detail="Database session factory not available")
|
||||
return factory
|
||||
|
||||
|
||||
async def get_db_session(request: Request) -> AsyncIterator[AsyncSession]:
|
||||
"""Open a session without auto-commit. Use for read-only endpoints."""
|
||||
session_factory = _get_session_factory(request)
|
||||
async with session_factory() as session:
|
||||
yield session
|
||||
|
||||
|
||||
async def get_db_session_transaction(request: Request) -> AsyncIterator[AsyncSession]:
|
||||
"""Open a session and commit on success, rollback on error."""
|
||||
session_factory = _get_session_factory(request)
|
||||
async with session_factory() as session:
|
||||
try:
|
||||
yield session
|
||||
await session.commit()
|
||||
except Exception:
|
||||
await session.rollback()
|
||||
raise
|
||||
|
||||
|
||||
CurrentSession = Annotated[AsyncSession, Depends(get_db_session)]
|
||||
CurrentSessionTransaction = Annotated[AsyncSession, Depends(get_db_session_transaction)]
|
||||
@@ -1,41 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Annotated
|
||||
|
||||
from fastapi import Depends, HTTPException, Request
|
||||
|
||||
from app.infra.storage import ThreadMetaStorage
|
||||
from store.repositories.contracts import (
|
||||
FeedbackRepositoryProtocol,
|
||||
RunRepositoryProtocol,
|
||||
ThreadMetaRepositoryProtocol,
|
||||
)
|
||||
|
||||
|
||||
def _require_state(request: Request, attr: str, label: str):
|
||||
value = getattr(request.app.state, attr, None)
|
||||
if value is None:
|
||||
raise HTTPException(status_code=503, detail=f"{label} not available")
|
||||
return value
|
||||
|
||||
|
||||
def get_run_repository(request: Request) -> RunRepositoryProtocol:
|
||||
return _require_state(request, "run_store", "Run store")
|
||||
|
||||
|
||||
def get_thread_meta_repository(request: Request) -> ThreadMetaRepositoryProtocol:
|
||||
return _require_state(request, "thread_meta_repo", "Thread metadata store")
|
||||
|
||||
|
||||
def get_thread_meta_storage(request: Request) -> ThreadMetaStorage:
|
||||
return _require_state(request, "thread_meta_storage", "Thread metadata storage")
|
||||
|
||||
|
||||
def get_feedback_repository(request: Request) -> FeedbackRepositoryProtocol:
|
||||
return _require_state(request, "feedback_repo", "Feedback")
|
||||
|
||||
|
||||
CurrentRunRepository = Annotated[RunRepositoryProtocol, Depends(get_run_repository)]
|
||||
CurrentThreadMetaRepository = Annotated[ThreadMetaRepositoryProtocol, Depends(get_thread_meta_repository)]
|
||||
CurrentThreadMetaStorage = Annotated[ThreadMetaStorage, Depends(get_thread_meta_storage)]
|
||||
CurrentFeedbackRepository = Annotated[FeedbackRepositoryProtocol, Depends(get_feedback_repository)]
|
||||
@@ -1,18 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Annotated
|
||||
|
||||
from fastapi import Depends, HTTPException, Request
|
||||
|
||||
from deerflow.runtime import StreamBridge
|
||||
|
||||
|
||||
def get_stream_bridge(request: Request) -> StreamBridge:
|
||||
"""Get stream bridge from app.state."""
|
||||
bridge = getattr(request.app.state, "stream_bridge", None)
|
||||
if bridge is None:
|
||||
raise HTTPException(status_code=503, detail="Stream bridge not available")
|
||||
return bridge
|
||||
|
||||
|
||||
CurrentStreamBridge = Annotated[StreamBridge, Depends(get_stream_bridge)]
|
||||
@@ -1,132 +0,0 @@
|
||||
from collections.abc import AsyncGenerator
|
||||
from contextlib import asynccontextmanager
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from fastapi import FastAPI
|
||||
from fastapi.responses import HTMLResponse
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
from scalar_fastapi import AgentScalarConfig, get_scalar_api_reference
|
||||
from starlette.middleware.cors import CORSMiddleware
|
||||
from store.persistence import create_persistence
|
||||
|
||||
from app.gateway.common import lifespan_manager
|
||||
from app.gateway.router import router as gateway_router
|
||||
from app.infra.run_events import build_run_event_store
|
||||
from app.infra.storage import FeedbackStoreAdapter, RunStoreAdapter, ThreadMetaStorage, ThreadMetaStoreAdapter
|
||||
from app.plugins.auth.authorization.hooks import build_authz_hooks
|
||||
from app.plugins.auth.injection import install_route_guards, load_route_policy_registry, validate_route_policy_registry
|
||||
from app.plugins.auth.security import AuthMiddleware, CSRFMiddleware
|
||||
|
||||
STATIC_DIR = Path(__file__).resolve().parents[1] / "static"
|
||||
STATIC_MOUNT = "/api/static"
|
||||
SCALAR_JS_URL = f"{STATIC_MOUNT}/scalar.js"
|
||||
|
||||
|
||||
@lifespan_manager.register
|
||||
@asynccontextmanager
|
||||
async def init_persistence(app: FastAPI) -> AsyncGenerator[dict[str, Any], None]:
|
||||
"""Initialize persistence layer (DB, checkpointer, store)."""
|
||||
app_persistence = await create_persistence()
|
||||
|
||||
await app_persistence.setup()
|
||||
run_store = RunStoreAdapter(app_persistence.session_factory)
|
||||
thread_meta_store = ThreadMetaStoreAdapter(app_persistence.session_factory)
|
||||
feedback_store = FeedbackStoreAdapter(app_persistence.session_factory)
|
||||
|
||||
try:
|
||||
yield {
|
||||
"persistence": app_persistence,
|
||||
"checkpointer": app_persistence.checkpointer,
|
||||
"store": None,
|
||||
"session_factory": app_persistence.session_factory,
|
||||
"run_store": run_store,
|
||||
"run_read_repo": run_store,
|
||||
"run_write_repo": run_store,
|
||||
"run_delete_repo": run_store,
|
||||
"feedback_repo": feedback_store,
|
||||
"thread_meta_repo": thread_meta_store,
|
||||
"thread_meta_storage": ThreadMetaStorage(thread_meta_store),
|
||||
"run_event_store": build_run_event_store(app_persistence.session_factory),
|
||||
}
|
||||
finally:
|
||||
await app_persistence.aclose()
|
||||
|
||||
|
||||
@lifespan_manager.register
|
||||
@asynccontextmanager
|
||||
async def init_runtime(app: FastAPI) -> AsyncGenerator[dict[str, Any], None]:
|
||||
"""Initialize StreamBridge for LangGraph-compatible runtime endpoints."""
|
||||
from app.infra.stream_bridge import build_stream_bridge
|
||||
|
||||
async with build_stream_bridge() as stream_bridge:
|
||||
yield {
|
||||
"stream_bridge": stream_bridge,
|
||||
}
|
||||
|
||||
|
||||
def register_app() -> FastAPI:
|
||||
app = FastAPI(
|
||||
title="DeerFlow API Gateway",
|
||||
version="0.1.0",
|
||||
docs_url=None,
|
||||
redoc_url=None,
|
||||
lifespan=lifespan_manager.build(),
|
||||
openapi_tags=[
|
||||
{
|
||||
"name": "threads",
|
||||
"description": "Endpoints for managing threads, which are conversations between a human and an assistant. A thread can have multiple runs as the conversation progresses."
|
||||
}
|
||||
]
|
||||
)
|
||||
|
||||
app.state.authz_hooks = build_authz_hooks()
|
||||
|
||||
_register_static(app)
|
||||
_register_routes(app)
|
||||
_register_scalar(app)
|
||||
_register_auth_route_policies(app)
|
||||
_register_middlewares(app)
|
||||
|
||||
return app
|
||||
|
||||
|
||||
def _register_static(app: FastAPI) -> None:
|
||||
app.mount(STATIC_MOUNT, StaticFiles(directory=STATIC_DIR), name="static")
|
||||
|
||||
|
||||
def _register_routes(app: FastAPI) -> None:
|
||||
app.include_router(gateway_router)
|
||||
|
||||
|
||||
def _register_auth_route_policies(app: FastAPI) -> None:
|
||||
registry = load_route_policy_registry()
|
||||
validate_route_policy_registry(app, registry)
|
||||
app.state.auth_route_policy_registry = registry
|
||||
install_route_guards(app)
|
||||
|
||||
|
||||
def _register_middlewares(app: FastAPI) -> None:
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=["*"],
|
||||
allow_credentials=True,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
expose_headers=["*"],
|
||||
)
|
||||
app.add_middleware(CSRFMiddleware)
|
||||
app.add_middleware(AuthMiddleware)
|
||||
|
||||
|
||||
def _register_scalar(app: FastAPI) -> None:
|
||||
@app.get("/docs", include_in_schema=False)
|
||||
def scalar_docs() -> HTMLResponse:
|
||||
return get_scalar_api_reference(
|
||||
openapi_url=app.openapi_url,
|
||||
title=app.title,
|
||||
scalar_js_url=SCALAR_JS_URL,
|
||||
agent=AgentScalarConfig(disabled=True),
|
||||
hide_client_button=True,
|
||||
overrides={"mcp": {"disabled": True}},
|
||||
)
|
||||
@@ -1,22 +0,0 @@
|
||||
from fastapi import APIRouter
|
||||
|
||||
from app.plugins.auth.api.router import router as auth_router
|
||||
|
||||
from .routers import artifacts, channels, mcp, models, skills, uploads
|
||||
from .routers.agents import router as agents_router
|
||||
from .routers.langgraph import feedback_router, runs_router, suggestion_router, threads_router
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
router.include_router(auth_router)
|
||||
router.include_router(threads_router, prefix="/api/threads")
|
||||
router.include_router(runs_router, prefix="/api/threads")
|
||||
router.include_router(feedback_router, prefix="/api/threads")
|
||||
router.include_router(suggestion_router)
|
||||
router.include_router(agents_router)
|
||||
router.include_router(channels.router)
|
||||
router.include_router(artifacts.router)
|
||||
router.include_router(mcp.router)
|
||||
router.include_router(models.router)
|
||||
router.include_router(skills.router)
|
||||
router.include_router(uploads.router)
|
||||
@@ -1,3 +0,0 @@
|
||||
from . import artifacts, mcp, models, skills, suggestions, uploads
|
||||
|
||||
__all__ = ["artifacts", "mcp", "models", "skills", "suggestions", "uploads"]
|
||||
@@ -1,52 +0,0 @@
|
||||
"""Gateway router for IM channel management."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from pydantic import BaseModel
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/channels", tags=["channels"])
|
||||
|
||||
|
||||
class ChannelStatusResponse(BaseModel):
|
||||
service_running: bool
|
||||
channels: dict[str, dict]
|
||||
|
||||
|
||||
class ChannelRestartResponse(BaseModel):
|
||||
success: bool
|
||||
message: str
|
||||
|
||||
|
||||
@router.get("/", response_model=ChannelStatusResponse)
|
||||
async def get_channels_status() -> ChannelStatusResponse:
|
||||
"""Get the status of all IM channels."""
|
||||
from app.channels.service import get_channel_service
|
||||
|
||||
service = get_channel_service()
|
||||
if service is None:
|
||||
return ChannelStatusResponse(service_running=False, channels={})
|
||||
status = service.get_status()
|
||||
return ChannelStatusResponse(**status)
|
||||
|
||||
|
||||
@router.post("/{name}/restart", response_model=ChannelRestartResponse)
|
||||
async def restart_channel(name: str) -> ChannelRestartResponse:
|
||||
"""Restart a specific IM channel."""
|
||||
from app.channels.service import get_channel_service
|
||||
|
||||
service = get_channel_service()
|
||||
if service is None:
|
||||
raise HTTPException(status_code=503, detail="Channel service is not running")
|
||||
|
||||
success = await service.restart_channel(name)
|
||||
if success:
|
||||
logger.info("Channel %s restarted successfully", name)
|
||||
return ChannelRestartResponse(success=True, message=f"Channel {name} restarted successfully")
|
||||
else:
|
||||
logger.warning("Failed to restart channel %s", name)
|
||||
return ChannelRestartResponse(success=False, message=f"Failed to restart channel {name}")
|
||||
@@ -1,6 +0,0 @@
|
||||
from .feedback import router as feedback_router
|
||||
from .runs import router as runs_router
|
||||
from .suggestions import router as suggestion_router
|
||||
from .threads import router as threads_router
|
||||
|
||||
__all__ = ["feedback_router", "runs_router", "threads_router", "suggestion_router"]
|
||||
@@ -1,179 +0,0 @@
|
||||
"""LangGraph-compatible run feedback endpoints."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Request
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.gateway.dependencies import get_feedback_repository, get_run_repository
|
||||
from app.plugins.auth.security.actor_context import bind_request_actor_context, resolve_request_user_id
|
||||
from app.plugins.auth.security.dependencies import get_current_user_id
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(tags=["feedback"])
|
||||
|
||||
|
||||
class FeedbackCreateRequest(BaseModel):
|
||||
rating: int = Field(..., description="Feedback rating: +1 (positive) or -1 (negative)")
|
||||
comment: str | None = Field(default=None, description="Optional text feedback")
|
||||
message_id: str | None = Field(default=None, description="Optional: scope feedback to a specific message")
|
||||
|
||||
|
||||
class FeedbackResponse(BaseModel):
|
||||
feedback_id: str
|
||||
run_id: str
|
||||
thread_id: str
|
||||
owner_id: str | None = None
|
||||
message_id: str | None = None
|
||||
rating: int
|
||||
comment: str | None = None
|
||||
created_at: str = ""
|
||||
|
||||
|
||||
class FeedbackStatsResponse(BaseModel):
|
||||
run_id: str
|
||||
total: int = 0
|
||||
positive: int = 0
|
||||
negative: int = 0
|
||||
|
||||
|
||||
async def _validate_run_scope(thread_id: str, run_id: str, request: Request) -> None:
|
||||
run_store = get_run_repository(request)
|
||||
if resolve_request_user_id(request) is None:
|
||||
run = await run_store.get(run_id, user_id=None)
|
||||
else:
|
||||
with bind_request_actor_context(request):
|
||||
run = await run_store.get(run_id)
|
||||
if run is None:
|
||||
raise HTTPException(status_code=404, detail=f"Run {run_id} not found")
|
||||
if run.get("thread_id") != thread_id:
|
||||
raise HTTPException(status_code=404, detail=f"Run {run_id} not found in thread {thread_id}")
|
||||
|
||||
|
||||
async def _get_current_user(request: Request) -> str | None:
|
||||
"""Extract current user id from auth dependencies when available."""
|
||||
return await get_current_user_id(request)
|
||||
|
||||
|
||||
async def _create_feedback(
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
body: FeedbackCreateRequest,
|
||||
request: Request,
|
||||
) -> dict[str, Any]:
|
||||
if body.rating not in (1, -1):
|
||||
raise HTTPException(status_code=400, detail="rating must be +1 or -1")
|
||||
|
||||
await _validate_run_scope(thread_id, run_id, request)
|
||||
user_id = await _get_current_user(request)
|
||||
feedback_repo = get_feedback_repository(request)
|
||||
return await feedback_repo.create(
|
||||
run_id=run_id,
|
||||
thread_id=thread_id,
|
||||
rating=body.rating,
|
||||
user_id=user_id,
|
||||
message_id=body.message_id,
|
||||
comment=body.comment,
|
||||
)
|
||||
|
||||
|
||||
@router.put("/{thread_id}/runs/{run_id}/feedback", response_model=FeedbackResponse)
|
||||
async def upsert_feedback(
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
body: FeedbackCreateRequest,
|
||||
request: Request,
|
||||
) -> dict[str, Any]:
|
||||
"""Create or replace the run-level feedback record."""
|
||||
feedback_repo = get_feedback_repository(request)
|
||||
user_id = await _get_current_user(request)
|
||||
if user_id is not None:
|
||||
return await feedback_repo.upsert(
|
||||
run_id=run_id,
|
||||
thread_id=thread_id,
|
||||
rating=body.rating,
|
||||
user_id=user_id,
|
||||
comment=body.comment,
|
||||
)
|
||||
existing = await feedback_repo.list_by_run(thread_id, run_id, limit=100, user_id=None)
|
||||
for item in existing:
|
||||
feedback_id = item.get("feedback_id")
|
||||
if isinstance(feedback_id, str):
|
||||
await feedback_repo.delete(feedback_id)
|
||||
return await _create_feedback(thread_id, run_id, body, request)
|
||||
|
||||
|
||||
@router.post("/{thread_id}/runs/{run_id}/feedback", response_model=FeedbackResponse)
|
||||
async def create_feedback(
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
body: FeedbackCreateRequest,
|
||||
request: Request,
|
||||
) -> dict[str, Any]:
|
||||
"""Submit feedback for a run."""
|
||||
return await _create_feedback(thread_id, run_id, body, request)
|
||||
|
||||
|
||||
@router.get("/{thread_id}/runs/{run_id}/feedback", response_model=list[FeedbackResponse])
|
||||
async def list_feedback(
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
request: Request,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""List all feedback for a run."""
|
||||
feedback_repo = get_feedback_repository(request)
|
||||
user_id = await _get_current_user(request)
|
||||
return await feedback_repo.list_by_run(thread_id, run_id, user_id=user_id)
|
||||
|
||||
|
||||
@router.get("/{thread_id}/runs/{run_id}/feedback/stats", response_model=FeedbackStatsResponse)
|
||||
async def feedback_stats(
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
request: Request,
|
||||
) -> dict[str, Any]:
|
||||
"""Get aggregated feedback stats for a run."""
|
||||
feedback_repo = get_feedback_repository(request)
|
||||
return await feedback_repo.aggregate_by_run(thread_id, run_id)
|
||||
|
||||
|
||||
@router.delete("/{thread_id}/runs/{run_id}/feedback")
|
||||
async def delete_run_feedback(
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
request: Request,
|
||||
) -> dict[str, bool]:
|
||||
"""Delete all feedback records for a run."""
|
||||
feedback_repo = get_feedback_repository(request)
|
||||
user_id = await _get_current_user(request)
|
||||
if user_id is not None:
|
||||
return {"success": await feedback_repo.delete_by_run(thread_id=thread_id, run_id=run_id, user_id=user_id)}
|
||||
existing = await feedback_repo.list_by_run(thread_id, run_id, limit=100, user_id=None)
|
||||
for item in existing:
|
||||
feedback_id = item.get("feedback_id")
|
||||
if isinstance(feedback_id, str):
|
||||
await feedback_repo.delete(feedback_id)
|
||||
return {"success": True}
|
||||
|
||||
|
||||
@router.delete("/{thread_id}/runs/{run_id}/feedback/{feedback_id}")
|
||||
async def delete_feedback(
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
feedback_id: str,
|
||||
request: Request,
|
||||
) -> dict[str, bool]:
|
||||
"""Delete a single feedback record."""
|
||||
feedback_repo = get_feedback_repository(request)
|
||||
existing = await feedback_repo.get(feedback_id)
|
||||
if existing is None:
|
||||
raise HTTPException(status_code=404, detail=f"Feedback {feedback_id} not found")
|
||||
if existing.get("thread_id") != thread_id or existing.get("run_id") != run_id:
|
||||
raise HTTPException(status_code=404, detail=f"Feedback {feedback_id} not found in run {run_id}")
|
||||
deleted = await feedback_repo.delete(feedback_id)
|
||||
if not deleted:
|
||||
raise HTTPException(status_code=404, detail=f"Feedback {feedback_id} not found")
|
||||
return {"success": True}
|
||||
@@ -1,501 +0,0 @@
|
||||
"""LangGraph-compatible runs endpoints backed by RunsFacade."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections.abc import AsyncIterator
|
||||
from typing import Literal
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Request
|
||||
from fastapi.responses import Response, StreamingResponse
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.plugins.auth.security.actor_context import bind_request_actor_context
|
||||
from app.gateway.services.runs.facade_factory import build_runs_facade_from_request
|
||||
from app.gateway.services.runs.input import (
|
||||
AdaptedRunRequest,
|
||||
RunSpecBuilder,
|
||||
UnsupportedRunFeatureError,
|
||||
adapt_create_run_request,
|
||||
adapt_create_stream_request,
|
||||
adapt_create_wait_request,
|
||||
adapt_join_stream_request,
|
||||
adapt_join_wait_request,
|
||||
)
|
||||
from deerflow.runtime.runs.types import RunRecord, RunSpec
|
||||
from deerflow.runtime.stream_bridge import JSONValue, StreamEvent
|
||||
|
||||
router = APIRouter(tags=["runs"])
|
||||
|
||||
|
||||
class RunCreateRequest(BaseModel):
|
||||
assistant_id: str | None = Field(default=None, description="Agent / assistant to use")
|
||||
follow_up_to_run_id: str | None = Field(default=None, description="Lineage link to the prior run")
|
||||
input: dict[str, JSONValue] | None = Field(default=None, description="Graph input (e.g. {messages: [...]})")
|
||||
command: dict[str, JSONValue] | None = Field(default=None, description="LangGraph Command")
|
||||
metadata: dict[str, JSONValue] | None = Field(default=None, description="Run metadata")
|
||||
config: dict[str, JSONValue] | None = Field(default=None, description="RunnableConfig overrides")
|
||||
context: dict[str, JSONValue] | None = Field(default=None, description="DeerFlow context overrides (model_name, thinking_enabled, etc.)")
|
||||
webhook: str | None = Field(default=None, description="Completion callback URL")
|
||||
checkpoint_id: str | None = Field(default=None, description="Resume from checkpoint")
|
||||
checkpoint: dict[str, JSONValue] | None = Field(default=None, description="Full checkpoint object")
|
||||
interrupt_before: list[str] | Literal["*"] | None = Field(default=None, description="Nodes to interrupt before")
|
||||
interrupt_after: list[str] | Literal["*"] | None = Field(default=None, description="Nodes to interrupt after")
|
||||
stream_mode: list[str] | str | None = Field(default=None, description="Stream mode(s)")
|
||||
stream_subgraphs: bool = Field(default=False, description="Include subgraph events")
|
||||
stream_resumable: bool | None = Field(default=None, description="SSE resumable mode")
|
||||
on_disconnect: Literal["cancel", "continue"] = Field(default="cancel", description="Behaviour on SSE disconnect")
|
||||
on_completion: Literal["delete", "keep"] = Field(default="keep", description="Delete temp thread on completion")
|
||||
multitask_strategy: Literal["reject", "rollback", "interrupt", "enqueue"] = Field(default="reject", description="Concurrency strategy")
|
||||
after_seconds: float | None = Field(default=None, description="Delayed execution")
|
||||
if_not_exists: Literal["reject", "create"] = Field(default="create", description="Thread creation policy")
|
||||
feedback_keys: list[str] | None = Field(default=None, description="LangSmith feedback keys")
|
||||
|
||||
|
||||
class RunResponse(BaseModel):
|
||||
run_id: str
|
||||
thread_id: str
|
||||
assistant_id: str | None = None
|
||||
status: str
|
||||
metadata: dict[str, JSONValue] = Field(default_factory=dict)
|
||||
multitask_strategy: str = "reject"
|
||||
created_at: str = ""
|
||||
updated_at: str = ""
|
||||
|
||||
|
||||
class RunDeleteResponse(BaseModel):
|
||||
deleted: bool
|
||||
|
||||
|
||||
class RunMessageResponse(BaseModel):
|
||||
run_id: str
|
||||
content: JSONValue
|
||||
metadata: dict[str, JSONValue] = Field(default_factory=dict)
|
||||
created_at: str
|
||||
seq: int
|
||||
|
||||
|
||||
class RunMessagesResponse(BaseModel):
|
||||
data: list[RunMessageResponse]
|
||||
hasMore: bool = False
|
||||
|
||||
|
||||
def format_sse(event: str, data: JSONValue, *, event_id: str | None = None) -> str:
|
||||
"""Format a single SSE frame."""
|
||||
payload = json.dumps(data, default=str, ensure_ascii=False)
|
||||
parts = [f"event: {event}", f"data: {payload}"]
|
||||
if event_id:
|
||||
parts.append(f"id: {event_id}")
|
||||
parts.append("")
|
||||
parts.append("")
|
||||
return "\n".join(parts)
|
||||
|
||||
|
||||
def _record_to_response(record: RunRecord) -> RunResponse:
|
||||
return RunResponse(
|
||||
run_id=record.run_id,
|
||||
thread_id=record.thread_id,
|
||||
assistant_id=record.assistant_id,
|
||||
status=record.status,
|
||||
metadata=record.metadata,
|
||||
multitask_strategy=record.multitask_strategy,
|
||||
created_at=record.created_at,
|
||||
updated_at=record.updated_at,
|
||||
)
|
||||
|
||||
|
||||
def _trim_paginated_rows(
|
||||
rows: list[dict],
|
||||
*,
|
||||
limit: int,
|
||||
after_seq: int | None,
|
||||
) -> tuple[list[dict], bool]:
|
||||
has_more = len(rows) > limit
|
||||
if not has_more:
|
||||
return rows, False
|
||||
if after_seq is not None:
|
||||
return rows[:limit], True
|
||||
return rows[-limit:], True
|
||||
|
||||
|
||||
def _event_to_run_message(event: dict) -> RunMessageResponse:
|
||||
return RunMessageResponse(
|
||||
run_id=str(event["run_id"]),
|
||||
content=event.get("content"),
|
||||
metadata=dict(event.get("metadata") or {}),
|
||||
created_at=str(event.get("created_at") or ""),
|
||||
seq=int(event["seq"]),
|
||||
)
|
||||
|
||||
|
||||
async def _sse_consumer(
|
||||
stream: AsyncIterator[StreamEvent],
|
||||
request: Request,
|
||||
*,
|
||||
cancel_on_disconnect: bool,
|
||||
cancel_run,
|
||||
run_id: str,
|
||||
) -> AsyncIterator[str]:
|
||||
try:
|
||||
async for event in stream:
|
||||
if await request.is_disconnected():
|
||||
break
|
||||
|
||||
if event.event == "__heartbeat__":
|
||||
yield ": heartbeat\n\n"
|
||||
continue
|
||||
|
||||
if event.event == "__end__":
|
||||
yield format_sse("end", None, event_id=event.id or None)
|
||||
return
|
||||
|
||||
if event.event == "__cancelled__":
|
||||
yield format_sse("cancel", None, event_id=event.id or None)
|
||||
return
|
||||
|
||||
yield format_sse(event.event, event.data, event_id=event.id or None)
|
||||
finally:
|
||||
if cancel_on_disconnect:
|
||||
await cancel_run(run_id)
|
||||
|
||||
|
||||
def _get_run_event_store(request: Request):
|
||||
event_store = getattr(request.app.state, "run_event_store", None)
|
||||
if event_store is None:
|
||||
raise HTTPException(status_code=503, detail="Run event store not available")
|
||||
return event_store
|
||||
|
||||
|
||||
@router.get("/{thread_id}/runs", response_model=list[RunResponse])
|
||||
async def list_runs(
|
||||
thread_id: str,
|
||||
request: Request,
|
||||
limit: int = 100,
|
||||
offset: int = 0,
|
||||
status: str | None = None,
|
||||
) -> list[RunResponse]:
|
||||
# Accepted for API compatibility; field projection is not implemented yet.
|
||||
facade = build_runs_facade_from_request(request)
|
||||
with bind_request_actor_context(request):
|
||||
records = await facade.list_runs(thread_id)
|
||||
if status is not None:
|
||||
records = [record for record in records if record.status == status]
|
||||
records = records[offset : offset + limit]
|
||||
return [_record_to_response(record) for record in records]
|
||||
|
||||
|
||||
@router.get("/{thread_id}/runs/{run_id}", response_model=RunResponse)
|
||||
async def get_run(thread_id: str, run_id: str, request: Request) -> RunResponse:
|
||||
facade = build_runs_facade_from_request(request)
|
||||
with bind_request_actor_context(request):
|
||||
record = await facade.get_run(run_id)
|
||||
if record is None or record.thread_id != thread_id:
|
||||
raise HTTPException(status_code=404, detail=f"Run {run_id} not found")
|
||||
return _record_to_response(record)
|
||||
|
||||
|
||||
@router.get("/{thread_id}/runs/{run_id}/messages", response_model=RunMessagesResponse)
|
||||
async def run_messages(
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
request: Request,
|
||||
limit: int = 50,
|
||||
before_seq: int | None = None,
|
||||
after_seq: int | None = None,
|
||||
) -> RunMessagesResponse:
|
||||
facade = build_runs_facade_from_request(request)
|
||||
with bind_request_actor_context(request):
|
||||
record = await facade.get_run(run_id)
|
||||
if record is None or record.thread_id != thread_id:
|
||||
raise HTTPException(status_code=404, detail=f"Run {run_id} not found")
|
||||
|
||||
event_store = _get_run_event_store(request)
|
||||
with bind_request_actor_context(request):
|
||||
rows = await event_store.list_messages_by_run(
|
||||
thread_id,
|
||||
run_id,
|
||||
limit=limit + 1,
|
||||
before_seq=before_seq,
|
||||
after_seq=after_seq,
|
||||
)
|
||||
page, has_more = _trim_paginated_rows(rows, limit=limit, after_seq=after_seq)
|
||||
return RunMessagesResponse(data=[_event_to_run_message(row) for row in page], hasMore=has_more)
|
||||
|
||||
|
||||
def _build_spec(
|
||||
*,
|
||||
adapted: AdaptedRunRequest,
|
||||
) -> RunSpec:
|
||||
try:
|
||||
return RunSpecBuilder().build(adapted)
|
||||
except UnsupportedRunFeatureError as exc:
|
||||
raise HTTPException(status_code=501, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@router.post("/{thread_id}/runs", response_model=RunResponse)
|
||||
async def create_run(
|
||||
thread_id: str,
|
||||
body: RunCreateRequest,
|
||||
request: Request,
|
||||
) -> Response:
|
||||
adapted = adapt_create_run_request(
|
||||
thread_id=thread_id,
|
||||
body=body.model_dump(),
|
||||
headers=dict(request.headers),
|
||||
query=dict(request.query_params),
|
||||
)
|
||||
spec = _build_spec(adapted=adapted)
|
||||
facade = build_runs_facade_from_request(request)
|
||||
with bind_request_actor_context(request):
|
||||
record = await facade.create_background(spec)
|
||||
return Response(
|
||||
content=_record_to_response(record).model_dump_json(),
|
||||
media_type="application/json",
|
||||
headers={"Content-Location": f"/api/threads/{thread_id}/runs/{record.run_id}"},
|
||||
)
|
||||
|
||||
|
||||
@router.post("/{thread_id}/runs/stream")
|
||||
async def stream_run(
|
||||
thread_id: str,
|
||||
body: RunCreateRequest,
|
||||
request: Request,
|
||||
) -> StreamingResponse:
|
||||
adapted = adapt_create_stream_request(
|
||||
thread_id=thread_id,
|
||||
body=body.model_dump(),
|
||||
headers=dict(request.headers),
|
||||
query=dict(request.query_params),
|
||||
)
|
||||
|
||||
spec = _build_spec(adapted=adapted)
|
||||
|
||||
facade = build_runs_facade_from_request(request)
|
||||
with bind_request_actor_context(request):
|
||||
record, stream = await facade.create_and_stream(spec)
|
||||
|
||||
return StreamingResponse(
|
||||
_sse_consumer(
|
||||
stream,
|
||||
request,
|
||||
cancel_on_disconnect=spec.on_disconnect == "cancel",
|
||||
cancel_run=facade.cancel,
|
||||
run_id=record.run_id,
|
||||
),
|
||||
media_type="text/event-stream",
|
||||
headers={
|
||||
"Cache-Control": "no-cache",
|
||||
"Connection": "keep-alive",
|
||||
"X-Accel-Buffering": "no",
|
||||
"Content-Location": f"/api/threads/{thread_id}/runs/{record.run_id}",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@router.post("/{thread_id}/runs/wait")
|
||||
async def wait_run(
|
||||
thread_id: str,
|
||||
body: RunCreateRequest,
|
||||
request: Request,
|
||||
) -> Response:
|
||||
adapted = adapt_create_wait_request(
|
||||
thread_id=thread_id,
|
||||
body=body.model_dump(),
|
||||
headers=dict(request.headers),
|
||||
query=dict(request.query_params),
|
||||
)
|
||||
spec = _build_spec(adapted=adapted)
|
||||
facade = build_runs_facade_from_request(request)
|
||||
with bind_request_actor_context(request):
|
||||
record, result = await facade.create_and_wait(spec)
|
||||
return Response(
|
||||
content=json.dumps(result, default=str, ensure_ascii=False),
|
||||
media_type="application/json",
|
||||
headers={"Content-Location": f"/api/threads/{thread_id}/runs/{record.run_id}"},
|
||||
)
|
||||
|
||||
|
||||
@router.post("/runs", response_model=RunResponse)
|
||||
async def create_stateless_run(body: RunCreateRequest, request: Request) -> Response:
|
||||
adapted = adapt_create_run_request(
|
||||
thread_id=None,
|
||||
body=body.model_dump(),
|
||||
headers=dict(request.headers),
|
||||
query=dict(request.query_params),
|
||||
)
|
||||
spec = _build_spec(adapted=adapted)
|
||||
facade = build_runs_facade_from_request(request)
|
||||
with bind_request_actor_context(request):
|
||||
record = await facade.create_background(spec)
|
||||
return Response(
|
||||
content=_record_to_response(record).model_dump_json(),
|
||||
media_type="application/json",
|
||||
headers={"Content-Location": f"/api/threads/{record.thread_id}/runs/{record.run_id}"},
|
||||
)
|
||||
|
||||
|
||||
@router.post("/runs/stream")
|
||||
async def create_stateless_stream_run(body: RunCreateRequest, request: Request) -> StreamingResponse:
|
||||
adapted = adapt_create_stream_request(
|
||||
thread_id=None,
|
||||
body=body.model_dump(),
|
||||
headers=dict(request.headers),
|
||||
query=dict(request.query_params),
|
||||
)
|
||||
spec = _build_spec(adapted=adapted)
|
||||
facade = build_runs_facade_from_request(request)
|
||||
with bind_request_actor_context(request):
|
||||
record, stream = await facade.create_and_stream(spec)
|
||||
|
||||
return StreamingResponse(
|
||||
_sse_consumer(
|
||||
stream,
|
||||
request,
|
||||
cancel_on_disconnect=spec.on_disconnect == "cancel",
|
||||
cancel_run=facade.cancel,
|
||||
run_id=record.run_id,
|
||||
),
|
||||
media_type="text/event-stream",
|
||||
headers={
|
||||
"Cache-Control": "no-cache",
|
||||
"Connection": "keep-alive",
|
||||
"X-Accel-Buffering": "no",
|
||||
"Content-Location": f"/api/threads/{record.thread_id}/runs/{record.run_id}",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@router.post("/runs/wait")
|
||||
async def wait_stateless_run(body: RunCreateRequest, request: Request) -> Response:
|
||||
adapted = adapt_create_wait_request(
|
||||
thread_id=None,
|
||||
body=body.model_dump(),
|
||||
headers=dict(request.headers),
|
||||
query=dict(request.query_params),
|
||||
)
|
||||
spec = _build_spec(adapted=adapted)
|
||||
facade = build_runs_facade_from_request(request)
|
||||
with bind_request_actor_context(request):
|
||||
record, result = await facade.create_and_wait(spec)
|
||||
return Response(
|
||||
content=json.dumps(result, default=str, ensure_ascii=False),
|
||||
media_type="application/json",
|
||||
headers={"Content-Location": f"/api/threads/{record.thread_id}/runs/{record.run_id}"},
|
||||
)
|
||||
|
||||
|
||||
@router.api_route("/{thread_id}/runs/{run_id}/stream", methods=["GET", "POST"], response_model=None)
|
||||
async def stream_existing_run(
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
request: Request,
|
||||
action: Literal["interrupt", "rollback"] | None = None,
|
||||
wait: bool = False,
|
||||
cancel_on_disconnect: bool = False,
|
||||
stream_mode: str | None = None,
|
||||
) -> StreamingResponse | Response:
|
||||
facade = build_runs_facade_from_request(request)
|
||||
with bind_request_actor_context(request):
|
||||
record = await facade.get_run(run_id)
|
||||
if record is None or record.thread_id != thread_id:
|
||||
raise HTTPException(status_code=404, detail=f"Run {run_id} not found")
|
||||
|
||||
if action is not None:
|
||||
with bind_request_actor_context(request):
|
||||
cancelled = await facade.cancel(run_id, action=action)
|
||||
if not cancelled:
|
||||
raise HTTPException(status_code=409, detail=f"Run {run_id} is not cancellable")
|
||||
if wait:
|
||||
with bind_request_actor_context(request):
|
||||
await facade.join_wait(run_id)
|
||||
return Response(status_code=204)
|
||||
|
||||
adapted = adapt_join_stream_request(
|
||||
thread_id=thread_id,
|
||||
run_id=run_id,
|
||||
headers=dict(request.headers),
|
||||
query=dict(request.query_params),
|
||||
)
|
||||
with bind_request_actor_context(request):
|
||||
stream = await facade.join_stream(run_id, last_event_id=adapted.last_event_id)
|
||||
|
||||
return StreamingResponse(
|
||||
_sse_consumer(
|
||||
stream,
|
||||
request,
|
||||
cancel_on_disconnect=cancel_on_disconnect,
|
||||
cancel_run=facade.cancel,
|
||||
run_id=run_id,
|
||||
),
|
||||
media_type="text/event-stream",
|
||||
headers={
|
||||
"Cache-Control": "no-cache",
|
||||
"Connection": "keep-alive",
|
||||
"X-Accel-Buffering": "no",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{thread_id}/runs/{run_id}/join")
|
||||
async def join_existing_run(
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
request: Request,
|
||||
cancel_on_disconnect: bool = False,
|
||||
) -> JSONValue:
|
||||
# Accepted for API compatibility; current join_wait path does not change
|
||||
# behavior based on client disconnect.
|
||||
facade = build_runs_facade_from_request(request)
|
||||
with bind_request_actor_context(request):
|
||||
record = await facade.get_run(run_id)
|
||||
if record is None or record.thread_id != thread_id:
|
||||
raise HTTPException(status_code=404, detail=f"Run {run_id} not found")
|
||||
|
||||
adapted = adapt_join_wait_request(
|
||||
thread_id=thread_id,
|
||||
run_id=run_id,
|
||||
headers=dict(request.headers),
|
||||
query=dict(request.query_params),
|
||||
)
|
||||
with bind_request_actor_context(request):
|
||||
return await facade.join_wait(run_id, last_event_id=adapted.last_event_id)
|
||||
|
||||
|
||||
@router.post("/{thread_id}/runs/{run_id}/cancel")
|
||||
async def cancel_existing_run(
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
request: Request,
|
||||
wait: bool = False,
|
||||
action: Literal["interrupt", "rollback"] = "interrupt",
|
||||
) -> JSONValue:
|
||||
facade = build_runs_facade_from_request(request)
|
||||
with bind_request_actor_context(request):
|
||||
record = await facade.get_run(run_id)
|
||||
if record is None or record.thread_id != thread_id:
|
||||
raise HTTPException(status_code=404, detail=f"Run {run_id} not found")
|
||||
|
||||
with bind_request_actor_context(request):
|
||||
cancelled = await facade.cancel(run_id, action=action)
|
||||
if not cancelled:
|
||||
raise HTTPException(status_code=409, detail=f"Run {run_id} is not cancellable")
|
||||
if wait:
|
||||
with bind_request_actor_context(request):
|
||||
return await facade.join_wait(run_id)
|
||||
return {}
|
||||
|
||||
|
||||
@router.delete("/{thread_id}/runs/{run_id}", response_model=RunDeleteResponse)
|
||||
async def delete_run(
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
request: Request,
|
||||
) -> RunDeleteResponse:
|
||||
facade = build_runs_facade_from_request(request)
|
||||
with bind_request_actor_context(request):
|
||||
record = await facade.get_run(run_id)
|
||||
if record is None or record.thread_id != thread_id:
|
||||
raise HTTPException(status_code=404, detail=f"Run {run_id} not found")
|
||||
with bind_request_actor_context(request):
|
||||
deleted = await facade.delete_run(run_id)
|
||||
return RunDeleteResponse(deleted=deleted)
|
||||
@@ -1,132 +0,0 @@
|
||||
import json
|
||||
import logging
|
||||
|
||||
from fastapi import APIRouter
|
||||
from langchain_core.messages import HumanMessage, SystemMessage
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from deerflow.models import create_chat_model
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api", tags=["suggestions"])
|
||||
|
||||
|
||||
class SuggestionMessage(BaseModel):
|
||||
role: str = Field(..., description="Message role: user|assistant")
|
||||
content: str = Field(..., description="Message content as plain text")
|
||||
|
||||
|
||||
class SuggestionsRequest(BaseModel):
|
||||
messages: list[SuggestionMessage] = Field(..., description="Recent conversation messages")
|
||||
n: int = Field(default=3, ge=1, le=5, description="Number of suggestions to generate")
|
||||
model_name: str | None = Field(default=None, description="Optional model override")
|
||||
|
||||
|
||||
class SuggestionsResponse(BaseModel):
|
||||
suggestions: list[str] = Field(default_factory=list, description="Suggested follow-up questions")
|
||||
|
||||
|
||||
def _strip_markdown_code_fence(text: str) -> str:
|
||||
stripped = text.strip()
|
||||
if not stripped.startswith("```"):
|
||||
return stripped
|
||||
lines = stripped.splitlines()
|
||||
if len(lines) >= 3 and lines[0].startswith("```") and lines[-1].startswith("```"):
|
||||
return "\n".join(lines[1:-1]).strip()
|
||||
return stripped
|
||||
|
||||
|
||||
def _parse_json_string_list(text: str) -> list[str] | None:
|
||||
candidate = _strip_markdown_code_fence(text)
|
||||
start = candidate.find("[")
|
||||
end = candidate.rfind("]")
|
||||
if start == -1 or end == -1 or end <= start:
|
||||
return None
|
||||
candidate = candidate[start : end + 1]
|
||||
try:
|
||||
data = json.loads(candidate)
|
||||
except Exception:
|
||||
return None
|
||||
if not isinstance(data, list):
|
||||
return None
|
||||
out: list[str] = []
|
||||
for item in data:
|
||||
if not isinstance(item, str):
|
||||
continue
|
||||
s = item.strip()
|
||||
if not s:
|
||||
continue
|
||||
out.append(s)
|
||||
return out
|
||||
|
||||
|
||||
def _extract_response_text(content: object) -> str:
|
||||
if isinstance(content, str):
|
||||
return content
|
||||
if isinstance(content, list):
|
||||
parts: list[str] = []
|
||||
for block in content:
|
||||
if isinstance(block, str):
|
||||
parts.append(block)
|
||||
elif isinstance(block, dict) and block.get("type") in {"text", "output_text"}:
|
||||
text = block.get("text")
|
||||
if isinstance(text, str):
|
||||
parts.append(text)
|
||||
return "\n".join(parts) if parts else ""
|
||||
if content is None:
|
||||
return ""
|
||||
return str(content)
|
||||
|
||||
|
||||
def _format_conversation(messages: list[SuggestionMessage]) -> str:
|
||||
parts: list[str] = []
|
||||
for m in messages:
|
||||
role = m.role.strip().lower()
|
||||
if role in ("user", "human"):
|
||||
parts.append(f"User: {m.content.strip()}")
|
||||
elif role in ("assistant", "ai"):
|
||||
parts.append(f"Assistant: {m.content.strip()}")
|
||||
else:
|
||||
parts.append(f"{m.role}: {m.content.strip()}")
|
||||
return "\n".join(parts).strip()
|
||||
|
||||
|
||||
@router.post(
|
||||
"/threads/{thread_id}/suggestions",
|
||||
response_model=SuggestionsResponse,
|
||||
summary="Generate Follow-up Questions",
|
||||
description="Generate short follow-up questions a user might ask next, based on recent conversation context.",
|
||||
)
|
||||
async def generate_suggestions(thread_id: str, request: SuggestionsRequest) -> SuggestionsResponse:
|
||||
if not request.messages:
|
||||
return SuggestionsResponse(suggestions=[])
|
||||
|
||||
n = request.n
|
||||
conversation = _format_conversation(request.messages)
|
||||
if not conversation:
|
||||
return SuggestionsResponse(suggestions=[])
|
||||
|
||||
system_instruction = (
|
||||
"You are generating follow-up questions to help the user continue the conversation.\n"
|
||||
f"Based on the conversation below, produce EXACTLY {n} short questions the user might ask next.\n"
|
||||
"Requirements:\n"
|
||||
"- Questions must be relevant to the preceding conversation.\n"
|
||||
"- Questions must be written in the same language as the user.\n"
|
||||
"- Keep each question concise (ideally <= 20 words / <= 40 Chinese characters).\n"
|
||||
"- Do NOT include numbering, markdown, or any extra text.\n"
|
||||
"- Output MUST be a JSON array of strings only.\n"
|
||||
)
|
||||
user_content = f"Conversation Context:\n{conversation}\n\nGenerate {n} follow-up questions"
|
||||
|
||||
try:
|
||||
model = create_chat_model(name=request.model_name, thinking_enabled=False)
|
||||
response = await model.ainvoke([SystemMessage(content=system_instruction), HumanMessage(content=user_content)])
|
||||
raw = _extract_response_text(response.content)
|
||||
suggestions = _parse_json_string_list(raw) or []
|
||||
cleaned = [s.replace("\n", " ").strip() for s in suggestions if s.strip()]
|
||||
cleaned = cleaned[:n]
|
||||
return SuggestionsResponse(suggestions=cleaned)
|
||||
except Exception as exc:
|
||||
logger.exception("Failed to generate suggestions: thread_id=%s err=%s", thread_id, exc)
|
||||
return SuggestionsResponse(suggestions=[])
|
||||
@@ -1,455 +0,0 @@
|
||||
"""Thread management endpoints.
|
||||
|
||||
Provides CRUD operations for threads and checkpoint state management.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import time
|
||||
import uuid
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Request
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.gateway.dependencies import CurrentCheckpointer, CurrentRunRepository, CurrentThreadMetaStorage
|
||||
from app.infra.storage import ThreadMetaStorage
|
||||
from app.plugins.auth.security.actor_context import bind_request_actor_context, resolve_request_user_id
|
||||
from deerflow.config.paths import Paths, get_paths
|
||||
from deerflow.runtime import serialize_channel_values
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(tags=["threads"])
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Request / Response Models
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class ThreadCreateRequest(BaseModel):
|
||||
thread_id: str | None = Field(default=None, description="Optional thread ID (auto-generated if omitted)")
|
||||
assistant_id: str | None = Field(default=None, description="Associate thread with an assistant")
|
||||
metadata: dict[str, Any] = Field(default_factory=dict, description="Initial metadata")
|
||||
|
||||
|
||||
class ThreadSearchRequest(BaseModel):
|
||||
metadata: dict[str, Any] = Field(default_factory=dict, description="Metadata filter (exact match)")
|
||||
limit: int = Field(default=100, ge=1, le=1000, description="Maximum results")
|
||||
offset: int = Field(default=0, ge=0, description="Pagination offset")
|
||||
status: str | None = Field(default=None, description="Filter by thread status")
|
||||
user_id: str | None = Field(default=None, description="Filter by user ID")
|
||||
assistant_id: str | None = Field(default=None, description="Filter by assistant ID")
|
||||
|
||||
|
||||
class ThreadResponse(BaseModel):
|
||||
thread_id: str = Field(description="Unique thread identifier")
|
||||
status: str = Field(default="idle", description="Thread status")
|
||||
created_at: str = Field(default="", description="ISO timestamp")
|
||||
updated_at: str = Field(default="", description="ISO timestamp")
|
||||
metadata: dict[str, Any] = Field(default_factory=dict, description="Thread metadata")
|
||||
values: dict[str, Any] = Field(default_factory=dict, description="Current state values")
|
||||
interrupts: dict[str, Any] = Field(default_factory=dict, description="Pending interrupts")
|
||||
|
||||
|
||||
class ThreadDeleteResponse(BaseModel):
|
||||
success: bool
|
||||
message: str
|
||||
|
||||
|
||||
class ThreadStateUpdateRequest(BaseModel):
|
||||
values: dict[str, Any] | None = Field(default=None, description="Channel values to merge")
|
||||
checkpoint_id: str | None = Field(default=None, description="Checkpoint to branch from")
|
||||
checkpoint: dict[str, Any] | None = Field(default=None, description="Full checkpoint object")
|
||||
as_node: str | None = Field(default=None, description="Node identity for the update")
|
||||
|
||||
|
||||
class ThreadStateResponse(BaseModel):
|
||||
values: dict[str, Any] = Field(default_factory=dict, description="Current channel values")
|
||||
next: list[str] = Field(default_factory=list, description="Next nodes to execute")
|
||||
tasks: list[dict[str, Any]] = Field(default_factory=list, description="Interrupted task details")
|
||||
checkpoint: dict[str, Any] = Field(default_factory=dict, description="Checkpoint info")
|
||||
checkpoint_id: str | None = Field(default=None, description="Current checkpoint ID")
|
||||
parent_checkpoint_id: str | None = Field(default=None, description="Parent checkpoint ID")
|
||||
metadata: dict[str, Any] = Field(default_factory=dict, description="Checkpoint metadata")
|
||||
created_at: str | None = Field(default=None, description="Checkpoint timestamp")
|
||||
|
||||
|
||||
class ThreadHistoryRequest(BaseModel):
|
||||
limit: int = Field(default=10, ge=1, le=100, description="Maximum entries")
|
||||
before: str | None = Field(default=None, description="Cursor for pagination (checkpoint_id)")
|
||||
|
||||
|
||||
class HistoryEntry(BaseModel):
|
||||
checkpoint_id: str
|
||||
parent_checkpoint_id: str | None = None
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
values: dict[str, Any] = Field(default_factory=dict)
|
||||
created_at: str | None = None
|
||||
next: list[str] = Field(default_factory=list)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def sanitize_log_param(value: str) -> str:
|
||||
"""Strip control characters to prevent log injection."""
|
||||
|
||||
return value.replace("\n", "").replace("\r", "").replace("\x00", "")
|
||||
|
||||
|
||||
def _delete_thread_data(thread_id: str, paths: Paths | None = None) -> ThreadDeleteResponse:
|
||||
"""Delete local filesystem data for a thread."""
|
||||
path_manager = paths or get_paths()
|
||||
try:
|
||||
path_manager.delete_thread_dir(thread_id)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=422, detail=str(exc)) from exc
|
||||
except FileNotFoundError:
|
||||
logger.debug("No local thread data to delete for %s", sanitize_log_param(thread_id))
|
||||
return ThreadDeleteResponse(success=True, message=f"No local data for {thread_id}")
|
||||
except Exception as exc:
|
||||
logger.exception("Failed to delete thread data for %s", sanitize_log_param(thread_id))
|
||||
raise HTTPException(status_code=500, detail="Failed to delete local thread data.") from exc
|
||||
|
||||
logger.info("Deleted local thread data for %s", sanitize_log_param(thread_id))
|
||||
return ThreadDeleteResponse(success=True, message=f"Deleted local thread data for {thread_id}")
|
||||
|
||||
|
||||
async def _thread_or_run_exists(
|
||||
*,
|
||||
request: Request,
|
||||
thread_id: str,
|
||||
thread_meta_storage: ThreadMetaStorage,
|
||||
run_repo,
|
||||
) -> bool:
|
||||
request_user_id = resolve_request_user_id(request)
|
||||
|
||||
if request_user_id is None:
|
||||
thread = await thread_meta_storage.get_thread(thread_id, user_id=None)
|
||||
if thread is not None:
|
||||
return True
|
||||
runs = await run_repo.list_by_thread(thread_id, limit=1, user_id=None)
|
||||
return bool(runs)
|
||||
|
||||
with bind_request_actor_context(request):
|
||||
thread = await thread_meta_storage.get_thread(thread_id)
|
||||
if thread is not None:
|
||||
return True
|
||||
runs = await run_repo.list_by_thread(thread_id, limit=1)
|
||||
return bool(runs)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Endpoints
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@router.post("", response_model=ThreadResponse)
|
||||
async def create_thread(
|
||||
body: ThreadCreateRequest,
|
||||
request: Request,
|
||||
thread_meta_storage: CurrentThreadMetaStorage,
|
||||
) -> ThreadResponse:
|
||||
"""Create a new thread."""
|
||||
thread_id = body.thread_id or str(uuid.uuid4())
|
||||
|
||||
request_user_id = resolve_request_user_id(request)
|
||||
if request_user_id is None:
|
||||
existing = await thread_meta_storage.get_thread(thread_id, user_id=None)
|
||||
else:
|
||||
with bind_request_actor_context(request):
|
||||
existing = await thread_meta_storage.get_thread(thread_id)
|
||||
if existing is not None:
|
||||
return ThreadResponse(
|
||||
thread_id=thread_id,
|
||||
status=existing.status,
|
||||
created_at=existing.created_time.isoformat() if existing.created_time else "",
|
||||
updated_at=existing.updated_time.isoformat() if existing.updated_time else "",
|
||||
metadata=existing.metadata,
|
||||
)
|
||||
|
||||
try:
|
||||
if request_user_id is None:
|
||||
created = await thread_meta_storage.ensure_thread(
|
||||
thread_id=thread_id,
|
||||
assistant_id=body.assistant_id,
|
||||
metadata=body.metadata,
|
||||
user_id=None,
|
||||
)
|
||||
else:
|
||||
with bind_request_actor_context(request):
|
||||
created = await thread_meta_storage.ensure_thread(
|
||||
thread_id=thread_id,
|
||||
assistant_id=body.assistant_id,
|
||||
metadata=body.metadata,
|
||||
)
|
||||
except Exception:
|
||||
logger.exception("Failed to create thread %s", sanitize_log_param(thread_id))
|
||||
raise HTTPException(status_code=500, detail="Failed to create thread")
|
||||
|
||||
logger.info("Thread created: %s", sanitize_log_param(thread_id))
|
||||
return ThreadResponse(
|
||||
thread_id=thread_id,
|
||||
status=created.status,
|
||||
created_at=created.created_time.isoformat() if created.created_time else "",
|
||||
updated_at=created.updated_time.isoformat() if created.updated_time else "",
|
||||
metadata=created.metadata,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/search", response_model=list[ThreadResponse])
|
||||
async def search_threads(
|
||||
body: ThreadSearchRequest,
|
||||
request: Request,
|
||||
thread_meta_storage: CurrentThreadMetaStorage,
|
||||
) -> list[ThreadResponse]:
|
||||
"""Search threads with filters."""
|
||||
try:
|
||||
request_user_id = resolve_request_user_id(request)
|
||||
if request_user_id is None:
|
||||
threads = await thread_meta_storage.search_threads(
|
||||
metadata=body.metadata or None,
|
||||
status=body.status,
|
||||
user_id=body.user_id,
|
||||
assistant_id=body.assistant_id,
|
||||
limit=body.limit,
|
||||
offset=body.offset,
|
||||
)
|
||||
else:
|
||||
with bind_request_actor_context(request):
|
||||
threads = await thread_meta_storage.search_threads(
|
||||
metadata=body.metadata or None,
|
||||
status=body.status,
|
||||
assistant_id=body.assistant_id,
|
||||
limit=body.limit,
|
||||
offset=body.offset,
|
||||
)
|
||||
except Exception:
|
||||
logger.exception("Failed to search threads")
|
||||
raise HTTPException(status_code=500, detail="Failed to search threads")
|
||||
|
||||
return [
|
||||
ThreadResponse(
|
||||
thread_id=t.thread_id,
|
||||
status=t.status,
|
||||
created_at=t.created_time.isoformat() if t.created_time else "",
|
||||
updated_at=t.updated_time.isoformat() if t.updated_time else "",
|
||||
metadata=t.metadata,
|
||||
values={"title": t.display_name} if t.display_name else {},
|
||||
interrupts={},
|
||||
)
|
||||
for t in threads
|
||||
]
|
||||
|
||||
|
||||
@router.delete("/{thread_id}", response_model=ThreadDeleteResponse)
|
||||
async def delete_thread(
|
||||
thread_id: str,
|
||||
checkpointer: CurrentCheckpointer,
|
||||
thread_meta_storage: CurrentThreadMetaStorage,
|
||||
) -> ThreadDeleteResponse:
|
||||
"""Delete a thread and all associated data."""
|
||||
response = _delete_thread_data(thread_id)
|
||||
|
||||
# Remove checkpoints (best-effort)
|
||||
try:
|
||||
if hasattr(checkpointer, "adelete_thread"):
|
||||
await checkpointer.adelete_thread(thread_id)
|
||||
except Exception:
|
||||
logger.debug("Could not delete checkpoints for thread %s", sanitize_log_param(thread_id))
|
||||
|
||||
# Remove thread_meta (best-effort)
|
||||
try:
|
||||
await thread_meta_storage.delete_thread(thread_id)
|
||||
except Exception:
|
||||
logger.debug("Could not delete thread_meta for %s", sanitize_log_param(thread_id))
|
||||
|
||||
return response
|
||||
|
||||
|
||||
@router.get("/{thread_id}/state", response_model=ThreadStateResponse)
|
||||
async def get_thread_state(
|
||||
thread_id: str,
|
||||
request: Request,
|
||||
checkpointer: CurrentCheckpointer,
|
||||
thread_meta_storage: CurrentThreadMetaStorage,
|
||||
run_repo: CurrentRunRepository,
|
||||
) -> ThreadStateResponse:
|
||||
"""Get the latest state snapshot for a thread."""
|
||||
config = {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}}
|
||||
|
||||
try:
|
||||
checkpoint_tuple = await checkpointer.aget_tuple(config)
|
||||
except Exception:
|
||||
logger.exception("Failed to get state for thread %s", sanitize_log_param(thread_id))
|
||||
raise HTTPException(status_code=500, detail="Failed to get thread state")
|
||||
|
||||
if checkpoint_tuple is None:
|
||||
if await _thread_or_run_exists(
|
||||
request=request,
|
||||
thread_id=thread_id,
|
||||
thread_meta_storage=thread_meta_storage,
|
||||
run_repo=run_repo,
|
||||
):
|
||||
return ThreadStateResponse()
|
||||
raise HTTPException(status_code=404, detail=f"Thread {thread_id} not found")
|
||||
|
||||
checkpoint = getattr(checkpoint_tuple, "checkpoint", {}) or {}
|
||||
metadata = getattr(checkpoint_tuple, "metadata", {}) or {}
|
||||
channel_values = checkpoint.get("channel_values", {})
|
||||
|
||||
ckpt_config = getattr(checkpoint_tuple, "config", {}) or {}
|
||||
checkpoint_id = ckpt_config.get("configurable", {}).get("checkpoint_id")
|
||||
|
||||
parent_config = getattr(checkpoint_tuple, "parent_config", None)
|
||||
parent_checkpoint_id = parent_config.get("configurable", {}).get("checkpoint_id") if parent_config else None
|
||||
|
||||
tasks_raw = getattr(checkpoint_tuple, "tasks", []) or []
|
||||
next_nodes = [t.name for t in tasks_raw if hasattr(t, "name")]
|
||||
tasks = [{"id": getattr(t, "id", ""), "name": getattr(t, "name", "")} for t in tasks_raw]
|
||||
|
||||
return ThreadStateResponse(
|
||||
values=serialize_channel_values(channel_values),
|
||||
next=next_nodes,
|
||||
tasks=tasks,
|
||||
checkpoint={"id": checkpoint_id, "ts": str(metadata.get("created_at", ""))},
|
||||
checkpoint_id=checkpoint_id,
|
||||
parent_checkpoint_id=parent_checkpoint_id,
|
||||
metadata=metadata,
|
||||
created_at=str(metadata.get("created_at", "")),
|
||||
)
|
||||
|
||||
|
||||
@router.post("/{thread_id}/state", response_model=ThreadStateResponse)
|
||||
async def update_thread_state(
|
||||
thread_id: str,
|
||||
body: ThreadStateUpdateRequest,
|
||||
checkpointer: CurrentCheckpointer,
|
||||
thread_meta_storage: CurrentThreadMetaStorage,
|
||||
) -> ThreadStateResponse:
|
||||
"""Update thread state (human-in-the-loop or title rename)."""
|
||||
read_config: dict[str, Any] = {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}}
|
||||
if body.checkpoint_id:
|
||||
read_config["configurable"]["checkpoint_id"] = body.checkpoint_id
|
||||
|
||||
try:
|
||||
checkpoint_tuple = await checkpointer.aget_tuple(read_config)
|
||||
except Exception:
|
||||
logger.exception("Failed to get state for thread %s", sanitize_log_param(thread_id))
|
||||
raise HTTPException(status_code=500, detail="Failed to get thread state")
|
||||
|
||||
if checkpoint_tuple is None:
|
||||
raise HTTPException(status_code=404, detail=f"Thread {thread_id} not found")
|
||||
|
||||
checkpoint: dict[str, Any] = dict(getattr(checkpoint_tuple, "checkpoint", {}) or {})
|
||||
metadata: dict[str, Any] = dict(getattr(checkpoint_tuple, "metadata", {}) or {})
|
||||
channel_values: dict[str, Any] = dict(checkpoint.get("channel_values", {}))
|
||||
|
||||
if body.values:
|
||||
channel_values.update(body.values)
|
||||
|
||||
checkpoint["channel_values"] = channel_values
|
||||
metadata["updated_at"] = time.time()
|
||||
|
||||
if body.as_node:
|
||||
metadata["source"] = "update"
|
||||
metadata["step"] = metadata.get("step", 0) + 1
|
||||
metadata["writes"] = {body.as_node: body.values}
|
||||
|
||||
write_config: dict[str, Any] = {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}}
|
||||
try:
|
||||
new_config = await checkpointer.aput(write_config, checkpoint, metadata, {})
|
||||
except Exception:
|
||||
logger.exception("Failed to update state for thread %s", sanitize_log_param(thread_id))
|
||||
raise HTTPException(status_code=500, detail="Failed to update thread state")
|
||||
|
||||
new_checkpoint_id: str | None = None
|
||||
if isinstance(new_config, dict):
|
||||
new_checkpoint_id = new_config.get("configurable", {}).get("checkpoint_id")
|
||||
|
||||
# Sync title to thread_meta
|
||||
if body.values and "title" in body.values:
|
||||
new_title = body.values["title"]
|
||||
if new_title:
|
||||
try:
|
||||
await thread_meta_storage.sync_thread_title(
|
||||
thread_id=thread_id,
|
||||
title=new_title,
|
||||
)
|
||||
except Exception:
|
||||
logger.debug("Failed to sync title for %s", sanitize_log_param(thread_id))
|
||||
|
||||
return ThreadStateResponse(
|
||||
values=serialize_channel_values(channel_values),
|
||||
next=[],
|
||||
metadata=metadata,
|
||||
checkpoint_id=new_checkpoint_id,
|
||||
created_at=str(metadata.get("created_at", "")),
|
||||
)
|
||||
|
||||
|
||||
@router.post("/{thread_id}/history", response_model=list[HistoryEntry])
|
||||
async def get_thread_history(
|
||||
thread_id: str,
|
||||
body: ThreadHistoryRequest,
|
||||
request: Request,
|
||||
checkpointer: CurrentCheckpointer,
|
||||
thread_meta_storage: CurrentThreadMetaStorage,
|
||||
run_repo: CurrentRunRepository,
|
||||
) -> list[HistoryEntry]:
|
||||
"""Get checkpoint history for a thread."""
|
||||
config: dict[str, Any] = {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}}
|
||||
if body.before:
|
||||
config["configurable"]["checkpoint_id"] = body.before
|
||||
|
||||
entries: list[HistoryEntry] = []
|
||||
is_first = True
|
||||
|
||||
try:
|
||||
async for checkpoint_tuple in checkpointer.alist(config, limit=body.limit):
|
||||
ckpt_config = getattr(checkpoint_tuple, "config", {}) or {}
|
||||
parent_config = getattr(checkpoint_tuple, "parent_config", None)
|
||||
metadata = getattr(checkpoint_tuple, "metadata", {}) or {}
|
||||
checkpoint = getattr(checkpoint_tuple, "checkpoint", {}) or {}
|
||||
|
||||
checkpoint_id = ckpt_config.get("configurable", {}).get("checkpoint_id", "")
|
||||
parent_id = parent_config.get("configurable", {}).get("checkpoint_id") if parent_config else None
|
||||
channel_values = checkpoint.get("channel_values", {})
|
||||
|
||||
values: dict[str, Any] = {}
|
||||
if title := channel_values.get("title"):
|
||||
values["title"] = title
|
||||
if is_first and (messages := channel_values.get("messages")):
|
||||
values["messages"] = serialize_channel_values({"messages": messages}).get("messages", [])
|
||||
is_first = False
|
||||
|
||||
tasks_raw = getattr(checkpoint_tuple, "tasks", []) or []
|
||||
next_nodes = [t.name for t in tasks_raw if hasattr(t, "name")]
|
||||
|
||||
entries.append(
|
||||
HistoryEntry(
|
||||
checkpoint_id=checkpoint_id,
|
||||
parent_checkpoint_id=parent_id,
|
||||
metadata=metadata,
|
||||
values=values,
|
||||
created_at=str(metadata.get("created_at", "")),
|
||||
next=next_nodes,
|
||||
)
|
||||
)
|
||||
except Exception:
|
||||
logger.exception("Failed to get history for thread %s", sanitize_log_param(thread_id))
|
||||
raise HTTPException(status_code=500, detail="Failed to get thread history")
|
||||
|
||||
if not entries and await _thread_or_run_exists(
|
||||
request=request,
|
||||
thread_id=thread_id,
|
||||
thread_meta_storage=thread_meta_storage,
|
||||
run_repo=run_repo,
|
||||
):
|
||||
return []
|
||||
|
||||
return entries
|
||||
@@ -1,366 +0,0 @@
|
||||
"""Memory API router for retrieving and managing global memory data."""
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Request
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.plugins.auth.security.actor_context import bind_request_actor_context
|
||||
from deerflow.agents.memory.updater import (
|
||||
clear_memory_data,
|
||||
create_memory_fact,
|
||||
delete_memory_fact,
|
||||
get_memory_data,
|
||||
import_memory_data,
|
||||
reload_memory_data,
|
||||
update_memory_fact,
|
||||
)
|
||||
from deerflow.config.memory_config import get_memory_config
|
||||
from deerflow.runtime.actor_context import get_effective_user_id
|
||||
|
||||
router = APIRouter(prefix="/api", tags=["memory"])
|
||||
|
||||
|
||||
class ContextSection(BaseModel):
|
||||
"""Model for context sections (user and history)."""
|
||||
|
||||
summary: str = Field(default="", description="Summary content")
|
||||
updatedAt: str = Field(default="", description="Last update timestamp")
|
||||
|
||||
|
||||
class UserContext(BaseModel):
|
||||
"""Model for user context."""
|
||||
|
||||
workContext: ContextSection = Field(default_factory=ContextSection)
|
||||
personalContext: ContextSection = Field(default_factory=ContextSection)
|
||||
topOfMind: ContextSection = Field(default_factory=ContextSection)
|
||||
|
||||
|
||||
class HistoryContext(BaseModel):
|
||||
"""Model for history context."""
|
||||
|
||||
recentMonths: ContextSection = Field(default_factory=ContextSection)
|
||||
earlierContext: ContextSection = Field(default_factory=ContextSection)
|
||||
longTermBackground: ContextSection = Field(default_factory=ContextSection)
|
||||
|
||||
|
||||
class Fact(BaseModel):
|
||||
"""Model for a memory fact."""
|
||||
|
||||
id: str = Field(..., description="Unique identifier for the fact")
|
||||
content: str = Field(..., description="Fact content")
|
||||
category: str = Field(default="context", description="Fact category")
|
||||
confidence: float = Field(default=0.5, description="Confidence score (0-1)")
|
||||
createdAt: str = Field(default="", description="Creation timestamp")
|
||||
source: str = Field(default="unknown", description="Source thread ID")
|
||||
sourceError: str | None = Field(default=None, description="Optional description of the prior mistake or wrong approach")
|
||||
|
||||
|
||||
class MemoryResponse(BaseModel):
|
||||
"""Response model for memory data."""
|
||||
|
||||
version: str = Field(default="1.0", description="Memory schema version")
|
||||
lastUpdated: str = Field(default="", description="Last update timestamp")
|
||||
user: UserContext = Field(default_factory=UserContext)
|
||||
history: HistoryContext = Field(default_factory=HistoryContext)
|
||||
facts: list[Fact] = Field(default_factory=list)
|
||||
|
||||
|
||||
def _map_memory_fact_value_error(exc: ValueError) -> HTTPException:
|
||||
"""Convert updater validation errors into stable API responses."""
|
||||
if exc.args and exc.args[0] == "confidence":
|
||||
detail = "Invalid confidence value; must be between 0 and 1."
|
||||
else:
|
||||
detail = "Memory fact content cannot be empty."
|
||||
return HTTPException(status_code=400, detail=detail)
|
||||
|
||||
|
||||
class FactCreateRequest(BaseModel):
|
||||
"""Request model for creating a memory fact."""
|
||||
|
||||
content: str = Field(..., min_length=1, description="Fact content")
|
||||
category: str = Field(default="context", description="Fact category")
|
||||
confidence: float = Field(default=0.5, ge=0.0, le=1.0, description="Confidence score (0-1)")
|
||||
|
||||
|
||||
class FactPatchRequest(BaseModel):
|
||||
"""PATCH request model that preserves existing values for omitted fields."""
|
||||
|
||||
content: str | None = Field(default=None, min_length=1, description="Fact content")
|
||||
category: str | None = Field(default=None, description="Fact category")
|
||||
confidence: float | None = Field(default=None, ge=0.0, le=1.0, description="Confidence score (0-1)")
|
||||
|
||||
|
||||
class MemoryConfigResponse(BaseModel):
|
||||
"""Response model for memory configuration."""
|
||||
|
||||
enabled: bool = Field(..., description="Whether memory is enabled")
|
||||
storage_path: str = Field(..., description="Path to memory storage file")
|
||||
debounce_seconds: int = Field(..., description="Debounce time for memory updates")
|
||||
max_facts: int = Field(..., description="Maximum number of facts to store")
|
||||
fact_confidence_threshold: float = Field(..., description="Minimum confidence threshold for facts")
|
||||
injection_enabled: bool = Field(..., description="Whether memory injection is enabled")
|
||||
max_injection_tokens: int = Field(..., description="Maximum tokens for memory injection")
|
||||
|
||||
|
||||
class MemoryStatusResponse(BaseModel):
|
||||
"""Response model for memory status."""
|
||||
|
||||
config: MemoryConfigResponse
|
||||
data: MemoryResponse
|
||||
|
||||
|
||||
@router.get(
|
||||
"/memory",
|
||||
response_model=MemoryResponse,
|
||||
response_model_exclude_none=True,
|
||||
summary="Get Memory Data",
|
||||
description="Retrieve the current global memory data including user context, history, and facts.",
|
||||
)
|
||||
async def get_memory(request: Request) -> MemoryResponse:
|
||||
"""Get the current global memory data.
|
||||
|
||||
Returns:
|
||||
The current memory data with user context, history, and facts.
|
||||
|
||||
Example Response:
|
||||
```json
|
||||
{
|
||||
"version": "1.0",
|
||||
"lastUpdated": "2024-01-15T10:30:00Z",
|
||||
"user": {
|
||||
"workContext": {"summary": "Working on DeerFlow project", "updatedAt": "..."},
|
||||
"personalContext": {"summary": "Prefers concise responses", "updatedAt": "..."},
|
||||
"topOfMind": {"summary": "Building memory API", "updatedAt": "..."}
|
||||
},
|
||||
"history": {
|
||||
"recentMonths": {"summary": "Recent development activities", "updatedAt": "..."},
|
||||
"earlierContext": {"summary": "", "updatedAt": ""},
|
||||
"longTermBackground": {"summary": "", "updatedAt": ""}
|
||||
},
|
||||
"facts": [
|
||||
{
|
||||
"id": "fact_abc123",
|
||||
"content": "User prefers TypeScript over JavaScript",
|
||||
"category": "preference",
|
||||
"confidence": 0.9,
|
||||
"createdAt": "2024-01-15T10:30:00Z",
|
||||
"source": "thread_xyz"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
"""
|
||||
with bind_request_actor_context(request):
|
||||
memory_data = get_memory_data(user_id=get_effective_user_id())
|
||||
return MemoryResponse(**memory_data)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/memory/reload",
|
||||
response_model=MemoryResponse,
|
||||
response_model_exclude_none=True,
|
||||
summary="Reload Memory Data",
|
||||
description="Reload memory data from the storage file, refreshing the in-memory cache.",
|
||||
)
|
||||
async def reload_memory(request: Request) -> MemoryResponse:
|
||||
"""Reload memory data from file.
|
||||
|
||||
This forces a reload of the memory data from the storage file,
|
||||
useful when the file has been modified externally.
|
||||
|
||||
Returns:
|
||||
The reloaded memory data.
|
||||
"""
|
||||
with bind_request_actor_context(request):
|
||||
memory_data = reload_memory_data(user_id=get_effective_user_id())
|
||||
return MemoryResponse(**memory_data)
|
||||
|
||||
|
||||
@router.delete(
|
||||
"/memory",
|
||||
response_model=MemoryResponse,
|
||||
response_model_exclude_none=True,
|
||||
summary="Clear All Memory Data",
|
||||
description="Delete all saved memory data and reset the memory structure to an empty state.",
|
||||
)
|
||||
async def clear_memory(request: Request) -> MemoryResponse:
|
||||
"""Clear all persisted memory data."""
|
||||
with bind_request_actor_context(request):
|
||||
try:
|
||||
memory_data = clear_memory_data(user_id=get_effective_user_id())
|
||||
except OSError as exc:
|
||||
raise HTTPException(status_code=500, detail="Failed to clear memory data.") from exc
|
||||
|
||||
return MemoryResponse(**memory_data)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/memory/facts",
|
||||
response_model=MemoryResponse,
|
||||
response_model_exclude_none=True,
|
||||
summary="Create Memory Fact",
|
||||
description="Create a single saved memory fact manually.",
|
||||
)
|
||||
async def create_memory_fact_endpoint(request: Request, payload: FactCreateRequest) -> MemoryResponse:
|
||||
"""Create a single fact manually."""
|
||||
with bind_request_actor_context(request):
|
||||
try:
|
||||
memory_data = create_memory_fact(
|
||||
content=payload.content,
|
||||
category=payload.category,
|
||||
confidence=payload.confidence,
|
||||
user_id=get_effective_user_id(),
|
||||
)
|
||||
except ValueError as exc:
|
||||
raise _map_memory_fact_value_error(exc) from exc
|
||||
except OSError as exc:
|
||||
raise HTTPException(status_code=500, detail="Failed to create memory fact.") from exc
|
||||
|
||||
return MemoryResponse(**memory_data)
|
||||
|
||||
|
||||
@router.delete(
|
||||
"/memory/facts/{fact_id}",
|
||||
response_model=MemoryResponse,
|
||||
response_model_exclude_none=True,
|
||||
summary="Delete Memory Fact",
|
||||
description="Delete a single saved memory fact by its fact id.",
|
||||
)
|
||||
async def delete_memory_fact_endpoint(fact_id: str, request: Request) -> MemoryResponse:
|
||||
"""Delete a single fact from memory by fact id."""
|
||||
with bind_request_actor_context(request):
|
||||
try:
|
||||
memory_data = delete_memory_fact(fact_id, user_id=get_effective_user_id())
|
||||
except KeyError as exc:
|
||||
raise HTTPException(status_code=404, detail=f"Memory fact '{fact_id}' not found.") from exc
|
||||
except OSError as exc:
|
||||
raise HTTPException(status_code=500, detail="Failed to delete memory fact.") from exc
|
||||
|
||||
return MemoryResponse(**memory_data)
|
||||
|
||||
|
||||
@router.patch(
|
||||
"/memory/facts/{fact_id}",
|
||||
response_model=MemoryResponse,
|
||||
response_model_exclude_none=True,
|
||||
summary="Patch Memory Fact",
|
||||
description="Partially update a single saved memory fact by its fact id while preserving omitted fields.",
|
||||
)
|
||||
async def update_memory_fact_endpoint(fact_id: str, request: Request, payload: FactPatchRequest) -> MemoryResponse:
|
||||
"""Partially update a single fact manually."""
|
||||
with bind_request_actor_context(request):
|
||||
try:
|
||||
memory_data = update_memory_fact(
|
||||
fact_id=fact_id,
|
||||
content=payload.content,
|
||||
category=payload.category,
|
||||
confidence=payload.confidence,
|
||||
user_id=get_effective_user_id(),
|
||||
)
|
||||
except ValueError as exc:
|
||||
raise _map_memory_fact_value_error(exc) from exc
|
||||
except KeyError as exc:
|
||||
raise HTTPException(status_code=404, detail=f"Memory fact '{fact_id}' not found.") from exc
|
||||
except OSError as exc:
|
||||
raise HTTPException(status_code=500, detail="Failed to update memory fact.") from exc
|
||||
|
||||
return MemoryResponse(**memory_data)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/memory/export",
|
||||
response_model=MemoryResponse,
|
||||
response_model_exclude_none=True,
|
||||
summary="Export Memory Data",
|
||||
description="Export the current global memory data as JSON for backup or transfer.",
|
||||
)
|
||||
async def export_memory(request: Request) -> MemoryResponse:
|
||||
"""Export the current memory data."""
|
||||
with bind_request_actor_context(request):
|
||||
memory_data = get_memory_data(user_id=get_effective_user_id())
|
||||
return MemoryResponse(**memory_data)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/memory/import",
|
||||
response_model=MemoryResponse,
|
||||
response_model_exclude_none=True,
|
||||
summary="Import Memory Data",
|
||||
description="Import and overwrite the current global memory data from a JSON payload.",
|
||||
)
|
||||
async def import_memory(request: Request, payload: MemoryResponse) -> MemoryResponse:
|
||||
"""Import and persist memory data."""
|
||||
with bind_request_actor_context(request):
|
||||
try:
|
||||
memory_data = import_memory_data(payload.model_dump(), user_id=get_effective_user_id())
|
||||
except OSError as exc:
|
||||
raise HTTPException(status_code=500, detail="Failed to import memory data.") from exc
|
||||
|
||||
return MemoryResponse(**memory_data)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/memory/config",
|
||||
response_model=MemoryConfigResponse,
|
||||
summary="Get Memory Configuration",
|
||||
description="Retrieve the current memory system configuration.",
|
||||
)
|
||||
async def get_memory_config_endpoint() -> MemoryConfigResponse:
|
||||
"""Get the memory system configuration.
|
||||
|
||||
Returns:
|
||||
The current memory configuration settings.
|
||||
|
||||
Example Response:
|
||||
```json
|
||||
{
|
||||
"enabled": true,
|
||||
"storage_path": ".deer-flow/memory.json",
|
||||
"debounce_seconds": 30,
|
||||
"max_facts": 100,
|
||||
"fact_confidence_threshold": 0.7,
|
||||
"injection_enabled": true,
|
||||
"max_injection_tokens": 2000
|
||||
}
|
||||
```
|
||||
"""
|
||||
config = get_memory_config()
|
||||
return MemoryConfigResponse(
|
||||
enabled=config.enabled,
|
||||
storage_path=config.storage_path,
|
||||
debounce_seconds=config.debounce_seconds,
|
||||
max_facts=config.max_facts,
|
||||
fact_confidence_threshold=config.fact_confidence_threshold,
|
||||
injection_enabled=config.injection_enabled,
|
||||
max_injection_tokens=config.max_injection_tokens,
|
||||
)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/memory/status",
|
||||
response_model=MemoryStatusResponse,
|
||||
response_model_exclude_none=True,
|
||||
summary="Get Memory Status",
|
||||
description="Retrieve both memory configuration and current data in a single request.",
|
||||
)
|
||||
async def get_memory_status(request: Request) -> MemoryStatusResponse:
|
||||
"""Get the memory system status including configuration and data.
|
||||
|
||||
Returns:
|
||||
Combined memory configuration and current data.
|
||||
"""
|
||||
with bind_request_actor_context(request):
|
||||
config = get_memory_config()
|
||||
memory_data = get_memory_data(user_id=get_effective_user_id())
|
||||
|
||||
return MemoryStatusResponse(
|
||||
config=MemoryConfigResponse(
|
||||
enabled=config.enabled,
|
||||
storage_path=config.storage_path,
|
||||
debounce_seconds=config.debounce_seconds,
|
||||
max_facts=config.max_facts,
|
||||
fact_confidence_threshold=config.fact_confidence_threshold,
|
||||
injection_enabled=config.injection_enabled,
|
||||
max_injection_tokens=config.max_injection_tokens,
|
||||
),
|
||||
data=MemoryResponse(**memory_data),
|
||||
)
|
||||
@@ -1,356 +0,0 @@
|
||||
import json
|
||||
import logging
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.gateway.path_utils import resolve_thread_virtual_path
|
||||
from deerflow.agents.lead_agent.prompt import refresh_skills_system_prompt_cache_async
|
||||
from deerflow.config.extensions_config import ExtensionsConfig, SkillStateConfig, get_extensions_config, reload_extensions_config
|
||||
from deerflow.skills import Skill, load_skills
|
||||
from deerflow.skills.installer import SkillAlreadyExistsError, install_skill_from_archive
|
||||
from deerflow.skills.manager import (
|
||||
append_history,
|
||||
atomic_write,
|
||||
custom_skill_exists,
|
||||
ensure_custom_skill_is_editable,
|
||||
get_custom_skill_dir,
|
||||
get_custom_skill_file,
|
||||
get_skill_history_file,
|
||||
read_custom_skill_content,
|
||||
read_history,
|
||||
validate_skill_markdown_content,
|
||||
)
|
||||
from deerflow.skills.security_scanner import scan_skill_content
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api", tags=["skills"])
|
||||
|
||||
|
||||
class SkillResponse(BaseModel):
|
||||
"""Response model for skill information."""
|
||||
|
||||
name: str = Field(..., description="Name of the skill")
|
||||
description: str = Field(..., description="Description of what the skill does")
|
||||
license: str | None = Field(None, description="License information")
|
||||
category: str = Field(..., description="Category of the skill (public or custom)")
|
||||
enabled: bool = Field(default=True, description="Whether this skill is enabled")
|
||||
|
||||
|
||||
class SkillsListResponse(BaseModel):
|
||||
"""Response model for listing all skills."""
|
||||
|
||||
skills: list[SkillResponse]
|
||||
|
||||
|
||||
class SkillUpdateRequest(BaseModel):
|
||||
"""Request model for updating a skill."""
|
||||
|
||||
enabled: bool = Field(..., description="Whether to enable or disable the skill")
|
||||
|
||||
|
||||
class SkillInstallRequest(BaseModel):
|
||||
"""Request model for installing a skill from a .skill file."""
|
||||
|
||||
thread_id: str = Field(..., description="The thread ID where the .skill file is located")
|
||||
path: str = Field(..., description="Virtual path to the .skill file (e.g., mnt/user-data/outputs/my-skill.skill)")
|
||||
|
||||
|
||||
class SkillInstallResponse(BaseModel):
|
||||
"""Response model for skill installation."""
|
||||
|
||||
success: bool = Field(..., description="Whether the installation was successful")
|
||||
skill_name: str = Field(..., description="Name of the installed skill")
|
||||
message: str = Field(..., description="Installation result message")
|
||||
|
||||
|
||||
class CustomSkillContentResponse(SkillResponse):
|
||||
content: str = Field(..., description="Raw SKILL.md content")
|
||||
|
||||
|
||||
class CustomSkillUpdateRequest(BaseModel):
|
||||
content: str = Field(..., description="Replacement SKILL.md content")
|
||||
|
||||
|
||||
class CustomSkillHistoryResponse(BaseModel):
|
||||
history: list[dict]
|
||||
|
||||
|
||||
class SkillRollbackRequest(BaseModel):
|
||||
history_index: int = Field(default=-1, description="History entry index to restore from, defaulting to the latest change.")
|
||||
|
||||
|
||||
def _skill_to_response(skill: Skill) -> SkillResponse:
|
||||
"""Convert a Skill object to a SkillResponse."""
|
||||
return SkillResponse(
|
||||
name=skill.name,
|
||||
description=skill.description,
|
||||
license=skill.license,
|
||||
category=skill.category,
|
||||
enabled=skill.enabled,
|
||||
)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/skills",
|
||||
response_model=SkillsListResponse,
|
||||
summary="List All Skills",
|
||||
description="Retrieve a list of all available skills from both public and custom directories.",
|
||||
)
|
||||
async def list_skills() -> SkillsListResponse:
|
||||
try:
|
||||
skills = load_skills(enabled_only=False)
|
||||
return SkillsListResponse(skills=[_skill_to_response(skill) for skill in skills])
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to load skills: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to load skills: {str(e)}")
|
||||
|
||||
|
||||
@router.post(
|
||||
"/skills/install",
|
||||
response_model=SkillInstallResponse,
|
||||
summary="Install Skill",
|
||||
description="Install a skill from a .skill file (ZIP archive) located in the thread's user-data directory.",
|
||||
)
|
||||
async def install_skill(request: SkillInstallRequest) -> SkillInstallResponse:
|
||||
try:
|
||||
skill_file_path = resolve_thread_virtual_path(request.thread_id, request.path)
|
||||
result = install_skill_from_archive(skill_file_path)
|
||||
await refresh_skills_system_prompt_cache_async()
|
||||
return SkillInstallResponse(**result)
|
||||
except FileNotFoundError as e:
|
||||
raise HTTPException(status_code=404, detail=str(e))
|
||||
except SkillAlreadyExistsError as e:
|
||||
raise HTTPException(status_code=409, detail=str(e))
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to install skill: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to install skill: {str(e)}")
|
||||
|
||||
|
||||
@router.get("/skills/custom", response_model=SkillsListResponse, summary="List Custom Skills")
|
||||
async def list_custom_skills() -> SkillsListResponse:
|
||||
try:
|
||||
skills = [skill for skill in load_skills(enabled_only=False) if skill.category == "custom"]
|
||||
return SkillsListResponse(skills=[_skill_to_response(skill) for skill in skills])
|
||||
except Exception as e:
|
||||
logger.error("Failed to list custom skills: %s", e, exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to list custom skills: {str(e)}")
|
||||
|
||||
|
||||
@router.get("/skills/custom/{skill_name}", response_model=CustomSkillContentResponse, summary="Get Custom Skill Content")
|
||||
async def get_custom_skill(skill_name: str) -> CustomSkillContentResponse:
|
||||
try:
|
||||
skills = load_skills(enabled_only=False)
|
||||
skill = next((s for s in skills if s.name == skill_name and s.category == "custom"), None)
|
||||
if skill is None:
|
||||
raise HTTPException(status_code=404, detail=f"Custom skill '{skill_name}' not found")
|
||||
return CustomSkillContentResponse(**_skill_to_response(skill).model_dump(), content=read_custom_skill_content(skill_name))
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error("Failed to get custom skill %s: %s", skill_name, e, exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to get custom skill: {str(e)}")
|
||||
|
||||
|
||||
@router.put("/skills/custom/{skill_name}", response_model=CustomSkillContentResponse, summary="Edit Custom Skill")
|
||||
async def update_custom_skill(skill_name: str, request: CustomSkillUpdateRequest) -> CustomSkillContentResponse:
|
||||
try:
|
||||
ensure_custom_skill_is_editable(skill_name)
|
||||
validate_skill_markdown_content(skill_name, request.content)
|
||||
scan = await scan_skill_content(request.content, executable=False, location=f"{skill_name}/SKILL.md")
|
||||
if scan.decision == "block":
|
||||
raise HTTPException(status_code=400, detail=f"Security scan blocked the edit: {scan.reason}")
|
||||
skill_file = get_custom_skill_dir(skill_name) / "SKILL.md"
|
||||
prev_content = skill_file.read_text(encoding="utf-8")
|
||||
atomic_write(skill_file, request.content)
|
||||
append_history(
|
||||
skill_name,
|
||||
{
|
||||
"action": "human_edit",
|
||||
"author": "human",
|
||||
"thread_id": None,
|
||||
"file_path": "SKILL.md",
|
||||
"prev_content": prev_content,
|
||||
"new_content": request.content,
|
||||
"scanner": {"decision": scan.decision, "reason": scan.reason},
|
||||
},
|
||||
)
|
||||
await refresh_skills_system_prompt_cache_async()
|
||||
return await get_custom_skill(skill_name)
|
||||
except HTTPException:
|
||||
raise
|
||||
except FileNotFoundError as e:
|
||||
raise HTTPException(status_code=404, detail=str(e))
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
except Exception as e:
|
||||
logger.error("Failed to update custom skill %s: %s", skill_name, e, exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to update custom skill: {str(e)}")
|
||||
|
||||
|
||||
@router.delete("/skills/custom/{skill_name}", summary="Delete Custom Skill")
|
||||
async def delete_custom_skill(skill_name: str) -> dict[str, bool]:
|
||||
try:
|
||||
ensure_custom_skill_is_editable(skill_name)
|
||||
skill_dir = get_custom_skill_dir(skill_name)
|
||||
prev_content = read_custom_skill_content(skill_name)
|
||||
append_history(
|
||||
skill_name,
|
||||
{
|
||||
"action": "human_delete",
|
||||
"author": "human",
|
||||
"thread_id": None,
|
||||
"file_path": "SKILL.md",
|
||||
"prev_content": prev_content,
|
||||
"new_content": None,
|
||||
"scanner": {"decision": "allow", "reason": "Deletion requested."},
|
||||
},
|
||||
)
|
||||
shutil.rmtree(skill_dir)
|
||||
await refresh_skills_system_prompt_cache_async()
|
||||
return {"success": True}
|
||||
except FileNotFoundError as e:
|
||||
raise HTTPException(status_code=404, detail=str(e))
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
except Exception as e:
|
||||
logger.error("Failed to delete custom skill %s: %s", skill_name, e, exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to delete custom skill: {str(e)}")
|
||||
|
||||
|
||||
@router.get("/skills/custom/{skill_name}/history", response_model=CustomSkillHistoryResponse, summary="Get Custom Skill History")
|
||||
async def get_custom_skill_history(skill_name: str) -> CustomSkillHistoryResponse:
|
||||
try:
|
||||
if not custom_skill_exists(skill_name) and not get_skill_history_file(skill_name).exists():
|
||||
raise HTTPException(status_code=404, detail=f"Custom skill '{skill_name}' not found")
|
||||
return CustomSkillHistoryResponse(history=read_history(skill_name))
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error("Failed to read history for %s: %s", skill_name, e, exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to read history: {str(e)}")
|
||||
|
||||
|
||||
@router.post("/skills/custom/{skill_name}/rollback", response_model=CustomSkillContentResponse, summary="Rollback Custom Skill")
|
||||
async def rollback_custom_skill(skill_name: str, request: SkillRollbackRequest) -> CustomSkillContentResponse:
|
||||
try:
|
||||
if not custom_skill_exists(skill_name) and not get_skill_history_file(skill_name).exists():
|
||||
raise HTTPException(status_code=404, detail=f"Custom skill '{skill_name}' not found")
|
||||
history = read_history(skill_name)
|
||||
if not history:
|
||||
raise HTTPException(status_code=400, detail=f"Custom skill '{skill_name}' has no history")
|
||||
record = history[request.history_index]
|
||||
target_content = record.get("prev_content")
|
||||
if target_content is None:
|
||||
raise HTTPException(status_code=400, detail="Selected history entry has no previous content to roll back to")
|
||||
validate_skill_markdown_content(skill_name, target_content)
|
||||
scan = await scan_skill_content(target_content, executable=False, location=f"{skill_name}/SKILL.md")
|
||||
skill_file = get_custom_skill_file(skill_name)
|
||||
current_content = skill_file.read_text(encoding="utf-8") if skill_file.exists() else None
|
||||
history_entry = {
|
||||
"action": "rollback",
|
||||
"author": "human",
|
||||
"thread_id": None,
|
||||
"file_path": "SKILL.md",
|
||||
"prev_content": current_content,
|
||||
"new_content": target_content,
|
||||
"rollback_from_ts": record.get("ts"),
|
||||
"scanner": {"decision": scan.decision, "reason": scan.reason},
|
||||
}
|
||||
if scan.decision == "block":
|
||||
append_history(skill_name, history_entry)
|
||||
raise HTTPException(status_code=400, detail=f"Rollback blocked by security scanner: {scan.reason}")
|
||||
atomic_write(skill_file, target_content)
|
||||
append_history(skill_name, history_entry)
|
||||
await refresh_skills_system_prompt_cache_async()
|
||||
return await get_custom_skill(skill_name)
|
||||
except HTTPException:
|
||||
raise
|
||||
except IndexError:
|
||||
raise HTTPException(status_code=400, detail="history_index is out of range")
|
||||
except FileNotFoundError as e:
|
||||
raise HTTPException(status_code=404, detail=str(e))
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
except Exception as e:
|
||||
logger.error("Failed to roll back custom skill %s: %s", skill_name, e, exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to roll back custom skill: {str(e)}")
|
||||
|
||||
|
||||
@router.get(
|
||||
"/skills/{skill_name}",
|
||||
response_model=SkillResponse,
|
||||
summary="Get Skill Details",
|
||||
description="Retrieve detailed information about a specific skill by its name.",
|
||||
)
|
||||
async def get_skill(skill_name: str) -> SkillResponse:
|
||||
try:
|
||||
skills = load_skills(enabled_only=False)
|
||||
skill = next((s for s in skills if s.name == skill_name), None)
|
||||
|
||||
if skill is None:
|
||||
raise HTTPException(status_code=404, detail=f"Skill '{skill_name}' not found")
|
||||
|
||||
return _skill_to_response(skill)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to get skill {skill_name}: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to get skill: {str(e)}")
|
||||
|
||||
|
||||
@router.put(
|
||||
"/skills/{skill_name}",
|
||||
response_model=SkillResponse,
|
||||
summary="Update Skill",
|
||||
description="Update a skill's enabled status by modifying the extensions_config.json file.",
|
||||
)
|
||||
async def update_skill(skill_name: str, request: SkillUpdateRequest) -> SkillResponse:
|
||||
try:
|
||||
skills = load_skills(enabled_only=False)
|
||||
skill = next((s for s in skills if s.name == skill_name), None)
|
||||
|
||||
if skill is None:
|
||||
raise HTTPException(status_code=404, detail=f"Skill '{skill_name}' not found")
|
||||
|
||||
config_path = ExtensionsConfig.resolve_config_path()
|
||||
if config_path is None:
|
||||
config_path = Path.cwd().parent / "extensions_config.json"
|
||||
logger.info(f"No existing extensions config found. Creating new config at: {config_path}")
|
||||
|
||||
extensions_config = get_extensions_config()
|
||||
extensions_config.skills[skill_name] = SkillStateConfig(enabled=request.enabled)
|
||||
|
||||
config_data = {
|
||||
"mcpServers": {name: server.model_dump() for name, server in extensions_config.mcp_servers.items()},
|
||||
"skills": {name: {"enabled": skill_config.enabled} for name, skill_config in extensions_config.skills.items()},
|
||||
}
|
||||
|
||||
with open(config_path, "w", encoding="utf-8") as f:
|
||||
json.dump(config_data, f, indent=2)
|
||||
|
||||
logger.info(f"Skills configuration updated and saved to: {config_path}")
|
||||
reload_extensions_config()
|
||||
await refresh_skills_system_prompt_cache_async()
|
||||
|
||||
skills = load_skills(enabled_only=False)
|
||||
updated_skill = next((s for s in skills if s.name == skill_name), None)
|
||||
|
||||
if updated_skill is None:
|
||||
raise HTTPException(status_code=500, detail=f"Failed to reload skill '{skill_name}' after update")
|
||||
|
||||
logger.info(f"Skill '{skill_name}' enabled status updated to {request.enabled}")
|
||||
return _skill_to_response(updated_skill)
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to update skill {skill_name}: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to update skill: {str(e)}")
|
||||
@@ -1,132 +0,0 @@
|
||||
import json
|
||||
import logging
|
||||
|
||||
from fastapi import APIRouter, Request
|
||||
from langchain_core.messages import HumanMessage, SystemMessage
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from deerflow.models import create_chat_model
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api", tags=["suggestions"])
|
||||
|
||||
|
||||
class SuggestionMessage(BaseModel):
|
||||
role: str = Field(..., description="Message role: user|assistant")
|
||||
content: str = Field(..., description="Message content as plain text")
|
||||
|
||||
|
||||
class SuggestionsRequest(BaseModel):
|
||||
messages: list[SuggestionMessage] = Field(..., description="Recent conversation messages")
|
||||
n: int = Field(default=3, ge=1, le=5, description="Number of suggestions to generate")
|
||||
model_name: str | None = Field(default=None, description="Optional model override")
|
||||
|
||||
|
||||
class SuggestionsResponse(BaseModel):
|
||||
suggestions: list[str] = Field(default_factory=list, description="Suggested follow-up questions")
|
||||
|
||||
|
||||
def _strip_markdown_code_fence(text: str) -> str:
|
||||
stripped = text.strip()
|
||||
if not stripped.startswith("```"):
|
||||
return stripped
|
||||
lines = stripped.splitlines()
|
||||
if len(lines) >= 3 and lines[0].startswith("```") and lines[-1].startswith("```"):
|
||||
return "\n".join(lines[1:-1]).strip()
|
||||
return stripped
|
||||
|
||||
|
||||
def _parse_json_string_list(text: str) -> list[str] | None:
|
||||
candidate = _strip_markdown_code_fence(text)
|
||||
start = candidate.find("[")
|
||||
end = candidate.rfind("]")
|
||||
if start == -1 or end == -1 or end <= start:
|
||||
return None
|
||||
candidate = candidate[start : end + 1]
|
||||
try:
|
||||
data = json.loads(candidate)
|
||||
except Exception:
|
||||
return None
|
||||
if not isinstance(data, list):
|
||||
return None
|
||||
out: list[str] = []
|
||||
for item in data:
|
||||
if not isinstance(item, str):
|
||||
continue
|
||||
s = item.strip()
|
||||
if not s:
|
||||
continue
|
||||
out.append(s)
|
||||
return out
|
||||
|
||||
|
||||
def _extract_response_text(content: object) -> str:
|
||||
if isinstance(content, str):
|
||||
return content
|
||||
if isinstance(content, list):
|
||||
parts: list[str] = []
|
||||
for block in content:
|
||||
if isinstance(block, str):
|
||||
parts.append(block)
|
||||
elif isinstance(block, dict) and block.get("type") in {"text", "output_text"}:
|
||||
text = block.get("text")
|
||||
if isinstance(text, str):
|
||||
parts.append(text)
|
||||
return "\n".join(parts) if parts else ""
|
||||
if content is None:
|
||||
return ""
|
||||
return str(content)
|
||||
|
||||
|
||||
def _format_conversation(messages: list[SuggestionMessage]) -> str:
|
||||
parts: list[str] = []
|
||||
for m in messages:
|
||||
role = m.role.strip().lower()
|
||||
if role in ("user", "human"):
|
||||
parts.append(f"User: {m.content.strip()}")
|
||||
elif role in ("assistant", "ai"):
|
||||
parts.append(f"Assistant: {m.content.strip()}")
|
||||
else:
|
||||
parts.append(f"{m.role}: {m.content.strip()}")
|
||||
return "\n".join(parts).strip()
|
||||
|
||||
|
||||
@router.post(
|
||||
"/threads/{thread_id}/suggestions",
|
||||
response_model=SuggestionsResponse,
|
||||
summary="Generate Follow-up Questions",
|
||||
description="Generate short follow-up questions a user might ask next, based on recent conversation context.",
|
||||
)
|
||||
async def generate_suggestions(thread_id: str, body: SuggestionsRequest, request: Request) -> SuggestionsResponse:
|
||||
if not body.messages:
|
||||
return SuggestionsResponse(suggestions=[])
|
||||
|
||||
n = body.n
|
||||
conversation = _format_conversation(body.messages)
|
||||
if not conversation:
|
||||
return SuggestionsResponse(suggestions=[])
|
||||
|
||||
system_instruction = (
|
||||
"You are generating follow-up questions to help the user continue the conversation.\n"
|
||||
f"Based on the conversation below, produce EXACTLY {n} short questions the user might ask next.\n"
|
||||
"Requirements:\n"
|
||||
"- Questions must be relevant to the preceding conversation.\n"
|
||||
"- Questions must be written in the same language as the user.\n"
|
||||
"- Keep each question concise (ideally <= 20 words / <= 40 Chinese characters).\n"
|
||||
"- Do NOT include numbering, markdown, or any extra text.\n"
|
||||
"- Output MUST be a JSON array of strings only.\n"
|
||||
)
|
||||
user_content = f"Conversation Context:\n{conversation}\n\nGenerate {n} follow-up questions"
|
||||
|
||||
try:
|
||||
model = create_chat_model(name=body.model_name, thinking_enabled=False)
|
||||
response = await model.ainvoke([SystemMessage(content=system_instruction), HumanMessage(content=user_content)])
|
||||
raw = _extract_response_text(response.content)
|
||||
suggestions = _parse_json_string_list(raw) or []
|
||||
cleaned = [s.replace("\n", " ").strip() for s in suggestions if s.strip()]
|
||||
cleaned = cleaned[:n]
|
||||
return SuggestionsResponse(suggestions=cleaned)
|
||||
except Exception as exc:
|
||||
logger.exception("Failed to generate suggestions: thread_id=%s err=%s", thread_id, exc)
|
||||
return SuggestionsResponse(suggestions=[])
|
||||
@@ -1,173 +0,0 @@
|
||||
"""Upload router for handling file uploads."""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import stat
|
||||
|
||||
from fastapi import APIRouter, File, HTTPException, Request, UploadFile
|
||||
from pydantic import BaseModel
|
||||
|
||||
from app.plugins.auth.security.actor_context import bind_request_actor_context
|
||||
from deerflow.sandbox.sandbox_provider import get_sandbox_provider
|
||||
from deerflow.config.paths import get_paths
|
||||
from deerflow.runtime.actor_context import get_effective_user_id
|
||||
from deerflow.uploads.manager import (
|
||||
PathTraversalError,
|
||||
delete_file_safe,
|
||||
enrich_file_listing,
|
||||
ensure_uploads_dir,
|
||||
get_uploads_dir,
|
||||
list_files_in_dir,
|
||||
normalize_filename,
|
||||
upload_artifact_url,
|
||||
upload_virtual_path,
|
||||
)
|
||||
from deerflow.utils.file_conversion import CONVERTIBLE_EXTENSIONS, convert_file_to_markdown
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/threads/{thread_id}/uploads", tags=["uploads"])
|
||||
|
||||
|
||||
class UploadResponse(BaseModel):
|
||||
"""Response model for file upload."""
|
||||
|
||||
success: bool
|
||||
files: list[dict[str, str]]
|
||||
message: str
|
||||
|
||||
|
||||
def _make_file_sandbox_writable(file_path: os.PathLike[str] | str) -> None:
|
||||
"""Ensure uploaded files remain writable when mounted into non-local sandboxes.
|
||||
|
||||
In AIO sandbox mode, the gateway writes the authoritative host-side file
|
||||
first, then the sandbox runtime may rewrite the same mounted path. Granting
|
||||
world-writable access here prevents permission mismatches between the
|
||||
gateway user and the sandbox runtime user.
|
||||
"""
|
||||
file_stat = os.lstat(file_path)
|
||||
if stat.S_ISLNK(file_stat.st_mode):
|
||||
logger.warning("Skipping sandbox chmod for symlinked upload path: %s", file_path)
|
||||
return
|
||||
|
||||
writable_mode = stat.S_IMODE(file_stat.st_mode) | stat.S_IWUSR | stat.S_IWGRP | stat.S_IWOTH
|
||||
chmod_kwargs = {"follow_symlinks": False} if os.chmod in os.supports_follow_symlinks else {}
|
||||
os.chmod(file_path, writable_mode, **chmod_kwargs)
|
||||
|
||||
|
||||
@router.post("", response_model=UploadResponse)
|
||||
async def upload_files(
|
||||
thread_id: str,
|
||||
request: Request,
|
||||
files: list[UploadFile] = File(...),
|
||||
) -> UploadResponse:
|
||||
"""Upload multiple files to a thread's uploads directory."""
|
||||
if not files:
|
||||
raise HTTPException(status_code=400, detail="No files provided")
|
||||
|
||||
with bind_request_actor_context(request):
|
||||
try:
|
||||
uploads_dir = ensure_uploads_dir(thread_id)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
sandbox_uploads = get_paths().sandbox_uploads_dir(thread_id, user_id=get_effective_user_id())
|
||||
uploaded_files = []
|
||||
|
||||
sandbox_provider = get_sandbox_provider()
|
||||
sandbox_id = sandbox_provider.acquire(thread_id)
|
||||
sandbox = sandbox_provider.get(sandbox_id)
|
||||
|
||||
for file in files:
|
||||
if not file.filename:
|
||||
continue
|
||||
|
||||
try:
|
||||
safe_filename = normalize_filename(file.filename)
|
||||
except ValueError:
|
||||
logger.warning(f"Skipping file with unsafe filename: {file.filename!r}")
|
||||
continue
|
||||
|
||||
try:
|
||||
content = await file.read()
|
||||
file_path = uploads_dir / safe_filename
|
||||
file_path.write_bytes(content)
|
||||
|
||||
virtual_path = upload_virtual_path(safe_filename)
|
||||
|
||||
if sandbox_id != "local":
|
||||
_make_file_sandbox_writable(file_path)
|
||||
sandbox.update_file(virtual_path, content)
|
||||
|
||||
file_info = {
|
||||
"filename": safe_filename,
|
||||
"size": str(len(content)),
|
||||
"path": str(sandbox_uploads / safe_filename),
|
||||
"virtual_path": virtual_path,
|
||||
"artifact_url": upload_artifact_url(thread_id, safe_filename),
|
||||
}
|
||||
|
||||
logger.info(f"Saved file: {safe_filename} ({len(content)} bytes) to {file_info['path']}")
|
||||
|
||||
file_ext = file_path.suffix.lower()
|
||||
if file_ext in CONVERTIBLE_EXTENSIONS:
|
||||
md_path = await convert_file_to_markdown(file_path)
|
||||
if md_path:
|
||||
md_virtual_path = upload_virtual_path(md_path.name)
|
||||
|
||||
if sandbox_id != "local":
|
||||
_make_file_sandbox_writable(md_path)
|
||||
sandbox.update_file(md_virtual_path, md_path.read_bytes())
|
||||
|
||||
file_info["markdown_file"] = md_path.name
|
||||
file_info["markdown_path"] = str(sandbox_uploads / md_path.name)
|
||||
file_info["markdown_virtual_path"] = md_virtual_path
|
||||
file_info["markdown_artifact_url"] = upload_artifact_url(thread_id, md_path.name)
|
||||
|
||||
uploaded_files.append(file_info)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to upload {file.filename}: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Failed to upload {file.filename}: {str(e)}")
|
||||
|
||||
return UploadResponse(
|
||||
success=True,
|
||||
files=uploaded_files,
|
||||
message=f"Successfully uploaded {len(uploaded_files)} file(s)",
|
||||
)
|
||||
|
||||
|
||||
@router.get("/list", response_model=dict)
|
||||
async def list_uploaded_files(thread_id: str, request: Request) -> dict:
|
||||
"""List all files in a thread's uploads directory."""
|
||||
with bind_request_actor_context(request):
|
||||
try:
|
||||
uploads_dir = get_uploads_dir(thread_id)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
result = list_files_in_dir(uploads_dir)
|
||||
enrich_file_listing(result, thread_id)
|
||||
|
||||
# Gateway additionally includes the sandbox-relative path.
|
||||
sandbox_uploads = get_paths().sandbox_uploads_dir(thread_id, user_id=get_effective_user_id())
|
||||
for f in result["files"]:
|
||||
f["path"] = str(sandbox_uploads / f["filename"])
|
||||
|
||||
return result
|
||||
|
||||
|
||||
@router.delete("/{filename}")
|
||||
async def delete_uploaded_file(thread_id: str, filename: str, request: Request) -> dict:
|
||||
"""Delete a file from a thread's uploads directory."""
|
||||
try:
|
||||
uploads_dir = get_uploads_dir(thread_id)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
try:
|
||||
return delete_file_safe(uploads_dir, filename, convertible_extensions=CONVERTIBLE_EXTENSIONS)
|
||||
except FileNotFoundError:
|
||||
raise HTTPException(status_code=404, detail=f"File not found: {filename}")
|
||||
except PathTraversalError:
|
||||
raise HTTPException(status_code=400, detail="Invalid path")
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to delete {filename}: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Failed to delete {filename}: {str(e)}")
|
||||
@@ -1,5 +0,0 @@
|
||||
"""Gateway service layer."""
|
||||
|
||||
"""Compatibility package for app service submodules."""
|
||||
|
||||
__all__: list[str] = []
|
||||
@@ -1,29 +0,0 @@
|
||||
"""Runs app layer services."""
|
||||
|
||||
from app.infra.storage import StorageRunObserver
|
||||
from .input import (
|
||||
AdaptedRunRequest,
|
||||
RunSpecBuilder,
|
||||
UnsupportedRunFeatureError,
|
||||
adapt_create_run_request,
|
||||
adapt_create_stream_request,
|
||||
adapt_create_wait_request,
|
||||
adapt_join_stream_request,
|
||||
adapt_join_wait_request,
|
||||
)
|
||||
from .store import AppRunCreateStore, AppRunDeleteStore, AppRunQueryStore
|
||||
|
||||
__all__ = [
|
||||
"AdaptedRunRequest",
|
||||
"AppRunCreateStore",
|
||||
"AppRunDeleteStore",
|
||||
"AppRunQueryStore",
|
||||
"RunSpecBuilder",
|
||||
"StorageRunObserver",
|
||||
"UnsupportedRunFeatureError",
|
||||
"adapt_create_run_request",
|
||||
"adapt_create_stream_request",
|
||||
"adapt_create_wait_request",
|
||||
"adapt_join_stream_request",
|
||||
"adapt_join_wait_request",
|
||||
]
|
||||
@@ -1,150 +0,0 @@
|
||||
"""Facade factory - assembles RunsFacade with dependencies."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Callable
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from fastapi import HTTPException, Request
|
||||
|
||||
from app.gateway.dependencies import get_checkpointer, get_stream_bridge
|
||||
from deerflow.runtime.runs.facade import RunsFacade
|
||||
from deerflow.runtime.runs.facade import RunsRuntime
|
||||
from deerflow.runtime.runs.internal.execution.supervisor import RunSupervisor
|
||||
from deerflow.runtime.runs.internal.planner import ExecutionPlanner
|
||||
from deerflow.runtime.runs.internal.registry import RunRegistry
|
||||
from deerflow.runtime.runs.internal.streams import RunStreamService
|
||||
from deerflow.runtime.runs.internal.wait import RunWaitService
|
||||
|
||||
from app.infra.storage import StorageRunObserver, ThreadMetaStorage
|
||||
from app.infra.storage.runs import RunDeleteRepository, RunReadRepository, RunWriteRepository
|
||||
from .store import AppRunCreateStore, AppRunDeleteStore, AppRunQueryStore
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from deerflow.runtime.stream_bridge import StreamBridge
|
||||
|
||||
|
||||
type AgentFactory = Callable[..., object]
|
||||
|
||||
|
||||
# Module-level singleton registry (shared across requests)
|
||||
_registry: RunRegistry | None = None
|
||||
_supervisor: RunSupervisor | None = None
|
||||
|
||||
|
||||
def _get_state(request: Request, attr: str, label: str):
|
||||
value = getattr(request.app.state, attr, None)
|
||||
if value is None:
|
||||
raise HTTPException(status_code=503, detail=f"{label} not available")
|
||||
return value
|
||||
|
||||
|
||||
def get_registry() -> RunRegistry:
|
||||
"""Get or create singleton registry."""
|
||||
global _registry
|
||||
if _registry is None:
|
||||
_registry = RunRegistry()
|
||||
return _registry
|
||||
|
||||
|
||||
def get_supervisor() -> RunSupervisor:
|
||||
"""Get or create singleton run supervisor."""
|
||||
global _supervisor
|
||||
if _supervisor is None:
|
||||
_supervisor = RunSupervisor()
|
||||
return _supervisor
|
||||
|
||||
|
||||
def resolve_agent_factory(assistant_id: str | None) -> AgentFactory:
|
||||
"""Resolve the agent factory callable from config."""
|
||||
from deerflow.agents.lead_agent.agent import make_lead_agent
|
||||
|
||||
return make_lead_agent
|
||||
|
||||
|
||||
def build_runs_facade(
|
||||
*,
|
||||
stream_bridge: "StreamBridge",
|
||||
checkpointer: object,
|
||||
store: object | None = None,
|
||||
run_read_repo: RunReadRepository | None = None,
|
||||
run_write_repo: RunWriteRepository | None = None,
|
||||
run_delete_repo: RunDeleteRepository | None = None,
|
||||
thread_meta_storage: ThreadMetaStorage | None = None,
|
||||
run_event_store: object | None = None,
|
||||
) -> RunsFacade:
|
||||
"""
|
||||
Build RunsFacade with all dependencies.
|
||||
|
||||
Args:
|
||||
stream_bridge: StreamBridge instance
|
||||
checkpointer: LangGraph checkpointer
|
||||
store: Optional LangGraph runtime store
|
||||
run_read_repo: Optional run repository for durable reads
|
||||
run_write_repo: Optional run repository for durable writes
|
||||
run_delete_repo: Optional run repository for durable deletes
|
||||
thread_meta_storage: Optional thread metadata storage adapter
|
||||
|
||||
Returns:
|
||||
Configured RunsFacade instance
|
||||
"""
|
||||
registry = get_registry()
|
||||
planner = ExecutionPlanner()
|
||||
supervisor = get_supervisor()
|
||||
|
||||
stream_service = RunStreamService(stream_bridge)
|
||||
wait_service = RunWaitService(stream_service)
|
||||
query_store = AppRunQueryStore(run_read_repo) if run_read_repo else None
|
||||
create_store = (
|
||||
AppRunCreateStore(run_write_repo, thread_meta_storage=thread_meta_storage)
|
||||
if run_write_repo
|
||||
else None
|
||||
)
|
||||
delete_store = AppRunDeleteStore(run_delete_repo) if run_delete_repo else None
|
||||
|
||||
# Build storage observer if repositories provided
|
||||
storage_observer = None
|
||||
if run_write_repo or thread_meta_storage:
|
||||
storage_observer = StorageRunObserver(
|
||||
run_write_repo=run_write_repo,
|
||||
thread_meta_storage=thread_meta_storage,
|
||||
)
|
||||
|
||||
return RunsFacade(
|
||||
registry=registry,
|
||||
planner=planner,
|
||||
supervisor=supervisor,
|
||||
stream_service=stream_service,
|
||||
wait_service=wait_service,
|
||||
runtime=RunsRuntime(
|
||||
bridge=stream_bridge,
|
||||
checkpointer=checkpointer,
|
||||
store=store,
|
||||
event_store=run_event_store,
|
||||
agent_factory_resolver=resolve_agent_factory,
|
||||
),
|
||||
observer=storage_observer,
|
||||
query_store=query_store,
|
||||
create_store=create_store,
|
||||
delete_store=delete_store,
|
||||
)
|
||||
|
||||
|
||||
def build_runs_facade_from_request(request: "Request") -> RunsFacade:
|
||||
"""
|
||||
Build RunsFacade from FastAPI request context.
|
||||
|
||||
Extracts dependencies from request.app.state.
|
||||
"""
|
||||
app_state = request.app.state
|
||||
|
||||
return build_runs_facade(
|
||||
stream_bridge=get_stream_bridge(request),
|
||||
checkpointer=get_checkpointer(request),
|
||||
store=getattr(request.app.state, "store", None),
|
||||
run_read_repo=getattr(app_state, "run_read_repo", None),
|
||||
run_write_repo=getattr(app_state, "run_write_repo", None),
|
||||
run_delete_repo=getattr(app_state, "run_delete_repo", None),
|
||||
thread_meta_storage=getattr(app_state, "thread_meta_storage", None),
|
||||
run_event_store=getattr(app_state, "run_event_store", None),
|
||||
)
|
||||
@@ -1,22 +0,0 @@
|
||||
"""Input adapters for app-owned runs entrypoints."""
|
||||
|
||||
from .request_adapter import (
|
||||
AdaptedRunRequest,
|
||||
adapt_create_run_request,
|
||||
adapt_create_stream_request,
|
||||
adapt_create_wait_request,
|
||||
adapt_join_stream_request,
|
||||
adapt_join_wait_request,
|
||||
)
|
||||
from .spec_builder import RunSpecBuilder, UnsupportedRunFeatureError
|
||||
|
||||
__all__ = [
|
||||
"AdaptedRunRequest",
|
||||
"RunSpecBuilder",
|
||||
"UnsupportedRunFeatureError",
|
||||
"adapt_create_run_request",
|
||||
"adapt_create_stream_request",
|
||||
"adapt_create_wait_request",
|
||||
"adapt_join_stream_request",
|
||||
"adapt_join_wait_request",
|
||||
]
|
||||
@@ -1,127 +0,0 @@
|
||||
"""App-owned request adapter for runs entrypoints."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
from deerflow.runtime.stream_bridge import JSONValue
|
||||
from deerflow.runtime.runs.types import RunIntent
|
||||
|
||||
type RequestBody = dict[str, JSONValue]
|
||||
type RequestQuery = dict[str, str]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AdaptedRunRequest:
|
||||
"""
|
||||
统一的内部请求 DTO.
|
||||
|
||||
路由层只负责提取 path/query/body，适配器负责转成稳定内部结构。
|
||||
"""
|
||||
|
||||
intent: RunIntent
|
||||
thread_id: str | None
|
||||
run_id: str | None
|
||||
body: RequestBody
|
||||
headers: dict[str, str]
|
||||
query: RequestQuery
|
||||
|
||||
@property
|
||||
def last_event_id(self) -> str | None:
|
||||
"""Extract Last-Event-ID from headers."""
|
||||
return self.headers.get("last-event-id") or self.headers.get("Last-Event-ID")
|
||||
|
||||
@property
|
||||
def is_stateless(self) -> bool:
|
||||
"""Check if this is a stateless request."""
|
||||
return self.thread_id is None
|
||||
|
||||
|
||||
def adapt_create_run_request(
|
||||
*,
|
||||
thread_id: str | None,
|
||||
body: RequestBody,
|
||||
headers: dict[str, str] | None = None,
|
||||
query: RequestQuery | None = None,
|
||||
) -> AdaptedRunRequest:
|
||||
"""Adapt POST /threads/{thread_id}/runs or POST /runs."""
|
||||
return AdaptedRunRequest(
|
||||
intent="create_background",
|
||||
thread_id=thread_id,
|
||||
run_id=None,
|
||||
body=body,
|
||||
headers=headers or {},
|
||||
query=query or {},
|
||||
)
|
||||
|
||||
|
||||
def adapt_create_stream_request(
|
||||
*,
|
||||
thread_id: str | None,
|
||||
body: RequestBody,
|
||||
headers: dict[str, str] | None = None,
|
||||
query: RequestQuery | None = None,
|
||||
) -> AdaptedRunRequest:
|
||||
"""Adapt POST /threads/{thread_id}/runs/stream or POST /runs/stream."""
|
||||
return AdaptedRunRequest(
|
||||
intent="create_and_stream",
|
||||
thread_id=thread_id,
|
||||
run_id=None,
|
||||
body=body,
|
||||
headers=headers or {},
|
||||
query=query or {},
|
||||
)
|
||||
|
||||
|
||||
def adapt_create_wait_request(
|
||||
*,
|
||||
thread_id: str | None,
|
||||
body: RequestBody,
|
||||
headers: dict[str, str] | None = None,
|
||||
query: RequestQuery | None = None,
|
||||
) -> AdaptedRunRequest:
|
||||
"""Adapt POST /threads/{thread_id}/runs/wait or POST /runs/wait."""
|
||||
return AdaptedRunRequest(
|
||||
intent="create_and_wait",
|
||||
thread_id=thread_id,
|
||||
run_id=None,
|
||||
body=body,
|
||||
headers=headers or {},
|
||||
query=query or {},
|
||||
)
|
||||
|
||||
|
||||
def adapt_join_stream_request(
|
||||
*,
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
headers: dict[str, str] | None = None,
|
||||
query: RequestQuery | None = None,
|
||||
) -> AdaptedRunRequest:
|
||||
"""Adapt GET /threads/{thread_id}/runs/{run_id}/stream."""
|
||||
return AdaptedRunRequest(
|
||||
intent="join_stream",
|
||||
thread_id=thread_id,
|
||||
run_id=run_id,
|
||||
body={},
|
||||
headers=headers or {},
|
||||
query=query or {},
|
||||
)
|
||||
|
||||
|
||||
def adapt_join_wait_request(
|
||||
*,
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
headers: dict[str, str] | None = None,
|
||||
query: RequestQuery | None = None,
|
||||
) -> AdaptedRunRequest:
|
||||
"""Adapt GET /threads/{thread_id}/runs/{run_id}/join."""
|
||||
return AdaptedRunRequest(
|
||||
intent="join_wait",
|
||||
thread_id=thread_id,
|
||||
run_id=run_id,
|
||||
body={},
|
||||
headers=headers or {},
|
||||
query=query or {},
|
||||
)
|
||||
@@ -1,254 +0,0 @@
|
||||
"""App-owned RunSpec builder."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import uuid
|
||||
|
||||
from langchain_core.messages import HumanMessage
|
||||
|
||||
from deerflow.runtime.runs.types import CheckpointRequest, RunScope, RunSpec
|
||||
from deerflow.runtime.stream_bridge import JSONValue
|
||||
|
||||
from .request_adapter import AdaptedRunRequest
|
||||
|
||||
type JSONMapping = dict[str, JSONValue]
|
||||
type GraphInput = dict[str, object]
|
||||
type RunnableConfigDict = dict[str, object]
|
||||
|
||||
|
||||
class UnsupportedRunFeatureError(ValueError):
|
||||
"""Raised when a phase1-unsupported feature is requested."""
|
||||
|
||||
pass
|
||||
|
||||
|
||||
class RunSpecBuilder:
|
||||
"""
|
||||
Build RunSpec from AdaptedRunRequest.
|
||||
|
||||
Phase 1 rules:
|
||||
1. messages-tuple normalized to messages
|
||||
2. enqueue not supported
|
||||
3. rollback not supported
|
||||
4. after_seconds not supported
|
||||
5. stream_resumable accepted
|
||||
6. stateless auto-generates temporary thread
|
||||
"""
|
||||
|
||||
# Phase 1 unsupported features
|
||||
UNSUPPORTED_MULTITASK_STRATEGIES = {"enqueue"}
|
||||
UNSUPPORTED_ACTIONS = {"rollback"}
|
||||
|
||||
# Default stream modes
|
||||
DEFAULT_STREAM_MODES = ["values", "messages"]
|
||||
CONTEXT_CONFIGURABLE_KEYS = frozenset({
|
||||
"model_name",
|
||||
"mode",
|
||||
"thinking_enabled",
|
||||
"reasoning_effort",
|
||||
"is_plan_mode",
|
||||
"subagent_enabled",
|
||||
"max_concurrent_subagents",
|
||||
})
|
||||
DEFAULT_ASSISTANT_ID = "lead_agent"
|
||||
|
||||
@staticmethod
|
||||
def _as_json_mapping(value: JSONValue | None) -> JSONMapping | None:
|
||||
return value if isinstance(value, dict) else None
|
||||
|
||||
@staticmethod
|
||||
def _as_string_list(value: JSONValue | None) -> list[str] | None:
|
||||
if not isinstance(value, list):
|
||||
return None
|
||||
return [item for item in value if isinstance(item, str)]
|
||||
|
||||
def build(self, request: AdaptedRunRequest) -> RunSpec:
|
||||
"""Build RunSpec from adapted request."""
|
||||
body = request.body
|
||||
|
||||
# Validate phase1 constraints
|
||||
self._validate_constraints(body)
|
||||
|
||||
# Build scope
|
||||
scope = self._build_scope(request)
|
||||
|
||||
# Normalize stream modes
|
||||
stream_modes = self._normalize_stream_modes(body.get("stream_mode"))
|
||||
|
||||
# Build checkpoint request
|
||||
checkpoint_request = self._build_checkpoint_request(body)
|
||||
|
||||
config = self._build_runnable_config(
|
||||
thread_id=scope.thread_id,
|
||||
request_config=self._as_json_mapping(body.get("config")),
|
||||
metadata=self._as_json_mapping(body.get("metadata")),
|
||||
assistant_id=body.get("assistant_id"),
|
||||
context=self._as_json_mapping(body.get("context")),
|
||||
)
|
||||
|
||||
return RunSpec(
|
||||
intent=request.intent,
|
||||
scope=scope,
|
||||
assistant_id=body.get("assistant_id") if isinstance(body.get("assistant_id"), str) else None,
|
||||
input=self._normalize_input(self._as_json_mapping(body.get("input"))),
|
||||
command=self._as_json_mapping(body.get("command")),
|
||||
runnable_config=config,
|
||||
context=self._as_json_mapping(body.get("context")),
|
||||
metadata=self._as_json_mapping(body.get("metadata")) or {},
|
||||
stream_modes=stream_modes,
|
||||
stream_subgraphs=bool(body.get("stream_subgraphs", False)),
|
||||
stream_resumable=bool(body.get("stream_resumable", False)),
|
||||
on_disconnect=body.get("on_disconnect", "cancel") if body.get("on_disconnect") in {"cancel", "continue"} else "cancel",
|
||||
on_completion=body.get("on_completion", "keep") if body.get("on_completion") in {"delete", "keep"} else "keep",
|
||||
multitask_strategy=body.get("multitask_strategy", "reject") if body.get("multitask_strategy") in {"reject", "interrupt"} else "reject",
|
||||
interrupt_before="*" if body.get("interrupt_before") == "*" else self._as_string_list(body.get("interrupt_before")),
|
||||
interrupt_after="*" if body.get("interrupt_after") == "*" else self._as_string_list(body.get("interrupt_after")),
|
||||
checkpoint_request=checkpoint_request,
|
||||
follow_up_to_run_id=body.get("follow_up_to_run_id") if isinstance(body.get("follow_up_to_run_id"), str) else None,
|
||||
webhook=body.get("webhook") if isinstance(body.get("webhook"), str) else None,
|
||||
feedback_keys=self._as_string_list(body.get("feedback_keys")),
|
||||
)
|
||||
|
||||
def _validate_constraints(self, body: JSONMapping) -> None:
|
||||
"""Validate phase1 constraints, raise UnsupportedRunFeatureError if violated."""
|
||||
# Check multitask_strategy
|
||||
strategy = body.get("multitask_strategy", "reject")
|
||||
if strategy in self.UNSUPPORTED_MULTITASK_STRATEGIES:
|
||||
raise UnsupportedRunFeatureError(
|
||||
f"multitask_strategy '{strategy}' is not supported in phase1. "
|
||||
f"Supported: reject, interrupt"
|
||||
)
|
||||
|
||||
# Check for rollback action
|
||||
command = self._as_json_mapping(body.get("command")) or {}
|
||||
if command.get("action") in self.UNSUPPORTED_ACTIONS:
|
||||
raise UnsupportedRunFeatureError(
|
||||
f"action '{command.get('action')}' is not supported in phase1"
|
||||
)
|
||||
|
||||
# Check for after_seconds
|
||||
if body.get("after_seconds") is not None:
|
||||
raise UnsupportedRunFeatureError("after_seconds is not supported in phase1")
|
||||
|
||||
def _build_scope(self, request: AdaptedRunRequest) -> RunScope:
|
||||
"""Build RunScope from request."""
|
||||
if request.is_stateless:
|
||||
# Stateless: generate temporary thread
|
||||
return RunScope(
|
||||
kind="stateless",
|
||||
thread_id=str(uuid.uuid4()),
|
||||
temporary=True,
|
||||
)
|
||||
else:
|
||||
assert request.thread_id is not None
|
||||
return RunScope(
|
||||
kind="stateful",
|
||||
thread_id=request.thread_id,
|
||||
temporary=False,
|
||||
)
|
||||
|
||||
def _normalize_stream_modes(self, stream_mode: JSONValue | None) -> list[str]:
|
||||
"""Normalize stream_mode to list, convert messages-tuple to messages."""
|
||||
if stream_mode is None:
|
||||
return self.DEFAULT_STREAM_MODES.copy()
|
||||
|
||||
if isinstance(stream_mode, str):
|
||||
modes = [stream_mode]
|
||||
elif isinstance(stream_mode, list):
|
||||
modes = [mode for mode in stream_mode if isinstance(mode, str)]
|
||||
else:
|
||||
return self.DEFAULT_STREAM_MODES.copy()
|
||||
|
||||
return ["messages" if m == "messages-tuple" else m for m in modes]
|
||||
|
||||
def _build_checkpoint_request(self, body: JSONMapping) -> CheckpointRequest | None:
|
||||
"""Build CheckpointRequest if checkpoint data is provided."""
|
||||
checkpoint_id = body.get("checkpoint_id")
|
||||
checkpoint = self._as_json_mapping(body.get("checkpoint"))
|
||||
|
||||
if not isinstance(checkpoint_id, str) and checkpoint is None:
|
||||
return None
|
||||
|
||||
return CheckpointRequest(
|
||||
checkpoint_id=checkpoint_id if isinstance(checkpoint_id, str) else None,
|
||||
checkpoint=checkpoint,
|
||||
)
|
||||
|
||||
def _normalize_input(self, raw_input: JSONMapping | None) -> GraphInput | None:
|
||||
"""Convert HTTP-friendly message dicts into LangChain message objects."""
|
||||
if raw_input is None:
|
||||
return None
|
||||
|
||||
messages = raw_input.get("messages")
|
||||
if not messages or not isinstance(messages, list):
|
||||
return raw_input
|
||||
|
||||
converted: list[object] = []
|
||||
for msg in messages:
|
||||
if isinstance(msg, dict):
|
||||
role = msg.get("role", msg.get("type", "user"))
|
||||
content = msg.get("content", "")
|
||||
if role in ("user", "human"):
|
||||
converted.append(HumanMessage(content=content))
|
||||
else:
|
||||
converted.append(HumanMessage(content=content))
|
||||
else:
|
||||
converted.append(msg)
|
||||
return {**raw_input, "messages": converted}
|
||||
|
||||
def _build_runnable_config(
|
||||
self,
|
||||
*,
|
||||
thread_id: str,
|
||||
request_config: JSONMapping | None,
|
||||
metadata: JSONMapping | None,
|
||||
assistant_id: str | None,
|
||||
context: JSONMapping | None,
|
||||
) -> RunnableConfigDict:
|
||||
"""Build RunnableConfig from request payload and app-side rules."""
|
||||
config: RunnableConfigDict = {"recursion_limit": 100}
|
||||
|
||||
if request_config:
|
||||
if "context" in request_config:
|
||||
config["context"] = request_config["context"]
|
||||
else:
|
||||
configurable = {"thread_id": thread_id}
|
||||
raw_configurable = request_config.get("configurable")
|
||||
if isinstance(raw_configurable, dict):
|
||||
configurable.update(raw_configurable)
|
||||
config["configurable"] = configurable
|
||||
|
||||
for key, value in request_config.items():
|
||||
if key not in ("configurable", "context"):
|
||||
config[key] = value
|
||||
else:
|
||||
config["configurable"] = {"thread_id": thread_id}
|
||||
|
||||
configurable = config.get("configurable")
|
||||
if (
|
||||
assistant_id
|
||||
and assistant_id != self.DEFAULT_ASSISTANT_ID
|
||||
and isinstance(configurable, dict)
|
||||
and "agent_name" not in configurable
|
||||
):
|
||||
normalized = assistant_id.strip().lower().replace("_", "-")
|
||||
if not normalized or not re.fullmatch(r"[a-z0-9-]+", normalized):
|
||||
raise ValueError(
|
||||
f"Invalid assistant_id {assistant_id!r}: must contain only letters, digits, and hyphens after normalization."
|
||||
)
|
||||
configurable["agent_name"] = normalized
|
||||
|
||||
if metadata:
|
||||
existing_metadata = config.get("metadata")
|
||||
if isinstance(existing_metadata, dict):
|
||||
existing_metadata.update(metadata)
|
||||
else:
|
||||
config["metadata"] = dict(metadata)
|
||||
|
||||
if context and isinstance(configurable, dict):
|
||||
for key in self.CONTEXT_CONFIGURABLE_KEYS:
|
||||
if key in context:
|
||||
configurable.setdefault(key, context[key])
|
||||
|
||||
return config
|
||||
@@ -1,5 +0,0 @@
|
||||
"""Compatibility wrapper for the app-owned storage observer."""
|
||||
|
||||
from app.infra.storage.runs import StorageRunObserver
|
||||
|
||||
__all__ = ["StorageRunObserver"]
|
||||
@@ -1,11 +0,0 @@
|
||||
"""App-owned runs store adapters."""
|
||||
|
||||
from .create_store import AppRunCreateStore
|
||||
from .delete_store import AppRunDeleteStore
|
||||
from .query_store import AppRunQueryStore
|
||||
|
||||
__all__ = [
|
||||
"AppRunCreateStore",
|
||||
"AppRunDeleteStore",
|
||||
"AppRunQueryStore",
|
||||
]
|
||||
@@ -1,38 +0,0 @@
|
||||
"""App-owned durable run creation adapter."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from deerflow.runtime.runs.store import RunCreateStore
|
||||
from deerflow.runtime.runs.types import RunRecord
|
||||
|
||||
from app.infra.storage import ThreadMetaStorage
|
||||
from app.infra.storage.runs import RunWriteRepository
|
||||
|
||||
|
||||
class AppRunCreateStore(RunCreateStore):
|
||||
"""Write the initial durable row for a newly created run."""
|
||||
|
||||
def __init__(self, repo: RunWriteRepository, thread_meta_storage: ThreadMetaStorage | None = None) -> None:
|
||||
self._repo = repo
|
||||
self._thread_meta_storage = thread_meta_storage
|
||||
|
||||
async def create_run(self, record: RunRecord) -> None:
|
||||
await self._repo.create(
|
||||
run_id=record.run_id,
|
||||
thread_id=record.thread_id,
|
||||
assistant_id=record.assistant_id,
|
||||
status=str(record.status),
|
||||
metadata=record.metadata,
|
||||
follow_up_to_run_id=record.follow_up_to_run_id,
|
||||
created_at=record.created_at,
|
||||
)
|
||||
if self._thread_meta_storage is not None and record.assistant_id:
|
||||
thread = await self._thread_meta_storage.ensure_thread(
|
||||
thread_id=record.thread_id,
|
||||
assistant_id=record.assistant_id,
|
||||
)
|
||||
if thread.assistant_id != record.assistant_id:
|
||||
await self._thread_meta_storage.sync_thread_assistant_id(
|
||||
thread_id=record.thread_id,
|
||||
assistant_id=record.assistant_id,
|
||||
)
|
||||
@@ -1,17 +0,0 @@
|
||||
"""App-owned durable run deletion adapter."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from deerflow.runtime.runs.store import RunDeleteStore
|
||||
|
||||
from app.infra.storage.runs import RunDeleteRepository
|
||||
|
||||
|
||||
class AppRunDeleteStore(RunDeleteStore):
|
||||
"""Delete durable run rows via the app storage adapter."""
|
||||
|
||||
def __init__(self, repo: RunDeleteRepository) -> None:
|
||||
self._repo = repo
|
||||
|
||||
async def delete_run(self, run_id: str) -> bool:
|
||||
return await self._repo.delete(run_id)
|
||||
@@ -1,47 +0,0 @@
|
||||
"""App-owned durable run query adapter."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from deerflow.runtime.runs.store import RunQueryStore
|
||||
from deerflow.runtime.runs.types import RunRecord, RunStatus
|
||||
|
||||
from app.infra.storage.runs import RunReadRepository, RunRow
|
||||
|
||||
|
||||
class AppRunQueryStore(RunQueryStore):
|
||||
"""Map app-side durable run rows into harness RunRecord DTOs."""
|
||||
|
||||
def __init__(self, repo: RunReadRepository) -> None:
|
||||
self._repo = repo
|
||||
|
||||
async def get_run(self, run_id: str) -> RunRecord | None:
|
||||
row = await self._repo.get(run_id)
|
||||
if row is None:
|
||||
return None
|
||||
return self._to_run_record(row)
|
||||
|
||||
async def list_runs(
|
||||
self,
|
||||
thread_id: str,
|
||||
*,
|
||||
limit: int = 100,
|
||||
) -> list[RunRecord]:
|
||||
rows = await self._repo.list_by_thread(thread_id, limit=limit)
|
||||
return [self._to_run_record(row) for row in rows]
|
||||
|
||||
def _to_run_record(self, row: RunRow) -> RunRecord:
|
||||
return RunRecord(
|
||||
run_id=row["run_id"],
|
||||
thread_id=row["thread_id"],
|
||||
assistant_id=row.get("assistant_id"),
|
||||
status=RunStatus(row.get("status", "pending")),
|
||||
temporary=False,
|
||||
multitask_strategy=row.get("multitask_strategy", "reject"),
|
||||
metadata=row.get("metadata", {}),
|
||||
follow_up_to_run_id=row.get("follow_up_to_run_id"),
|
||||
created_at=row.get("created_at", ""),
|
||||
updated_at=row.get("updated_at", ""),
|
||||
started_at=row.get("started_at"),
|
||||
ended_at=row.get("ended_at"),
|
||||
error=row.get("error"),
|
||||
)
|
||||
@@ -1 +0,0 @@
|
||||
"""Application-owned infrastructure adapters and wiring."""
|
||||
@@ -1,6 +0,0 @@
|
||||
"""Run event store backends owned by app infrastructure."""
|
||||
|
||||
from .factory import build_run_event_store
|
||||
from .jsonl_store import JsonlRunEventStore
|
||||
|
||||
__all__ = ["JsonlRunEventStore", "build_run_event_store"]
|
||||
@@ -1,25 +0,0 @@
|
||||
"""Factory for app-owned run event store backends."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
|
||||
from app.infra.storage import AppRunEventStore
|
||||
from deerflow.config import get_app_config
|
||||
|
||||
from .jsonl_store import JsonlRunEventStore
|
||||
|
||||
|
||||
def build_run_event_store(session_factory: async_sessionmaker[AsyncSession]) -> AppRunEventStore | JsonlRunEventStore:
|
||||
"""Build the run event store selected by app configuration."""
|
||||
|
||||
config = get_app_config().run_events
|
||||
if config.backend == "db":
|
||||
return AppRunEventStore(session_factory)
|
||||
if config.backend == "jsonl":
|
||||
return JsonlRunEventStore(
|
||||
base_dir=Path(config.jsonl_base_dir),
|
||||
)
|
||||
raise ValueError(f"Unsupported run event backend: {config.backend}")
|
||||
@@ -1,210 +0,0 @@
|
||||
"""JSONL run event store backend owned by app infrastructure."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import shutil
|
||||
from collections.abc import Iterable
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
class JsonlRunEventStore:
|
||||
"""Append-only JSONL implementation of the runs RunEventStore protocol."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
base_dir: Path | str = ".deer-flow/run-events",
|
||||
) -> None:
|
||||
self._base_dir = Path(base_dir)
|
||||
self._locks: dict[str, asyncio.Lock] = {}
|
||||
self._locks_guard = asyncio.Lock()
|
||||
|
||||
async def put_batch(self, events: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
if not events:
|
||||
return []
|
||||
|
||||
grouped: dict[str, list[dict[str, Any]]] = {}
|
||||
for event in events:
|
||||
grouped.setdefault(str(event["thread_id"]), []).append(event)
|
||||
|
||||
records_by_thread: dict[str, list[dict[str, Any]]] = {}
|
||||
for thread_id, thread_events in grouped.items():
|
||||
async with await self._thread_lock(thread_id):
|
||||
records_by_thread[thread_id] = self._append_thread_events(thread_id, thread_events)
|
||||
|
||||
indexes = {thread_id: 0 for thread_id in records_by_thread}
|
||||
ordered: list[dict[str, Any]] = []
|
||||
for event in events:
|
||||
thread_id = str(event["thread_id"])
|
||||
index = indexes[thread_id]
|
||||
ordered.append(records_by_thread[thread_id][index])
|
||||
indexes[thread_id] = index + 1
|
||||
return ordered
|
||||
|
||||
async def list_messages(
|
||||
self,
|
||||
thread_id: str,
|
||||
*,
|
||||
limit: int = 50,
|
||||
before_seq: int | None = None,
|
||||
after_seq: int | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
events = [event for event in await self._read_thread_events(thread_id) if event.get("category") == "message"]
|
||||
if before_seq is not None:
|
||||
events = [event for event in events if int(event["seq"]) < before_seq]
|
||||
return events[-limit:]
|
||||
if after_seq is not None:
|
||||
events = [event for event in events if int(event["seq"]) > after_seq]
|
||||
return events[:limit]
|
||||
return events[-limit:]
|
||||
|
||||
async def list_events(
|
||||
self,
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
*,
|
||||
event_types: list[str] | None = None,
|
||||
limit: int = 500,
|
||||
) -> list[dict[str, Any]]:
|
||||
event_type_set = set(event_types or [])
|
||||
events = [
|
||||
event
|
||||
for event in await self._read_thread_events(thread_id)
|
||||
if event.get("run_id") == run_id and (not event_type_set or event.get("event_type") in event_type_set)
|
||||
]
|
||||
return events[:limit]
|
||||
|
||||
async def list_messages_by_run(
|
||||
self,
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
*,
|
||||
limit: int = 50,
|
||||
before_seq: int | None = None,
|
||||
after_seq: int | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
events = [
|
||||
event
|
||||
for event in await self._read_thread_events(thread_id)
|
||||
if event.get("run_id") == run_id and event.get("category") == "message"
|
||||
]
|
||||
if before_seq is not None:
|
||||
events = [event for event in events if int(event["seq"]) < before_seq]
|
||||
return events[-limit:]
|
||||
if after_seq is not None:
|
||||
events = [event for event in events if int(event["seq"]) > after_seq]
|
||||
return events[:limit]
|
||||
return events[-limit:]
|
||||
|
||||
async def count_messages(self, thread_id: str) -> int:
|
||||
return len(await self.list_messages(thread_id, limit=10**9))
|
||||
|
||||
async def delete_by_thread(self, thread_id: str) -> int:
|
||||
async with await self._thread_lock(thread_id):
|
||||
count = len(self._read_thread_events_sync(thread_id))
|
||||
shutil.rmtree(self._thread_dir(thread_id), ignore_errors=True)
|
||||
return count
|
||||
|
||||
async def delete_by_run(self, thread_id: str, run_id: str) -> int:
|
||||
async with await self._thread_lock(thread_id):
|
||||
events = self._read_thread_events_sync(thread_id)
|
||||
kept = [event for event in events if event.get("run_id") != run_id]
|
||||
deleted = len(events) - len(kept)
|
||||
if deleted:
|
||||
self._write_thread_events(thread_id, kept)
|
||||
return deleted
|
||||
|
||||
async def _thread_lock(self, thread_id: str) -> asyncio.Lock:
|
||||
async with self._locks_guard:
|
||||
lock = self._locks.get(thread_id)
|
||||
if lock is None:
|
||||
lock = asyncio.Lock()
|
||||
self._locks[thread_id] = lock
|
||||
return lock
|
||||
|
||||
def _append_thread_events(self, thread_id: str, events: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
thread_dir = self._thread_dir(thread_id)
|
||||
thread_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
seq = self._read_seq(thread_id)
|
||||
records: list[dict[str, Any]] = []
|
||||
with self._events_path(thread_id).open("a", encoding="utf-8") as file:
|
||||
for event in events:
|
||||
seq += 1
|
||||
record = self._normalize_event(event, seq=seq)
|
||||
file.write(json.dumps(record, ensure_ascii=False, default=str))
|
||||
file.write("\n")
|
||||
records.append(record)
|
||||
self._write_seq(thread_id, seq)
|
||||
return records
|
||||
|
||||
def _normalize_event(self, event: dict[str, Any], *, seq: int) -> dict[str, Any]:
|
||||
created_at = event.get("created_at")
|
||||
if isinstance(created_at, datetime):
|
||||
created_at_value = created_at.isoformat()
|
||||
elif created_at:
|
||||
created_at_value = str(created_at)
|
||||
else:
|
||||
created_at_value = datetime.now(UTC).isoformat()
|
||||
|
||||
return {
|
||||
"thread_id": str(event["thread_id"]),
|
||||
"run_id": str(event["run_id"]),
|
||||
"seq": seq,
|
||||
"event_type": str(event["event_type"]),
|
||||
"category": str(event["category"]),
|
||||
"content": event.get("content", ""),
|
||||
"metadata": dict(event.get("metadata") or {}),
|
||||
"created_at": created_at_value,
|
||||
}
|
||||
|
||||
async def _read_thread_events(self, thread_id: str) -> list[dict[str, Any]]:
|
||||
async with await self._thread_lock(thread_id):
|
||||
return self._read_thread_events_sync(thread_id)
|
||||
|
||||
def _read_thread_events_sync(self, thread_id: str) -> list[dict[str, Any]]:
|
||||
path = self._events_path(thread_id)
|
||||
if not path.exists():
|
||||
return []
|
||||
|
||||
events: list[dict[str, Any]] = []
|
||||
with path.open(encoding="utf-8") as file:
|
||||
for line in file:
|
||||
stripped = line.strip()
|
||||
if stripped:
|
||||
events.append(json.loads(stripped))
|
||||
return events
|
||||
|
||||
def _write_thread_events(self, thread_id: str, events: Iterable[dict[str, Any]]) -> None:
|
||||
thread_dir = self._thread_dir(thread_id)
|
||||
thread_dir.mkdir(parents=True, exist_ok=True)
|
||||
temp_path = self._events_path(thread_id).with_suffix(".jsonl.tmp")
|
||||
with temp_path.open("w", encoding="utf-8") as file:
|
||||
for event in events:
|
||||
file.write(json.dumps(event, ensure_ascii=False, default=str))
|
||||
file.write("\n")
|
||||
temp_path.replace(self._events_path(thread_id))
|
||||
|
||||
def _read_seq(self, thread_id: str) -> int:
|
||||
path = self._seq_path(thread_id)
|
||||
if not path.exists():
|
||||
return 0
|
||||
try:
|
||||
return int(path.read_text(encoding="utf-8").strip() or "0")
|
||||
except ValueError:
|
||||
return 0
|
||||
|
||||
def _write_seq(self, thread_id: str, seq: int) -> None:
|
||||
self._seq_path(thread_id).write_text(str(seq), encoding="utf-8")
|
||||
|
||||
def _thread_dir(self, thread_id: str) -> Path:
|
||||
return self._base_dir / "threads" / thread_id
|
||||
|
||||
def _events_path(self, thread_id: str) -> Path:
|
||||
return self._thread_dir(thread_id) / "events.jsonl"
|
||||
|
||||
def _seq_path(self, thread_id: str) -> Path:
|
||||
return self._thread_dir(thread_id) / "seq"
|
||||
@@ -1,14 +0,0 @@
|
||||
"""Storage-facing adapters owned by the app layer."""
|
||||
|
||||
from .run_events import AppRunEventStore
|
||||
from .runs import FeedbackStoreAdapter, RunStoreAdapter, StorageRunObserver
|
||||
from .thread_meta import ThreadMetaStorage, ThreadMetaStoreAdapter
|
||||
|
||||
__all__ = [
|
||||
"AppRunEventStore",
|
||||
"FeedbackStoreAdapter",
|
||||
"RunStoreAdapter",
|
||||
"StorageRunObserver",
|
||||
"ThreadMetaStorage",
|
||||
"ThreadMetaStoreAdapter",
|
||||
]
|
||||
@@ -1,166 +0,0 @@
|
||||
"""App-owned adapter from runs callbacks to storage run event repository."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
from store.repositories import RunEvent, RunEventCreate, build_run_event_repository, build_thread_meta_repository
|
||||
|
||||
from deerflow.runtime.actor_context import get_actor_context
|
||||
|
||||
|
||||
class AppRunEventStore:
|
||||
"""Implements the harness RunEventStore protocol using storage repositories."""
|
||||
|
||||
def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:
|
||||
self._session_factory = session_factory
|
||||
|
||||
async def put_batch(self, events: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
if not events:
|
||||
return []
|
||||
|
||||
denied = {str(event["thread_id"]) for event in events if not await self._thread_visible(str(event["thread_id"]))}
|
||||
if denied:
|
||||
raise PermissionError(f"actor is not allowed to append events for thread(s): {', '.join(sorted(denied))}")
|
||||
|
||||
async with self._session_factory() as session:
|
||||
try:
|
||||
repo = build_run_event_repository(session)
|
||||
rows = await repo.append_batch([_event_create_from_dict(event) for event in events])
|
||||
await session.commit()
|
||||
except Exception:
|
||||
await session.rollback()
|
||||
raise
|
||||
|
||||
return [_event_to_dict(row) for row in rows]
|
||||
|
||||
async def list_messages(
|
||||
self,
|
||||
thread_id: str,
|
||||
*,
|
||||
limit: int = 50,
|
||||
before_seq: int | None = None,
|
||||
after_seq: int | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
if not await self._thread_visible(thread_id):
|
||||
return []
|
||||
async with self._session_factory() as session:
|
||||
repo = build_run_event_repository(session)
|
||||
rows = await repo.list_messages(
|
||||
thread_id,
|
||||
limit=limit,
|
||||
before_seq=before_seq,
|
||||
after_seq=after_seq,
|
||||
)
|
||||
return [_event_to_dict(row) for row in rows]
|
||||
|
||||
async def list_events(
|
||||
self,
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
*,
|
||||
event_types: list[str] | None = None,
|
||||
limit: int = 500,
|
||||
) -> list[dict[str, Any]]:
|
||||
if not await self._thread_visible(thread_id):
|
||||
return []
|
||||
async with self._session_factory() as session:
|
||||
repo = build_run_event_repository(session)
|
||||
rows = await repo.list_events(thread_id, run_id, event_types=event_types, limit=limit)
|
||||
return [_event_to_dict(row) for row in rows]
|
||||
|
||||
async def list_messages_by_run(
|
||||
self,
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
*,
|
||||
limit: int = 50,
|
||||
before_seq: int | None = None,
|
||||
after_seq: int | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
if not await self._thread_visible(thread_id):
|
||||
return []
|
||||
async with self._session_factory() as session:
|
||||
repo = build_run_event_repository(session)
|
||||
rows = await repo.list_messages_by_run(
|
||||
thread_id,
|
||||
run_id,
|
||||
limit=limit,
|
||||
before_seq=before_seq,
|
||||
after_seq=after_seq,
|
||||
)
|
||||
return [_event_to_dict(row) for row in rows]
|
||||
|
||||
async def count_messages(self, thread_id: str) -> int:
|
||||
if not await self._thread_visible(thread_id):
|
||||
return 0
|
||||
async with self._session_factory() as session:
|
||||
repo = build_run_event_repository(session)
|
||||
return await repo.count_messages(thread_id)
|
||||
|
||||
async def delete_by_thread(self, thread_id: str) -> int:
|
||||
if not await self._thread_visible(thread_id):
|
||||
return 0
|
||||
async with self._session_factory() as session:
|
||||
try:
|
||||
repo = build_run_event_repository(session)
|
||||
count = await repo.delete_by_thread(thread_id)
|
||||
await session.commit()
|
||||
except Exception:
|
||||
await session.rollback()
|
||||
raise
|
||||
return count
|
||||
|
||||
async def delete_by_run(self, thread_id: str, run_id: str) -> int:
|
||||
if not await self._thread_visible(thread_id):
|
||||
return 0
|
||||
async with self._session_factory() as session:
|
||||
try:
|
||||
repo = build_run_event_repository(session)
|
||||
count = await repo.delete_by_run(thread_id, run_id)
|
||||
await session.commit()
|
||||
except Exception:
|
||||
await session.rollback()
|
||||
raise
|
||||
return count
|
||||
|
||||
async def _thread_visible(self, thread_id: str) -> bool:
|
||||
actor = get_actor_context()
|
||||
if actor is None or actor.user_id is None:
|
||||
return True
|
||||
|
||||
async with self._session_factory() as session:
|
||||
thread_repo = build_thread_meta_repository(session)
|
||||
thread = await thread_repo.get_thread_meta(thread_id)
|
||||
|
||||
if thread is None:
|
||||
return True
|
||||
return thread.user_id is None or thread.user_id == actor.user_id
|
||||
|
||||
|
||||
def _event_create_from_dict(event: dict[str, Any]) -> RunEventCreate:
|
||||
created_at = event.get("created_at")
|
||||
return RunEventCreate(
|
||||
thread_id=str(event["thread_id"]),
|
||||
run_id=str(event["run_id"]),
|
||||
event_type=str(event["event_type"]),
|
||||
category=str(event["category"]),
|
||||
content=event.get("content", ""),
|
||||
metadata=dict(event.get("metadata") or {}),
|
||||
created_at=datetime.fromisoformat(created_at) if isinstance(created_at, str) else created_at,
|
||||
)
|
||||
|
||||
|
||||
def _event_to_dict(event: RunEvent) -> dict[str, Any]:
|
||||
return {
|
||||
"thread_id": event.thread_id,
|
||||
"run_id": event.run_id,
|
||||
"event_type": event.event_type,
|
||||
"category": event.category,
|
||||
"content": event.content,
|
||||
"metadata": event.metadata,
|
||||
"seq": event.seq,
|
||||
"created_at": event.created_at.isoformat(),
|
||||
}
|
||||
@@ -1,515 +0,0 @@
|
||||
"""Run lifecycle persistence adapters owned by the app layer."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import uuid
|
||||
from collections.abc import Callable
|
||||
from typing import Protocol, TypedDict, Unpack, cast
|
||||
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
from store.repositories import FeedbackCreate, Run, RunCreate, build_feedback_repository, build_run_repository
|
||||
|
||||
from deerflow.runtime.actor_context import AUTO, resolve_user_id
|
||||
from deerflow.runtime.serialization import serialize_lc_object
|
||||
from deerflow.runtime.runs.observer import LifecycleEventType, RunLifecycleEvent, RunObserver
|
||||
from deerflow.runtime.stream_bridge import JSONValue
|
||||
|
||||
from .thread_meta import ThreadMetaStorage
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class RunCreateFields(TypedDict, total=False):
|
||||
status: str
|
||||
created_at: str
|
||||
started_at: str
|
||||
ended_at: str
|
||||
assistant_id: str | None
|
||||
user_id: str | None
|
||||
follow_up_to_run_id: str | None
|
||||
metadata: dict[str, JSONValue]
|
||||
kwargs: dict[str, JSONValue]
|
||||
|
||||
|
||||
class RunStatusUpdateFields(TypedDict, total=False):
|
||||
started_at: str
|
||||
ended_at: str
|
||||
metadata: dict[str, JSONValue]
|
||||
|
||||
|
||||
class RunCompletionFields(TypedDict, total=False):
|
||||
total_input_tokens: int
|
||||
total_output_tokens: int
|
||||
total_tokens: int
|
||||
llm_call_count: int
|
||||
lead_agent_tokens: int
|
||||
subagent_tokens: int
|
||||
middleware_tokens: int
|
||||
message_count: int
|
||||
last_ai_message: str | None
|
||||
first_human_message: str | None
|
||||
error: str | None
|
||||
|
||||
|
||||
class RunRow(TypedDict, total=False):
|
||||
run_id: str
|
||||
thread_id: str
|
||||
assistant_id: str | None
|
||||
status: str
|
||||
multitask_strategy: str
|
||||
follow_up_to_run_id: str | None
|
||||
metadata: dict[str, JSONValue]
|
||||
created_at: str
|
||||
updated_at: str
|
||||
started_at: str | None
|
||||
ended_at: str | None
|
||||
error: str | None
|
||||
|
||||
|
||||
class RunReadRepository(Protocol):
|
||||
"""Protocol for durable run queries."""
|
||||
|
||||
async def get(self, run_id: str, *, user_id: str | None | object = AUTO) -> RunRow | None: ...
|
||||
|
||||
async def list_by_thread(
|
||||
self,
|
||||
thread_id: str,
|
||||
*,
|
||||
limit: int = 100,
|
||||
user_id: str | None | object = AUTO,
|
||||
) -> list[RunRow]: ...
|
||||
|
||||
|
||||
class RunWriteRepository(Protocol):
|
||||
"""Protocol for durable run writes."""
|
||||
|
||||
async def create(self, run_id: str, thread_id: str, **kwargs: Unpack[RunCreateFields]) -> None: ...
|
||||
async def update_status(
|
||||
self,
|
||||
run_id: str,
|
||||
status: str,
|
||||
**kwargs: Unpack[RunStatusUpdateFields],
|
||||
) -> None: ...
|
||||
async def set_error(self, run_id: str, error: str) -> None: ...
|
||||
async def update_run_completion(
|
||||
self,
|
||||
run_id: str,
|
||||
*,
|
||||
status: str,
|
||||
**kwargs: Unpack[RunCompletionFields],
|
||||
) -> None: ...
|
||||
|
||||
|
||||
class RunDeleteRepository(Protocol):
|
||||
"""Protocol for durable run deletion."""
|
||||
|
||||
async def delete(self, run_id: str) -> bool: ...
|
||||
|
||||
|
||||
class _RepositoryContext:
|
||||
def __init__(
|
||||
self,
|
||||
session_factory: async_sessionmaker[AsyncSession],
|
||||
build_repo: Callable[[AsyncSession], object],
|
||||
*,
|
||||
commit: bool,
|
||||
) -> None:
|
||||
self._session_factory = session_factory
|
||||
self._build_repo = build_repo
|
||||
self._commit = commit
|
||||
self._session: AsyncSession | None = None
|
||||
|
||||
async def __aenter__(self):
|
||||
self._session = self._session_factory()
|
||||
return self._build_repo(self._session)
|
||||
|
||||
async def __aexit__(self, exc_type, exc, tb) -> None:
|
||||
if self._session is None:
|
||||
return
|
||||
try:
|
||||
if self._commit:
|
||||
if exc_type is None:
|
||||
await self._session.commit()
|
||||
else:
|
||||
await self._session.rollback()
|
||||
finally:
|
||||
await self._session.close()
|
||||
|
||||
|
||||
def _run_to_row(row: Run) -> RunRow:
|
||||
return {
|
||||
"run_id": row.run_id,
|
||||
"thread_id": row.thread_id,
|
||||
"assistant_id": row.assistant_id,
|
||||
"user_id": row.user_id,
|
||||
"status": row.status,
|
||||
"model_name": row.model_name,
|
||||
"multitask_strategy": row.multitask_strategy,
|
||||
"follow_up_to_run_id": row.follow_up_to_run_id,
|
||||
"metadata": cast(dict[str, JSONValue], row.metadata),
|
||||
"kwargs": cast(dict[str, JSONValue], row.kwargs),
|
||||
"created_at": row.created_time.isoformat(),
|
||||
"updated_at": row.updated_time.isoformat() if row.updated_time else "",
|
||||
"total_input_tokens": row.total_input_tokens,
|
||||
"total_output_tokens": row.total_output_tokens,
|
||||
"total_tokens": row.total_tokens,
|
||||
"llm_call_count": row.llm_call_count,
|
||||
"lead_agent_tokens": row.lead_agent_tokens,
|
||||
"subagent_tokens": row.subagent_tokens,
|
||||
"middleware_tokens": row.middleware_tokens,
|
||||
"message_count": row.message_count,
|
||||
"first_human_message": row.first_human_message,
|
||||
"last_ai_message": row.last_ai_message,
|
||||
"error": row.error,
|
||||
}
|
||||
|
||||
|
||||
class FeedbackStoreAdapter:
|
||||
"""Expose feedback route semantics on top of storage package repositories."""
|
||||
|
||||
def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:
|
||||
self._session_factory = session_factory
|
||||
|
||||
async def create(
|
||||
self,
|
||||
*,
|
||||
run_id: str,
|
||||
thread_id: str,
|
||||
rating: int,
|
||||
owner_id: str | None = None,
|
||||
user_id: str | None = None,
|
||||
message_id: str | None = None,
|
||||
comment: str | None = None,
|
||||
) -> dict[str, object]:
|
||||
if rating not in (1, -1):
|
||||
raise ValueError(f"rating must be +1 or -1, got {rating}")
|
||||
effective_user_id = user_id if user_id is not None else owner_id
|
||||
async with self._transaction() as repo:
|
||||
row = await repo.create_feedback(
|
||||
FeedbackCreate(
|
||||
feedback_id=str(uuid.uuid4()),
|
||||
run_id=run_id,
|
||||
thread_id=thread_id,
|
||||
rating=rating,
|
||||
user_id=effective_user_id,
|
||||
message_id=message_id,
|
||||
comment=comment,
|
||||
)
|
||||
)
|
||||
return _feedback_to_dict(row)
|
||||
|
||||
async def get(self, feedback_id: str) -> dict[str, object] | None:
|
||||
async with self._read() as repo:
|
||||
row = await repo.get_feedback(feedback_id)
|
||||
return _feedback_to_dict(row) if row is not None else None
|
||||
|
||||
async def list_by_run(
|
||||
self,
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
*,
|
||||
limit: int = 100,
|
||||
user_id: str | None = None,
|
||||
) -> list[dict[str, object]]:
|
||||
async with self._read() as repo:
|
||||
rows = await repo.list_feedback_by_run(run_id)
|
||||
filtered = [row for row in rows if row.thread_id == thread_id]
|
||||
if user_id is not None:
|
||||
filtered = [row for row in filtered if row.user_id == user_id]
|
||||
return [_feedback_to_dict(row) for row in filtered][:limit]
|
||||
|
||||
async def list_by_thread(self, thread_id: str, *, limit: int = 100) -> list[dict[str, object]]:
|
||||
async with self._read() as repo:
|
||||
rows = await repo.list_feedback_by_thread(thread_id)
|
||||
return [_feedback_to_dict(row) for row in rows][:limit]
|
||||
|
||||
async def aggregate_by_run(self, thread_id: str, run_id: str) -> dict[str, object]:
|
||||
rows = await self.list_by_run(thread_id, run_id)
|
||||
positive = sum(1 for row in rows if row["rating"] == 1)
|
||||
negative = sum(1 for row in rows if row["rating"] == -1)
|
||||
return {"run_id": run_id, "total": len(rows), "positive": positive, "negative": negative}
|
||||
|
||||
async def delete(self, feedback_id: str) -> bool:
|
||||
async with self._transaction() as repo:
|
||||
return await repo.delete_feedback(feedback_id)
|
||||
|
||||
async def upsert(
|
||||
self,
|
||||
*,
|
||||
run_id: str,
|
||||
thread_id: str,
|
||||
rating: int,
|
||||
user_id: str,
|
||||
comment: str | None = None,
|
||||
) -> dict[str, object]:
|
||||
if rating not in (1, -1):
|
||||
raise ValueError(f"rating must be +1 or -1, got {rating}")
|
||||
async with self._transaction() as repo:
|
||||
rows = await repo.list_feedback_by_run(run_id)
|
||||
existing = next((row for row in rows if row.thread_id == thread_id and row.user_id == user_id), None)
|
||||
feedback_id = existing.feedback_id if existing is not None else str(uuid.uuid4())
|
||||
if existing is not None:
|
||||
await repo.delete_feedback(existing.feedback_id)
|
||||
row = await repo.create_feedback(
|
||||
FeedbackCreate(
|
||||
feedback_id=feedback_id,
|
||||
run_id=run_id,
|
||||
thread_id=thread_id,
|
||||
rating=rating,
|
||||
user_id=user_id,
|
||||
comment=comment,
|
||||
)
|
||||
)
|
||||
return _feedback_to_dict(row)
|
||||
|
||||
async def delete_by_run(self, *, thread_id: str, run_id: str, user_id: str) -> bool:
|
||||
async with self._transaction() as repo:
|
||||
rows = await repo.list_feedback_by_run(run_id)
|
||||
existing = next((row for row in rows if row.thread_id == thread_id and row.user_id == user_id), None)
|
||||
if existing is None:
|
||||
return False
|
||||
return await repo.delete_feedback(existing.feedback_id)
|
||||
|
||||
async def list_by_thread_grouped(self, thread_id: str, *, user_id: str) -> dict[str, dict[str, object]]:
|
||||
rows = await self.list_by_thread(thread_id)
|
||||
return {
|
||||
row["run_id"]: row
|
||||
for row in rows
|
||||
if row["user_id"] == user_id
|
||||
}
|
||||
|
||||
def _read(self) -> _RepositoryContext:
|
||||
return _RepositoryContext(self._session_factory, build_feedback_repository, commit=False)
|
||||
|
||||
def _transaction(self) -> _RepositoryContext:
|
||||
return _RepositoryContext(self._session_factory, build_feedback_repository, commit=True)
|
||||
|
||||
|
||||
def _feedback_to_dict(row) -> dict[str, object]:
|
||||
return {
|
||||
"feedback_id": row.feedback_id,
|
||||
"run_id": row.run_id,
|
||||
"thread_id": row.thread_id,
|
||||
"user_id": row.user_id,
|
||||
"owner_id": row.user_id,
|
||||
"message_id": row.message_id,
|
||||
"rating": row.rating,
|
||||
"comment": row.comment,
|
||||
"created_at": row.created_time.isoformat(),
|
||||
}
|
||||
|
||||
|
||||
class RunStoreAdapter:
|
||||
"""Expose runs facade storage semantics on top of storage package repositories."""
|
||||
|
||||
def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:
|
||||
self._session_factory = session_factory
|
||||
|
||||
async def get(self, run_id: str, *, user_id: str | None | object = AUTO) -> RunRow | None:
|
||||
effective_user_id = resolve_user_id(user_id, method_name="RunStoreAdapter.get")
|
||||
async with self._read() as repo:
|
||||
row = await repo.get_run(run_id)
|
||||
if row is None:
|
||||
return None
|
||||
if effective_user_id is not None and row.user_id != effective_user_id:
|
||||
return None
|
||||
return _run_to_row(row)
|
||||
|
||||
async def list_by_thread(
|
||||
self,
|
||||
thread_id: str,
|
||||
*,
|
||||
limit: int = 100,
|
||||
user_id: str | None | object = AUTO,
|
||||
) -> list[RunRow]:
|
||||
effective_user_id = resolve_user_id(user_id, method_name="RunStoreAdapter.list_by_thread")
|
||||
async with self._read() as repo:
|
||||
rows = await repo.list_runs_by_thread(thread_id, limit=limit, offset=0)
|
||||
if effective_user_id is not None:
|
||||
rows = [row for row in rows if row.user_id == effective_user_id]
|
||||
return [_run_to_row(row) for row in rows]
|
||||
|
||||
async def create(self, run_id: str, thread_id: str, **kwargs: Unpack[RunCreateFields]) -> None:
|
||||
metadata = cast(dict[str, JSONValue], serialize_lc_object(kwargs.get("metadata") or {}))
|
||||
run_kwargs = cast(dict[str, JSONValue], serialize_lc_object(kwargs.get("kwargs") or {}))
|
||||
effective_user_id = resolve_user_id(kwargs.get("user_id", AUTO), method_name="RunStoreAdapter.create")
|
||||
async with self._transaction() as repo:
|
||||
await repo.create_run(
|
||||
RunCreate(
|
||||
run_id=run_id,
|
||||
thread_id=thread_id,
|
||||
assistant_id=kwargs.get("assistant_id"),
|
||||
user_id=effective_user_id,
|
||||
status=kwargs.get("status", "pending"),
|
||||
metadata=dict(metadata),
|
||||
kwargs=dict(run_kwargs),
|
||||
follow_up_to_run_id=kwargs.get("follow_up_to_run_id"),
|
||||
)
|
||||
)
|
||||
|
||||
async def delete(self, run_id: str, *, user_id: str | None | object = AUTO) -> bool:
|
||||
async with self._transaction() as repo:
|
||||
existing = await repo.get_run(run_id)
|
||||
if existing is None:
|
||||
return False
|
||||
effective_user_id = resolve_user_id(user_id, method_name="RunStoreAdapter.delete")
|
||||
if effective_user_id is not None and existing.user_id != effective_user_id:
|
||||
return False
|
||||
await repo.delete_run(run_id)
|
||||
return True
|
||||
|
||||
async def update_status(
|
||||
self,
|
||||
run_id: str,
|
||||
status: str,
|
||||
**kwargs: Unpack[RunStatusUpdateFields],
|
||||
) -> None:
|
||||
async with self._transaction() as repo:
|
||||
await repo.update_run_status(run_id, status)
|
||||
|
||||
async def set_error(self, run_id: str, error: str) -> None:
|
||||
async with self._transaction() as repo:
|
||||
await repo.update_run_status(run_id, "error", error=error)
|
||||
|
||||
async def update_run_completion(
|
||||
self,
|
||||
run_id: str,
|
||||
*,
|
||||
status: str,
|
||||
**kwargs: Unpack[RunCompletionFields],
|
||||
) -> None:
|
||||
async with self._transaction() as repo:
|
||||
await repo.update_run_completion(
|
||||
run_id,
|
||||
status=status,
|
||||
total_input_tokens=kwargs.get("total_input_tokens", 0),
|
||||
total_output_tokens=kwargs.get("total_output_tokens", 0),
|
||||
total_tokens=kwargs.get("total_tokens", 0),
|
||||
llm_call_count=kwargs.get("llm_call_count", 0),
|
||||
lead_agent_tokens=kwargs.get("lead_agent_tokens", 0),
|
||||
subagent_tokens=kwargs.get("subagent_tokens", 0),
|
||||
middleware_tokens=kwargs.get("middleware_tokens", 0),
|
||||
message_count=kwargs.get("message_count", 0),
|
||||
last_ai_message=kwargs.get("last_ai_message"),
|
||||
first_human_message=kwargs.get("first_human_message"),
|
||||
error=kwargs.get("error"),
|
||||
)
|
||||
|
||||
def _read(self) -> _RepositoryContext:
|
||||
return _RepositoryContext(self._session_factory, build_run_repository, commit=False)
|
||||
|
||||
def _transaction(self) -> _RepositoryContext:
|
||||
return _RepositoryContext(self._session_factory, build_run_repository, commit=True)
|
||||
|
||||
|
||||
class StorageRunObserver(RunObserver):
|
||||
"""Persist run lifecycle state into app-owned repositories."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
run_write_repo: RunWriteRepository | None = None,
|
||||
thread_meta_storage: ThreadMetaStorage | None = None,
|
||||
) -> None:
|
||||
self._run_write_repo = run_write_repo
|
||||
self._thread_meta_storage = thread_meta_storage
|
||||
|
||||
async def on_event(self, event: RunLifecycleEvent) -> None:
|
||||
try:
|
||||
await self._dispatch(event)
|
||||
except Exception:
|
||||
logger.exception(
|
||||
"StorageRunObserver failed to persist event %s for run %s",
|
||||
event.event_type,
|
||||
event.run_id,
|
||||
)
|
||||
|
||||
async def _dispatch(self, event: RunLifecycleEvent) -> None:
|
||||
handlers = {
|
||||
LifecycleEventType.RUN_STARTED: self._handle_run_started,
|
||||
LifecycleEventType.RUN_COMPLETED: self._handle_run_completed,
|
||||
LifecycleEventType.RUN_FAILED: self._handle_run_failed,
|
||||
LifecycleEventType.RUN_CANCELLED: self._handle_run_cancelled,
|
||||
LifecycleEventType.THREAD_STATUS_UPDATED: self._handle_thread_status,
|
||||
}
|
||||
|
||||
handler = handlers.get(event.event_type)
|
||||
if handler:
|
||||
await handler(event)
|
||||
|
||||
async def _handle_run_started(self, event: RunLifecycleEvent) -> None:
|
||||
if self._run_write_repo:
|
||||
await self._run_write_repo.update_status(
|
||||
run_id=event.run_id,
|
||||
status="running",
|
||||
started_at=event.occurred_at.isoformat(),
|
||||
)
|
||||
|
||||
async def _handle_run_completed(self, event: RunLifecycleEvent) -> None:
|
||||
payload = dict(event.payload) if event.payload else {}
|
||||
if self._run_write_repo:
|
||||
completion_data = payload.get("completion_data")
|
||||
if isinstance(completion_data, dict):
|
||||
await self._run_write_repo.update_run_completion(
|
||||
run_id=event.run_id,
|
||||
status="success",
|
||||
**cast(RunCompletionFields, completion_data),
|
||||
)
|
||||
else:
|
||||
await self._run_write_repo.update_status(
|
||||
run_id=event.run_id,
|
||||
status="success",
|
||||
ended_at=event.occurred_at.isoformat(),
|
||||
)
|
||||
|
||||
if self._thread_meta_storage and "title" in payload:
|
||||
await self._thread_meta_storage.sync_thread_title(
|
||||
thread_id=event.thread_id,
|
||||
title=payload["title"],
|
||||
)
|
||||
|
||||
async def _handle_run_failed(self, event: RunLifecycleEvent) -> None:
|
||||
if self._run_write_repo:
|
||||
payload = dict(event.payload) if event.payload else {}
|
||||
error = payload.get("error", "Unknown error")
|
||||
completion_data = payload.get("completion_data")
|
||||
if isinstance(completion_data, dict):
|
||||
await self._run_write_repo.update_run_completion(
|
||||
run_id=event.run_id,
|
||||
status="error",
|
||||
error=str(error),
|
||||
**cast(RunCompletionFields, completion_data),
|
||||
)
|
||||
else:
|
||||
await self._run_write_repo.update_status(
|
||||
run_id=event.run_id,
|
||||
status="error",
|
||||
ended_at=event.occurred_at.isoformat(),
|
||||
)
|
||||
await self._run_write_repo.set_error(run_id=event.run_id, error=str(error))
|
||||
|
||||
async def _handle_run_cancelled(self, event: RunLifecycleEvent) -> None:
|
||||
if self._run_write_repo:
|
||||
payload = dict(event.payload) if event.payload else {}
|
||||
completion_data = payload.get("completion_data")
|
||||
if isinstance(completion_data, dict):
|
||||
await self._run_write_repo.update_run_completion(
|
||||
run_id=event.run_id,
|
||||
status="interrupted",
|
||||
**cast(RunCompletionFields, completion_data),
|
||||
)
|
||||
else:
|
||||
await self._run_write_repo.update_status(
|
||||
run_id=event.run_id,
|
||||
status="interrupted",
|
||||
ended_at=event.occurred_at.isoformat(),
|
||||
)
|
||||
|
||||
async def _handle_thread_status(self, event: RunLifecycleEvent) -> None:
|
||||
if self._thread_meta_storage:
|
||||
payload = dict(event.payload) if event.payload else {}
|
||||
status = payload.get("status", "idle")
|
||||
await self._thread_meta_storage.sync_thread_status(
|
||||
thread_id=event.thread_id,
|
||||
status=status,
|
||||
)
|
||||
@@ -1,208 +0,0 @@
|
||||
"""Thread metadata storage adapter owned by the app layer."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
from store.repositories import build_thread_meta_repository
|
||||
from store.repositories.contracts import (
|
||||
ThreadMeta,
|
||||
ThreadMetaCreate,
|
||||
ThreadMetaRepositoryProtocol,
|
||||
)
|
||||
from deerflow.runtime.actor_context import AUTO, resolve_user_id
|
||||
|
||||
|
||||
class ThreadMetaStoreAdapter:
|
||||
"""Use storage package thread repositories with per-call sessions."""
|
||||
|
||||
def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:
|
||||
self._session_factory = session_factory
|
||||
|
||||
async def create_thread_meta(self, data: ThreadMetaCreate) -> ThreadMeta:
|
||||
async with self._transaction() as repo:
|
||||
return await repo.create_thread_meta(data)
|
||||
|
||||
async def get_thread_meta(self, thread_id: str) -> ThreadMeta | None:
|
||||
async with self._read() as repo:
|
||||
return await repo.get_thread_meta(thread_id)
|
||||
|
||||
async def update_thread_meta(
|
||||
self,
|
||||
thread_id: str,
|
||||
*,
|
||||
assistant_id: str | None = None,
|
||||
display_name: str | None = None,
|
||||
status: str | None = None,
|
||||
metadata: dict[str, Any] | None = None,
|
||||
) -> None:
|
||||
async with self._transaction() as repo:
|
||||
await repo.update_thread_meta(
|
||||
thread_id,
|
||||
assistant_id=assistant_id,
|
||||
display_name=display_name,
|
||||
status=status,
|
||||
metadata=metadata,
|
||||
)
|
||||
|
||||
async def delete_thread(self, thread_id: str) -> None:
|
||||
async with self._transaction() as repo:
|
||||
await repo.delete_thread(thread_id)
|
||||
|
||||
async def search_threads(
|
||||
self,
|
||||
*,
|
||||
metadata: dict[str, Any] | None = None,
|
||||
status: str | None = None,
|
||||
user_id: str | None = None,
|
||||
assistant_id: str | None = None,
|
||||
limit: int = 100,
|
||||
offset: int = 0,
|
||||
) -> list[ThreadMeta]:
|
||||
async with self._read() as repo:
|
||||
return await repo.search_threads(
|
||||
metadata=metadata,
|
||||
status=status,
|
||||
user_id=user_id,
|
||||
assistant_id=assistant_id,
|
||||
limit=limit,
|
||||
offset=offset,
|
||||
)
|
||||
|
||||
def _read(self):
|
||||
return _ThreadMetaRepositoryContext(self._session_factory, commit=False)
|
||||
|
||||
def _transaction(self):
|
||||
return _ThreadMetaRepositoryContext(self._session_factory, commit=True)
|
||||
|
||||
|
||||
class _ThreadMetaRepositoryContext:
|
||||
def __init__(self, session_factory: async_sessionmaker[AsyncSession], *, commit: bool) -> None:
|
||||
self._session_factory = session_factory
|
||||
self._commit = commit
|
||||
self._session: AsyncSession | None = None
|
||||
|
||||
async def __aenter__(self):
|
||||
self._session = self._session_factory()
|
||||
return build_thread_meta_repository(self._session)
|
||||
|
||||
async def __aexit__(self, exc_type, exc, tb) -> None:
|
||||
if self._session is None:
|
||||
return
|
||||
try:
|
||||
if self._commit:
|
||||
if exc_type is None:
|
||||
await self._session.commit()
|
||||
else:
|
||||
await self._session.rollback()
|
||||
finally:
|
||||
await self._session.close()
|
||||
|
||||
|
||||
class ThreadMetaStorage:
|
||||
"""App-facing adapter around the storage thread metadata contract."""
|
||||
|
||||
def __init__(self, repo: ThreadMetaRepositoryProtocol) -> None:
|
||||
self._repo = repo
|
||||
|
||||
async def get_thread(self, thread_id: str, *, user_id: str | None | object = AUTO) -> ThreadMeta | None:
|
||||
thread = await self._repo.get_thread_meta(thread_id)
|
||||
if thread is None:
|
||||
return None
|
||||
effective_user_id = resolve_user_id(user_id, method_name="ThreadMetaStorage.get_thread")
|
||||
if effective_user_id is not None and thread.user_id != effective_user_id:
|
||||
return None
|
||||
return thread
|
||||
|
||||
async def ensure_thread(
|
||||
self,
|
||||
*,
|
||||
thread_id: str,
|
||||
assistant_id: str | None = None,
|
||||
metadata: dict[str, Any] | None = None,
|
||||
user_id: str | None | object = AUTO,
|
||||
) -> ThreadMeta:
|
||||
effective_user_id = resolve_user_id(user_id, method_name="ThreadMetaStorage.ensure_thread")
|
||||
existing = await self.get_thread(thread_id, user_id=effective_user_id)
|
||||
if existing is not None:
|
||||
return existing
|
||||
|
||||
return await self._repo.create_thread_meta(
|
||||
ThreadMetaCreate(
|
||||
thread_id=thread_id,
|
||||
assistant_id=assistant_id,
|
||||
user_id=effective_user_id,
|
||||
metadata=metadata or {},
|
||||
)
|
||||
)
|
||||
|
||||
async def ensure_thread_running(
|
||||
self,
|
||||
*,
|
||||
thread_id: str,
|
||||
assistant_id: str | None = None,
|
||||
metadata: dict[str, Any] | None = None,
|
||||
) -> ThreadMeta | None:
|
||||
existing = await self._repo.get_thread_meta(thread_id)
|
||||
if existing is None:
|
||||
return await self._repo.create_thread_meta(
|
||||
ThreadMetaCreate(
|
||||
thread_id=thread_id,
|
||||
assistant_id=assistant_id,
|
||||
status="running",
|
||||
metadata=metadata or {},
|
||||
)
|
||||
)
|
||||
|
||||
await self._repo.update_thread_meta(thread_id, status="running")
|
||||
return await self._repo.get_thread_meta(thread_id)
|
||||
|
||||
async def sync_thread_title(self, *, thread_id: str, title: str) -> None:
|
||||
await self._repo.update_thread_meta(thread_id, display_name=title)
|
||||
|
||||
async def sync_thread_assistant_id(self, *, thread_id: str, assistant_id: str) -> None:
|
||||
await self._repo.update_thread_meta(thread_id, assistant_id=assistant_id)
|
||||
|
||||
async def sync_thread_status(self, *, thread_id: str, status: str) -> None:
|
||||
await self._repo.update_thread_meta(thread_id, status=status)
|
||||
|
||||
async def sync_thread_metadata(
|
||||
self,
|
||||
*,
|
||||
thread_id: str,
|
||||
metadata: dict[str, Any],
|
||||
) -> None:
|
||||
await self._repo.update_thread_meta(thread_id, metadata=metadata)
|
||||
|
||||
async def delete_thread(self, thread_id: str) -> None:
|
||||
await self._repo.delete_thread(thread_id)
|
||||
|
||||
async def search_threads(
|
||||
self,
|
||||
*,
|
||||
metadata: dict[str, Any] | None = None,
|
||||
status: str | None = None,
|
||||
user_id: str | None | object = AUTO,
|
||||
assistant_id: str | None = None,
|
||||
limit: int = 100,
|
||||
offset: int = 0,
|
||||
) -> list[ThreadMeta]:
|
||||
normalized_status = status.strip() if status is not None else None
|
||||
resolved_user_id = resolve_user_id(user_id, method_name="ThreadMetaStorage.search_threads")
|
||||
normalized_user_id = resolved_user_id.strip() if resolved_user_id is not None else None
|
||||
normalized_assistant_id = (
|
||||
assistant_id.strip() if assistant_id is not None else None
|
||||
)
|
||||
|
||||
return await self._repo.search_threads(
|
||||
metadata=metadata,
|
||||
status=normalized_status or None,
|
||||
user_id=normalized_user_id or None,
|
||||
assistant_id=normalized_assistant_id or None,
|
||||
limit=limit,
|
||||
offset=offset,
|
||||
)
|
||||
|
||||
|
||||
__all__ = ["ThreadMetaStorage", "ThreadMetaStoreAdapter"]
|
||||
@@ -1,6 +0,0 @@
|
||||
"""App-owned stream bridge adapters and factory."""
|
||||
|
||||
from .factory import build_stream_bridge
|
||||
from .adapters import MemoryStreamBridge, RedisStreamBridge
|
||||
|
||||
__all__ = ["MemoryStreamBridge", "RedisStreamBridge", "build_stream_bridge"]
|
||||
@@ -1,6 +0,0 @@
|
||||
"""Concrete stream bridge adapters owned by the app layer."""
|
||||
|
||||
from .memory import MemoryStreamBridge
|
||||
from .redis import RedisStreamBridge
|
||||
|
||||
__all__ = ["MemoryStreamBridge", "RedisStreamBridge"]
|
||||
@@ -1,450 +0,0 @@
|
||||
"""In-memory stream bridge implementation owned by the app layer."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
from collections.abc import AsyncIterator
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Literal
|
||||
|
||||
from deerflow.runtime.stream_bridge import (
|
||||
CANCELLED_SENTINEL,
|
||||
END_SENTINEL,
|
||||
HEARTBEAT_SENTINEL,
|
||||
TERMINAL_STATES,
|
||||
ResumeResult,
|
||||
StreamBridge,
|
||||
StreamEvent,
|
||||
StreamStatus,
|
||||
)
|
||||
from deerflow.runtime.stream_bridge.exceptions import (
|
||||
BridgeClosedError,
|
||||
StreamCapacityExceededError,
|
||||
StreamTerminatedError,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class _RunStream:
|
||||
condition: asyncio.Condition = field(default_factory=asyncio.Condition)
|
||||
events: list[StreamEvent] = field(default_factory=list)
|
||||
id_to_offset: dict[str, int] = field(default_factory=dict)
|
||||
start_offset: int = 0
|
||||
current_bytes: int = 0
|
||||
seq: int = 0
|
||||
status: StreamStatus = StreamStatus.ACTIVE
|
||||
created_at: float = field(default_factory=time.monotonic)
|
||||
last_publish_at: float | None = None
|
||||
ended_at: float | None = None
|
||||
subscriber_count: int = 0
|
||||
last_subscribe_at: float | None = None
|
||||
awaiting_input: bool = False
|
||||
awaiting_since: float | None = None
|
||||
|
||||
|
||||
class MemoryStreamBridge(StreamBridge):
|
||||
"""Per-run in-memory event log implementation."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
max_events_per_stream: int = 256,
|
||||
max_bytes_per_stream: int = 10 * 1024 * 1024,
|
||||
max_active_streams: int = 1000,
|
||||
stream_eviction_policy: Literal["reject", "lru"] = "lru",
|
||||
terminal_retention_ttl: float = 300.0,
|
||||
active_no_publish_timeout: float = 600.0,
|
||||
orphan_timeout: float = 60.0,
|
||||
max_stream_age: float = 86400.0,
|
||||
hitl_extended_timeout: float = 7200.0,
|
||||
cleanup_interval: float = 30.0,
|
||||
queue_maxsize: int | None = None,
|
||||
) -> None:
|
||||
if queue_maxsize is not None:
|
||||
max_events_per_stream = queue_maxsize
|
||||
|
||||
self._max_events = max_events_per_stream
|
||||
self._max_bytes = max_bytes_per_stream
|
||||
self._max_streams = max_active_streams
|
||||
self._eviction_policy = stream_eviction_policy
|
||||
self._terminal_ttl = terminal_retention_ttl
|
||||
self._active_timeout = active_no_publish_timeout
|
||||
self._orphan_timeout = orphan_timeout
|
||||
self._max_age = max_stream_age
|
||||
self._hitl_timeout = hitl_extended_timeout
|
||||
self._cleanup_interval = cleanup_interval
|
||||
self._streams: dict[str, _RunStream] = {}
|
||||
self._registry_lock = asyncio.Lock()
|
||||
self._closed = False
|
||||
self._cleanup_task: asyncio.Task[None] | None = None
|
||||
|
||||
async def start(self) -> None:
|
||||
if self._cleanup_task is None:
|
||||
self._cleanup_task = asyncio.create_task(self._cleanup_loop())
|
||||
logger.info(
|
||||
"MemoryStreamBridge started (max_events=%d, max_bytes=%d, max_streams=%d)",
|
||||
self._max_events,
|
||||
self._max_bytes,
|
||||
self._max_streams,
|
||||
)
|
||||
|
||||
async def close(self) -> None:
|
||||
async with self._registry_lock:
|
||||
self._closed = True
|
||||
if self._cleanup_task is not None:
|
||||
self._cleanup_task.cancel()
|
||||
try:
|
||||
await self._cleanup_task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
self._cleanup_task = None
|
||||
|
||||
for stream in self._streams.values():
|
||||
async with stream.condition:
|
||||
stream.status = StreamStatus.CLOSED
|
||||
stream.condition.notify_all()
|
||||
|
||||
self._streams.clear()
|
||||
logger.info("MemoryStreamBridge closed")
|
||||
|
||||
async def _get_or_create_stream(self, run_id: str) -> _RunStream:
|
||||
stream = self._streams.get(run_id)
|
||||
if stream is not None:
|
||||
return stream
|
||||
|
||||
async with self._registry_lock:
|
||||
if self._closed:
|
||||
raise BridgeClosedError("Stream bridge is closed")
|
||||
|
||||
stream = self._streams.get(run_id)
|
||||
if stream is not None:
|
||||
return stream
|
||||
|
||||
if len(self._streams) >= self._max_streams:
|
||||
if self._eviction_policy == "reject":
|
||||
raise StreamCapacityExceededError(
|
||||
f"Max {self._max_streams} active streams reached"
|
||||
)
|
||||
evicted = self._evict_oldest_terminal()
|
||||
if evicted is None:
|
||||
raise StreamCapacityExceededError("All streams active, cannot evict")
|
||||
logger.info("Evicted stream %s to make room", evicted)
|
||||
|
||||
stream = _RunStream()
|
||||
self._streams[run_id] = stream
|
||||
logger.debug("Created stream for run %s", run_id)
|
||||
return stream
|
||||
|
||||
def _evict_oldest_terminal(self) -> str | None:
|
||||
oldest_run_id: str | None = None
|
||||
oldest_ended_at: float = float("inf")
|
||||
for run_id, stream in self._streams.items():
|
||||
if stream.status in TERMINAL_STATES and stream.ended_at is not None:
|
||||
if stream.ended_at < oldest_ended_at:
|
||||
oldest_ended_at = stream.ended_at
|
||||
oldest_run_id = run_id
|
||||
if oldest_run_id is not None:
|
||||
del self._streams[oldest_run_id]
|
||||
return oldest_run_id
|
||||
return None
|
||||
|
||||
def _next_id(self, stream: _RunStream) -> str:
|
||||
stream.seq += 1
|
||||
return f"{int(time.time() * 1000)}-{stream.seq}"
|
||||
|
||||
def _estimate_size(self, event: StreamEvent) -> int:
|
||||
base = len(event.id) + len(event.event) + 100
|
||||
if event.data is None:
|
||||
return base
|
||||
if isinstance(event.data, str):
|
||||
return base + len(event.data)
|
||||
if isinstance(event.data, (dict, list)):
|
||||
try:
|
||||
return base + len(json.dumps(event.data, default=str))
|
||||
except (TypeError, ValueError):
|
||||
return base + 200
|
||||
return base + 50
|
||||
|
||||
def _evict_overflow(self, stream: _RunStream) -> None:
|
||||
while len(stream.events) > self._max_events or stream.current_bytes > self._max_bytes:
|
||||
if not stream.events:
|
||||
break
|
||||
evicted = stream.events.pop(0)
|
||||
stream.id_to_offset.pop(evicted.id, None)
|
||||
stream.current_bytes -= self._estimate_size(evicted)
|
||||
stream.start_offset += 1
|
||||
|
||||
async def publish(self, run_id: str, event: str, data: Any) -> str:
|
||||
stream = await self._get_or_create_stream(run_id)
|
||||
async with stream.condition:
|
||||
if stream.status != StreamStatus.ACTIVE:
|
||||
raise StreamTerminatedError(
|
||||
f"Cannot publish to {stream.status.value} stream"
|
||||
)
|
||||
|
||||
entry = StreamEvent(id=self._next_id(stream), event=event, data=data)
|
||||
absolute_offset = stream.start_offset + len(stream.events)
|
||||
stream.events.append(entry)
|
||||
stream.id_to_offset[entry.id] = absolute_offset
|
||||
stream.current_bytes += self._estimate_size(entry)
|
||||
stream.last_publish_at = time.monotonic()
|
||||
self._evict_overflow(stream)
|
||||
stream.condition.notify_all()
|
||||
return entry.id
|
||||
|
||||
async def publish_end(self, run_id: str) -> str:
|
||||
return await self.publish_terminal(run_id, StreamStatus.ENDED)
|
||||
|
||||
async def publish_terminal(
|
||||
self,
|
||||
run_id: str,
|
||||
kind: StreamStatus,
|
||||
data: Any = None,
|
||||
) -> str:
|
||||
if kind not in TERMINAL_STATES:
|
||||
raise ValueError(f"Invalid terminal kind: {kind}")
|
||||
|
||||
stream = await self._get_or_create_stream(run_id)
|
||||
async with stream.condition:
|
||||
if stream.status != StreamStatus.ACTIVE:
|
||||
for evt in reversed(stream.events):
|
||||
if evt.event in ("end", "cancel", "error", "dead_letter"):
|
||||
return evt.id
|
||||
return ""
|
||||
|
||||
event_name = {
|
||||
StreamStatus.ENDED: "end",
|
||||
StreamStatus.CANCELLED: "cancel",
|
||||
StreamStatus.ERRORED: "error",
|
||||
}[kind]
|
||||
entry = StreamEvent(id=self._next_id(stream), event=event_name, data=data)
|
||||
absolute_offset = stream.start_offset + len(stream.events)
|
||||
stream.events.append(entry)
|
||||
stream.id_to_offset[entry.id] = absolute_offset
|
||||
stream.current_bytes += self._estimate_size(entry)
|
||||
stream.status = kind
|
||||
stream.ended_at = time.monotonic()
|
||||
stream.awaiting_input = False
|
||||
stream.condition.notify_all()
|
||||
logger.debug("Stream %s terminal: %s", run_id, kind.value)
|
||||
return entry.id
|
||||
|
||||
async def cancel(self, run_id: str) -> None:
|
||||
await self.publish_terminal(run_id, StreamStatus.CANCELLED)
|
||||
|
||||
async def subscribe(
|
||||
self,
|
||||
run_id: str,
|
||||
*,
|
||||
last_event_id: str | None = None,
|
||||
heartbeat_interval: float = 15.0,
|
||||
) -> AsyncIterator[StreamEvent]:
|
||||
stream = await self._get_or_create_stream(run_id)
|
||||
resume = self._resolve_resume_point(stream, last_event_id)
|
||||
next_offset = resume.next_offset
|
||||
|
||||
async with stream.condition:
|
||||
stream.subscriber_count += 1
|
||||
stream.last_subscribe_at = time.monotonic()
|
||||
|
||||
try:
|
||||
while True:
|
||||
entry_to_yield: StreamEvent | None = None
|
||||
sentinel_to_yield: StreamEvent | None = None
|
||||
should_return = False
|
||||
should_wait = False
|
||||
|
||||
async with stream.condition:
|
||||
if self._closed or stream.status == StreamStatus.CLOSED:
|
||||
sentinel_to_yield = CANCELLED_SENTINEL
|
||||
should_return = True
|
||||
elif next_offset < stream.start_offset:
|
||||
next_offset = stream.start_offset
|
||||
else:
|
||||
local_index = next_offset - stream.start_offset
|
||||
if 0 <= local_index < len(stream.events):
|
||||
entry_to_yield = stream.events[local_index]
|
||||
next_offset += 1
|
||||
if entry_to_yield.event in ("end", "cancel", "error", "dead_letter"):
|
||||
should_return = True
|
||||
elif stream.status in TERMINAL_STATES:
|
||||
sentinel_to_yield = END_SENTINEL
|
||||
should_return = True
|
||||
else:
|
||||
should_wait = True
|
||||
try:
|
||||
await asyncio.wait_for(
|
||||
stream.condition.wait(),
|
||||
timeout=heartbeat_interval,
|
||||
)
|
||||
except TimeoutError:
|
||||
pass
|
||||
|
||||
if sentinel_to_yield is not None:
|
||||
yield sentinel_to_yield
|
||||
if should_return:
|
||||
return
|
||||
continue
|
||||
|
||||
if entry_to_yield is not None:
|
||||
yield entry_to_yield
|
||||
if should_return:
|
||||
return
|
||||
continue
|
||||
|
||||
if should_wait:
|
||||
async with stream.condition:
|
||||
local_index = next_offset - stream.start_offset
|
||||
has_events = 0 <= local_index < len(stream.events)
|
||||
is_terminal = stream.status in TERMINAL_STATES
|
||||
if not has_events and not is_terminal:
|
||||
yield HEARTBEAT_SENTINEL
|
||||
|
||||
finally:
|
||||
async with stream.condition:
|
||||
stream.subscriber_count = max(0, stream.subscriber_count - 1)
|
||||
|
||||
async def mark_awaiting_input(self, run_id: str) -> None:
|
||||
stream = self._streams.get(run_id)
|
||||
if stream is None:
|
||||
return
|
||||
async with stream.condition:
|
||||
if stream.status == StreamStatus.ACTIVE:
|
||||
stream.awaiting_input = True
|
||||
stream.awaiting_since = time.monotonic()
|
||||
logger.debug("Stream %s marked as awaiting input", run_id)
|
||||
|
||||
async def cleanup(self, run_id: str, *, delay: float = 0) -> None:
|
||||
if delay > 0:
|
||||
await asyncio.sleep(delay)
|
||||
await self._do_cleanup(run_id, "manual")
|
||||
|
||||
async def _do_cleanup(self, run_id: str, reason: str) -> None:
|
||||
async with self._registry_lock:
|
||||
stream = self._streams.pop(run_id, None)
|
||||
if stream is not None:
|
||||
async with stream.condition:
|
||||
stream.status = StreamStatus.CLOSED
|
||||
stream.condition.notify_all()
|
||||
logger.debug("Cleaned up stream %s (reason: %s)", run_id, reason)
|
||||
|
||||
async def _mark_dead_letter(self, run_id: str, reason: str) -> None:
|
||||
stream = self._streams.get(run_id)
|
||||
if stream is None:
|
||||
return
|
||||
async with stream.condition:
|
||||
if stream.status != StreamStatus.ACTIVE:
|
||||
return
|
||||
entry = StreamEvent(
|
||||
id=self._next_id(stream),
|
||||
event="dead_letter",
|
||||
data={"reason": reason, "timestamp": time.time()},
|
||||
)
|
||||
absolute_offset = stream.start_offset + len(stream.events)
|
||||
stream.events.append(entry)
|
||||
stream.id_to_offset[entry.id] = absolute_offset
|
||||
stream.current_bytes += self._estimate_size(entry)
|
||||
stream.status = StreamStatus.ERRORED
|
||||
stream.ended_at = time.monotonic()
|
||||
stream.condition.notify_all()
|
||||
logger.warning("Stream %s marked as dead letter: %s", run_id, reason)
|
||||
|
||||
async def _cleanup_loop(self) -> None:
|
||||
while not self._closed:
|
||||
try:
|
||||
await asyncio.sleep(self._cleanup_interval)
|
||||
except asyncio.CancelledError:
|
||||
break
|
||||
|
||||
now = time.monotonic()
|
||||
to_cleanup: list[tuple[str, str]] = []
|
||||
to_mark_dead: list[tuple[str, str]] = []
|
||||
|
||||
async with self._registry_lock:
|
||||
for run_id, stream in list(self._streams.items()):
|
||||
if now - stream.created_at > self._max_age:
|
||||
to_cleanup.append((run_id, "max_age_exceeded"))
|
||||
continue
|
||||
|
||||
if stream.status == StreamStatus.ACTIVE:
|
||||
timeout = self._hitl_timeout if stream.awaiting_input else self._active_timeout
|
||||
last_activity = stream.last_publish_at or stream.created_at
|
||||
if now - last_activity > timeout:
|
||||
to_mark_dead.append((run_id, "no_publish_timeout"))
|
||||
continue
|
||||
|
||||
if stream.status in TERMINAL_STATES and stream.ended_at:
|
||||
if stream.subscriber_count > 0:
|
||||
continue
|
||||
last_sub = stream.last_subscribe_at or stream.ended_at
|
||||
if now - last_sub > self._orphan_timeout:
|
||||
to_cleanup.append((run_id, "orphan"))
|
||||
continue
|
||||
if now - stream.ended_at > self._terminal_ttl:
|
||||
to_cleanup.append((run_id, "ttl_expired"))
|
||||
|
||||
for run_id, reason in to_mark_dead:
|
||||
await self._mark_dead_letter(run_id, reason)
|
||||
for run_id, reason in to_cleanup:
|
||||
await self._do_cleanup(run_id, reason)
|
||||
|
||||
def get_stats(self) -> dict[str, Any]:
|
||||
active = sum(1 for s in self._streams.values() if s.status == StreamStatus.ACTIVE)
|
||||
terminal = sum(1 for s in self._streams.values() if s.status in TERMINAL_STATES)
|
||||
total_events = sum(len(s.events) for s in self._streams.values())
|
||||
total_bytes = sum(s.current_bytes for s in self._streams.values())
|
||||
total_subs = sum(s.subscriber_count for s in self._streams.values())
|
||||
return {
|
||||
"total_streams": len(self._streams),
|
||||
"active_streams": active,
|
||||
"terminal_streams": terminal,
|
||||
"total_events": total_events,
|
||||
"total_bytes": total_bytes,
|
||||
"total_subscribers": total_subs,
|
||||
"closed": self._closed,
|
||||
}
|
||||
|
||||
def _resolve_resume_point(
|
||||
self,
|
||||
stream: _RunStream,
|
||||
last_event_id: str | None,
|
||||
) -> ResumeResult:
|
||||
if last_event_id is None:
|
||||
return ResumeResult(next_offset=stream.start_offset, status="fresh")
|
||||
if last_event_id in stream.id_to_offset:
|
||||
return ResumeResult(
|
||||
next_offset=stream.id_to_offset[last_event_id] + 1,
|
||||
status="resumed",
|
||||
)
|
||||
|
||||
parts = last_event_id.split("-")
|
||||
if len(parts) != 2:
|
||||
return ResumeResult(next_offset=stream.start_offset, status="invalid")
|
||||
try:
|
||||
event_ts = int(parts[0])
|
||||
_event_seq = int(parts[1])
|
||||
except ValueError:
|
||||
return ResumeResult(next_offset=stream.start_offset, status="invalid")
|
||||
|
||||
if stream.events:
|
||||
try:
|
||||
oldest_parts = stream.events[0].id.split("-")
|
||||
oldest_ts = int(oldest_parts[0])
|
||||
if event_ts < oldest_ts:
|
||||
return ResumeResult(
|
||||
next_offset=stream.start_offset,
|
||||
status="evicted",
|
||||
gap_count=stream.start_offset,
|
||||
)
|
||||
except (ValueError, IndexError):
|
||||
pass
|
||||
|
||||
return ResumeResult(next_offset=stream.start_offset, status="unknown")
|
||||
|
||||
|
||||
__all__ = ["MemoryStreamBridge"]
|
||||
@@ -1,37 +0,0 @@
|
||||
"""Redis-backed stream bridge placeholder owned by the app layer."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import AsyncIterator
|
||||
from typing import Any
|
||||
|
||||
from deerflow.runtime.stream_bridge import StreamBridge, StreamEvent
|
||||
|
||||
|
||||
class RedisStreamBridge(StreamBridge):
|
||||
"""Reserved app-owned Redis implementation.
|
||||
|
||||
Phase 1 intentionally keeps Redis out of the harness package. The concrete
|
||||
implementation will live here once cross-process streaming is introduced.
|
||||
"""
|
||||
|
||||
def __init__(self, *, redis_url: str) -> None:
|
||||
self._redis_url = redis_url
|
||||
|
||||
async def publish(self, run_id: str, event: str, data: Any) -> str:
|
||||
raise NotImplementedError("Redis stream bridge will be implemented in app infra")
|
||||
|
||||
async def publish_end(self, run_id: str) -> str:
|
||||
raise NotImplementedError("Redis stream bridge will be implemented in app infra")
|
||||
|
||||
def subscribe(
|
||||
self,
|
||||
run_id: str,
|
||||
*,
|
||||
last_event_id: str | None = None,
|
||||
heartbeat_interval: float = 15.0,
|
||||
) -> AsyncIterator[StreamEvent]:
|
||||
raise NotImplementedError("Redis stream bridge will be implemented in app infra")
|
||||
|
||||
async def cleanup(self, run_id: str, *, delay: float = 0) -> None:
|
||||
raise NotImplementedError("Redis stream bridge will be implemented in app infra")
|
||||
@@ -1,50 +0,0 @@
|
||||
"""App-owned stream bridge factory."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from collections.abc import AsyncIterator
|
||||
from contextlib import AbstractAsyncContextManager, asynccontextmanager
|
||||
|
||||
from deerflow.config.stream_bridge_config import get_stream_bridge_config
|
||||
from deerflow.runtime.stream_bridge import StreamBridge
|
||||
|
||||
from .adapters import MemoryStreamBridge, RedisStreamBridge
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def build_stream_bridge(config=None) -> AbstractAsyncContextManager[StreamBridge]:
|
||||
"""Build the configured app-owned stream bridge."""
|
||||
return _build_stream_bridge_impl(config)
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def _build_stream_bridge_impl(config=None) -> AsyncIterator[StreamBridge]:
|
||||
if config is None:
|
||||
config = get_stream_bridge_config()
|
||||
|
||||
if config is None or config.type == "memory":
|
||||
maxsize = config.queue_maxsize if config is not None else 256
|
||||
bridge = MemoryStreamBridge(queue_maxsize=maxsize)
|
||||
await bridge.start()
|
||||
logger.info("Stream bridge initialised: memory (queue_maxsize=%d)", maxsize)
|
||||
try:
|
||||
yield bridge
|
||||
finally:
|
||||
await bridge.close()
|
||||
return
|
||||
|
||||
if config.type == "redis":
|
||||
if not config.redis_url:
|
||||
raise ValueError("Redis stream bridge requires redis_url")
|
||||
bridge = RedisStreamBridge(redis_url=config.redis_url)
|
||||
await bridge.start()
|
||||
logger.info("Stream bridge initialised: redis (%s)", config.redis_url)
|
||||
try:
|
||||
yield bridge
|
||||
finally:
|
||||
await bridge.close()
|
||||
return
|
||||
|
||||
raise ValueError(f"Unknown stream bridge type: {config.type!r}")
|
||||
@@ -1,15 +0,0 @@
|
||||
"""Entry point for running the Gateway API via `python app/main.py`.
|
||||
|
||||
Useful for IDE debugging (e.g., PyCharm / VS Code debug configurations).
|
||||
Equivalent to: PYTHONPATH=. uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001
|
||||
"""
|
||||
|
||||
import uvicorn
|
||||
|
||||
if __name__ == "__main__":
|
||||
uvicorn.run(
|
||||
"app.gateway.app:app",
|
||||
host="0.0.0.0",
|
||||
port=8001,
|
||||
reload=True,
|
||||
)
|
||||
@@ -1,314 +0,0 @@
|
||||
# app.plugins Design Overview
|
||||
|
||||
This document describes the current role of `backend/app/plugins`, its plugin design contract, dependency boundaries, and how the current `auth` plugin provides services with minimal intrusion into the host application.
|
||||
|
||||
## 1. Overall Role
|
||||
|
||||
`app.plugins` is the application-side plugin boundary.
|
||||
|
||||
Its purpose is not to implement a generic plugin marketplace. Instead, it provides a clear boundary inside `app` for separable business capabilities, so that a capability can:
|
||||
|
||||
1. carry its own domain model, runtime state, and adapters inside the plugin
|
||||
2. interact with the host application only through a limited set of seams
|
||||
3. remain replaceable, removable, and extensible over time
|
||||
|
||||
The only real plugin currently implemented under this directory is [`auth`](/Users/rayhpeng/workspace/open-source/deer-flow/backend/app/plugins/auth).
|
||||
|
||||
The current direction is not “put all logic into app”. It is:
|
||||
|
||||
1. the host application owns unified bootstrap, shared infrastructure, and top-level router assembly
|
||||
2. each plugin owns its own business contract, persistence definitions, runtime state, and outward-facing adapters
|
||||
|
||||
## 2. Plugin Design Contract
|
||||
|
||||
### 2.1 A plugin should carry its own implementation
|
||||
|
||||
The primary contract visible in the current codebase is:
|
||||
|
||||
A plugin’s own ORM, runtime, domain, and adapters should be implemented inside the plugin itself. Core business behavior should not be scattered into unrelated external modules.
|
||||
|
||||
The `auth` plugin already follows that pattern with a fairly complete internal structure:
|
||||
|
||||
1. `domain`
|
||||
- config, errors, JWT, password logic, domain models, service
|
||||
2. `storage`
|
||||
- plugin-owned ORM models, repository contracts, and repository implementations
|
||||
3. `runtime`
|
||||
- plugin-owned runtime config state
|
||||
4. `api`
|
||||
- plugin-owned HTTP router and schemas
|
||||
5. `security`
|
||||
- plugin-owned middleware, dependencies, CSRF logic, and LangGraph adapter
|
||||
6. `authorization`
|
||||
- plugin-owned permission model, policy resolution, and hooks
|
||||
7. `injection`
|
||||
- plugin-owned route-policy loading, injection, and validation
|
||||
|
||||
In other words, a plugin should be a self-contained capability module, not a bag of helpers.
|
||||
|
||||
### 2.2 The host app should provide shared infrastructure, not plugin internals
|
||||
|
||||
The current contract is not that every plugin must be fully infrastructure-independent.
|
||||
|
||||
It is:
|
||||
|
||||
1. a plugin may reuse the application’s shared `engine`, `session_factory`, FastAPI app, and router tree
|
||||
2. but the plugin must still own its table definitions, repositories, runtime config, and business/auth behavior
|
||||
|
||||
This is stated explicitly in [`auth/plugin.toml`](/Users/rayhpeng/workspace/open-source/deer-flow/backend/app/plugins/auth/plugin.toml):
|
||||
|
||||
1. `storage.mode = "shared_infrastructure"`
|
||||
2. the plugin owns its storage definitions and repositories
|
||||
3. but it reuses the application’s shared persistence infrastructure
|
||||
|
||||
So the real rule is not “never reuse infrastructure”. The real rule is “do not outsource plugin business semantics to the rest of the app”.
|
||||
|
||||
### 2.3 Dependencies should remain one-way
|
||||
|
||||
The intended dependency direction in the current design is:
|
||||
|
||||
```text
|
||||
gateway / app bootstrap
|
||||
-> plugin public adapters
|
||||
-> plugin domain / storage / runtime
|
||||
```
|
||||
|
||||
Not:
|
||||
|
||||
```text
|
||||
plugin domain
|
||||
-> depends on app business modules
|
||||
```
|
||||
|
||||
A plugin may depend on:
|
||||
|
||||
1. shared persistence infrastructure
|
||||
2. `app.state` provided by the host application
|
||||
3. generic framework capabilities such as FastAPI / Starlette
|
||||
|
||||
But its core business rules should not depend on unrelated app business modules, otherwise hot-swappability becomes unrealistic.
|
||||
|
||||
## 3. The Current auth Plugin Structure
|
||||
|
||||
The current `auth` plugin is effectively a self-contained authentication and authorization package with its own models, services, and adapters.
|
||||
|
||||
### 3.1 domain
|
||||
|
||||
[`auth/domain`](/Users/rayhpeng/workspace/open-source/deer-flow/backend/app/plugins/auth/domain) owns:
|
||||
|
||||
1. `config.py`
|
||||
- auth-related configuration definition and loading
|
||||
2. `errors.py`
|
||||
- error codes and response contracts
|
||||
3. `jwt.py`
|
||||
- token encoding and decoding
|
||||
4. `password.py`
|
||||
- password hashing and verification
|
||||
5. `models.py`
|
||||
- auth domain models
|
||||
6. `service.py`
|
||||
- `AuthService` as the core business service
|
||||
|
||||
`AuthService` depends only on the plugin’s own `DbUserRepository` plus the shared session factory. The auth business logic is not reimplemented in `gateway`.
|
||||
|
||||
### 3.2 storage
|
||||
|
||||
[`auth/storage`](/Users/rayhpeng/workspace/open-source/deer-flow/backend/app/plugins/auth/storage) clearly shows the “ORM is owned by the plugin” contract:
|
||||
|
||||
1. [`models.py`](/Users/rayhpeng/workspace/open-source/deer-flow/backend/app/plugins/auth/storage/models.py)
|
||||
- defines the plugin-owned `users` table model
|
||||
2. [`contracts.py`](/Users/rayhpeng/workspace/open-source/deer-flow/backend/app/plugins/auth/storage/contracts.py)
|
||||
- defines `User`, `UserCreate`, and `UserRepositoryProtocol`
|
||||
3. [`repositories.py`](/Users/rayhpeng/workspace/open-source/deer-flow/backend/app/plugins/auth/storage/repositories.py)
|
||||
- implements `DbUserRepository`
|
||||
|
||||
The key point is:
|
||||
|
||||
1. the plugin defines its own ORM model
|
||||
2. the plugin defines its own repository protocol
|
||||
3. the plugin implements its own repository
|
||||
4. external code only needs to provide a session or session factory
|
||||
|
||||
That is the minimal shared seam the boundary should preserve.
|
||||
|
||||
### 3.3 runtime
|
||||
|
||||
[`auth/runtime/config_state.py`](/Users/rayhpeng/workspace/open-source/deer-flow/backend/app/plugins/auth/runtime/config_state.py) keeps plugin-owned runtime config state:
|
||||
|
||||
1. `get_auth_config()`
|
||||
2. `set_auth_config()`
|
||||
3. `reset_auth_config()`
|
||||
|
||||
This matters because runtime state is also part of the plugin boundary. If future plugins need their own caches, state holders, or feature flags, they should follow the same pattern and keep them inside the plugin.
|
||||
|
||||
### 3.4 adapters
|
||||
|
||||
The `auth` plugin exposes capability through four main adapter groups:
|
||||
|
||||
1. `api/router.py`
|
||||
- HTTP endpoints
|
||||
2. `security/*`
|
||||
- middleware, dependencies, request-user resolution, actor-context bridge
|
||||
3. `authorization/*`
|
||||
- capabilities, policy evaluators, auth hooks
|
||||
4. `injection/*`
|
||||
- route-policy registry, guard injection, startup validation
|
||||
|
||||
These adapters all follow the same rule:
|
||||
|
||||
1. entry-point behavior is defined inside the plugin
|
||||
2. the host app only assembles and wires it
|
||||
|
||||
## 4. How a Plugin Interacts with the Host App
|
||||
|
||||
### 4.1 The top-level router only includes plugin routers
|
||||
|
||||
[`app/gateway/router.py`](/Users/rayhpeng/workspace/open-source/deer-flow/backend/app/gateway/router.py) simply:
|
||||
|
||||
1. imports `app.plugins.auth.api.router`
|
||||
2. calls `include_router(auth_router)`
|
||||
|
||||
That means the host app integrates auth HTTP behavior by assembly, not by duplicating login/register logic in `gateway`.
|
||||
|
||||
### 4.2 registrar performs wiring, not takeover
|
||||
|
||||
In [`app/gateway/registrar.py`](/Users/rayhpeng/workspace/open-source/deer-flow/backend/app/gateway/registrar.py), the host app mainly does this:
|
||||
|
||||
1. `app.state.authz_hooks = build_authz_hooks()`
|
||||
2. loads and validates the route-policy registry
|
||||
3. calls `install_route_guards(app)`
|
||||
4. calls `app.add_middleware(CSRFMiddleware)`
|
||||
5. calls `app.add_middleware(AuthMiddleware)`
|
||||
|
||||
So the host app only wires the plugin in:
|
||||
|
||||
1. register middleware
|
||||
2. install route guards
|
||||
3. expose hooks and registries through `app.state`
|
||||
|
||||
The actual auth logic, authz logic, and route-policy semantics still live inside the plugin.
|
||||
|
||||
### 4.3 The plugin reuses shared sessions, but still owns business repositories
|
||||
|
||||
In [`auth/security/dependencies.py`](/Users/rayhpeng/workspace/open-source/deer-flow/backend/app/plugins/auth/security/dependencies.py):
|
||||
|
||||
1. the plugin reads the shared session factory from `request.app.state.persistence.session_factory`
|
||||
2. constructs `DbUserRepository` itself
|
||||
3. constructs `AuthService` itself
|
||||
|
||||
This is a good low-intrusion seam:
|
||||
|
||||
1. the outside world provides only shared infrastructure handles
|
||||
2. the plugin decides how to instantiate its internal dependencies
|
||||
|
||||
## 5. Hot-Swappability and Low-Intrusion Principles
|
||||
|
||||
### 5.1 If a plugin serves other modules, it should minimize intrusion
|
||||
|
||||
When a plugin provides services to the rest of the app, the preferred patterns are:
|
||||
|
||||
1. expose a router
|
||||
2. expose middleware or dependencies
|
||||
3. expose hooks or protocols
|
||||
4. inject a small number of shared objects through `app.state`
|
||||
5. use config-driven route policies or capabilities instead of hardcoding checks inside business routes
|
||||
|
||||
Patterns to avoid:
|
||||
|
||||
1. large plugin-specific branches spread across `gateway`
|
||||
2. unrelated business modules importing plugin ORM internals and rebuilding plugin logic themselves
|
||||
3. plugin state being maintained across many global modules
|
||||
|
||||
### 5.2 Low-intrusion seams already visible in auth
|
||||
|
||||
The current `auth` plugin already uses four important low-intrusion seams:
|
||||
|
||||
1. router integration
|
||||
- `gateway.router` only calls `include_router`
|
||||
2. middleware integration
|
||||
- `registrar` only registers `AuthMiddleware` and `CSRFMiddleware`
|
||||
3. policy injection
|
||||
- `install_route_guards(app)` appends `Depends(enforce_route_policy)` uniformly to routes
|
||||
4. hook seam
|
||||
- `authz_hooks` is exposed via `app.state`, so permission providers and policy builders can be replaced
|
||||
|
||||
This structure has three practical benefits:
|
||||
|
||||
1. host-app changes stay concentrated in the assembly layer
|
||||
2. plugin core logic stays concentrated inside the plugin directory
|
||||
3. swapping implementations does not require editing business routes one by one
|
||||
|
||||
### 5.3 Route policy is a key low-intrusion mechanism
|
||||
|
||||
[`auth/injection/registry_loader.py`](/Users/rayhpeng/workspace/open-source/deer-flow/backend/app/plugins/auth/injection/registry_loader.py), [`validation.py`](/Users/rayhpeng/workspace/open-source/deer-flow/backend/app/plugins/auth/injection/validation.py), and [`route_injector.py`](/Users/rayhpeng/workspace/open-source/deer-flow/backend/app/plugins/auth/injection/route_injector.py) together form an important contract:
|
||||
|
||||
1. route policies live in the plugin-owned `route_policies.yaml`
|
||||
2. startup validates that policy entries and real routes stay aligned
|
||||
3. guards are attached by uniform injection instead of manual per-endpoint code
|
||||
|
||||
That allows the plugin to:
|
||||
|
||||
1. describe which routes are public, which capabilities are required, and which owner policies apply
|
||||
2. avoid large invasive changes to the host routing layer
|
||||
3. remain easier to replace or trim down later
|
||||
|
||||
## 6. What “ORM and runtime are implemented inside the plugin” Should Mean
|
||||
|
||||
That contract should be read as three concrete rules:
|
||||
|
||||
1. data models belong to the plugin
|
||||
- the plugin’s own tables, Pydantic contracts, repository protocols, and repository implementations stay inside the plugin directory
|
||||
2. runtime state belongs to the plugin
|
||||
- plugin-owned config caches, context bridges, and plugin-level hooks stay inside the plugin
|
||||
3. the outside world exposes infrastructure, not plugin semantics
|
||||
- for example shared `session_factory`, FastAPI app, and `app.state`
|
||||
|
||||
Using `auth` as the example:
|
||||
|
||||
1. the `users` table is defined inside the plugin, not in `app.infra`
|
||||
2. `AuthService` is implemented inside the plugin, not in `gateway`
|
||||
3. `get_auth_config()` is maintained inside the plugin, not cached elsewhere
|
||||
4. `AuthMiddleware`, `route_guard`, and `AuthzHooks` are all provided by the plugin itself
|
||||
|
||||
This is the structural prerequisite for meaningful pluginization later.
|
||||
|
||||
## 7. Current Scope and Non-Goals
|
||||
|
||||
At the current stage, the role of `app.plugins` is mainly:
|
||||
|
||||
1. to create module boundaries for separable application-side capabilities
|
||||
2. to let each plugin own its own domain/storage/runtime/adapters
|
||||
3. to connect plugins to the host app through assembly-oriented seams
|
||||
|
||||
The current non-goals are also clear:
|
||||
|
||||
1. this is not yet a full generic plugin discovery/installation system
|
||||
2. plugins are not dynamically enabled or disabled at runtime
|
||||
3. shared infrastructure is not being duplicated into every plugin
|
||||
|
||||
So at this stage, “hot-swappable” should be interpreted more precisely as:
|
||||
|
||||
1. plugin boundaries stay as independent as possible
|
||||
2. integration points stay concentrated in the assembly layer
|
||||
3. replacing or removing a plugin should mostly affect a small number of places such as `registrar`, router includes, and `app.state` hooks
|
||||
|
||||
## 8. Suggested Evolution Rules
|
||||
|
||||
If `app.plugins` is going to become a more stable plugin boundary, the codebase should keep following these rules:
|
||||
|
||||
1. each plugin directory should keep a `domain` / `storage` / `runtime` / `adapter` split
|
||||
2. plugin-owned ORM and repositories should not drift into shared business directories
|
||||
3. when a plugin serves the rest of the app, it should prefer exposing protocols, hooks, routers, and middleware over forcing external code to import internal implementation details
|
||||
4. seams between a plugin and the host app should stay mostly limited to:
|
||||
- `router.include_router(...)`
|
||||
- `app.add_middleware(...)`
|
||||
- `app.state.*`
|
||||
- lifespan/bootstrap wiring
|
||||
5. config-driven integration should be preferred over scattered hardcoded integration
|
||||
6. startup validation should be preferred over implicit runtime failure
|
||||
|
||||
## 9. Summary
|
||||
|
||||
The current `app.plugins` contract can be summarized in one sentence:
|
||||
|
||||
Each plugin owns its own business implementation, ORM, and runtime; the host application provides shared infrastructure and assembly seams; and services should be integrated through low-intrusion, replaceable boundaries so the system can evolve toward real hot-swappability.
|
||||
@@ -1,310 +0,0 @@
|
||||
# app.plugins 设计说明
|
||||
|
||||
本文基于当前代码实现，说明 `backend/app/plugins` 的定位、插件设计契约、依赖边界，以及当前 `auth` 插件是如何在尽量少侵入宿主应用的前提下提供服务的。
|
||||
|
||||
## 1. 总体定位
|
||||
|
||||
`app.plugins` 是应用侧插件边界。它的目标不是做一个通用插件市场，而是在 `app` 这一层给可拆分的业务能力预留清晰边界，使某一类能力可以：
|
||||
|
||||
1. 在插件内部自带领域模型、运行时状态和适配器
|
||||
2. 只通过有限的接缝与宿主应用交互
|
||||
3. 在未来保持“可替换、可裁剪、可扩展”
|
||||
|
||||
当前目录下实际落地的插件是 [`auth`](/Users/rayhpeng/workspace/open-source/deer-flow/backend/app/plugins/auth)。
|
||||
|
||||
从当前实现看，`app.plugins` 的方向不是“所有逻辑都塞进 app”，而是：
|
||||
|
||||
1. 宿主应用负责统一启动、共享基础设施和总路由装配
|
||||
2. 插件负责自己的业务契约、持久化定义、运行时状态和外部适配器
|
||||
|
||||
## 2. 插件设计契约
|
||||
|
||||
### 2.1 插件内部要自带完整能力
|
||||
|
||||
当前代码体现出的首要契约是：
|
||||
|
||||
插件自己的 ORM、runtime、domain、adapter，原则上都应由插件内部实现，不要把核心业务依赖散落到外部模块。
|
||||
|
||||
以 `auth` 插件为例，它内部已经自带了完整分层：
|
||||
|
||||
1. `domain`
|
||||
- 配置、错误、JWT、密码、领域模型、服务
|
||||
2. `storage`
|
||||
- 插件自己的 ORM 模型、仓储契约和仓储实现
|
||||
3. `runtime`
|
||||
- 插件自己的运行时配置状态
|
||||
4. `api`
|
||||
- 插件自己的 HTTP router 和 schema
|
||||
5. `security`
|
||||
- 插件自己的 middleware、dependency、csrf、LangGraph 适配
|
||||
6. `authorization`
|
||||
- 插件自己的权限模型、policy 解析和 hook
|
||||
7. `injection`
|
||||
- 插件自己的路由策略注册、注入和校验逻辑
|
||||
|
||||
换句话说，插件不是一组零散 helper，而应该是一个自闭合的功能模块。
|
||||
|
||||
### 2.2 宿主应用只提供共享基础设施，不承接插件内部逻辑
|
||||
|
||||
当前约束不是“插件完全独立进程”，而是：
|
||||
|
||||
1. 插件可以复用应用共享的 `engine`、`session_factory`、FastAPI app、路由树
|
||||
2. 但插件自己的表结构、仓储、运行时配置、鉴权逻辑，仍然应由插件自己拥有
|
||||
|
||||
这一点在 [`auth/plugin.toml`](/Users/rayhpeng/workspace/open-source/deer-flow/backend/app/plugins/auth/plugin.toml) 里写得很明确：
|
||||
|
||||
1. `storage.mode = "shared_infrastructure"`
|
||||
2. 说明插件拥有自己的 storage definitions 和 repositories
|
||||
3. 但复用应用共享的 persistence infrastructure
|
||||
|
||||
所以这里的契约不是“禁止复用基础设施”，而是“不要把插件内部业务实现外包给 app 其他模块”。
|
||||
|
||||
### 2.3 依赖方向要单向
|
||||
|
||||
按当前实现，比较理想的依赖方向是：
|
||||
|
||||
```text
|
||||
gateway / app bootstrap
|
||||
-> plugin public adapters
|
||||
-> plugin domain / storage / runtime
|
||||
```
|
||||
|
||||
而不是：
|
||||
|
||||
```text
|
||||
plugin domain
|
||||
-> 依赖 app 里的业务模块
|
||||
```
|
||||
|
||||
插件可以使用：
|
||||
|
||||
1. 共享持久化基础设施
|
||||
2. 宿主应用提供的 `app.state`
|
||||
3. FastAPI / Starlette 等通用框架能力
|
||||
|
||||
但不应该把自己的核心业务规则建立在别的业务模块之上，否则后续无法热插拔。
|
||||
|
||||
## 3. 当前 auth 插件的实际结构
|
||||
|
||||
当前 `auth` 插件可以概括为一套“自带模型、自带服务、自带适配器”的认证授权包。
|
||||
|
||||
### 3.1 domain
|
||||
|
||||
[`auth/domain`](/Users/rayhpeng/workspace/open-source/deer-flow/backend/app/plugins/auth/domain) 负责：
|
||||
|
||||
1. `config.py`
|
||||
- 认证相关配置定义与加载
|
||||
2. `errors.py`
|
||||
- 错误码和错误响应契约
|
||||
3. `jwt.py`
|
||||
- token 编解码
|
||||
4. `password.py`
|
||||
- 密码哈希和校验
|
||||
5. `models.py`
|
||||
- auth 域模型
|
||||
6. `service.py`
|
||||
- `AuthService`，作为核心业务服务
|
||||
|
||||
`AuthService` 本身只依赖插件内部的 `DbUserRepository` 和共享 session factory，没有把认证逻辑散到 `gateway`。
|
||||
|
||||
### 3.2 storage
|
||||
|
||||
[`auth/storage`](/Users/rayhpeng/workspace/open-source/deer-flow/backend/app/plugins/auth/storage) 明确体现了“ORM 由插件自己内部实现”的契约：
|
||||
|
||||
1. [`models.py`](/Users/rayhpeng/workspace/open-source/deer-flow/backend/app/plugins/auth/storage/models.py)
|
||||
- 定义插件自己的 `users` 表模型
|
||||
2. [`contracts.py`](/Users/rayhpeng/workspace/open-source/deer-flow/backend/app/plugins/auth/storage/contracts.py)
|
||||
- 定义 `User`、`UserCreate` 和 `UserRepositoryProtocol`
|
||||
3. [`repositories.py`](/Users/rayhpeng/workspace/open-source/deer-flow/backend/app/plugins/auth/storage/repositories.py)
|
||||
- 实现 `DbUserRepository`
|
||||
|
||||
这里的关键点是：
|
||||
|
||||
1. 插件自己定义 ORM model
|
||||
2. 插件自己定义 repository protocol
|
||||
3. 插件自己实现 repository
|
||||
4. 外部只需要给它 session / session_factory
|
||||
|
||||
这就是插件边界应该保持的最小共享面。
|
||||
|
||||
### 3.3 runtime
|
||||
|
||||
[`auth/runtime/config_state.py`](/Users/rayhpeng/workspace/open-source/deer-flow/backend/app/plugins/auth/runtime/config_state.py) 维护插件自己的 runtime config state：
|
||||
|
||||
1. `get_auth_config()`
|
||||
2. `set_auth_config()`
|
||||
3. `reset_auth_config()`
|
||||
|
||||
这说明运行时配置状态也属于插件内部，而不是由外部模块代持。后续如果别的插件需要自己的缓存、状态机、feature flag，也应沿这个模式内聚在插件内部。
|
||||
|
||||
### 3.4 adapters
|
||||
|
||||
`auth` 插件对外暴露能力主要通过四类 adapter：
|
||||
|
||||
1. `api/router.py`
|
||||
- HTTP 接口
|
||||
2. `security/*`
|
||||
- middleware、dependency、request user 解析、actor context bridge
|
||||
3. `authorization/*`
|
||||
- capability、policy evaluator、auth hooks
|
||||
4. `injection/*`
|
||||
- route policy registry、guard 注入、启动校验
|
||||
|
||||
这类 adapter 的共同特征是：
|
||||
|
||||
1. 入口能力在插件内定义
|
||||
2. 宿主应用只负责调用和装配
|
||||
|
||||
## 4. 插件如何与宿主应用交互
|
||||
|
||||
### 4.1 总路由只 include，不重写插件逻辑
|
||||
|
||||
[`app/gateway/router.py`](/Users/rayhpeng/workspace/open-source/deer-flow/backend/app/gateway/router.py) 只是：
|
||||
|
||||
1. 引入 `app.plugins.auth.api.router`
|
||||
2. `include_router(auth_router)`
|
||||
|
||||
这说明宿主应用对 auth HTTP 能力的接入是装配式的，而不是在 `gateway` 里重写一套登录/注册逻辑。
|
||||
|
||||
### 4.2 registrar 负责启动装配，不负责接管插件实现
|
||||
|
||||
[`app/gateway/registrar.py`](/Users/rayhpeng/workspace/open-source/deer-flow/backend/app/gateway/registrar.py) 里，宿主应用做的事情主要是：
|
||||
|
||||
1. `app.state.authz_hooks = build_authz_hooks()`
|
||||
2. 加载并校验 route policy registry
|
||||
3. `install_route_guards(app)`
|
||||
4. `app.add_middleware(CSRFMiddleware)`
|
||||
5. `app.add_middleware(AuthMiddleware)`
|
||||
|
||||
也就是说，宿主应用只负责把插件接进来：
|
||||
|
||||
1. 注册 middleware
|
||||
2. 安装 route guard
|
||||
3. 把 hooks 和 registry 放到 `app.state`
|
||||
|
||||
真正的鉴权逻辑、认证逻辑、路由策略语义仍然在插件内部。
|
||||
|
||||
### 4.3 共享会话工厂，但业务仓储仍归插件
|
||||
|
||||
在 [`auth/security/dependencies.py`](/Users/rayhpeng/workspace/open-source/deer-flow/backend/app/plugins/auth/security/dependencies.py) 中：
|
||||
|
||||
1. 插件从 `request.app.state.persistence.session_factory` 取得共享 session factory
|
||||
2. 然后自己构造 `DbUserRepository`
|
||||
3. 再自己构造 `AuthService`
|
||||
|
||||
这就是一个很典型的低侵入接缝：
|
||||
|
||||
1. 外部只提供共享基础设施句柄
|
||||
2. 插件自己决定如何实例化内部依赖
|
||||
|
||||
## 5. 热插拔与低侵入原则
|
||||
|
||||
### 5.1 如果要向其他模块提供服务，应尽量减少入侵
|
||||
|
||||
插件给其他模块提供服务时，优先选下面这些方式：
|
||||
|
||||
1. 暴露 router
|
||||
2. 暴露 middleware / dependency
|
||||
3. 暴露 hook 或 protocol
|
||||
4. 通过 `app.state` 注入少量共享对象
|
||||
5. 使用配置驱动的 route policy / capability，而不是把判断逻辑硬编码进业务路由
|
||||
|
||||
不推荐的方式是：
|
||||
|
||||
1. 在 `gateway` 大量写插件特定分支
|
||||
2. 让别的业务模块直接 import 插件内部 ORM 细节后自行拼逻辑
|
||||
3. 把插件状态散落到全局多个模块中共同维护
|
||||
|
||||
### 5.2 当前 auth 插件已经体现出的低侵入点
|
||||
|
||||
当前 `auth` 插件的低侵入接入点主要有四个：
|
||||
|
||||
1. 路由接入
|
||||
- `gateway.router` 只 `include_router`
|
||||
2. 中间件接入
|
||||
- `registrar` 只注册 `AuthMiddleware` / `CSRFMiddleware`
|
||||
3. 策略注入
|
||||
- `install_route_guards(app)` 给路由统一追加 `Depends(enforce_route_policy)`
|
||||
4. hook 接缝
|
||||
- `authz_hooks` 通过 `app.state` 暴露，策略构建和权限提供器可以替换
|
||||
|
||||
这套结构的好处是：
|
||||
|
||||
1. 宿主应用改动面集中在装配层
|
||||
2. 插件核心实现集中在插件目录内部
|
||||
3. 替换实现时，不需要在业务路由里逐个修改
|
||||
|
||||
### 5.3 route policy 是低侵入的关键机制
|
||||
|
||||
[`auth/injection/registry_loader.py`](/Users/rayhpeng/workspace/open-source/deer-flow/backend/app/plugins/auth/injection/registry_loader.py)、[`validation.py`](/Users/rayhpeng/workspace/open-source/deer-flow/backend/app/plugins/auth/injection/validation.py) 和 [`route_injector.py`](/Users/rayhpeng/workspace/open-source/deer-flow/backend/app/plugins/auth/injection/route_injector.py) 共同形成了一套很关键的契约：
|
||||
|
||||
1. 路由策略写在插件自己的 `route_policies.yaml`
|
||||
2. 启动时会校验策略表和真实路由是否一致
|
||||
3. guard 通过统一注入附着到路由，而不是每个 endpoint 手写一遍
|
||||
|
||||
这使得插件能够：
|
||||
|
||||
1. 用配置描述“哪些路由公开、需要哪些 capability、需要哪些 owner policy”
|
||||
2. 避免对宿主路由层做大规模侵入
|
||||
3. 在未来更容易替换或裁剪某个插件
|
||||
|
||||
## 6. 关于“ORM、runtime 都由自己内部实现”的具体说明
|
||||
|
||||
这条契约建议明确理解为以下三点：
|
||||
|
||||
1. 数据模型归插件
|
||||
- 插件自己的表、Pydantic contract、repository protocol、repository implementation 都放在插件目录内
|
||||
2. 运行时状态归插件
|
||||
- 插件自己的配置缓存、上下文桥、插件级 hooks 都在插件内部维护
|
||||
3. 外部只暴露基础设施，不接管插件语义
|
||||
- 例如共享 `session_factory`、FastAPI app、`app.state`
|
||||
|
||||
拿 `auth` 举例：
|
||||
|
||||
1. `users` 表在插件里定义，不在 `app.infra` 定义
|
||||
2. `AuthService` 在插件里实现，不在 `gateway` 实现
|
||||
3. `get_auth_config()` 在插件里维护，不由别的模块缓存
|
||||
4. `AuthMiddleware`、`route_guard`、`AuthzHooks` 都由插件自己提供
|
||||
|
||||
这是后续做插件化时最重要的结构前提。
|
||||
|
||||
## 7. 当前作用范围与非目标
|
||||
|
||||
就当前实现而言，`app.plugins` 的作用范围主要是：
|
||||
|
||||
1. 为应用侧可拆分能力建立模块边界
|
||||
2. 让插件拥有自己的 domain/storage/runtime/adapter
|
||||
3. 通过装配式接缝接入宿主应用
|
||||
|
||||
当前非目标也很明确：
|
||||
|
||||
1. 还不是一个完整的通用插件发现/安装系统
|
||||
2. 还没有做到运行时动态启停插件
|
||||
3. 也不是把共享基础设施完全复制进每个插件
|
||||
|
||||
所以“热插拔”在当前阶段更准确的含义是：
|
||||
|
||||
1. 插件边界尽量独立
|
||||
2. 接入点尽量集中在装配层
|
||||
3. 替换或移除时，改动尽量局限在 `registrar`、`router include`、`app.state` hooks 这些少数位置
|
||||
|
||||
## 8. 后续演进建议
|
||||
|
||||
如果后续要继续把 `app.plugins` 做成更稳定的插件边界，建议保持这些规则：
|
||||
|
||||
1. 每个插件目录内部都保持 `domain` / `storage` / `runtime` / `adapter` 分层
|
||||
2. 插件自己的 ORM 与 repository 不要下沉到共享业务目录
|
||||
3. 插件向外提供服务时优先暴露 protocol、hook、router、middleware，而不是要求外部 import 内部实现细节
|
||||
4. 插件与宿主应用的接缝尽量限制在：
|
||||
- `router.include_router(...)`
|
||||
- `app.add_middleware(...)`
|
||||
- `app.state.*`
|
||||
- 生命周期装配
|
||||
5. 配置驱动优先于散落的硬编码接入
|
||||
6. 启动期校验优先于运行时隐式失败
|
||||
|
||||
## 9. 设计总结
|
||||
|
||||
可以把当前 `app.plugins` 的契约总结为一句话：
|
||||
|
||||
插件内部拥有自己的业务实现、ORM 和 runtime；宿主应用只提供共享基础设施和装配接缝；对外服务时尽量通过低侵入、可替换的方式接入，以便后续做到真正的热插拔和边界演进。
|
||||
@@ -1 +0,0 @@
|
||||
"""Application plugin packages."""
|
||||
@@ -1,21 +0,0 @@
|
||||
# Auth Plugin
|
||||
|
||||
This package is the future Level 2 auth plugin boundary for DeerFlow.
|
||||
|
||||
Scope:
|
||||
|
||||
- Auth domain logic: config, errors, models, JWT, password hashing, service
|
||||
- Auth adapters: HTTP router, FastAPI dependencies, middleware, LangGraph adapter
|
||||
- Auth storage: user/account models and repositories
|
||||
|
||||
Non-scope:
|
||||
|
||||
- Shared app/container bootstrap
|
||||
- Shared persistence engine/session lifecycle
|
||||
- Generic plugin discovery/registration framework
|
||||
|
||||
Target architecture:
|
||||
|
||||
- The plugin owns its storage definitions and business logic
|
||||
- The plugin reuses the application's shared persistence infrastructure
|
||||
- The gateway only assembles the plugin instead of owning auth logic directly
|
||||
@@ -1,14 +0,0 @@
|
||||
"""Auth plugin package.
|
||||
|
||||
Level 2 plugin goal:
|
||||
- Own auth domain logic
|
||||
- Own auth adapters (router, dependencies, middleware, LangGraph adapter)
|
||||
- Own auth storage definitions
|
||||
- Reuse the application's shared persistence/session infrastructure
|
||||
"""
|
||||
|
||||
from app.plugins.auth.authorization.hooks import build_authz_hooks
|
||||
|
||||
__all__ = [
|
||||
"build_authz_hooks",
|
||||
]
|
||||
@@ -1,17 +0,0 @@
|
||||
"""HTTP API layer for the auth plugin."""
|
||||
|
||||
from app.plugins.auth.api.router import (
|
||||
ChangePasswordRequest,
|
||||
LoginResponse,
|
||||
MessageResponse,
|
||||
RegisterRequest,
|
||||
router,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"ChangePasswordRequest",
|
||||
"LoginResponse",
|
||||
"MessageResponse",
|
||||
"RegisterRequest",
|
||||
"router",
|
||||
]
|
||||
@@ -1,171 +0,0 @@
|
||||
"""Authentication endpoints for the auth plugin."""
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request, Response, status
|
||||
from fastapi.security import OAuth2PasswordRequestForm
|
||||
|
||||
from app.plugins.auth.api.schemas import (
|
||||
ChangePasswordRequest,
|
||||
InitializeAdminRequest,
|
||||
LoginResponse,
|
||||
MessageResponse,
|
||||
RegisterRequest,
|
||||
_check_rate_limit,
|
||||
_get_client_ip,
|
||||
_login_attempts,
|
||||
_record_login_failure,
|
||||
_record_login_success,
|
||||
)
|
||||
from app.plugins.auth.domain.errors import AuthErrorResponse
|
||||
from app.plugins.auth.domain.jwt import create_access_token
|
||||
from app.plugins.auth.domain.models import UserResponse
|
||||
from app.plugins.auth.domain.service import AuthServiceError
|
||||
from app.plugins.auth.runtime.config_state import get_auth_config
|
||||
from app.plugins.auth.security.csrf import is_secure_request
|
||||
from app.plugins.auth.security.dependencies import CurrentAuthService, get_current_user_from_request
|
||||
|
||||
router = APIRouter(prefix="/api/v1/auth", tags=["auth"])
|
||||
|
||||
|
||||
def _set_session_cookie(response: Response, token: str, request: Request) -> None:
|
||||
config = get_auth_config()
|
||||
is_https = is_secure_request(request)
|
||||
response.set_cookie(
|
||||
key="access_token",
|
||||
value=token,
|
||||
httponly=True,
|
||||
secure=is_https,
|
||||
samesite="lax",
|
||||
max_age=config.token_expiry_days * 24 * 3600 if is_https else None,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/login/local", response_model=LoginResponse)
|
||||
async def login_local(
|
||||
request: Request,
|
||||
response: Response,
|
||||
auth_service: CurrentAuthService,
|
||||
form_data: OAuth2PasswordRequestForm = Depends(),
|
||||
):
|
||||
client_ip = _get_client_ip(request)
|
||||
_check_rate_limit(client_ip)
|
||||
try:
|
||||
user = await auth_service.login_local(form_data.username, form_data.password)
|
||||
except AuthServiceError as exc:
|
||||
_record_login_failure(client_ip)
|
||||
raise HTTPException(
|
||||
status_code=exc.status_code,
|
||||
detail=AuthErrorResponse(code=exc.code, message=exc.message).model_dump(),
|
||||
) from exc
|
||||
|
||||
_record_login_success(client_ip)
|
||||
token = create_access_token(str(user.id), token_version=user.token_version)
|
||||
_set_session_cookie(response, token, request)
|
||||
return LoginResponse(
|
||||
expires_in=get_auth_config().token_expiry_days * 24 * 3600,
|
||||
needs_setup=user.needs_setup,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/register", response_model=UserResponse, status_code=status.HTTP_201_CREATED)
|
||||
async def register(request: Request, response: Response, body: RegisterRequest, auth_service: CurrentAuthService):
|
||||
try:
|
||||
user = await auth_service.register(body.email, body.password)
|
||||
except AuthServiceError as exc:
|
||||
raise HTTPException(
|
||||
status_code=exc.status_code,
|
||||
detail=AuthErrorResponse(code=exc.code, message=exc.message).model_dump(),
|
||||
) from exc
|
||||
|
||||
token = create_access_token(str(user.id), token_version=user.token_version)
|
||||
_set_session_cookie(response, token, request)
|
||||
return UserResponse(id=str(user.id), email=user.email, system_role=user.system_role)
|
||||
|
||||
|
||||
@router.post("/logout", response_model=MessageResponse)
|
||||
async def logout(request: Request, response: Response):
|
||||
response.delete_cookie(key="access_token", secure=is_secure_request(request), samesite="lax")
|
||||
return MessageResponse(message="Successfully logged out")
|
||||
|
||||
|
||||
@router.post("/change-password", response_model=MessageResponse)
|
||||
async def change_password(
|
||||
request: Request,
|
||||
response: Response,
|
||||
body: ChangePasswordRequest,
|
||||
auth_service: CurrentAuthService,
|
||||
):
|
||||
user = await get_current_user_from_request(request)
|
||||
try:
|
||||
user = await auth_service.change_password(
|
||||
user,
|
||||
current_password=body.current_password,
|
||||
new_password=body.new_password,
|
||||
new_email=body.new_email,
|
||||
)
|
||||
except AuthServiceError as exc:
|
||||
raise HTTPException(
|
||||
status_code=exc.status_code,
|
||||
detail=AuthErrorResponse(code=exc.code, message=exc.message).model_dump(),
|
||||
) from exc
|
||||
|
||||
token = create_access_token(str(user.id), token_version=user.token_version)
|
||||
_set_session_cookie(response, token, request)
|
||||
return MessageResponse(message="Password changed successfully")
|
||||
|
||||
|
||||
@router.get("/me", response_model=UserResponse)
|
||||
async def get_me(request: Request):
|
||||
user = await get_current_user_from_request(request)
|
||||
return UserResponse(id=str(user.id), email=user.email, system_role=user.system_role, needs_setup=user.needs_setup)
|
||||
|
||||
|
||||
@router.get("/setup-status")
|
||||
async def setup_status(auth_service: CurrentAuthService):
|
||||
return {"needs_setup": await auth_service.get_setup_status()}
|
||||
|
||||
|
||||
@router.post("/initialize", response_model=UserResponse, status_code=status.HTTP_201_CREATED)
|
||||
async def initialize_admin(
|
||||
request: Request,
|
||||
response: Response,
|
||||
body: InitializeAdminRequest,
|
||||
auth_service: CurrentAuthService,
|
||||
):
|
||||
try:
|
||||
user = await auth_service.initialize_admin(body.email, body.password)
|
||||
except AuthServiceError as exc:
|
||||
raise HTTPException(
|
||||
status_code=exc.status_code,
|
||||
detail=AuthErrorResponse(code=exc.code, message=exc.message).model_dump(),
|
||||
) from exc
|
||||
|
||||
token = create_access_token(str(user.id), token_version=user.token_version)
|
||||
_set_session_cookie(response, token, request)
|
||||
return UserResponse(id=str(user.id), email=user.email, system_role=user.system_role)
|
||||
|
||||
|
||||
@router.get("/oauth/{provider}")
|
||||
async def oauth_login(provider: str):
|
||||
if provider not in ["github", "google"]:
|
||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=f"Unsupported OAuth provider: {provider}")
|
||||
raise HTTPException(status_code=status.HTTP_501_NOT_IMPLEMENTED, detail="OAuth login not yet implemented")
|
||||
|
||||
|
||||
@router.get("/callback/{provider}")
|
||||
async def oauth_callback(provider: str, code: str, state: str):
|
||||
raise HTTPException(status_code=status.HTTP_501_NOT_IMPLEMENTED, detail="OAuth callback not yet implemented")
|
||||
|
||||
|
||||
__all__ = [
|
||||
"ChangePasswordRequest",
|
||||
"InitializeAdminRequest",
|
||||
"LoginResponse",
|
||||
"MessageResponse",
|
||||
"RegisterRequest",
|
||||
"_check_rate_limit",
|
||||
"_get_client_ip",
|
||||
"_login_attempts",
|
||||
"_record_login_failure",
|
||||
"_record_login_success",
|
||||
"router",
|
||||
]
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user