Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 xing.liu
|
d0c6f450d4 |
@@ -1,181 +0,0 @@
|
||||
---
|
||||
name: smoke-test
|
||||
description: End-to-end smoke test skill for DeerFlow. Guides through: 1) Pulling latest code, 2) Docker OR Local installation and deployment (user preference, default to Local if Docker network issues), 3) Service availability verification, 4) Health check, 5) Final test report. Use when the user says "run smoke test", "smoke test deployment", "verify installation", "test service availability", "end-to-end test", or similar.
|
||||
---
|
||||
|
||||
# DeerFlow Smoke Test Skill
|
||||
|
||||
This skill guides the Agent through DeerFlow's full end-to-end smoke test workflow, including code updates, deployment (supporting both Docker and local installation modes), service availability verification, and health checks.
|
||||
|
||||
## Deployment Mode Selection
|
||||
|
||||
This skill supports two deployment modes:
|
||||
- **Local installation mode** (recommended, especially when network issues occur) - Run all services directly on the local machine
|
||||
- **Docker mode** - Run all services inside Docker containers
|
||||
|
||||
**Selection strategy**:
|
||||
- If the user explicitly asks for Docker mode, use Docker
|
||||
- If network issues occur (such as slow image pulls), automatically switch to local mode
|
||||
- Default to local mode whenever possible
|
||||
|
||||
## Structure
|
||||
|
||||
```
|
||||
smoke-test/
|
||||
├── SKILL.md ← You are here - core workflow and logic
|
||||
├── scripts/
|
||||
│ ├── check_docker.sh ← Check the Docker environment
|
||||
│ ├── check_local_env.sh ← Check local environment dependencies
|
||||
│ ├── frontend_check.sh ← Frontend page smoke check
|
||||
│ ├── pull_code.sh ← Pull the latest code
|
||||
│ ├── deploy_docker.sh ← Docker deployment
|
||||
│ ├── deploy_local.sh ← Local deployment
|
||||
│ └── health_check.sh ← Service health check
|
||||
├── references/
|
||||
│ ├── SOP.md ← Standard operating procedure
|
||||
│ └── troubleshooting.md ← Troubleshooting guide
|
||||
└── templates/
|
||||
├── report.local.template.md ← Local mode smoke test report template
|
||||
└── report.docker.template.md ← Docker mode smoke test report template
|
||||
```
|
||||
|
||||
## Standard Operating Procedure (SOP)
|
||||
|
||||
### Phase 1: Code Update Check
|
||||
|
||||
1. **Confirm current directory** - Verify that the current working directory is the DeerFlow project root
|
||||
2. **Check Git status** - See whether there are uncommitted changes
|
||||
3. **Pull the latest code** - Use `git pull origin main` to get the latest updates
|
||||
4. **Confirm code update** - Verify that the latest code was pulled successfully
|
||||
|
||||
### Phase 2: Deployment Mode Selection and Environment Check
|
||||
|
||||
**Choose deployment mode**:
|
||||
- Ask for user preference, or choose automatically based on network conditions
|
||||
- Default to local installation mode
|
||||
|
||||
**Local mode environment check**:
|
||||
1. **Check Node.js version** - Requires 22+
|
||||
2. **Check pnpm** - Package manager
|
||||
3. **Check uv** - Python package manager
|
||||
4. **Check nginx** - Reverse proxy
|
||||
5. **Check required ports** - Confirm that ports 2026, 3000, 8001, and 2024 are not occupied
|
||||
|
||||
**Docker mode environment check** (if Docker is selected):
|
||||
1. **Check whether Docker is installed** - Run `docker --version`
|
||||
2. **Check Docker daemon status** - Run `docker info`
|
||||
3. **Check Docker Compose availability** - Run `docker compose version`
|
||||
4. **Check required ports** - Confirm that port 2026 is not occupied
|
||||
|
||||
### Phase 3: Configuration Preparation
|
||||
|
||||
1. **Check whether config.yaml exists**
|
||||
- If it does not exist, run `make config` to generate it
|
||||
- If it already exists, check whether it needs an upgrade with `make config-upgrade`
|
||||
2. **Check the .env file**
|
||||
- Verify that required environment variables are configured
|
||||
- Especially model API keys such as `OPENAI_API_KEY`
|
||||
|
||||
### Phase 4: Deployment Execution
|
||||
|
||||
**Local mode deployment**:
|
||||
1. **Check dependencies** - Run `make check`
|
||||
2. **Install dependencies** - Run `make install`
|
||||
3. **(Optional) Pre-pull the sandbox image** - If needed, run `make setup-sandbox`
|
||||
4. **Start services** - Run `make dev-daemon` (background mode, recommended) or `make dev` (foreground mode)
|
||||
5. **Wait for startup** - Give all services enough time to start completely (90-120 seconds recommended)
|
||||
|
||||
**Docker mode deployment** (if Docker is selected):
|
||||
1. **Initialize Docker environment** - Run `make docker-init`
|
||||
2. **Start Docker services** - Run `make docker-start`
|
||||
3. **Wait for startup** - Give all containers enough time to start completely (60 seconds recommended)
|
||||
|
||||
### Phase 5: Service Health Check
|
||||
|
||||
**Local mode health check**:
|
||||
1. **Check process status** - Confirm that LangGraph, Gateway, Frontend, and Nginx processes are all running
|
||||
2. **Check frontend service** - Visit `http://localhost:2026` and verify that the page loads
|
||||
3. **Check API Gateway** - Verify the `http://localhost:2026/health` endpoint
|
||||
4. **Check LangGraph service** - Verify the availability of relevant endpoints
|
||||
5. **Frontend route smoke check** - Run `bash .agent/skills/smoke-test/scripts/frontend_check.sh` to verify key routes under `/workspace`
|
||||
|
||||
**Docker mode health check** (when using Docker):
|
||||
1. **Check container status** - Run `docker ps` and confirm that all containers are running
|
||||
2. **Check frontend service** - Visit `http://localhost:2026` and verify that the page loads
|
||||
3. **Check API Gateway** - Verify the `http://localhost:2026/health` endpoint
|
||||
4. **Check LangGraph service** - Verify the availability of relevant endpoints
|
||||
5. **Frontend route smoke check** - Run `bash .agent/skills/smoke-test/scripts/frontend_check.sh` to verify key routes under `/workspace`
|
||||
|
||||
### Optional Functional Verification
|
||||
|
||||
1. **List available models** - Verify that model configuration loads correctly
|
||||
2. **List available skills** - Verify that the skill directory is mounted correctly
|
||||
3. **Simple chat test** - Send a simple message to verify the end-to-end flow
|
||||
|
||||
### Phase 6: Generate Test Report
|
||||
|
||||
1. **Collect all test results** - Summarize execution status for each phase
|
||||
2. **Record encountered issues** - If anything fails, record the error details
|
||||
3. **Generate the final report** - Use the template that matches the selected deployment mode to create the complete test report, including overall conclusion, detailed key test cases, and explicit frontend page / route results
|
||||
4. **Provide follow-up recommendations** - Offer suggestions based on the test results
|
||||
|
||||
## Execution Rules
|
||||
|
||||
- **Follow the sequence** - Execute strictly in the order described above
|
||||
- **Idempotency** - Every step should be safe to repeat
|
||||
- **Error handling** - If a step fails, stop and report the issue, then provide troubleshooting suggestions
|
||||
- **Detailed logging** - Record the execution result and status of each step
|
||||
- **User confirmation** - Ask for confirmation before potentially risky operations such as overwriting config
|
||||
- **Mode preference** - Prefer local mode to avoid network-related issues
|
||||
- **Template requirement** - The final report must use the matching template under `templates/`; do not output a free-form summary instead of the template-based report
|
||||
- **Report clarity** - The execution summary must include the overall pass/fail conclusion plus per-case result explanations, and frontend smoke check results must be listed explicitly in the report
|
||||
- **Optional phase handling** - If functional verification is not executed, do not present it as a separate skipped phase in the final report
|
||||
|
||||
## Known Acceptable Warnings
|
||||
|
||||
The following warnings can appear during smoke testing and do not block a successful result:
|
||||
- Feishu/Lark SSL errors in Gateway logs (certificate verification failure) can be ignored if that channel is not enabled
|
||||
- Warnings in LangGraph logs about missing methods in the custom checkpointer, such as `adelete_for_runs` or `aprune`, do not affect the core functionality
|
||||
|
||||
## Key Tools
|
||||
|
||||
Use the following tools during execution:
|
||||
|
||||
1. **bash** - Run shell commands
|
||||
2. **present_file** - Show generated reports and important files
|
||||
3. **task_tool** - Organize complex steps with subtasks when needed
|
||||
|
||||
## Success Criteria
|
||||
|
||||
Smoke test pass criteria (local mode):
|
||||
- [x] Latest code is pulled successfully
|
||||
- [x] Local environment check passes (Node.js 22+, pnpm, uv, nginx)
|
||||
- [x] Configuration files are set up correctly
|
||||
- [x] `make check` passes
|
||||
- [x] `make install` completes successfully
|
||||
- [x] `make dev` starts successfully
|
||||
- [x] All service processes run normally
|
||||
- [x] Frontend page is accessible
|
||||
- [x] Frontend route smoke check passes (`/workspace` key routes)
|
||||
- [x] API Gateway health check passes
|
||||
- [x] Test report is generated completely
|
||||
|
||||
Smoke test pass criteria (Docker mode):
|
||||
- [x] Latest code is pulled successfully
|
||||
- [x] Docker environment check passes
|
||||
- [x] Configuration files are set up correctly
|
||||
- [x] `make docker-init` completes successfully
|
||||
- [x] `make docker-start` completes successfully
|
||||
- [x] All Docker containers run normally
|
||||
- [x] Frontend page is accessible
|
||||
- [x] Frontend route smoke check passes (`/workspace` key routes)
|
||||
- [x] API Gateway health check passes
|
||||
- [x] Test report is generated completely
|
||||
|
||||
## Read Reference Files
|
||||
|
||||
Before starting execution, read the following reference files:
|
||||
1. `references/SOP.md` - Detailed step-by-step operating instructions
|
||||
2. `references/troubleshooting.md` - Common issues and solutions
|
||||
3. `templates/report.local.template.md` - Local mode test report template
|
||||
4. `templates/report.docker.template.md` - Docker mode test report template
|
||||
@@ -1,452 +0,0 @@
|
||||
# DeerFlow Smoke Test Standard Operating Procedure (SOP)
|
||||
|
||||
This document describes the detailed operating steps for each phase of the DeerFlow smoke test.
|
||||
|
||||
## Phase 1: Code Update Check
|
||||
|
||||
### 1.1 Confirm Current Directory
|
||||
|
||||
**Objective**: Verify that the current working directory is the DeerFlow project root.
|
||||
|
||||
**Steps**:
|
||||
1. Run `pwd` to view the current working directory
|
||||
2. Check whether the directory contains the following files/directories:
|
||||
- `Makefile`
|
||||
- `backend/`
|
||||
- `frontend/`
|
||||
- `config.example.yaml`
|
||||
|
||||
**Success Criteria**: The current directory contains all of the files/directories listed above.
|
||||
|
||||
---
|
||||
|
||||
### 1.2 Check Git Status
|
||||
|
||||
**Objective**: Check whether there are uncommitted changes.
|
||||
|
||||
**Steps**:
|
||||
1. Run `git status`
|
||||
2. Check whether the output includes "Changes not staged for commit" or "Untracked files"
|
||||
|
||||
**Notes**:
|
||||
- If there are uncommitted changes, recommend that the user commit or stash them first to avoid conflicts while pulling
|
||||
- If the user confirms that they want to continue, this step can be skipped
|
||||
|
||||
---
|
||||
|
||||
### 1.3 Pull the Latest Code
|
||||
|
||||
**Objective**: Fetch the latest code updates.
|
||||
|
||||
**Steps**:
|
||||
1. Run `git fetch origin main`
|
||||
2. Run `git pull origin main`
|
||||
|
||||
**Success Criteria**:
|
||||
- The commands succeed without errors
|
||||
- The output shows "Already up to date" or indicates that new commits were pulled successfully
|
||||
|
||||
---
|
||||
|
||||
### 1.4 Confirm Code Update
|
||||
|
||||
**Objective**: Verify that the latest code was pulled successfully.
|
||||
|
||||
**Steps**:
|
||||
1. Run `git log -1 --oneline` to view the latest commit
|
||||
2. Record the commit hash and message
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Deployment Mode Selection and Environment Check
|
||||
|
||||
### 2.1 Choose Deployment Mode
|
||||
|
||||
**Objective**: Decide whether to use local mode or Docker mode.
|
||||
|
||||
**Decision Flow**:
|
||||
1. Prefer local mode first to avoid network-related issues
|
||||
2. If the user explicitly requests Docker, use Docker
|
||||
3. If Docker network issues occur, switch to local mode automatically
|
||||
|
||||
---
|
||||
|
||||
### 2.2 Local Mode Environment Check
|
||||
|
||||
**Objective**: Verify that local development environment dependencies are satisfied.
|
||||
|
||||
#### 2.2.1 Check Node.js Version
|
||||
|
||||
**Steps**:
|
||||
1. If nvm is used, run `nvm use 22` to switch to Node 22+
|
||||
2. Run `node --version`
|
||||
|
||||
**Success Criteria**: Version >= 22.x
|
||||
|
||||
**Failure Handling**:
|
||||
- If the version is too low, ask the user to install/switch Node.js with nvm:
|
||||
```bash
|
||||
nvm install 22
|
||||
nvm use 22
|
||||
```
|
||||
- Or install it from the official website: https://nodejs.org/
|
||||
|
||||
---
|
||||
|
||||
#### 2.2.2 Check pnpm
|
||||
|
||||
**Steps**:
|
||||
1. Run `pnpm --version`
|
||||
|
||||
**Success Criteria**: The command returns pnpm version information.
|
||||
|
||||
**Failure Handling**:
|
||||
- If pnpm is not installed, ask the user to install it with `npm install -g pnpm`
|
||||
|
||||
---
|
||||
|
||||
#### 2.2.3 Check uv
|
||||
|
||||
**Steps**:
|
||||
1. Run `uv --version`
|
||||
|
||||
**Success Criteria**: The command returns uv version information.
|
||||
|
||||
**Failure Handling**:
|
||||
- If uv is not installed, ask the user to install uv
|
||||
|
||||
---
|
||||
|
||||
#### 2.2.4 Check nginx
|
||||
|
||||
**Steps**:
|
||||
1. Run `nginx -v`
|
||||
|
||||
**Success Criteria**: The command returns nginx version information.
|
||||
|
||||
**Failure Handling**:
|
||||
- macOS: install with Homebrew using `brew install nginx`
|
||||
- Linux: install using the system package manager
|
||||
|
||||
---
|
||||
|
||||
#### 2.2.5 Check Required Ports
|
||||
|
||||
**Steps**:
|
||||
1. Run the following commands to check ports:
|
||||
```bash
|
||||
lsof -i :2026 # Main port
|
||||
lsof -i :3000 # Frontend
|
||||
lsof -i :8001 # Gateway
|
||||
lsof -i :2024 # LangGraph
|
||||
```
|
||||
|
||||
**Success Criteria**: All ports are free, or they are occupied only by DeerFlow-related processes.
|
||||
|
||||
**Failure Handling**:
|
||||
- If a port is occupied, ask the user to stop the related process
|
||||
|
||||
---
|
||||
|
||||
### 2.3 Docker Mode Environment Check (If Docker Is Selected)
|
||||
|
||||
#### 2.3.1 Check Whether Docker Is Installed
|
||||
|
||||
**Steps**:
|
||||
1. Run `docker --version`
|
||||
|
||||
**Success Criteria**: The command returns Docker version information, such as "Docker version 24.x.x".
|
||||
|
||||
---
|
||||
|
||||
#### 2.3.2 Check Docker Daemon Status
|
||||
|
||||
**Steps**:
|
||||
1. Run `docker info`
|
||||
|
||||
**Success Criteria**: The command runs successfully and shows Docker system information.
|
||||
|
||||
**Failure Handling**:
|
||||
- If it fails, ask the user to start Docker Desktop or the Docker service
|
||||
|
||||
---
|
||||
|
||||
#### 2.3.3 Check Docker Compose Availability
|
||||
|
||||
**Steps**:
|
||||
1. Run `docker compose version`
|
||||
|
||||
**Success Criteria**: The command returns Docker Compose version information.
|
||||
|
||||
---
|
||||
|
||||
#### 2.3.4 Check Required Ports
|
||||
|
||||
**Steps**:
|
||||
1. Run `lsof -i :2026` (macOS/Linux) or `netstat -ano | findstr :2026` (Windows)
|
||||
|
||||
**Success Criteria**: Port 2026 is free, or it is occupied only by a DeerFlow-related process.
|
||||
|
||||
**Failure Handling**:
|
||||
- If the port is occupied by another process, ask the user to stop that process or change the configuration
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Configuration Preparation
|
||||
|
||||
### 3.1 Check config.yaml
|
||||
|
||||
**Steps**:
|
||||
1. Check whether `config.yaml` exists
|
||||
2. If it does not exist, run `make config`
|
||||
3. If it already exists, consider running `make config-upgrade` to merge new fields
|
||||
|
||||
**Validation**:
|
||||
- Check whether at least one model is configured in config.yaml
|
||||
- Check whether the model configuration references the correct environment variables
|
||||
|
||||
---
|
||||
|
||||
### 3.2 Check the .env File
|
||||
|
||||
**Steps**:
|
||||
1. Check whether the `.env` file exists
|
||||
2. If it does not exist, copy it from `.env.example`
|
||||
3. Check whether the following environment variables are configured:
|
||||
- `OPENAI_API_KEY` (or other model API keys)
|
||||
- Other required settings
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Deployment Execution
|
||||
|
||||
### 4.1 Local Mode Deployment
|
||||
|
||||
#### 4.1.1 Check Dependencies
|
||||
|
||||
**Steps**:
|
||||
1. Run `make check`
|
||||
|
||||
**Description**: This command validates all required tools (Node.js 22+, pnpm, uv, nginx).
|
||||
|
||||
---
|
||||
|
||||
#### 4.1.2 Install Dependencies
|
||||
|
||||
**Steps**:
|
||||
1. Run `make install`
|
||||
|
||||
**Description**: This command installs both backend and frontend dependencies.
|
||||
|
||||
**Notes**:
|
||||
- This step may take some time
|
||||
- If network issues cause failures, try using a closer or mirrored package registry
|
||||
|
||||
---
|
||||
|
||||
#### 4.1.3 (Optional) Pre-pull the Sandbox Image
|
||||
|
||||
**Steps**:
|
||||
1. If Docker / Container sandbox is used, run `make setup-sandbox`
|
||||
|
||||
**Description**: This step is optional and not needed for local sandbox mode.
|
||||
|
||||
---
|
||||
|
||||
#### 4.1.4 Start Services
|
||||
|
||||
**Steps**:
|
||||
1. Run `make dev-daemon` (background mode)
|
||||
|
||||
**Description**: This command starts all services (LangGraph, Gateway, Frontend, Nginx).
|
||||
|
||||
**Notes**:
|
||||
- `make dev` runs in the foreground and stops with Ctrl+C
|
||||
- `make dev-daemon` runs in the background
|
||||
- Use `make stop` to stop services
|
||||
|
||||
---
|
||||
|
||||
#### 4.1.5 Wait for Services to Start
|
||||
|
||||
**Steps**:
|
||||
1. Wait 90-120 seconds for all services to start completely
|
||||
2. You can monitor startup progress by checking these log files:
|
||||
- `logs/langgraph.log`
|
||||
- `logs/gateway.log`
|
||||
- `logs/frontend.log`
|
||||
- `logs/nginx.log`
|
||||
|
||||
---
|
||||
|
||||
### 4.2 Docker Mode Deployment (If Docker Is Selected)
|
||||
|
||||
#### 4.2.1 Initialize the Docker Environment
|
||||
|
||||
**Steps**:
|
||||
1. Run `make docker-init`
|
||||
|
||||
**Description**: This command pulls the sandbox image if needed.
|
||||
|
||||
---
|
||||
|
||||
#### 4.2.2 Start Docker Services
|
||||
|
||||
**Steps**:
|
||||
1. Run `make docker-start`
|
||||
|
||||
**Description**: This command builds and starts all required Docker containers.
|
||||
|
||||
---
|
||||
|
||||
#### 4.2.3 Wait for Services to Start
|
||||
|
||||
**Steps**:
|
||||
1. Wait 60-90 seconds for all services to start completely
|
||||
2. You can run `make docker-logs` to monitor startup progress
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: Service Health Check
|
||||
|
||||
### 5.1 Local Mode Health Check
|
||||
|
||||
#### 5.1.1 Check Process Status
|
||||
|
||||
**Steps**:
|
||||
1. Run the following command to check processes:
|
||||
```bash
|
||||
ps aux | grep -E "(langgraph|uvicorn|next|nginx)" | grep -v grep
|
||||
```
|
||||
|
||||
**Success Criteria**: Confirm that the following processes are running:
|
||||
- LangGraph (`langgraph dev`)
|
||||
- Gateway (`uvicorn app.gateway.app:app`)
|
||||
- Frontend (`next dev` or `next start`)
|
||||
- Nginx (`nginx`)
|
||||
|
||||
---
|
||||
|
||||
#### 5.1.2 Check Frontend Service
|
||||
|
||||
**Steps**:
|
||||
1. Use curl or a browser to visit `http://localhost:2026`
|
||||
2. Verify that the page loads normally
|
||||
|
||||
**Example curl command**:
|
||||
```bash
|
||||
curl -I http://localhost:2026
|
||||
```
|
||||
|
||||
**Success Criteria**: Returns an HTTP 200 status code.
|
||||
|
||||
---
|
||||
|
||||
#### 5.1.3 Check API Gateway
|
||||
|
||||
**Steps**:
|
||||
1. Visit `http://localhost:2026/health`
|
||||
|
||||
**Example curl command**:
|
||||
```bash
|
||||
curl http://localhost:2026/health
|
||||
```
|
||||
|
||||
**Success Criteria**: Returns health status JSON.
|
||||
|
||||
---
|
||||
|
||||
#### 5.1.4 Check LangGraph Service
|
||||
|
||||
**Steps**:
|
||||
1. Visit relevant LangGraph endpoints to verify availability
|
||||
|
||||
---
|
||||
|
||||
### 5.2 Docker Mode Health Check (When Using Docker)
|
||||
|
||||
#### 5.2.1 Check Container Status
|
||||
|
||||
**Steps**:
|
||||
1. Run `docker ps`
|
||||
2. Confirm that the following containers are running:
|
||||
- `deer-flow-nginx`
|
||||
- `deer-flow-frontend`
|
||||
- `deer-flow-gateway`
|
||||
- `deer-flow-langgraph` (if not in gateway mode)
|
||||
|
||||
---
|
||||
|
||||
#### 5.2.2 Check Frontend Service
|
||||
|
||||
**Steps**:
|
||||
1. Use curl or a browser to visit `http://localhost:2026`
|
||||
2. Verify that the page loads normally
|
||||
|
||||
**Example curl command**:
|
||||
```bash
|
||||
curl -I http://localhost:2026
|
||||
```
|
||||
|
||||
**Success Criteria**: Returns an HTTP 200 status code.
|
||||
|
||||
---
|
||||
|
||||
#### 5.2.3 Check API Gateway
|
||||
|
||||
**Steps**:
|
||||
1. Visit `http://localhost:2026/health`
|
||||
|
||||
**Example curl command**:
|
||||
```bash
|
||||
curl http://localhost:2026/health
|
||||
```
|
||||
|
||||
**Success Criteria**: Returns health status JSON.
|
||||
|
||||
---
|
||||
|
||||
#### 5.2.4 Check LangGraph Service
|
||||
|
||||
**Steps**:
|
||||
1. Visit relevant LangGraph endpoints to verify availability
|
||||
|
||||
---
|
||||
|
||||
## Optional Functional Verification
|
||||
|
||||
### 6.1 List Available Models
|
||||
|
||||
**Steps**: Verify the model list through the API or UI.
|
||||
|
||||
---
|
||||
|
||||
### 6.2 List Available Skills
|
||||
|
||||
**Steps**: Verify the skill list through the API or UI.
|
||||
|
||||
---
|
||||
|
||||
### 6.3 Simple Chat Test
|
||||
|
||||
**Steps**: Send a simple message to test the complete workflow.
|
||||
|
||||
---
|
||||
|
||||
## Phase 6: Generate the Test Report
|
||||
|
||||
### 6.1 Collect Test Results
|
||||
|
||||
Summarize the execution status of each phase and record successful and failed items.
|
||||
|
||||
### 6.2 Record Issues
|
||||
|
||||
If anything fails, record detailed error information.
|
||||
|
||||
### 6.3 Generate the Report
|
||||
|
||||
Use the template to create a complete test report.
|
||||
|
||||
### 6.4 Provide Recommendations
|
||||
|
||||
Provide follow-up recommendations based on the test results.
|
||||
@@ -1,612 +0,0 @@
|
||||
# Troubleshooting Guide
|
||||
|
||||
This document lists common issues encountered during DeerFlow smoke testing and how to resolve them.
|
||||
|
||||
## Code Update Issues
|
||||
|
||||
### Issue: `git pull` Fails with a Merge Conflict Warning
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
error: Your local changes to the following files would be overwritten by merge
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
1. Option A: Commit local changes first
|
||||
```bash
|
||||
git add .
|
||||
git commit -m "Save local changes"
|
||||
git pull origin main
|
||||
```
|
||||
|
||||
2. Option B: Stash local changes
|
||||
```bash
|
||||
git stash
|
||||
git pull origin main
|
||||
git stash pop # Restore changes later if needed
|
||||
```
|
||||
|
||||
3. Option C: Discard local changes (use with caution)
|
||||
```bash
|
||||
git reset --hard HEAD
|
||||
git pull origin main
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Local Mode Environment Issues
|
||||
|
||||
### Issue: Node.js Version Is Too Old
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
Node.js version is too old. Requires 22+, got x.x.x
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
1. Install or upgrade Node.js with nvm:
|
||||
```bash
|
||||
nvm install 22
|
||||
nvm use 22
|
||||
```
|
||||
|
||||
2. Or download and install it from the official website: https://nodejs.org/
|
||||
|
||||
3. Verify the version:
|
||||
```bash
|
||||
node --version
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: pnpm Is Not Installed
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
command not found: pnpm
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
1. Install pnpm with npm:
|
||||
```bash
|
||||
npm install -g pnpm
|
||||
```
|
||||
|
||||
2. Or use the official installation script:
|
||||
```bash
|
||||
curl -fsSL https://get.pnpm.io/install.sh | sh -
|
||||
```
|
||||
|
||||
3. Verify the installation:
|
||||
```bash
|
||||
pnpm --version
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: uv Is Not Installed
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
command not found: uv
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
1. Use the official installation script:
|
||||
```bash
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
```
|
||||
|
||||
2. macOS users can also install it with Homebrew:
|
||||
```bash
|
||||
brew install uv
|
||||
```
|
||||
|
||||
3. Verify the installation:
|
||||
```bash
|
||||
uv --version
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: nginx Is Not Installed
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
command not found: nginx
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
1. macOS (Homebrew):
|
||||
```bash
|
||||
brew install nginx
|
||||
```
|
||||
|
||||
2. Ubuntu/Debian:
|
||||
```bash
|
||||
sudo apt update
|
||||
sudo apt install nginx
|
||||
```
|
||||
|
||||
3. CentOS/RHEL:
|
||||
```bash
|
||||
sudo yum install nginx
|
||||
```
|
||||
|
||||
4. Verify the installation:
|
||||
```bash
|
||||
nginx -v
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: Port Is Already in Use
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
Error: listen EADDRINUSE: address already in use :::2026
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
1. Find the process using the port:
|
||||
```bash
|
||||
lsof -i :2026 # macOS/Linux
|
||||
netstat -ano | findstr :2026 # Windows
|
||||
```
|
||||
|
||||
2. Stop that process:
|
||||
```bash
|
||||
kill -9 <PID> # macOS/Linux
|
||||
taskkill /PID <PID> /F # Windows
|
||||
```
|
||||
|
||||
3. Or stop DeerFlow services first:
|
||||
```bash
|
||||
make stop
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Local Mode Dependency Installation Issues
|
||||
|
||||
### Issue: `make install` Fails Due to Network Timeout
|
||||
|
||||
**Symptoms**:
|
||||
Network timeouts or connection failures occur during dependency installation.
|
||||
|
||||
**Solutions**:
|
||||
1. Configure pnpm to use a mirror registry:
|
||||
```bash
|
||||
pnpm config set registry https://registry.npmmirror.com
|
||||
```
|
||||
|
||||
2. Configure uv to use a mirror registry:
|
||||
```bash
|
||||
uv pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
|
||||
```
|
||||
|
||||
3. Retry the installation:
|
||||
```bash
|
||||
make install
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: Python Dependency Installation Fails
|
||||
|
||||
**Symptoms**:
|
||||
Errors occur during `uv sync`.
|
||||
|
||||
**Solutions**:
|
||||
1. Clean the uv cache:
|
||||
```bash
|
||||
cd backend
|
||||
uv cache clean
|
||||
```
|
||||
|
||||
2. Resync dependencies:
|
||||
```bash
|
||||
cd backend
|
||||
uv sync
|
||||
```
|
||||
|
||||
3. View detailed error logs:
|
||||
```bash
|
||||
cd backend
|
||||
uv sync --verbose
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: Frontend Dependency Installation Fails
|
||||
|
||||
**Symptoms**:
|
||||
Errors occur during `pnpm install`.
|
||||
|
||||
**Solutions**:
|
||||
1. Clean the pnpm cache:
|
||||
```bash
|
||||
cd frontend
|
||||
pnpm store prune
|
||||
```
|
||||
|
||||
2. Remove node_modules and the lock file:
|
||||
```bash
|
||||
cd frontend
|
||||
rm -rf node_modules pnpm-lock.yaml
|
||||
```
|
||||
|
||||
3. Reinstall:
|
||||
```bash
|
||||
cd frontend
|
||||
pnpm install
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Local Mode Service Startup Issues
|
||||
|
||||
### Issue: Services Exit Immediately After Startup
|
||||
|
||||
**Symptoms**:
|
||||
Processes exit quickly after running `make dev-daemon`.
|
||||
|
||||
**Solutions**:
|
||||
1. Check log files:
|
||||
```bash
|
||||
tail -f logs/langgraph.log
|
||||
tail -f logs/gateway.log
|
||||
tail -f logs/frontend.log
|
||||
tail -f logs/nginx.log
|
||||
```
|
||||
|
||||
2. Check whether config.yaml is configured correctly
|
||||
3. Check environment variables in the .env file
|
||||
4. Confirm that required ports are not occupied
|
||||
5. Stop all services and restart:
|
||||
```bash
|
||||
make stop
|
||||
make dev-daemon
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: Nginx Fails to Start Because Temp Directories Do Not Exist
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
nginx: [emerg] mkdir() "/opt/homebrew/var/run/nginx/client_body_temp" failed (2: No such file or directory)
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
Add local temp directory configuration to `docker/nginx/nginx.local.conf` so nginx uses the repository's temp directory.
|
||||
|
||||
Add the following at the beginning of the `http` block:
|
||||
```nginx
|
||||
client_body_temp_path temp/client_body_temp;
|
||||
proxy_temp_path temp/proxy_temp;
|
||||
fastcgi_temp_path temp/fastcgi_temp;
|
||||
uwsgi_temp_path temp/uwsgi_temp;
|
||||
scgi_temp_path temp/scgi_temp;
|
||||
```
|
||||
|
||||
Note: The `temp/` directory under the repository root is created automatically by `make dev` or `make dev-daemon`.
|
||||
|
||||
---
|
||||
|
||||
### Issue: Nginx Fails to Start (General)
|
||||
|
||||
**Symptoms**:
|
||||
The nginx process fails to start or reports an error.
|
||||
|
||||
**Solutions**:
|
||||
1. Check the nginx configuration:
|
||||
```bash
|
||||
nginx -t -c docker/nginx/nginx.local.conf -p .
|
||||
```
|
||||
|
||||
2. Check nginx logs:
|
||||
```bash
|
||||
tail -f logs/nginx.log
|
||||
```
|
||||
|
||||
3. Ensure no other nginx process is running:
|
||||
```bash
|
||||
ps aux | grep nginx
|
||||
```
|
||||
|
||||
4. If needed, stop existing nginx processes:
|
||||
```bash
|
||||
pkill -9 nginx
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: Frontend Compilation Fails
|
||||
|
||||
**Symptoms**:
|
||||
Compilation errors appear in `frontend.log`.
|
||||
|
||||
**Solutions**:
|
||||
1. Check frontend logs:
|
||||
```bash
|
||||
tail -f logs/frontend.log
|
||||
```
|
||||
|
||||
2. Check whether Node.js version is 22+
|
||||
3. Reinstall frontend dependencies:
|
||||
```bash
|
||||
cd frontend
|
||||
rm -rf node_modules .next
|
||||
pnpm install
|
||||
```
|
||||
|
||||
4. Restart services:
|
||||
```bash
|
||||
make stop
|
||||
make dev-daemon
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Issue: Gateway Fails to Start
|
||||
|
||||
**Symptoms**:
|
||||
Errors appear in `gateway.log`.
|
||||
|
||||
**Solutions**:
|
||||
1. Check gateway logs:
|
||||
```bash
|
||||
tail -f logs/gateway.log
|
||||
```
|
||||
|
||||
2. Check whether config.yaml exists and has valid formatting
|
||||
3. Check whether Python dependencies are complete:
|
||||
```bash
|
||||
cd backend
|
||||
uv sync
|
||||
```
|
||||
|
||||
4. Confirm that the LangGraph service is running normally (if not in gateway mode)
|
||||
|
||||
---
|
||||
|
||||
### Issue: LangGraph Fails to Start
|
||||
|
||||
**Symptoms**:
|
||||
Errors appear in `langgraph.log`.
|
||||
|
||||
**Solutions**:
|
||||
1. Check LangGraph logs:
|
||||
```bash
|
||||
tail -f logs/langgraph.log
|
||||
```
|
||||
|
||||
2. Check config.yaml
|
||||
3. Check whether Python dependencies are complete
|
||||
4. Confirm that port 2024 is not occupied
|
||||
|
||||
---
|
||||
|
||||
## Docker-Related Issues
|
||||
|
||||
### Issue: Docker Commands Cannot Run
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
Cannot connect to the Docker daemon
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
1. Confirm that Docker Desktop is running
|
||||
2. macOS: check whether the Docker icon appears in the top menu bar
|
||||
3. Linux: run `sudo systemctl start docker`
|
||||
4. Run `docker info` again to verify
|
||||
|
||||
---
|
||||
|
||||
### Issue: `make docker-init` Fails to Pull the Image
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
Error pulling image: connection refused
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
1. Check network connectivity
|
||||
2. Configure a Docker image mirror if needed
|
||||
3. Check whether a proxy is required
|
||||
4. Switch to local installation mode if necessary (recommended)
|
||||
|
||||
---
|
||||
|
||||
## Configuration File Issues
|
||||
|
||||
### Issue: config.yaml Is Missing or Invalid
|
||||
|
||||
**Symptoms**:
|
||||
```
|
||||
Error: could not read config.yaml
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
1. Regenerate the configuration file:
|
||||
```bash
|
||||
make config
|
||||
```
|
||||
|
||||
2. Check YAML syntax:
|
||||
- Make sure indentation is correct (use 2 spaces)
|
||||
- Make sure there are no tab characters
|
||||
- Check that there is a space after each colon
|
||||
|
||||
3. Use a YAML validation tool to check the format
|
||||
|
||||
---
|
||||
|
||||
### Issue: Model API Key Is Not Configured
|
||||
|
||||
**Symptoms**:
|
||||
After services start, API requests fail with authentication errors.
|
||||
|
||||
**Solutions**:
|
||||
1. Edit the .env file and add the API key:
|
||||
```bash
|
||||
OPENAI_API_KEY=your-actual-api-key-here
|
||||
```
|
||||
|
||||
2. Restart services (local mode):
|
||||
```bash
|
||||
make stop
|
||||
make dev-daemon
|
||||
```
|
||||
|
||||
3. Restart services (Docker mode):
|
||||
```bash
|
||||
make docker-stop
|
||||
make docker-start
|
||||
```
|
||||
|
||||
4. Confirm that the model configuration in config.yaml references the environment variable correctly
|
||||
|
||||
---
|
||||
|
||||
## Service Health Check Issues
|
||||
|
||||
### Issue: Frontend Page Is Not Accessible
|
||||
|
||||
**Symptoms**:
|
||||
The browser shows a connection failure when visiting http://localhost:2026.
|
||||
|
||||
**Solutions** (local mode):
|
||||
1. Confirm that the nginx process is running:
|
||||
```bash
|
||||
ps aux | grep nginx
|
||||
```
|
||||
|
||||
2. Check nginx logs:
|
||||
```bash
|
||||
tail -f logs/nginx.log
|
||||
```
|
||||
|
||||
3. Check firewall settings
|
||||
|
||||
**Solutions** (Docker mode):
|
||||
1. Confirm that the nginx container is running:
|
||||
```bash
|
||||
docker ps | grep nginx
|
||||
```
|
||||
|
||||
2. Check nginx logs:
|
||||
```bash
|
||||
cd docker && docker compose -p deer-flow-dev -f docker-compose-dev.yaml logs nginx
|
||||
```
|
||||
|
||||
3. Check firewall settings
|
||||
|
||||
---
|
||||
|
||||
### Issue: API Gateway Health Check Fails
|
||||
|
||||
**Symptoms**:
|
||||
Accessing `/health` returns an error or times out.
|
||||
|
||||
**Solutions** (local mode):
|
||||
1. Check gateway logs:
|
||||
```bash
|
||||
tail -f logs/gateway.log
|
||||
```
|
||||
|
||||
2. Confirm that config.yaml exists and has valid formatting
|
||||
3. Check whether Python dependencies are complete
|
||||
4. Confirm that the LangGraph service is running normally
|
||||
|
||||
**Solutions** (Docker mode):
|
||||
1. Check gateway container logs:
|
||||
```bash
|
||||
make docker-logs-gateway
|
||||
```
|
||||
|
||||
2. Confirm that config.yaml is mounted correctly
|
||||
3. Check whether Python dependencies are complete
|
||||
4. Confirm that the LangGraph service is running normally
|
||||
|
||||
---
|
||||
|
||||
## Common Diagnostic Commands
|
||||
|
||||
### Local Mode Diagnostics
|
||||
|
||||
#### View All Service Processes
|
||||
```bash
|
||||
ps aux | grep -E "(langgraph|uvicorn|next|nginx)" | grep -v grep
|
||||
```
|
||||
|
||||
#### View Service Logs
|
||||
```bash
|
||||
# View all logs
|
||||
tail -f logs/*.log
|
||||
|
||||
# View specific service logs
|
||||
tail -f logs/langgraph.log
|
||||
tail -f logs/gateway.log
|
||||
tail -f logs/frontend.log
|
||||
tail -f logs/nginx.log
|
||||
```
|
||||
|
||||
#### Stop All Services
|
||||
```bash
|
||||
make stop
|
||||
```
|
||||
|
||||
#### Fully Reset the Local Environment
|
||||
```bash
|
||||
make stop
|
||||
make clean
|
||||
make config
|
||||
make install
|
||||
make dev-daemon
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Docker Mode Diagnostics
|
||||
|
||||
#### View All Container Status
|
||||
```bash
|
||||
docker ps -a
|
||||
```
|
||||
|
||||
#### View Container Resource Usage
|
||||
```bash
|
||||
docker stats
|
||||
```
|
||||
|
||||
#### Enter a Container for Debugging
|
||||
```bash
|
||||
docker exec -it deer-flow-gateway sh
|
||||
```
|
||||
|
||||
#### Clean Up All DeerFlow-Related Containers and Images
|
||||
```bash
|
||||
make docker-stop
|
||||
cd docker && docker compose -p deer-flow-dev -f docker-compose-dev.yaml down -v
|
||||
```
|
||||
|
||||
#### Fully Reset the Docker Environment
|
||||
```bash
|
||||
make docker-stop
|
||||
make clean
|
||||
make config
|
||||
make docker-init
|
||||
make docker-start
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Get More Help
|
||||
|
||||
If the solutions above do not resolve the issue:
|
||||
1. Check the GitHub issues for the project: https://github.com/bytedance/deer-flow/issues
|
||||
2. Review the project documentation: README.md and the `backend/docs/` directory
|
||||
3. Open a new issue and include detailed error logs
|
||||
@@ -1,80 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
set -e
|
||||
|
||||
echo "=========================================="
|
||||
echo " Checking Docker Environment"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
|
||||
# Check whether Docker is installed
|
||||
if command -v docker >/dev/null 2>&1; then
|
||||
echo "✓ Docker is installed"
|
||||
docker --version
|
||||
else
|
||||
echo "✗ Docker is not installed"
|
||||
exit 1
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check the Docker daemon
|
||||
if docker info >/dev/null 2>&1; then
|
||||
echo "✓ Docker daemon is running normally"
|
||||
else
|
||||
echo "✗ Docker daemon is not running"
|
||||
echo " Please start Docker Desktop or the Docker service"
|
||||
exit 1
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check Docker Compose
|
||||
if docker compose version >/dev/null 2>&1; then
|
||||
echo "✓ Docker Compose is available"
|
||||
docker compose version
|
||||
else
|
||||
echo "✗ Docker Compose is not available"
|
||||
exit 1
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check port 2026
|
||||
if ! command -v lsof >/dev/null 2>&1; then
|
||||
echo "✗ lsof is required to check whether port 2026 is available"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
port_2026_usage="$(lsof -nP -iTCP:2026 -sTCP:LISTEN 2>/dev/null || true)"
|
||||
if [ -n "$port_2026_usage" ]; then
|
||||
echo "⚠ Port 2026 is already in use"
|
||||
echo " Occupying process:"
|
||||
echo "$port_2026_usage"
|
||||
|
||||
deerflow_process_found=0
|
||||
while IFS= read -r pid; do
|
||||
if [ -z "$pid" ]; then
|
||||
continue
|
||||
fi
|
||||
|
||||
process_command="$(ps -p "$pid" -o command= 2>/dev/null || true)"
|
||||
case "$process_command" in
|
||||
*[Dd]eer[Ff]low*|*[Dd]eerflow*|*[Nn]ginx*deerflow*|*deerflow/*[Nn]ginx*)
|
||||
deerflow_process_found=1
|
||||
;;
|
||||
esac
|
||||
done <<EOF
|
||||
$(printf '%s\n' "$port_2026_usage" | awk 'NR > 1 {print $2}')
|
||||
EOF
|
||||
|
||||
if [ "$deerflow_process_found" -eq 1 ]; then
|
||||
echo "✓ Port 2026 is occupied by DeerFlow"
|
||||
else
|
||||
echo "✗ Port 2026 must be free before starting DeerFlow"
|
||||
exit 1
|
||||
fi
|
||||
else
|
||||
echo "✓ Port 2026 is available"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
echo "=========================================="
|
||||
echo " Docker Environment Check Complete"
|
||||
echo "=========================================="
|
||||
@@ -1,93 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
set -e
|
||||
|
||||
echo "=========================================="
|
||||
echo " Checking Local Development Environment"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
|
||||
all_passed=true
|
||||
|
||||
# Check Node.js
|
||||
echo "1. Checking Node.js..."
|
||||
if command -v node >/dev/null 2>&1; then
|
||||
NODE_VERSION=$(node --version | sed 's/v//')
|
||||
NODE_MAJOR=$(echo "$NODE_VERSION" | cut -d. -f1)
|
||||
if [ "$NODE_MAJOR" -ge 22 ]; then
|
||||
echo "✓ Node.js is installed (version: $NODE_VERSION)"
|
||||
else
|
||||
echo "✗ Node.js version is too old (current: $NODE_VERSION, required: 22+)"
|
||||
all_passed=false
|
||||
fi
|
||||
else
|
||||
echo "✗ Node.js is not installed"
|
||||
all_passed=false
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check pnpm
|
||||
echo "2. Checking pnpm..."
|
||||
if command -v pnpm >/dev/null 2>&1; then
|
||||
echo "✓ pnpm is installed (version: $(pnpm --version))"
|
||||
else
|
||||
echo "✗ pnpm is not installed"
|
||||
echo " Install command: npm install -g pnpm"
|
||||
all_passed=false
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check uv
|
||||
echo "3. Checking uv..."
|
||||
if command -v uv >/dev/null 2>&1; then
|
||||
echo "✓ uv is installed (version: $(uv --version))"
|
||||
else
|
||||
echo "✗ uv is not installed"
|
||||
all_passed=false
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check nginx
|
||||
echo "4. Checking nginx..."
|
||||
if command -v nginx >/dev/null 2>&1; then
|
||||
echo "✓ nginx is installed (version: $(nginx -v 2>&1))"
|
||||
else
|
||||
echo "✗ nginx is not installed"
|
||||
echo " macOS: brew install nginx"
|
||||
echo " Linux: install it with the system package manager"
|
||||
all_passed=false
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check ports
|
||||
echo "5. Checking ports..."
|
||||
if ! command -v lsof >/dev/null 2>&1; then
|
||||
echo "✗ lsof is not installed, so port availability cannot be verified"
|
||||
echo " Install lsof and rerun this check"
|
||||
all_passed=false
|
||||
else
|
||||
for port in 2026 3000 8001 2024; do
|
||||
if lsof -i :$port >/dev/null 2>&1; then
|
||||
echo "⚠ Port $port is already in use:"
|
||||
lsof -i :$port | head -2
|
||||
all_passed=false
|
||||
else
|
||||
echo "✓ Port $port is available"
|
||||
fi
|
||||
done
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Summary
|
||||
echo "=========================================="
|
||||
echo " Environment Check Summary"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
if [ "$all_passed" = true ]; then
|
||||
echo "✅ All environment checks passed!"
|
||||
echo ""
|
||||
echo "Next step: run make install to install dependencies"
|
||||
exit 0
|
||||
else
|
||||
echo "❌ Some checks failed. Please fix the issues above first"
|
||||
exit 1
|
||||
fi
|
||||
@@ -1,65 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
set -e
|
||||
|
||||
echo "=========================================="
|
||||
echo " Docker Deployment"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
|
||||
# Check config.yaml
|
||||
if [ ! -f "config.yaml" ]; then
|
||||
echo "config.yaml does not exist. Generating it..."
|
||||
make config
|
||||
echo ""
|
||||
echo "⚠ Please edit config.yaml to configure your models and API keys"
|
||||
echo " Then run this script again"
|
||||
exit 1
|
||||
else
|
||||
echo "✓ config.yaml exists"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check the .env file
|
||||
if [ ! -f ".env" ]; then
|
||||
echo ".env does not exist. Copying it from the example..."
|
||||
if [ -f ".env.example" ]; then
|
||||
cp .env.example .env
|
||||
echo "✓ Created the .env file"
|
||||
else
|
||||
echo "⚠ .env.example does not exist. Please create the .env file manually"
|
||||
fi
|
||||
else
|
||||
echo "✓ .env file exists"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check the frontend .env file
|
||||
if [ ! -f "frontend/.env" ]; then
|
||||
echo "frontend/.env does not exist. Copying it from the example..."
|
||||
if [ -f "frontend/.env.example" ]; then
|
||||
cp frontend/.env.example frontend/.env
|
||||
echo "✓ Created the frontend/.env file"
|
||||
else
|
||||
echo "⚠ frontend/.env.example does not exist. Please create frontend/.env manually"
|
||||
fi
|
||||
else
|
||||
echo "✓ frontend/.env file exists"
|
||||
fi
|
||||
echo ""
|
||||
# Initialize the Docker environment
|
||||
echo "Initializing the Docker environment..."
|
||||
make docker-init
|
||||
echo ""
|
||||
|
||||
# Start Docker services
|
||||
echo "Starting Docker services..."
|
||||
make docker-start
|
||||
echo ""
|
||||
|
||||
echo "=========================================="
|
||||
echo " Deployment Complete"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
echo "🌐 Access URL: http://localhost:2026"
|
||||
echo "📋 View logs: make docker-logs"
|
||||
echo "🛑 Stop services: make docker-stop"
|
||||
@@ -1,63 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
set -e
|
||||
|
||||
echo "=========================================="
|
||||
echo " Local Mode Deployment"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
|
||||
# Check config.yaml
|
||||
if [ ! -f "config.yaml" ]; then
|
||||
echo "config.yaml does not exist. Generating it..."
|
||||
make config
|
||||
echo ""
|
||||
echo "⚠ Please edit config.yaml to configure your models and API keys"
|
||||
echo " Then run this script again"
|
||||
exit 1
|
||||
else
|
||||
echo "✓ config.yaml exists"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check the .env file
|
||||
if [ ! -f ".env" ]; then
|
||||
echo ".env does not exist. Copying it from the example..."
|
||||
if [ -f ".env.example" ]; then
|
||||
cp .env.example .env
|
||||
echo "✓ Created the .env file"
|
||||
else
|
||||
echo "⚠ .env.example does not exist. Please create the .env file manually"
|
||||
fi
|
||||
else
|
||||
echo "✓ .env file exists"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Check dependencies
|
||||
echo "Checking dependencies..."
|
||||
make check
|
||||
echo ""
|
||||
|
||||
# Install dependencies
|
||||
echo "Installing dependencies..."
|
||||
make install
|
||||
echo ""
|
||||
|
||||
# Start services
|
||||
echo "Starting services (background mode)..."
|
||||
make dev-daemon
|
||||
echo ""
|
||||
|
||||
echo "=========================================="
|
||||
echo " Deployment Complete"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
echo "🌐 Access URL: http://localhost:2026"
|
||||
echo "📋 View logs:"
|
||||
echo " - logs/langgraph.log"
|
||||
echo " - logs/gateway.log"
|
||||
echo " - logs/frontend.log"
|
||||
echo " - logs/nginx.log"
|
||||
echo "🛑 Stop services: make stop"
|
||||
echo ""
|
||||
echo "Please wait 90-120 seconds for all services to start completely, then run the health check"
|
||||
@@ -1,70 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
set +e
|
||||
|
||||
echo "=========================================="
|
||||
echo " Frontend Page Smoke Check"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
|
||||
BASE_URL="${BASE_URL:-http://localhost:2026}"
|
||||
DOC_PATH="${DOC_PATH:-/en/docs}"
|
||||
|
||||
all_passed=true
|
||||
|
||||
check_status() {
|
||||
local name="$1"
|
||||
local url="$2"
|
||||
local expected_re="$3"
|
||||
|
||||
local status
|
||||
status="$(curl -s -o /dev/null -w "%{http_code}" -L "$url")"
|
||||
if echo "$status" | grep -Eq "$expected_re"; then
|
||||
echo "✓ $name ($url) -> $status"
|
||||
else
|
||||
echo "✗ $name ($url) -> $status (expected: $expected_re)"
|
||||
all_passed=false
|
||||
fi
|
||||
}
|
||||
|
||||
check_final_url() {
|
||||
local name="$1"
|
||||
local url="$2"
|
||||
local expected_path_re="$3"
|
||||
|
||||
local effective
|
||||
effective="$(curl -s -o /dev/null -w "%{url_effective}" -L "$url")"
|
||||
if echo "$effective" | grep -Eq "$expected_path_re"; then
|
||||
echo "✓ $name redirect target -> $effective"
|
||||
else
|
||||
echo "✗ $name redirect target -> $effective (expected path: $expected_path_re)"
|
||||
all_passed=false
|
||||
fi
|
||||
}
|
||||
|
||||
echo "1. Checking entry pages..."
|
||||
check_status "Landing page" "${BASE_URL}/" "200"
|
||||
check_status "Workspace redirect" "${BASE_URL}/workspace" "200|301|302|307|308"
|
||||
check_final_url "Workspace redirect" "${BASE_URL}/workspace" "/workspace/chats/"
|
||||
echo ""
|
||||
|
||||
echo "2. Checking key workspace routes..."
|
||||
check_status "New chat page" "${BASE_URL}/workspace/chats/new" "200"
|
||||
check_status "Chats list page" "${BASE_URL}/workspace/chats" "200"
|
||||
check_status "Agents gallery page" "${BASE_URL}/workspace/agents" "200"
|
||||
echo ""
|
||||
|
||||
echo "3. Checking docs route (optional)..."
|
||||
check_status "Docs page" "${BASE_URL}${DOC_PATH}" "200|404"
|
||||
echo ""
|
||||
|
||||
echo "=========================================="
|
||||
echo " Frontend Smoke Check Summary"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
if [ "$all_passed" = true ]; then
|
||||
echo "✅ Frontend smoke checks passed!"
|
||||
exit 0
|
||||
else
|
||||
echo "❌ Frontend smoke checks failed"
|
||||
exit 1
|
||||
fi
|
||||
@@ -1,125 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
set +e
|
||||
|
||||
echo "=========================================="
|
||||
echo " Service Health Check"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
|
||||
all_passed=true
|
||||
mode="${SMOKE_TEST_MODE:-auto}"
|
||||
summary_hint="make logs"
|
||||
|
||||
print_step() {
|
||||
echo "$1"
|
||||
}
|
||||
|
||||
check_http_status() {
|
||||
local name="$1"
|
||||
local url="$2"
|
||||
local expected_re="$3"
|
||||
local status
|
||||
|
||||
status="$(curl -s -o /dev/null -w "%{http_code}" "$url" 2>/dev/null)"
|
||||
if echo "$status" | grep -Eq "$expected_re"; then
|
||||
echo "✓ $name is accessible ($url -> $status)"
|
||||
else
|
||||
echo "✗ $name is not accessible ($url -> ${status:-000})"
|
||||
all_passed=false
|
||||
fi
|
||||
}
|
||||
|
||||
check_listen_port() {
|
||||
local name="$1"
|
||||
local port="$2"
|
||||
|
||||
if lsof -nP -iTCP:"$port" -sTCP:LISTEN >/dev/null 2>&1; then
|
||||
echo "✓ $name is listening on port $port"
|
||||
else
|
||||
echo "✗ $name is not listening on port $port"
|
||||
all_passed=false
|
||||
fi
|
||||
}
|
||||
|
||||
docker_available() {
|
||||
command -v docker >/dev/null 2>&1 && docker info >/dev/null 2>&1
|
||||
}
|
||||
|
||||
detect_mode() {
|
||||
case "$mode" in
|
||||
local|docker)
|
||||
echo "$mode"
|
||||
return
|
||||
;;
|
||||
esac
|
||||
|
||||
if docker_available && docker ps --format "{{.Names}}" | grep -q "deer-flow"; then
|
||||
echo "docker"
|
||||
else
|
||||
echo "local"
|
||||
fi
|
||||
}
|
||||
|
||||
mode="$(detect_mode)"
|
||||
|
||||
echo "Deployment mode: $mode"
|
||||
echo ""
|
||||
|
||||
if [ "$mode" = "docker" ]; then
|
||||
summary_hint="make docker-logs"
|
||||
print_step "1. Checking container status..."
|
||||
if docker ps --format "{{.Names}}" | grep -q "deer-flow"; then
|
||||
echo "✓ Containers are running:"
|
||||
docker ps --format " - {{.Names}} ({{.Status}})"
|
||||
else
|
||||
echo "✗ No DeerFlow-related containers are running"
|
||||
all_passed=false
|
||||
fi
|
||||
else
|
||||
summary_hint="logs/{langgraph,gateway,frontend,nginx}.log"
|
||||
print_step "1. Checking local service ports..."
|
||||
check_listen_port "Nginx" 2026
|
||||
check_listen_port "Frontend" 3000
|
||||
check_listen_port "Gateway" 8001
|
||||
check_listen_port "LangGraph" 2024
|
||||
fi
|
||||
echo ""
|
||||
|
||||
echo "2. Waiting for services to fully start (30 seconds)..."
|
||||
sleep 30
|
||||
echo ""
|
||||
|
||||
echo "3. Checking frontend service..."
|
||||
check_http_status "Frontend service" "http://localhost:2026" "200|301|302|307|308"
|
||||
echo ""
|
||||
|
||||
echo "4. Checking API Gateway..."
|
||||
health_response=$(curl -s http://localhost:2026/health 2>/dev/null)
|
||||
if [ $? -eq 0 ] && [ -n "$health_response" ]; then
|
||||
echo "✓ API Gateway health check passed"
|
||||
echo " Response: $health_response"
|
||||
else
|
||||
echo "✗ API Gateway health check failed"
|
||||
all_passed=false
|
||||
fi
|
||||
echo ""
|
||||
|
||||
echo "5. Checking LangGraph service..."
|
||||
check_http_status "LangGraph service" "http://localhost:2024/" "200|301|302|307|308|404"
|
||||
echo ""
|
||||
|
||||
echo "=========================================="
|
||||
echo " Health Check Summary"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
if [ "$all_passed" = true ]; then
|
||||
echo "✅ All checks passed!"
|
||||
echo ""
|
||||
echo "🌐 Application URL: http://localhost:2026"
|
||||
exit 0
|
||||
else
|
||||
echo "❌ Some checks failed"
|
||||
echo ""
|
||||
echo "Please review: $summary_hint"
|
||||
exit 1
|
||||
fi
|
||||
@@ -1,49 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
set -e
|
||||
|
||||
echo "=========================================="
|
||||
echo " Pulling the Latest Code"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
|
||||
# Check whether the current directory is a Git repository
|
||||
if [ ! -d ".git" ]; then
|
||||
echo "✗ The current directory is not a Git repository"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check Git status
|
||||
echo "Checking Git status..."
|
||||
if git status --porcelain | grep -q .; then
|
||||
echo "⚠ Uncommitted changes detected:"
|
||||
git status --short
|
||||
echo ""
|
||||
echo "Please commit or stash your changes before continuing"
|
||||
echo "Options:"
|
||||
echo " 1. git add . && git commit -m 'Save changes'"
|
||||
echo " 2. git stash (stash changes and restore them later)"
|
||||
echo " 3. git reset --hard HEAD (discard local changes - use with caution)"
|
||||
exit 1
|
||||
else
|
||||
echo "✓ Working tree is clean"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# Fetch remote updates
|
||||
echo "Fetching remote updates..."
|
||||
git fetch origin main
|
||||
echo ""
|
||||
|
||||
# Pull the latest code
|
||||
echo "Pulling the latest code..."
|
||||
git pull origin main
|
||||
echo ""
|
||||
|
||||
# Show the latest commit
|
||||
echo "Latest commit:"
|
||||
git log -1 --oneline
|
||||
echo ""
|
||||
|
||||
echo "=========================================="
|
||||
echo " Code Update Complete"
|
||||
echo "=========================================="
|
||||
@@ -1,180 +0,0 @@
|
||||
# DeerFlow Smoke Test Report
|
||||
|
||||
**Test Date**: {{test_date}}
|
||||
**Test Environment**: {{test_environment}}
|
||||
**Deployment Mode**: Docker
|
||||
**Test Version**: {{git_commit}}
|
||||
|
||||
---
|
||||
|
||||
## Execution Summary
|
||||
|
||||
| Metric | Status |
|
||||
|------|------|
|
||||
| Total Test Phases | 6 |
|
||||
| Passed Phases | {{passed_stages}} |
|
||||
| Failed Phases | {{failed_stages}} |
|
||||
| Overall Conclusion | **{{overall_status}}** |
|
||||
|
||||
### Key Test Cases
|
||||
|
||||
| Case | Result | Details |
|
||||
|------|--------|---------|
|
||||
| Code update check | {{case_code_update}} | {{case_code_update_details}} |
|
||||
| Environment check | {{case_env_check}} | {{case_env_check_details}} |
|
||||
| Configuration preparation | {{case_config_prep}} | {{case_config_prep_details}} |
|
||||
| Deployment | {{case_deploy}} | {{case_deploy_details}} |
|
||||
| Health check | {{case_health_check}} | {{case_health_check_details}} |
|
||||
| Frontend routes | {{case_frontend_routes_overall}} | {{case_frontend_routes_details}} |
|
||||
|
||||
---
|
||||
|
||||
## Detailed Test Results
|
||||
|
||||
### Phase 1: Code Update Check
|
||||
|
||||
- [x] Confirm current directory - {{status_dir_check}}
|
||||
- [x] Check Git status - {{status_git_status}}
|
||||
- [x] Pull latest code - {{status_git_pull}}
|
||||
- [x] Confirm code update - {{status_git_verify}}
|
||||
|
||||
**Phase Status**: {{stage1_status}}
|
||||
|
||||
---
|
||||
|
||||
### Phase 2: Docker Environment Check
|
||||
|
||||
- [x] Docker version - {{status_docker_version}}
|
||||
- [x] Docker daemon - {{status_docker_daemon}}
|
||||
- [x] Docker Compose - {{status_docker_compose}}
|
||||
- [x] Port check - {{status_port_check}}
|
||||
|
||||
**Phase Status**: {{stage2_status}}
|
||||
|
||||
---
|
||||
|
||||
### Phase 3: Configuration Preparation
|
||||
|
||||
- [x] config.yaml - {{status_config_yaml}}
|
||||
- [x] .env file - {{status_env_file}}
|
||||
- [x] Model configuration - {{status_model_config}}
|
||||
|
||||
**Phase Status**: {{stage3_status}}
|
||||
|
||||
---
|
||||
|
||||
### Phase 4: Docker Deployment
|
||||
|
||||
- [x] docker-init - {{status_docker_init}}
|
||||
- [x] docker-start - {{status_docker_start}}
|
||||
- [x] Service startup wait - {{status_wait_startup}}
|
||||
|
||||
**Phase Status**: {{stage4_status}}
|
||||
|
||||
---
|
||||
|
||||
### Phase 5: Service Health Check
|
||||
|
||||
- [x] Container status - {{status_containers}}
|
||||
- [x] Frontend service - {{status_frontend}}
|
||||
- [x] API Gateway - {{status_api_gateway}}
|
||||
- [x] LangGraph service - {{status_langgraph}}
|
||||
|
||||
**Phase Status**: {{stage5_status}}
|
||||
|
||||
---
|
||||
|
||||
### Frontend Routes Smoke Results
|
||||
|
||||
| Route | Status | Details |
|
||||
|-------|--------|---------|
|
||||
| Landing `/` | {{landing_status}} | {{landing_details}} |
|
||||
| Workspace redirect `/workspace` | {{workspace_redirect_status}} | target {{workspace_redirect_target}} |
|
||||
| New chat `/workspace/chats/new` | {{new_chat_status}} | {{new_chat_details}} |
|
||||
| Chats list `/workspace/chats` | {{chats_list_status}} | {{chats_list_details}} |
|
||||
| Agents gallery `/workspace/agents` | {{agents_gallery_status}} | {{agents_gallery_details}} |
|
||||
| Docs `{{docs_path}}` | {{docs_status}} | {{docs_details}} |
|
||||
|
||||
**Summary**: {{frontend_routes_summary}}
|
||||
|
||||
---
|
||||
|
||||
### Phase 6: Test Report Generation
|
||||
|
||||
- [x] Result summary - {{status_summary}}
|
||||
- [x] Issue log - {{status_issues}}
|
||||
- [x] Report generation - {{status_report}}
|
||||
|
||||
**Phase Status**: {{stage6_status}}
|
||||
|
||||
---
|
||||
|
||||
## Issue Log
|
||||
|
||||
### Issue 1
|
||||
**Description**: {{issue1_description}}
|
||||
**Severity**: {{issue1_severity}}
|
||||
**Solution**: {{issue1_solution}}
|
||||
|
||||
---
|
||||
|
||||
## Environment Information
|
||||
|
||||
### Docker Version
|
||||
```text
|
||||
{{docker_version_output}}
|
||||
```
|
||||
|
||||
### Git Information
|
||||
```text
|
||||
Repository: {{git_repo}}
|
||||
Branch: {{git_branch}}
|
||||
Commit: {{git_commit}}
|
||||
Commit Message: {{git_commit_message}}
|
||||
```
|
||||
|
||||
### Configuration Summary
|
||||
- config.yaml exists: {{config_exists}}
|
||||
- .env file exists: {{env_exists}}
|
||||
- Number of configured models: {{model_count}}
|
||||
|
||||
---
|
||||
|
||||
## Container Status
|
||||
|
||||
| Container Name | Status | Uptime |
|
||||
|----------|------|----------|
|
||||
| deer-flow-nginx | {{nginx_status}} | {{nginx_uptime}} |
|
||||
| deer-flow-frontend | {{frontend_status}} | {{frontend_uptime}} |
|
||||
| deer-flow-gateway | {{gateway_status}} | {{gateway_uptime}} |
|
||||
| deer-flow-langgraph | {{langgraph_status}} | {{langgraph_uptime}} |
|
||||
|
||||
---
|
||||
|
||||
## Recommendations and Next Steps
|
||||
|
||||
### If the Test Passes
|
||||
1. [ ] Visit http://localhost:2026 to start using DeerFlow
|
||||
2. [ ] Configure your preferred model if it is not configured yet
|
||||
3. [ ] Explore available skills
|
||||
4. [ ] Refer to the documentation to learn more features
|
||||
|
||||
### If the Test Fails
|
||||
1. [ ] Review references/troubleshooting.md for common solutions
|
||||
2. [ ] Check Docker logs: `make docker-logs`
|
||||
3. [ ] Verify configuration file format and content
|
||||
4. [ ] If needed, fully reset the environment: `make clean && make config && make docker-init && make docker-start`
|
||||
|
||||
---
|
||||
|
||||
## Appendix
|
||||
|
||||
### Full Logs
|
||||
{{full_logs}}
|
||||
|
||||
### Tester
|
||||
{{tester_name}}
|
||||
|
||||
---
|
||||
|
||||
*Report generated at: {{report_time}}*
|
||||
@@ -1,185 +0,0 @@
|
||||
# DeerFlow Smoke Test Report
|
||||
|
||||
**Test Date**: {{test_date}}
|
||||
**Test Environment**: {{test_environment}}
|
||||
**Deployment Mode**: Local
|
||||
**Test Version**: {{git_commit}}
|
||||
|
||||
---
|
||||
|
||||
## Execution Summary
|
||||
|
||||
| Metric | Status |
|
||||
|------|------|
|
||||
| Total Test Phases | 6 |
|
||||
| Passed Phases | {{passed_stages}} |
|
||||
| Failed Phases | {{failed_stages}} |
|
||||
| Overall Conclusion | **{{overall_status}}** |
|
||||
|
||||
### Key Test Cases
|
||||
|
||||
| Case | Result | Details |
|
||||
|------|--------|---------|
|
||||
| Code update check | {{case_code_update}} | {{case_code_update_details}} |
|
||||
| Environment check | {{case_env_check}} | {{case_env_check_details}} |
|
||||
| Configuration preparation | {{case_config_prep}} | {{case_config_prep_details}} |
|
||||
| Deployment | {{case_deploy}} | {{case_deploy_details}} |
|
||||
| Health check | {{case_health_check}} | {{case_health_check_details}} |
|
||||
| Frontend routes | {{case_frontend_routes_overall}} | {{case_frontend_routes_details}} |
|
||||
|
||||
---
|
||||
|
||||
## Detailed Test Results
|
||||
|
||||
### Phase 1: Code Update Check
|
||||
|
||||
- [x] Confirm current directory - {{status_dir_check}}
|
||||
- [x] Check Git status - {{status_git_status}}
|
||||
- [x] Pull latest code - {{status_git_pull}}
|
||||
- [x] Confirm code update - {{status_git_verify}}
|
||||
|
||||
**Phase Status**: {{stage1_status}}
|
||||
|
||||
---
|
||||
|
||||
### Phase 2: Local Environment Check
|
||||
|
||||
- [x] Node.js version - {{status_node_version}}
|
||||
- [x] pnpm - {{status_pnpm}}
|
||||
- [x] uv - {{status_uv}}
|
||||
- [x] nginx - {{status_nginx}}
|
||||
- [x] Port check - {{status_port_check}}
|
||||
|
||||
**Phase Status**: {{stage2_status}}
|
||||
|
||||
---
|
||||
|
||||
### Phase 3: Configuration Preparation
|
||||
|
||||
- [x] config.yaml - {{status_config_yaml}}
|
||||
- [x] .env file - {{status_env_file}}
|
||||
- [x] Model configuration - {{status_model_config}}
|
||||
|
||||
**Phase Status**: {{stage3_status}}
|
||||
|
||||
---
|
||||
|
||||
### Phase 4: Local Deployment
|
||||
|
||||
- [x] make check - {{status_make_check}}
|
||||
- [x] make install - {{status_make_install}}
|
||||
- [x] make dev-daemon / make dev - {{status_local_start}}
|
||||
- [x] Service startup wait - {{status_wait_startup}}
|
||||
|
||||
**Phase Status**: {{stage4_status}}
|
||||
|
||||
---
|
||||
|
||||
### Phase 5: Service Health Check
|
||||
|
||||
- [x] Process status - {{status_processes}}
|
||||
- [x] Frontend service - {{status_frontend}}
|
||||
- [x] API Gateway - {{status_api_gateway}}
|
||||
- [x] LangGraph service - {{status_langgraph}}
|
||||
|
||||
**Phase Status**: {{stage5_status}}
|
||||
|
||||
---
|
||||
|
||||
### Frontend Routes Smoke Results
|
||||
|
||||
| Route | Status | Details |
|
||||
|-------|--------|---------|
|
||||
| Landing `/` | {{landing_status}} | {{landing_details}} |
|
||||
| Workspace redirect `/workspace` | {{workspace_redirect_status}} | target {{workspace_redirect_target}} |
|
||||
| New chat `/workspace/chats/new` | {{new_chat_status}} | {{new_chat_details}} |
|
||||
| Chats list `/workspace/chats` | {{chats_list_status}} | {{chats_list_details}} |
|
||||
| Agents gallery `/workspace/agents` | {{agents_gallery_status}} | {{agents_gallery_details}} |
|
||||
| Docs `{{docs_path}}` | {{docs_status}} | {{docs_details}} |
|
||||
|
||||
**Summary**: {{frontend_routes_summary}}
|
||||
|
||||
---
|
||||
|
||||
### Phase 6: Test Report Generation
|
||||
|
||||
- [x] Result summary - {{status_summary}}
|
||||
- [x] Issue log - {{status_issues}}
|
||||
- [x] Report generation - {{status_report}}
|
||||
|
||||
**Phase Status**: {{stage6_status}}
|
||||
|
||||
---
|
||||
|
||||
## Issue Log
|
||||
|
||||
### Issue 1
|
||||
**Description**: {{issue1_description}}
|
||||
**Severity**: {{issue1_severity}}
|
||||
**Solution**: {{issue1_solution}}
|
||||
|
||||
---
|
||||
|
||||
## Environment Information
|
||||
|
||||
### Local Dependency Versions
|
||||
```text
|
||||
Node.js: {{node_version_output}}
|
||||
pnpm: {{pnpm_version_output}}
|
||||
uv: {{uv_version_output}}
|
||||
nginx: {{nginx_version_output}}
|
||||
```
|
||||
|
||||
### Git Information
|
||||
```text
|
||||
Repository: {{git_repo}}
|
||||
Branch: {{git_branch}}
|
||||
Commit: {{git_commit}}
|
||||
Commit Message: {{git_commit_message}}
|
||||
```
|
||||
|
||||
### Configuration Summary
|
||||
- config.yaml exists: {{config_exists}}
|
||||
- .env file exists: {{env_exists}}
|
||||
- Number of configured models: {{model_count}}
|
||||
|
||||
---
|
||||
|
||||
## Local Service Status
|
||||
|
||||
| Service | Status | Endpoint |
|
||||
|---------|--------|----------|
|
||||
| Nginx | {{nginx_status}} | {{nginx_endpoint}} |
|
||||
| Frontend | {{frontend_status}} | {{frontend_endpoint}} |
|
||||
| Gateway | {{gateway_status}} | {{gateway_endpoint}} |
|
||||
| LangGraph | {{langgraph_status}} | {{langgraph_endpoint}} |
|
||||
|
||||
---
|
||||
|
||||
## Recommendations and Next Steps
|
||||
|
||||
### If the Test Passes
|
||||
1. [ ] Visit http://localhost:2026 to start using DeerFlow
|
||||
2. [ ] Configure your preferred model if it is not configured yet
|
||||
3. [ ] Explore available skills
|
||||
4. [ ] Refer to the documentation to learn more features
|
||||
|
||||
### If the Test Fails
|
||||
1. [ ] Review references/troubleshooting.md for common solutions
|
||||
2. [ ] Check local logs: `logs/{langgraph,gateway,frontend,nginx}.log`
|
||||
3. [ ] Verify configuration file format and content
|
||||
4. [ ] If needed, fully reset the environment: `make stop && make clean && make install && make dev-daemon`
|
||||
|
||||
---
|
||||
|
||||
## Appendix
|
||||
|
||||
### Full Logs
|
||||
{{full_logs}}
|
||||
|
||||
### Tester
|
||||
{{tester_name}}
|
||||
|
||||
---
|
||||
|
||||
*Report generated at: {{report_time}}*
|
||||
@@ -3,7 +3,6 @@ Dockerfile
|
||||
.dockerignore
|
||||
.git
|
||||
.gitignore
|
||||
docker/
|
||||
|
||||
# Python
|
||||
__pycache__/
|
||||
@@ -27,7 +26,6 @@ wheels/
|
||||
*.egg-info/
|
||||
.installed.cfg
|
||||
*.egg
|
||||
.venv/
|
||||
|
||||
# Web
|
||||
node_modules
|
||||
@@ -52,20 +50,3 @@ examples/
|
||||
assets/
|
||||
tests/
|
||||
*.log
|
||||
|
||||
# Exclude directories not needed in Docker context
|
||||
# Frontend build only needs frontend/
|
||||
# Backend build only needs backend/
|
||||
scripts/
|
||||
logs/
|
||||
docker/
|
||||
skills/
|
||||
frontend/.next
|
||||
frontend/node_modules
|
||||
backend/.venv
|
||||
backend/htmlcov
|
||||
backend/.coverage
|
||||
*.md
|
||||
!README.md
|
||||
!frontend/README.md
|
||||
!backend/README.md
|
||||
|
||||
+21
-56
@@ -1,62 +1,27 @@
|
||||
# Serper API Key (Google Search) - https://serper.dev
|
||||
SERPER_API_KEY=your-serper-api-key
|
||||
# Application Settings
|
||||
DEBUG=True
|
||||
APP_ENV=development
|
||||
|
||||
# TAVILY API Key
|
||||
TAVILY_API_KEY=your-tavily-api-key
|
||||
# docker build args
|
||||
NEXT_PUBLIC_API_URL="http://localhost:8000/api"
|
||||
|
||||
# Jina API Key
|
||||
JINA_API_KEY=your-jina-api-key
|
||||
# Search Engine, Supported values: tavily (recommended), duckduckgo, brave_search, arxiv
|
||||
SEARCH_API=tavily
|
||||
TAVILY_API_KEY=tvly-xxx
|
||||
# BRAVE_SEARCH_API_KEY=xxx # Required only if SEARCH_API is brave_search
|
||||
# JINA_API_KEY=jina_xxx # Optional, default is None
|
||||
|
||||
# InfoQuest API Key
|
||||
INFOQUEST_API_KEY=your-infoquest-api-key
|
||||
# Browser CORS allowlist for split-origin or port-forwarded deployments (comma-separated exact origins).
|
||||
# Leave unset when using the unified nginx endpoint, e.g. http://localhost:2026.
|
||||
# GATEWAY_CORS_ORIGINS=http://localhost:3000,http://127.0.0.1:3000
|
||||
# Optional, volcengine TTS for generating podcast
|
||||
VOLCENGINE_TTS_APPID=xxx
|
||||
VOLCENGINE_TTS_ACCESS_TOKEN=xxx
|
||||
# VOLCENGINE_TTS_CLUSTER=volcano_tts # Optional, default is volcano_tts
|
||||
# VOLCENGINE_TTS_VOICE_TYPE=BV700_V2_streaming # Optional, default is BV700_V2_streaming
|
||||
|
||||
# Optional:
|
||||
# FIRECRAWL_API_KEY=your-firecrawl-api-key
|
||||
# VOLCENGINE_API_KEY=your-volcengine-api-key
|
||||
# OPENAI_API_KEY=your-openai-api-key
|
||||
# GEMINI_API_KEY=your-gemini-api-key
|
||||
# DEEPSEEK_API_KEY=your-deepseek-api-key
|
||||
# NOVITA_API_KEY=your-novita-api-key # OpenAI-compatible, see https://novita.ai
|
||||
# MINIMAX_API_KEY=your-minimax-api-key # OpenAI-compatible, see https://platform.minimax.io
|
||||
# VLLM_API_KEY=your-vllm-api-key # OpenAI-compatible
|
||||
# FEISHU_APP_ID=your-feishu-app-id
|
||||
# FEISHU_APP_SECRET=your-feishu-app-secret
|
||||
|
||||
# SLACK_BOT_TOKEN=your-slack-bot-token
|
||||
# SLACK_APP_TOKEN=your-slack-app-token
|
||||
# TELEGRAM_BOT_TOKEN=your-telegram-bot-token
|
||||
# DISCORD_BOT_TOKEN=your-discord-bot-token
|
||||
|
||||
# Enable LangSmith to monitor and debug your LLM calls, agent runs, and tool executions.
|
||||
# Option, for langsmith tracing and monitoring
|
||||
# LANGSMITH_TRACING=true
|
||||
# LANGSMITH_ENDPOINT=https://api.smith.langchain.com
|
||||
# LANGSMITH_API_KEY=your-langsmith-api-key
|
||||
# LANGSMITH_PROJECT=your-langsmith-project
|
||||
# LANGSMITH_ENDPOINT="https://api.smith.langchain.com"
|
||||
# LANGSMITH_API_KEY="xxx"
|
||||
# LANGSMITH_PROJECT="xxx"
|
||||
|
||||
# GitHub API Token
|
||||
# GITHUB_TOKEN=your-github-token
|
||||
|
||||
# Database (only needed when config.yaml has database.backend: postgres)
|
||||
# DATABASE_URL=postgresql://deerflow:password@localhost:5432/deerflow
|
||||
#
|
||||
# WECOM_BOT_ID=your-wecom-bot-id
|
||||
# WECOM_BOT_SECRET=your-wecom-bot-secret
|
||||
# DINGTALK_CLIENT_ID=your-dingtalk-client-id
|
||||
# DINGTALK_CLIENT_SECRET=your-dingtalk-client-secret
|
||||
|
||||
# Set to "false" to disable Swagger UI, ReDoc, and OpenAPI schema in production
|
||||
# GATEWAY_ENABLE_DOCS=false
|
||||
|
||||
# ── Frontend SSR → Gateway wiring ─────────────────────────────────────────────
|
||||
# The Next.js server uses these to reach the Gateway during SSR (auth checks,
|
||||
# /api/* rewrites). They default to localhost values that match `make dev` and
|
||||
# `make start`, so most local users do not need to set them.
|
||||
#
|
||||
# Override only when the Gateway is not on localhost:8001 (e.g. when the
|
||||
# frontend and gateway run on different hosts, in containers with a service
|
||||
# alias, or behind a different port). docker-compose already sets these.
|
||||
# DEER_FLOW_INTERNAL_GATEWAY_BASE_URL=http://localhost:8001
|
||||
# DEER_FLOW_TRUSTED_ORIGINS=http://localhost:3000,http://localhost:2026
|
||||
# [!NOTE]
|
||||
# For model settings and other configurations, please refer to `docs/configuration_guide.md`
|
||||
|
||||
@@ -1,43 +0,0 @@
|
||||
# Normalize line endings to LF for all text files
|
||||
* text=auto eol=lf
|
||||
|
||||
# Shell scripts and makefiles must always use LF
|
||||
*.sh text eol=lf
|
||||
Makefile text eol=lf
|
||||
**/Makefile text eol=lf
|
||||
|
||||
# Common config/source files
|
||||
*.yml text eol=lf
|
||||
*.yaml text eol=lf
|
||||
*.toml text eol=lf
|
||||
*.json text eol=lf
|
||||
*.md text eol=lf
|
||||
*.py text eol=lf
|
||||
*.ts text eol=lf
|
||||
*.tsx text eol=lf
|
||||
*.js text eol=lf
|
||||
*.jsx text eol=lf
|
||||
*.css text eol=lf
|
||||
*.scss text eol=lf
|
||||
*.html text eol=lf
|
||||
*.env text eol=lf
|
||||
|
||||
# Windows scripts
|
||||
*.bat text eol=crlf
|
||||
*.cmd text eol=crlf
|
||||
|
||||
# Binary assets
|
||||
*.png binary
|
||||
*.jpg binary
|
||||
*.jpeg binary
|
||||
*.gif binary
|
||||
*.webp binary
|
||||
*.ico binary
|
||||
*.pdf binary
|
||||
*.zip binary
|
||||
*.tar binary
|
||||
*.gz binary
|
||||
*.mp4 binary
|
||||
*.mov binary
|
||||
*.woff binary
|
||||
*.woff2 binary
|
||||
@@ -1,128 +0,0 @@
|
||||
name: Runtime Information
|
||||
description: Report runtime/environment details to help reproduce an issue.
|
||||
title: "[runtime] "
|
||||
labels:
|
||||
- needs-triage
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
Thanks for sharing runtime details.
|
||||
Complete this form so maintainers can quickly reproduce and diagnose the problem.
|
||||
|
||||
- type: input
|
||||
id: summary
|
||||
attributes:
|
||||
label: Problem summary
|
||||
description: Short summary of the issue.
|
||||
placeholder: e.g. make dev fails to start gateway service
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: expected
|
||||
attributes:
|
||||
label: Expected behavior
|
||||
placeholder: What did you expect to happen?
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: actual
|
||||
attributes:
|
||||
label: Actual behavior
|
||||
placeholder: What happened instead? Include key error lines.
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: dropdown
|
||||
id: os
|
||||
attributes:
|
||||
label: Operating system
|
||||
options:
|
||||
- macOS
|
||||
- Linux
|
||||
- Windows
|
||||
- Other
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: input
|
||||
id: platform_details
|
||||
attributes:
|
||||
label: Platform details
|
||||
description: Add architecture and shell if relevant.
|
||||
placeholder: e.g. arm64, zsh
|
||||
|
||||
- type: input
|
||||
id: python_version
|
||||
attributes:
|
||||
label: Python version
|
||||
placeholder: e.g. Python 3.12.9
|
||||
|
||||
- type: input
|
||||
id: node_version
|
||||
attributes:
|
||||
label: Node.js version
|
||||
placeholder: e.g. v23.11.0
|
||||
|
||||
- type: input
|
||||
id: pnpm_version
|
||||
attributes:
|
||||
label: pnpm version
|
||||
placeholder: e.g. 10.26.2
|
||||
|
||||
- type: input
|
||||
id: uv_version
|
||||
attributes:
|
||||
label: uv version
|
||||
placeholder: e.g. 0.7.20
|
||||
|
||||
- type: dropdown
|
||||
id: run_mode
|
||||
attributes:
|
||||
label: How are you running DeerFlow?
|
||||
options:
|
||||
- Local (make dev)
|
||||
- Docker (make docker-dev)
|
||||
- CI
|
||||
- Other
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: reproduce
|
||||
attributes:
|
||||
label: Reproduction steps
|
||||
description: Provide exact commands and sequence.
|
||||
placeholder: |
|
||||
1. make check
|
||||
2. make install
|
||||
3. make dev
|
||||
4. ...
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: logs
|
||||
attributes:
|
||||
label: Relevant logs
|
||||
description: Paste key lines from logs (for example logs/gateway.log, logs/frontend.log).
|
||||
render: shell
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: git_info
|
||||
attributes:
|
||||
label: Git state
|
||||
description: Share output of git branch and latest commit SHA.
|
||||
placeholder: |
|
||||
branch: feature/my-branch
|
||||
commit: abcdef1
|
||||
|
||||
- type: textarea
|
||||
id: additional
|
||||
attributes:
|
||||
label: Additional context
|
||||
description: Add anything else that might help triage.
|
||||
@@ -1,213 +0,0 @@
|
||||
# Copilot Onboarding Instructions for DeerFlow
|
||||
|
||||
Use this file as the default operating guide for this repository. Follow it first, and only search the codebase when this file is incomplete or incorrect.
|
||||
|
||||
## 1) Repository Summary
|
||||
|
||||
DeerFlow is a full-stack "super agent harness".
|
||||
|
||||
- Backend: Python 3.12, LangGraph + FastAPI gateway, sandbox/tool system, memory, MCP integration.
|
||||
- Frontend: Next.js 16 + React 19 + TypeScript + pnpm.
|
||||
- Local dev entrypoint: root `Makefile` starts backend + frontend + nginx on `http://localhost:2026`.
|
||||
- Docker dev entrypoint: `make docker-*` (mode-aware provisioner startup from `config.yaml`).
|
||||
|
||||
Current repo footprint is medium-large (backend service, frontend app, docker stack, skills library, docs).
|
||||
|
||||
## 2) Runtime and Toolchain Requirements
|
||||
|
||||
Validated in this repo on macOS:
|
||||
|
||||
- Node.js `>=22` (validated with Node `23.11.0`)
|
||||
- pnpm (repo expects lockfile generated by pnpm 10; validated with pnpm `10.26.2` and `10.15.0`)
|
||||
- Python `>=3.12` (CI uses `3.12`)
|
||||
- `uv` (validated with `0.7.20`)
|
||||
- `nginx` (required for `make dev` unified local endpoint)
|
||||
|
||||
Always run from repo root unless a command explicitly says otherwise.
|
||||
|
||||
## 3) Build/Test/Lint/Run - Verified Command Sequences
|
||||
|
||||
These were executed and validated in this repository.
|
||||
|
||||
### A. Bootstrap and install
|
||||
|
||||
1. Check prerequisites:
|
||||
|
||||
```bash
|
||||
make check
|
||||
```
|
||||
|
||||
Observed: passes when required tools are installed.
|
||||
|
||||
2. Install dependencies (recommended order: backend then frontend, as implemented by `make install`):
|
||||
|
||||
```bash
|
||||
make install
|
||||
```
|
||||
|
||||
### B. Backend CI-equivalent validation
|
||||
|
||||
Run from `backend/`:
|
||||
|
||||
```bash
|
||||
make lint
|
||||
make test
|
||||
```
|
||||
|
||||
Validated results:
|
||||
|
||||
- `make lint`: pass (`ruff check .`)
|
||||
- `make test`: pass (`277 passed, 15 warnings in ~76.6s`)
|
||||
|
||||
CI parity:
|
||||
|
||||
- `.github/workflows/backend-unit-tests.yml` runs on pull requests.
|
||||
- CI executes `uv sync --group dev`, then `make lint`, then `make test` in `backend/`.
|
||||
|
||||
### C. Frontend validation
|
||||
|
||||
Run from `frontend/`.
|
||||
|
||||
Recommended reliable sequence:
|
||||
|
||||
```bash
|
||||
pnpm lint
|
||||
pnpm typecheck
|
||||
BETTER_AUTH_SECRET=local-dev-secret pnpm build
|
||||
```
|
||||
|
||||
Observed failure modes and workarounds:
|
||||
|
||||
- `pnpm build` fails without `BETTER_AUTH_SECRET` in production-mode env validation.
|
||||
- Workaround: set `BETTER_AUTH_SECRET` (best) or set `SKIP_ENV_VALIDATION=1`.
|
||||
- Even with `SKIP_ENV_VALIDATION=1`, Better Auth can still warn/error in logs about default secret; prefer setting a real non-default secret.
|
||||
- `pnpm check` currently fails (`next lint` invocation is incompatible here and resolves to an invalid directory). Do not rely on `pnpm check`; run `pnpm lint` and `pnpm typecheck` explicitly.
|
||||
|
||||
### D. Run locally (all services)
|
||||
|
||||
From root:
|
||||
|
||||
```bash
|
||||
make dev
|
||||
```
|
||||
|
||||
Behavior:
|
||||
|
||||
- Stops existing local services first.
|
||||
- Starts LangGraph (`2024`), Gateway (`8001`), Frontend (`3000`), nginx (`2026`).
|
||||
- Unified app endpoint: `http://localhost:2026`.
|
||||
- Logs: `logs/langgraph.log`, `logs/gateway.log`, `logs/frontend.log`, `logs/nginx.log`.
|
||||
|
||||
Stop services:
|
||||
|
||||
```bash
|
||||
make stop
|
||||
```
|
||||
|
||||
If tool sessions/timeouts interrupt `make dev`, run `make stop` again to ensure cleanup.
|
||||
|
||||
### E. Config bootstrap
|
||||
|
||||
From root:
|
||||
|
||||
```bash
|
||||
make config
|
||||
```
|
||||
|
||||
Important behavior:
|
||||
|
||||
- This intentionally aborts if `config.yaml` (or `config.yml`/`configure.yml`) already exists.
|
||||
- Use `make config` only for first-time setup in a clean clone.
|
||||
|
||||
## 4) Command Order That Minimizes Failures
|
||||
|
||||
Use this exact order for local code changes:
|
||||
|
||||
1. `make check`
|
||||
2. `make install` (if frontend fails with proxy errors, rerun frontend install with proxy vars unset)
|
||||
3. Backend checks: `cd backend && make lint && make test`
|
||||
4. Frontend checks: `cd frontend && pnpm lint && pnpm typecheck`
|
||||
5. Frontend build (if UI changes or release-sensitive changes): `BETTER_AUTH_SECRET=... pnpm build`
|
||||
|
||||
Always run backend lint/tests before opening PRs because that is what CI enforces.
|
||||
|
||||
## 5) Project Layout and Architecture (High-Value Paths)
|
||||
|
||||
Root-level orchestration and config:
|
||||
|
||||
- `Makefile` - main local/dev/docker command entrypoints
|
||||
- `config.example.yaml` - primary app config template
|
||||
- `config.yaml` - local active config (gitignored)
|
||||
- `docker/docker-compose-dev.yaml` - Docker dev topology
|
||||
- `.github/workflows/backend-unit-tests.yml` - PR validation workflow
|
||||
|
||||
Backend core:
|
||||
|
||||
- `backend/packages/harness/deerflow/agents/` - lead agent, middleware chain, memory
|
||||
- `backend/app/gateway/` - FastAPI gateway API
|
||||
- `backend/packages/harness/deerflow/sandbox/` - sandbox provider + tool wrappers
|
||||
- `backend/packages/harness/deerflow/subagents/` - subagent registry/execution
|
||||
- `backend/packages/harness/deerflow/mcp/` - MCP integration
|
||||
- `backend/langgraph.json` - graph entrypoint (`deerflow.agents:make_lead_agent`)
|
||||
- `backend/pyproject.toml` - Python deps and `requires-python`
|
||||
- `backend/ruff.toml` - lint/format policy
|
||||
- `backend/tests/` - backend unit and integration-like tests
|
||||
|
||||
Frontend core:
|
||||
|
||||
- `frontend/src/app/` - Next.js routes/pages
|
||||
- `frontend/src/components/` - UI components
|
||||
- `frontend/src/core/` - app logic (threads, tools, API, models)
|
||||
- `frontend/src/env.js` - env schema/validation (critical for build behavior)
|
||||
- `frontend/package.json` - scripts/deps
|
||||
- `frontend/eslint.config.js` - lint rules
|
||||
- `frontend/tsconfig.json` - TS config
|
||||
|
||||
Skills and assets:
|
||||
|
||||
- `skills/public/` - built-in skill packs loaded by agent runtime
|
||||
|
||||
## 6) Pre-Checkin / Validation Expectations
|
||||
|
||||
Before submitting changes, run at minimum:
|
||||
|
||||
- Backend: `cd backend && make lint && make test`
|
||||
- Frontend (if touched): `cd frontend && pnpm lint && pnpm typecheck`
|
||||
- Frontend build when changing env/auth/routing/build-sensitive files: `BETTER_AUTH_SECRET=... pnpm build`
|
||||
|
||||
If touching orchestration/config (`Makefile`, `docker/*`, `config*.yaml`), also run `make dev` and verify the four services start.
|
||||
|
||||
## 7) Non-Obvious Dependencies and Gotchas
|
||||
|
||||
- Proxy env vars can silently break frontend network operations (`pnpm install`/registry access).
|
||||
- `BETTER_AUTH_SECRET` is effectively required for reliable frontend production build validation.
|
||||
- Next.js may warn about multiple lockfiles and workspace root inference; this is currently a warning, not a build blocker.
|
||||
- `make config` is non-idempotent by design when config already exists.
|
||||
- `make dev` includes process cleanup and can emit shutdown logs/noise if interrupted; this is expected.
|
||||
|
||||
## 8) Root Inventory (quick reference)
|
||||
|
||||
Important root entries:
|
||||
|
||||
- `.github/`
|
||||
- `backend/`
|
||||
- `frontend/`
|
||||
- `docker/`
|
||||
- `skills/`
|
||||
- `scripts/`
|
||||
- `docs/`
|
||||
- `README.md`
|
||||
- `CONTRIBUTING.md`
|
||||
- `Makefile`
|
||||
- `config.example.yaml`
|
||||
- `extensions_config.example.json`
|
||||
|
||||
## 9) Instruction Priority
|
||||
|
||||
Trust this onboarding guide first.
|
||||
|
||||
Only do broad repo searches (`grep/find/code search`) when:
|
||||
|
||||
- you need file-level implementation details not listed here,
|
||||
- a command here fails and you need updated replacement behavior,
|
||||
- or CI/workflow definitions have changed since this file was written.
|
||||
@@ -1,40 +0,0 @@
|
||||
name: Unit Tests
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [ 'main' ]
|
||||
pull_request:
|
||||
types: [opened, synchronize, reopened, ready_for_review]
|
||||
|
||||
concurrency:
|
||||
group: unit-tests-${{ github.event.pull_request.number || github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
backend-unit-tests:
|
||||
if: github.event.pull_request.draft == false
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v7
|
||||
|
||||
- name: Install backend dependencies
|
||||
working-directory: backend
|
||||
run: uv sync --group dev
|
||||
|
||||
- name: Run unit tests of backend
|
||||
working-directory: backend
|
||||
run: make test
|
||||
@@ -1,101 +0,0 @@
|
||||
name: Publish Containers
|
||||
|
||||
on:
|
||||
push:
|
||||
tags:
|
||||
- "v*"
|
||||
|
||||
jobs:
|
||||
|
||||
backend-container:
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
attestations: write
|
||||
id-token: write
|
||||
env:
|
||||
REGISTRY: ghcr.io
|
||||
IMAGE_NAME: ${{ github.repository }}-backend
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v6
|
||||
- name: Log in to the Container registry
|
||||
uses: docker/login-action@74a5d142397b4f367a81961eba4e8cd7edddf772 #v3.4.0
|
||||
with:
|
||||
registry: ${{ env.REGISTRY }}
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
- name: Extract metadata (tags, labels) for Docker
|
||||
id: meta
|
||||
uses: docker/metadata-action@902fa8ec7d6ecbf8d84d538b9b233a880e428804 #v5.7.0
|
||||
with:
|
||||
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
|
||||
tags: |
|
||||
type=ref,event=tag
|
||||
type=ref,event=branch
|
||||
type=sha
|
||||
type=raw,value=latest,enable={{is_default_branch}}
|
||||
- name: Build and push Docker image
|
||||
id: push
|
||||
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 #v6.18.0
|
||||
with:
|
||||
context: .
|
||||
file: backend/Dockerfile
|
||||
push: true
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
|
||||
- name: Generate artifact attestation
|
||||
uses: actions/attest-build-provenance@v2
|
||||
with:
|
||||
subject-name: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME}}
|
||||
subject-digest: ${{ steps.push.outputs.digest }}
|
||||
push-to-registry: true
|
||||
|
||||
frontend-container:
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
attestations: write
|
||||
id-token: write
|
||||
env:
|
||||
REGISTRY: ghcr.io
|
||||
IMAGE_NAME: ${{ github.repository }}-frontend
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v6
|
||||
- name: Log in to the Container registry
|
||||
uses: docker/login-action@74a5d142397b4f367a81961eba4e8cd7edddf772 #v3.4.0
|
||||
with:
|
||||
registry: ${{ env.REGISTRY }}
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
- name: Extract metadata (tags, labels) for Docker
|
||||
id: meta
|
||||
uses: docker/metadata-action@902fa8ec7d6ecbf8d84d538b9b233a880e428804 #v5.7.0
|
||||
with:
|
||||
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
|
||||
tags: |
|
||||
type=ref,event=tag
|
||||
type=ref,event=branch
|
||||
type=sha
|
||||
type=raw,value=latest,enable={{is_default_branch}}
|
||||
- name: Build and push Docker image
|
||||
id: push
|
||||
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 #v6.18.0
|
||||
with:
|
||||
context: .
|
||||
file: frontend/Dockerfile
|
||||
push: true
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
|
||||
- name: Generate artifact attestation
|
||||
uses: actions/attest-build-provenance@v2
|
||||
with:
|
||||
subject-name: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME}}
|
||||
subject-digest: ${{ steps.push.outputs.digest }}
|
||||
push-to-registry: true
|
||||
@@ -1,63 +0,0 @@
|
||||
name: E2E Tests
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [ 'main' ]
|
||||
paths:
|
||||
- 'frontend/**'
|
||||
- '.github/workflows/e2e-tests.yml'
|
||||
pull_request:
|
||||
types: [opened, synchronize, reopened, ready_for_review]
|
||||
paths:
|
||||
- 'frontend/**'
|
||||
- '.github/workflows/e2e-tests.yml'
|
||||
|
||||
concurrency:
|
||||
group: e2e-tests-${{ github.event.pull_request.number || github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
e2e-tests:
|
||||
if: ${{ github.event_name != 'pull_request' || github.event.pull_request.draft == false }}
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '22'
|
||||
|
||||
- name: Enable Corepack
|
||||
run: corepack enable
|
||||
|
||||
- name: Use pinned pnpm version
|
||||
run: corepack prepare pnpm@10.26.2 --activate
|
||||
|
||||
- name: Install frontend dependencies
|
||||
working-directory: frontend
|
||||
run: pnpm install --frozen-lockfile
|
||||
|
||||
- name: Install Playwright Chromium
|
||||
working-directory: frontend
|
||||
run: npx playwright install chromium --with-deps
|
||||
|
||||
- name: Run E2E tests
|
||||
working-directory: frontend
|
||||
run: pnpm exec playwright test
|
||||
env:
|
||||
SKIP_ENV_VALIDATION: '1'
|
||||
|
||||
- name: Upload Playwright report
|
||||
uses: actions/upload-artifact@v4
|
||||
if: ${{ !cancelled() }}
|
||||
with:
|
||||
name: playwright-report
|
||||
path: frontend/playwright-report/
|
||||
retention-days: 7
|
||||
@@ -1,43 +0,0 @@
|
||||
name: Frontend Unit Tests
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [ 'main' ]
|
||||
pull_request:
|
||||
types: [opened, synchronize, reopened, ready_for_review]
|
||||
|
||||
concurrency:
|
||||
group: frontend-unit-tests-${{ github.event.pull_request.number || github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
frontend-unit-tests:
|
||||
if: github.event.pull_request.draft == false
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '22'
|
||||
|
||||
- name: Enable Corepack
|
||||
run: corepack enable
|
||||
|
||||
- name: Use pinned pnpm version
|
||||
run: corepack prepare pnpm@10.26.2 --activate
|
||||
|
||||
- name: Install frontend dependencies
|
||||
working-directory: frontend
|
||||
run: pnpm install --frozen-lockfile
|
||||
|
||||
- name: Run unit tests of frontend
|
||||
working-directory: frontend
|
||||
run: make test
|
||||
@@ -1,74 +0,0 @@
|
||||
name: Lint Check
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [ 'main' ]
|
||||
pull_request:
|
||||
branches: [ '*' ]
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
lint:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v7
|
||||
|
||||
- name: Install dependencies
|
||||
working-directory: backend
|
||||
run: |
|
||||
uv sync --group dev
|
||||
|
||||
- name: Lint backend
|
||||
working-directory: backend
|
||||
run: make lint
|
||||
|
||||
lint-frontend:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '22'
|
||||
|
||||
- name: Enable Corepack
|
||||
run: corepack enable
|
||||
|
||||
- name: Use pinned pnpm version
|
||||
run: corepack prepare pnpm@10.26.2 --activate
|
||||
|
||||
- name: Install frontend dependencies
|
||||
run: |
|
||||
cd frontend
|
||||
pnpm install --frozen-lockfile
|
||||
|
||||
- name: Check frontend formatting
|
||||
run: |
|
||||
cd frontend
|
||||
pnpm format
|
||||
|
||||
- name: Run frontend linting
|
||||
run: |
|
||||
cd frontend
|
||||
pnpm lint
|
||||
|
||||
- name: Check TypeScript types
|
||||
run: |
|
||||
cd frontend
|
||||
pnpm typecheck
|
||||
|
||||
- name: Build frontend
|
||||
run: |
|
||||
cd frontend
|
||||
BETTER_AUTH_SECRET=local-dev-secret pnpm build
|
||||
@@ -0,0 +1,28 @@
|
||||
name: Lint Check
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [ 'main' ]
|
||||
pull_request:
|
||||
branches: [ '*' ]
|
||||
|
||||
jobs:
|
||||
lint:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
|
||||
- name: Install the latest version of uv
|
||||
uses: astral-sh/setup-uv@v5
|
||||
with:
|
||||
version: "latest"
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
uv venv --python 3.12
|
||||
uv pip install -e ".[dev]"
|
||||
|
||||
- name: Run linters
|
||||
run: |
|
||||
source .venv/bin/activate
|
||||
make lint
|
||||
@@ -0,0 +1,45 @@
|
||||
name: Test Cases Check
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [ 'main' ]
|
||||
pull_request:
|
||||
branches: [ '*' ]
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
|
||||
- name: Install the latest version of uv
|
||||
uses: astral-sh/setup-uv@v5
|
||||
with:
|
||||
version: "latest"
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
uv venv --python 3.12
|
||||
uv pip install -e ".[dev]"
|
||||
uv pip install -e ".[test]"
|
||||
|
||||
- name: Run test cases with coverage
|
||||
run: |
|
||||
source .venv/bin/activate
|
||||
TAVILY_API_KEY=mock-key make coverage
|
||||
|
||||
- name: Generate HTML Coverage Report
|
||||
run: |
|
||||
source .venv/bin/activate
|
||||
python -m coverage html -d coverage_html
|
||||
|
||||
- name: Upload Coverage Report
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: coverage-report
|
||||
path: coverage_html/
|
||||
|
||||
- name: Display Coverage Summary
|
||||
run: |
|
||||
source .venv/bin/activate
|
||||
python -m coverage report
|
||||
+13
-49
@@ -1,62 +1,26 @@
|
||||
# DeerFlow docker image cache
|
||||
docker/.cache/
|
||||
# oh-my-claudecode state
|
||||
.omc/
|
||||
# OS generated files
|
||||
.DS_Store
|
||||
*.local
|
||||
._*
|
||||
.Spotlight-V100
|
||||
.Trashes
|
||||
ehthumbs.db
|
||||
Thumbs.db
|
||||
|
||||
# Python cache
|
||||
# Python-generated files
|
||||
__pycache__/
|
||||
*.pyc
|
||||
*.pyo
|
||||
*.py[oc]
|
||||
build/
|
||||
dist/
|
||||
wheels/
|
||||
*.egg-info
|
||||
.coverage
|
||||
agent_history.gif
|
||||
static/browser_history/*.gif
|
||||
|
||||
# Virtual environments
|
||||
.venv
|
||||
venv/
|
||||
|
||||
# Environment variables
|
||||
.env
|
||||
|
||||
# Configuration files
|
||||
config.yaml
|
||||
mcp_config.json
|
||||
extensions_config.json
|
||||
# user conf
|
||||
conf.yaml
|
||||
|
||||
# IDE
|
||||
.idea/
|
||||
.vscode/
|
||||
.langgraph_api/
|
||||
|
||||
# Coverage report
|
||||
# coverage report
|
||||
coverage.xml
|
||||
coverage/
|
||||
.deer-flow/
|
||||
.claude/
|
||||
skills/custom/*
|
||||
logs/
|
||||
log/
|
||||
debug.log
|
||||
|
||||
# Local git hooks (keep only on this machine, do not push)
|
||||
.githooks/
|
||||
|
||||
# pnpm
|
||||
.pnpm-store
|
||||
sandbox_image_cache.tar
|
||||
|
||||
# ignore the legacy `web` folder
|
||||
web/
|
||||
|
||||
# Deployment artifacts
|
||||
backend/Dockerfile.langgraph
|
||||
config.yaml.bak
|
||||
.playwright-mcp
|
||||
/frontend/test-results/
|
||||
/frontend/playwright-report/
|
||||
.gstack/
|
||||
.worktrees
|
||||
|
||||
@@ -1,33 +0,0 @@
|
||||
repos:
|
||||
# Backend: ruff lint + format via uv (uses the same ruff version as backend deps)
|
||||
- repo: local
|
||||
hooks:
|
||||
- id: ruff
|
||||
name: ruff lint
|
||||
entry: bash -c 'cd backend && uv run ruff check --fix "${@/#backend\//}"' --
|
||||
language: system
|
||||
types_or: [python]
|
||||
files: ^backend/
|
||||
- id: ruff-format
|
||||
name: ruff format
|
||||
entry: bash -c 'cd backend && uv run ruff format "${@/#backend\//}"' --
|
||||
language: system
|
||||
types_or: [python]
|
||||
files: ^backend/
|
||||
|
||||
# Frontend: eslint + prettier (must run from frontend/ for node_modules resolution)
|
||||
- repo: local
|
||||
hooks:
|
||||
- id: frontend-eslint
|
||||
name: eslint (frontend)
|
||||
entry: bash -c 'cd frontend && npx eslint --fix "${@/#frontend\//}"' --
|
||||
language: system
|
||||
types_or: [javascript, tsx, ts]
|
||||
files: ^frontend/
|
||||
|
||||
- id: frontend-prettier
|
||||
name: prettier (frontend)
|
||||
entry: bash -c 'cd frontend && npx prettier --write "${@/#frontend\//}"' --
|
||||
language: system
|
||||
files: ^frontend/
|
||||
types_or: [javascript, tsx, ts, json, css]
|
||||
Vendored
+60
@@ -0,0 +1,60 @@
|
||||
{
|
||||
"version": "0.2.0",
|
||||
"configurations": [
|
||||
{
|
||||
"name": "Python: 当前文件",
|
||||
"type": "debugpy",
|
||||
"request": "launch",
|
||||
"program": "${file}",
|
||||
"console": "integratedTerminal",
|
||||
"justMyCode": true
|
||||
},
|
||||
{
|
||||
"name": "Python: main.py",
|
||||
"type": "debugpy",
|
||||
"request": "launch",
|
||||
"program": "${workspaceFolder}/main.py",
|
||||
"console": "integratedTerminal",
|
||||
"justMyCode": false,
|
||||
"env": {
|
||||
"PYTHONPATH": "${workspaceFolder}"
|
||||
},
|
||||
"args": [
|
||||
"--debug", "--max_plan_iterations", "1", "--max_step_num", "1"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Python: llm.py",
|
||||
"type": "debugpy",
|
||||
"request": "launch",
|
||||
"program": "${workspaceFolder}/src/llms/llm.py",
|
||||
"console": "integratedTerminal",
|
||||
"justMyCode": true,
|
||||
"env": {
|
||||
"PYTHONPATH": "${workspaceFolder}"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Python: server.py",
|
||||
"type": "debugpy",
|
||||
"request": "launch",
|
||||
"program": "${workspaceFolder}/server.py",
|
||||
"console": "integratedTerminal",
|
||||
"justMyCode": false,
|
||||
"env": {
|
||||
"PYTHONPATH": "${workspaceFolder}"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Python: graph.py",
|
||||
"type": "debugpy",
|
||||
"request": "launch",
|
||||
"program": "${workspaceFolder}/src/ppt/graph/builder.py",
|
||||
"console": "integratedTerminal",
|
||||
"justMyCode": false,
|
||||
"env": {
|
||||
"PYTHONPATH": "${workspaceFolder}"
|
||||
}
|
||||
},
|
||||
]
|
||||
}
|
||||
@@ -1,128 +0,0 @@
|
||||
# Contributor Covenant Code of Conduct
|
||||
|
||||
## Our Pledge
|
||||
|
||||
We as members, contributors, and leaders pledge to make participation in our
|
||||
community a harassment-free experience for everyone, regardless of age, body
|
||||
size, visible or invisible disability, ethnicity, sex characteristics, gender
|
||||
identity and expression, level of experience, education, socio-economic status,
|
||||
nationality, personal appearance, race, religion, or sexual identity
|
||||
and orientation.
|
||||
|
||||
We pledge to act and interact in ways that contribute to an open, welcoming,
|
||||
diverse, inclusive, and healthy community.
|
||||
|
||||
## Our Standards
|
||||
|
||||
Examples of behavior that contributes to a positive environment for our
|
||||
community include:
|
||||
|
||||
* Demonstrating empathy and kindness toward other people
|
||||
* Being respectful of differing opinions, viewpoints, and experiences
|
||||
* Giving and gracefully accepting constructive feedback
|
||||
* Accepting responsibility and apologizing to those affected by our mistakes,
|
||||
and learning from the experience
|
||||
* Focusing on what is best not just for us as individuals, but for the
|
||||
overall community
|
||||
|
||||
Examples of unacceptable behavior include:
|
||||
|
||||
* The use of sexualized language or imagery, and sexual attention or
|
||||
advances of any kind
|
||||
* Trolling, insulting or derogatory comments, and personal or political attacks
|
||||
* Public or private harassment
|
||||
* Publishing others' private information, such as a physical or email
|
||||
address, without their explicit permission
|
||||
* Other conduct which could reasonably be considered inappropriate in a
|
||||
professional setting
|
||||
|
||||
## Enforcement Responsibilities
|
||||
|
||||
Community leaders are responsible for clarifying and enforcing our standards of
|
||||
acceptable behavior and will take appropriate and fair corrective action in
|
||||
response to any behavior that they deem inappropriate, threatening, offensive,
|
||||
or harmful.
|
||||
|
||||
Community leaders have the right and responsibility to remove, edit, or reject
|
||||
comments, commits, code, wiki edits, issues, and other contributions that are
|
||||
not aligned to this Code of Conduct, and will communicate reasons for moderation
|
||||
decisions when appropriate.
|
||||
|
||||
## Scope
|
||||
|
||||
This Code of Conduct applies within all community spaces, and also applies when
|
||||
an individual is officially representing the community in public spaces.
|
||||
Examples of representing our community include using an official e-mail address,
|
||||
posting via an official social media account, or acting as an appointed
|
||||
representative at an online or offline event.
|
||||
|
||||
## Enforcement
|
||||
|
||||
Instances of abusive, harassing, or otherwise unacceptable behavior may be
|
||||
reported to the community leaders responsible for enforcement at
|
||||
willem.jiang@gmail.com.
|
||||
All complaints will be reviewed and investigated promptly and fairly.
|
||||
|
||||
All community leaders are obligated to respect the privacy and security of the
|
||||
reporter of any incident.
|
||||
|
||||
## Enforcement Guidelines
|
||||
|
||||
Community leaders will follow these Community Impact Guidelines in determining
|
||||
the consequences for any action they deem in violation of this Code of Conduct:
|
||||
|
||||
### 1. Correction
|
||||
|
||||
**Community Impact**: Use of inappropriate language or other behavior deemed
|
||||
unprofessional or unwelcome in the community.
|
||||
|
||||
**Consequence**: A private, written warning from community leaders, providing
|
||||
clarity around the nature of the violation and an explanation of why the
|
||||
behavior was inappropriate. A public apology may be requested.
|
||||
|
||||
### 2. Warning
|
||||
|
||||
**Community Impact**: A violation through a single incident or series
|
||||
of actions.
|
||||
|
||||
**Consequence**: A warning with consequences for continued behavior. No
|
||||
interaction with the people involved, including unsolicited interaction with
|
||||
those enforcing the Code of Conduct, for a specified period of time. This
|
||||
includes avoiding interactions in community spaces as well as external channels
|
||||
like social media. Violating these terms may lead to a temporary or
|
||||
permanent ban.
|
||||
|
||||
### 3. Temporary Ban
|
||||
|
||||
**Community Impact**: A serious violation of community standards, including
|
||||
sustained inappropriate behavior.
|
||||
|
||||
**Consequence**: A temporary ban from any sort of interaction or public
|
||||
communication with the community for a specified period of time. No public or
|
||||
private interaction with the people involved, including unsolicited interaction
|
||||
with those enforcing the Code of Conduct, is allowed during this period.
|
||||
Violating these terms may lead to a permanent ban.
|
||||
|
||||
### 4. Permanent Ban
|
||||
|
||||
**Community Impact**: Demonstrating a pattern of violation of community
|
||||
standards, including sustained inappropriate behavior, harassment of an
|
||||
individual, or aggression toward or disparagement of classes of individuals.
|
||||
|
||||
**Consequence**: A permanent ban from any sort of public interaction within
|
||||
the community.
|
||||
|
||||
## Attribution
|
||||
|
||||
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
|
||||
version 2.0, available at
|
||||
https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
|
||||
|
||||
Community Impact Guidelines were inspired by [Mozilla's code of conduct
|
||||
enforcement ladder](https://github.com/mozilla/diversity).
|
||||
|
||||
[homepage]: https://www.contributor-covenant.org
|
||||
|
||||
For answers to common questions about this code of conduct, see the FAQ at
|
||||
https://www.contributor-covenant.org/faq. Translations are available at
|
||||
https://www.contributor-covenant.org/translations.
|
||||
+133
@@ -0,0 +1,133 @@
|
||||
# Contributing to DeerFlow
|
||||
|
||||
Thank you for your interest in contributing to DeerFlow! We welcome contributions of all kinds from the community.
|
||||
|
||||
## Ways to Contribute
|
||||
|
||||
There are many ways you can contribute to DeerFlow:
|
||||
|
||||
- **Code Contributions**: Add new features, fix bugs, or improve performance
|
||||
- **Documentation**: Improve README, add code comments, or create examples
|
||||
- **Bug Reports**: Submit detailed bug reports through issues
|
||||
- **Feature Requests**: Suggest new features or improvements
|
||||
- **Code Reviews**: Review pull requests from other contributors
|
||||
- **Community Support**: Help others in discussions and issues
|
||||
|
||||
## Development Setup
|
||||
|
||||
1. Fork the repository
|
||||
2. Clone your fork:
|
||||
```bash
|
||||
git clone https://github.com/bytedance/deer-flow.git
|
||||
cd deer-flow
|
||||
```
|
||||
3. Set up your development environment:
|
||||
```bash
|
||||
# Install dependencies, uv will take care of the python interpreter and venv creation
|
||||
uv sync
|
||||
|
||||
# For development, install additional dependencies
|
||||
uv pip install -e ".[dev]"
|
||||
uv pip install -e ".[test]"
|
||||
```
|
||||
4. Configure pre-commit hooks:
|
||||
```bash
|
||||
chmod +x pre-commit
|
||||
ln -s ../../pre-commit .git/hooks/pre-commit
|
||||
```
|
||||
|
||||
## Development Process
|
||||
|
||||
1. Create a new branch:
|
||||
```bash
|
||||
git checkout -b feature/amazing-feature
|
||||
```
|
||||
|
||||
2. Make your changes following our coding standards:
|
||||
- Write clear, documented code
|
||||
- Follow PEP 8 style guidelines
|
||||
- Add tests for new features
|
||||
- Update documentation as needed
|
||||
|
||||
3. Run tests and checks:
|
||||
```bash
|
||||
make test # Run tests
|
||||
make lint # Run linting
|
||||
make format # Format code
|
||||
make coverage # Check test coverage
|
||||
```
|
||||
|
||||
4. Commit your changes:
|
||||
```bash
|
||||
git commit -m 'Add some amazing feature'
|
||||
```
|
||||
|
||||
5. Push to your fork:
|
||||
```bash
|
||||
git push origin feature/amazing-feature
|
||||
```
|
||||
|
||||
6. Open a Pull Request
|
||||
|
||||
## Pull Request Guidelines
|
||||
|
||||
- Fill in the pull request template completely
|
||||
- Include tests for new features
|
||||
- Update documentation as needed
|
||||
- Ensure all tests pass and there are no linting errors
|
||||
- Keep pull requests focused on a single feature or fix
|
||||
- Reference any related issues
|
||||
|
||||
## Code Style
|
||||
|
||||
- Follow PEP 8 guidelines
|
||||
- Use type hints where possible
|
||||
- Write descriptive docstrings
|
||||
- Keep functions and methods focused and single-purpose
|
||||
- Comment complex logic
|
||||
- Python version requirement: >= 3.12
|
||||
|
||||
## Testing
|
||||
|
||||
Run the test suite:
|
||||
```bash
|
||||
# Run all tests
|
||||
make test
|
||||
|
||||
# Run specific test file
|
||||
pytest tests/integration/test_workflow.py
|
||||
|
||||
# Run with coverage
|
||||
make coverage
|
||||
```
|
||||
|
||||
## Code Quality
|
||||
|
||||
```bash
|
||||
# Run linting
|
||||
make lint
|
||||
|
||||
# Format code
|
||||
make format
|
||||
```
|
||||
|
||||
## Community Guidelines
|
||||
|
||||
- Be respectful and inclusive
|
||||
- Follow our code of conduct
|
||||
- Help others learn and grow
|
||||
- Give constructive feedback
|
||||
- Stay focused on improving the project
|
||||
|
||||
## Need Help?
|
||||
|
||||
If you need help with anything:
|
||||
- Check existing issues and discussions
|
||||
- Join our community channels
|
||||
- Ask questions in discussions
|
||||
|
||||
## License
|
||||
|
||||
By contributing to DeerFlow, you agree that your contributions will be licensed under the MIT License.
|
||||
|
||||
We appreciate your contributions to making DeerFlow better!
|
||||
-334
@@ -1,334 +0,0 @@
|
||||
# Contributing to DeerFlow
|
||||
|
||||
Thank you for your interest in contributing to DeerFlow! This guide will help you set up your development environment and understand our development workflow.
|
||||
|
||||
## Development Environment Setup
|
||||
|
||||
We offer two development environments. **Docker is recommended** for the most consistent and hassle-free experience.
|
||||
|
||||
### Option 1: Docker Development (Recommended)
|
||||
|
||||
Docker provides a consistent, isolated environment with all dependencies pre-configured. No need to install Node.js, Python, or nginx on your local machine.
|
||||
|
||||
#### Prerequisites
|
||||
|
||||
- Docker Desktop or Docker Engine
|
||||
- pnpm (for caching optimization)
|
||||
|
||||
#### Setup Steps
|
||||
|
||||
1. **Configure the application**:
|
||||
```bash
|
||||
# Copy example configuration
|
||||
cp config.example.yaml config.yaml
|
||||
|
||||
# Set your API keys
|
||||
export OPENAI_API_KEY="your-key-here"
|
||||
# or edit config.yaml directly
|
||||
```
|
||||
|
||||
2. **Initialize Docker environment** (first time only):
|
||||
```bash
|
||||
make docker-init
|
||||
```
|
||||
This will:
|
||||
- Build Docker images
|
||||
- Install frontend dependencies (pnpm)
|
||||
- Install backend dependencies (uv)
|
||||
- Share pnpm cache with host for faster builds
|
||||
|
||||
3. **Start development services**:
|
||||
```bash
|
||||
make docker-start
|
||||
```
|
||||
`make docker-start` reads `config.yaml` and starts `provisioner` only for provisioner/Kubernetes sandbox mode.
|
||||
|
||||
All services will start with hot-reload enabled:
|
||||
- Frontend changes are automatically reloaded
|
||||
- Backend changes trigger automatic restart
|
||||
- Gateway-hosted LangGraph-compatible runtime supports hot-reload
|
||||
|
||||
4. **Access the application**:
|
||||
- Web Interface: http://localhost:2026
|
||||
- API Gateway: http://localhost:2026/api/*
|
||||
- LangGraph-compatible API: http://localhost:2026/api/langgraph/*
|
||||
|
||||
#### Docker Commands
|
||||
|
||||
```bash
|
||||
# Build the custom k3s image (with pre-cached sandbox image)
|
||||
make docker-init
|
||||
# Start Docker services (mode-aware, localhost:2026)
|
||||
make docker-start
|
||||
# Stop Docker development services
|
||||
make docker-stop
|
||||
# View Docker development logs
|
||||
make docker-logs
|
||||
# View Docker frontend logs
|
||||
make docker-logs-frontend
|
||||
# View Docker gateway logs
|
||||
make docker-logs-gateway
|
||||
```
|
||||
|
||||
If Docker builds are slow in your network, you can override the default package registries before running `make docker-init` or `make docker-start`:
|
||||
|
||||
```bash
|
||||
export UV_INDEX_URL=https://pypi.org/simple
|
||||
export NPM_REGISTRY=https://registry.npmjs.org
|
||||
```
|
||||
|
||||
#### Recommended host resources
|
||||
|
||||
Use these as practical starting points for development and review environments:
|
||||
|
||||
| Scenario | Starting point | Recommended | Notes |
|
||||
|---------|-----------|------------|-------|
|
||||
| `make dev` on one machine | 4 vCPU, 8 GB RAM | 8 vCPU, 16 GB RAM | Best when DeerFlow uses hosted model APIs. |
|
||||
| `make docker-start` review environment | 4 vCPU, 8 GB RAM | 8 vCPU, 16 GB RAM | Docker image builds and sandbox containers need extra headroom. |
|
||||
| Shared Linux test server | 8 vCPU, 16 GB RAM | 16 vCPU, 32 GB RAM | Prefer this for heavier multi-agent runs or multiple reviewers. |
|
||||
|
||||
`2 vCPU / 4 GB` environments often fail to start reliably or become unresponsive under normal DeerFlow workloads.
|
||||
|
||||
#### Linux: Docker daemon permission denied
|
||||
|
||||
If `make docker-init`, `make docker-start`, or `make docker-stop` fails on Linux with an error like below, your current user likely does not have permission to access the Docker daemon socket:
|
||||
|
||||
```text
|
||||
unable to get image 'deer-flow-gateway': permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock
|
||||
```
|
||||
|
||||
Recommended fix: add your current user to the `docker` group so Docker commands work without `sudo`.
|
||||
|
||||
1. Confirm the `docker` group exists:
|
||||
```bash
|
||||
getent group docker
|
||||
```
|
||||
2. Add your current user to the `docker` group:
|
||||
```bash
|
||||
sudo usermod -aG docker $USER
|
||||
```
|
||||
3. Apply the new group membership. The most reliable option is to log out completely and then log back in. If you want to refresh the current shell session instead, run:
|
||||
```bash
|
||||
newgrp docker
|
||||
```
|
||||
4. Verify Docker access:
|
||||
```bash
|
||||
docker ps
|
||||
```
|
||||
5. Retry the DeerFlow command:
|
||||
```bash
|
||||
make docker-stop
|
||||
make docker-start
|
||||
```
|
||||
|
||||
If `docker ps` still reports a permission error after `usermod`, fully log out and log back in before retrying.
|
||||
|
||||
#### Docker Architecture
|
||||
|
||||
```
|
||||
Host Machine
|
||||
↓
|
||||
Docker Compose (deer-flow-dev)
|
||||
├→ nginx (port 2026) ← Reverse proxy
|
||||
├→ web (port 3000) ← Frontend with hot-reload
|
||||
├→ gateway (port 8001) ← Gateway API + LangGraph-compatible runtime with hot-reload
|
||||
└→ provisioner (optional, port 8002) ← Started only in provisioner/K8s sandbox mode
|
||||
```
|
||||
|
||||
**Benefits of Docker Development**:
|
||||
- ✅ Consistent environment across different machines
|
||||
- ✅ No need to install Node.js, Python, or nginx locally
|
||||
- ✅ Isolated dependencies and services
|
||||
- ✅ Easy cleanup and reset
|
||||
- ✅ Hot-reload for all services
|
||||
- ✅ Production-like environment
|
||||
|
||||
### Option 2: Local Development
|
||||
|
||||
If you prefer to run services directly on your machine:
|
||||
|
||||
#### Prerequisites
|
||||
|
||||
Check that you have all required tools installed:
|
||||
|
||||
```bash
|
||||
make check
|
||||
```
|
||||
|
||||
Required tools:
|
||||
- Node.js 22+
|
||||
- pnpm
|
||||
- uv (Python package manager)
|
||||
- nginx
|
||||
|
||||
#### Setup Steps
|
||||
|
||||
1. **Configure the application** (same as Docker setup above)
|
||||
|
||||
2. **Install dependencies** (this also sets up pre-commit hooks):
|
||||
```bash
|
||||
make install
|
||||
```
|
||||
|
||||
3. **Run development server** (starts all services with nginx):
|
||||
```bash
|
||||
make dev
|
||||
```
|
||||
|
||||
4. **Access the application**:
|
||||
- Web Interface: http://localhost:2026
|
||||
- All API requests are automatically proxied through nginx
|
||||
|
||||
#### Manual Service Control
|
||||
|
||||
If you need to start services individually:
|
||||
|
||||
1. **Start backend service**:
|
||||
```bash
|
||||
# Terminal 1: Start Gateway API + embedded agent runtime (port 8001)
|
||||
cd backend
|
||||
make dev
|
||||
|
||||
# Terminal 2: Start Frontend (port 3000)
|
||||
cd frontend
|
||||
pnpm dev
|
||||
```
|
||||
|
||||
2. **Start nginx**:
|
||||
```bash
|
||||
make nginx
|
||||
# or directly: nginx -c $(pwd)/docker/nginx/nginx.local.conf -g 'daemon off;'
|
||||
```
|
||||
|
||||
3. **Access the application**:
|
||||
- Web Interface: http://localhost:2026
|
||||
|
||||
#### Nginx Configuration
|
||||
|
||||
The nginx configuration provides:
|
||||
- Unified entry point on port 2026
|
||||
- Rewrites `/api/langgraph/*` to Gateway's LangGraph-compatible API (8001)
|
||||
- Routes other `/api/*` endpoints to Gateway API (8001)
|
||||
- Routes non-API requests to Frontend (3000)
|
||||
- Same-origin API routing; split-origin or port-forwarded browser clients should use the Gateway `GATEWAY_CORS_ORIGINS` allowlist
|
||||
- SSE/streaming support for real-time agent responses
|
||||
- Optimized timeouts for long-running operations
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
deer-flow/
|
||||
├── config.example.yaml # Configuration template
|
||||
├── extensions_config.example.json # MCP and Skills configuration template
|
||||
├── Makefile # Build and development commands
|
||||
├── scripts/
|
||||
│ └── docker.sh # Docker management script
|
||||
├── docker/
|
||||
│ ├── docker-compose-dev.yaml # Docker Compose configuration
|
||||
│ └── nginx/
|
||||
│ ├── nginx.conf # Nginx config for Docker
|
||||
│ └── nginx.local.conf # Nginx config for local dev
|
||||
├── backend/ # Backend application
|
||||
│ ├── src/
|
||||
│ │ ├── gateway/ # Gateway API and LangGraph-compatible runtime (port 8001)
|
||||
│ │ ├── agents/ # LangGraph agent runtime used by Gateway
|
||||
│ │ ├── mcp/ # Model Context Protocol integration
|
||||
│ │ ├── skills/ # Skills system
|
||||
│ │ └── sandbox/ # Sandbox execution
|
||||
│ ├── docs/ # Backend documentation
|
||||
│ └── Makefile # Backend commands
|
||||
├── frontend/ # Frontend application
|
||||
│ └── Makefile # Frontend commands
|
||||
└── skills/ # Agent skills
|
||||
├── public/ # Public skills
|
||||
└── custom/ # Custom skills
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Browser
|
||||
↓
|
||||
Nginx (port 2026) ← Unified entry point
|
||||
├→ Frontend (port 3000) ← / (non-API requests)
|
||||
└→ Gateway API (port 8001) ← /api/* and /api/langgraph/* (LangGraph-compatible agent interactions)
|
||||
```
|
||||
|
||||
## Development Workflow
|
||||
|
||||
1. **Create a feature branch**:
|
||||
```bash
|
||||
git checkout -b feature/your-feature-name
|
||||
```
|
||||
|
||||
2. **Make your changes** with hot-reload enabled
|
||||
|
||||
3. **Format and lint your code** (CI will reject unformatted code):
|
||||
```bash
|
||||
# Backend
|
||||
cd backend
|
||||
make format # ruff check --fix + ruff format
|
||||
|
||||
# Frontend
|
||||
cd frontend
|
||||
pnpm format:write # Prettier
|
||||
```
|
||||
|
||||
4. **Test your changes** thoroughly
|
||||
|
||||
5. **Commit your changes**:
|
||||
```bash
|
||||
git add .
|
||||
git commit -m "feat: description of your changes"
|
||||
```
|
||||
|
||||
6. **Push and create a Pull Request**:
|
||||
```bash
|
||||
git push origin feature/your-feature-name
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
# Backend tests
|
||||
cd backend
|
||||
make test
|
||||
|
||||
# Frontend unit tests
|
||||
cd frontend
|
||||
make test
|
||||
|
||||
# Frontend E2E tests (requires Chromium; builds and auto-starts the Next.js production server)
|
||||
cd frontend
|
||||
make test-e2e
|
||||
```
|
||||
|
||||
### PR Regression Checks
|
||||
|
||||
Every pull request triggers the following CI workflows:
|
||||
|
||||
- **Backend unit tests** — [.github/workflows/backend-unit-tests.yml](.github/workflows/backend-unit-tests.yml)
|
||||
- **Frontend unit tests** — [.github/workflows/frontend-unit-tests.yml](.github/workflows/frontend-unit-tests.yml)
|
||||
- **Frontend E2E tests** — [.github/workflows/e2e-tests.yml](.github/workflows/e2e-tests.yml) (triggered only when `frontend/` files change)
|
||||
|
||||
## Code Style
|
||||
|
||||
- **Backend (Python)**: We use `ruff` for linting and formatting. Run `make format` before committing.
|
||||
- **Frontend (TypeScript)**: We use ESLint and Prettier. Run `pnpm format:write` before committing.
|
||||
- CI enforces formatting — PRs with unformatted code will fail the lint check.
|
||||
|
||||
## Documentation
|
||||
|
||||
- [Configuration Guide](backend/docs/CONFIGURATION.md) - Setup and configuration
|
||||
- [Architecture Overview](backend/CLAUDE.md) - Technical architecture
|
||||
- [MCP Setup Guide](backend/docs/MCP_SERVER.md) - Model Context Protocol configuration
|
||||
|
||||
## Need Help?
|
||||
|
||||
- Check existing [Issues](https://github.com/bytedance/deer-flow/issues)
|
||||
- Read the [Documentation](backend/docs/)
|
||||
- Ask questions in [Discussions](https://github.com/bytedance/deer-flow/discussions)
|
||||
|
||||
## License
|
||||
|
||||
By contributing to DeerFlow, you agree that your contributions will be licensed under the [MIT License](./LICENSE).
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
FROM ghcr.io/astral-sh/uv:python3.12-bookworm-slim
|
||||
|
||||
# Install uv.
|
||||
COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/uv
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Copy the application into the container.
|
||||
COPY . /app
|
||||
|
||||
# Install the application dependencies.
|
||||
RUN uv sync --frozen --no-cache
|
||||
|
||||
EXPOSE 8000
|
||||
|
||||
# Run the application.
|
||||
CMD ["uv", "run", "python", "server.py", "--host", "0.0.0.0", "--port", "8000"]
|
||||
-87
@@ -1,87 +0,0 @@
|
||||
# DeerFlow Install
|
||||
|
||||
This file is for coding agents. If the DeerFlow repository is not already cloned and open, clone `https://github.com/bytedance/deer-flow.git` first, then continue from the repository root.
|
||||
|
||||
## Goal
|
||||
|
||||
Bootstrap a DeerFlow local development workspace on the user's machine with the least risky path available.
|
||||
|
||||
Default preference:
|
||||
|
||||
1. Docker development environment
|
||||
2. Local development environment
|
||||
|
||||
Do not assume API keys or model credentials exist. Set up everything that can be prepared safely, then stop with a concise summary of what the user still needs to provide.
|
||||
|
||||
## Operating Rules
|
||||
|
||||
- Be idempotent. Re-running this document should not damage an existing setup.
|
||||
- Prefer existing repo commands over ad hoc shell commands.
|
||||
- Do not use `sudo` or install system packages without explicit user approval.
|
||||
- Do not overwrite existing user config values unless the user asks.
|
||||
- If a step fails, stop, explain the blocker, and provide the smallest next action.
|
||||
- If multiple setup paths are possible, prefer Docker when Docker is already available.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
Consider the setup successful when all of the following are true:
|
||||
|
||||
- The DeerFlow repository is cloned and the current working directory is the repo root.
|
||||
- `config.yaml` exists.
|
||||
- For Docker setup, `make docker-init` completed successfully and Docker prerequisites are prepared, but services are not assumed to be running yet.
|
||||
- For local setup, `make check` passed or reported no missing prerequisites, and `make install` completed successfully.
|
||||
- The user receives the exact next command to launch DeerFlow.
|
||||
- The user also receives any missing model configuration or referenced environment variable names from `config.yaml`, without inspecting secret-bearing files for actual values.
|
||||
|
||||
## Steps
|
||||
|
||||
- If the current directory is not the DeerFlow repository root, clone `https://github.com/bytedance/deer-flow.git` if needed, then change into the repository root.
|
||||
- Confirm the current directory is the DeerFlow repository root by checking that `Makefile`, `backend/`, `frontend/`, and `config.example.yaml` exist.
|
||||
- Detect whether `config.yaml` already exists.
|
||||
- If `config.yaml` does not exist, run `make config`.
|
||||
- Detect whether Docker is available and the daemon is reachable with `docker info`.
|
||||
- If Docker is available:
|
||||
- Run `make docker-init`.
|
||||
- Treat this as Docker prerequisite preparation only. Do not claim that app services, compose validation, or image builds have already succeeded.
|
||||
- Do not start long-running services unless the user explicitly asks or this setup request clearly includes launch verification.
|
||||
- Tell the user the recommended next command is `make docker-start`.
|
||||
- If Docker is not available:
|
||||
- Run `make check`.
|
||||
- If `make check` reports missing system dependencies such as `node`, `pnpm`, `uv`, or `nginx`, stop and report the missing tools instead of attempting privileged installs.
|
||||
- If prerequisites are satisfied, run `make install`.
|
||||
- Tell the user the recommended next command is `make dev`.
|
||||
- Inspect `config.yaml` only for missing model entries or referenced environment variable placeholders. Do not read `.env`, `frontend/.env`, or other secret-bearing files.
|
||||
- If no model is configured, tell the user they must add at least one entry under `models` in `config.yaml`.
|
||||
- If `config.yaml` references variables such as `$OPENAI_API_KEY`, tell the user which variable names still need real values, but do not verify them by opening secret-bearing files.
|
||||
- If the repository already appears configured, avoid repeating expensive work unless it is necessary to verify the environment.
|
||||
|
||||
## Verification
|
||||
|
||||
Use the lightest verification that matches the chosen setup path.
|
||||
|
||||
For Docker setup:
|
||||
|
||||
- Confirm `make docker-init` completed successfully.
|
||||
- Confirm `config.yaml` exists.
|
||||
- State explicitly that Docker services were not started and `make docker-start` is still the first real launch step.
|
||||
- Do not leave background services running unless the user asked for that.
|
||||
|
||||
For local setup:
|
||||
|
||||
- Confirm `make install` completed successfully.
|
||||
- Confirm `config.yaml` exists.
|
||||
- Do not leave background services running unless the user asked for that.
|
||||
|
||||
## Final Response Format
|
||||
|
||||
Return a short status report with:
|
||||
|
||||
1. Setup path used: Docker or local
|
||||
2. Setup level reached: Docker prerequisites prepared or local dependencies installed
|
||||
3. Files created or detected: for example `config.yaml`
|
||||
4. Remaining user action: model config, env var values, auth files, or nothing
|
||||
5. Exact next command to start DeerFlow
|
||||
|
||||
## EXECUTE NOW
|
||||
|
||||
Complete the steps above. When finished, stop after the setup boundary and report status instead of continuing into unrelated project work.
|
||||
@@ -1,7 +1,6 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2025 Bytedance Ltd. and/or its affiliates
|
||||
Copyright (c) 2025-2026 DeerFlow Authors
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
@@ -19,4 +18,4 @@ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
SOFTWARE.
|
||||
@@ -1,183 +1,22 @@
|
||||
# DeerFlow - Unified Development Environment
|
||||
.PHONY: lint format install-dev serve test coverage
|
||||
|
||||
.PHONY: help config config-upgrade check install setup doctor dev dev-daemon start start-daemon stop up down clean docker-init docker-start docker-stop docker-logs docker-logs-frontend docker-logs-gateway
|
||||
install-dev:
|
||||
uv pip install -e ".[dev]" && uv pip install -e ".[test]"
|
||||
|
||||
BASH ?= bash
|
||||
BACKEND_UV_RUN = cd backend && uv run
|
||||
format:
|
||||
uv run black --preview .
|
||||
|
||||
# Detect OS for Windows compatibility
|
||||
ifeq ($(OS),Windows_NT)
|
||||
SHELL := cmd.exe
|
||||
PYTHON ?= python
|
||||
# Run repo shell scripts through Git Bash when Make is launched from cmd.exe / PowerShell.
|
||||
RUN_WITH_GIT_BASH = call scripts\run-with-git-bash.cmd
|
||||
else
|
||||
PYTHON ?= python3
|
||||
RUN_WITH_GIT_BASH =
|
||||
endif
|
||||
lint:
|
||||
uv run black --check .
|
||||
|
||||
help:
|
||||
@echo "DeerFlow Development Commands:"
|
||||
@echo " make setup - Interactive setup wizard (recommended for new users)"
|
||||
@echo " make doctor - Check configuration and system requirements"
|
||||
@echo " make config - Generate local config files (aborts if config already exists)"
|
||||
@echo " make config-upgrade - Merge new fields from config.example.yaml into config.yaml"
|
||||
@echo " make check - Check if all required tools are installed"
|
||||
@echo " make install - Install all dependencies (frontend + backend + pre-commit hooks)"
|
||||
@echo " make setup-sandbox - Pre-pull sandbox container image (recommended)"
|
||||
@echo " make dev - Start all services in development mode (with hot-reloading)"
|
||||
@echo " make dev-daemon - Start dev services in background (daemon mode)"
|
||||
@echo " make start - Start all services in production mode (optimized, no hot-reloading)"
|
||||
@echo " make start-daemon - Start prod services in background (daemon mode)"
|
||||
@echo " make stop - Stop all running services"
|
||||
@echo " make clean - Clean up processes and temporary files"
|
||||
@echo ""
|
||||
@echo "Docker Production Commands:"
|
||||
@echo " make up - Build and start production Docker services (localhost:2026)"
|
||||
@echo " make down - Stop and remove production Docker containers"
|
||||
@echo ""
|
||||
@echo "Docker Development Commands:"
|
||||
@echo " make docker-init - Pull the sandbox image"
|
||||
@echo " make docker-start - Start Docker services (mode-aware from config.yaml, localhost:2026)"
|
||||
@echo " make docker-stop - Stop Docker development services"
|
||||
@echo " make docker-logs - View Docker development logs"
|
||||
@echo " make docker-logs-frontend - View Docker frontend logs"
|
||||
@echo " make docker-logs-gateway - View Docker gateway logs"
|
||||
serve:
|
||||
uv run server.py --reload
|
||||
|
||||
## Setup & Diagnosis
|
||||
setup:
|
||||
@$(BACKEND_UV_RUN) python ../scripts/setup_wizard.py
|
||||
test:
|
||||
uv run pytest tests/
|
||||
|
||||
doctor:
|
||||
@$(BACKEND_UV_RUN) python ../scripts/doctor.py
|
||||
langgraph-dev:
|
||||
uvx --refresh --from "langgraph-cli[inmem]" --with-editable . --python 3.12 langgraph dev --allow-blocking
|
||||
|
||||
config:
|
||||
@$(PYTHON) ./scripts/configure.py
|
||||
|
||||
config-upgrade:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/config-upgrade.sh
|
||||
|
||||
# Check required tools
|
||||
check:
|
||||
@$(PYTHON) ./scripts/check.py
|
||||
|
||||
# Install all dependencies
|
||||
install:
|
||||
@echo "Installing backend dependencies..."
|
||||
@cd backend && uv sync
|
||||
@echo "Installing frontend dependencies..."
|
||||
@cd frontend && pnpm install
|
||||
@echo "Installing pre-commit hooks..."
|
||||
@$(BACKEND_UV_RUN) --with pre-commit pre-commit install
|
||||
@echo "✓ All dependencies installed"
|
||||
@echo ""
|
||||
@echo "=========================================="
|
||||
@echo " Optional: Pre-pull Sandbox Image"
|
||||
@echo "=========================================="
|
||||
@echo ""
|
||||
@echo "If you plan to use Docker/Container-based sandbox, you can pre-pull the image:"
|
||||
@echo " make setup-sandbox"
|
||||
@echo ""
|
||||
|
||||
# Pre-pull sandbox Docker image (optional but recommended)
|
||||
setup-sandbox:
|
||||
@echo "=========================================="
|
||||
@echo " Pre-pulling Sandbox Container Image"
|
||||
@echo "=========================================="
|
||||
@echo ""
|
||||
@IMAGE=$$(grep -A 20 "# sandbox:" config.yaml 2>/dev/null | grep "image:" | awk '{print $$2}' | head -1); \
|
||||
if [ -z "$$IMAGE" ]; then \
|
||||
IMAGE="enterprise-public-cn-beijing.cr.volces.com/vefaas-public/all-in-one-sandbox:latest"; \
|
||||
echo "Using default image: $$IMAGE"; \
|
||||
else \
|
||||
echo "Using configured image: $$IMAGE"; \
|
||||
fi; \
|
||||
echo ""; \
|
||||
if command -v container >/dev/null 2>&1 && [ "$$(uname)" = "Darwin" ]; then \
|
||||
echo "Detected Apple Container on macOS, pulling image..."; \
|
||||
container image pull "$$IMAGE" || echo "⚠ Apple Container pull failed, will try Docker"; \
|
||||
fi; \
|
||||
if command -v docker >/dev/null 2>&1; then \
|
||||
echo "Pulling image using Docker..."; \
|
||||
if docker pull "$$IMAGE"; then \
|
||||
echo ""; \
|
||||
echo "✓ Sandbox image pulled successfully"; \
|
||||
else \
|
||||
echo ""; \
|
||||
echo "⚠ Failed to pull sandbox image (this is OK for local sandbox mode)"; \
|
||||
fi; \
|
||||
else \
|
||||
echo "✗ Neither Docker nor Apple Container is available"; \
|
||||
echo " Please install Docker: https://docs.docker.com/get-docker/"; \
|
||||
exit 1; \
|
||||
fi
|
||||
|
||||
# Start all services in development mode (with hot-reloading)
|
||||
dev:
|
||||
@$(PYTHON) ./scripts/check.py
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --dev
|
||||
|
||||
# Start all services in production mode (with optimizations)
|
||||
start:
|
||||
@$(PYTHON) ./scripts/check.py
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --prod
|
||||
|
||||
# Start all services in daemon mode (background)
|
||||
dev-daemon:
|
||||
@$(PYTHON) ./scripts/check.py
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --dev --daemon
|
||||
|
||||
# Start prod services in daemon mode (background)
|
||||
start-daemon:
|
||||
@$(PYTHON) ./scripts/check.py
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --prod --daemon
|
||||
|
||||
# Stop all services
|
||||
stop:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --stop
|
||||
|
||||
# Clean up
|
||||
clean: stop
|
||||
@echo "Cleaning up..."
|
||||
@-rm -rf backend/.deer-flow 2>/dev/null || true
|
||||
@-rm -rf backend/.langgraph_api 2>/dev/null || true
|
||||
@-rm -rf logs/*.log 2>/dev/null || true
|
||||
@echo "✓ Cleanup complete"
|
||||
|
||||
# ==========================================
|
||||
# Docker Development Commands
|
||||
# ==========================================
|
||||
|
||||
# Initialize Docker containers and install dependencies
|
||||
docker-init:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/docker.sh init
|
||||
|
||||
# Start Docker development environment
|
||||
docker-start:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/docker.sh start
|
||||
|
||||
# Stop Docker development environment
|
||||
docker-stop:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/docker.sh stop
|
||||
|
||||
# View Docker development logs
|
||||
docker-logs:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/docker.sh logs
|
||||
|
||||
# View Docker development logs
|
||||
docker-logs-frontend:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/docker.sh logs --frontend
|
||||
docker-logs-gateway:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/docker.sh logs --gateway
|
||||
|
||||
# ==========================================
|
||||
# Production Docker Commands
|
||||
# ==========================================
|
||||
|
||||
# Build and start production services
|
||||
up:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/deploy.sh
|
||||
|
||||
# Stop and remove production containers
|
||||
down:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/deploy.sh down
|
||||
coverage:
|
||||
uv run pytest --cov=src tests/ --cov-report=term-missing --cov-report=xml
|
||||
|
||||
+495
@@ -0,0 +1,495 @@
|
||||
# 🦌 DeerFlow
|
||||
|
||||
[](https://www.python.org/downloads/)
|
||||
[](https://opensource.org/licenses/MIT)
|
||||
[](https://deepwiki.com/bytedance/deer-flow)
|
||||
<!-- DeepWiki badge generated by https://deepwiki.ryoppippi.com/ -->
|
||||
|
||||
[English](./README.md) | [简体中文](./README_zh.md) | [日本語](./README_ja.md) | [Deutsch](./README_de.md)
|
||||
|
||||
> Aus Open Source entstanden, an Open Source zurückgeben.
|
||||
|
||||
**DeerFlow** (**D**eep **E**xploration and **E**fficient **R**esearch **Flow**) ist ein Community-getriebenes Framework für tiefgehende Recherche, das auf der großartigen Arbeit der Open-Source-Community aufbaut. Unser Ziel ist es, Sprachmodelle mit spezialisierten Werkzeugen für Aufgaben wie Websuche, Crawling und Python-Code-Ausführung zu kombinieren und gleichzeitig der Community, die dies möglich gemacht hat, etwas zurückzugeben.
|
||||
|
||||
Besuchen Sie [unsere offizielle Website](https://deerflow.tech/) für weitere Details.
|
||||
|
||||
## Demo
|
||||
|
||||
### Video
|
||||
|
||||
https://github.com/user-attachments/assets/f3786598-1f2a-4d07-919e-8b99dfa1de3e
|
||||
|
||||
In dieser Demo zeigen wir, wie man DeerFlow nutzt, um:
|
||||
- Nahtlos mit MCP-Diensten zu integrieren
|
||||
- Den Prozess der tiefgehenden Recherche durchzuführen und einen umfassenden Bericht mit Bildern zu erstellen
|
||||
- Podcast-Audio basierend auf dem generierten Bericht zu erstellen
|
||||
|
||||
### Wiedergaben
|
||||
|
||||
- [Wie hoch ist der Eiffelturm im Vergleich zum höchsten Gebäude?](https://deerflow.tech/chat?replay=eiffel-tower-vs-tallest-building)
|
||||
- [Was sind die angesagtesten Repositories auf GitHub?](https://deerflow.tech/chat?replay=github-top-trending-repo)
|
||||
- [Einen Artikel über traditionelle Gerichte aus Nanjing schreiben](https://deerflow.tech/chat?replay=nanjing-traditional-dishes)
|
||||
- [Wie dekoriert man eine Mietwohnung?](https://deerflow.tech/chat?replay=rental-apartment-decoration)
|
||||
- [Besuchen Sie unsere offizielle Website, um weitere Wiedergaben zu entdecken.](https://deerflow.tech/#case-studies)
|
||||
|
||||
---
|
||||
|
||||
|
||||
## 📑 Inhaltsverzeichnis
|
||||
|
||||
- [🚀 Schnellstart](#schnellstart)
|
||||
- [🌟 Funktionen](#funktionen)
|
||||
- [🏗️ Architektur](#architektur)
|
||||
- [🛠️ Entwicklung](#entwicklung)
|
||||
- [🗣️ Text-zu-Sprache-Integration](#text-zu-sprache-integration)
|
||||
- [📚 Beispiele](#beispiele)
|
||||
- [❓ FAQ](#faq)
|
||||
- [📜 Lizenz](#lizenz)
|
||||
- [💖 Danksagungen](#danksagungen)
|
||||
- [⭐ Star-Verlauf](#star-verlauf)
|
||||
|
||||
|
||||
## Schnellstart
|
||||
|
||||
DeerFlow ist in Python entwickelt und kommt mit einer in Node.js geschriebenen Web-UI. Um einen reibungslosen Einrichtungsprozess zu gewährleisten, empfehlen wir die Verwendung der folgenden Tools:
|
||||
|
||||
### Empfohlene Tools
|
||||
- **[`uv`](https://docs.astral.sh/uv/getting-started/installation/):**
|
||||
Vereinfacht die Verwaltung von Python-Umgebungen und Abhängigkeiten. `uv` erstellt automatisch eine virtuelle Umgebung im Stammverzeichnis und installiert alle erforderlichen Pakete für Sie—keine manuelle Installation von Python-Umgebungen notwendig.
|
||||
|
||||
- **[`nvm`](https://github.com/nvm-sh/nvm):**
|
||||
Verwalten Sie mühelos mehrere Versionen der Node.js-Laufzeit.
|
||||
|
||||
- **[`pnpm`](https://pnpm.io/installation):**
|
||||
Installieren und verwalten Sie Abhängigkeiten des Node.js-Projekts.
|
||||
|
||||
### Umgebungsanforderungen
|
||||
Stellen Sie sicher, dass Ihr System die folgenden Mindestanforderungen erfüllt:
|
||||
- **[Python](https://www.python.org/downloads/):** Version `3.12+`
|
||||
- **[Node.js](https://nodejs.org/en/download/):** Version `22+`
|
||||
|
||||
### Installation
|
||||
```bash
|
||||
# Repository klonen
|
||||
git clone https://github.com/bytedance/deer-flow.git
|
||||
cd deer-flow
|
||||
|
||||
# Abhängigkeiten installieren, uv kümmert sich um den Python-Interpreter und die Erstellung der venv sowie die Installation der erforderlichen Pakete
|
||||
uv sync
|
||||
|
||||
# Konfigurieren Sie .env mit Ihren API-Schlüsseln
|
||||
# Tavily: https://app.tavily.com/home
|
||||
# Brave_SEARCH: https://brave.com/search/api/
|
||||
# volcengine TTS: Fügen Sie Ihre TTS-Anmeldedaten hinzu, falls vorhanden
|
||||
cp .env.example .env
|
||||
|
||||
# Siehe die Abschnitte 'Unterstützte Suchmaschinen' und 'Text-zu-Sprache-Integration' unten für alle verfügbaren Optionen
|
||||
|
||||
# Konfigurieren Sie conf.yaml für Ihr LLM-Modell und API-Schlüssel
|
||||
# Weitere Details finden Sie unter 'docs/configuration_guide.md'
|
||||
cp conf.yaml.example conf.yaml
|
||||
|
||||
# Installieren Sie marp für PPT-Generierung
|
||||
# https://github.com/marp-team/marp-cli?tab=readme-ov-file#use-package-manager
|
||||
brew install marp-cli
|
||||
```
|
||||
|
||||
Optional können Sie Web-UI-Abhängigkeiten über [pnpm](https://pnpm.io/installation) installieren:
|
||||
|
||||
```bash
|
||||
cd deer-flow/web
|
||||
pnpm install
|
||||
```
|
||||
|
||||
### Konfigurationen
|
||||
|
||||
Weitere Informationen finden Sie im [Konfigurationsleitfaden](docs/configuration_guide.md).
|
||||
|
||||
> [!HINWEIS]
|
||||
> Lesen Sie den Leitfaden sorgfältig, bevor Sie das Projekt starten, und aktualisieren Sie die Konfigurationen entsprechend Ihren spezifischen Einstellungen und Anforderungen.
|
||||
|
||||
### Konsolen-UI
|
||||
|
||||
Der schnellste Weg, um das Projekt auszuführen, ist die Verwendung der Konsolen-UI.
|
||||
|
||||
```bash
|
||||
# Führen Sie das Projekt in einer bash-ähnlichen Shell aus
|
||||
uv run main.py
|
||||
```
|
||||
|
||||
### Web-UI
|
||||
|
||||
Dieses Projekt enthält auch eine Web-UI, die ein dynamischeres und ansprechenderes interaktives Erlebnis bietet.
|
||||
> [!HINWEIS]
|
||||
> Sie müssen zuerst die Abhängigkeiten der Web-UI installieren.
|
||||
|
||||
```bash
|
||||
# Führen Sie sowohl den Backend- als auch den Frontend-Server im Entwicklungsmodus aus
|
||||
# Unter macOS/Linux
|
||||
./bootstrap.sh -d
|
||||
|
||||
# Unter Windows
|
||||
bootstrap.bat -d
|
||||
```
|
||||
|
||||
Öffnen Sie Ihren Browser und besuchen Sie [`http://localhost:3000`](http://localhost:3000), um die Web-UI zu erkunden.
|
||||
|
||||
Weitere Details finden Sie im Verzeichnis [`web`](./web/).
|
||||
|
||||
|
||||
## Unterstützte Suchmaschinen
|
||||
|
||||
DeerFlow unterstützt mehrere Suchmaschinen, die in Ihrer `.env`-Datei über die Variable `SEARCH_API` konfiguriert werden können:
|
||||
|
||||
- **Tavily** (Standard): Eine spezialisierte Such-API für KI-Anwendungen
|
||||
- Erfordert `TAVILY_API_KEY` in Ihrer `.env`-Datei
|
||||
- Registrieren Sie sich unter: https://app.tavily.com/home
|
||||
|
||||
- **DuckDuckGo**: Datenschutzorientierte Suchmaschine
|
||||
- Kein API-Schlüssel erforderlich
|
||||
|
||||
- **Brave Search**: Datenschutzorientierte Suchmaschine mit erweiterten Funktionen
|
||||
- Erfordert `BRAVE_SEARCH_API_KEY` in Ihrer `.env`-Datei
|
||||
- Registrieren Sie sich unter: https://brave.com/search/api/
|
||||
|
||||
- **Arxiv**: Wissenschaftliche Papiersuche für akademische Forschung
|
||||
- Kein API-Schlüssel erforderlich
|
||||
- Spezialisiert auf wissenschaftliche und akademische Papiere
|
||||
|
||||
Um Ihre bevorzugte Suchmaschine zu konfigurieren, setzen Sie die Variable `SEARCH_API` in Ihrer `.env`-Datei:
|
||||
|
||||
```bash
|
||||
# Wählen Sie eine: tavily, duckduckgo, brave_search, arxiv
|
||||
SEARCH_API=tavily
|
||||
```
|
||||
|
||||
## Funktionen
|
||||
|
||||
### Kernfähigkeiten
|
||||
|
||||
- 🤖 **LLM-Integration**
|
||||
- Unterstützt die Integration der meisten Modelle über [litellm](https://docs.litellm.ai/docs/providers).
|
||||
- Unterstützung für Open-Source-Modelle wie Qwen
|
||||
- OpenAI-kompatible API-Schnittstelle
|
||||
- Mehrstufiges LLM-System für unterschiedliche Aufgabenkomplexitäten
|
||||
|
||||
### Tools und MCP-Integrationen
|
||||
|
||||
- 🔍 **Suche und Abruf**
|
||||
- Websuche über Tavily, Brave Search und mehr
|
||||
- Crawling mit Jina
|
||||
- Fortgeschrittene Inhaltsextraktion
|
||||
|
||||
- 🔗 **MCP Nahtlose Integration**
|
||||
- Erweiterte Fähigkeiten für privaten Domänenzugriff, Wissensgraphen, Webbrowsing und mehr
|
||||
- Erleichtert die Integration verschiedener Forschungswerkzeuge und -methoden
|
||||
|
||||
### Menschliche Zusammenarbeit
|
||||
|
||||
- 🧠 **Mensch-in-der-Schleife**
|
||||
- Unterstützt interaktive Modifikation von Forschungsplänen mit natürlicher Sprache
|
||||
- Unterstützt automatische Akzeptanz von Forschungsplänen
|
||||
|
||||
- 📝 **Bericht-Nachbearbeitung**
|
||||
- Unterstützt Notion-ähnliche Blockbearbeitung
|
||||
- Ermöglicht KI-Verfeinerungen, einschließlich KI-unterstützter Polierung, Satzkürzung und -erweiterung
|
||||
- Angetrieben von [tiptap](https://tiptap.dev/)
|
||||
|
||||
### Inhaltserstellung
|
||||
|
||||
- 🎙️ **Podcast- und Präsentationserstellung**
|
||||
- KI-gestützte Podcast-Skripterstellung und Audiosynthese
|
||||
- Automatisierte Erstellung einfacher PowerPoint-Präsentationen
|
||||
- Anpassbare Vorlagen für maßgeschneiderte Inhalte
|
||||
|
||||
|
||||
## Architektur
|
||||
|
||||
DeerFlow implementiert eine modulare Multi-Agenten-Systemarchitektur, die für automatisierte Forschung und Codeanalyse konzipiert ist. Das System basiert auf LangGraph und ermöglicht einen flexiblen zustandsbasierten Workflow, bei dem Komponenten über ein klar definiertes Nachrichtenübermittlungssystem kommunizieren.
|
||||
|
||||
|
||||
> Sehen Sie es live auf [deerflow.tech](https://deerflow.tech/#multi-agent-architecture)
|
||||
|
||||
Das System verwendet einen optimierten Workflow mit den folgenden Komponenten:
|
||||
|
||||
1. **Koordinator**: Der Einstiegspunkt, der den Workflow-Lebenszyklus verwaltet
|
||||
- Initiiert den Forschungsprozess basierend auf Benutzereingaben
|
||||
- Delegiert Aufgaben bei Bedarf an den Planer
|
||||
- Fungiert als primäre Schnittstelle zwischen dem Benutzer und dem System
|
||||
|
||||
2. **Planer**: Strategische Komponente für Aufgabenzerlegung und -planung
|
||||
- Analysiert Forschungsziele und erstellt strukturierte Ausführungspläne
|
||||
- Bestimmt, ob ausreichend Kontext verfügbar ist oder ob weitere Forschung benötigt wird
|
||||
- Verwaltet den Forschungsablauf und entscheidet, wann der endgültige Bericht erstellt wird
|
||||
|
||||
3. **Forschungsteam**: Eine Sammlung spezialisierter Agenten, die den Plan ausführen:
|
||||
- **Forscher**: Führt Websuchen und Informationssammlung mit Tools wie Websuchmaschinen, Crawling und sogar MCP-Diensten durch.
|
||||
- **Codierer**: Behandelt Codeanalyse, -ausführung und technische Aufgaben mit dem Python REPL Tool.
|
||||
Jeder Agent hat Zugriff auf spezifische Tools, die für seine Rolle optimiert sind, und operiert innerhalb des LangGraph-Frameworks
|
||||
|
||||
4. **Reporter**: Endphasenprozessor für Forschungsergebnisse
|
||||
- Aggregiert Erkenntnisse vom Forschungsteam
|
||||
- Verarbeitet und strukturiert die gesammelten Informationen
|
||||
- Erstellt umfassende Forschungsberichte
|
||||
|
||||
## Text-zu-Sprache-Integration
|
||||
|
||||
DeerFlow enthält jetzt eine Text-zu-Sprache (TTS)-Funktion, mit der Sie Forschungsberichte in Sprache umwandeln können. Diese Funktion verwendet die volcengine TTS API, um hochwertige Audios aus Text zu generieren. Funktionen wie Geschwindigkeit, Lautstärke und Tonhöhe können ebenfalls angepasst werden.
|
||||
|
||||
### Verwendung der TTS API
|
||||
|
||||
Sie können auf die TTS-Funktionalität über den Endpunkt `/api/tts` zugreifen:
|
||||
|
||||
```bash
|
||||
# Beispiel API-Aufruf mit curl
|
||||
curl --location 'http://localhost:8000/api/tts' \
|
||||
--header 'Content-Type: application/json' \
|
||||
--data '{
|
||||
"text": "Dies ist ein Test der Text-zu-Sprache-Funktionalität.",
|
||||
"speed_ratio": 1.0,
|
||||
"volume_ratio": 1.0,
|
||||
"pitch_ratio": 1.0
|
||||
}' \
|
||||
--output speech.mp3
|
||||
```
|
||||
|
||||
|
||||
## Entwicklung
|
||||
|
||||
### Testen
|
||||
|
||||
Führen Sie die Testsuite aus:
|
||||
|
||||
```bash
|
||||
# Alle Tests ausführen
|
||||
make test
|
||||
|
||||
# Spezifische Testdatei ausführen
|
||||
pytest tests/integration/test_workflow.py
|
||||
|
||||
# Mit Abdeckung ausführen
|
||||
make coverage
|
||||
```
|
||||
|
||||
### Codequalität
|
||||
|
||||
```bash
|
||||
# Lint ausführen
|
||||
make lint
|
||||
|
||||
# Code formatieren
|
||||
make format
|
||||
```
|
||||
|
||||
### Debugging mit LangGraph Studio
|
||||
|
||||
DeerFlow verwendet LangGraph für seine Workflow-Architektur. Sie können LangGraph Studio verwenden, um den Workflow in Echtzeit zu debuggen und zu visualisieren.
|
||||
|
||||
#### LangGraph Studio lokal ausführen
|
||||
|
||||
DeerFlow enthält eine `langgraph.json`-Konfigurationsdatei, die die Graphstruktur und Abhängigkeiten für das LangGraph Studio definiert. Diese Datei verweist auf die im Projekt definierten Workflow-Graphen und lädt automatisch Umgebungsvariablen aus der `.env`-Datei.
|
||||
|
||||
##### Mac
|
||||
|
||||
```bash
|
||||
# Installieren Sie den uv-Paketmanager, wenn Sie ihn noch nicht haben
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
|
||||
# Installieren Sie Abhängigkeiten und starten Sie den LangGraph-Server
|
||||
uvx --refresh --from "langgraph-cli[inmem]" --with-editable . --python 3.12 langgraph dev --allow-blocking
|
||||
```
|
||||
|
||||
##### Windows / Linux
|
||||
|
||||
```bash
|
||||
# Abhängigkeiten installieren
|
||||
pip install -e .
|
||||
pip install -U "langgraph-cli[inmem]"
|
||||
|
||||
# LangGraph-Server starten
|
||||
langgraph dev
|
||||
```
|
||||
|
||||
Nach dem Start des LangGraph-Servers sehen Sie mehrere URLs im Terminal:
|
||||
- API: http://127.0.0.1:2024
|
||||
- Studio UI: https://smith.langchain.com/studio/?baseUrl=http://127.0.0.1:2024
|
||||
- API-Dokumentation: http://127.0.0.1:2024/docs
|
||||
|
||||
Öffnen Sie den Studio UI-Link in Ihrem Browser, um auf die Debugging-Schnittstelle zuzugreifen.
|
||||
|
||||
#### Verwendung von LangGraph Studio
|
||||
|
||||
In der Studio UI können Sie:
|
||||
|
||||
1. Den Workflow-Graphen visualisieren und sehen, wie Komponenten verbunden sind
|
||||
2. Die Ausführung in Echtzeit verfolgen, um zu sehen, wie Daten durch das System fließen
|
||||
3. Den Zustand in jedem Schritt des Workflows inspizieren
|
||||
4. Probleme durch Untersuchung von Ein- und Ausgaben jeder Komponente debuggen
|
||||
5. Feedback während der Planungsphase geben, um Forschungspläne zu verfeinern
|
||||
|
||||
Wenn Sie ein Forschungsthema in der Studio UI einreichen, können Sie die gesamte Workflow-Ausführung sehen, einschließlich:
|
||||
- Die Planungsphase, in der der Forschungsplan erstellt wird
|
||||
- Die Feedback-Schleife, in der Sie den Plan ändern können
|
||||
- Die Forschungs- und Schreibphasen für jeden Abschnitt
|
||||
- Die Erstellung des endgültigen Berichts
|
||||
|
||||
### Aktivieren von LangSmith-Tracing
|
||||
|
||||
DeerFlow unterstützt LangSmith-Tracing, um Ihnen beim Debuggen und Überwachen Ihrer Workflows zu helfen. Um LangSmith-Tracing zu aktivieren:
|
||||
|
||||
1. Stellen Sie sicher, dass Ihre `.env`-Datei die folgenden Konfigurationen enthält (siehe `.env.example`):
|
||||
```bash
|
||||
LANGSMITH_TRACING=true
|
||||
LANGSMITH_ENDPOINT="https://api.smith.langchain.com"
|
||||
LANGSMITH_API_KEY="xxx"
|
||||
LANGSMITH_PROJECT="xxx"
|
||||
```
|
||||
|
||||
2. Starten Sie das Tracing mit LangSmith lokal, indem Sie folgenden Befehl ausführen:
|
||||
```bash
|
||||
langgraph dev
|
||||
```
|
||||
|
||||
Dies aktiviert die Trace-Visualisierung in LangGraph Studio und sendet Ihre Traces zur Überwachung und Analyse an LangSmith.
|
||||
|
||||
## Beispiele
|
||||
|
||||
Die folgenden Beispiele demonstrieren die Fähigkeiten von DeerFlow:
|
||||
|
||||
### Forschungsberichte
|
||||
|
||||
1. **OpenAI Sora Bericht** - Analyse von OpenAIs Sora KI-Tool
|
||||
- Diskutiert Funktionen, Zugang, Prompt-Engineering, Einschränkungen und ethische Überlegungen
|
||||
- [Vollständigen Bericht ansehen](examples/openai_sora_report.md)
|
||||
|
||||
2. **Googles Agent-to-Agent-Protokoll Bericht** - Überblick über Googles Agent-to-Agent (A2A)-Protokoll
|
||||
- Diskutiert seine Rolle in der KI-Agentenkommunikation und seine Beziehung zum Model Context Protocol (MCP) von Anthropic
|
||||
- [Vollständigen Bericht ansehen](examples/what_is_agent_to_agent_protocol.md)
|
||||
|
||||
3. **Was ist MCP?** - Eine umfassende Analyse des Begriffs "MCP" in mehreren Kontexten
|
||||
- Untersucht Model Context Protocol in KI, Monocalciumphosphat in der Chemie und Micro-channel Plate in der Elektronik
|
||||
- [Vollständigen Bericht ansehen](examples/what_is_mcp.md)
|
||||
|
||||
4. **Bitcoin-Preisschwankungen** - Analyse der jüngsten Bitcoin-Preisbewegungen
|
||||
- Untersucht Markttrends, regulatorische Einflüsse und technische Indikatoren
|
||||
- Bietet Empfehlungen basierend auf historischen Daten
|
||||
- [Vollständigen Bericht ansehen](examples/bitcoin_price_fluctuation.md)
|
||||
|
||||
5. **Was ist LLM?** - Eine eingehende Erforschung großer Sprachmodelle
|
||||
- Diskutiert Architektur, Training, Anwendungen und ethische Überlegungen
|
||||
- [Vollständigen Bericht ansehen](examples/what_is_llm.md)
|
||||
|
||||
6. **Wie nutzt man Claude für tiefgehende Recherche?** - Best Practices und Workflows für die Verwendung von Claude in der tiefgehenden Forschung
|
||||
- Behandelt Prompt-Engineering, Datenanalyse und Integration mit anderen Tools
|
||||
- [Vollständigen Bericht ansehen](examples/how_to_use_claude_deep_research.md)
|
||||
|
||||
7. **KI-Adoption im Gesundheitswesen: Einflussfaktoren** - Analyse der Faktoren, die die KI-Adoption im Gesundheitswesen vorantreiben
|
||||
- Diskutiert KI-Technologien, Datenqualität, ethische Überlegungen, wirtschaftliche Bewertungen, organisatorische Bereitschaft und digitale Infrastruktur
|
||||
- [Vollständigen Bericht ansehen](examples/AI_adoption_in_healthcare.md)
|
||||
|
||||
8. **Auswirkungen des Quantencomputing auf die Kryptographie** - Analyse der Auswirkungen des Quantencomputing auf die Kryptographie
|
||||
- Diskutiert Schwachstellen der klassischen Kryptographie, Post-Quanten-Kryptographie und quantenresistente kryptographische Lösungen
|
||||
- [Vollständigen Bericht ansehen](examples/Quantum_Computing_Impact_on_Cryptography.md)
|
||||
|
||||
9. **Cristiano Ronaldos Leistungshöhepunkte** - Analyse der Leistungshöhepunkte von Cristiano Ronaldo
|
||||
- Diskutiert seine Karriereerfolge, internationalen Tore und Leistungen in verschiedenen Spielen
|
||||
- [Vollständigen Bericht ansehen](examples/Cristiano_Ronaldo's_Performance_Highlights.md)
|
||||
|
||||
Um diese Beispiele auszuführen oder Ihre eigenen Forschungsberichte zu erstellen, können Sie die folgenden Befehle verwenden:
|
||||
|
||||
```bash
|
||||
# Mit einer spezifischen Anfrage ausführen
|
||||
uv run main.py "Welche Faktoren beeinflussen die KI-Adoption im Gesundheitswesen?"
|
||||
|
||||
# Mit benutzerdefinierten Planungsparametern ausführen
|
||||
uv run main.py --max_plan_iterations 3 "Wie wirkt sich Quantencomputing auf die Kryptographie aus?"
|
||||
|
||||
# Im interaktiven Modus mit eingebauten Fragen ausführen
|
||||
uv run main.py --interactive
|
||||
|
||||
# Oder mit grundlegendem interaktiven Prompt ausführen
|
||||
uv run main.py
|
||||
|
||||
# Alle verfügbaren Optionen anzeigen
|
||||
uv run main.py --help
|
||||
```
|
||||
|
||||
### Interaktiver Modus
|
||||
|
||||
Die Anwendung unterstützt jetzt einen interaktiven Modus mit eingebauten Fragen in Englisch und Chinesisch:
|
||||
|
||||
1. Starten Sie den interaktiven Modus:
|
||||
```bash
|
||||
uv run main.py --interactive
|
||||
```
|
||||
|
||||
2. Wählen Sie Ihre bevorzugte Sprache (English oder 中文)
|
||||
|
||||
3. Wählen Sie aus einer Liste von eingebauten Fragen oder wählen Sie die Option, Ihre eigene Frage zu stellen
|
||||
|
||||
4. Das System wird Ihre Frage verarbeiten und einen umfassenden Forschungsbericht generieren
|
||||
|
||||
### Mensch-in-der-Schleife
|
||||
|
||||
DeerFlow enthält einen Mensch-in-der-Schleife-Mechanismus, der es Ihnen ermöglicht, Forschungspläne vor ihrer Ausführung zu überprüfen, zu bearbeiten und zu genehmigen:
|
||||
|
||||
1. **Planüberprüfung**: Wenn Mensch-in-der-Schleife aktiviert ist, präsentiert das System den generierten Forschungsplan zur Überprüfung vor der Ausführung
|
||||
|
||||
2. **Feedback geben**: Sie können:
|
||||
- Den Plan akzeptieren, indem Sie mit `[ACCEPTED]` antworten
|
||||
- Den Plan bearbeiten, indem Sie Feedback geben (z.B., `[EDIT PLAN] Fügen Sie mehr Schritte zur technischen Implementierung hinzu`)
|
||||
- Das System wird Ihr Feedback einarbeiten und einen überarbeiteten Plan generieren
|
||||
|
||||
3. **Automatische Akzeptanz**: Sie können die automatische Akzeptanz aktivieren, um den Überprüfungsprozess zu überspringen:
|
||||
- Über API: Setzen Sie `auto_accepted_plan: true` in Ihrer Anfrage
|
||||
|
||||
4. **API-Integration**: Bei Verwendung der API können Sie Feedback über den Parameter `feedback` geben:
|
||||
```json
|
||||
{
|
||||
"messages": [{"role": "user", "content": "Was ist Quantencomputing?"}],
|
||||
"thread_id": "my_thread_id",
|
||||
"auto_accepted_plan": false,
|
||||
"feedback": "[EDIT PLAN] Mehr über Quantenalgorithmen aufnehmen"
|
||||
}
|
||||
```
|
||||
|
||||
### Kommandozeilenargumente
|
||||
|
||||
Die Anwendung unterstützt mehrere Kommandozeilenargumente, um ihr Verhalten anzupassen:
|
||||
|
||||
- **query**: Die zu verarbeitende Forschungsanfrage (kann mehrere Wörter umfassen)
|
||||
- **--interactive**: Im interaktiven Modus mit eingebauten Fragen ausführen
|
||||
- **--max_plan_iterations**: Maximale Anzahl von Planungszyklen (Standard: 1)
|
||||
- **--max_step_num**: Maximale Anzahl von Schritten in einem Forschungsplan (Standard: 3)
|
||||
- **--debug**: Detaillierte Debug-Protokollierung aktivieren
|
||||
|
||||
## FAQ
|
||||
|
||||
Weitere Informationen finden Sie in der [FAQ.md](docs/FAQ.md).
|
||||
|
||||
## Lizenz
|
||||
|
||||
Dieses Projekt ist Open Source und unter der [MIT-Lizenz](./LICENSE) verfügbar.
|
||||
|
||||
## Danksagungen
|
||||
|
||||
DeerFlow baut auf der unglaublichen Arbeit der Open-Source-Community auf. Wir sind allen Projekten und Mitwirkenden zutiefst dankbar, deren Bemühungen DeerFlow möglich gemacht haben. Wahrhaftig stehen wir auf den Schultern von Riesen.
|
||||
|
||||
Wir möchten unsere aufrichtige Wertschätzung den folgenden Projekten für ihre unschätzbaren Beiträge aussprechen:
|
||||
|
||||
- **[LangChain](https://github.com/langchain-ai/langchain)**: Ihr außergewöhnliches Framework unterstützt unsere LLM-Interaktionen und -Ketten und ermöglicht nahtlose Integration und Funktionalität.
|
||||
- **[LangGraph](https://github.com/langchain-ai/langgraph)**: Ihr innovativer Ansatz zur Multi-Agenten-Orchestrierung war maßgeblich für die Ermöglichung der ausgeklügelten Workflows von DeerFlow.
|
||||
|
||||
Diese Projekte veranschaulichen die transformative Kraft der Open-Source-Zusammenarbeit, und wir sind stolz darauf, auf ihren Grundlagen aufzubauen.
|
||||
|
||||
### Hauptmitwirkende
|
||||
Ein herzliches Dankeschön geht an die Hauptautoren von `DeerFlow`, deren Vision, Leidenschaft und Engagement dieses Projekt zum Leben erweckt haben:
|
||||
|
||||
- **[Daniel Walnut](https://github.com/hetaoBackend/)**
|
||||
- **[Henry Li](https://github.com/magiccube/)**
|
||||
|
||||
Ihr unerschütterliches Engagement und Fachwissen waren die treibende Kraft hinter dem Erfolg von DeerFlow. Wir fühlen uns geehrt, Sie an der Spitze dieser Reise zu haben.
|
||||
|
||||
## Star-Verlauf
|
||||
|
||||
[](https://star-history.com/#bytedance/deer-flow&Date)
|
||||
-629
@@ -1,629 +0,0 @@
|
||||
# 🦌 DeerFlow - 2.0
|
||||
|
||||
[English](./README.md) | [中文](./README_zh.md) | [日本語](./README_ja.md) | Français | [Русский](./README_ru.md)
|
||||
|
||||
[](./backend/pyproject.toml)
|
||||
[](./Makefile)
|
||||
[](./LICENSE)
|
||||
|
||||
<a href="https://trendshift.io/repositories/14699" target="_blank"><img src="https://trendshift.io/api/badge/repositories/14699" alt="bytedance%2Fdeer-flow | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
|
||||
> Le 28 février 2026, DeerFlow a décroché la 🏆 1re place sur GitHub Trending suite au lancement de la version 2. Un immense merci à notre incroyable communauté — c'est grâce à vous ! 💪🔥
|
||||
|
||||
DeerFlow (**D**eep **E**xploration and **E**fficient **R**esearch **Flow**) est un **super agent harness** open source qui orchestre des **sub-agents**, de la **mémoire** et des **sandboxes** pour accomplir pratiquement n'importe quelle tâche — le tout propulsé par des **skills extensibles**.
|
||||
|
||||
https://github.com/user-attachments/assets/a8bcadc4-e040-4cf2-8fda-dd768b999c18
|
||||
|
||||
> [!NOTE]
|
||||
> **DeerFlow 2.0 est une réécriture complète.** Il ne partage aucun code avec la v1. Si vous cherchez le framework Deep Research original, il est maintenu sur la [branche `1.x`](https://github.com/bytedance/deer-flow/tree/main-1.x) — les contributions y sont toujours les bienvenues. Le développement actif a migré vers la 2.0.
|
||||
|
||||
## Site officiel
|
||||
|
||||
[<img width="2880" height="1600" alt="image" src="https://github.com/user-attachments/assets/a598c49f-3b2f-41ea-a052-05e21349188a" />](https://deerflow.tech)
|
||||
|
||||
Découvrez-en plus et regardez des **démos réelles** sur notre [**site officiel**](https://deerflow.tech).
|
||||
|
||||
## Coding Plan de ByteDance Volcengine
|
||||
|
||||
<img width="4808" height="2400" alt="英文方舟" src="https://github.com/user-attachments/assets/2ecc7b9d-50be-4185-b1f7-5542d222fb2d" />
|
||||
|
||||
- Nous recommandons fortement d'utiliser Doubao-Seed-2.0-Code, DeepSeek v3.2 et Kimi 2.5 pour exécuter DeerFlow
|
||||
- [En savoir plus](https://www.byteplus.com/en/activity/codingplan?utm_campaign=deer_flow&utm_content=deer_flow&utm_medium=devrel&utm_source=OWO&utm_term=deer_flow)
|
||||
- [Développeurs en Chine continentale, cliquez ici](https://www.volcengine.com/activity/codingplan?utm_campaign=deer_flow&utm_content=deer_flow&utm_medium=devrel&utm_source=OWO&utm_term=deer_flow)
|
||||
|
||||
## InfoQuest
|
||||
|
||||
DeerFlow intègre désormais le toolkit de recherche et de crawling intelligent développé par BytePlus — [InfoQuest (essai gratuit en ligne)](https://docs.byteplus.com/en/docs/InfoQuest/What_is_Info_Quest)
|
||||
|
||||
<a href="https://docs.byteplus.com/en/docs/InfoQuest/What_is_Info_Quest" target="_blank">
|
||||
<img
|
||||
src="https://sf16-sg.tiktokcdn.com/obj/eden-sg/hubseh7bsbps/20251208-160108.png" alt="InfoQuest_banner"
|
||||
/>
|
||||
</a>
|
||||
|
||||
---
|
||||
|
||||
## Table des matières
|
||||
|
||||
- [🦌 DeerFlow - 2.0](#-deerflow---20)
|
||||
- [Site officiel](#site-officiel)
|
||||
- [InfoQuest](#infoquest)
|
||||
- [Table des matières](#table-des-matières)
|
||||
- [Installation en une phrase pour un coding agent](#installation-en-une-phrase-pour-un-coding-agent)
|
||||
- [Démarrage rapide](#démarrage-rapide)
|
||||
- [Configuration](#configuration)
|
||||
- [Lancer l'application](#lancer-lapplication)
|
||||
- [Option 1 : Docker (recommandé)](#option-1--docker-recommandé)
|
||||
- [Option 2 : Développement local](#option-2--développement-local)
|
||||
- [Avancé](#avancé)
|
||||
- [Mode Sandbox](#mode-sandbox)
|
||||
- [Serveur MCP](#serveur-mcp)
|
||||
- [Canaux de messagerie](#canaux-de-messagerie)
|
||||
- [Traçage LangSmith](#traçage-langsmith)
|
||||
- [Du Deep Research au Super Agent Harness](#du-deep-research-au-super-agent-harness)
|
||||
- [Fonctionnalités principales](#fonctionnalités-principales)
|
||||
- [Skills et outils](#skills-et-outils)
|
||||
- [Intégration Claude Code](#intégration-claude-code)
|
||||
- [Sub-Agents](#sub-agents)
|
||||
- [Sandbox et système de fichiers](#sandbox-et-système-de-fichiers)
|
||||
- [Context Engineering](#context-engineering)
|
||||
- [Mémoire à long terme](#mémoire-à-long-terme)
|
||||
- [Modèles recommandés](#modèles-recommandés)
|
||||
- [Client Python intégré](#client-python-intégré)
|
||||
- [Documentation](#documentation)
|
||||
- [⚠️ Avertissement de sécurité](#️-avertissement-de-sécurité)
|
||||
- [Contribuer](#contribuer)
|
||||
- [Licence](#licence)
|
||||
- [Remerciements](#remerciements)
|
||||
- [Contributeurs principaux](#contributeurs-principaux)
|
||||
- [Star History](#star-history)
|
||||
|
||||
## Installation en une phrase pour un coding agent
|
||||
|
||||
Si vous utilisez Claude Code, Codex, Cursor, Windsurf ou un autre coding agent, vous pouvez simplement lui envoyer cette phrase :
|
||||
|
||||
```text
|
||||
Aide-moi à cloner DeerFlow si nécessaire, puis à initialiser son environnement de développement local en suivant https://raw.githubusercontent.com/bytedance/deer-flow/main/Install.md
|
||||
```
|
||||
|
||||
Ce prompt est destiné aux coding agents. Il leur demande de cloner le dépôt si nécessaire, de privilégier Docker quand il est disponible, puis de s'arrêter avec la commande exacte pour lancer DeerFlow et la liste des configurations encore manquantes.
|
||||
|
||||
## Démarrage rapide
|
||||
|
||||
### Configuration
|
||||
|
||||
1. **Cloner le dépôt DeerFlow**
|
||||
|
||||
```bash
|
||||
git clone https://github.com/bytedance/deer-flow.git
|
||||
cd deer-flow
|
||||
```
|
||||
|
||||
2. **Générer les fichiers de configuration locaux**
|
||||
|
||||
Depuis le répertoire racine du projet (`deer-flow/`), exécutez :
|
||||
|
||||
```bash
|
||||
make config
|
||||
```
|
||||
|
||||
Cette commande crée les fichiers de configuration locaux à partir des templates fournis.
|
||||
|
||||
3. **Configurer le(s) modèle(s) de votre choix**
|
||||
|
||||
Éditez `config.yaml` et définissez au moins un modèle :
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- name: gpt-4 # Internal identifier
|
||||
display_name: GPT-4 # Human-readable name
|
||||
use: langchain_openai:ChatOpenAI # LangChain class path
|
||||
model: gpt-4 # Model identifier for API
|
||||
api_key: $OPENAI_API_KEY # API key (recommended: use env var)
|
||||
max_tokens: 4096 # Maximum tokens per request
|
||||
temperature: 0.7 # Sampling temperature
|
||||
|
||||
- name: openrouter-gemini-2.5-flash
|
||||
display_name: Gemini 2.5 Flash (OpenRouter)
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: google/gemini-2.5-flash-preview
|
||||
api_key: $OPENAI_API_KEY # OpenRouter still uses the OpenAI-compatible field name here
|
||||
base_url: https://openrouter.ai/api/v1
|
||||
|
||||
- name: gpt-5-responses
|
||||
display_name: GPT-5 (Responses API)
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: gpt-5
|
||||
api_key: $OPENAI_API_KEY
|
||||
use_responses_api: true
|
||||
output_version: responses/v1
|
||||
```
|
||||
|
||||
OpenRouter et les passerelles compatibles OpenAI similaires doivent être configurés avec `langchain_openai:ChatOpenAI` et `base_url`. Si vous préférez utiliser un nom de variable d'environnement propre au fournisseur, pointez `api_key` vers cette variable explicitement (par exemple `api_key: $OPENROUTER_API_KEY`).
|
||||
|
||||
Pour router les modèles OpenAI via `/v1/responses`, continuez d'utiliser `langchain_openai:ChatOpenAI` et définissez `use_responses_api: true` avec `output_version: responses/v1`.
|
||||
|
||||
Exemples de providers basés sur un CLI :
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- name: gpt-5.4
|
||||
display_name: GPT-5.4 (Codex CLI)
|
||||
use: deerflow.models.openai_codex_provider:CodexChatModel
|
||||
model: gpt-5.4
|
||||
supports_thinking: true
|
||||
supports_reasoning_effort: true
|
||||
|
||||
- name: claude-sonnet-4.6
|
||||
display_name: Claude Sonnet 4.6 (Claude Code OAuth)
|
||||
use: deerflow.models.claude_provider:ClaudeChatModel
|
||||
model: claude-sonnet-4-6
|
||||
max_tokens: 4096
|
||||
supports_thinking: true
|
||||
```
|
||||
|
||||
- Codex CLI lit `~/.codex/auth.json`
|
||||
- L'endpoint Responses de Codex rejette actuellement `max_tokens` et `max_output_tokens`, donc `CodexChatModel` n'expose pas de limite de tokens par requête
|
||||
- Claude Code accepte `CLAUDE_CODE_OAUTH_TOKEN`, `ANTHROPIC_AUTH_TOKEN`, `CLAUDE_CODE_OAUTH_TOKEN_FILE_DESCRIPTOR`, `CLAUDE_CODE_CREDENTIALS_PATH`, ou en clair `~/.claude/.credentials.json`
|
||||
- Sur macOS, DeerFlow ne sonde pas le Keychain automatiquement. Exportez l'auth Claude Code explicitement si nécessaire :
|
||||
|
||||
```bash
|
||||
eval "$(python3 scripts/export_claude_code_oauth.py --print-export)"
|
||||
```
|
||||
|
||||
4. **Définir les clés API pour le(s) modèle(s) configuré(s)**
|
||||
|
||||
Choisissez l'une des méthodes suivantes :
|
||||
|
||||
- Option A : Éditer le fichier `.env` à la racine du projet (recommandé)
|
||||
|
||||
|
||||
```bash
|
||||
TAVILY_API_KEY=your-tavily-api-key
|
||||
OPENAI_API_KEY=your-openai-api-key
|
||||
# OpenRouter also uses OPENAI_API_KEY when your config uses langchain_openai:ChatOpenAI + base_url.
|
||||
# Add other provider keys as needed
|
||||
INFOQUEST_API_KEY=your-infoquest-api-key
|
||||
```
|
||||
|
||||
- Option B : Exporter les variables d'environnement dans votre shell
|
||||
|
||||
```bash
|
||||
export OPENAI_API_KEY=your-openai-api-key
|
||||
```
|
||||
|
||||
Pour les providers basés sur un CLI :
|
||||
- Codex CLI : `~/.codex/auth.json`
|
||||
- Claude Code OAuth : handoff explicite via env/fichier ou `~/.claude/.credentials.json`
|
||||
|
||||
- Option C : Éditer `config.yaml` directement (non recommandé en production)
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- name: gpt-4
|
||||
api_key: your-actual-api-key-here # Replace placeholder
|
||||
```
|
||||
|
||||
### Lancer l'application
|
||||
|
||||
#### Option 1 : Docker (recommandé)
|
||||
|
||||
**Développement** (hot-reload, montage des sources) :
|
||||
|
||||
```bash
|
||||
make docker-init # Pull sandbox image (only once or when image updates)
|
||||
make docker-start # Start services (auto-detects sandbox mode from config.yaml)
|
||||
```
|
||||
|
||||
`make docker-start` ne lance `provisioner` que si `config.yaml` utilise le mode provisioner (`sandbox.use: deerflow.community.aio_sandbox:AioSandboxProvider` avec `provisioner_url`).
|
||||
Les processus backend récupèrent automatiquement les changements dans `config.yaml` au prochain accès à la configuration, donc les mises à jour de métadonnées des modèles ne nécessitent pas de redémarrage manuel en développement.
|
||||
|
||||
> [!TIP]
|
||||
> Sous Linux, si les commandes Docker échouent avec `permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock`, ajoutez votre utilisateur au groupe `docker` et reconnectez-vous avant de réessayer. Voir [CONTRIBUTING.md](CONTRIBUTING.md#linux-docker-daemon-permission-denied) pour la solution complète.
|
||||
|
||||
**Production** (build des images en local, montage de la config et des données) :
|
||||
|
||||
```bash
|
||||
make up # Build images and start all production services
|
||||
make down # Stop and remove containers
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> Le runtime d'agent s'exécute actuellement dans la Gateway. nginx réécrit `/api/langgraph/*` vers l'API compatible LangGraph servie par la Gateway.
|
||||
|
||||
Accès : http://localhost:2026
|
||||
|
||||
Voir [CONTRIBUTING.md](CONTRIBUTING.md) pour le guide complet de développement avec Docker.
|
||||
|
||||
#### Option 2 : Développement local
|
||||
|
||||
Si vous préférez lancer les services en local :
|
||||
|
||||
Prérequis : complétez d'abord les étapes de « Configuration » ci-dessus (`make config` et clés API des modèles). `make dev` nécessite un fichier de configuration valide (par défaut `config.yaml` à la racine du projet ; modifiable via `DEER_FLOW_CONFIG_PATH`).
|
||||
|
||||
1. **Vérifier les prérequis** :
|
||||
```bash
|
||||
make check # Verifies Node.js 22+, pnpm, uv, nginx
|
||||
```
|
||||
|
||||
2. **Installer les dépendances** :
|
||||
```bash
|
||||
make install # Install backend + frontend dependencies
|
||||
```
|
||||
|
||||
3. **(Optionnel) Pré-télécharger l'image sandbox** :
|
||||
```bash
|
||||
# Recommended if using Docker/Container-based sandbox
|
||||
make setup-sandbox
|
||||
```
|
||||
|
||||
4. **Démarrer les services** :
|
||||
```bash
|
||||
make dev
|
||||
```
|
||||
|
||||
5. **Accès** : http://localhost:2026
|
||||
|
||||
### Avancé
|
||||
#### Mode Sandbox
|
||||
|
||||
DeerFlow supporte plusieurs modes d'exécution sandbox :
|
||||
- **Exécution locale** (exécute le code sandbox directement sur la machine hôte)
|
||||
- **Exécution Docker** (exécute le code sandbox dans des conteneurs Docker isolés)
|
||||
- **Exécution Docker avec Kubernetes** (exécute le code sandbox dans des pods Kubernetes via le service provisioner)
|
||||
|
||||
En développement Docker, le démarrage des services suit le mode sandbox défini dans `config.yaml`. En mode Local/Docker, `provisioner` n'est pas démarré.
|
||||
|
||||
Voir le [Guide de configuration Sandbox](backend/docs/CONFIGURATION.md#sandbox) pour configurer le mode de votre choix.
|
||||
|
||||
#### Serveur MCP
|
||||
|
||||
DeerFlow supporte des serveurs MCP et des skills configurables pour étendre ses capacités.
|
||||
Pour les serveurs MCP HTTP/SSE, les flux de tokens OAuth sont supportés (`client_credentials`, `refresh_token`).
|
||||
Voir le [Guide MCP Server](backend/docs/MCP_SERVER.md) pour les instructions détaillées.
|
||||
|
||||
#### Canaux de messagerie
|
||||
|
||||
DeerFlow peut recevoir des tâches depuis des applications de messagerie. Les canaux démarrent automatiquement une fois configurés — aucune IP publique n'est requise.
|
||||
|
||||
| Canal | Transport | Difficulté |
|
||||
|---------|-----------|------------|
|
||||
| Telegram | Bot API (long-polling) | Facile |
|
||||
| Slack | Socket Mode | Modérée |
|
||||
| Feishu / Lark | WebSocket | Modérée |
|
||||
| DingTalk | Stream Push (WebSocket) | Modérée |
|
||||
|
||||
**Configuration dans `config.yaml` :**
|
||||
|
||||
```yaml
|
||||
channels:
|
||||
# LangGraph-compatible Gateway API base URL (default: http://localhost:8001/api)
|
||||
langgraph_url: http://localhost:8001/api
|
||||
# Gateway API URL (default: http://localhost:8001)
|
||||
gateway_url: http://localhost:8001
|
||||
|
||||
# Optional: global session defaults for all mobile channels
|
||||
session:
|
||||
assistant_id: lead_agent
|
||||
config:
|
||||
recursion_limit: 100
|
||||
context:
|
||||
thinking_enabled: true
|
||||
is_plan_mode: false
|
||||
subagent_enabled: false
|
||||
|
||||
feishu:
|
||||
enabled: true
|
||||
app_id: $FEISHU_APP_ID
|
||||
app_secret: $FEISHU_APP_SECRET
|
||||
# domain: https://open.feishu.cn # China (default)
|
||||
# domain: https://open.larksuite.com # International
|
||||
|
||||
slack:
|
||||
enabled: true
|
||||
bot_token: $SLACK_BOT_TOKEN # xoxb-...
|
||||
app_token: $SLACK_APP_TOKEN # xapp-... (Socket Mode)
|
||||
allowed_users: [] # empty = allow all
|
||||
|
||||
telegram:
|
||||
enabled: true
|
||||
bot_token: $TELEGRAM_BOT_TOKEN
|
||||
allowed_users: [] # empty = allow all
|
||||
|
||||
# Optional: per-channel / per-user session settings
|
||||
session:
|
||||
assistant_id: mobile_agent
|
||||
context:
|
||||
thinking_enabled: false
|
||||
users:
|
||||
"123456789":
|
||||
assistant_id: vip_agent
|
||||
config:
|
||||
recursion_limit: 150
|
||||
context:
|
||||
thinking_enabled: true
|
||||
subagent_enabled: true
|
||||
|
||||
dingtalk:
|
||||
enabled: true
|
||||
client_id: $DINGTALK_CLIENT_ID # ClientId depuis DingTalk Open Platform
|
||||
client_secret: $DINGTALK_CLIENT_SECRET # ClientSecret depuis DingTalk Open Platform
|
||||
allowed_users: [] # vide = tout le monde autorisé
|
||||
card_template_id: "" # Optionnel : ID de modèle AI Card pour l'effet machine à écrire en streaming
|
||||
```
|
||||
|
||||
Définissez les clés API correspondantes dans votre fichier `.env` :
|
||||
|
||||
```bash
|
||||
# Telegram
|
||||
TELEGRAM_BOT_TOKEN=123456789:ABCdefGHIjklMNOpqrSTUvwxYZ
|
||||
|
||||
# Slack
|
||||
SLACK_BOT_TOKEN=xoxb-...
|
||||
SLACK_APP_TOKEN=xapp-...
|
||||
|
||||
# Feishu / Lark
|
||||
FEISHU_APP_ID=cli_xxxx
|
||||
FEISHU_APP_SECRET=your_app_secret
|
||||
|
||||
# DingTalk
|
||||
DINGTALK_CLIENT_ID=your_client_id
|
||||
DINGTALK_CLIENT_SECRET=your_client_secret
|
||||
```
|
||||
|
||||
**Configuration Telegram**
|
||||
|
||||
1. Ouvrez une conversation avec [@BotFather](https://t.me/BotFather), envoyez `/newbot`, et copiez le token HTTP API.
|
||||
2. Définissez `TELEGRAM_BOT_TOKEN` dans `.env` et activez le canal dans `config.yaml`.
|
||||
|
||||
**Configuration Slack**
|
||||
|
||||
1. Créez une Slack App sur [api.slack.com/apps](https://api.slack.com/apps) → Create New App → From scratch.
|
||||
2. Dans **OAuth & Permissions**, ajoutez les Bot Token Scopes : `app_mentions:read`, `chat:write`, `im:history`, `im:read`, `im:write`, `files:write`.
|
||||
3. Activez le **Socket Mode** → générez un App-Level Token (`xapp-…`) avec le scope `connections:write`.
|
||||
4. Dans **Event Subscriptions**, abonnez-vous aux bot events : `app_mention`, `message.im`.
|
||||
5. Définissez `SLACK_BOT_TOKEN` et `SLACK_APP_TOKEN` dans `.env` et activez le canal dans `config.yaml`.
|
||||
|
||||
**Configuration Feishu / Lark**
|
||||
|
||||
1. Créez une application sur [Feishu Open Platform](https://open.feishu.cn/) → activez la capacité **Bot**.
|
||||
2. Ajoutez les permissions : `im:message`, `im:message.p2p_msg:readonly`, `im:resource`.
|
||||
3. Dans **Events**, abonnez-vous à `im.message.receive_v1` et sélectionnez le mode **Long Connection**.
|
||||
4. Copiez l'App ID et l'App Secret. Définissez `FEISHU_APP_ID` et `FEISHU_APP_SECRET` dans `.env` et activez le canal dans `config.yaml`.
|
||||
|
||||
**Configuration DingTalk**
|
||||
|
||||
1. Créez une application sur [DingTalk Open Platform](https://open.dingtalk.com/) et activez la capacité **Robot**.
|
||||
2. Dans la page de configuration du robot, définissez le mode de réception des messages sur **Stream**.
|
||||
3. Copiez le `Client ID` et le `Client Secret`. Définissez `DINGTALK_CLIENT_ID` et `DINGTALK_CLIENT_SECRET` dans `.env` et activez le canal dans `config.yaml`.
|
||||
4. *(Optionnel)* Pour activer les réponses en streaming AI Card (effet machine à écrire), créez un modèle **AI Card** sur la [plateforme de cartes DingTalk](https://open.dingtalk.com/document/dingstart/typewriter-effect-streaming-ai-card), puis définissez `card_template_id` dans `config.yaml` avec l'ID du modèle. Vous devez également demander les permissions `Card.Streaming.Write` et `Card.Instance.Write`.
|
||||
|
||||
**Commandes**
|
||||
|
||||
Une fois un canal connecté, vous pouvez interagir avec DeerFlow directement depuis le chat :
|
||||
|
||||
| Commande | Description |
|
||||
|---------|-------------|
|
||||
| `/new` | Démarrer une nouvelle conversation |
|
||||
| `/status` | Afficher les infos du thread en cours |
|
||||
| `/models` | Lister les modèles disponibles |
|
||||
| `/memory` | Consulter la mémoire |
|
||||
| `/help` | Afficher l'aide |
|
||||
|
||||
> Les messages sans préfixe de commande sont traités comme du chat classique — DeerFlow crée un thread et répond de manière conversationnelle.
|
||||
|
||||
#### Traçage LangSmith
|
||||
|
||||
DeerFlow intègre nativement [LangSmith](https://smith.langchain.com) pour l'observabilité. Une fois activé, tous les appels LLM, les exécutions d'agents et les exécutions d'outils sont tracés et visibles dans le tableau de bord LangSmith.
|
||||
|
||||
Ajoutez les lignes suivantes à votre fichier `.env` :
|
||||
|
||||
```bash
|
||||
LANGSMITH_TRACING=true
|
||||
LANGSMITH_ENDPOINT=https://api.smith.langchain.com
|
||||
LANGSMITH_API_KEY=lsv2_pt_xxxxxxxxxxxxxxxx
|
||||
LANGSMITH_PROJECT=xxx
|
||||
```
|
||||
|
||||
Pour les déploiements Docker, le traçage est désactivé par défaut. Définissez `LANGSMITH_TRACING=true` et `LANGSMITH_API_KEY` dans votre `.env` pour l'activer.
|
||||
|
||||
## Du Deep Research au Super Agent Harness
|
||||
|
||||
DeerFlow a démarré comme un framework de Deep Research — et la communauté s'en est emparée. Depuis le lancement, les développeurs l'ont poussé bien au-delà de la recherche : construction de pipelines de données, génération de présentations, mise en place de dashboards, automatisation de workflows de contenu. Des usages qu'on n'avait jamais anticipés.
|
||||
|
||||
Ça nous a révélé quelque chose d'important : DeerFlow n'était pas qu'un simple outil de recherche. C'était un **harness** — un runtime qui donne aux agents l'infrastructure nécessaire pour vraiment accomplir du travail.
|
||||
|
||||
On l'a donc reconstruit de zéro.
|
||||
|
||||
DeerFlow 2.0 n'est plus un framework à assembler soi-même. C'est un super agent harness — clé en main et entièrement extensible. Construit sur LangGraph et LangChain, il embarque tout ce dont un agent a besoin out of the box : un système de fichiers, de la mémoire, des skills, une exécution sandboxée, et la capacité de planifier et de lancer des sub-agents pour les tâches complexes et multi-étapes.
|
||||
|
||||
Utilisez-le tel quel. Ou démontez-le et faites-en le vôtre.
|
||||
|
||||
## Fonctionnalités principales
|
||||
|
||||
### Skills et outils
|
||||
|
||||
Les skills sont ce qui permet à DeerFlow de faire *pratiquement n'importe quoi*.
|
||||
|
||||
Un Agent Skill standard est un module de capacité structuré — un fichier Markdown qui définit un workflow, des bonnes pratiques et des références vers des ressources associées. DeerFlow est livré avec des skills intégrés pour la recherche, la génération de rapports, la création de présentations, les pages web, la génération d'images et de vidéos, et bien plus. Mais la vraie force réside dans l'extensibilité : ajoutez vos propres skills, remplacez ceux fournis, ou combinez-les en workflows composites.
|
||||
|
||||
Les skills sont chargés progressivement — uniquement quand la tâche le nécessite, pas tous en même temps. Ça permet de garder la fenêtre de contexte légère et de bien fonctionner même avec des modèles sensibles au nombre de tokens.
|
||||
|
||||
Quand vous installez des archives `.skill` via le Gateway, DeerFlow accepte les métadonnées frontmatter optionnelles standard comme `version`, `author` et `compatibility`, plutôt que de rejeter des skills externes par ailleurs valides.
|
||||
|
||||
Les outils suivent la même philosophie. DeerFlow est livré avec un ensemble d'outils de base — recherche web, fetch de pages web, opérations sur les fichiers, exécution bash — et supporte les outils custom via des serveurs MCP et des fonctions Python. Remplacez n'importe quoi. Ajoutez n'importe quoi.
|
||||
|
||||
Les suggestions de suivi générées par le Gateway normalisent désormais aussi bien la sortie texte brut du modèle que le contenu riche au format bloc/liste avant de parser la réponse en tableau JSON, de sorte que les wrappers de contenu propres à chaque provider ne suppriment plus silencieusement les suggestions.
|
||||
|
||||
```
|
||||
# Paths inside the sandbox container
|
||||
/mnt/skills/public
|
||||
├── research/SKILL.md
|
||||
├── report-generation/SKILL.md
|
||||
├── slide-creation/SKILL.md
|
||||
├── web-page/SKILL.md
|
||||
└── image-generation/SKILL.md
|
||||
|
||||
/mnt/skills/custom
|
||||
└── your-custom-skill/SKILL.md ← yours
|
||||
```
|
||||
|
||||
#### Intégration Claude Code
|
||||
|
||||
Le skill `claude-to-deerflow` vous permet d'interagir avec une instance DeerFlow en cours d'exécution directement depuis [Claude Code](https://docs.anthropic.com/en/docs/claude-code). Envoyez des tâches de recherche, vérifiez le statut, gérez les threads — le tout sans quitter le terminal.
|
||||
|
||||
**Installer le skill** :
|
||||
|
||||
```bash
|
||||
npx skills add https://github.com/bytedance/deer-flow --skill claude-to-deerflow
|
||||
```
|
||||
|
||||
Assurez-vous ensuite que DeerFlow tourne (par défaut sur `http://localhost:2026`) et utilisez la commande `/claude-to-deerflow` dans Claude Code.
|
||||
|
||||
**Ce que vous pouvez faire** :
|
||||
- Envoyer des messages à DeerFlow et recevoir des réponses en streaming
|
||||
- Choisir le mode d'exécution : flash (rapide), standard, pro (planification), ultra (sub-agents)
|
||||
- Vérifier la santé de DeerFlow, lister les modèles/skills/agents
|
||||
- Gérer les threads et l'historique des conversations
|
||||
- Upload des fichiers pour analyse
|
||||
|
||||
**Variables d'environnement** (optionnel, pour des endpoints custom) :
|
||||
|
||||
```bash
|
||||
DEERFLOW_URL=http://localhost:2026 # Unified proxy base URL
|
||||
DEERFLOW_GATEWAY_URL=http://localhost:2026 # Gateway API
|
||||
DEERFLOW_LANGGRAPH_URL=http://localhost:2026/api/langgraph # LangGraph API
|
||||
```
|
||||
|
||||
Voir [`skills/public/claude-to-deerflow/SKILL.md`](skills/public/claude-to-deerflow/SKILL.md) pour la référence API complète.
|
||||
|
||||
### Sub-Agents
|
||||
|
||||
Les tâches complexes tiennent rarement en une seule passe. DeerFlow les décompose.
|
||||
|
||||
L'agent principal peut lancer des sub-agents à la volée — chacun avec son propre contexte délimité, ses outils et ses conditions d'arrêt. Les sub-agents s'exécutent en parallèle quand c'est possible, remontent des résultats structurés, et l'agent principal synthétise le tout en une sortie cohérente.
|
||||
|
||||
C'est comme ça que DeerFlow gère les tâches qui prennent de quelques minutes à plusieurs heures : une tâche de recherche peut se déployer en une dizaine de sub-agents, chacun explorant un angle différent, puis converger vers un seul rapport — ou un site web — ou un jeu de slides avec des visuels générés. Un seul harness, de nombreuses mains.
|
||||
|
||||
### Sandbox et système de fichiers
|
||||
|
||||
DeerFlow ne se contente pas de *parler* de faire les choses. Il dispose de son propre ordinateur.
|
||||
|
||||
Chaque tâche s'exécute dans un conteneur Docker isolé avec un système de fichiers complet — skills, workspace, uploads, outputs. L'agent lit, écrit et édite des fichiers. Il exécute des commandes bash et du code. Il visualise des images. Le tout sandboxé, le tout auditable, zéro contamination entre les sessions.
|
||||
|
||||
C'est la différence entre un chatbot avec accès à des outils et un agent doté d'un véritable environnement d'exécution.
|
||||
|
||||
```
|
||||
# Paths inside the sandbox container
|
||||
/mnt/user-data/
|
||||
├── uploads/ ← your files
|
||||
├── workspace/ ← agents' working directory
|
||||
└── outputs/ ← final deliverables
|
||||
```
|
||||
|
||||
### Context Engineering
|
||||
|
||||
**Contexte isolé des Sub-Agents** : chaque sub-agent s'exécute dans son propre contexte isolé. Il ne peut voir ni le contexte de l'agent principal, ni celui des autres sub-agents. L'objectif est de garantir que chaque sub-agent reste concentré sur sa tâche sans être parasité par des informations non pertinentes.
|
||||
|
||||
**Résumé** : au sein d'une session, DeerFlow gère le contexte de manière agressive — en résumant les sous-tâches terminées, en déchargeant les résultats intermédiaires vers le système de fichiers, en compressant ce qui n'est plus immédiatement pertinent. Ça lui permet de rester efficace sur des tâches longues et multi-étapes sans faire exploser la fenêtre de contexte.
|
||||
|
||||
### Mémoire à long terme
|
||||
|
||||
La plupart des agents oublient tout dès qu'une conversation se termine. DeerFlow, lui, se souvient.
|
||||
|
||||
D'une session à l'autre, DeerFlow construit une mémoire persistante de votre profil, de vos préférences et de vos connaissances accumulées. Plus vous l'utilisez, mieux il vous connaît — votre style d'écriture, votre stack technique, vos workflows récurrents. La mémoire est stockée localement et reste sous votre contrôle.
|
||||
|
||||
Les mises à jour de la mémoire ignorent désormais les entrées de faits en double au moment de l'application, de sorte que les préférences et le contexte répétés ne s'accumulent plus indéfiniment entre les sessions.
|
||||
|
||||
## Modèles recommandés
|
||||
|
||||
DeerFlow est agnostique en termes de modèle — il fonctionne avec n'importe quel LLM implémentant l'API compatible OpenAI. Cela dit, il offre de meilleures performances avec des modèles qui supportent :
|
||||
|
||||
- **De longues fenêtres de contexte** (100k+ tokens) pour la recherche approfondie et les tâches multi-étapes
|
||||
- **Des capacités de raisonnement** pour la planification adaptative et la décomposition de tâches complexes
|
||||
- **Des entrées multimodales** pour la compréhension d'images et de vidéos
|
||||
- **Un usage fiable des outils (tool use)** pour des appels de fonctions et des sorties structurées fiables
|
||||
|
||||
## Client Python intégré
|
||||
|
||||
DeerFlow peut être utilisé comme bibliothèque Python intégrée sans lancer l'ensemble des services HTTP. Le `DeerFlowClient` fournit un accès direct in-process à toutes les capacités d'agent et de Gateway, en retournant les mêmes schémas de réponse que l'API HTTP Gateway. Le HTTP Gateway expose également `DELETE /api/threads/{thread_id}` pour supprimer les données de thread locales gérées par DeerFlow après la suppression du thread LangGraph :
|
||||
|
||||
```python
|
||||
from deerflow.client import DeerFlowClient
|
||||
|
||||
client = DeerFlowClient()
|
||||
|
||||
# Chat
|
||||
response = client.chat("Analyze this paper for me", thread_id="my-thread")
|
||||
|
||||
# Streaming (LangGraph SSE protocol: values, messages-tuple, end)
|
||||
for event in client.stream("hello"):
|
||||
if event.type == "messages-tuple" and event.data.get("type") == "ai":
|
||||
print(event.data["content"])
|
||||
|
||||
# Configuration & management — returns Gateway-aligned dicts
|
||||
models = client.list_models() # {"models": [...]}
|
||||
skills = client.list_skills() # {"skills": [...]}
|
||||
client.update_skill("web-search", enabled=True)
|
||||
client.upload_files("thread-1", ["./report.pdf"]) # {"success": True, "files": [...]}
|
||||
```
|
||||
|
||||
Toutes les méthodes retournant des dicts sont validées en CI contre les modèles de réponse Pydantic du Gateway (`TestGatewayConformance`), garantissant que le client intégré reste synchronisé avec les schémas de l'API HTTP. Voir `backend/packages/harness/deerflow/client.py` pour la documentation API complète.
|
||||
|
||||
## Documentation
|
||||
|
||||
- [Guide de contribution](CONTRIBUTING.md) - Mise en place de l'environnement de développement et workflow
|
||||
- [Guide de configuration](backend/docs/CONFIGURATION.md) - Instructions d'installation et de configuration
|
||||
- [Vue d'ensemble de l'architecture](backend/CLAUDE.md) - Détails de l'architecture technique
|
||||
- [Architecture backend](backend/README.md) - Architecture backend et référence API
|
||||
|
||||
## ⚠️ Avertissement de sécurité
|
||||
|
||||
### Un déploiement inapproprié peut introduire des risques de sécurité
|
||||
|
||||
DeerFlow dispose de capacités clés à hauts privilèges, notamment **l'exécution de commandes système, les opérations sur les ressources et l'invocation de logique métier**. Il est conçu par défaut pour être **déployé dans un environnement local de confiance (accessible uniquement via l'interface de loopback 127.0.0.1)**. Si vous déployez l'agent dans des environnements non fiables — tels que des réseaux LAN, des serveurs cloud publics ou d'autres environnements accessibles depuis plusieurs terminaux — sans mesures de sécurité strictes, cela peut introduire des risques, notamment :
|
||||
|
||||
- **Invocation non autorisée** : les fonctionnalités de l'agent pourraient être découvertes par des tiers non autorisés ou des scanners malveillants, déclenchant des requêtes non autorisées en masse qui exécutent des opérations à haut risque (commandes système, lecture/écriture de fichiers), pouvant causer de graves conséquences.
|
||||
- **Risques juridiques et de conformité** : si l'agent est utilisé illégalement pour mener des cyberattaques, du vol de données ou d'autres activités illicites, cela peut entraîner des responsabilités juridiques et des risques de conformité.
|
||||
|
||||
### Recommandations de sécurité
|
||||
|
||||
**Note : nous recommandons fortement de déployer DeerFlow dans un environnement réseau local de confiance.** Si vous avez besoin d'un déploiement multi-appareils ou multi-réseaux, vous devez mettre en place des mesures de sécurité strictes, par exemple :
|
||||
|
||||
- **Liste blanche d'IP** : utilisez `iptables`, ou déployez des pare-feux matériels / commutateurs avec ACL, pour **configurer des règles de liste blanche d'IP** et refuser l'accès à toutes les autres adresses IP.
|
||||
- **Passerelle d'authentification** : configurez un proxy inverse (ex. nginx) et **activez une authentification forte en amont**, bloquant tout accès non authentifié.
|
||||
- **Isolation réseau** : si possible, placez l'agent et les appareils de confiance dans le **même VLAN dédié**, isolé des autres équipements réseau.
|
||||
- **Restez informé** : continuez à suivre les mises à jour de sécurité du projet DeerFlow.
|
||||
|
||||
## Contribuer
|
||||
|
||||
Les contributions sont les bienvenues ! Consultez [CONTRIBUTING.md](CONTRIBUTING.md) pour la mise en place de l'environnement de développement, le workflow et les conventions.
|
||||
|
||||
La couverture de tests de régression inclut la détection du mode sandbox Docker et les tests de gestion du kubeconfig-path du provisioner dans `backend/tests/`.
|
||||
|
||||
## Licence
|
||||
|
||||
Ce projet est open source et disponible sous la [Licence MIT](./LICENSE).
|
||||
|
||||
## Remerciements
|
||||
|
||||
DeerFlow est construit sur le travail remarquable de la communauté open source. Nous sommes profondément reconnaissants envers tous les projets et contributeurs dont les efforts ont rendu DeerFlow possible. Nous nous tenons véritablement sur les épaules de géants.
|
||||
|
||||
Nous tenons à exprimer notre sincère gratitude aux projets suivants pour leurs contributions inestimables :
|
||||
|
||||
- **[LangChain](https://github.com/langchain-ai/langchain)** : leur excellent framework propulse nos interactions LLM et nos chaînes, permettant une intégration et des fonctionnalités fluides.
|
||||
- **[LangGraph](https://github.com/langchain-ai/langgraph)** : leur approche innovante de l'orchestration multi-agents a été déterminante pour les workflows sophistiqués de DeerFlow.
|
||||
|
||||
Ces projets illustrent le pouvoir transformateur de la collaboration open source, et nous sommes fiers de bâtir sur leurs fondations.
|
||||
|
||||
### Contributeurs principaux
|
||||
|
||||
Un grand merci aux auteurs principaux de `DeerFlow`, dont la vision, la passion et le dévouement ont donné vie à ce projet :
|
||||
|
||||
- **[Daniel Walnut](https://github.com/hetaoBackend/)**
|
||||
- **[Henry Li](https://github.com/magiccube/)**
|
||||
|
||||
Votre engagement sans faille et votre expertise sont le moteur du succès de DeerFlow. Nous sommes honorés de vous avoir à la barre de cette aventure.
|
||||
|
||||
## Star History
|
||||
|
||||
[](https://star-history.com/#bytedance/deer-flow&Date)
|
||||
+481
-510
File diff suppressed because it is too large
Load Diff
-505
@@ -1,505 +0,0 @@
|
||||
# 🦌 DeerFlow - 2.0
|
||||
|
||||
[English](./README.md) | [中文](./README_zh.md) | [日本語](./README_ja.md) | [Français](./README_fr.md) | Русский
|
||||
|
||||
[](./backend/pyproject.toml)
|
||||
[](./Makefile)
|
||||
[](./LICENSE)
|
||||
|
||||
<a href="https://trendshift.io/repositories/14699" target="_blank"><img src="https://trendshift.io/api/badge/repositories/14699" alt="bytedance%2Fdeer-flow | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
|
||||
|
||||
> 28 февраля 2026 года DeerFlow занял 🏆 #1 в GitHub Trending после релиза версии 2. Спасибо огромное нашему сообществу — всё благодаря вам! 💪🔥
|
||||
|
||||
DeerFlow (**D**eep **E**xploration and **E**fficient **R**esearch **Flow**) — open-source **Super Agent Harness**, который управляет **Sub-Agents**, **Memory** и **Sandbox** для решения почти любой задачи. Всё на основе расширяемых **Skills**.
|
||||
|
||||
https://github.com/user-attachments/assets/a8bcadc4-e040-4cf2-8fda-dd768b999c18
|
||||
|
||||
> [!NOTE]
|
||||
> **DeerFlow 2.0 — проект переписан с нуля.** Общего кода с v1 нет. Если нужен оригинальный Deep Research фреймворк — он живёт в ветке [`1.x`](https://github.com/bytedance/deer-flow/tree/main-1.x), туда тоже принимают контрибьюты. Активная разработка идёт в 2.0.
|
||||
|
||||
## Официальный сайт
|
||||
|
||||
[<img width="2880" height="1600" alt="image" src="https://github.com/user-attachments/assets/a598c49f-3b2f-41ea-a052-05e21349188a" />](https://deerflow.tech)
|
||||
|
||||
Больше информации и живые демо на [**официальном сайте**](https://deerflow.tech).
|
||||
|
||||
## Coding Plan от ByteDance Volcengine
|
||||
|
||||
<img width="4808" height="2400" alt="英文方舟" src="https://github.com/user-attachments/assets/2ecc7b9d-50be-4185-b1f7-5542d222fb2d" />
|
||||
|
||||
- Рекомендуем Doubao-Seed-2.0-Code, DeepSeek v3.2 и Kimi 2.5 для запуска DeerFlow
|
||||
- [Подробнее](https://www.byteplus.com/en/activity/codingplan?utm_campaign=deer_flow&utm_content=deer_flow&utm_medium=devrel&utm_source=OWO&utm_term=deer_flow)
|
||||
- [Для разработчиков из материкового Китая](https://www.volcengine.com/activity/codingplan?utm_campaign=deer_flow&utm_content=deer_flow&utm_medium=devrel&utm_source=OWO&utm_term=deer_flow)
|
||||
|
||||
## InfoQuest
|
||||
|
||||
DeerFlow интегрирован с инструментарием для умного поиска и краулинга от BytePlus — [InfoQuest (есть бесплатный онлайн-доступ)](https://docs.byteplus.com/en/docs/InfoQuest/What_is_Info_Quest)
|
||||
|
||||
<a href="https://docs.byteplus.com/en/docs/InfoQuest/What_is_Info_Quest" target="_blank">
|
||||
<img
|
||||
src="https://sf16-sg.tiktokcdn.com/obj/eden-sg/hubseh7bsbps/20251208-160108.png"
|
||||
alt="InfoQuest_banner"
|
||||
/>
|
||||
</a>
|
||||
|
||||
---
|
||||
|
||||
## Содержание
|
||||
|
||||
- [🦌 DeerFlow - 2.0](#-deerflow---20)
|
||||
- [Официальный сайт](#официальный-сайт)
|
||||
- [InfoQuest](#infoquest)
|
||||
- [Содержание](#содержание)
|
||||
- [Установка одной фразой для coding agent](#установка-одной-фразой-для-coding-agent)
|
||||
- [Быстрый старт](#быстрый-старт)
|
||||
- [Конфигурация](#конфигурация)
|
||||
- [Запуск](#запуск)
|
||||
- [Вариант 1: Docker (рекомендуется)](#вариант-1-docker-рекомендуется)
|
||||
- [Вариант 2: Локальная разработка](#вариант-2-локальная-разработка)
|
||||
- [Дополнительно](#дополнительно)
|
||||
- [Режим Sandbox](#режим-sandbox)
|
||||
- [MCP-сервер](#mcp-сервер)
|
||||
- [Мессенджеры](#мессенджеры)
|
||||
- [Трассировка LangSmith](#трассировка-langsmith)
|
||||
- [От Deep Research к Super Agent Harness](#от-deep-research-к-super-agent-harness)
|
||||
- [Core Features](#core-features)
|
||||
- [Skills & Tools](#skills--tools)
|
||||
- [Интеграция с Claude Code](#интеграция-с-claude-code)
|
||||
- [Sub-Agents](#sub-agents)
|
||||
- [Sandbox & файловая система](#sandbox--файловая-система)
|
||||
- [Context Engineering](#context-engineering)
|
||||
- [Long-Term Memory](#long-term-memory)
|
||||
- [Рекомендуемые модели](#рекомендуемые-модели)
|
||||
- [Встроенный Python-клиент](#встроенный-python-клиент)
|
||||
- [Документация](#документация)
|
||||
- [⚠️ Безопасность](#️-безопасность)
|
||||
- [Участие в разработке](#участие-в-разработке)
|
||||
- [Лицензия](#лицензия)
|
||||
- [Благодарности](#благодарности)
|
||||
- [Ключевые контрибьюторы](#ключевые-контрибьюторы)
|
||||
- [История звёзд](#история-звёзд)
|
||||
|
||||
## Установка одной фразой для coding agent
|
||||
|
||||
Если вы используете Claude Code, Codex, Cursor, Windsurf или другой coding agent, просто отправьте ему эту фразу:
|
||||
|
||||
```text
|
||||
Если DeerFlow еще не клонирован, сначала клонируй его, а затем подготовь локальное окружение разработки по инструкции https://raw.githubusercontent.com/bytedance/deer-flow/main/Install.md
|
||||
```
|
||||
|
||||
Этот prompt предназначен для coding agent. Он просит агента при необходимости сначала клонировать репозиторий, предпочесть Docker, если он доступен, и в конце вернуть точную команду запуска и список недостающих настроек.
|
||||
|
||||
## Быстрый старт
|
||||
|
||||
### Конфигурация
|
||||
|
||||
1. **Склонировать репозиторий DeerFlow**
|
||||
|
||||
```bash
|
||||
git clone https://github.com/bytedance/deer-flow.git
|
||||
cd deer-flow
|
||||
```
|
||||
|
||||
2. **Сгенерировать локальные конфиги**
|
||||
|
||||
Из корня проекта (`deer-flow/`) запустите:
|
||||
|
||||
```bash
|
||||
make config
|
||||
```
|
||||
|
||||
Команда создаёт локальные конфиги на основе шаблонов.
|
||||
|
||||
3. **Настроить модель**
|
||||
|
||||
Отредактируйте `config.yaml` и задайте хотя бы одну модель:
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- name: gpt-4 # Внутренний идентификатор
|
||||
display_name: GPT-4 # Отображаемое имя
|
||||
use: langchain_openai:ChatOpenAI # Путь к классу LangChain
|
||||
model: gpt-4 # Идентификатор модели для API
|
||||
api_key: $OPENAI_API_KEY # API-ключ (рекомендуется: переменная окружения)
|
||||
max_tokens: 4096 # Максимальное количество токенов на запрос
|
||||
temperature: 0.7 # Температура сэмплирования
|
||||
|
||||
- name: openrouter-gemini-2.5-flash
|
||||
display_name: Gemini 2.5 Flash (OpenRouter)
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: google/gemini-2.5-flash-preview
|
||||
api_key: $OPENAI_API_KEY
|
||||
base_url: https://openrouter.ai/api/v1
|
||||
|
||||
- name: gpt-5-responses
|
||||
display_name: GPT-5 (Responses API)
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: gpt-5
|
||||
api_key: $OPENAI_API_KEY
|
||||
use_responses_api: true
|
||||
output_version: responses/v1
|
||||
```
|
||||
|
||||
OpenRouter и аналогичные OpenAI-совместимые шлюзы настраиваются через `langchain_openai:ChatOpenAI` с параметром `base_url`. Для CLI-провайдеров:
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- name: gpt-5.4
|
||||
display_name: GPT-5.4 (Codex CLI)
|
||||
use: deerflow.models.openai_codex_provider:CodexChatModel
|
||||
model: gpt-5.4
|
||||
supports_thinking: true
|
||||
supports_reasoning_effort: true
|
||||
|
||||
- name: claude-sonnet-4.6
|
||||
display_name: Claude Sonnet 4.6 (Claude Code OAuth)
|
||||
use: deerflow.models.claude_provider:ClaudeChatModel
|
||||
model: claude-sonnet-4-6
|
||||
max_tokens: 4096
|
||||
supports_thinking: true
|
||||
```
|
||||
|
||||
- Codex CLI читает `~/.codex/auth.json`
|
||||
- Claude Code принимает `CLAUDE_CODE_OAUTH_TOKEN`, `ANTHROPIC_AUTH_TOKEN` или `~/.claude/.credentials.json`
|
||||
- На macOS при необходимости экспортируйте аутентификацию Claude Code явно:
|
||||
|
||||
```bash
|
||||
eval "$(python3 scripts/export_claude_code_oauth.py --print-export)"
|
||||
```
|
||||
|
||||
4. **Указать API-ключи**
|
||||
|
||||
- **Вариант А**: файл `.env` в корне проекта (рекомендуется)
|
||||
|
||||
```bash
|
||||
TAVILY_API_KEY=your-tavily-api-key
|
||||
OPENAI_API_KEY=your-openai-api-key
|
||||
INFOQUEST_API_KEY=your-infoquest-api-key
|
||||
```
|
||||
|
||||
- **Вариант Б**: переменные окружения в терминале
|
||||
|
||||
```bash
|
||||
export OPENAI_API_KEY=your-openai-api-key
|
||||
```
|
||||
|
||||
- **Вариант В**: напрямую в `config.yaml` (не рекомендуется для продакшена)
|
||||
|
||||
### Запуск
|
||||
|
||||
#### Вариант 1: Docker (рекомендуется)
|
||||
|
||||
**Разработка** (hot-reload, монтирование исходников):
|
||||
|
||||
```bash
|
||||
make docker-init # Загрузить образ Sandbox (один раз или при обновлении)
|
||||
make docker-start # Запустить сервисы
|
||||
```
|
||||
|
||||
**Продакшен** (собирает образы локально):
|
||||
|
||||
```bash
|
||||
make up # Собрать образы и запустить все сервисы
|
||||
make down # Остановить и удалить контейнеры
|
||||
```
|
||||
|
||||
> [!TIP]
|
||||
> На Linux при ошибке `permission denied` для Docker daemon добавьте пользователя в группу `docker` и перелогиньтесь. Подробнее в [CONTRIBUTING.md](CONTRIBUTING.md#linux-docker-daemon-permission-denied).
|
||||
|
||||
Адрес: http://localhost:2026
|
||||
|
||||
#### Вариант 2: Локальная разработка
|
||||
|
||||
1. **Проверить зависимости**:
|
||||
```bash
|
||||
make check # Проверяет Node.js 22+, pnpm, uv, nginx
|
||||
```
|
||||
|
||||
2. **Установить зависимости**:
|
||||
```bash
|
||||
make install
|
||||
```
|
||||
|
||||
3. **(Опционально) Загрузить образ Sandbox заранее**:
|
||||
```bash
|
||||
make setup-sandbox
|
||||
```
|
||||
|
||||
4. **Запустить сервисы**:
|
||||
```bash
|
||||
make dev
|
||||
```
|
||||
|
||||
5. **Адрес**: http://localhost:2026
|
||||
|
||||
### Дополнительно
|
||||
|
||||
#### Режим Sandbox
|
||||
|
||||
DeerFlow поддерживает несколько режимов выполнения:
|
||||
- **Локальное выполнение** — код запускается прямо на хосте
|
||||
- **Docker** — код выполняется в изолированных Docker-контейнерах
|
||||
- **Docker + Kubernetes** — выполнение в Kubernetes-подах через provisioner
|
||||
|
||||
Подробнее в [руководстве по конфигурации Sandbox](backend/docs/CONFIGURATION.md#sandbox).
|
||||
|
||||
#### MCP-сервер
|
||||
|
||||
DeerFlow поддерживает настраиваемые MCP-серверы для расширения возможностей. Для HTTP/SSE MCP-серверов поддерживаются OAuth-токены (`client_credentials`, `refresh_token`). Подробнее в [руководстве по MCP-серверу](backend/docs/MCP_SERVER.md).
|
||||
|
||||
#### Мессенджеры
|
||||
|
||||
DeerFlow принимает задачи прямо из мессенджеров. Каналы запускаются автоматически при настройке, публичный IP не нужен.
|
||||
|
||||
| Канал | Транспорт | Сложность |
|
||||
|-------|-----------|-----------|
|
||||
| Telegram | Bot API (long-polling) | Просто |
|
||||
| Slack | Socket Mode | Средне |
|
||||
| Feishu / Lark | WebSocket | Средне |
|
||||
| DingTalk | Stream Push (WebSocket) | Средне |
|
||||
|
||||
**Конфигурация в `config.yaml`:**
|
||||
|
||||
```yaml
|
||||
channels:
|
||||
feishu:
|
||||
enabled: true
|
||||
app_id: $FEISHU_APP_ID
|
||||
app_secret: $FEISHU_APP_SECRET
|
||||
# domain: https://open.feishu.cn # China (default)
|
||||
# domain: https://open.larksuite.com # International
|
||||
|
||||
slack:
|
||||
enabled: true
|
||||
bot_token: $SLACK_BOT_TOKEN
|
||||
app_token: $SLACK_APP_TOKEN
|
||||
allowed_users: []
|
||||
|
||||
telegram:
|
||||
enabled: true
|
||||
bot_token: $TELEGRAM_BOT_TOKEN
|
||||
allowed_users: []
|
||||
|
||||
dingtalk:
|
||||
enabled: true
|
||||
client_id: $DINGTALK_CLIENT_ID # ClientId с DingTalk Open Platform
|
||||
client_secret: $DINGTALK_CLIENT_SECRET # ClientSecret с DingTalk Open Platform
|
||||
allowed_users: [] # пусто = разрешить всем
|
||||
card_template_id: "" # Опционально: ID шаблона AI Card для потокового эффекта печатной машинки
|
||||
```
|
||||
|
||||
**Настройка Telegram**
|
||||
|
||||
1. Напишите [@BotFather](https://t.me/BotFather), отправьте `/newbot` и скопируйте HTTP API-токен.
|
||||
2. Укажите `TELEGRAM_BOT_TOKEN` в `.env` и включите канал в `config.yaml`.
|
||||
|
||||
**Настройка DingTalk**
|
||||
|
||||
1. Создайте приложение на [DingTalk Open Platform](https://open.dingtalk.com/) и включите возможность **Робот**.
|
||||
2. На странице настроек робота установите режим приёма сообщений на **Stream**.
|
||||
3. Скопируйте `Client ID` и `Client Secret`. Укажите `DINGTALK_CLIENT_ID` и `DINGTALK_CLIENT_SECRET` в `.env` и включите канал в `config.yaml`.
|
||||
4. *(Опционально)* Для включения потоковых ответов AI Card (эффект печатной машинки) создайте шаблон **AI Card** на [платформе карточек DingTalk](https://open.dingtalk.com/document/dingstart/typewriter-effect-streaming-ai-card), затем укажите `card_template_id` в `config.yaml` с ID шаблона. Также необходимо запросить разрешения `Card.Streaming.Write` и `Card.Instance.Write`.
|
||||
|
||||
**Доступные команды**
|
||||
|
||||
| Команда | Описание |
|
||||
|---------|----------|
|
||||
| `/new` | Начать новый диалог |
|
||||
| `/status` | Показать информацию о текущем треде |
|
||||
| `/models` | Список доступных моделей |
|
||||
| `/memory` | Просмотреть память |
|
||||
| `/help` | Показать справку |
|
||||
|
||||
> Сообщения без команды воспринимаются как обычный чат — DeerFlow создаёт тред и отвечает.
|
||||
|
||||
#### Трассировка LangSmith
|
||||
|
||||
DeerFlow имеет встроенную интеграцию с [LangSmith](https://smith.langchain.com) для наблюдаемости. При включении все вызовы LLM, запуски агентов и выполнения инструментов отслеживаются и отображаются в дашборде LangSmith.
|
||||
|
||||
Добавьте в файл `.env` в корне проекта:
|
||||
|
||||
```bash
|
||||
LANGSMITH_TRACING=true
|
||||
LANGSMITH_API_KEY=lsv2_pt_xxxxxxxxxxxxxxxx
|
||||
LANGSMITH_PROJECT=deer-flow
|
||||
```
|
||||
|
||||
`LANGSMITH_ENDPOINT` по умолчанию `https://api.smith.langchain.com` и может быть переопределён при необходимости. Устаревшие переменные `LANGCHAIN_*` (`LANGCHAIN_TRACING_V2`, `LANGCHAIN_API_KEY` и т.д.) также поддерживаются для обратной совместимости; `LANGSMITH_*` имеет приоритет, когда заданы обе.
|
||||
|
||||
В Docker-развёртываниях трассировка отключена по умолчанию. Установите `LANGSMITH_TRACING=true` и `LANGSMITH_API_KEY` в `.env` для включения.
|
||||
|
||||
## От Deep Research к Super Agent Harness
|
||||
|
||||
DeerFlow начинался как фреймворк для Deep Research, и сообщество вышло далеко за эти рамки. После запуска разработчики строили пайплайны, генерировали презентации, поднимали дашборды, автоматизировали контент. То, чего мы не ожидали.
|
||||
|
||||
Стало понятно: DeerFlow не просто research-инструмент. Это **harness**: runtime, который даёт агентам необходимую инфраструктуру.
|
||||
|
||||
Поэтому мы переписали всё с нуля.
|
||||
|
||||
DeerFlow 2.0 — это Super Agent Harness «из коробки». Batteries included, полностью расширяемый. Построен на LangGraph и LangChain. По умолчанию есть всё, что нужно агенту: файловая система, memory, skills, sandbox-выполнение и возможность планировать и запускать sub-agents для сложных многошаговых задач.
|
||||
|
||||
Используйте как есть. Или разберите и переделайте под себя.
|
||||
|
||||
## Core Features
|
||||
|
||||
### Skills & Tools
|
||||
|
||||
Skills — это то, что позволяет DeerFlow делать почти что угодно.
|
||||
|
||||
Agent Skill — это структурированный модуль: Markdown-файл с описанием воркфлоу, лучших практик и ссылок на ресурсы. DeerFlow поставляется со встроенными skills для ресёрча, генерации отчётов, слайдов, веб-страниц, изображений и видео. Но главное — расширяемость: добавляйте свои skills, заменяйте встроенные или собирайте из них составные воркфлоу.
|
||||
|
||||
Skills загружаются по мере необходимости, только когда задача их требует. Это держит контекстное окно чистым.
|
||||
|
||||
```
|
||||
# Пути внутри контейнера sandbox
|
||||
/mnt/skills/public
|
||||
├── research/SKILL.md
|
||||
├── report-generation/SKILL.md
|
||||
├── slide-creation/SKILL.md
|
||||
├── web-page/SKILL.md
|
||||
└── image-generation/SKILL.md
|
||||
|
||||
/mnt/skills/custom
|
||||
└── your-custom-skill/SKILL.md ← ваш skill
|
||||
```
|
||||
|
||||
#### Интеграция с Claude Code
|
||||
|
||||
Skill `claude-to-deerflow` позволяет работать с DeerFlow прямо из [Claude Code](https://docs.anthropic.com/en/docs/claude-code). Отправляйте задачи, проверяйте статус, управляйте тредами, не выходя из терминала.
|
||||
|
||||
**Установка скилла**:
|
||||
|
||||
```bash
|
||||
npx skills add https://github.com/bytedance/deer-flow --skill claude-to-deerflow
|
||||
```
|
||||
|
||||
**Что можно делать**:
|
||||
- Отправлять сообщения в DeerFlow и получать потоковые ответы
|
||||
- Выбирать режимы выполнения: flash (быстро), standard, pro (planning), ultra (sub-agents)
|
||||
- Проверять статус DeerFlow, просматривать модели, скиллы, агентов
|
||||
- Управлять тредами и историей диалога
|
||||
- Загружать файлы для анализа
|
||||
|
||||
Полный справочник API в [`skills/public/claude-to-deerflow/SKILL.md`](skills/public/claude-to-deerflow/SKILL.md).
|
||||
|
||||
### Sub-Agents
|
||||
|
||||
Сложные задачи редко решаются за один проход. DeerFlow их декомпозирует.
|
||||
|
||||
Lead agent запускает sub-agents на лету, каждый со своим изолированным контекстом, инструментами и условиями завершения. Sub-agents работают параллельно, возвращают структурированные результаты, а lead agent собирает всё в единый итог.
|
||||
|
||||
Вот как DeerFlow справляется с задачами на минуты и часы: research-задача разветвляется в дюжину sub-agents, каждый копает свой угол, потом всё сходится в один отчёт, или сайт, или слайддек со сгенерированными визуалами. Один harness, много рук.
|
||||
|
||||
### Sandbox & файловая система
|
||||
|
||||
DeerFlow не просто *говорит* о том, что умеет что-то делать. У него есть собственный компьютер.
|
||||
|
||||
Каждая задача выполняется внутри изолированного Docker-контейнера с полной файловой системой: skills, workspace, uploads, outputs. Агент читает, пишет и редактирует файлы. Выполняет bash-команды и пишет код. Смотрит на изображения. Всё изолировано, всё прозрачно, никакого пересечения между сессиями.
|
||||
|
||||
Это разница между чатботом с доступом к инструментам и агентом с реальной средой выполнения.
|
||||
|
||||
```
|
||||
# Пути внутри контейнера sandbox
|
||||
/mnt/user-data/
|
||||
├── uploads/ ← ваши файлы
|
||||
├── workspace/ ← рабочая директория агентов
|
||||
└── outputs/ ← результаты
|
||||
```
|
||||
|
||||
### Context Engineering
|
||||
|
||||
**Изолированный контекст**: каждый sub-agent работает в своём контексте и не видит контекст главного агента или других sub-agents. Агент фокусируется на своей задаче.
|
||||
|
||||
**Управление контекстом**: внутри сессии DeerFlow агрессивно сжимает контекст и суммирует завершённые подзадачи, выгружает промежуточные результаты в файловую систему, сжимает то, что уже не актуально. На длинных многошаговых задачах контекстное окно не переполняется.
|
||||
|
||||
### Long-Term Memory
|
||||
|
||||
Большинство агентов забывают всё, когда диалог заканчивается. DeerFlow помнит.
|
||||
|
||||
DeerFlow сохраняет ваш профиль, предпочтения и накопленные знания между сессиями. Чем больше используете, тем лучше он вас знает: стиль, технологический стек, повторяющиеся воркфлоу. Всё хранится локально и остаётся под вашим контролем.
|
||||
|
||||
## Рекомендуемые модели
|
||||
|
||||
DeerFlow работает с любым LLM через OpenAI-совместимый API. Лучше всего — с моделями, которые поддерживают:
|
||||
|
||||
- **Большое контекстное окно** (100k+ токенов) — для deep research и многошаговых задач
|
||||
- **Reasoning capabilities** — для адаптивного планирования и сложной декомпозиции
|
||||
- **Multimodal inputs** — для работы с изображениями и видео
|
||||
- **Strong tool-use** — для надёжного вызова функций и структурированных ответов
|
||||
|
||||
## Встроенный Python-клиент
|
||||
|
||||
DeerFlow можно использовать как Python-библиотеку прямо в коде — без запуска HTTP-сервисов. `DeerFlowClient` даёт доступ ко всем возможностям агента и Gateway, возвращает те же схемы ответов, что и HTTP Gateway API:
|
||||
|
||||
```python
|
||||
from deerflow.client import DeerFlowClient
|
||||
|
||||
client = DeerFlowClient()
|
||||
|
||||
# Chat
|
||||
response = client.chat("Analyze this paper for me", thread_id="my-thread")
|
||||
|
||||
# Streaming (LangGraph SSE protocol: values, messages-tuple, end)
|
||||
for event in client.stream("hello"):
|
||||
if event.type == "messages-tuple" and event.data.get("type") == "ai":
|
||||
print(event.data["content"])
|
||||
|
||||
# Configuration & management — returns Gateway-aligned dicts
|
||||
models = client.list_models() # {"models": [...]}
|
||||
skills = client.list_skills() # {"skills": [...]}
|
||||
client.update_skill("web-search", enabled=True)
|
||||
client.upload_files("thread-1", ["./report.pdf"]) # {"success": True, "files": [...]}
|
||||
```
|
||||
|
||||
## Документация
|
||||
|
||||
- [Руководство по участию](CONTRIBUTING.md) — настройка среды разработки, воркфлоу и гайдлайны
|
||||
- [Руководство по конфигурации](backend/docs/CONFIGURATION.md) — инструкции по настройке
|
||||
- [Обзор архитектуры](backend/CLAUDE.md) — технические детали
|
||||
- [Архитектура бэкенда](backend/README.md) — бэкенд и справочник API
|
||||
|
||||
## ⚠️ Безопасность
|
||||
|
||||
### Неправильное развёртывание может привести к угрозам безопасности
|
||||
|
||||
DeerFlow обладает ключевыми высокопривилегированными возможностями, включая **выполнение системных команд, операции с ресурсами и вызов бизнес-логики**. По умолчанию он рассчитан на **развёртывание в локальной доверенной среде (доступ только через loopback-адрес 127.0.0.1)**. Если вы разворачиваете агент в недоверенных средах — локальных сетях, публичных облачных серверах или других окружениях, доступных с нескольких устройств — без строгих мер безопасности, это может привести к следующим угрозам:
|
||||
|
||||
- **Несанкционированные вызовы**: функциональность агента может быть обнаружена неавторизованными третьими лицами или вредоносными сканерами, что приведёт к массовым несанкционированным запросам с выполнением высокорисковых операций (системные команды, чтение/запись файлов) и серьёзным последствиям для безопасности.
|
||||
- **Юридические и compliance-риски**: если агент будет незаконно использован для кибератак, кражи данных или других противоправных действий, это может повлечь юридическую ответственность и compliance-риски.
|
||||
|
||||
### Рекомендации по безопасности
|
||||
|
||||
**Примечание: настоятельно рекомендуем развёртывать DeerFlow только в локальной доверенной сети.** Если вам необходимо развёртывание через несколько устройств или сетей, обязательно реализуйте строгие меры безопасности, например:
|
||||
|
||||
- **Белый список IP-адресов**: используйте `iptables` или аппаратные межсетевые экраны / коммутаторы с ACL, чтобы **настроить правила белого списка IP** и заблокировать доступ со всех остальных адресов.
|
||||
- **Шлюз аутентификации**: настройте обратный прокси (nginx и др.) и **включите строгую предварительную аутентификацию**, запрещающую любой доступ без авторизации.
|
||||
- **Сетевая изоляция**: по возможности разместите агент и доверенные устройства в **одном выделенном VLAN**, изолированном от остальной сети.
|
||||
- **Следите за обновлениями**: регулярно отслеживайте обновления безопасности проекта DeerFlow.
|
||||
|
||||
## Участие в разработке
|
||||
|
||||
Приветствуем контрибьюторов! Настройка среды разработки, воркфлоу и гайдлайны — в [CONTRIBUTING.md](CONTRIBUTING.md).
|
||||
|
||||
## Лицензия
|
||||
|
||||
Проект распространяется под [лицензией MIT](./LICENSE).
|
||||
|
||||
## Благодарности
|
||||
|
||||
DeerFlow стоит на плечах open-source сообщества. Спасибо всем проектам и разработчикам, чья работа сделала его возможным.
|
||||
|
||||
Отдельная благодарность:
|
||||
|
||||
- **[LangChain](https://github.com/langchain-ai/langchain)** — фреймворк для взаимодействия с LLM и построения цепочек.
|
||||
- **[LangGraph](https://github.com/langchain-ai/langgraph)** — многоагентная оркестрация, на которой держатся сложные воркфлоу DeerFlow.
|
||||
|
||||
### Ключевые контрибьюторы
|
||||
|
||||
Авторы DeerFlow, без которых проекта бы не было:
|
||||
|
||||
- **[Daniel Walnut](https://github.com/hetaoBackend/)**
|
||||
- **[Henry Li](https://github.com/magiccube/)**
|
||||
|
||||
## История звёзд
|
||||
|
||||
[](https://star-history.com/#bytedance/deer-flow&Date)
|
||||
+479
-532
File diff suppressed because it is too large
Load Diff
-12
@@ -1,12 +0,0 @@
|
||||
# Security Policy
|
||||
|
||||
## Supported Versions
|
||||
|
||||
As deer-flow doesn't provide an official release yet, please use the latest version for the security updates.
|
||||
Currently, we have two branches to maintain:
|
||||
* main branch for deer-flow 2.x
|
||||
* main-1.x branch for deer-flow 1.x
|
||||
|
||||
## Reporting a Vulnerability
|
||||
|
||||
Please go to https://github.com/bytedance/deer-flow/security to report the vulnerability you find.
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 151 KiB |
@@ -1,28 +0,0 @@
|
||||
# Python-generated files
|
||||
__pycache__/
|
||||
*.py[oc]
|
||||
build/
|
||||
dist/
|
||||
wheels/
|
||||
*.egg-info
|
||||
.coverage
|
||||
.coverage.*
|
||||
.ruff_cache
|
||||
agent_history.gif
|
||||
static/browser_history/*.gif
|
||||
|
||||
log/
|
||||
log/*
|
||||
|
||||
# Virtual environments
|
||||
.venv
|
||||
venv/
|
||||
|
||||
# User config file
|
||||
config.yaml
|
||||
|
||||
# Langgraph
|
||||
.langgraph_api
|
||||
|
||||
# Claude Code settings
|
||||
.claude/settings.local.json
|
||||
Vendored
-3
@@ -1,3 +0,0 @@
|
||||
{
|
||||
"recommendations": ["charliermarsh.ruff"]
|
||||
}
|
||||
Vendored
-11
@@ -1,11 +0,0 @@
|
||||
{
|
||||
"window.title": "${activeEditorShort}${separator}${separator}deer-flow/backend",
|
||||
"[python]": {
|
||||
"editor.formatOnSave": true,
|
||||
"editor.codeActionsOnSave": {
|
||||
"source.fixAll": "explicit",
|
||||
"source.organizeImports": "explicit"
|
||||
},
|
||||
"editor.defaultFormatter": "charliermarsh.ruff"
|
||||
}
|
||||
}
|
||||
@@ -1,2 +0,0 @@
|
||||
For the backend architecture and design patterns:
|
||||
@./CLAUDE.md
|
||||
@@ -1,572 +0,0 @@
|
||||
# CLAUDE.md
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
## Project Overview
|
||||
|
||||
DeerFlow is a LangGraph-based AI super agent system with a full-stack architecture. The backend provides a "super agent" with sandbox execution, persistent memory, subagent delegation, and extensible tool integration - all operating in per-thread isolated environments.
|
||||
|
||||
**Architecture**:
|
||||
- **Gateway API** (port 8001): REST API plus embedded LangGraph-compatible agent runtime
|
||||
- **Frontend** (port 3000): Next.js web interface
|
||||
- **Nginx** (port 2026): Unified reverse proxy entry point
|
||||
- **Provisioner** (port 8002, optional in Docker dev): Started only when sandbox is configured for provisioner/Kubernetes mode
|
||||
|
||||
**Runtime**:
|
||||
- `make dev`, Docker dev, and production all run the agent runtime in Gateway via `RunManager` + `run_agent()` + `StreamBridge` (`packages/harness/deerflow/runtime/`). Nginx exposes that runtime at `/api/langgraph/*` and rewrites it to Gateway's native `/api/*` routers.
|
||||
|
||||
**Project Structure**:
|
||||
```
|
||||
deer-flow/
|
||||
├── Makefile # Root commands (check, install, dev, stop)
|
||||
├── config.yaml # Main application configuration
|
||||
├── extensions_config.json # MCP servers and skills configuration
|
||||
├── backend/ # Backend application (this directory)
|
||||
│ ├── Makefile # Backend-only commands (dev, gateway, lint)
|
||||
│ ├── langgraph.json # LangGraph Studio graph configuration
|
||||
│ ├── packages/
|
||||
│ │ └── harness/ # deerflow-harness package (import: deerflow.*)
|
||||
│ │ ├── pyproject.toml
|
||||
│ │ └── deerflow/
|
||||
│ │ ├── agents/ # LangGraph agent system
|
||||
│ │ │ ├── lead_agent/ # Main agent (factory + system prompt)
|
||||
│ │ │ ├── middlewares/ # 10 middleware components
|
||||
│ │ │ ├── memory/ # Memory extraction, queue, prompts
|
||||
│ │ │ └── thread_state.py # ThreadState schema
|
||||
│ │ ├── sandbox/ # Sandbox execution system
|
||||
│ │ │ ├── local/ # Local filesystem provider
|
||||
│ │ │ ├── sandbox.py # Abstract Sandbox interface
|
||||
│ │ │ ├── tools.py # bash, ls, read/write/str_replace
|
||||
│ │ │ └── middleware.py # Sandbox lifecycle management
|
||||
│ │ ├── subagents/ # Subagent delegation system
|
||||
│ │ │ ├── builtins/ # general-purpose, bash agents
|
||||
│ │ │ ├── executor.py # Background execution engine
|
||||
│ │ │ └── registry.py # Agent registry
|
||||
│ │ ├── tools/builtins/ # Built-in tools (present_files, ask_clarification, view_image)
|
||||
│ │ ├── mcp/ # MCP integration (tools, cache, client)
|
||||
│ │ ├── models/ # Model factory with thinking/vision support
|
||||
│ │ ├── skills/ # Skills discovery, loading, parsing
|
||||
│ │ ├── config/ # Configuration system (app, model, sandbox, tool, etc.)
|
||||
│ │ ├── community/ # Community tools (tavily, jina_ai, firecrawl, image_search, aio_sandbox)
|
||||
│ │ ├── reflection/ # Dynamic module loading (resolve_variable, resolve_class)
|
||||
│ │ ├── utils/ # Utilities (network, readability)
|
||||
│ │ └── client.py # Embedded Python client (DeerFlowClient)
|
||||
│ ├── app/ # Application layer (import: app.*)
|
||||
│ │ ├── gateway/ # FastAPI Gateway API
|
||||
│ │ │ ├── app.py # FastAPI application
|
||||
│ │ │ └── routers/ # FastAPI route modules (models, mcp, memory, skills, uploads, threads, artifacts, agents, suggestions, channels)
|
||||
│ │ └── channels/ # IM platform integrations
|
||||
│ ├── tests/ # Test suite
|
||||
│ └── docs/ # Documentation
|
||||
├── frontend/ # Next.js frontend application
|
||||
└── skills/ # Agent skills directory
|
||||
├── public/ # Public skills (committed)
|
||||
└── custom/ # Custom skills (gitignored)
|
||||
```
|
||||
|
||||
## Important Development Guidelines
|
||||
|
||||
### Documentation Update Policy
|
||||
**CRITICAL: Always update README.md and CLAUDE.md after every code change**
|
||||
|
||||
When making code changes, you MUST update the relevant documentation:
|
||||
- Update `README.md` for user-facing changes (features, setup, usage instructions)
|
||||
- Update `CLAUDE.md` for development changes (architecture, commands, workflows, internal systems)
|
||||
- Keep documentation synchronized with the codebase at all times
|
||||
- Ensure accuracy and timeliness of all documentation
|
||||
|
||||
## Commands
|
||||
|
||||
**Root directory** (for full application):
|
||||
```bash
|
||||
make check # Check system requirements
|
||||
make install # Install all dependencies (frontend + backend)
|
||||
make dev # Start all services (Gateway + Frontend + Nginx), with config.yaml preflight
|
||||
make start # Start production services locally
|
||||
make stop # Stop all services
|
||||
```
|
||||
|
||||
**Backend directory** (for backend development only):
|
||||
```bash
|
||||
make install # Install backend dependencies
|
||||
make dev # Run Gateway API with reload (port 8001)
|
||||
make gateway # Run Gateway API only (port 8001)
|
||||
make test # Run all backend tests
|
||||
make lint # Lint with ruff
|
||||
make format # Format code with ruff
|
||||
```
|
||||
|
||||
Regression tests related to Docker/provisioner behavior:
|
||||
- `tests/test_docker_sandbox_mode_detection.py` (mode detection from `config.yaml`)
|
||||
- `tests/test_provisioner_kubeconfig.py` (kubeconfig file/directory handling)
|
||||
|
||||
Boundary check (harness → app import firewall):
|
||||
- `tests/test_harness_boundary.py` — ensures `packages/harness/deerflow/` never imports from `app.*`
|
||||
|
||||
CI runs these regression tests for every pull request via [.github/workflows/backend-unit-tests.yml](../.github/workflows/backend-unit-tests.yml).
|
||||
|
||||
## Architecture
|
||||
|
||||
### Harness / App Split
|
||||
|
||||
The backend is split into two layers with a strict dependency direction:
|
||||
|
||||
- **Harness** (`packages/harness/deerflow/`): Publishable agent framework package (`deerflow-harness`). Import prefix: `deerflow.*`. Contains agent orchestration, tools, sandbox, models, MCP, skills, config — everything needed to build and run agents.
|
||||
- **App** (`app/`): Unpublished application code. Import prefix: `app.*`. Contains the FastAPI Gateway API and IM channel integrations (Feishu, Slack, Telegram, DingTalk).
|
||||
|
||||
**Dependency rule**: App imports deerflow, but deerflow never imports app. This boundary is enforced by `tests/test_harness_boundary.py` which runs in CI.
|
||||
|
||||
**Import conventions**:
|
||||
```python
|
||||
# Harness internal
|
||||
from deerflow.agents import make_lead_agent
|
||||
from deerflow.models import create_chat_model
|
||||
|
||||
# App internal
|
||||
from app.gateway.app import app
|
||||
from app.channels.service import start_channel_service
|
||||
|
||||
# App → Harness (allowed)
|
||||
from deerflow.config import get_app_config
|
||||
|
||||
# Harness → App (FORBIDDEN — enforced by test_harness_boundary.py)
|
||||
# from app.gateway.routers.uploads import ... # ← will fail CI
|
||||
```
|
||||
|
||||
### Agent System
|
||||
|
||||
**Lead Agent** (`packages/harness/deerflow/agents/lead_agent/agent.py`):
|
||||
- Entry point: `make_lead_agent(config: RunnableConfig)` registered in `langgraph.json`
|
||||
- Dynamic model selection via `create_chat_model()` with thinking/vision support
|
||||
- Tools loaded via `get_available_tools()` - combines sandbox, built-in, MCP, community, and subagent tools
|
||||
- System prompt generated by `apply_prompt_template()` with skills, memory, and subagent instructions
|
||||
|
||||
**ThreadState** (`packages/harness/deerflow/agents/thread_state.py`):
|
||||
- Extends `AgentState` with: `sandbox`, `thread_data`, `title`, `artifacts`, `todos`, `uploaded_files`, `viewed_images`
|
||||
- Uses custom reducers: `merge_artifacts` (deduplicate), `merge_viewed_images` (merge/clear)
|
||||
|
||||
**Runtime Configuration** (via `config.configurable`):
|
||||
- `thinking_enabled` - Enable model's extended thinking
|
||||
- `model_name` - Select specific LLM model
|
||||
- `is_plan_mode` - Enable TodoList middleware
|
||||
- `subagent_enabled` - Enable task delegation tool
|
||||
|
||||
### Middleware Chain
|
||||
|
||||
Lead-agent middlewares are assembled in strict append order across `packages/harness/deerflow/agents/middlewares/tool_error_handling_middleware.py` (`build_lead_runtime_middlewares`) and `packages/harness/deerflow/agents/lead_agent/agent.py` (`_build_middlewares`):
|
||||
|
||||
1. **ThreadDataMiddleware** - Creates per-thread directories under the user's isolation scope (`backend/.deer-flow/users/{user_id}/threads/{thread_id}/user-data/{workspace,uploads,outputs}`); resolves `user_id` via `get_effective_user_id()` (falls back to `"default"` in no-auth mode); Web UI thread deletion now follows LangGraph thread removal with Gateway cleanup of the local thread directory
|
||||
2. **UploadsMiddleware** - Tracks and injects newly uploaded files into conversation
|
||||
3. **SandboxMiddleware** - Acquires sandbox, stores `sandbox_id` in state
|
||||
4. **DanglingToolCallMiddleware** - Injects placeholder ToolMessages for AIMessage tool_calls that lack responses (e.g., due to user interruption), including raw provider tool-call payloads preserved only in `additional_kwargs["tool_calls"]`
|
||||
5. **LLMErrorHandlingMiddleware** - Normalizes provider/model invocation failures into recoverable assistant-facing errors before later middleware/tool stages run
|
||||
6. **GuardrailMiddleware** - Pre-tool-call authorization via pluggable `GuardrailProvider` protocol (optional, if `guardrails.enabled` in config). Evaluates each tool call and returns error ToolMessage on deny. Three provider options: built-in `AllowlistProvider` (zero deps), OAP policy providers (e.g. `aport-agent-guardrails`), or custom providers. See [docs/GUARDRAILS.md](docs/GUARDRAILS.md) for setup, usage, and how to implement a provider.
|
||||
7. **SandboxAuditMiddleware** - Audits sandboxed shell/file operations for security logging before tool execution continues
|
||||
8. **ToolErrorHandlingMiddleware** - Converts tool exceptions into error `ToolMessage`s so the run can continue instead of aborting
|
||||
9. **SummarizationMiddleware** - Context reduction when approaching token limits (optional, if enabled)
|
||||
10. **TodoListMiddleware** - Task tracking with `write_todos` tool (optional, if plan_mode)
|
||||
11. **TokenUsageMiddleware** - Records token usage metrics when token tracking is enabled (optional)
|
||||
12. **TitleMiddleware** - Auto-generates thread title after first complete exchange and normalizes structured message content before prompting the title model
|
||||
13. **MemoryMiddleware** - Queues conversations for async memory update (filters to user + final AI responses)
|
||||
14. **ViewImageMiddleware** - Injects base64 image data before LLM call (conditional on vision support)
|
||||
15. **DeferredToolFilterMiddleware** - Hides deferred tool schemas from the bound model until tool search is enabled (optional)
|
||||
16. **SubagentLimitMiddleware** - Truncates excess `task` tool calls from model response to enforce `MAX_CONCURRENT_SUBAGENTS` limit (optional, if `subagent_enabled`)
|
||||
17. **LoopDetectionMiddleware** - Detects repeated tool-call loops; hard-stop responses clear both structured `tool_calls` and raw provider tool-call metadata before forcing a final text answer
|
||||
18. **ClarificationMiddleware** - Intercepts `ask_clarification` tool calls, interrupts via `Command(goto=END)` (must be last)
|
||||
|
||||
### Configuration System
|
||||
|
||||
**Main Configuration** (`config.yaml`):
|
||||
|
||||
Setup: Copy `config.example.yaml` to `config.yaml` in the **project root** directory.
|
||||
|
||||
**Config Versioning**: `config.example.yaml` has a `config_version` field. On startup, `AppConfig.from_file()` compares user version vs example version and emits a warning if outdated. Missing `config_version` = version 0. Run `make config-upgrade` to auto-merge missing fields. When changing the config schema, bump `config_version` in `config.example.yaml`.
|
||||
|
||||
**Config Caching**: `get_app_config()` caches the parsed config, but automatically reloads it when the resolved config path changes or the file's mtime increases. This keeps Gateway and LangGraph reads aligned with `config.yaml` edits without requiring a manual process restart.
|
||||
|
||||
Configuration priority:
|
||||
1. Explicit `config_path` argument
|
||||
2. `DEER_FLOW_CONFIG_PATH` environment variable
|
||||
3. `config.yaml` in current directory (backend/)
|
||||
4. `config.yaml` in parent directory (project root - **recommended location**)
|
||||
|
||||
Config values starting with `$` are resolved as environment variables (e.g., `$OPENAI_API_KEY`).
|
||||
`ModelConfig` also declares `use_responses_api` and `output_version` so OpenAI `/v1/responses` can be enabled explicitly while still using `langchain_openai:ChatOpenAI`.
|
||||
|
||||
**Extensions Configuration** (`extensions_config.json`):
|
||||
|
||||
MCP servers and skills are configured together in `extensions_config.json` in project root:
|
||||
|
||||
Configuration priority:
|
||||
1. Explicit `config_path` argument
|
||||
2. `DEER_FLOW_EXTENSIONS_CONFIG_PATH` environment variable
|
||||
3. `extensions_config.json` in current directory (backend/)
|
||||
4. `extensions_config.json` in parent directory (project root - **recommended location**)
|
||||
|
||||
### Gateway API (`app/gateway/`)
|
||||
|
||||
FastAPI application on port 8001 with health check at `GET /health`. Set `GATEWAY_ENABLE_DOCS=false` to disable `/docs`, `/redoc`, and `/openapi.json` in production (default: enabled).
|
||||
|
||||
CORS is same-origin by default when requests enter through nginx on port 2026. Split-origin or port-forwarded browser clients must opt in with `GATEWAY_CORS_ORIGINS` (comma-separated exact origins); Gateway `CORSMiddleware` and `CSRFMiddleware` both read that variable so browser CORS and auth-origin checks stay aligned.
|
||||
|
||||
**Routers**:
|
||||
|
||||
| Router | Endpoints |
|
||||
|--------|-----------|
|
||||
| **Models** (`/api/models`) | `GET /` - list models; `GET /{name}` - model details |
|
||||
| **MCP** (`/api/mcp`) | `GET /config` - get config; `PUT /config` - update config (saves to extensions_config.json) |
|
||||
| **Skills** (`/api/skills`) | `GET /` - list skills; `GET /{name}` - details; `PUT /{name}` - update enabled; `POST /install` - install from .skill archive (accepts standard optional frontmatter like `version`, `author`, `compatibility`) |
|
||||
| **Memory** (`/api/memory`) | `GET /` - memory data; `POST /reload` - force reload; `GET /config` - config; `GET /status` - config + data |
|
||||
| **Uploads** (`/api/threads/{id}/uploads`) | `POST /` - upload files (auto-converts PDF/PPT/Excel/Word); `GET /list` - list; `DELETE /{filename}` - delete |
|
||||
| **Threads** (`/api/threads/{id}`) | `DELETE /` - remove DeerFlow-managed local thread data after LangGraph thread deletion; unexpected failures are logged server-side and return a generic 500 detail |
|
||||
| **Artifacts** (`/api/threads/{id}/artifacts`) | `GET /{path}` - serve artifacts; active content types (`text/html`, `application/xhtml+xml`, `image/svg+xml`) are always forced as download attachments to reduce XSS risk; `?download=true` still forces download for other file types |
|
||||
| **Suggestions** (`/api/threads/{id}/suggestions`) | `POST /` - generate follow-up questions; rich list/block model content is normalized before JSON parsing |
|
||||
| **Thread Runs** (`/api/threads/{id}/runs`) | `POST /` - create background run; `POST /stream` - create + SSE stream; `POST /wait` - create + block; `GET /` - list runs; `GET /{rid}` - run details; `POST /{rid}/cancel` - cancel; `GET /{rid}/join` - join SSE; `GET /{rid}/messages` - paginated messages `{data, has_more}`; `GET /{rid}/events` - full event stream; `GET /../messages` - thread messages with feedback; `GET /../token-usage` - aggregate tokens |
|
||||
| **Feedback** (`/api/threads/{id}/runs/{rid}/feedback`) | `PUT /` - upsert feedback; `DELETE /` - delete user feedback; `POST /` - create feedback; `GET /` - list feedback; `GET /stats` - aggregate stats; `DELETE /{fid}` - delete specific |
|
||||
| **Runs** (`/api/runs`) | `POST /stream` - stateless run + SSE; `POST /wait` - stateless run + block; `GET /{rid}/messages` - paginated messages by run_id `{data, has_more}` (cursor: `after_seq`/`before_seq`); `GET /{rid}/feedback` - list feedback by run_id |
|
||||
|
||||
Proxied through nginx: `/api/langgraph/*` → Gateway LangGraph-compatible runtime, all other `/api/*` → Gateway REST APIs.
|
||||
|
||||
### Sandbox System (`packages/harness/deerflow/sandbox/`)
|
||||
|
||||
**Interface**: Abstract `Sandbox` with `execute_command`, `read_file`, `write_file`, `list_dir`
|
||||
**Provider Pattern**: `SandboxProvider` with `acquire`, `get`, `release` lifecycle
|
||||
**Implementations**:
|
||||
- `LocalSandboxProvider` - Singleton local filesystem execution with path mappings
|
||||
- `AioSandboxProvider` (`packages/harness/deerflow/community/`) - Docker-based isolation
|
||||
|
||||
**Virtual Path System**:
|
||||
- Agent sees: `/mnt/user-data/{workspace,uploads,outputs}`, `/mnt/skills`
|
||||
- Physical: `backend/.deer-flow/users/{user_id}/threads/{thread_id}/user-data/...`, `deer-flow/skills/`
|
||||
- Translation: `replace_virtual_path()` / `replace_virtual_paths_in_command()`
|
||||
- Detection: `is_local_sandbox()` checks `sandbox_id == "local"`
|
||||
|
||||
**Sandbox Tools** (in `packages/harness/deerflow/sandbox/tools.py`):
|
||||
- `bash` - Execute commands with path translation and error handling
|
||||
- `ls` - Directory listing (tree format, max 2 levels)
|
||||
- `read_file` - Read file contents with optional line range
|
||||
- `write_file` - Write/append to files, creates directories; overwrites by default and exposes the `append` argument in the model-facing schema for end-of-file writes
|
||||
- `str_replace` - Substring replacement (single or all occurrences); same-path serialization is scoped to `(sandbox.id, path)` so isolated sandboxes do not contend on identical virtual paths inside one process
|
||||
|
||||
### Subagent System (`packages/harness/deerflow/subagents/`)
|
||||
|
||||
**Built-in Agents**: `general-purpose` (all tools except `task`) and `bash` (command specialist)
|
||||
**Execution**: Dual thread pool - `_scheduler_pool` (3 workers) + `_execution_pool` (3 workers)
|
||||
**Concurrency**: `MAX_CONCURRENT_SUBAGENTS = 3` enforced by `SubagentLimitMiddleware` (truncates excess tool calls in `after_model`), 15-minute timeout
|
||||
**Flow**: `task()` tool → `SubagentExecutor` → background thread → poll 5s → SSE events → result
|
||||
**Events**: `task_started`, `task_running`, `task_completed`/`task_failed`/`task_timed_out`
|
||||
|
||||
### Tool System (`packages/harness/deerflow/tools/`)
|
||||
|
||||
`get_available_tools(groups, include_mcp, model_name, subagent_enabled)` assembles:
|
||||
1. **Config-defined tools** - Resolved from `config.yaml` via `resolve_variable()`
|
||||
2. **MCP tools** - From enabled MCP servers (lazy initialized, cached with mtime invalidation)
|
||||
3. **Built-in tools**:
|
||||
- `present_files` - Make output files visible to user (only `/mnt/user-data/outputs`)
|
||||
- `ask_clarification` - Request clarification (intercepted by ClarificationMiddleware → interrupts)
|
||||
- `view_image` - Read image as base64 (added only if model supports vision)
|
||||
- `setup_agent` - Bootstrap-only: persist a brand-new custom agent's `SOUL.md` and `config.yaml`. Bound only when `is_bootstrap=True`.
|
||||
- `update_agent` - Custom-agent-only: persist self-updates to the current agent's `SOUL.md` / `config.yaml` from inside a normal chat (partial update + atomic write). Bound when `agent_name` is set and `is_bootstrap=False`.
|
||||
4. **Subagent tool** (if enabled):
|
||||
- `task` - Delegate to subagent (description, prompt, subagent_type)
|
||||
|
||||
**Community tools** (`packages/harness/deerflow/community/`):
|
||||
- `tavily/` - Web search (5 results default) and web fetch (4KB limit)
|
||||
- `jina_ai/` - Web fetch via Jina reader API with readability extraction
|
||||
- `firecrawl/` - Web scraping via Firecrawl API
|
||||
|
||||
**ACP agent tools**:
|
||||
- `invoke_acp_agent` - Invokes external ACP-compatible agents from `config.yaml`
|
||||
- ACP launchers must be real ACP adapters. The standard `codex` CLI is not ACP-compatible by itself; configure a wrapper such as `npx -y @zed-industries/codex-acp` or an installed `codex-acp` binary
|
||||
- Missing ACP executables now return an actionable error message instead of a raw `[Errno 2]`
|
||||
- Each ACP agent uses a per-thread workspace at `{base_dir}/users/{user_id}/threads/{thread_id}/acp-workspace/`. The workspace is accessible to the lead agent via the virtual path `/mnt/acp-workspace/` (read-only). In docker sandbox mode, the directory is volume-mounted into the container at `/mnt/acp-workspace` (read-only); in local sandbox mode, path translation is handled by `tools.py`
|
||||
- `image_search/` - Image search via DuckDuckGo
|
||||
|
||||
### MCP System (`packages/harness/deerflow/mcp/`)
|
||||
|
||||
- Uses `langchain-mcp-adapters` `MultiServerMCPClient` for multi-server management
|
||||
- **Lazy initialization**: Tools loaded on first use via `get_cached_mcp_tools()`
|
||||
- **Cache invalidation**: Detects config file changes via mtime comparison
|
||||
- **Transports**: stdio (command-based), SSE, HTTP
|
||||
- **OAuth (HTTP/SSE)**: Supports token endpoint flows (`client_credentials`, `refresh_token`) with automatic token refresh + Authorization header injection
|
||||
- **Runtime updates**: Gateway API saves to extensions_config.json; LangGraph detects via mtime
|
||||
|
||||
### Skills System (`packages/harness/deerflow/skills/`)
|
||||
|
||||
- **Location**: `deer-flow/skills/{public,custom}/`
|
||||
- **Format**: Directory with `SKILL.md` (YAML frontmatter: name, description, license, allowed-tools)
|
||||
- **Loading**: `load_skills()` recursively scans `skills/{public,custom}` for `SKILL.md`, parses metadata, and reads enabled state from extensions_config.json
|
||||
- **Injection**: Enabled skills listed in agent system prompt with container paths
|
||||
- **Installation**: `POST /api/skills/install` extracts .skill ZIP archive to custom/ directory
|
||||
|
||||
### Model Factory (`packages/harness/deerflow/models/factory.py`)
|
||||
|
||||
- `create_chat_model(name, thinking_enabled)` instantiates LLM from config via reflection
|
||||
- Supports `thinking_enabled` flag with per-model `when_thinking_enabled` overrides
|
||||
- Supports vLLM-style thinking toggles via `when_thinking_enabled.extra_body.chat_template_kwargs.enable_thinking` for Qwen reasoning models, while normalizing legacy `thinking` configs for backward compatibility
|
||||
- Supports `supports_vision` flag for image understanding models
|
||||
- Config values starting with `$` resolved as environment variables
|
||||
- Missing provider modules surface actionable install hints from reflection resolvers (for example `uv add langchain-google-genai`)
|
||||
|
||||
### vLLM Provider (`packages/harness/deerflow/models/vllm_provider.py`)
|
||||
|
||||
- `VllmChatModel` subclasses `langchain_openai:ChatOpenAI` for vLLM 0.19.0 OpenAI-compatible endpoints
|
||||
- Preserves vLLM's non-standard assistant `reasoning` field on full responses, streaming deltas, and follow-up tool-call turns
|
||||
- Designed for configs that enable thinking through `extra_body.chat_template_kwargs.enable_thinking` on vLLM 0.19.0 Qwen reasoning models, while accepting the older `thinking` alias
|
||||
|
||||
### IM Channels System (`app/channels/`)
|
||||
|
||||
Bridges external messaging platforms (Feishu, Slack, Telegram, DingTalk) to the DeerFlow agent via the LangGraph Server.
|
||||
|
||||
|
||||
**Architecture**: Channels communicate with Gateway through the `langgraph-sdk` HTTP client (same as the frontend), ensuring threads are created and managed server-side. The internal SDK client injects process-local internal auth plus a matching CSRF cookie/header pair so Gateway accepts state-changing thread/run requests from channel workers without relying on browser session cookies.
|
||||
|
||||
**Components**:
|
||||
- `message_bus.py` - Async pub/sub hub (`InboundMessage` → queue → dispatcher; `OutboundMessage` → callbacks → channels)
|
||||
- `store.py` - JSON-file persistence mapping `channel_name:chat_id[:topic_id]` → `thread_id` (keys are `channel:chat` for root conversations and `channel:chat:topic` for threaded conversations)
|
||||
- `manager.py` - Core dispatcher: creates threads via `client.threads.create()`, routes commands, keeps Slack/Telegram on `client.runs.wait()`, and uses `client.runs.stream(["messages-tuple", "values"])` for Feishu incremental outbound updates
|
||||
- `base.py` - Abstract `Channel` base class (start/stop/send lifecycle)
|
||||
- `service.py` - Manages lifecycle of all configured channels from `config.yaml`
|
||||
- `slack.py` / `feishu.py` / `telegram.py` / `dingtalk.py` - Platform-specific implementations (`feishu.py` tracks the running card `message_id` in memory and patches the same card in place; `dingtalk.py` optionally uses AI Card streaming for in-place updates when `card_template_id` is configured)
|
||||
|
||||
**Message Flow**:
|
||||
1. External platform -> Channel impl -> `MessageBus.publish_inbound()`
|
||||
2. `ChannelManager._dispatch_loop()` consumes from queue
|
||||
3. For chat: look up/create thread through Gateway's LangGraph-compatible API
|
||||
4. Feishu chat: `runs.stream()` → accumulate AI text → publish multiple outbound updates (`is_final=False`) → publish final outbound (`is_final=True`)
|
||||
5. Slack/Telegram chat: `runs.wait()` → extract final response → publish outbound
|
||||
6. Feishu channel sends one running reply card up front, then patches the same card for each outbound update (card JSON sets `config.update_multi=true` for Feishu's patch API requirement)
|
||||
7. DingTalk AI Card mode (when `card_template_id` configured): `runs.stream()` → create card with initial text → stream updates via `PUT /v1.0/card/streaming` → finalize on `is_final=True`. Falls back to `sampleMarkdown` if card creation or streaming fails
|
||||
8. For commands (`/new`, `/status`, `/models`, `/memory`, `/help`): handle locally or query Gateway API
|
||||
9. Outbound → channel callbacks → platform reply
|
||||
|
||||
**Configuration** (`config.yaml` -> `channels`):
|
||||
- `langgraph_url` - LangGraph-compatible Gateway API base URL (default: `http://localhost:8001/api`)
|
||||
- `gateway_url` - Gateway API URL for auxiliary commands (default: `http://localhost:8001`)
|
||||
- In Docker Compose, IM channels run inside the `gateway` container, so `localhost` points back to that container. Use `http://gateway:8001/api` for `langgraph_url` and `http://gateway:8001` for `gateway_url`, or set `DEER_FLOW_CHANNELS_LANGGRAPH_URL` / `DEER_FLOW_CHANNELS_GATEWAY_URL`.
|
||||
- Per-channel configs: `feishu` (app_id, app_secret), `slack` (bot_token, app_token), `telegram` (bot_token), `dingtalk` (client_id, client_secret, optional `card_template_id` for AI Card streaming)
|
||||
|
||||
|
||||
### Memory System (`packages/harness/deerflow/agents/memory/`)
|
||||
|
||||
**Components**:
|
||||
- `updater.py` - LLM-based memory updates with fact extraction, whitespace-normalized fact deduplication (trims leading/trailing whitespace before comparing), and atomic file I/O
|
||||
- `queue.py` - Debounced update queue (per-thread deduplication, configurable wait time); captures `user_id` at enqueue time so it survives the `threading.Timer` boundary
|
||||
- `prompt.py` - Prompt templates for memory updates
|
||||
- `storage.py` - File-based storage with per-user isolation; cache keyed by `(user_id, agent_name)` tuple
|
||||
|
||||
**Per-User Isolation**:
|
||||
- Memory is stored per-user at `{base_dir}/users/{user_id}/memory.json`
|
||||
- Per-agent per-user memory at `{base_dir}/users/{user_id}/agents/{agent_name}/memory.json`
|
||||
- Custom agent definitions (`SOUL.md` + `config.yaml`) are also per-user at `{base_dir}/users/{user_id}/agents/{agent_name}/`. The legacy shared layout `{base_dir}/agents/{agent_name}/` remains read-only fallback for unmigrated installations
|
||||
- `user_id` is resolved via `get_effective_user_id()` from `deerflow.runtime.user_context`
|
||||
- In no-auth mode, `user_id` defaults to `"default"` (constant `DEFAULT_USER_ID`)
|
||||
- Absolute `storage_path` in config opts out of per-user isolation
|
||||
- **Migration**: Run `PYTHONPATH=. python scripts/migrate_user_isolation.py` to move legacy `memory.json`, `threads/`, and `agents/` into per-user layout. Supports `--dry-run` (preview changes) and `--user-id USER_ID` (assign unowned legacy data to a user, defaults to `default`).
|
||||
|
||||
**Data Structure** (stored in `{base_dir}/users/{user_id}/memory.json`):
|
||||
- **User Context**: `workContext`, `personalContext`, `topOfMind` (1-3 sentence summaries)
|
||||
- **History**: `recentMonths`, `earlierContext`, `longTermBackground`
|
||||
- **Facts**: Discrete facts with `id`, `content`, `category` (preference/knowledge/context/behavior/goal), `confidence` (0-1), `createdAt`, `source`
|
||||
|
||||
**Workflow**:
|
||||
1. `MemoryMiddleware` filters messages (user inputs + final AI responses), captures `user_id` via `get_effective_user_id()`, and queues conversation with the captured `user_id`
|
||||
2. Queue debounces (30s default), batches updates, deduplicates per-thread
|
||||
3. Background thread invokes LLM to extract context updates and facts, using the stored `user_id` (not the contextvar, which is unavailable on timer threads)
|
||||
4. Applies updates atomically (temp file + rename) with cache invalidation, skipping duplicate fact content before append
|
||||
5. Next interaction injects top 15 facts + context into `<memory>` tags in system prompt
|
||||
|
||||
Focused regression coverage for the updater lives in `backend/tests/test_memory_updater.py`.
|
||||
|
||||
**Configuration** (`config.yaml` → `memory`):
|
||||
- `enabled` / `injection_enabled` - Master switches
|
||||
- `storage_path` - Path to memory.json (absolute path opts out of per-user isolation)
|
||||
- `debounce_seconds` - Wait time before processing (default: 30)
|
||||
- `model_name` - LLM for updates (null = default model)
|
||||
- `max_facts` / `fact_confidence_threshold` - Fact storage limits (100 / 0.7)
|
||||
- `max_injection_tokens` - Token limit for prompt injection (2000)
|
||||
|
||||
### Reflection System (`packages/harness/deerflow/reflection/`)
|
||||
|
||||
- `resolve_variable(path)` - Import module and return variable (e.g., `module.path:variable_name`)
|
||||
- `resolve_class(path, base_class)` - Import and validate class against base class
|
||||
|
||||
### Config Schema
|
||||
|
||||
**`config.yaml`** key sections:
|
||||
- `models[]` - LLM configs with `use` class path, `supports_thinking`, `supports_vision`, provider-specific fields
|
||||
- vLLM reasoning models should use `deerflow.models.vllm_provider:VllmChatModel`; for Qwen-style parsers prefer `when_thinking_enabled.extra_body.chat_template_kwargs.enable_thinking`, and DeerFlow will also normalize the older `thinking` alias
|
||||
- `tools[]` - Tool configs with `use` variable path and `group`
|
||||
- `tool_groups[]` - Logical groupings for tools
|
||||
- `sandbox.use` - Sandbox provider class path
|
||||
- `skills.path` / `skills.container_path` - Host and container paths to skills directory
|
||||
- `title` - Auto-title generation (enabled, max_words, max_chars, prompt_template)
|
||||
- `summarization` - Context summarization (enabled, trigger conditions, keep policy)
|
||||
- `subagents.enabled` - Master switch for subagent delegation
|
||||
- `memory` - Memory system (enabled, storage_path, debounce_seconds, model_name, max_facts, fact_confidence_threshold, injection_enabled, max_injection_tokens)
|
||||
|
||||
**`extensions_config.json`**:
|
||||
- `mcpServers` - Map of server name → config (enabled, type, command, args, env, url, headers, oauth, description)
|
||||
- `skills` - Map of skill name → state (enabled)
|
||||
|
||||
Both can be modified at runtime via Gateway API endpoints or `DeerFlowClient` methods.
|
||||
|
||||
### Embedded Client (`packages/harness/deerflow/client.py`)
|
||||
|
||||
`DeerFlowClient` provides direct in-process access to all DeerFlow capabilities without HTTP services. All return types align with the Gateway API response schemas, so consumer code works identically in HTTP and embedded modes.
|
||||
|
||||
**Architecture**: Imports the same `deerflow` modules that Gateway API uses. Shares the same config files and data directories. No FastAPI dependency.
|
||||
|
||||
**Agent Conversation**:
|
||||
- `chat(message, thread_id)` — synchronous, accumulates streaming deltas per message-id and returns the final AI text
|
||||
- `stream(message, thread_id)` — subscribes to LangGraph `stream_mode=["values", "messages", "custom"]` and yields `StreamEvent`:
|
||||
- `"values"` — full state snapshot (title, messages, artifacts); AI text already delivered via `messages` mode is **not** re-synthesized here to avoid duplicate deliveries
|
||||
- `"messages-tuple"` — per-chunk update: for AI text this is a **delta** (concat per `id` to rebuild the full message); tool calls and tool results are emitted once each
|
||||
- `"custom"` — forwarded from `StreamWriter`
|
||||
- `"end"` — stream finished (carries cumulative `usage` counted once per message id)
|
||||
- Agent created lazily via `create_agent()` + `_build_middlewares()`, same as `make_lead_agent`
|
||||
- Supports `checkpointer` parameter for state persistence across turns
|
||||
- `reset_agent()` forces agent recreation (e.g. after memory or skill changes)
|
||||
- See [docs/STREAMING.md](docs/STREAMING.md) for the full design: why Gateway and DeerFlowClient are parallel paths, LangGraph's `stream_mode` semantics, the per-id dedup invariants, and regression testing strategy
|
||||
|
||||
**Gateway Equivalent Methods** (replaces Gateway API):
|
||||
|
||||
| Category | Methods | Return format |
|
||||
|----------|---------|---------------|
|
||||
| Models | `list_models()`, `get_model(name)` | `{"models": [...]}`, `{name, display_name, ...}` |
|
||||
| MCP | `get_mcp_config()`, `update_mcp_config(servers)` | `{"mcp_servers": {...}}` |
|
||||
| Skills | `list_skills()`, `get_skill(name)`, `update_skill(name, enabled)`, `install_skill(path)` | `{"skills": [...]}` |
|
||||
| Memory | `get_memory()`, `reload_memory()`, `get_memory_config()`, `get_memory_status()` | dict |
|
||||
| Uploads | `upload_files(thread_id, files)`, `list_uploads(thread_id)`, `delete_upload(thread_id, filename)` | `{"success": true, "files": [...]}`, `{"files": [...], "count": N}` |
|
||||
| Artifacts | `get_artifact(thread_id, path)` → `(bytes, mime_type)` | tuple |
|
||||
|
||||
**Key difference from Gateway**: Upload accepts local `Path` objects instead of HTTP `UploadFile`, rejects directory paths before copying, and reuses a single worker when document conversion must run inside an active event loop. Artifact returns `(bytes, mime_type)` instead of HTTP Response. The new Gateway-only thread cleanup route deletes `.deer-flow/threads/{thread_id}` after LangGraph thread deletion; there is no matching `DeerFlowClient` method yet. `update_mcp_config()` and `update_skill()` automatically invalidate the cached agent.
|
||||
|
||||
**Tests**: `tests/test_client.py` (77 unit tests including `TestGatewayConformance`), `tests/test_client_live.py` (live integration tests, requires config.yaml)
|
||||
|
||||
**Gateway Conformance Tests** (`TestGatewayConformance`): Validate that every dict-returning client method conforms to the corresponding Gateway Pydantic response model. Each test parses the client output through the Gateway model — if Gateway adds a required field that the client doesn't provide, Pydantic raises `ValidationError` and CI catches the drift. Covers: `ModelsListResponse`, `ModelResponse`, `SkillsListResponse`, `SkillResponse`, `SkillInstallResponse`, `McpConfigResponse`, `UploadResponse`, `MemoryConfigResponse`, `MemoryStatusResponse`.
|
||||
|
||||
## Development Workflow
|
||||
|
||||
### Test-Driven Development (TDD) — MANDATORY
|
||||
|
||||
**Every new feature or bug fix MUST be accompanied by unit tests. No exceptions.**
|
||||
|
||||
- Write tests in `backend/tests/` following the existing naming convention `test_<feature>.py`
|
||||
- Run the full suite before and after your change: `make test`
|
||||
- Tests must pass before a feature is considered complete
|
||||
- For lightweight config/utility modules, prefer pure unit tests with no external dependencies
|
||||
- If a module causes circular import issues in tests, add a `sys.modules` mock in `tests/conftest.py` (see existing example for `deerflow.subagents.executor`)
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
make test
|
||||
|
||||
# Run a specific test file
|
||||
PYTHONPATH=. uv run pytest tests/test_<feature>.py -v
|
||||
```
|
||||
|
||||
### Running the Full Application
|
||||
|
||||
From the **project root** directory:
|
||||
```bash
|
||||
make dev
|
||||
```
|
||||
|
||||
This starts all services and makes the application available at `http://localhost:2026`.
|
||||
|
||||
**All startup modes:**
|
||||
|
||||
| | **Local Foreground** | **Local Daemon** | **Docker Dev** | **Docker Prod** |
|
||||
|---|---|---|---|---|
|
||||
| **Dev** | `./scripts/serve.sh --dev`<br/>`make dev` | `./scripts/serve.sh --dev --daemon`<br/>`make dev-daemon` | `./scripts/docker.sh start`<br/>`make docker-start` | — |
|
||||
| **Prod** | `./scripts/serve.sh --prod`<br/>`make start` | `./scripts/serve.sh --prod --daemon`<br/>`make start-daemon` | — | `./scripts/deploy.sh`<br/>`make up` |
|
||||
|
||||
| Action | Local | Docker Dev | Docker Prod |
|
||||
|---|---|---|---|
|
||||
| **Stop** | `./scripts/serve.sh --stop`<br/>`make stop` | `./scripts/docker.sh stop`<br/>`make docker-stop` | `./scripts/deploy.sh down`<br/>`make down` |
|
||||
| **Restart** | `./scripts/serve.sh --restart [flags]` | `./scripts/docker.sh restart` | — |
|
||||
|
||||
**Nginx routing**:
|
||||
- `/api/langgraph/*` → Gateway embedded runtime (8001), rewritten to `/api/*`
|
||||
- `/api/*` (other) → Gateway API (8001)
|
||||
- `/` (non-API) → Frontend (3000)
|
||||
|
||||
### Running Backend Services Separately
|
||||
|
||||
From the **backend** directory:
|
||||
|
||||
```bash
|
||||
# Gateway API
|
||||
make gateway
|
||||
```
|
||||
|
||||
Direct access (without nginx):
|
||||
- Gateway: `http://localhost:8001`
|
||||
|
||||
### Frontend Configuration
|
||||
|
||||
The frontend uses environment variables to connect to backend services:
|
||||
- `NEXT_PUBLIC_LANGGRAPH_BASE_URL` - Defaults to `/api/langgraph` (through nginx)
|
||||
- `NEXT_PUBLIC_BACKEND_BASE_URL` - Defaults to empty string (through nginx)
|
||||
|
||||
When using `make dev` from root, the frontend automatically connects through nginx.
|
||||
|
||||
## Key Features
|
||||
|
||||
### File Upload
|
||||
|
||||
Multi-file upload with automatic document conversion:
|
||||
- Endpoint: `POST /api/threads/{thread_id}/uploads`
|
||||
- Supports: PDF, PPT, Excel, Word documents (converted via `markitdown`)
|
||||
- Rejects directory inputs before copying so uploads stay all-or-nothing
|
||||
- Reuses one conversion worker per request when called from an active event loop
|
||||
- Files stored in thread-isolated directories
|
||||
- Duplicate filenames in a single upload request are auto-renamed with `_N` suffixes so later files do not truncate earlier files
|
||||
- Agent receives uploaded file list via `UploadsMiddleware`
|
||||
|
||||
See [docs/FILE_UPLOAD.md](docs/FILE_UPLOAD.md) for details.
|
||||
|
||||
### Plan Mode
|
||||
|
||||
TodoList middleware for complex multi-step tasks:
|
||||
- Controlled via runtime config: `config.configurable.is_plan_mode = True`
|
||||
- Provides `write_todos` tool for task tracking
|
||||
- One task in_progress at a time, real-time updates
|
||||
|
||||
See [docs/plan_mode_usage.md](docs/plan_mode_usage.md) for details.
|
||||
|
||||
### Context Summarization
|
||||
|
||||
Automatic conversation summarization when approaching token limits:
|
||||
- Configured in `config.yaml` under `summarization` key
|
||||
- Trigger types: tokens, messages, or fraction of max input
|
||||
- Keeps recent messages while summarizing older ones
|
||||
|
||||
See [docs/summarization.md](docs/summarization.md) for details.
|
||||
|
||||
### Vision Support
|
||||
|
||||
For models with `supports_vision: true`:
|
||||
- `ViewImageMiddleware` processes images in conversation
|
||||
- `view_image_tool` added to agent's toolset
|
||||
- Images automatically converted to base64 and injected into state
|
||||
|
||||
## Code Style
|
||||
|
||||
- Uses `ruff` for linting and formatting
|
||||
- Line length: 240 characters
|
||||
- Python 3.12+ with type hints
|
||||
- Double quotes, space indentation
|
||||
|
||||
## Documentation
|
||||
|
||||
See `docs/` directory for detailed documentation:
|
||||
- [CONFIGURATION.md](docs/CONFIGURATION.md) - Configuration options
|
||||
- [ARCHITECTURE.md](docs/ARCHITECTURE.md) - Architecture details
|
||||
- [API.md](docs/API.md) - API reference
|
||||
- [SETUP.md](docs/SETUP.md) - Setup guide
|
||||
- [FILE_UPLOAD.md](docs/FILE_UPLOAD.md) - File upload feature
|
||||
- [PATH_EXAMPLES.md](docs/PATH_EXAMPLES.md) - Path types and usage
|
||||
- [summarization.md](docs/summarization.md) - Context summarization
|
||||
- [plan_mode_usage.md](docs/plan_mode_usage.md) - Plan mode with TodoList
|
||||
@@ -1,423 +0,0 @@
|
||||
# Contributing to DeerFlow Backend
|
||||
|
||||
Thank you for your interest in contributing to DeerFlow! This document provides guidelines and instructions for contributing to the backend codebase.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Getting Started](#getting-started)
|
||||
- [Development Setup](#development-setup)
|
||||
- [Project Structure](#project-structure)
|
||||
- [Code Style](#code-style)
|
||||
- [Making Changes](#making-changes)
|
||||
- [Testing](#testing)
|
||||
- [Pull Request Process](#pull-request-process)
|
||||
- [Architecture Guidelines](#architecture-guidelines)
|
||||
|
||||
## Getting Started
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Python 3.12 or higher
|
||||
- [uv](https://docs.astral.sh/uv/) package manager
|
||||
- Git
|
||||
- Docker (optional, for Docker sandbox testing)
|
||||
|
||||
### Fork and Clone
|
||||
|
||||
1. Fork the repository on GitHub
|
||||
2. Clone your fork locally:
|
||||
```bash
|
||||
git clone https://github.com/YOUR_USERNAME/deer-flow.git
|
||||
cd deer-flow
|
||||
```
|
||||
|
||||
## Development Setup
|
||||
|
||||
### Install Dependencies
|
||||
|
||||
```bash
|
||||
# From project root
|
||||
cp config.example.yaml config.yaml
|
||||
|
||||
# Install backend dependencies
|
||||
cd backend
|
||||
make install
|
||||
```
|
||||
|
||||
### Configure Environment
|
||||
|
||||
Set up your API keys for testing:
|
||||
|
||||
```bash
|
||||
export OPENAI_API_KEY="your-api-key"
|
||||
# Add other keys as needed
|
||||
```
|
||||
|
||||
### Run the Development Server
|
||||
|
||||
```bash
|
||||
# Gateway API + embedded agent runtime
|
||||
make dev
|
||||
```
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
backend/src/
|
||||
├── agents/ # Agent system
|
||||
│ ├── lead_agent/ # Main agent implementation
|
||||
│ │ └── agent.py # Agent factory and creation
|
||||
│ ├── middlewares/ # Agent middlewares
|
||||
│ │ ├── thread_data_middleware.py
|
||||
│ │ ├── sandbox_middleware.py
|
||||
│ │ ├── title_middleware.py
|
||||
│ │ ├── uploads_middleware.py
|
||||
│ │ ├── view_image_middleware.py
|
||||
│ │ └── clarification_middleware.py
|
||||
│ └── thread_state.py # Thread state definition
|
||||
│
|
||||
├── gateway/ # FastAPI Gateway
|
||||
│ ├── app.py # FastAPI application
|
||||
│ └── routers/ # Route handlers
|
||||
│ ├── models.py # /api/models endpoints
|
||||
│ ├── mcp.py # /api/mcp endpoints
|
||||
│ ├── skills.py # /api/skills endpoints
|
||||
│ ├── artifacts.py # /api/threads/.../artifacts
|
||||
│ └── uploads.py # /api/threads/.../uploads
|
||||
│
|
||||
├── sandbox/ # Sandbox execution
|
||||
│ ├── __init__.py # Sandbox interface
|
||||
│ ├── local.py # Local sandbox provider
|
||||
│ └── tools.py # Sandbox tools (bash, file ops)
|
||||
│
|
||||
├── tools/ # Agent tools
|
||||
│ └── builtins/ # Built-in tools
|
||||
│ ├── present_file_tool.py
|
||||
│ ├── ask_clarification_tool.py
|
||||
│ └── view_image_tool.py
|
||||
│
|
||||
├── mcp/ # MCP integration
|
||||
│ └── manager.py # MCP server management
|
||||
│
|
||||
├── models/ # Model system
|
||||
│ └── factory.py # Model factory
|
||||
│
|
||||
├── skills/ # Skills system
|
||||
│ └── loader.py # Skills loader
|
||||
│
|
||||
├── config/ # Configuration
|
||||
│ ├── app_config.py # Main app config
|
||||
│ ├── extensions_config.py # Extensions config
|
||||
│ └── summarization_config.py
|
||||
│
|
||||
├── community/ # Community tools
|
||||
│ ├── tavily/ # Tavily web search
|
||||
│ ├── jina/ # Jina web fetch
|
||||
│ ├── firecrawl/ # Firecrawl scraping
|
||||
│ └── aio_sandbox/ # Docker sandbox
|
||||
│
|
||||
├── reflection/ # Dynamic loading
|
||||
│ └── __init__.py # Module resolution
|
||||
│
|
||||
└── utils/ # Utilities
|
||||
└── __init__.py
|
||||
```
|
||||
|
||||
## Code Style
|
||||
|
||||
### Linting and Formatting
|
||||
|
||||
We use `ruff` for both linting and formatting:
|
||||
|
||||
```bash
|
||||
# Check for issues
|
||||
make lint
|
||||
|
||||
# Auto-fix and format
|
||||
make format
|
||||
```
|
||||
|
||||
### Style Guidelines
|
||||
|
||||
- **Line length**: 240 characters maximum
|
||||
- **Python version**: 3.12+ features allowed
|
||||
- **Type hints**: Use type hints for function signatures
|
||||
- **Quotes**: Double quotes for strings
|
||||
- **Indentation**: 4 spaces (no tabs)
|
||||
- **Imports**: Group by standard library, third-party, local
|
||||
|
||||
### Docstrings
|
||||
|
||||
Use docstrings for public functions and classes:
|
||||
|
||||
```python
|
||||
def create_chat_model(name: str, thinking_enabled: bool = False) -> BaseChatModel:
|
||||
"""Create a chat model instance from configuration.
|
||||
|
||||
Args:
|
||||
name: The model name as defined in config.yaml
|
||||
thinking_enabled: Whether to enable extended thinking
|
||||
|
||||
Returns:
|
||||
A configured LangChain chat model instance
|
||||
|
||||
Raises:
|
||||
ValueError: If the model name is not found in configuration
|
||||
"""
|
||||
...
|
||||
```
|
||||
|
||||
## Making Changes
|
||||
|
||||
### Branch Naming
|
||||
|
||||
Use descriptive branch names:
|
||||
|
||||
- `feature/add-new-tool` - New features
|
||||
- `fix/sandbox-timeout` - Bug fixes
|
||||
- `docs/update-readme` - Documentation
|
||||
- `refactor/config-system` - Code refactoring
|
||||
|
||||
### Commit Messages
|
||||
|
||||
Write clear, concise commit messages:
|
||||
|
||||
```
|
||||
feat: add support for Claude 3.5 model
|
||||
|
||||
- Add model configuration in config.yaml
|
||||
- Update model factory to handle Claude-specific settings
|
||||
- Add tests for new model
|
||||
```
|
||||
|
||||
Prefix types:
|
||||
- `feat:` - New feature
|
||||
- `fix:` - Bug fix
|
||||
- `docs:` - Documentation
|
||||
- `refactor:` - Code refactoring
|
||||
- `test:` - Tests
|
||||
- `chore:` - Build/config changes
|
||||
|
||||
## Testing
|
||||
|
||||
### Running Tests
|
||||
|
||||
```bash
|
||||
uv run pytest
|
||||
```
|
||||
|
||||
### Writing Tests
|
||||
|
||||
Place tests in the `tests/` directory mirroring the source structure:
|
||||
|
||||
```
|
||||
tests/
|
||||
├── test_models/
|
||||
│ └── test_factory.py
|
||||
├── test_sandbox/
|
||||
│ └── test_local.py
|
||||
└── test_gateway/
|
||||
└── test_models_router.py
|
||||
```
|
||||
|
||||
Example test:
|
||||
|
||||
```python
|
||||
import pytest
|
||||
from deerflow.models.factory import create_chat_model
|
||||
|
||||
def test_create_chat_model_with_valid_name():
|
||||
"""Test that a valid model name creates a model instance."""
|
||||
model = create_chat_model("gpt-4")
|
||||
assert model is not None
|
||||
|
||||
def test_create_chat_model_with_invalid_name():
|
||||
"""Test that an invalid model name raises ValueError."""
|
||||
with pytest.raises(ValueError):
|
||||
create_chat_model("nonexistent-model")
|
||||
```
|
||||
|
||||
## Pull Request Process
|
||||
|
||||
### Before Submitting
|
||||
|
||||
1. **Ensure tests pass**: `uv run pytest`
|
||||
2. **Run linter**: `make lint`
|
||||
3. **Format code**: `make format`
|
||||
4. **Update documentation** if needed
|
||||
|
||||
### PR Description
|
||||
|
||||
Include in your PR description:
|
||||
|
||||
- **What**: Brief description of changes
|
||||
- **Why**: Motivation for the change
|
||||
- **How**: Implementation approach
|
||||
- **Testing**: How you tested the changes
|
||||
|
||||
### Review Process
|
||||
|
||||
1. Submit PR with clear description
|
||||
2. Address review feedback
|
||||
3. Ensure CI passes
|
||||
4. Maintainer will merge when approved
|
||||
|
||||
## Architecture Guidelines
|
||||
|
||||
### Adding New Tools
|
||||
|
||||
1. Create tool in `packages/harness/deerflow/tools/builtins/` or `packages/harness/deerflow/community/`:
|
||||
|
||||
```python
|
||||
# packages/harness/deerflow/tools/builtins/my_tool.py
|
||||
from langchain_core.tools import tool
|
||||
|
||||
@tool
|
||||
def my_tool(param: str) -> str:
|
||||
"""Tool description for the agent.
|
||||
|
||||
Args:
|
||||
param: Description of the parameter
|
||||
|
||||
Returns:
|
||||
Description of return value
|
||||
"""
|
||||
return f"Result: {param}"
|
||||
```
|
||||
|
||||
2. Register in `config.yaml`:
|
||||
|
||||
```yaml
|
||||
tools:
|
||||
- name: my_tool
|
||||
group: my_group
|
||||
use: deerflow.tools.builtins.my_tool:my_tool
|
||||
```
|
||||
|
||||
### Adding New Middleware
|
||||
|
||||
1. Create middleware in `packages/harness/deerflow/agents/middlewares/`:
|
||||
|
||||
```python
|
||||
# packages/harness/deerflow/agents/middlewares/my_middleware.py
|
||||
from langchain.agents.middleware import BaseMiddleware
|
||||
from langchain_core.runnables import RunnableConfig
|
||||
|
||||
class MyMiddleware(BaseMiddleware):
|
||||
"""Middleware description."""
|
||||
|
||||
def transform_state(self, state: dict, config: RunnableConfig) -> dict:
|
||||
"""Transform the state before agent execution."""
|
||||
# Modify state as needed
|
||||
return state
|
||||
```
|
||||
|
||||
2. Register in `packages/harness/deerflow/agents/lead_agent/agent.py`:
|
||||
|
||||
```python
|
||||
middlewares = [
|
||||
ThreadDataMiddleware(),
|
||||
SandboxMiddleware(),
|
||||
MyMiddleware(), # Add your middleware
|
||||
TitleMiddleware(),
|
||||
ClarificationMiddleware(),
|
||||
]
|
||||
```
|
||||
|
||||
### Adding New API Endpoints
|
||||
|
||||
1. Create router in `app/gateway/routers/`:
|
||||
|
||||
```python
|
||||
# app/gateway/routers/my_router.py
|
||||
from fastapi import APIRouter
|
||||
|
||||
router = APIRouter(prefix="/my-endpoint", tags=["my-endpoint"])
|
||||
|
||||
@router.get("/")
|
||||
async def get_items():
|
||||
"""Get all items."""
|
||||
return {"items": []}
|
||||
|
||||
@router.post("/")
|
||||
async def create_item(data: dict):
|
||||
"""Create a new item."""
|
||||
return {"created": data}
|
||||
```
|
||||
|
||||
2. Register in `app/gateway/app.py`:
|
||||
|
||||
```python
|
||||
from app.gateway.routers import my_router
|
||||
|
||||
app.include_router(my_router.router)
|
||||
```
|
||||
|
||||
### Configuration Changes
|
||||
|
||||
When adding new configuration options:
|
||||
|
||||
1. Update `packages/harness/deerflow/config/app_config.py` with new fields
|
||||
2. Add default values in `config.example.yaml`
|
||||
3. Document in `docs/CONFIGURATION.md`
|
||||
|
||||
### MCP Server Integration
|
||||
|
||||
To add support for a new MCP server:
|
||||
|
||||
1. Add configuration in `extensions_config.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"my-server": {
|
||||
"enabled": true,
|
||||
"type": "stdio",
|
||||
"command": "npx",
|
||||
"args": ["-y", "@my-org/mcp-server"],
|
||||
"description": "My MCP Server"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. Update `extensions_config.example.json` with the new server
|
||||
|
||||
### Skills Development
|
||||
|
||||
To create a new skill:
|
||||
|
||||
1. Create directory in `skills/public/` or `skills/custom/`:
|
||||
|
||||
```
|
||||
skills/public/my-skill/
|
||||
└── SKILL.md
|
||||
```
|
||||
|
||||
2. Write `SKILL.md` with YAML front matter:
|
||||
|
||||
```markdown
|
||||
---
|
||||
name: My Skill
|
||||
description: What this skill does
|
||||
license: MIT
|
||||
allowed-tools:
|
||||
- read_file
|
||||
- write_file
|
||||
- bash
|
||||
---
|
||||
|
||||
# My Skill
|
||||
|
||||
Instructions for the agent when this skill is enabled...
|
||||
```
|
||||
|
||||
## Questions?
|
||||
|
||||
If you have questions about contributing:
|
||||
|
||||
1. Check existing documentation in `docs/`
|
||||
2. Look for similar issues or PRs on GitHub
|
||||
3. Open a discussion or issue on GitHub
|
||||
|
||||
Thank you for contributing to DeerFlow!
|
||||
@@ -1,101 +0,0 @@
|
||||
# Backend Dockerfile — multi-stage build
|
||||
# Stage 1 (builder): compiles native Python extensions with build-essential
|
||||
# Stage 2 (dev): retains toolchain for dev containers (uv sync at startup)
|
||||
# Stage 3 (runtime): clean image without compiler toolchain for production
|
||||
|
||||
# UV source image (override for restricted networks that cannot reach ghcr.io)
|
||||
ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.7.20
|
||||
FROM ${UV_IMAGE} AS uv-source
|
||||
|
||||
# ── Stage 1: Builder ──────────────────────────────────────────────────────────
|
||||
FROM python:3.12-slim-bookworm AS builder
|
||||
|
||||
ARG NODE_MAJOR=22
|
||||
ARG APT_MIRROR
|
||||
ARG UV_INDEX_URL
|
||||
# Optional extras to install (e.g. "postgres" for PostgreSQL support)
|
||||
# Usage: docker build --build-arg UV_EXTRAS=postgres ...
|
||||
ARG UV_EXTRAS
|
||||
|
||||
# Optionally override apt mirror for restricted networks (e.g. APT_MIRROR=mirrors.aliyun.com)
|
||||
RUN if [ -n "${APT_MIRROR}" ]; then \
|
||||
sed -i "s|deb.debian.org|${APT_MIRROR}|g" /etc/apt/sources.list.d/debian.sources 2>/dev/null || true; \
|
||||
sed -i "s|deb.debian.org|${APT_MIRROR}|g" /etc/apt/sources.list 2>/dev/null || true; \
|
||||
fi
|
||||
|
||||
# Install build tools + Node.js (build-essential needed for native Python extensions)
|
||||
RUN apt-get update && apt-get install -y \
|
||||
curl \
|
||||
build-essential \
|
||||
gnupg \
|
||||
ca-certificates \
|
||||
&& mkdir -p /etc/apt/keyrings \
|
||||
&& curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg \
|
||||
&& echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_${NODE_MAJOR}.x nodistro main" > /etc/apt/sources.list.d/nodesource.list \
|
||||
&& apt-get update \
|
||||
&& apt-get install -y nodejs \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install uv (source image overridable via UV_IMAGE build arg)
|
||||
COPY --from=uv-source /uv /uvx /usr/local/bin/
|
||||
|
||||
# Set working directory
|
||||
WORKDIR /app
|
||||
|
||||
# Copy backend source code
|
||||
COPY backend ./backend
|
||||
|
||||
# Install dependencies with cache mount
|
||||
# When UV_EXTRAS is set (e.g. "postgres"), installs optional dependencies.
|
||||
RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
sh -c "cd backend && UV_INDEX_URL=${UV_INDEX_URL:-https://pypi.org/simple} uv sync ${UV_EXTRAS:+--extra $UV_EXTRAS}"
|
||||
|
||||
# UTF-8 locale prevents UnicodeEncodeError on Chinese/emoji content in minimal
|
||||
# containers where locale configuration may be missing and the default encoding is not UTF-8.
|
||||
ENV LANG=C.UTF-8
|
||||
ENV LC_ALL=C.UTF-8
|
||||
ENV PYTHONIOENCODING=utf-8
|
||||
|
||||
# ── Stage 2: Dev ──────────────────────────────────────────────────────────────
|
||||
# Retains compiler toolchain from builder so startup-time `uv sync` can build
|
||||
# source distributions in development containers.
|
||||
FROM builder AS dev
|
||||
|
||||
# Install Docker CLI (for DooD: allows starting sandbox containers via host Docker socket)
|
||||
COPY --from=docker:cli /usr/local/bin/docker /usr/local/bin/docker
|
||||
|
||||
EXPOSE 8001 2024
|
||||
|
||||
CMD ["sh", "-c", "cd backend && PYTHONPATH=. uv run uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001"]
|
||||
|
||||
# ── Stage 3: Runtime ──────────────────────────────────────────────────────────
|
||||
# Clean image without build-essential — reduces size (~200 MB) and attack surface.
|
||||
FROM python:3.12-slim-bookworm
|
||||
|
||||
ENV LANG=C.UTF-8
|
||||
ENV LC_ALL=C.UTF-8
|
||||
ENV PYTHONIOENCODING=utf-8
|
||||
|
||||
# Copy Node.js runtime from builder (provides npx for MCP servers)
|
||||
COPY --from=builder /usr/bin/node /usr/bin/node
|
||||
COPY --from=builder /usr/lib/node_modules /usr/lib/node_modules
|
||||
RUN ln -s ../lib/node_modules/npm/bin/npm-cli.js /usr/bin/npm \
|
||||
&& ln -s ../lib/node_modules/npm/bin/npx-cli.js /usr/bin/npx
|
||||
|
||||
# Install Docker CLI (for DooD: allows starting sandbox containers via host Docker socket)
|
||||
COPY --from=docker:cli /usr/local/bin/docker /usr/local/bin/docker
|
||||
|
||||
# Install uv (source image overridable via UV_IMAGE build arg)
|
||||
COPY --from=uv-source /uv /uvx /usr/local/bin/
|
||||
|
||||
# Set working directory
|
||||
WORKDIR /app
|
||||
|
||||
# Copy backend with pre-built virtualenv from builder
|
||||
COPY --from=builder /app/backend ./backend
|
||||
|
||||
# Expose ports (gateway: 8001, langgraph: 2024)
|
||||
EXPOSE 8001 2024
|
||||
|
||||
# Default command (can be overridden in docker-compose)
|
||||
CMD ["sh", "-c", "cd backend && PYTHONPATH=. uv run --no-sync uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001"]
|
||||
@@ -1,18 +0,0 @@
|
||||
install:
|
||||
uv sync
|
||||
|
||||
dev:
|
||||
PYTHONPATH=. uv run uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001 --reload
|
||||
|
||||
gateway:
|
||||
PYTHONPATH=. uv run uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001
|
||||
|
||||
test:
|
||||
PYTHONPATH=. uv run pytest tests/ -v
|
||||
|
||||
lint:
|
||||
uvx ruff check .
|
||||
uvx ruff format --check .
|
||||
|
||||
format:
|
||||
uvx ruff check . --fix && uvx ruff format .
|
||||
@@ -1,414 +0,0 @@
|
||||
# DeerFlow Backend
|
||||
|
||||
DeerFlow is a LangGraph-based AI super agent with sandbox execution, persistent memory, and extensible tool integration. The backend enables AI agents to execute code, browse the web, manage files, delegate tasks to subagents, and retain context across conversations - all in isolated, per-thread environments.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────┐
|
||||
│ Nginx (Port 2026) │
|
||||
│ Unified reverse proxy │
|
||||
└───────┬──────────────────┬───────────┘
|
||||
│
|
||||
/api/langgraph/* │ /api/* (other)
|
||||
rewritten to /api/* │
|
||||
▼
|
||||
┌────────────────────────────────────────┐
|
||||
│ Gateway API (8001) │
|
||||
│ FastAPI REST + agent runtime │
|
||||
│ │
|
||||
│ Models, MCP, Skills, Memory, Uploads, │
|
||||
│ Artifacts, Threads, Runs, Streaming │
|
||||
│ │
|
||||
│ ┌────────────────────────────────────┐ │
|
||||
│ │ Lead Agent │ │
|
||||
│ │ Middleware Chain, Tools, Subagents │ │
|
||||
│ └────────────────────────────────────┘ │
|
||||
└────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Request Routing** (via Nginx):
|
||||
- `/api/langgraph/*` → Gateway LangGraph-compatible API - agent interactions, threads, streaming
|
||||
- `/api/*` (other) → Gateway API - models, MCP, skills, memory, artifacts, uploads, thread-local cleanup
|
||||
- `/` (non-API) → Frontend - Next.js web interface
|
||||
|
||||
---
|
||||
|
||||
## Core Components
|
||||
|
||||
### Lead Agent
|
||||
|
||||
The single LangGraph agent (`lead_agent`) is the runtime entry point, created via `make_lead_agent(config)`. It combines:
|
||||
|
||||
- **Dynamic model selection** with thinking and vision support
|
||||
- **Middleware chain** for cross-cutting concerns (9 middlewares)
|
||||
- **Tool system** with sandbox, MCP, community, and built-in tools
|
||||
- **Subagent delegation** for parallel task execution
|
||||
- **System prompt** with skills injection, memory context, and working directory guidance
|
||||
|
||||
### Middleware Chain
|
||||
|
||||
Middlewares execute in strict order, each handling a specific concern:
|
||||
|
||||
| # | Middleware | Purpose |
|
||||
|---|-----------|---------|
|
||||
| 1 | **ThreadDataMiddleware** | Creates per-thread isolated directories (workspace, uploads, outputs) |
|
||||
| 2 | **UploadsMiddleware** | Injects newly uploaded files into conversation context |
|
||||
| 3 | **SandboxMiddleware** | Acquires sandbox environment for code execution |
|
||||
| 4 | **SummarizationMiddleware** | Reduces context when approaching token limits (optional) |
|
||||
| 5 | **TodoListMiddleware** | Tracks multi-step tasks in plan mode (optional) |
|
||||
| 6 | **TitleMiddleware** | Auto-generates conversation titles after first exchange |
|
||||
| 7 | **MemoryMiddleware** | Queues conversations for async memory extraction |
|
||||
| 8 | **ViewImageMiddleware** | Injects image data for vision-capable models (conditional) |
|
||||
| 9 | **ClarificationMiddleware** | Intercepts clarification requests and interrupts execution (must be last) |
|
||||
|
||||
### Sandbox System
|
||||
|
||||
Per-thread isolated execution with virtual path translation:
|
||||
|
||||
- **Abstract interface**: `execute_command`, `read_file`, `write_file`, `list_dir`
|
||||
- **Providers**: `LocalSandboxProvider` (filesystem) and `AioSandboxProvider` (Docker, in community/)
|
||||
- **Virtual paths**: `/mnt/user-data/{workspace,uploads,outputs}` → thread-specific physical directories
|
||||
- **Skills path**: `/mnt/skills` → `deer-flow/skills/` directory
|
||||
- **Skills loading**: Recursively discovers nested `SKILL.md` files under `skills/{public,custom}` and preserves nested container paths
|
||||
- **File-write safety**: `str_replace` serializes read-modify-write per `(sandbox.id, path)` so isolated sandboxes keep concurrency even when virtual paths match
|
||||
- **Tools**: `bash`, `ls`, `read_file`, `write_file`, `str_replace` (`write_file` overwrites by default and exposes `append` for end-of-file writes; `bash` is disabled by default when using `LocalSandboxProvider`; use `AioSandboxProvider` for isolated shell access)
|
||||
|
||||
### Subagent System
|
||||
|
||||
Async task delegation with concurrent execution:
|
||||
|
||||
- **Built-in agents**: `general-purpose` (full toolset) and `bash` (command specialist, exposed only when shell access is available)
|
||||
- **Concurrency**: Max 3 subagents per turn, 15-minute timeout
|
||||
- **Execution**: Background thread pools with status tracking and SSE events
|
||||
- **Flow**: Agent calls `task()` tool → executor runs subagent in background → polls for completion → returns result
|
||||
|
||||
### Memory System
|
||||
|
||||
LLM-powered persistent context retention across conversations:
|
||||
|
||||
- **Automatic extraction**: Analyzes conversations for user context, facts, and preferences
|
||||
- **Structured storage**: User context (work, personal, top-of-mind), history, and confidence-scored facts
|
||||
- **Debounced updates**: Batches updates to minimize LLM calls (configurable wait time)
|
||||
- **System prompt injection**: Top facts + context injected into agent prompts
|
||||
- **Storage**: JSON file with mtime-based cache invalidation
|
||||
|
||||
### Tool Ecosystem
|
||||
|
||||
| Category | Tools |
|
||||
|----------|-------|
|
||||
| **Sandbox** | `bash`, `ls`, `read_file`, `write_file`, `str_replace` |
|
||||
| **Built-in** | `present_files`, `ask_clarification`, `view_image`, `task` (subagent) |
|
||||
| **Community** | Tavily (web search), Jina AI (web fetch), Firecrawl (scraping), DuckDuckGo (image search) |
|
||||
| **MCP** | Any Model Context Protocol server (stdio, SSE, HTTP transports) |
|
||||
| **Skills** | Domain-specific workflows injected via system prompt |
|
||||
|
||||
### Gateway API
|
||||
|
||||
FastAPI application providing REST endpoints for frontend integration:
|
||||
|
||||
| Route | Purpose |
|
||||
|-------|---------|
|
||||
| `GET /api/models` | List available LLM models |
|
||||
| `GET/PUT /api/mcp/config` | Manage MCP server configurations |
|
||||
| `GET/PUT /api/skills` | List and manage skills |
|
||||
| `POST /api/skills/install` | Install skill from `.skill` archive |
|
||||
| `GET /api/memory` | Retrieve memory data |
|
||||
| `POST /api/memory/reload` | Force memory reload |
|
||||
| `GET /api/memory/config` | Memory configuration |
|
||||
| `GET /api/memory/status` | Combined config + data |
|
||||
| `POST /api/threads/{id}/uploads` | Upload files (auto-converts PDF/PPT/Excel/Word to Markdown, rejects directory paths, auto-renames duplicate filenames in one request) |
|
||||
| `GET /api/threads/{id}/uploads/list` | List uploaded files |
|
||||
| `DELETE /api/threads/{id}` | Delete DeerFlow-managed local thread data after LangGraph thread deletion; unexpected failures are logged server-side and return a generic 500 detail |
|
||||
| `GET /api/threads/{id}/artifacts/{path}` | Serve generated artifacts |
|
||||
|
||||
### IM Channels
|
||||
|
||||
The IM bridge supports Feishu, Slack, and Telegram. Slack and Telegram still use the final `runs.wait()` response path, while Feishu now streams through `runs.stream(["messages-tuple", "values"])` and updates a single in-thread card in place.
|
||||
|
||||
For Feishu card updates, DeerFlow stores the running card's `message_id` per inbound message and patches that same card until the run finishes, preserving the existing `OK` / `DONE` reaction flow.
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Python 3.12+
|
||||
- [uv](https://docs.astral.sh/uv/) package manager
|
||||
- API keys for your chosen LLM provider
|
||||
|
||||
### Installation
|
||||
|
||||
```bash
|
||||
cd deer-flow
|
||||
|
||||
# Copy configuration files
|
||||
cp config.example.yaml config.yaml
|
||||
|
||||
# Install backend dependencies
|
||||
cd backend
|
||||
make install
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
Edit `config.yaml` in the project root:
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- name: gpt-4o
|
||||
display_name: GPT-4o
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: gpt-4o
|
||||
api_key: $OPENAI_API_KEY
|
||||
supports_thinking: false
|
||||
supports_vision: true
|
||||
|
||||
- name: gpt-5-responses
|
||||
display_name: GPT-5 (Responses API)
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: gpt-5
|
||||
api_key: $OPENAI_API_KEY
|
||||
use_responses_api: true
|
||||
output_version: responses/v1
|
||||
supports_vision: true
|
||||
```
|
||||
|
||||
Set your API keys:
|
||||
|
||||
```bash
|
||||
export OPENAI_API_KEY="your-api-key-here"
|
||||
```
|
||||
|
||||
### Running
|
||||
|
||||
**Full Application** (from project root):
|
||||
|
||||
```bash
|
||||
make dev # Starts Gateway + Frontend + Nginx
|
||||
```
|
||||
|
||||
Access at: http://localhost:2026
|
||||
|
||||
**Backend Only** (from backend directory):
|
||||
|
||||
```bash
|
||||
# Gateway API + embedded agent runtime
|
||||
make dev
|
||||
```
|
||||
|
||||
Direct access: Gateway at http://localhost:8001
|
||||
|
||||
---
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
backend/
|
||||
├── src/
|
||||
│ ├── agents/ # Agent system
|
||||
│ │ ├── lead_agent/ # Main agent (factory, prompts)
|
||||
│ │ ├── middlewares/ # 9 middleware components
|
||||
│ │ ├── memory/ # Memory extraction & storage
|
||||
│ │ └── thread_state.py # ThreadState schema
|
||||
│ ├── gateway/ # FastAPI Gateway API
|
||||
│ │ ├── app.py # Application setup
|
||||
│ │ └── routers/ # 6 route modules
|
||||
│ ├── sandbox/ # Sandbox execution
|
||||
│ │ ├── local/ # Local filesystem provider
|
||||
│ │ ├── sandbox.py # Abstract interface
|
||||
│ │ ├── tools.py # bash, ls, read/write/str_replace
|
||||
│ │ └── middleware.py # Sandbox lifecycle
|
||||
│ ├── subagents/ # Subagent delegation
|
||||
│ │ ├── builtins/ # general-purpose, bash agents
|
||||
│ │ ├── executor.py # Background execution engine
|
||||
│ │ └── registry.py # Agent registry
|
||||
│ ├── tools/builtins/ # Built-in tools
|
||||
│ ├── mcp/ # MCP protocol integration
|
||||
│ ├── models/ # Model factory
|
||||
│ ├── skills/ # Skill discovery & loading
|
||||
│ ├── config/ # Configuration system
|
||||
│ ├── community/ # Community tools & providers
|
||||
│ ├── reflection/ # Dynamic module loading
|
||||
│ └── utils/ # Utilities
|
||||
├── docs/ # Documentation
|
||||
├── tests/ # Test suite
|
||||
├── langgraph.json # LangGraph graph registry for tooling/Studio compatibility
|
||||
├── pyproject.toml # Python dependencies
|
||||
├── Makefile # Development commands
|
||||
└── Dockerfile # Container build
|
||||
```
|
||||
|
||||
`langgraph.json` is not the default service entrypoint. The scripts and Docker
|
||||
deployments run the Gateway embedded runtime; the file is kept for LangGraph
|
||||
tooling, Studio, or direct LangGraph Server compatibility.
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
### Main Configuration (`config.yaml`)
|
||||
|
||||
Place in project root. Config values starting with `$` resolve as environment variables.
|
||||
|
||||
Key sections:
|
||||
- `models` - LLM configurations with class paths, API keys, thinking/vision flags
|
||||
- `tools` - Tool definitions with module paths and groups
|
||||
- `tool_groups` - Logical tool groupings
|
||||
- `sandbox` - Execution environment provider
|
||||
- `skills` - Skills directory paths
|
||||
- `title` - Auto-title generation settings
|
||||
- `summarization` - Context summarization settings
|
||||
- `subagents` - Subagent system (enabled/disabled)
|
||||
- `memory` - Memory system settings (enabled, storage, debounce, facts limits)
|
||||
|
||||
Provider note:
|
||||
- `models[*].use` references provider classes by module path (for example `langchain_openai:ChatOpenAI`).
|
||||
- If a provider module is missing, DeerFlow now returns an actionable error with install guidance (for example `uv add langchain-google-genai`).
|
||||
|
||||
### Extensions Configuration (`extensions_config.json`)
|
||||
|
||||
MCP servers and skill states in a single file:
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"github": {
|
||||
"enabled": true,
|
||||
"type": "stdio",
|
||||
"command": "npx",
|
||||
"args": ["-y", "@modelcontextprotocol/server-github"],
|
||||
"env": {"GITHUB_TOKEN": "$GITHUB_TOKEN"}
|
||||
},
|
||||
"secure-http": {
|
||||
"enabled": true,
|
||||
"type": "http",
|
||||
"url": "https://api.example.com/mcp",
|
||||
"oauth": {
|
||||
"enabled": true,
|
||||
"token_url": "https://auth.example.com/oauth/token",
|
||||
"grant_type": "client_credentials",
|
||||
"client_id": "$MCP_OAUTH_CLIENT_ID",
|
||||
"client_secret": "$MCP_OAUTH_CLIENT_SECRET"
|
||||
}
|
||||
}
|
||||
},
|
||||
"skills": {
|
||||
"pdf-processing": {"enabled": true}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
- `DEER_FLOW_CONFIG_PATH` - Override config.yaml location
|
||||
- `DEER_FLOW_EXTENSIONS_CONFIG_PATH` - Override extensions_config.json location
|
||||
- Model API keys: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `DEEPSEEK_API_KEY`, etc.
|
||||
- Tool API keys: `TAVILY_API_KEY`, `GITHUB_TOKEN`, etc.
|
||||
|
||||
### LangSmith Tracing
|
||||
|
||||
DeerFlow has built-in [LangSmith](https://smith.langchain.com) integration for observability. When enabled, all LLM calls, agent runs, tool executions, and middleware processing are traced and visible in the LangSmith dashboard.
|
||||
|
||||
**Setup:**
|
||||
|
||||
1. Sign up at [smith.langchain.com](https://smith.langchain.com) and create a project.
|
||||
2. Add the following to your `.env` file in the project root:
|
||||
|
||||
```bash
|
||||
LANGSMITH_TRACING=true
|
||||
LANGSMITH_ENDPOINT=https://api.smith.langchain.com
|
||||
LANGSMITH_API_KEY=lsv2_pt_xxxxxxxxxxxxxxxx
|
||||
LANGSMITH_PROJECT=xxx
|
||||
```
|
||||
|
||||
**Legacy variables:** The `LANGCHAIN_TRACING_V2`, `LANGCHAIN_API_KEY`, `LANGCHAIN_PROJECT`, and `LANGCHAIN_ENDPOINT` variables are also supported for backward compatibility. `LANGSMITH_*` variables take precedence when both are set.
|
||||
|
||||
### Langfuse Tracing
|
||||
|
||||
DeerFlow also supports [Langfuse](https://langfuse.com) observability for LangChain-compatible runs.
|
||||
|
||||
Add the following to your `.env` file:
|
||||
|
||||
```bash
|
||||
LANGFUSE_TRACING=true
|
||||
LANGFUSE_PUBLIC_KEY=pk-lf-xxxxxxxxxxxxxxxx
|
||||
LANGFUSE_SECRET_KEY=sk-lf-xxxxxxxxxxxxxxxx
|
||||
LANGFUSE_BASE_URL=https://cloud.langfuse.com
|
||||
```
|
||||
|
||||
If you are using a self-hosted Langfuse deployment, set `LANGFUSE_BASE_URL` to your Langfuse host.
|
||||
|
||||
### Dual Provider Behavior
|
||||
|
||||
If both LangSmith and Langfuse are enabled, DeerFlow initializes and attaches both callbacks so the same run data is reported to both systems.
|
||||
|
||||
If a provider is explicitly enabled but required credentials are missing, or the provider callback cannot be initialized, DeerFlow raises an error when tracing is initialized during model creation instead of silently disabling tracing.
|
||||
|
||||
**Docker:** In `docker-compose.yaml`, tracing is disabled by default (`LANGSMITH_TRACING=false`). Set `LANGSMITH_TRACING=true` and/or `LANGFUSE_TRACING=true` in your `.env`, together with the required credentials, to enable tracing in containerized deployments.
|
||||
|
||||
---
|
||||
|
||||
## Development
|
||||
|
||||
### Commands
|
||||
|
||||
```bash
|
||||
make install # Install dependencies
|
||||
make dev # Run Gateway API + embedded agent runtime (port 8001)
|
||||
make gateway # Run Gateway API without reload (port 8001)
|
||||
make lint # Run linter (ruff)
|
||||
make format # Format code (ruff)
|
||||
```
|
||||
|
||||
### Code Style
|
||||
|
||||
- **Linter/Formatter**: `ruff`
|
||||
- **Line length**: 240 characters
|
||||
- **Python**: 3.12+ with type hints
|
||||
- **Quotes**: Double quotes
|
||||
- **Indentation**: 4 spaces
|
||||
|
||||
### Testing
|
||||
|
||||
```bash
|
||||
uv run pytest
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Technology Stack
|
||||
|
||||
- **LangGraph** (1.0.6+) - Agent framework and multi-agent orchestration
|
||||
- **LangChain** (1.2.3+) - LLM abstractions and tool system
|
||||
- **FastAPI** (0.115.0+) - Gateway REST API
|
||||
- **langchain-mcp-adapters** - Model Context Protocol support
|
||||
- **agent-sandbox** - Sandboxed code execution
|
||||
- **markitdown** - Multi-format document conversion
|
||||
- **tavily-python** / **firecrawl-py** - Web search and scraping
|
||||
|
||||
---
|
||||
|
||||
## Documentation
|
||||
|
||||
- [Configuration Guide](docs/CONFIGURATION.md)
|
||||
- [Architecture Details](docs/ARCHITECTURE.md)
|
||||
- [API Reference](docs/API.md)
|
||||
- [File Upload](docs/FILE_UPLOAD.md)
|
||||
- [Path Examples](docs/PATH_EXAMPLES.md)
|
||||
- [Context Summarization](docs/summarization.md)
|
||||
- [Plan Mode](docs/plan_mode_usage.md)
|
||||
- [Setup Guide](docs/SETUP.md)
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
See the [LICENSE](../LICENSE) file in the project root.
|
||||
|
||||
## Contributing
|
||||
|
||||
See [CONTRIBUTING.md](CONTRIBUTING.md) for contribution guidelines.
|
||||
@@ -1,16 +0,0 @@
|
||||
"""IM Channel integration for DeerFlow.
|
||||
|
||||
Provides a pluggable channel system that connects external messaging platforms
|
||||
(Feishu/Lark, Slack, Telegram) to the DeerFlow agent via the ChannelManager,
|
||||
which uses ``langgraph-sdk`` to communicate with Gateway's LangGraph-compatible API.
|
||||
"""
|
||||
|
||||
from app.channels.base import Channel
|
||||
from app.channels.message_bus import InboundMessage, MessageBus, OutboundMessage
|
||||
|
||||
__all__ = [
|
||||
"Channel",
|
||||
"InboundMessage",
|
||||
"MessageBus",
|
||||
"OutboundMessage",
|
||||
]
|
||||
@@ -1,130 +0,0 @@
|
||||
"""Abstract base class for IM channels."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import Any
|
||||
|
||||
from app.channels.message_bus import InboundMessage, InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class Channel(ABC):
|
||||
"""Base class for all IM channel implementations.
|
||||
|
||||
Each channel connects to an external messaging platform and:
|
||||
1. Receives messages, wraps them as InboundMessage, publishes to the bus.
|
||||
2. Subscribes to outbound messages and sends replies back to the platform.
|
||||
|
||||
Subclasses must implement ``start``, ``stop``, and ``send``.
|
||||
"""
|
||||
|
||||
def __init__(self, name: str, bus: MessageBus, config: dict[str, Any]) -> None:
|
||||
self.name = name
|
||||
self.bus = bus
|
||||
self.config = config
|
||||
self._running = False
|
||||
|
||||
@property
|
||||
def is_running(self) -> bool:
|
||||
return self._running
|
||||
|
||||
@property
|
||||
def supports_streaming(self) -> bool:
|
||||
return False
|
||||
|
||||
# -- lifecycle ---------------------------------------------------------
|
||||
|
||||
@abstractmethod
|
||||
async def start(self) -> None:
|
||||
"""Start listening for messages from the external platform."""
|
||||
|
||||
@abstractmethod
|
||||
async def stop(self) -> None:
|
||||
"""Gracefully stop the channel."""
|
||||
|
||||
# -- outbound ----------------------------------------------------------
|
||||
|
||||
@abstractmethod
|
||||
async def send(self, msg: OutboundMessage) -> None:
|
||||
"""Send a message back to the external platform.
|
||||
|
||||
The implementation should use ``msg.chat_id`` and ``msg.thread_ts``
|
||||
to route the reply to the correct conversation/thread.
|
||||
"""
|
||||
|
||||
async def send_file(self, msg: OutboundMessage, attachment: ResolvedAttachment) -> bool:
|
||||
"""Upload a single file attachment to the platform.
|
||||
|
||||
Returns True if the upload succeeded, False otherwise.
|
||||
Default implementation returns False (no file upload support).
|
||||
"""
|
||||
return False
|
||||
|
||||
# -- helpers -----------------------------------------------------------
|
||||
|
||||
def _make_inbound(
|
||||
self,
|
||||
chat_id: str,
|
||||
user_id: str,
|
||||
text: str,
|
||||
*,
|
||||
msg_type: InboundMessageType = InboundMessageType.CHAT,
|
||||
thread_ts: str | None = None,
|
||||
files: list[dict[str, Any]] | None = None,
|
||||
metadata: dict[str, Any] | None = None,
|
||||
) -> InboundMessage:
|
||||
"""Convenience factory for creating InboundMessage instances."""
|
||||
return InboundMessage(
|
||||
channel_name=self.name,
|
||||
chat_id=chat_id,
|
||||
user_id=user_id,
|
||||
text=text,
|
||||
msg_type=msg_type,
|
||||
thread_ts=thread_ts,
|
||||
files=files or [],
|
||||
metadata=metadata or {},
|
||||
)
|
||||
|
||||
async def _on_outbound(self, msg: OutboundMessage) -> None:
|
||||
"""Outbound callback registered with the bus.
|
||||
|
||||
Only forwards messages targeted at this channel.
|
||||
Sends the text message first, then uploads any file attachments.
|
||||
File uploads are skipped entirely when the text send fails to avoid
|
||||
partial deliveries (files without accompanying text).
|
||||
"""
|
||||
if msg.channel_name == self.name:
|
||||
try:
|
||||
await self.send(msg)
|
||||
except Exception:
|
||||
logger.exception("Failed to send outbound message on channel %s", self.name)
|
||||
return # Do not attempt file uploads when the text message failed
|
||||
|
||||
for attachment in msg.attachments:
|
||||
try:
|
||||
success = await self.send_file(msg, attachment)
|
||||
if not success:
|
||||
logger.warning("[%s] file upload skipped for %s", self.name, attachment.filename)
|
||||
except Exception:
|
||||
logger.exception("[%s] failed to upload file %s", self.name, attachment.filename)
|
||||
|
||||
async def receive_file(self, msg: InboundMessage, thread_id: str) -> InboundMessage:
|
||||
"""
|
||||
Optionally process and materialize inbound file attachments for this channel.
|
||||
|
||||
By default, this method does nothing and simply returns the original message.
|
||||
Subclasses (e.g. FeishuChannel) may override this to download files (images, documents, etc)
|
||||
referenced in msg.files, save them to the sandbox, and update msg.text to include
|
||||
the sandbox file paths for downstream model consumption.
|
||||
|
||||
Args:
|
||||
msg: The inbound message, possibly containing file metadata in msg.files.
|
||||
thread_id: The resolved DeerFlow thread ID for sandbox path context.
|
||||
|
||||
Returns:
|
||||
The (possibly modified) InboundMessage, with text and/or files updated as needed.
|
||||
"""
|
||||
return msg
|
||||
@@ -1,20 +0,0 @@
|
||||
"""Shared command definitions used by all channel implementations.
|
||||
|
||||
Keeping the authoritative command set in one place ensures that channel
|
||||
parsers (e.g. Feishu) and the ChannelManager dispatcher stay in sync
|
||||
automatically — adding or removing a command here is the single edit
|
||||
required.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
KNOWN_CHANNEL_COMMANDS: frozenset[str] = frozenset(
|
||||
{
|
||||
"/bootstrap",
|
||||
"/new",
|
||||
"/status",
|
||||
"/models",
|
||||
"/memory",
|
||||
"/help",
|
||||
}
|
||||
)
|
||||
@@ -1,740 +0,0 @@
|
||||
"""DingTalk channel implementation."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
from app.channels.base import Channel
|
||||
from app.channels.commands import KNOWN_CHANNEL_COMMANDS
|
||||
from app.channels.message_bus import InboundMessage, InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
DINGTALK_API_BASE = "https://api.dingtalk.com"
|
||||
|
||||
_TOKEN_REFRESH_MARGIN_SECONDS = 300
|
||||
|
||||
_CONVERSATION_TYPE_P2P = "1"
|
||||
_CONVERSATION_TYPE_GROUP = "2"
|
||||
|
||||
_MAX_UPLOAD_SIZE_BYTES = 20 * 1024 * 1024
|
||||
|
||||
|
||||
def _normalize_conversation_type(raw: Any) -> str:
|
||||
"""Normalize ``conversationType`` to ``"1"`` (P2P) or ``"2"`` (group).
|
||||
|
||||
Stream payloads may send int or string values.
|
||||
"""
|
||||
if raw is None:
|
||||
return _CONVERSATION_TYPE_P2P
|
||||
s = str(raw).strip()
|
||||
if s == _CONVERSATION_TYPE_GROUP:
|
||||
return _CONVERSATION_TYPE_GROUP
|
||||
return _CONVERSATION_TYPE_P2P
|
||||
|
||||
|
||||
def _normalize_allowed_users(allowed_users: Any) -> set[str]:
|
||||
if allowed_users is None:
|
||||
return set()
|
||||
if isinstance(allowed_users, str):
|
||||
values = [allowed_users]
|
||||
elif isinstance(allowed_users, (list, tuple, set)):
|
||||
values = allowed_users
|
||||
else:
|
||||
logger.warning(
|
||||
"DingTalk allowed_users should be a list of user IDs; treating %s as one string value",
|
||||
type(allowed_users).__name__,
|
||||
)
|
||||
values = [allowed_users]
|
||||
return {str(uid) for uid in values if str(uid)}
|
||||
|
||||
|
||||
def _is_dingtalk_command(text: str) -> bool:
|
||||
if not text.startswith("/"):
|
||||
return False
|
||||
return text.split(maxsplit=1)[0].lower() in KNOWN_CHANNEL_COMMANDS
|
||||
|
||||
|
||||
def _extract_text_from_rich_text(rich_text_list: list) -> str:
|
||||
parts: list[str] = []
|
||||
for item in rich_text_list:
|
||||
if isinstance(item, dict) and "text" in item:
|
||||
parts.append(item["text"])
|
||||
return " ".join(parts)
|
||||
|
||||
|
||||
_FENCED_CODE_BLOCK_RE = re.compile(r"```(\w*)\n(.*?)```", re.DOTALL)
|
||||
_INLINE_CODE_RE = re.compile(r"`([^`\n]+)`")
|
||||
_HORIZONTAL_RULE_RE = re.compile(r"^-{3,}$", re.MULTILINE)
|
||||
_TABLE_SEPARATOR_RE = re.compile(r"^\|[-:| ]+\|$", re.MULTILINE)
|
||||
|
||||
|
||||
def _convert_markdown_table(text: str) -> str:
|
||||
# DingTalk sampleMarkdown does not render pipe-delimited tables.
|
||||
lines = text.split("\n")
|
||||
result: list[str] = []
|
||||
i = 0
|
||||
while i < len(lines):
|
||||
line = lines[i]
|
||||
# Detect table: header row followed by separator row
|
||||
if i + 1 < len(lines) and line.strip().startswith("|") and _TABLE_SEPARATOR_RE.match(lines[i + 1].strip()):
|
||||
headers = [h.strip() for h in line.strip().strip("|").split("|")]
|
||||
i += 2 # skip header + separator
|
||||
while i < len(lines) and lines[i].strip().startswith("|"):
|
||||
cells = [c.strip() for c in lines[i].strip().strip("|").split("|")]
|
||||
for h, c in zip(headers, cells):
|
||||
result.append(f"> **{h}**: {c}")
|
||||
result.append("")
|
||||
i += 1
|
||||
else:
|
||||
result.append(line)
|
||||
i += 1
|
||||
return "\n".join(result)
|
||||
|
||||
|
||||
def _adapt_markdown_for_dingtalk(text: str) -> str:
|
||||
"""Adapt markdown for DingTalk's limited sampleMarkdown renderer."""
|
||||
|
||||
def _code_block_to_quote(match: re.Match) -> str:
|
||||
lang = match.group(1)
|
||||
code = match.group(2).rstrip("\n")
|
||||
prefix = f"> **{lang}**\n" if lang else ""
|
||||
quoted_lines = "\n".join(f"> {line}" for line in code.split("\n"))
|
||||
return f"{prefix}{quoted_lines}\n"
|
||||
|
||||
text = _FENCED_CODE_BLOCK_RE.sub(_code_block_to_quote, text)
|
||||
text = _INLINE_CODE_RE.sub(r"**\1**", text)
|
||||
text = _convert_markdown_table(text)
|
||||
text = _HORIZONTAL_RULE_RE.sub("───────────", text)
|
||||
return text
|
||||
|
||||
|
||||
class DingTalkChannel(Channel):
|
||||
"""DingTalk IM channel using Stream Push (WebSocket, no public IP needed)."""
|
||||
|
||||
def __init__(self, bus: MessageBus, config: dict[str, Any]) -> None:
|
||||
super().__init__(name="dingtalk", bus=bus, config=config)
|
||||
self._thread: threading.Thread | None = None
|
||||
self._main_loop: asyncio.AbstractEventLoop | None = None
|
||||
self._client_id: str = ""
|
||||
self._client_secret: str = ""
|
||||
self._allowed_users: set[str] = _normalize_allowed_users(config.get("allowed_users"))
|
||||
self._cached_token: str = ""
|
||||
self._token_expires_at: float = 0.0
|
||||
self._token_lock = asyncio.Lock()
|
||||
self._card_template_id: str = config.get("card_template_id", "")
|
||||
self._card_track_ids: dict[str, str] = {}
|
||||
self._dingtalk_client: Any = None
|
||||
self._stream_client: Any = None
|
||||
self._incoming_messages: dict[str, Any] = {}
|
||||
self._incoming_messages_lock = threading.Lock()
|
||||
self._card_repliers: dict[str, Any] = {}
|
||||
|
||||
@property
|
||||
def supports_streaming(self) -> bool:
|
||||
return bool(self._card_template_id)
|
||||
|
||||
async def start(self) -> None:
|
||||
if self._running:
|
||||
return
|
||||
|
||||
try:
|
||||
import dingtalk_stream # noqa: F401
|
||||
except ImportError:
|
||||
logger.error("dingtalk-stream is not installed. Install it with: uv add dingtalk-stream")
|
||||
return
|
||||
|
||||
client_id = self.config.get("client_id", "")
|
||||
client_secret = self.config.get("client_secret", "")
|
||||
|
||||
if not client_id or not client_secret:
|
||||
logger.error("DingTalk channel requires client_id and client_secret")
|
||||
return
|
||||
|
||||
self._client_id = client_id
|
||||
self._client_secret = client_secret
|
||||
self._main_loop = asyncio.get_running_loop()
|
||||
|
||||
if self._card_template_id:
|
||||
logger.info("[DingTalk] AI Card mode enabled (template=%s)", self._card_template_id)
|
||||
|
||||
self._running = True
|
||||
self.bus.subscribe_outbound(self._on_outbound)
|
||||
|
||||
self._thread = threading.Thread(
|
||||
target=self._run_stream,
|
||||
args=(client_id, client_secret),
|
||||
daemon=True,
|
||||
)
|
||||
self._thread.start()
|
||||
logger.info("DingTalk channel started")
|
||||
|
||||
async def stop(self) -> None:
|
||||
self._running = False
|
||||
self.bus.unsubscribe_outbound(self._on_outbound)
|
||||
|
||||
stream_client = self._stream_client
|
||||
if stream_client is not None:
|
||||
try:
|
||||
if hasattr(stream_client, "disconnect"):
|
||||
stream_client.disconnect()
|
||||
except Exception:
|
||||
logger.debug("[DingTalk] error disconnecting stream client", exc_info=True)
|
||||
|
||||
self._dingtalk_client = None
|
||||
self._stream_client = None
|
||||
with self._incoming_messages_lock:
|
||||
self._incoming_messages.clear()
|
||||
self._card_repliers.clear()
|
||||
self._card_track_ids.clear()
|
||||
if self._thread:
|
||||
self._thread.join(timeout=5)
|
||||
self._thread = None
|
||||
logger.info("DingTalk channel stopped")
|
||||
|
||||
def _resolve_routing(self, msg: OutboundMessage) -> tuple[str, str, str]:
|
||||
"""Return (conversation_type, sender_staff_id, conversation_id).
|
||||
|
||||
Uses msg.chat_id as the primary routing key; metadata as fallback.
|
||||
"""
|
||||
conversation_type = _normalize_conversation_type(msg.metadata.get("conversation_type"))
|
||||
sender_staff_id = msg.metadata.get("sender_staff_id", "")
|
||||
conversation_id = msg.metadata.get("conversation_id", "")
|
||||
if conversation_type == _CONVERSATION_TYPE_GROUP:
|
||||
conversation_id = msg.chat_id or conversation_id
|
||||
else:
|
||||
sender_staff_id = msg.chat_id or sender_staff_id
|
||||
return conversation_type, sender_staff_id, conversation_id
|
||||
|
||||
async def send(self, msg: OutboundMessage, *, _max_retries: int = 3) -> None:
|
||||
conversation_type, sender_staff_id, conversation_id = self._resolve_routing(msg)
|
||||
robot_code = self._client_id
|
||||
|
||||
# Card mode: stream update to existing AI card
|
||||
source_key = self._make_card_source_key_from_outbound(msg)
|
||||
out_track_id = self._card_track_ids.get(source_key)
|
||||
|
||||
# ``card_template_id`` enables ``runs.stream`` (non-final + final outbounds).
|
||||
# If card creation failed, skip non-final chunks to avoid duplicate messages.
|
||||
if self._card_template_id and not out_track_id and not msg.is_final:
|
||||
return
|
||||
|
||||
if out_track_id:
|
||||
try:
|
||||
await self._stream_update_card(
|
||||
out_track_id,
|
||||
msg.text,
|
||||
is_finalize=msg.is_final,
|
||||
)
|
||||
except Exception:
|
||||
logger.warning("[DingTalk] card stream failed, falling back to sampleMarkdown")
|
||||
if msg.is_final:
|
||||
self._card_track_ids.pop(source_key, None)
|
||||
self._card_repliers.pop(out_track_id, None)
|
||||
await self._send_markdown_fallback(robot_code, conversation_type, sender_staff_id, conversation_id, msg.text)
|
||||
return
|
||||
if msg.is_final:
|
||||
self._card_track_ids.pop(source_key, None)
|
||||
self._card_repliers.pop(out_track_id, None)
|
||||
return
|
||||
|
||||
# Non-card mode: send sampleMarkdown with retry
|
||||
last_exc: Exception | None = None
|
||||
for attempt in range(_max_retries):
|
||||
try:
|
||||
if conversation_type == _CONVERSATION_TYPE_GROUP:
|
||||
await self._send_group_message(robot_code, conversation_id, msg.text, at_user_ids=[sender_staff_id] if sender_staff_id else None)
|
||||
else:
|
||||
await self._send_p2p_message(robot_code, sender_staff_id, msg.text)
|
||||
return
|
||||
except Exception as exc:
|
||||
last_exc = exc
|
||||
if attempt < _max_retries - 1:
|
||||
delay = 2**attempt
|
||||
logger.warning(
|
||||
"[DingTalk] send failed (attempt %d/%d), retrying in %ds: %s",
|
||||
attempt + 1,
|
||||
_max_retries,
|
||||
delay,
|
||||
exc,
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
|
||||
logger.error("[DingTalk] send failed after %d attempts: %s", _max_retries, last_exc)
|
||||
if last_exc is None:
|
||||
raise RuntimeError("DingTalk send failed without an exception from any attempt")
|
||||
raise last_exc
|
||||
|
||||
async def _send_markdown_fallback(
|
||||
self,
|
||||
robot_code: str,
|
||||
conversation_type: str,
|
||||
sender_staff_id: str,
|
||||
conversation_id: str,
|
||||
text: str,
|
||||
) -> None:
|
||||
try:
|
||||
if conversation_type == _CONVERSATION_TYPE_GROUP:
|
||||
await self._send_group_message(robot_code, conversation_id, text)
|
||||
else:
|
||||
await self._send_p2p_message(robot_code, sender_staff_id, text)
|
||||
except Exception:
|
||||
logger.exception("[DingTalk] markdown fallback also failed")
|
||||
raise
|
||||
|
||||
async def send_file(self, msg: OutboundMessage, attachment: ResolvedAttachment) -> bool:
|
||||
if attachment.size > _MAX_UPLOAD_SIZE_BYTES:
|
||||
logger.warning("[DingTalk] file too large (%d bytes), skipping: %s", attachment.size, attachment.filename)
|
||||
return False
|
||||
|
||||
conversation_type, sender_staff_id, conversation_id = self._resolve_routing(msg)
|
||||
robot_code = self._client_id
|
||||
|
||||
try:
|
||||
media_id = await self._upload_media(attachment.actual_path, "image" if attachment.is_image else "file")
|
||||
if not media_id:
|
||||
return False
|
||||
|
||||
if attachment.is_image:
|
||||
msg_key = "sampleImageMsg"
|
||||
msg_param = json.dumps({"photoURL": media_id})
|
||||
else:
|
||||
msg_key = "sampleFile"
|
||||
msg_param = json.dumps(
|
||||
{
|
||||
"fileUrl": media_id,
|
||||
"fileName": attachment.filename,
|
||||
"fileSize": str(attachment.size),
|
||||
}
|
||||
)
|
||||
|
||||
token = await self._get_access_token()
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(30.0)) as client:
|
||||
if conversation_type == _CONVERSATION_TYPE_GROUP:
|
||||
response = await client.post(
|
||||
f"{DINGTALK_API_BASE}/v1.0/robot/groupMessages/send",
|
||||
headers=self._api_headers(token),
|
||||
json={
|
||||
"msgKey": msg_key,
|
||||
"msgParam": msg_param,
|
||||
"robotCode": robot_code,
|
||||
"openConversationId": conversation_id,
|
||||
},
|
||||
)
|
||||
else:
|
||||
response = await client.post(
|
||||
f"{DINGTALK_API_BASE}/v1.0/robot/oToMessages/batchSend",
|
||||
headers=self._api_headers(token),
|
||||
json={
|
||||
"msgKey": msg_key,
|
||||
"msgParam": msg_param,
|
||||
"robotCode": robot_code,
|
||||
"userIds": [sender_staff_id],
|
||||
},
|
||||
)
|
||||
response.raise_for_status()
|
||||
|
||||
logger.info("[DingTalk] file sent: %s", attachment.filename)
|
||||
return True
|
||||
except (httpx.HTTPError, OSError, ValueError, TypeError, AttributeError):
|
||||
logger.exception("[DingTalk] failed to send file: %s", attachment.filename)
|
||||
return False
|
||||
|
||||
# -- stream client (runs in dedicated thread) --------------------------
|
||||
|
||||
def _run_stream(self, client_id: str, client_secret: str) -> None:
|
||||
try:
|
||||
import dingtalk_stream
|
||||
|
||||
credential = dingtalk_stream.Credential(client_id, client_secret)
|
||||
client = dingtalk_stream.DingTalkStreamClient(credential)
|
||||
self._stream_client = client
|
||||
client.register_callback_handler(
|
||||
dingtalk_stream.chatbot.ChatbotMessage.TOPIC,
|
||||
_DingTalkMessageHandler(self),
|
||||
)
|
||||
client.start_forever()
|
||||
except Exception:
|
||||
if self._running:
|
||||
logger.exception("DingTalk Stream Push error")
|
||||
finally:
|
||||
self._stream_client = None
|
||||
|
||||
def _on_chatbot_message(self, message: Any) -> None:
|
||||
if not self._running:
|
||||
return
|
||||
try:
|
||||
sender_staff_id = message.sender_staff_id or ""
|
||||
conversation_type = _normalize_conversation_type(message.conversation_type)
|
||||
conversation_id = message.conversation_id or ""
|
||||
msg_id = message.message_id or ""
|
||||
sender_nick = message.sender_nick or ""
|
||||
|
||||
if self._allowed_users and sender_staff_id not in self._allowed_users:
|
||||
logger.debug("[DingTalk] ignoring message from non-allowed user: %s", sender_staff_id)
|
||||
return
|
||||
|
||||
text = self._extract_text(message)
|
||||
if not text:
|
||||
logger.info("[DingTalk] empty text, ignoring message")
|
||||
return
|
||||
|
||||
logger.info(
|
||||
"[DingTalk] parsed message: conv_type=%s, msg_id=%s, sender=%s(%s), text=%r",
|
||||
conversation_type,
|
||||
msg_id,
|
||||
sender_staff_id,
|
||||
sender_nick,
|
||||
text[:100],
|
||||
)
|
||||
|
||||
if _is_dingtalk_command(text):
|
||||
msg_type = InboundMessageType.COMMAND
|
||||
else:
|
||||
msg_type = InboundMessageType.CHAT
|
||||
|
||||
# P2P: topic_id=None (single thread per user, like Telegram private chat)
|
||||
# Group: topic_id=msg_id (each new message starts a new topic, like Feishu)
|
||||
topic_id: str | None = msg_id if conversation_type == _CONVERSATION_TYPE_GROUP else None
|
||||
|
||||
# chat_id uses conversation_id for groups, sender_staff_id for P2P
|
||||
chat_id = conversation_id if conversation_type == _CONVERSATION_TYPE_GROUP else sender_staff_id
|
||||
|
||||
inbound = self._make_inbound(
|
||||
chat_id=chat_id,
|
||||
user_id=sender_staff_id,
|
||||
text=text,
|
||||
msg_type=msg_type,
|
||||
thread_ts=msg_id,
|
||||
metadata={
|
||||
"conversation_type": conversation_type,
|
||||
"conversation_id": conversation_id,
|
||||
"sender_staff_id": sender_staff_id,
|
||||
"sender_nick": sender_nick,
|
||||
"message_id": msg_id,
|
||||
},
|
||||
)
|
||||
inbound.topic_id = topic_id
|
||||
|
||||
if self._card_template_id:
|
||||
source_key = self._make_card_source_key(inbound)
|
||||
with self._incoming_messages_lock:
|
||||
self._incoming_messages[source_key] = message
|
||||
|
||||
if self._main_loop and self._main_loop.is_running():
|
||||
logger.info("[DingTalk] publishing inbound message to bus (type=%s, msg_id=%s)", msg_type.value, msg_id)
|
||||
fut = asyncio.run_coroutine_threadsafe(
|
||||
self._prepare_inbound(chat_id, inbound),
|
||||
self._main_loop,
|
||||
)
|
||||
fut.add_done_callback(lambda f, mid=msg_id: self._log_future_error(f, "prepare_inbound", mid))
|
||||
else:
|
||||
logger.warning("[DingTalk] main loop not running, cannot publish inbound message")
|
||||
except Exception:
|
||||
logger.exception("[DingTalk] error processing chatbot message")
|
||||
|
||||
@staticmethod
|
||||
def _extract_text(message: Any) -> str:
|
||||
msg_type = message.message_type
|
||||
if msg_type == "text" and message.text:
|
||||
return message.text.content.strip()
|
||||
if msg_type == "richText" and message.rich_text_content:
|
||||
return _extract_text_from_rich_text(message.rich_text_content.rich_text_list).strip()
|
||||
return ""
|
||||
|
||||
async def _prepare_inbound(self, chat_id: str, inbound: InboundMessage) -> None:
|
||||
# Running reply must finish before publish_inbound so AI card tracks are
|
||||
# registered before the manager emits streaming outbounds.
|
||||
await self._send_running_reply(chat_id, inbound)
|
||||
await self.bus.publish_inbound(inbound)
|
||||
|
||||
async def _send_running_reply(self, chat_id: str, inbound: InboundMessage) -> None:
|
||||
conversation_type = inbound.metadata.get("conversation_type", _CONVERSATION_TYPE_P2P)
|
||||
sender_staff_id = inbound.metadata.get("sender_staff_id", "")
|
||||
conversation_id = inbound.metadata.get("conversation_id", "")
|
||||
text = "\u23f3 Working on it..."
|
||||
|
||||
try:
|
||||
if self._card_template_id:
|
||||
source_key = self._make_card_source_key(inbound)
|
||||
with self._incoming_messages_lock:
|
||||
chatbot_message = self._incoming_messages.pop(source_key, None)
|
||||
out_track_id = await self._create_and_deliver_card(
|
||||
text,
|
||||
chatbot_message=chatbot_message,
|
||||
)
|
||||
if out_track_id:
|
||||
self._card_track_ids[source_key] = out_track_id
|
||||
logger.info("[DingTalk] AI card running reply sent for chat=%s", chat_id)
|
||||
return
|
||||
|
||||
robot_code = self._client_id
|
||||
if conversation_type == _CONVERSATION_TYPE_GROUP:
|
||||
await self._send_text_message_to_group(robot_code, conversation_id, text)
|
||||
else:
|
||||
await self._send_text_message_to_user(robot_code, sender_staff_id, text)
|
||||
logger.info("[DingTalk] 'Working on it...' reply sent for chat=%s", chat_id)
|
||||
except Exception:
|
||||
logger.exception("[DingTalk] failed to send running reply for chat=%s", chat_id)
|
||||
|
||||
# -- DingTalk API helpers ----------------------------------------------
|
||||
|
||||
async def _get_access_token(self) -> str:
|
||||
if self._cached_token and time.monotonic() < self._token_expires_at:
|
||||
return self._cached_token
|
||||
async with self._token_lock:
|
||||
if self._cached_token and time.monotonic() < self._token_expires_at:
|
||||
return self._cached_token
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(10.0)) as client:
|
||||
response = await client.post(
|
||||
f"{DINGTALK_API_BASE}/v1.0/oauth2/accessToken",
|
||||
json={"appKey": self._client_id, "appSecret": self._client_secret}, # DingTalk API field names
|
||||
)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
|
||||
if not isinstance(data, dict):
|
||||
raise ValueError(f"DingTalk access token response must be a JSON object, got {type(data).__name__}")
|
||||
|
||||
access_token = data.get("accessToken")
|
||||
if not isinstance(access_token, str) or not access_token.strip():
|
||||
raise ValueError("DingTalk access token response did not contain a usable accessToken")
|
||||
|
||||
raw_expires_in = data.get("expireIn", 7200)
|
||||
try:
|
||||
expires_in = int(raw_expires_in)
|
||||
except (TypeError, ValueError):
|
||||
logger.warning("[DingTalk] invalid expireIn value %r, using default 7200s", raw_expires_in)
|
||||
expires_in = 7200
|
||||
|
||||
self._cached_token = access_token.strip()
|
||||
self._token_expires_at = time.monotonic() + expires_in - _TOKEN_REFRESH_MARGIN_SECONDS
|
||||
return self._cached_token
|
||||
|
||||
@staticmethod
|
||||
def _api_headers(token: str) -> dict[str, str]:
|
||||
return {
|
||||
"x-acs-dingtalk-access-token": token,
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
async def _send_text_message_to_user(self, robot_code: str, user_id: str, text: str) -> None:
|
||||
token = await self._get_access_token()
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(30.0)) as client:
|
||||
response = await client.post(
|
||||
f"{DINGTALK_API_BASE}/v1.0/robot/oToMessages/batchSend",
|
||||
headers=self._api_headers(token),
|
||||
json={
|
||||
"msgKey": "sampleText",
|
||||
"msgParam": json.dumps({"content": text}),
|
||||
"robotCode": robot_code,
|
||||
"userIds": [user_id],
|
||||
},
|
||||
)
|
||||
response.raise_for_status()
|
||||
|
||||
async def _send_text_message_to_group(self, robot_code: str, conversation_id: str, text: str) -> None:
|
||||
token = await self._get_access_token()
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(30.0)) as client:
|
||||
response = await client.post(
|
||||
f"{DINGTALK_API_BASE}/v1.0/robot/groupMessages/send",
|
||||
headers=self._api_headers(token),
|
||||
json={
|
||||
"msgKey": "sampleText",
|
||||
"msgParam": json.dumps({"content": text}),
|
||||
"robotCode": robot_code,
|
||||
"openConversationId": conversation_id,
|
||||
},
|
||||
)
|
||||
response.raise_for_status()
|
||||
|
||||
async def _send_p2p_message(self, robot_code: str, user_id: str, text: str) -> None:
|
||||
text = _adapt_markdown_for_dingtalk(text)
|
||||
token = await self._get_access_token()
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(30.0)) as client:
|
||||
response = await client.post(
|
||||
f"{DINGTALK_API_BASE}/v1.0/robot/oToMessages/batchSend",
|
||||
headers=self._api_headers(token),
|
||||
json={
|
||||
"msgKey": "sampleMarkdown",
|
||||
"msgParam": json.dumps({"title": "DeerFlow", "text": text}),
|
||||
"robotCode": robot_code,
|
||||
"userIds": [user_id],
|
||||
},
|
||||
)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
if data.get("processQueryKey"):
|
||||
logger.info("[DingTalk] P2P message sent to user=%s", user_id)
|
||||
else:
|
||||
logger.warning("[DingTalk] P2P send response: %s", data)
|
||||
|
||||
async def _send_group_message(
|
||||
self,
|
||||
robot_code: str,
|
||||
conversation_id: str,
|
||||
text: str,
|
||||
*,
|
||||
at_user_ids: list[str] | None = None, # noqa: ARG002
|
||||
) -> None:
|
||||
# at_user_ids accepted for call-site compatibility but not passed to the API
|
||||
# (sampleMarkdown does not support @mentions).
|
||||
text = _adapt_markdown_for_dingtalk(text)
|
||||
token = await self._get_access_token()
|
||||
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(30.0)) as client:
|
||||
response = await client.post(
|
||||
f"{DINGTALK_API_BASE}/v1.0/robot/groupMessages/send",
|
||||
headers=self._api_headers(token),
|
||||
json={
|
||||
"msgKey": "sampleMarkdown",
|
||||
"msgParam": json.dumps({"title": "DeerFlow", "text": text}),
|
||||
"robotCode": robot_code,
|
||||
"openConversationId": conversation_id,
|
||||
},
|
||||
)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
if data.get("processQueryKey"):
|
||||
logger.info("[DingTalk] group message sent to conversation=%s", conversation_id)
|
||||
else:
|
||||
logger.warning("[DingTalk] group send response: %s", data)
|
||||
|
||||
# -- AI Card streaming helpers -------------------------------------------
|
||||
|
||||
def _make_card_source_key(self, inbound: InboundMessage) -> str:
|
||||
m = inbound.metadata
|
||||
return f"{m.get('conversation_type', '')}:{m.get('sender_staff_id', '')}:{m.get('conversation_id', '')}:{m.get('message_id', '')}"
|
||||
|
||||
def _make_card_source_key_from_outbound(self, msg: OutboundMessage) -> str:
|
||||
m = msg.metadata
|
||||
correlation_id = m.get("message_id") or msg.thread_ts or ""
|
||||
return f"{m.get('conversation_type', '')}:{m.get('sender_staff_id', '')}:{m.get('conversation_id', '')}:{correlation_id}"
|
||||
|
||||
async def _create_and_deliver_card(
|
||||
self,
|
||||
initial_text: str,
|
||||
*,
|
||||
chatbot_message: Any = None,
|
||||
) -> str | None:
|
||||
if self._dingtalk_client is None or chatbot_message is None:
|
||||
logger.warning("[DingTalk] SDK client or chatbot_message unavailable, skipping AI card")
|
||||
return None
|
||||
|
||||
try:
|
||||
from dingtalk_stream.card_replier import AICardReplier
|
||||
except ImportError:
|
||||
logger.warning("[DingTalk] dingtalk-stream card_replier not available")
|
||||
return None
|
||||
|
||||
try:
|
||||
replier = AICardReplier(self._dingtalk_client, chatbot_message)
|
||||
card_instance_id = await replier.async_create_and_deliver_card(
|
||||
card_template_id=self._card_template_id,
|
||||
card_data={"content": initial_text},
|
||||
)
|
||||
if not card_instance_id:
|
||||
return None
|
||||
|
||||
self._card_repliers[card_instance_id] = replier
|
||||
logger.info("[DingTalk] AI card created: outTrackId=%s", card_instance_id)
|
||||
return card_instance_id
|
||||
except Exception:
|
||||
logger.exception("[DingTalk] failed to create AI card")
|
||||
return None
|
||||
|
||||
async def _stream_update_card(
|
||||
self,
|
||||
out_track_id: str,
|
||||
content: str,
|
||||
*,
|
||||
is_finalize: bool = False,
|
||||
is_error: bool = False,
|
||||
) -> None:
|
||||
replier = self._card_repliers.get(out_track_id)
|
||||
if not replier:
|
||||
raise RuntimeError(f"No AICardReplier found for track ID {out_track_id}")
|
||||
|
||||
await replier.async_streaming(
|
||||
card_instance_id=out_track_id,
|
||||
content_key="content",
|
||||
content_value=content,
|
||||
append=False,
|
||||
finished=is_finalize,
|
||||
failed=is_error,
|
||||
)
|
||||
|
||||
# -- media upload --------------------------------------------------------
|
||||
|
||||
async def _upload_media(self, file_path: str | Path, media_type: str) -> str | None:
|
||||
try:
|
||||
file_bytes = await asyncio.to_thread(Path(file_path).read_bytes)
|
||||
token = await self._get_access_token()
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(60.0)) as client:
|
||||
response = await client.post(
|
||||
f"{DINGTALK_API_BASE}/v1.0/files/upload",
|
||||
headers={"x-acs-dingtalk-access-token": token},
|
||||
files={"file": ("upload", file_bytes)},
|
||||
data={"type": media_type},
|
||||
)
|
||||
response.raise_for_status()
|
||||
try:
|
||||
payload = response.json()
|
||||
except json.JSONDecodeError:
|
||||
logger.exception("[DingTalk] failed to decode upload response JSON: %s", file_path)
|
||||
return None
|
||||
if not isinstance(payload, dict):
|
||||
logger.warning("[DingTalk] unexpected upload response type %s for %s", type(payload).__name__, file_path)
|
||||
return None
|
||||
return payload.get("mediaId")
|
||||
except (httpx.HTTPError, OSError):
|
||||
logger.exception("[DingTalk] failed to upload media: %s", file_path)
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def _log_future_error(fut: Any, name: str, msg_id: str) -> None:
|
||||
try:
|
||||
exc = fut.exception()
|
||||
if exc:
|
||||
logger.error("[DingTalk] %s failed for msg_id=%s: %s", name, msg_id, exc)
|
||||
except (asyncio.CancelledError, asyncio.InvalidStateError):
|
||||
pass
|
||||
|
||||
|
||||
class _DingTalkMessageHandler:
|
||||
"""Callback handler registered with dingtalk-stream."""
|
||||
|
||||
def __init__(self, channel: DingTalkChannel) -> None:
|
||||
self._channel = channel
|
||||
|
||||
def pre_start(self) -> None:
|
||||
if hasattr(self, "dingtalk_client") and self.dingtalk_client is not None:
|
||||
self._channel._dingtalk_client = self.dingtalk_client
|
||||
|
||||
async def raw_process(self, callback_message: Any) -> Any:
|
||||
import dingtalk_stream
|
||||
from dingtalk_stream.frames import Headers
|
||||
|
||||
code, message = await self.process(callback_message)
|
||||
ack_message = dingtalk_stream.AckMessage()
|
||||
ack_message.code = code
|
||||
ack_message.headers.message_id = callback_message.headers.message_id
|
||||
ack_message.headers.content_type = Headers.CONTENT_TYPE_APPLICATION_JSON
|
||||
ack_message.data = {"response": message}
|
||||
return ack_message
|
||||
|
||||
async def process(self, callback: Any) -> tuple[int, str]:
|
||||
import dingtalk_stream
|
||||
|
||||
incoming_message = dingtalk_stream.ChatbotMessage.from_dict(callback.data)
|
||||
self._channel._on_chatbot_message(incoming_message)
|
||||
return dingtalk_stream.AckMessage.STATUS_OK, "OK"
|
||||
@@ -1,273 +0,0 @@
|
||||
"""Discord channel integration using discord.py."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import threading
|
||||
from typing import Any
|
||||
|
||||
from app.channels.base import Channel
|
||||
from app.channels.message_bus import InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_DISCORD_MAX_MESSAGE_LEN = 2000
|
||||
|
||||
|
||||
class DiscordChannel(Channel):
|
||||
"""Discord bot channel.
|
||||
|
||||
Configuration keys (in ``config.yaml`` under ``channels.discord``):
|
||||
- ``bot_token``: Discord Bot token.
|
||||
- ``allowed_guilds``: (optional) List of allowed Discord guild IDs. Empty = allow all.
|
||||
"""
|
||||
|
||||
def __init__(self, bus: MessageBus, config: dict[str, Any]) -> None:
|
||||
super().__init__(name="discord", bus=bus, config=config)
|
||||
self._bot_token = str(config.get("bot_token", "")).strip()
|
||||
self._allowed_guilds: set[int] = set()
|
||||
for guild_id in config.get("allowed_guilds", []):
|
||||
try:
|
||||
self._allowed_guilds.add(int(guild_id))
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
|
||||
self._client = None
|
||||
self._thread: threading.Thread | None = None
|
||||
self._discord_loop: asyncio.AbstractEventLoop | None = None
|
||||
self._main_loop: asyncio.AbstractEventLoop | None = None
|
||||
self._discord_module = None
|
||||
|
||||
async def start(self) -> None:
|
||||
if self._running:
|
||||
return
|
||||
|
||||
try:
|
||||
import discord
|
||||
except ImportError:
|
||||
logger.error("discord.py is not installed. Install it with: uv add discord.py")
|
||||
return
|
||||
|
||||
if not self._bot_token:
|
||||
logger.error("Discord channel requires bot_token")
|
||||
return
|
||||
|
||||
intents = discord.Intents.default()
|
||||
intents.messages = True
|
||||
intents.guilds = True
|
||||
intents.message_content = True
|
||||
|
||||
client = discord.Client(
|
||||
intents=intents,
|
||||
allowed_mentions=discord.AllowedMentions.none(),
|
||||
)
|
||||
self._client = client
|
||||
self._discord_module = discord
|
||||
self._main_loop = asyncio.get_event_loop()
|
||||
|
||||
@client.event
|
||||
async def on_message(message) -> None:
|
||||
await self._on_message(message)
|
||||
|
||||
self._running = True
|
||||
self.bus.subscribe_outbound(self._on_outbound)
|
||||
|
||||
self._thread = threading.Thread(target=self._run_client, daemon=True)
|
||||
self._thread.start()
|
||||
logger.info("Discord channel started")
|
||||
|
||||
async def stop(self) -> None:
|
||||
self._running = False
|
||||
self.bus.unsubscribe_outbound(self._on_outbound)
|
||||
|
||||
if self._client and self._discord_loop and self._discord_loop.is_running():
|
||||
close_future = asyncio.run_coroutine_threadsafe(self._client.close(), self._discord_loop)
|
||||
try:
|
||||
await asyncio.wait_for(asyncio.wrap_future(close_future), timeout=10)
|
||||
except TimeoutError:
|
||||
logger.warning("[Discord] client close timed out after 10s")
|
||||
except Exception:
|
||||
logger.exception("[Discord] error while closing client")
|
||||
|
||||
if self._thread:
|
||||
self._thread.join(timeout=10)
|
||||
self._thread = None
|
||||
|
||||
self._client = None
|
||||
self._discord_loop = None
|
||||
self._discord_module = None
|
||||
logger.info("Discord channel stopped")
|
||||
|
||||
async def send(self, msg: OutboundMessage) -> None:
|
||||
target = await self._resolve_target(msg)
|
||||
if target is None:
|
||||
logger.error("[Discord] target not found for chat_id=%s thread_ts=%s", msg.chat_id, msg.thread_ts)
|
||||
return
|
||||
|
||||
text = msg.text or ""
|
||||
for chunk in self._split_text(text):
|
||||
send_future = asyncio.run_coroutine_threadsafe(target.send(chunk), self._discord_loop)
|
||||
await asyncio.wrap_future(send_future)
|
||||
|
||||
async def send_file(self, msg: OutboundMessage, attachment: ResolvedAttachment) -> bool:
|
||||
target = await self._resolve_target(msg)
|
||||
if target is None:
|
||||
logger.error("[Discord] target not found for file upload chat_id=%s thread_ts=%s", msg.chat_id, msg.thread_ts)
|
||||
return False
|
||||
|
||||
if self._discord_module is None:
|
||||
return False
|
||||
|
||||
try:
|
||||
fp = open(str(attachment.actual_path), "rb") # noqa: SIM115
|
||||
file = self._discord_module.File(fp, filename=attachment.filename)
|
||||
send_future = asyncio.run_coroutine_threadsafe(target.send(file=file), self._discord_loop)
|
||||
await asyncio.wrap_future(send_future)
|
||||
logger.info("[Discord] file uploaded: %s", attachment.filename)
|
||||
return True
|
||||
except Exception:
|
||||
logger.exception("[Discord] failed to upload file: %s", attachment.filename)
|
||||
return False
|
||||
|
||||
async def _on_message(self, message) -> None:
|
||||
if not self._running or not self._client:
|
||||
return
|
||||
|
||||
if message.author.bot:
|
||||
return
|
||||
|
||||
if self._client.user and message.author.id == self._client.user.id:
|
||||
return
|
||||
|
||||
guild = message.guild
|
||||
if self._allowed_guilds:
|
||||
if guild is None or guild.id not in self._allowed_guilds:
|
||||
return
|
||||
|
||||
text = (message.content or "").strip()
|
||||
if not text:
|
||||
return
|
||||
|
||||
if self._discord_module is None:
|
||||
return
|
||||
|
||||
if isinstance(message.channel, self._discord_module.Thread):
|
||||
chat_id = str(message.channel.parent_id or message.channel.id)
|
||||
thread_id = str(message.channel.id)
|
||||
else:
|
||||
thread = await self._create_thread(message)
|
||||
if thread is None:
|
||||
return
|
||||
chat_id = str(message.channel.id)
|
||||
thread_id = str(thread.id)
|
||||
|
||||
msg_type = InboundMessageType.COMMAND if text.startswith("/") else InboundMessageType.CHAT
|
||||
inbound = self._make_inbound(
|
||||
chat_id=chat_id,
|
||||
user_id=str(message.author.id),
|
||||
text=text,
|
||||
msg_type=msg_type,
|
||||
thread_ts=thread_id,
|
||||
metadata={
|
||||
"guild_id": str(guild.id) if guild else None,
|
||||
"channel_id": str(message.channel.id),
|
||||
"message_id": str(message.id),
|
||||
},
|
||||
)
|
||||
inbound.topic_id = thread_id
|
||||
|
||||
if self._main_loop and self._main_loop.is_running():
|
||||
future = asyncio.run_coroutine_threadsafe(self.bus.publish_inbound(inbound), self._main_loop)
|
||||
future.add_done_callback(lambda f: logger.exception("[Discord] publish_inbound failed", exc_info=f.exception()) if f.exception() else None)
|
||||
|
||||
def _run_client(self) -> None:
|
||||
self._discord_loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(self._discord_loop)
|
||||
try:
|
||||
self._discord_loop.run_until_complete(self._client.start(self._bot_token))
|
||||
except Exception:
|
||||
if self._running:
|
||||
logger.exception("Discord client error")
|
||||
finally:
|
||||
try:
|
||||
if self._client and not self._client.is_closed():
|
||||
self._discord_loop.run_until_complete(self._client.close())
|
||||
except Exception:
|
||||
logger.exception("Error during Discord shutdown")
|
||||
|
||||
async def _create_thread(self, message):
|
||||
try:
|
||||
thread_name = f"deerflow-{message.author.display_name}-{message.id}"[:100]
|
||||
return await message.create_thread(name=thread_name)
|
||||
except Exception:
|
||||
logger.exception("[Discord] failed to create thread for message=%s (threads may be disabled or missing permissions)", message.id)
|
||||
try:
|
||||
await message.channel.send("Could not create a thread for your message. Please check that threads are enabled in this channel.")
|
||||
except Exception:
|
||||
pass
|
||||
return None
|
||||
|
||||
async def _resolve_target(self, msg: OutboundMessage):
|
||||
if not self._client or not self._discord_loop:
|
||||
return None
|
||||
|
||||
target_ids: list[str] = []
|
||||
if msg.thread_ts:
|
||||
target_ids.append(msg.thread_ts)
|
||||
if msg.chat_id and msg.chat_id not in target_ids:
|
||||
target_ids.append(msg.chat_id)
|
||||
|
||||
for raw_id in target_ids:
|
||||
target = await self._get_channel_or_thread(raw_id)
|
||||
if target is not None:
|
||||
return target
|
||||
return None
|
||||
|
||||
async def _get_channel_or_thread(self, raw_id: str):
|
||||
if not self._client or not self._discord_loop:
|
||||
return None
|
||||
|
||||
try:
|
||||
target_id = int(raw_id)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
get_future = asyncio.run_coroutine_threadsafe(self._fetch_channel(target_id), self._discord_loop)
|
||||
try:
|
||||
return await asyncio.wrap_future(get_future)
|
||||
except Exception:
|
||||
logger.exception("[Discord] failed to resolve target id=%s", raw_id)
|
||||
return None
|
||||
|
||||
async def _fetch_channel(self, target_id: int):
|
||||
if not self._client:
|
||||
return None
|
||||
|
||||
channel = self._client.get_channel(target_id)
|
||||
if channel is not None:
|
||||
return channel
|
||||
|
||||
try:
|
||||
return await self._client.fetch_channel(target_id)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def _split_text(text: str) -> list[str]:
|
||||
if not text:
|
||||
return [""]
|
||||
|
||||
chunks: list[str] = []
|
||||
remaining = text
|
||||
while len(remaining) > _DISCORD_MAX_MESSAGE_LEN:
|
||||
split_at = remaining.rfind("\n", 0, _DISCORD_MAX_MESSAGE_LEN)
|
||||
if split_at <= 0:
|
||||
split_at = _DISCORD_MAX_MESSAGE_LEN
|
||||
chunks.append(remaining[:split_at])
|
||||
remaining = remaining[split_at:].lstrip("\n")
|
||||
|
||||
if remaining:
|
||||
chunks.append(remaining)
|
||||
|
||||
return chunks
|
||||
@@ -1,698 +0,0 @@
|
||||
"""Feishu/Lark channel — connects to Feishu via WebSocket (no public IP needed)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import threading
|
||||
from typing import Any, Literal
|
||||
|
||||
from app.channels.base import Channel
|
||||
from app.channels.commands import KNOWN_CHANNEL_COMMANDS
|
||||
from app.channels.message_bus import InboundMessage, InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
|
||||
from deerflow.config.paths import VIRTUAL_PATH_PREFIX, get_paths
|
||||
from deerflow.runtime.user_context import get_effective_user_id
|
||||
from deerflow.sandbox.sandbox_provider import get_sandbox_provider
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _is_feishu_command(text: str) -> bool:
|
||||
if not text.startswith("/"):
|
||||
return False
|
||||
return text.split(maxsplit=1)[0].lower() in KNOWN_CHANNEL_COMMANDS
|
||||
|
||||
|
||||
class FeishuChannel(Channel):
|
||||
"""Feishu/Lark IM channel using the ``lark-oapi`` WebSocket client.
|
||||
|
||||
Configuration keys (in ``config.yaml`` under ``channels.feishu``):
|
||||
- ``app_id``: Feishu app ID.
|
||||
- ``app_secret``: Feishu app secret.
|
||||
- ``verification_token``: (optional) Event verification token.
|
||||
|
||||
The channel uses WebSocket long-connection mode so no public IP is required.
|
||||
|
||||
Message flow:
|
||||
1. User sends a message → bot adds "OK" emoji reaction
|
||||
2. Bot replies in thread: "Working on it......"
|
||||
3. Agent processes the message and returns a result
|
||||
4. Bot replies in thread with the result
|
||||
5. Bot adds "DONE" emoji reaction to the original message
|
||||
"""
|
||||
|
||||
def __init__(self, bus: MessageBus, config: dict[str, Any]) -> None:
|
||||
super().__init__(name="feishu", bus=bus, config=config)
|
||||
self._thread: threading.Thread | None = None
|
||||
self._main_loop: asyncio.AbstractEventLoop | None = None
|
||||
self._api_client = None
|
||||
self._CreateMessageReactionRequest = None
|
||||
self._CreateMessageReactionRequestBody = None
|
||||
self._Emoji = None
|
||||
self._PatchMessageRequest = None
|
||||
self._PatchMessageRequestBody = None
|
||||
self._background_tasks: set[asyncio.Task] = set()
|
||||
self._running_card_ids: dict[str, str] = {}
|
||||
self._running_card_tasks: dict[str, asyncio.Task] = {}
|
||||
self._CreateFileRequest = None
|
||||
self._CreateFileRequestBody = None
|
||||
self._CreateImageRequest = None
|
||||
self._CreateImageRequestBody = None
|
||||
self._GetMessageResourceRequest = None
|
||||
self._thread_lock = threading.Lock()
|
||||
|
||||
@property
|
||||
def supports_streaming(self) -> bool:
|
||||
return True
|
||||
|
||||
async def start(self) -> None:
|
||||
if self._running:
|
||||
return
|
||||
|
||||
try:
|
||||
import lark_oapi as lark
|
||||
from lark_oapi.api.im.v1 import (
|
||||
CreateFileRequest,
|
||||
CreateFileRequestBody,
|
||||
CreateImageRequest,
|
||||
CreateImageRequestBody,
|
||||
CreateMessageReactionRequest,
|
||||
CreateMessageReactionRequestBody,
|
||||
CreateMessageRequest,
|
||||
CreateMessageRequestBody,
|
||||
Emoji,
|
||||
GetMessageResourceRequest,
|
||||
PatchMessageRequest,
|
||||
PatchMessageRequestBody,
|
||||
ReplyMessageRequest,
|
||||
ReplyMessageRequestBody,
|
||||
)
|
||||
except ImportError:
|
||||
logger.error("lark-oapi is not installed. Install it with: uv add lark-oapi")
|
||||
return
|
||||
|
||||
self._lark = lark
|
||||
self._CreateMessageRequest = CreateMessageRequest
|
||||
self._CreateMessageRequestBody = CreateMessageRequestBody
|
||||
self._ReplyMessageRequest = ReplyMessageRequest
|
||||
self._ReplyMessageRequestBody = ReplyMessageRequestBody
|
||||
self._CreateMessageReactionRequest = CreateMessageReactionRequest
|
||||
self._CreateMessageReactionRequestBody = CreateMessageReactionRequestBody
|
||||
self._Emoji = Emoji
|
||||
self._PatchMessageRequest = PatchMessageRequest
|
||||
self._PatchMessageRequestBody = PatchMessageRequestBody
|
||||
self._CreateFileRequest = CreateFileRequest
|
||||
self._CreateFileRequestBody = CreateFileRequestBody
|
||||
self._CreateImageRequest = CreateImageRequest
|
||||
self._CreateImageRequestBody = CreateImageRequestBody
|
||||
self._GetMessageResourceRequest = GetMessageResourceRequest
|
||||
|
||||
app_id = self.config.get("app_id", "")
|
||||
app_secret = self.config.get("app_secret", "")
|
||||
domain = self.config.get("domain", "https://open.feishu.cn")
|
||||
|
||||
if not app_id or not app_secret:
|
||||
logger.error("Feishu channel requires app_id and app_secret")
|
||||
return
|
||||
|
||||
self._api_client = lark.Client.builder().app_id(app_id).app_secret(app_secret).domain(domain).build()
|
||||
logger.info("[Feishu] using domain: %s", domain)
|
||||
self._main_loop = asyncio.get_event_loop()
|
||||
|
||||
self._running = True
|
||||
self.bus.subscribe_outbound(self._on_outbound)
|
||||
|
||||
# Both ws.Client construction and start() must happen in a dedicated
|
||||
# thread with its own event loop. lark-oapi caches the running loop
|
||||
# at construction time and later calls loop.run_until_complete(),
|
||||
# which conflicts with an already-running uvloop.
|
||||
self._thread = threading.Thread(
|
||||
target=self._run_ws,
|
||||
args=(app_id, app_secret, domain),
|
||||
daemon=True,
|
||||
)
|
||||
self._thread.start()
|
||||
logger.info("Feishu channel started")
|
||||
|
||||
def _run_ws(self, app_id: str, app_secret: str, domain: str) -> None:
|
||||
"""Construct and run the lark WS client in a thread with a fresh event loop.
|
||||
|
||||
The lark-oapi SDK captures a module-level event loop at import time
|
||||
(``lark_oapi.ws.client.loop``). When uvicorn uses uvloop, that
|
||||
captured loop is the *main* thread's uvloop — which is already
|
||||
running, so ``loop.run_until_complete()`` inside ``Client.start()``
|
||||
raises ``RuntimeError``.
|
||||
|
||||
We work around this by creating a plain asyncio event loop for this
|
||||
thread and patching the SDK's module-level reference before calling
|
||||
``start()``.
|
||||
"""
|
||||
loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(loop)
|
||||
try:
|
||||
import lark_oapi as lark
|
||||
import lark_oapi.ws.client as _ws_client_mod
|
||||
|
||||
# Replace the SDK's module-level loop so Client.start() uses
|
||||
# this thread's (non-running) event loop instead of the main
|
||||
# thread's uvloop.
|
||||
_ws_client_mod.loop = loop
|
||||
|
||||
event_handler = lark.EventDispatcherHandler.builder("", "").register_p2_im_message_receive_v1(self._on_message).build()
|
||||
ws_client = lark.ws.Client(
|
||||
app_id=app_id,
|
||||
app_secret=app_secret,
|
||||
event_handler=event_handler,
|
||||
log_level=lark.LogLevel.INFO,
|
||||
domain=domain,
|
||||
)
|
||||
ws_client.start()
|
||||
except Exception:
|
||||
if self._running:
|
||||
logger.exception("Feishu WebSocket error")
|
||||
|
||||
async def stop(self) -> None:
|
||||
self._running = False
|
||||
self.bus.unsubscribe_outbound(self._on_outbound)
|
||||
for task in list(self._background_tasks):
|
||||
task.cancel()
|
||||
self._background_tasks.clear()
|
||||
for task in list(self._running_card_tasks.values()):
|
||||
task.cancel()
|
||||
self._running_card_tasks.clear()
|
||||
if self._thread:
|
||||
self._thread.join(timeout=5)
|
||||
self._thread = None
|
||||
logger.info("Feishu channel stopped")
|
||||
|
||||
async def send(self, msg: OutboundMessage, *, _max_retries: int = 3) -> None:
|
||||
if not self._api_client:
|
||||
logger.warning("[Feishu] send called but no api_client available")
|
||||
return
|
||||
|
||||
logger.info(
|
||||
"[Feishu] sending reply: chat_id=%s, thread_ts=%s, text_len=%d",
|
||||
msg.chat_id,
|
||||
msg.thread_ts,
|
||||
len(msg.text),
|
||||
)
|
||||
|
||||
last_exc: Exception | None = None
|
||||
for attempt in range(_max_retries):
|
||||
try:
|
||||
await self._send_card_message(msg)
|
||||
return # success
|
||||
except Exception as exc:
|
||||
last_exc = exc
|
||||
if attempt < _max_retries - 1:
|
||||
delay = 2**attempt # 1s, 2s
|
||||
logger.warning(
|
||||
"[Feishu] send failed (attempt %d/%d), retrying in %ds: %s",
|
||||
attempt + 1,
|
||||
_max_retries,
|
||||
delay,
|
||||
exc,
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
|
||||
logger.error("[Feishu] send failed after %d attempts: %s", _max_retries, last_exc)
|
||||
if last_exc is None:
|
||||
raise RuntimeError("Feishu send failed without an exception from any attempt")
|
||||
raise last_exc
|
||||
|
||||
async def send_file(self, msg: OutboundMessage, attachment: ResolvedAttachment) -> bool:
|
||||
if not self._api_client:
|
||||
return False
|
||||
|
||||
# Check size limits (image: 10MB, file: 30MB)
|
||||
if attachment.is_image and attachment.size > 10 * 1024 * 1024:
|
||||
logger.warning("[Feishu] image too large (%d bytes), skipping: %s", attachment.size, attachment.filename)
|
||||
return False
|
||||
if not attachment.is_image and attachment.size > 30 * 1024 * 1024:
|
||||
logger.warning("[Feishu] file too large (%d bytes), skipping: %s", attachment.size, attachment.filename)
|
||||
return False
|
||||
|
||||
try:
|
||||
if attachment.is_image:
|
||||
file_key = await self._upload_image(attachment.actual_path)
|
||||
msg_type = "image"
|
||||
content = json.dumps({"image_key": file_key})
|
||||
else:
|
||||
file_key = await self._upload_file(attachment.actual_path, attachment.filename)
|
||||
msg_type = "file"
|
||||
content = json.dumps({"file_key": file_key})
|
||||
|
||||
if msg.thread_ts:
|
||||
request = self._ReplyMessageRequest.builder().message_id(msg.thread_ts).request_body(self._ReplyMessageRequestBody.builder().msg_type(msg_type).content(content).reply_in_thread(True).build()).build()
|
||||
await asyncio.to_thread(self._api_client.im.v1.message.reply, request)
|
||||
else:
|
||||
request = self._CreateMessageRequest.builder().receive_id_type("chat_id").request_body(self._CreateMessageRequestBody.builder().receive_id(msg.chat_id).msg_type(msg_type).content(content).build()).build()
|
||||
await asyncio.to_thread(self._api_client.im.v1.message.create, request)
|
||||
|
||||
logger.info("[Feishu] file sent: %s (type=%s)", attachment.filename, msg_type)
|
||||
return True
|
||||
except Exception:
|
||||
logger.exception("[Feishu] failed to upload/send file: %s", attachment.filename)
|
||||
return False
|
||||
|
||||
async def _upload_image(self, path) -> str:
|
||||
"""Upload an image to Feishu and return the image_key."""
|
||||
with open(str(path), "rb") as f:
|
||||
request = self._CreateImageRequest.builder().request_body(self._CreateImageRequestBody.builder().image_type("message").image(f).build()).build()
|
||||
response = await asyncio.to_thread(self._api_client.im.v1.image.create, request)
|
||||
if not response.success():
|
||||
raise RuntimeError(f"Feishu image upload failed: code={response.code}, msg={response.msg}")
|
||||
return response.data.image_key
|
||||
|
||||
async def _upload_file(self, path, filename: str) -> str:
|
||||
"""Upload a file to Feishu and return the file_key."""
|
||||
suffix = path.suffix.lower() if hasattr(path, "suffix") else ""
|
||||
if suffix in (".xls", ".xlsx", ".csv"):
|
||||
file_type = "xls"
|
||||
elif suffix in (".ppt", ".pptx"):
|
||||
file_type = "ppt"
|
||||
elif suffix == ".pdf":
|
||||
file_type = "pdf"
|
||||
elif suffix in (".doc", ".docx"):
|
||||
file_type = "doc"
|
||||
else:
|
||||
file_type = "stream"
|
||||
|
||||
with open(str(path), "rb") as f:
|
||||
request = self._CreateFileRequest.builder().request_body(self._CreateFileRequestBody.builder().file_type(file_type).file_name(filename).file(f).build()).build()
|
||||
response = await asyncio.to_thread(self._api_client.im.v1.file.create, request)
|
||||
if not response.success():
|
||||
raise RuntimeError(f"Feishu file upload failed: code={response.code}, msg={response.msg}")
|
||||
return response.data.file_key
|
||||
|
||||
async def receive_file(self, msg: InboundMessage, thread_id: str) -> InboundMessage:
|
||||
"""Download a Feishu file into the thread uploads directory.
|
||||
|
||||
Returns the sandbox virtual path when the image is persisted successfully.
|
||||
"""
|
||||
if not msg.thread_ts:
|
||||
logger.warning("[Feishu] received file message without thread_ts, cannot associate with conversation: %s", msg)
|
||||
return msg
|
||||
files = msg.files
|
||||
if not files:
|
||||
logger.warning("[Feishu] received message with no files: %s", msg)
|
||||
return msg
|
||||
text = msg.text
|
||||
for file in files:
|
||||
if file.get("image_key"):
|
||||
virtual_path = await self._receive_single_file(msg.thread_ts, file["image_key"], "image", thread_id)
|
||||
text = text.replace("[image]", virtual_path, 1)
|
||||
elif file.get("file_key"):
|
||||
virtual_path = await self._receive_single_file(msg.thread_ts, file["file_key"], "file", thread_id)
|
||||
text = text.replace("[file]", virtual_path, 1)
|
||||
msg.text = text
|
||||
return msg
|
||||
|
||||
async def _receive_single_file(self, message_id: str, file_key: str, type: Literal["image", "file"], thread_id: str) -> str:
|
||||
request = self._GetMessageResourceRequest.builder().message_id(message_id).file_key(file_key).type(type).build()
|
||||
|
||||
def inner():
|
||||
return self._api_client.im.v1.message_resource.get(request)
|
||||
|
||||
try:
|
||||
response = await asyncio.to_thread(inner)
|
||||
except Exception:
|
||||
logger.exception("[Feishu] resource get request failed for resource_key=%s type=%s", file_key, type)
|
||||
return f"Failed to obtain the [{type}]"
|
||||
|
||||
if not response.success():
|
||||
logger.warning(
|
||||
"[Feishu] resource get failed: resource_key=%s, type=%s, code=%s, msg=%s, log_id=%s ",
|
||||
file_key,
|
||||
type,
|
||||
response.code,
|
||||
response.msg,
|
||||
response.get_log_id(),
|
||||
)
|
||||
return f"Failed to obtain the [{type}]"
|
||||
|
||||
image_stream = getattr(response, "file", None)
|
||||
if image_stream is None:
|
||||
logger.warning("[Feishu] resource get returned no file stream: resource_key=%s, type=%s", file_key, type)
|
||||
return f"Failed to obtain the [{type}]"
|
||||
|
||||
try:
|
||||
content: bytes = await asyncio.to_thread(image_stream.read)
|
||||
except Exception:
|
||||
logger.exception("[Feishu] failed to read resource stream: resource_key=%s, type=%s", file_key, type)
|
||||
return f"Failed to obtain the [{type}]"
|
||||
|
||||
if not content:
|
||||
logger.warning("[Feishu] empty resource content: resource_key=%s, type=%s", file_key, type)
|
||||
return f"Failed to obtain the [{type}]"
|
||||
|
||||
paths = get_paths()
|
||||
user_id = get_effective_user_id()
|
||||
paths.ensure_thread_dirs(thread_id, user_id=user_id)
|
||||
uploads_dir = paths.sandbox_uploads_dir(thread_id, user_id=user_id).resolve()
|
||||
|
||||
ext = "png" if type == "image" else "bin"
|
||||
raw_filename = getattr(response, "file_name", "") or f"feishu_{file_key[-12:]}.{ext}"
|
||||
|
||||
# Sanitize filename: preserve extension, replace path chars in name part
|
||||
if "." in raw_filename:
|
||||
name_part, ext = raw_filename.rsplit(".", 1)
|
||||
name_part = re.sub(r"[./\\]", "_", name_part)
|
||||
filename = f"{name_part}.{ext}"
|
||||
else:
|
||||
filename = re.sub(r"[./\\]", "_", raw_filename)
|
||||
resolved_target = uploads_dir / filename
|
||||
|
||||
def down_load():
|
||||
# use thread_lock to avoid filename conflicts when writing
|
||||
with self._thread_lock:
|
||||
resolved_target.write_bytes(content)
|
||||
|
||||
try:
|
||||
await asyncio.to_thread(down_load)
|
||||
except Exception:
|
||||
logger.exception("[Feishu] failed to persist downloaded resource: %s, type=%s", resolved_target, type)
|
||||
return f"Failed to obtain the [{type}]"
|
||||
|
||||
virtual_path = f"{VIRTUAL_PATH_PREFIX}/uploads/{resolved_target.name}"
|
||||
|
||||
try:
|
||||
sandbox_provider = get_sandbox_provider()
|
||||
sandbox_id = sandbox_provider.acquire(thread_id)
|
||||
if sandbox_id != "local":
|
||||
sandbox = sandbox_provider.get(sandbox_id)
|
||||
if sandbox is None:
|
||||
logger.warning("[Feishu] sandbox not found for thread_id=%s", thread_id)
|
||||
return f"Failed to obtain the [{type}]"
|
||||
sandbox.update_file(virtual_path, content)
|
||||
except Exception:
|
||||
logger.exception("[Feishu] failed to sync resource into non-local sandbox: %s", virtual_path)
|
||||
return f"Failed to obtain the [{type}]"
|
||||
|
||||
logger.info("[Feishu] downloaded resource mapped: file_key=%s -> %s", file_key, virtual_path)
|
||||
return virtual_path
|
||||
|
||||
# -- message formatting ------------------------------------------------
|
||||
|
||||
@staticmethod
|
||||
def _build_card_content(text: str) -> str:
|
||||
"""Build a Feishu interactive card with markdown content.
|
||||
|
||||
Feishu's interactive card format natively renders markdown, including
|
||||
headers, bold/italic, code blocks, lists, and links.
|
||||
"""
|
||||
card = {
|
||||
"config": {"wide_screen_mode": True, "update_multi": True},
|
||||
"elements": [{"tag": "markdown", "content": text}],
|
||||
}
|
||||
return json.dumps(card)
|
||||
|
||||
# -- reaction helpers --------------------------------------------------
|
||||
|
||||
async def _add_reaction(self, message_id: str, emoji_type: str = "THUMBSUP") -> None:
|
||||
"""Add an emoji reaction to a message."""
|
||||
if not self._api_client or not self._CreateMessageReactionRequest:
|
||||
return
|
||||
try:
|
||||
request = self._CreateMessageReactionRequest.builder().message_id(message_id).request_body(self._CreateMessageReactionRequestBody.builder().reaction_type(self._Emoji.builder().emoji_type(emoji_type).build()).build()).build()
|
||||
await asyncio.to_thread(self._api_client.im.v1.message_reaction.create, request)
|
||||
logger.info("[Feishu] reaction '%s' added to message %s", emoji_type, message_id)
|
||||
except Exception:
|
||||
logger.exception("[Feishu] failed to add reaction '%s' to message %s", emoji_type, message_id)
|
||||
|
||||
async def _reply_card(self, message_id: str, text: str) -> str | None:
|
||||
"""Reply with an interactive card and return the created card message ID."""
|
||||
if not self._api_client:
|
||||
return None
|
||||
|
||||
content = self._build_card_content(text)
|
||||
request = self._ReplyMessageRequest.builder().message_id(message_id).request_body(self._ReplyMessageRequestBody.builder().msg_type("interactive").content(content).reply_in_thread(True).build()).build()
|
||||
response = await asyncio.to_thread(self._api_client.im.v1.message.reply, request)
|
||||
response_data = getattr(response, "data", None)
|
||||
return getattr(response_data, "message_id", None)
|
||||
|
||||
async def _create_card(self, chat_id: str, text: str) -> None:
|
||||
"""Create a new card message in the target chat."""
|
||||
if not self._api_client:
|
||||
return
|
||||
|
||||
content = self._build_card_content(text)
|
||||
request = self._CreateMessageRequest.builder().receive_id_type("chat_id").request_body(self._CreateMessageRequestBody.builder().receive_id(chat_id).msg_type("interactive").content(content).build()).build()
|
||||
await asyncio.to_thread(self._api_client.im.v1.message.create, request)
|
||||
|
||||
async def _update_card(self, message_id: str, text: str) -> None:
|
||||
"""Patch an existing card message in place."""
|
||||
if not self._api_client or not self._PatchMessageRequest:
|
||||
return
|
||||
|
||||
content = self._build_card_content(text)
|
||||
request = self._PatchMessageRequest.builder().message_id(message_id).request_body(self._PatchMessageRequestBody.builder().content(content).build()).build()
|
||||
await asyncio.to_thread(self._api_client.im.v1.message.patch, request)
|
||||
|
||||
def _track_background_task(self, task: asyncio.Task, *, name: str, msg_id: str) -> None:
|
||||
"""Keep a strong reference to fire-and-forget tasks and surface errors."""
|
||||
self._background_tasks.add(task)
|
||||
task.add_done_callback(lambda done_task, task_name=name, mid=msg_id: self._finalize_background_task(done_task, task_name, mid))
|
||||
|
||||
def _finalize_background_task(self, task: asyncio.Task, name: str, msg_id: str) -> None:
|
||||
self._background_tasks.discard(task)
|
||||
self._log_task_error(task, name, msg_id)
|
||||
|
||||
async def _create_running_card(self, source_message_id: str, text: str) -> str | None:
|
||||
"""Create the running card and cache its message ID when available."""
|
||||
running_card_id = await self._reply_card(source_message_id, text)
|
||||
if running_card_id:
|
||||
self._running_card_ids[source_message_id] = running_card_id
|
||||
logger.info("[Feishu] running card created: source=%s card=%s", source_message_id, running_card_id)
|
||||
else:
|
||||
logger.warning("[Feishu] running card creation returned no message_id for source=%s, subsequent updates will fall back to new replies", source_message_id)
|
||||
return running_card_id
|
||||
|
||||
def _ensure_running_card_started(self, source_message_id: str, text: str = "Working on it...") -> asyncio.Task | None:
|
||||
"""Start running-card creation once per source message."""
|
||||
running_card_id = self._running_card_ids.get(source_message_id)
|
||||
if running_card_id:
|
||||
return None
|
||||
|
||||
running_card_task = self._running_card_tasks.get(source_message_id)
|
||||
if running_card_task:
|
||||
return running_card_task
|
||||
|
||||
running_card_task = asyncio.create_task(self._create_running_card(source_message_id, text))
|
||||
self._running_card_tasks[source_message_id] = running_card_task
|
||||
running_card_task.add_done_callback(lambda done_task, mid=source_message_id: self._finalize_running_card_task(mid, done_task))
|
||||
return running_card_task
|
||||
|
||||
def _finalize_running_card_task(self, source_message_id: str, task: asyncio.Task) -> None:
|
||||
if self._running_card_tasks.get(source_message_id) is task:
|
||||
self._running_card_tasks.pop(source_message_id, None)
|
||||
self._log_task_error(task, "create_running_card", source_message_id)
|
||||
|
||||
async def _ensure_running_card(self, source_message_id: str, text: str = "Working on it...") -> str | None:
|
||||
"""Ensure the in-thread running card exists and track its message ID."""
|
||||
running_card_id = self._running_card_ids.get(source_message_id)
|
||||
if running_card_id:
|
||||
return running_card_id
|
||||
|
||||
running_card_task = self._ensure_running_card_started(source_message_id, text)
|
||||
if running_card_task is None:
|
||||
return self._running_card_ids.get(source_message_id)
|
||||
return await running_card_task
|
||||
|
||||
async def _send_running_reply(self, message_id: str) -> None:
|
||||
"""Reply to a message in-thread with a running card."""
|
||||
try:
|
||||
await self._ensure_running_card(message_id)
|
||||
except Exception:
|
||||
logger.exception("[Feishu] failed to send running reply for message %s", message_id)
|
||||
|
||||
async def _send_card_message(self, msg: OutboundMessage) -> None:
|
||||
"""Send or update the Feishu card tied to the current request."""
|
||||
source_message_id = msg.thread_ts
|
||||
if source_message_id:
|
||||
running_card_id = self._running_card_ids.get(source_message_id)
|
||||
awaited_running_card_task = False
|
||||
|
||||
if not running_card_id:
|
||||
running_card_task = self._running_card_tasks.get(source_message_id)
|
||||
if running_card_task:
|
||||
awaited_running_card_task = True
|
||||
running_card_id = await running_card_task
|
||||
|
||||
if running_card_id:
|
||||
try:
|
||||
await self._update_card(running_card_id, msg.text)
|
||||
except Exception:
|
||||
if not msg.is_final:
|
||||
raise
|
||||
logger.exception(
|
||||
"[Feishu] failed to patch running card %s, falling back to final reply",
|
||||
running_card_id,
|
||||
)
|
||||
await self._reply_card(source_message_id, msg.text)
|
||||
else:
|
||||
logger.info("[Feishu] running card updated: source=%s card=%s", source_message_id, running_card_id)
|
||||
elif msg.is_final:
|
||||
await self._reply_card(source_message_id, msg.text)
|
||||
elif awaited_running_card_task:
|
||||
logger.warning(
|
||||
"[Feishu] running card task finished without message_id for source=%s, skipping duplicate non-final creation",
|
||||
source_message_id,
|
||||
)
|
||||
else:
|
||||
await self._ensure_running_card(source_message_id, msg.text)
|
||||
|
||||
if msg.is_final:
|
||||
self._running_card_ids.pop(source_message_id, None)
|
||||
await self._add_reaction(source_message_id, "DONE")
|
||||
return
|
||||
|
||||
await self._create_card(msg.chat_id, msg.text)
|
||||
|
||||
# -- internal ----------------------------------------------------------
|
||||
|
||||
@staticmethod
|
||||
def _log_future_error(fut, name: str, msg_id: str) -> None:
|
||||
"""Callback for run_coroutine_threadsafe futures to surface errors."""
|
||||
try:
|
||||
exc = fut.exception()
|
||||
if exc:
|
||||
logger.error("[Feishu] %s failed for msg_id=%s: %s", name, msg_id, exc)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
@staticmethod
|
||||
def _log_task_error(task: asyncio.Task, name: str, msg_id: str) -> None:
|
||||
"""Callback for background asyncio tasks to surface errors."""
|
||||
try:
|
||||
exc = task.exception()
|
||||
if exc:
|
||||
logger.error("[Feishu] %s failed for msg_id=%s: %s", name, msg_id, exc)
|
||||
except asyncio.CancelledError:
|
||||
logger.info("[Feishu] %s cancelled for msg_id=%s", name, msg_id)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
async def _prepare_inbound(self, msg_id: str, inbound) -> None:
|
||||
"""Kick off Feishu side effects without delaying inbound dispatch."""
|
||||
reaction_task = asyncio.create_task(self._add_reaction(msg_id, "OK"))
|
||||
self._track_background_task(reaction_task, name="add_reaction", msg_id=msg_id)
|
||||
self._ensure_running_card_started(msg_id)
|
||||
await self.bus.publish_inbound(inbound)
|
||||
|
||||
def _on_message(self, event) -> None:
|
||||
"""Called by lark-oapi when a message is received (runs in lark thread)."""
|
||||
try:
|
||||
logger.info("[Feishu] raw event received: type=%s", type(event).__name__)
|
||||
message = event.event.message
|
||||
chat_id = message.chat_id
|
||||
msg_id = message.message_id
|
||||
sender_id = event.event.sender.sender_id.open_id
|
||||
|
||||
# root_id is set when the message is a reply within a Feishu thread.
|
||||
# Use it as topic_id so all replies share the same DeerFlow thread.
|
||||
root_id = getattr(message, "root_id", None) or None
|
||||
|
||||
# Parse message content
|
||||
content = json.loads(message.content)
|
||||
|
||||
# files_list store the any-file-key in feishu messages, which can be used to download the file content later
|
||||
# In Feishu channel, image_keys are independent of file_keys.
|
||||
# The file_key includes files, videos, and audio, but does not include stickers.
|
||||
files_list = []
|
||||
|
||||
if "text" in content:
|
||||
# Handle plain text messages
|
||||
text = content["text"]
|
||||
elif "file_key" in content:
|
||||
file_key = content.get("file_key")
|
||||
if isinstance(file_key, str) and file_key:
|
||||
files_list.append({"file_key": file_key})
|
||||
text = "[file]"
|
||||
else:
|
||||
text = ""
|
||||
elif "image_key" in content:
|
||||
image_key = content.get("image_key")
|
||||
if isinstance(image_key, str) and image_key:
|
||||
files_list.append({"image_key": image_key})
|
||||
text = "[image]"
|
||||
else:
|
||||
text = ""
|
||||
elif "content" in content and isinstance(content["content"], list):
|
||||
# Handle rich-text messages with a top-level "content" list (e.g., topic groups/posts)
|
||||
text_paragraphs: list[str] = []
|
||||
for paragraph in content["content"]:
|
||||
if isinstance(paragraph, list):
|
||||
paragraph_text_parts: list[str] = []
|
||||
for element in paragraph:
|
||||
if isinstance(element, dict):
|
||||
# Include both normal text and @ mentions
|
||||
if element.get("tag") in ("text", "at"):
|
||||
text_value = element.get("text", "")
|
||||
if text_value:
|
||||
paragraph_text_parts.append(text_value)
|
||||
elif element.get("tag") == "img":
|
||||
image_key = element.get("image_key")
|
||||
if isinstance(image_key, str) and image_key:
|
||||
files_list.append({"image_key": image_key})
|
||||
paragraph_text_parts.append("[image]")
|
||||
elif element.get("tag") in ("file", "media"):
|
||||
file_key = element.get("file_key")
|
||||
if isinstance(file_key, str) and file_key:
|
||||
files_list.append({"file_key": file_key})
|
||||
paragraph_text_parts.append("[file]")
|
||||
if paragraph_text_parts:
|
||||
# Join text segments within a paragraph with spaces to avoid "helloworld"
|
||||
text_paragraphs.append(" ".join(paragraph_text_parts))
|
||||
|
||||
# Join paragraphs with blank lines to preserve paragraph boundaries
|
||||
text = "\n\n".join(text_paragraphs)
|
||||
else:
|
||||
text = ""
|
||||
text = text.strip()
|
||||
|
||||
logger.info(
|
||||
"[Feishu] parsed message: chat_id=%s, msg_id=%s, root_id=%s, sender=%s, text=%r",
|
||||
chat_id,
|
||||
msg_id,
|
||||
root_id,
|
||||
sender_id,
|
||||
text[:100] if text else "",
|
||||
)
|
||||
|
||||
if not (text or files_list):
|
||||
logger.info("[Feishu] empty text, ignoring message")
|
||||
return
|
||||
|
||||
# Only treat known slash commands as commands; absolute paths and
|
||||
# other slash-prefixed text should be handled as normal chat.
|
||||
if _is_feishu_command(text):
|
||||
msg_type = InboundMessageType.COMMAND
|
||||
else:
|
||||
msg_type = InboundMessageType.CHAT
|
||||
|
||||
# topic_id: use root_id for replies (same topic), msg_id for new messages (new topic)
|
||||
topic_id = root_id or msg_id
|
||||
|
||||
inbound = self._make_inbound(
|
||||
chat_id=chat_id,
|
||||
user_id=sender_id,
|
||||
text=text,
|
||||
msg_type=msg_type,
|
||||
thread_ts=msg_id,
|
||||
files=files_list,
|
||||
metadata={"message_id": msg_id, "root_id": root_id},
|
||||
)
|
||||
inbound.topic_id = topic_id
|
||||
|
||||
# Schedule on the async event loop
|
||||
if self._main_loop and self._main_loop.is_running():
|
||||
logger.info("[Feishu] publishing inbound message to bus (type=%s, msg_id=%s)", msg_type.value, msg_id)
|
||||
fut = asyncio.run_coroutine_threadsafe(self._prepare_inbound(msg_id, inbound), self._main_loop)
|
||||
fut.add_done_callback(lambda f, mid=msg_id: self._log_future_error(f, "prepare_inbound", mid))
|
||||
else:
|
||||
logger.warning("[Feishu] main loop not running, cannot publish inbound message")
|
||||
except Exception:
|
||||
logger.exception("[Feishu] error processing message")
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,173 +0,0 @@
|
||||
"""MessageBus — async pub/sub hub that decouples channels from the agent dispatcher."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import time
|
||||
from collections.abc import Callable, Coroutine
|
||||
from dataclasses import dataclass, field
|
||||
from enum import StrEnum
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Message types
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class InboundMessageType(StrEnum):
|
||||
"""Types of messages arriving from IM channels."""
|
||||
|
||||
CHAT = "chat"
|
||||
COMMAND = "command"
|
||||
|
||||
|
||||
@dataclass
|
||||
class InboundMessage:
|
||||
"""A message arriving from an IM channel toward the agent dispatcher.
|
||||
|
||||
Attributes:
|
||||
channel_name: Name of the source channel (e.g. "feishu", "slack").
|
||||
chat_id: Platform-specific chat/conversation identifier.
|
||||
user_id: Platform-specific user identifier.
|
||||
text: The message text.
|
||||
msg_type: Whether this is a regular chat message or a command.
|
||||
thread_ts: Optional platform thread identifier (for threaded replies).
|
||||
topic_id: Conversation topic identifier used to map to a DeerFlow thread.
|
||||
Messages sharing the same ``topic_id`` within a ``chat_id`` will
|
||||
reuse the same DeerFlow thread. When ``None``, each message
|
||||
creates a new thread (one-shot Q&A).
|
||||
files: Optional list of file attachments (platform-specific dicts).
|
||||
metadata: Arbitrary extra data from the channel.
|
||||
created_at: Unix timestamp when the message was created.
|
||||
"""
|
||||
|
||||
channel_name: str
|
||||
chat_id: str
|
||||
user_id: str
|
||||
text: str
|
||||
msg_type: InboundMessageType = InboundMessageType.CHAT
|
||||
thread_ts: str | None = None
|
||||
topic_id: str | None = None
|
||||
files: list[dict[str, Any]] = field(default_factory=list)
|
||||
metadata: dict[str, Any] = field(default_factory=dict)
|
||||
created_at: float = field(default_factory=time.time)
|
||||
|
||||
|
||||
@dataclass
|
||||
class ResolvedAttachment:
|
||||
"""A file attachment resolved to a host filesystem path, ready for upload.
|
||||
|
||||
Attributes:
|
||||
virtual_path: Original virtual path (e.g. /mnt/user-data/outputs/report.pdf).
|
||||
actual_path: Resolved host filesystem path.
|
||||
filename: Basename of the file.
|
||||
mime_type: MIME type (e.g. "application/pdf").
|
||||
size: File size in bytes.
|
||||
is_image: True for image/* MIME types (platforms may handle images differently).
|
||||
"""
|
||||
|
||||
virtual_path: str
|
||||
actual_path: Path
|
||||
filename: str
|
||||
mime_type: str
|
||||
size: int
|
||||
is_image: bool
|
||||
|
||||
|
||||
@dataclass
|
||||
class OutboundMessage:
|
||||
"""A message from the agent dispatcher back to a channel.
|
||||
|
||||
Attributes:
|
||||
channel_name: Target channel name (used for routing).
|
||||
chat_id: Target chat/conversation identifier.
|
||||
thread_id: DeerFlow thread ID that produced this response.
|
||||
text: The response text.
|
||||
artifacts: List of artifact paths produced by the agent.
|
||||
is_final: Whether this is the final message in the response stream.
|
||||
thread_ts: Optional platform thread identifier for threaded replies.
|
||||
metadata: Arbitrary extra data.
|
||||
created_at: Unix timestamp.
|
||||
"""
|
||||
|
||||
channel_name: str
|
||||
chat_id: str
|
||||
thread_id: str
|
||||
text: str
|
||||
artifacts: list[str] = field(default_factory=list)
|
||||
attachments: list[ResolvedAttachment] = field(default_factory=list)
|
||||
is_final: bool = True
|
||||
thread_ts: str | None = None
|
||||
metadata: dict[str, Any] = field(default_factory=dict)
|
||||
created_at: float = field(default_factory=time.time)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# MessageBus
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
OutboundCallback = Callable[[OutboundMessage], Coroutine[Any, Any, None]]
|
||||
|
||||
|
||||
class MessageBus:
|
||||
"""Async pub/sub hub connecting channels and the agent dispatcher.
|
||||
|
||||
Channels publish inbound messages; the dispatcher consumes them.
|
||||
The dispatcher publishes outbound messages; channels receive them
|
||||
via registered callbacks.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._inbound_queue: asyncio.Queue[InboundMessage] = asyncio.Queue()
|
||||
self._outbound_listeners: list[OutboundCallback] = []
|
||||
|
||||
# -- inbound -----------------------------------------------------------
|
||||
|
||||
async def publish_inbound(self, msg: InboundMessage) -> None:
|
||||
"""Enqueue an inbound message from a channel."""
|
||||
await self._inbound_queue.put(msg)
|
||||
logger.info(
|
||||
"[Bus] inbound enqueued: channel=%s, chat_id=%s, type=%s, queue_size=%d",
|
||||
msg.channel_name,
|
||||
msg.chat_id,
|
||||
msg.msg_type.value,
|
||||
self._inbound_queue.qsize(),
|
||||
)
|
||||
|
||||
async def get_inbound(self) -> InboundMessage:
|
||||
"""Block until the next inbound message is available."""
|
||||
return await self._inbound_queue.get()
|
||||
|
||||
@property
|
||||
def inbound_queue(self) -> asyncio.Queue[InboundMessage]:
|
||||
return self._inbound_queue
|
||||
|
||||
# -- outbound ----------------------------------------------------------
|
||||
|
||||
def subscribe_outbound(self, callback: OutboundCallback) -> None:
|
||||
"""Register an async callback for outbound messages."""
|
||||
self._outbound_listeners.append(callback)
|
||||
|
||||
def unsubscribe_outbound(self, callback: OutboundCallback) -> None:
|
||||
"""Remove a previously registered outbound callback."""
|
||||
self._outbound_listeners = [cb for cb in self._outbound_listeners if cb is not callback]
|
||||
|
||||
async def publish_outbound(self, msg: OutboundMessage) -> None:
|
||||
"""Dispatch an outbound message to all registered listeners."""
|
||||
logger.info(
|
||||
"[Bus] outbound dispatching: channel=%s, chat_id=%s, listeners=%d, text_len=%d",
|
||||
msg.channel_name,
|
||||
msg.chat_id,
|
||||
len(self._outbound_listeners),
|
||||
len(msg.text),
|
||||
)
|
||||
for callback in self._outbound_listeners:
|
||||
try:
|
||||
await callback(msg)
|
||||
except Exception:
|
||||
logger.exception("Error in outbound callback for channel=%s", msg.channel_name)
|
||||
@@ -1,230 +0,0 @@
|
||||
"""ChannelService — manages the lifecycle of all IM channels."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from app.channels.base import Channel
|
||||
from app.channels.manager import DEFAULT_GATEWAY_URL, DEFAULT_LANGGRAPH_URL, ChannelManager
|
||||
from app.channels.message_bus import MessageBus
|
||||
from app.channels.store import ChannelStore
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from deerflow.config.app_config import AppConfig
|
||||
|
||||
# Channel name → import path for lazy loading
|
||||
_CHANNEL_REGISTRY: dict[str, str] = {
|
||||
"dingtalk": "app.channels.dingtalk:DingTalkChannel",
|
||||
"discord": "app.channels.discord:DiscordChannel",
|
||||
"feishu": "app.channels.feishu:FeishuChannel",
|
||||
"slack": "app.channels.slack:SlackChannel",
|
||||
"telegram": "app.channels.telegram:TelegramChannel",
|
||||
"wechat": "app.channels.wechat:WechatChannel",
|
||||
"wecom": "app.channels.wecom:WeComChannel",
|
||||
}
|
||||
|
||||
# Keys that indicate a user has configured credentials for a channel.
|
||||
_CHANNEL_CREDENTIAL_KEYS: dict[str, list[str]] = {
|
||||
"dingtalk": ["client_id", "client_secret"],
|
||||
"discord": ["bot_token"],
|
||||
"feishu": ["app_id", "app_secret"],
|
||||
"slack": ["bot_token", "app_token"],
|
||||
"telegram": ["bot_token"],
|
||||
"wecom": ["bot_id", "bot_secret"],
|
||||
"wechat": ["bot_token"],
|
||||
}
|
||||
|
||||
_CHANNELS_LANGGRAPH_URL_ENV = "DEER_FLOW_CHANNELS_LANGGRAPH_URL"
|
||||
_CHANNELS_GATEWAY_URL_ENV = "DEER_FLOW_CHANNELS_GATEWAY_URL"
|
||||
|
||||
|
||||
def _resolve_service_url(config: dict[str, Any], config_key: str, env_key: str, default: str) -> str:
|
||||
value = config.pop(config_key, None)
|
||||
if isinstance(value, str) and value.strip():
|
||||
return value
|
||||
env_value = os.getenv(env_key, "").strip()
|
||||
if env_value:
|
||||
return env_value
|
||||
return default
|
||||
|
||||
|
||||
class ChannelService:
|
||||
"""Manages the lifecycle of all configured IM channels.
|
||||
|
||||
Reads configuration from ``config.yaml`` under the ``channels`` key,
|
||||
instantiates enabled channels, and starts the ChannelManager dispatcher.
|
||||
"""
|
||||
|
||||
def __init__(self, channels_config: dict[str, Any] | None = None) -> None:
|
||||
self.bus = MessageBus()
|
||||
self.store = ChannelStore()
|
||||
config = dict(channels_config or {})
|
||||
langgraph_url = _resolve_service_url(config, "langgraph_url", _CHANNELS_LANGGRAPH_URL_ENV, DEFAULT_LANGGRAPH_URL)
|
||||
gateway_url = _resolve_service_url(config, "gateway_url", _CHANNELS_GATEWAY_URL_ENV, DEFAULT_GATEWAY_URL)
|
||||
default_session = config.pop("session", None)
|
||||
channel_sessions = {name: channel_config.get("session") for name, channel_config in config.items() if isinstance(channel_config, dict)}
|
||||
self.manager = ChannelManager(
|
||||
bus=self.bus,
|
||||
store=self.store,
|
||||
langgraph_url=langgraph_url,
|
||||
gateway_url=gateway_url,
|
||||
default_session=default_session if isinstance(default_session, dict) else None,
|
||||
channel_sessions=channel_sessions,
|
||||
)
|
||||
self._channels: dict[str, Any] = {} # name -> Channel instance
|
||||
self._config = config
|
||||
self._running = False
|
||||
|
||||
@classmethod
|
||||
def from_app_config(cls, app_config: AppConfig | None = None) -> ChannelService:
|
||||
"""Create a ChannelService from the application config."""
|
||||
if app_config is None:
|
||||
from deerflow.config.app_config import get_app_config
|
||||
|
||||
app_config = get_app_config()
|
||||
channels_config = {}
|
||||
# extra fields are allowed by AppConfig (extra="allow")
|
||||
extra = app_config.model_extra or {}
|
||||
if "channels" in extra:
|
||||
channels_config = extra["channels"]
|
||||
return cls(channels_config=channels_config)
|
||||
|
||||
async def start(self) -> None:
|
||||
"""Start the manager and all enabled channels."""
|
||||
if self._running:
|
||||
return
|
||||
|
||||
await self.manager.start()
|
||||
|
||||
for name, channel_config in self._config.items():
|
||||
if not isinstance(channel_config, dict):
|
||||
continue
|
||||
if not channel_config.get("enabled", False):
|
||||
cred_keys = _CHANNEL_CREDENTIAL_KEYS.get(name, [])
|
||||
has_creds = any(not isinstance(channel_config.get(k), bool) and channel_config.get(k) is not None and str(channel_config[k]).strip() for k in cred_keys)
|
||||
if has_creds:
|
||||
logger.warning(
|
||||
"Channel '%s' has credentials configured but is disabled. Set enabled: true under channels.%s in config.yaml to activate it.",
|
||||
name,
|
||||
name,
|
||||
)
|
||||
else:
|
||||
logger.info("Channel %s is disabled, skipping", name)
|
||||
continue
|
||||
|
||||
await self._start_channel(name, channel_config)
|
||||
|
||||
self._running = True
|
||||
logger.info("ChannelService started with channels: %s", list(self._channels.keys()))
|
||||
|
||||
async def stop(self) -> None:
|
||||
"""Stop all channels and the manager."""
|
||||
for name, channel in list(self._channels.items()):
|
||||
try:
|
||||
await channel.stop()
|
||||
logger.info("Channel %s stopped", name)
|
||||
except Exception:
|
||||
logger.exception("Error stopping channel %s", name)
|
||||
self._channels.clear()
|
||||
|
||||
await self.manager.stop()
|
||||
self._running = False
|
||||
logger.info("ChannelService stopped")
|
||||
|
||||
async def restart_channel(self, name: str) -> bool:
|
||||
"""Restart a specific channel. Returns True if successful."""
|
||||
if name in self._channels:
|
||||
try:
|
||||
await self._channels[name].stop()
|
||||
except Exception:
|
||||
logger.exception("Error stopping channel %s for restart", name)
|
||||
del self._channels[name]
|
||||
|
||||
config = self._config.get(name)
|
||||
if not config or not isinstance(config, dict):
|
||||
logger.warning("No config for channel %s", name)
|
||||
return False
|
||||
|
||||
return await self._start_channel(name, config)
|
||||
|
||||
async def _start_channel(self, name: str, config: dict[str, Any]) -> bool:
|
||||
"""Instantiate and start a single channel."""
|
||||
import_path = _CHANNEL_REGISTRY.get(name)
|
||||
if not import_path:
|
||||
logger.warning("Unknown channel type: %s", name)
|
||||
return False
|
||||
|
||||
try:
|
||||
from deerflow.reflection import resolve_class
|
||||
|
||||
channel_cls = resolve_class(import_path, base_class=None)
|
||||
except Exception:
|
||||
logger.exception("Failed to import channel class for %s", name)
|
||||
return False
|
||||
|
||||
try:
|
||||
channel = channel_cls(bus=self.bus, config=config)
|
||||
self._channels[name] = channel
|
||||
await channel.start()
|
||||
if not channel.is_running:
|
||||
self._channels.pop(name, None)
|
||||
logger.error("Channel %s did not enter a running state after start()", name)
|
||||
return False
|
||||
logger.info("Channel %s started", name)
|
||||
return True
|
||||
except Exception:
|
||||
self._channels.pop(name, None)
|
||||
logger.exception("Failed to start channel %s", name)
|
||||
return False
|
||||
|
||||
def get_status(self) -> dict[str, Any]:
|
||||
"""Return status information for all channels."""
|
||||
channels_status = {}
|
||||
for name in _CHANNEL_REGISTRY:
|
||||
config = self._config.get(name, {})
|
||||
enabled = isinstance(config, dict) and config.get("enabled", False)
|
||||
running = name in self._channels and self._channels[name].is_running
|
||||
channels_status[name] = {
|
||||
"enabled": enabled,
|
||||
"running": running,
|
||||
}
|
||||
return {
|
||||
"service_running": self._running,
|
||||
"channels": channels_status,
|
||||
}
|
||||
|
||||
def get_channel(self, name: str) -> Channel | None:
|
||||
"""Return a running channel instance by name when available."""
|
||||
return self._channels.get(name)
|
||||
|
||||
|
||||
# -- singleton access -------------------------------------------------------
|
||||
|
||||
_channel_service: ChannelService | None = None
|
||||
|
||||
|
||||
def get_channel_service() -> ChannelService | None:
|
||||
"""Get the singleton ChannelService instance (if started)."""
|
||||
return _channel_service
|
||||
|
||||
|
||||
async def start_channel_service(app_config: AppConfig | None = None) -> ChannelService:
|
||||
"""Create and start the global ChannelService from app config."""
|
||||
global _channel_service
|
||||
if _channel_service is not None:
|
||||
return _channel_service
|
||||
_channel_service = ChannelService.from_app_config(app_config)
|
||||
await _channel_service.start()
|
||||
return _channel_service
|
||||
|
||||
|
||||
async def stop_channel_service() -> None:
|
||||
"""Stop the global ChannelService."""
|
||||
global _channel_service
|
||||
if _channel_service is not None:
|
||||
await _channel_service.stop()
|
||||
_channel_service = None
|
||||
@@ -1,264 +0,0 @@
|
||||
"""Slack channel — connects via Socket Mode (no public IP needed)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from markdown_to_mrkdwn import SlackMarkdownConverter
|
||||
|
||||
from app.channels.base import Channel
|
||||
from app.channels.message_bus import InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_slack_md_converter = SlackMarkdownConverter()
|
||||
|
||||
|
||||
def _normalize_allowed_users(allowed_users: Any) -> set[str]:
|
||||
if allowed_users is None:
|
||||
return set()
|
||||
if isinstance(allowed_users, str):
|
||||
values = [allowed_users]
|
||||
elif isinstance(allowed_users, list | tuple | set):
|
||||
values = allowed_users
|
||||
else:
|
||||
logger.warning(
|
||||
"Slack allowed_users should be a list of Slack user IDs or a single Slack user ID string; treating %s as one string value",
|
||||
type(allowed_users).__name__,
|
||||
)
|
||||
values = [allowed_users]
|
||||
return {str(user_id) for user_id in values if str(user_id)}
|
||||
|
||||
|
||||
class SlackChannel(Channel):
|
||||
"""Slack IM channel using Socket Mode (WebSocket, no public IP).
|
||||
|
||||
Configuration keys (in ``config.yaml`` under ``channels.slack``):
|
||||
- ``bot_token``: Slack Bot User OAuth Token (xoxb-...).
|
||||
- ``app_token``: Slack App-Level Token (xapp-...) for Socket Mode.
|
||||
- ``allowed_users``: (optional) List of allowed Slack user IDs, or a
|
||||
single Slack user ID string as shorthand. Empty = allow all. Other
|
||||
scalar values are treated as a single string with a warning.
|
||||
"""
|
||||
|
||||
def __init__(self, bus: MessageBus, config: dict[str, Any]) -> None:
|
||||
super().__init__(name="slack", bus=bus, config=config)
|
||||
self._socket_client = None
|
||||
self._web_client = None
|
||||
self._loop: asyncio.AbstractEventLoop | None = None
|
||||
self._allowed_users = _normalize_allowed_users(config.get("allowed_users", []))
|
||||
|
||||
async def start(self) -> None:
|
||||
if self._running:
|
||||
return
|
||||
|
||||
try:
|
||||
from slack_sdk import WebClient
|
||||
from slack_sdk.socket_mode import SocketModeClient
|
||||
from slack_sdk.socket_mode.response import SocketModeResponse
|
||||
except ImportError:
|
||||
logger.error("slack-sdk is not installed. Install it with: uv add slack-sdk")
|
||||
return
|
||||
|
||||
self._SocketModeResponse = SocketModeResponse
|
||||
|
||||
bot_token = self.config.get("bot_token", "")
|
||||
app_token = self.config.get("app_token", "")
|
||||
|
||||
if not bot_token or not app_token:
|
||||
logger.error("Slack channel requires bot_token and app_token")
|
||||
return
|
||||
|
||||
self._web_client = WebClient(token=bot_token)
|
||||
self._socket_client = SocketModeClient(
|
||||
app_token=app_token,
|
||||
web_client=self._web_client,
|
||||
)
|
||||
self._loop = asyncio.get_event_loop()
|
||||
|
||||
self._socket_client.socket_mode_request_listeners.append(self._on_socket_event)
|
||||
|
||||
self._running = True
|
||||
self.bus.subscribe_outbound(self._on_outbound)
|
||||
|
||||
# Start socket mode in background thread
|
||||
asyncio.get_event_loop().run_in_executor(None, self._socket_client.connect)
|
||||
logger.info("Slack channel started")
|
||||
|
||||
async def stop(self) -> None:
|
||||
self._running = False
|
||||
self.bus.unsubscribe_outbound(self._on_outbound)
|
||||
if self._socket_client:
|
||||
self._socket_client.close()
|
||||
self._socket_client = None
|
||||
logger.info("Slack channel stopped")
|
||||
|
||||
async def send(self, msg: OutboundMessage, *, _max_retries: int = 3) -> None:
|
||||
if not self._web_client:
|
||||
return
|
||||
|
||||
kwargs: dict[str, Any] = {
|
||||
"channel": msg.chat_id,
|
||||
"text": _slack_md_converter.convert(msg.text),
|
||||
}
|
||||
if msg.thread_ts:
|
||||
kwargs["thread_ts"] = msg.thread_ts
|
||||
|
||||
last_exc: Exception | None = None
|
||||
for attempt in range(_max_retries):
|
||||
try:
|
||||
await asyncio.to_thread(self._web_client.chat_postMessage, **kwargs)
|
||||
# Add a completion reaction to the thread root
|
||||
if msg.thread_ts:
|
||||
await asyncio.to_thread(
|
||||
self._add_reaction,
|
||||
msg.chat_id,
|
||||
msg.thread_ts,
|
||||
"white_check_mark",
|
||||
)
|
||||
return
|
||||
except Exception as exc:
|
||||
last_exc = exc
|
||||
if attempt < _max_retries - 1:
|
||||
delay = 2**attempt # 1s, 2s
|
||||
logger.warning(
|
||||
"[Slack] send failed (attempt %d/%d), retrying in %ds: %s",
|
||||
attempt + 1,
|
||||
_max_retries,
|
||||
delay,
|
||||
exc,
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
|
||||
logger.error("[Slack] send failed after %d attempts: %s", _max_retries, last_exc)
|
||||
# Add failure reaction on error
|
||||
if msg.thread_ts:
|
||||
try:
|
||||
await asyncio.to_thread(
|
||||
self._add_reaction,
|
||||
msg.chat_id,
|
||||
msg.thread_ts,
|
||||
"x",
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
if last_exc is None:
|
||||
raise RuntimeError("Slack send failed without an exception from any attempt")
|
||||
raise last_exc
|
||||
|
||||
async def send_file(self, msg: OutboundMessage, attachment: ResolvedAttachment) -> bool:
|
||||
if not self._web_client:
|
||||
return False
|
||||
|
||||
try:
|
||||
kwargs: dict[str, Any] = {
|
||||
"channel": msg.chat_id,
|
||||
"file": str(attachment.actual_path),
|
||||
"filename": attachment.filename,
|
||||
"title": attachment.filename,
|
||||
}
|
||||
if msg.thread_ts:
|
||||
kwargs["thread_ts"] = msg.thread_ts
|
||||
|
||||
await asyncio.to_thread(self._web_client.files_upload_v2, **kwargs)
|
||||
logger.info("[Slack] file uploaded: %s to channel=%s", attachment.filename, msg.chat_id)
|
||||
return True
|
||||
except Exception:
|
||||
logger.exception("[Slack] failed to upload file: %s", attachment.filename)
|
||||
return False
|
||||
|
||||
# -- internal ----------------------------------------------------------
|
||||
|
||||
def _add_reaction(self, channel_id: str, timestamp: str, emoji: str) -> None:
|
||||
"""Add an emoji reaction to a message (best-effort, non-blocking)."""
|
||||
if not self._web_client:
|
||||
return
|
||||
try:
|
||||
self._web_client.reactions_add(
|
||||
channel=channel_id,
|
||||
timestamp=timestamp,
|
||||
name=emoji,
|
||||
)
|
||||
except Exception as exc:
|
||||
if "already_reacted" not in str(exc):
|
||||
logger.warning("[Slack] failed to add reaction %s: %s", emoji, exc)
|
||||
|
||||
def _send_running_reply(self, channel_id: str, thread_ts: str) -> None:
|
||||
"""Send a 'Working on it......' reply in the thread (called from SDK thread)."""
|
||||
if not self._web_client:
|
||||
return
|
||||
try:
|
||||
self._web_client.chat_postMessage(
|
||||
channel=channel_id,
|
||||
text=":hourglass_flowing_sand: Working on it...",
|
||||
thread_ts=thread_ts,
|
||||
)
|
||||
logger.info("[Slack] 'Working on it...' reply sent in channel=%s, thread_ts=%s", channel_id, thread_ts)
|
||||
except Exception:
|
||||
logger.exception("[Slack] failed to send running reply in channel=%s", channel_id)
|
||||
|
||||
def _on_socket_event(self, client, req) -> None:
|
||||
"""Called by slack-sdk for each Socket Mode event."""
|
||||
try:
|
||||
# Acknowledge the event
|
||||
response = self._SocketModeResponse(envelope_id=req.envelope_id)
|
||||
client.send_socket_mode_response(response)
|
||||
|
||||
event_type = req.type
|
||||
if event_type != "events_api":
|
||||
return
|
||||
|
||||
event = req.payload.get("event", {})
|
||||
etype = event.get("type", "")
|
||||
|
||||
# Handle message events (DM or @mention)
|
||||
if etype in ("message", "app_mention"):
|
||||
self._handle_message_event(event)
|
||||
|
||||
except Exception:
|
||||
logger.exception("Error processing Slack event")
|
||||
|
||||
def _handle_message_event(self, event: dict) -> None:
|
||||
# Ignore bot messages
|
||||
if event.get("bot_id") or event.get("subtype"):
|
||||
return
|
||||
|
||||
user_id = event.get("user", "")
|
||||
|
||||
# Check allowed users
|
||||
if self._allowed_users and user_id not in self._allowed_users:
|
||||
logger.debug("Ignoring message from non-allowed user: %s", user_id)
|
||||
return
|
||||
|
||||
text = event.get("text", "").strip()
|
||||
if not text:
|
||||
return
|
||||
|
||||
channel_id = event.get("channel", "")
|
||||
thread_ts = event.get("thread_ts") or event.get("ts", "")
|
||||
|
||||
if text.startswith("/"):
|
||||
msg_type = InboundMessageType.COMMAND
|
||||
else:
|
||||
msg_type = InboundMessageType.CHAT
|
||||
|
||||
# topic_id: use thread_ts as the topic identifier.
|
||||
# For threaded messages, thread_ts is the root message ts (shared topic).
|
||||
# For non-threaded messages, thread_ts is the message's own ts (new topic).
|
||||
inbound = self._make_inbound(
|
||||
chat_id=channel_id,
|
||||
user_id=user_id,
|
||||
text=text,
|
||||
msg_type=msg_type,
|
||||
thread_ts=thread_ts,
|
||||
)
|
||||
inbound.topic_id = thread_ts
|
||||
|
||||
if self._loop and self._loop.is_running():
|
||||
# Acknowledge with an eyes reaction
|
||||
self._add_reaction(channel_id, event.get("ts", thread_ts), "eyes")
|
||||
# Send "running" reply first (fire-and-forget from SDK thread)
|
||||
self._send_running_reply(channel_id, thread_ts)
|
||||
asyncio.run_coroutine_threadsafe(self.bus.publish_inbound(inbound), self._loop)
|
||||
@@ -1,153 +0,0 @@
|
||||
"""ChannelStore — persists IM chat-to-DeerFlow thread mappings."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class ChannelStore:
|
||||
"""JSON-file-backed store that maps IM conversations to DeerFlow threads.
|
||||
|
||||
Data layout (on disk)::
|
||||
|
||||
{
|
||||
"<channel_name>:<chat_id>": {
|
||||
"thread_id": "<uuid>",
|
||||
"user_id": "<platform_user>",
|
||||
"created_at": 1700000000.0,
|
||||
"updated_at": 1700000000.0
|
||||
},
|
||||
...
|
||||
}
|
||||
|
||||
The store is intentionally simple — a single JSON file that is atomically
|
||||
rewritten on every mutation. For production workloads with high concurrency,
|
||||
this can be swapped for a proper database backend.
|
||||
"""
|
||||
|
||||
def __init__(self, path: str | Path | None = None) -> None:
|
||||
if path is None:
|
||||
from deerflow.config.paths import get_paths
|
||||
|
||||
path = Path(get_paths().base_dir) / "channels" / "store.json"
|
||||
self._path = Path(path)
|
||||
self._path.parent.mkdir(parents=True, exist_ok=True)
|
||||
self._data: dict[str, dict[str, Any]] = self._load()
|
||||
self._lock = threading.Lock()
|
||||
|
||||
# -- persistence -------------------------------------------------------
|
||||
|
||||
def _load(self) -> dict[str, dict[str, Any]]:
|
||||
if self._path.exists():
|
||||
try:
|
||||
return json.loads(self._path.read_text(encoding="utf-8"))
|
||||
except (json.JSONDecodeError, OSError):
|
||||
logger.warning("Corrupt channel store at %s, starting fresh", self._path)
|
||||
return {}
|
||||
|
||||
def _save(self) -> None:
|
||||
fd = tempfile.NamedTemporaryFile(
|
||||
mode="w",
|
||||
dir=self._path.parent,
|
||||
suffix=".tmp",
|
||||
delete=False,
|
||||
)
|
||||
try:
|
||||
json.dump(self._data, fd, indent=2)
|
||||
fd.close()
|
||||
Path(fd.name).replace(self._path)
|
||||
except BaseException:
|
||||
fd.close()
|
||||
Path(fd.name).unlink(missing_ok=True)
|
||||
raise
|
||||
|
||||
# -- key helpers -------------------------------------------------------
|
||||
|
||||
@staticmethod
|
||||
def _key(channel_name: str, chat_id: str, topic_id: str | None = None) -> str:
|
||||
if topic_id:
|
||||
return f"{channel_name}:{chat_id}:{topic_id}"
|
||||
return f"{channel_name}:{chat_id}"
|
||||
|
||||
# -- public API --------------------------------------------------------
|
||||
|
||||
def get_thread_id(self, channel_name: str, chat_id: str, topic_id: str | None = None) -> str | None:
|
||||
"""Look up the DeerFlow thread_id for a given IM conversation/topic."""
|
||||
entry = self._data.get(self._key(channel_name, chat_id, topic_id))
|
||||
return entry["thread_id"] if entry else None
|
||||
|
||||
def set_thread_id(
|
||||
self,
|
||||
channel_name: str,
|
||||
chat_id: str,
|
||||
thread_id: str,
|
||||
*,
|
||||
topic_id: str | None = None,
|
||||
user_id: str = "",
|
||||
) -> None:
|
||||
"""Create or update the mapping for an IM conversation/topic."""
|
||||
with self._lock:
|
||||
key = self._key(channel_name, chat_id, topic_id)
|
||||
now = time.time()
|
||||
existing = self._data.get(key)
|
||||
self._data[key] = {
|
||||
"thread_id": thread_id,
|
||||
"user_id": user_id,
|
||||
"created_at": existing["created_at"] if existing else now,
|
||||
"updated_at": now,
|
||||
}
|
||||
self._save()
|
||||
|
||||
def remove(self, channel_name: str, chat_id: str, topic_id: str | None = None) -> bool:
|
||||
"""Remove a mapping.
|
||||
|
||||
If ``topic_id`` is provided, only that specific conversation/topic mapping is removed.
|
||||
If ``topic_id`` is omitted, all mappings whose key starts with
|
||||
``"<channel_name>:<chat_id>"`` (including topic-specific ones) are removed.
|
||||
|
||||
Returns True if at least one mapping was removed.
|
||||
"""
|
||||
with self._lock:
|
||||
# Remove a specific conversation/topic mapping.
|
||||
if topic_id is not None:
|
||||
key = self._key(channel_name, chat_id, topic_id)
|
||||
if key in self._data:
|
||||
del self._data[key]
|
||||
self._save()
|
||||
return True
|
||||
return False
|
||||
|
||||
# Remove all mappings for this channel/chat_id (base and any topic-specific keys).
|
||||
prefix = self._key(channel_name, chat_id)
|
||||
keys_to_delete = [k for k in self._data if k == prefix or k.startswith(prefix + ":")]
|
||||
if not keys_to_delete:
|
||||
return False
|
||||
|
||||
for k in keys_to_delete:
|
||||
del self._data[k]
|
||||
self._save()
|
||||
return True
|
||||
|
||||
def list_entries(self, channel_name: str | None = None) -> list[dict[str, Any]]:
|
||||
"""List all stored mappings, optionally filtered by channel."""
|
||||
results = []
|
||||
for key, entry in self._data.items():
|
||||
parts = key.split(":", 2)
|
||||
ch = parts[0]
|
||||
chat = parts[1] if len(parts) > 1 else ""
|
||||
topic = parts[2] if len(parts) > 2 else None
|
||||
if channel_name and ch != channel_name:
|
||||
continue
|
||||
item: dict[str, Any] = {"channel_name": ch, "chat_id": chat, **entry}
|
||||
if topic is not None:
|
||||
item["topic_id"] = topic
|
||||
results.append(item)
|
||||
return results
|
||||
@@ -1,317 +0,0 @@
|
||||
"""Telegram channel — connects via long-polling (no public IP needed)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import threading
|
||||
from typing import Any
|
||||
|
||||
from app.channels.base import Channel
|
||||
from app.channels.message_bus import InboundMessage, InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class TelegramChannel(Channel):
|
||||
"""Telegram bot channel using long-polling.
|
||||
|
||||
Configuration keys (in ``config.yaml`` under ``channels.telegram``):
|
||||
- ``bot_token``: Telegram Bot API token (from @BotFather).
|
||||
- ``allowed_users``: (optional) List of allowed Telegram user IDs. Empty = allow all.
|
||||
"""
|
||||
|
||||
def __init__(self, bus: MessageBus, config: dict[str, Any]) -> None:
|
||||
super().__init__(name="telegram", bus=bus, config=config)
|
||||
self._application = None
|
||||
self._thread: threading.Thread | None = None
|
||||
self._tg_loop: asyncio.AbstractEventLoop | None = None
|
||||
self._main_loop: asyncio.AbstractEventLoop | None = None
|
||||
self._allowed_users: set[int] = set()
|
||||
for uid in config.get("allowed_users", []):
|
||||
try:
|
||||
self._allowed_users.add(int(uid))
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
# chat_id -> last sent message_id for threaded replies
|
||||
self._last_bot_message: dict[str, int] = {}
|
||||
|
||||
async def start(self) -> None:
|
||||
if self._running:
|
||||
return
|
||||
|
||||
try:
|
||||
from telegram.ext import ApplicationBuilder, CommandHandler, MessageHandler, filters
|
||||
except ImportError:
|
||||
logger.error("python-telegram-bot is not installed. Install it with: uv add python-telegram-bot")
|
||||
return
|
||||
|
||||
bot_token = self.config.get("bot_token", "")
|
||||
if not bot_token:
|
||||
logger.error("Telegram channel requires bot_token")
|
||||
return
|
||||
|
||||
self._main_loop = asyncio.get_event_loop()
|
||||
self._running = True
|
||||
self.bus.subscribe_outbound(self._on_outbound)
|
||||
|
||||
# Build the application
|
||||
app = ApplicationBuilder().token(bot_token).build()
|
||||
|
||||
# Command handlers
|
||||
app.add_handler(CommandHandler("start", self._cmd_start))
|
||||
app.add_handler(CommandHandler("new", self._cmd_generic))
|
||||
app.add_handler(CommandHandler("status", self._cmd_generic))
|
||||
app.add_handler(CommandHandler("models", self._cmd_generic))
|
||||
app.add_handler(CommandHandler("memory", self._cmd_generic))
|
||||
app.add_handler(CommandHandler("help", self._cmd_generic))
|
||||
|
||||
# General message handler
|
||||
app.add_handler(MessageHandler(filters.TEXT & ~filters.COMMAND, self._on_text))
|
||||
|
||||
self._application = app
|
||||
|
||||
# Run polling in a dedicated thread with its own event loop
|
||||
self._thread = threading.Thread(target=self._run_polling, daemon=True)
|
||||
self._thread.start()
|
||||
logger.info("Telegram channel started")
|
||||
|
||||
async def stop(self) -> None:
|
||||
self._running = False
|
||||
self.bus.unsubscribe_outbound(self._on_outbound)
|
||||
if self._tg_loop and self._tg_loop.is_running():
|
||||
self._tg_loop.call_soon_threadsafe(self._tg_loop.stop)
|
||||
if self._thread:
|
||||
self._thread.join(timeout=10)
|
||||
self._thread = None
|
||||
self._application = None
|
||||
logger.info("Telegram channel stopped")
|
||||
|
||||
async def send(self, msg: OutboundMessage, *, _max_retries: int = 3) -> None:
|
||||
if not self._application:
|
||||
return
|
||||
|
||||
try:
|
||||
chat_id = int(msg.chat_id)
|
||||
except (ValueError, TypeError):
|
||||
logger.error("Invalid Telegram chat_id: %s", msg.chat_id)
|
||||
return
|
||||
|
||||
kwargs: dict[str, Any] = {"chat_id": chat_id, "text": msg.text}
|
||||
|
||||
# Reply to the last bot message in this chat for threading
|
||||
reply_to = self._last_bot_message.get(msg.chat_id)
|
||||
if reply_to:
|
||||
kwargs["reply_to_message_id"] = reply_to
|
||||
|
||||
bot = self._application.bot
|
||||
last_exc: Exception | None = None
|
||||
for attempt in range(_max_retries):
|
||||
try:
|
||||
sent = await bot.send_message(**kwargs)
|
||||
self._last_bot_message[msg.chat_id] = sent.message_id
|
||||
return
|
||||
except Exception as exc:
|
||||
last_exc = exc
|
||||
if attempt < _max_retries - 1:
|
||||
delay = 2**attempt # 1s, 2s
|
||||
logger.warning(
|
||||
"[Telegram] send failed (attempt %d/%d), retrying in %ds: %s",
|
||||
attempt + 1,
|
||||
_max_retries,
|
||||
delay,
|
||||
exc,
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
|
||||
logger.error("[Telegram] send failed after %d attempts: %s", _max_retries, last_exc)
|
||||
if last_exc is None:
|
||||
raise RuntimeError("Telegram send failed without an exception from any attempt")
|
||||
raise last_exc
|
||||
|
||||
async def send_file(self, msg: OutboundMessage, attachment: ResolvedAttachment) -> bool:
|
||||
if not self._application:
|
||||
return False
|
||||
|
||||
try:
|
||||
chat_id = int(msg.chat_id)
|
||||
except (ValueError, TypeError):
|
||||
logger.error("[Telegram] Invalid chat_id: %s", msg.chat_id)
|
||||
return False
|
||||
|
||||
# Telegram limits: 10MB for photos, 50MB for documents
|
||||
if attachment.size > 50 * 1024 * 1024:
|
||||
logger.warning("[Telegram] file too large (%d bytes), skipping: %s", attachment.size, attachment.filename)
|
||||
return False
|
||||
|
||||
bot = self._application.bot
|
||||
reply_to = self._last_bot_message.get(msg.chat_id)
|
||||
|
||||
try:
|
||||
if attachment.is_image and attachment.size <= 10 * 1024 * 1024:
|
||||
with open(attachment.actual_path, "rb") as f:
|
||||
kwargs: dict[str, Any] = {"chat_id": chat_id, "photo": f}
|
||||
if reply_to:
|
||||
kwargs["reply_to_message_id"] = reply_to
|
||||
sent = await bot.send_photo(**kwargs)
|
||||
else:
|
||||
from telegram import InputFile
|
||||
|
||||
with open(attachment.actual_path, "rb") as f:
|
||||
input_file = InputFile(f, filename=attachment.filename)
|
||||
kwargs = {"chat_id": chat_id, "document": input_file}
|
||||
if reply_to:
|
||||
kwargs["reply_to_message_id"] = reply_to
|
||||
sent = await bot.send_document(**kwargs)
|
||||
|
||||
self._last_bot_message[msg.chat_id] = sent.message_id
|
||||
logger.info("[Telegram] file sent: %s to chat=%s", attachment.filename, msg.chat_id)
|
||||
return True
|
||||
except Exception:
|
||||
logger.exception("[Telegram] failed to send file: %s", attachment.filename)
|
||||
return False
|
||||
|
||||
# -- helpers -----------------------------------------------------------
|
||||
|
||||
async def _send_running_reply(self, chat_id: str, reply_to_message_id: int) -> None:
|
||||
"""Send a 'Working on it...' reply to the user's message."""
|
||||
if not self._application:
|
||||
return
|
||||
try:
|
||||
bot = self._application.bot
|
||||
await bot.send_message(
|
||||
chat_id=int(chat_id),
|
||||
text="Working on it...",
|
||||
reply_to_message_id=reply_to_message_id,
|
||||
)
|
||||
logger.info("[Telegram] 'Working on it...' reply sent in chat=%s", chat_id)
|
||||
except Exception:
|
||||
logger.exception("[Telegram] failed to send running reply in chat=%s", chat_id)
|
||||
|
||||
# -- internal ----------------------------------------------------------
|
||||
@staticmethod
|
||||
def _log_future_error(fut, name: str, msg_id: str):
|
||||
try:
|
||||
exc = fut.exception()
|
||||
if exc:
|
||||
logger.error("[Telegram] %s failed for msg_id=%s: %s", name, msg_id, exc)
|
||||
except Exception:
|
||||
logger.exception("[Telegram] Failed to inspect future for %s (msg_id=%s)", name, msg_id)
|
||||
|
||||
def _run_polling(self) -> None:
|
||||
"""Run telegram polling in a dedicated thread."""
|
||||
self._tg_loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(self._tg_loop)
|
||||
try:
|
||||
# Cannot use run_polling() because it calls add_signal_handler(),
|
||||
# which only works in the main thread. Instead, manually
|
||||
# initialize the application and start the updater.
|
||||
self._tg_loop.run_until_complete(self._application.initialize())
|
||||
self._tg_loop.run_until_complete(self._application.start())
|
||||
self._tg_loop.run_until_complete(self._application.updater.start_polling())
|
||||
self._tg_loop.run_forever()
|
||||
except Exception:
|
||||
if self._running:
|
||||
logger.exception("Telegram polling error")
|
||||
finally:
|
||||
# Graceful shutdown
|
||||
try:
|
||||
if self._application.updater.running:
|
||||
self._tg_loop.run_until_complete(self._application.updater.stop())
|
||||
self._tg_loop.run_until_complete(self._application.stop())
|
||||
self._tg_loop.run_until_complete(self._application.shutdown())
|
||||
except Exception:
|
||||
logger.exception("Error during Telegram shutdown")
|
||||
|
||||
def _check_user(self, user_id: int) -> bool:
|
||||
if not self._allowed_users:
|
||||
return True
|
||||
return user_id in self._allowed_users
|
||||
|
||||
async def _cmd_start(self, update, context) -> None:
|
||||
"""Handle /start command."""
|
||||
if not self._check_user(update.effective_user.id):
|
||||
return
|
||||
await update.message.reply_text("Welcome to DeerFlow! Send me a message to start a conversation.\nType /help for available commands.")
|
||||
|
||||
async def _process_incoming_with_reply(self, chat_id: str, msg_id: int, inbound: InboundMessage) -> None:
|
||||
await self._send_running_reply(chat_id, msg_id)
|
||||
await self.bus.publish_inbound(inbound)
|
||||
|
||||
async def _cmd_generic(self, update, context) -> None:
|
||||
"""Forward slash commands to the channel manager."""
|
||||
if not self._check_user(update.effective_user.id):
|
||||
return
|
||||
|
||||
text = update.message.text
|
||||
chat_id = str(update.effective_chat.id)
|
||||
user_id = str(update.effective_user.id)
|
||||
msg_id = str(update.message.message_id)
|
||||
|
||||
# Use the same topic_id logic as _on_text so that commands
|
||||
# like /new target the correct thread mapping.
|
||||
if update.effective_chat.type == "private":
|
||||
topic_id = None
|
||||
else:
|
||||
reply_to = update.message.reply_to_message
|
||||
if reply_to:
|
||||
topic_id = str(reply_to.message_id)
|
||||
else:
|
||||
topic_id = msg_id
|
||||
|
||||
inbound = self._make_inbound(
|
||||
chat_id=chat_id,
|
||||
user_id=user_id,
|
||||
text=text,
|
||||
msg_type=InboundMessageType.COMMAND,
|
||||
thread_ts=msg_id,
|
||||
)
|
||||
inbound.topic_id = topic_id
|
||||
|
||||
if self._main_loop and self._main_loop.is_running():
|
||||
fut = asyncio.run_coroutine_threadsafe(self._process_incoming_with_reply(chat_id, update.message.message_id, inbound), self._main_loop)
|
||||
fut.add_done_callback(lambda f: self._log_future_error(f, "process_incoming_with_reply", update.message.message_id))
|
||||
else:
|
||||
logger.warning("[Telegram] Main loop not running. Cannot publish inbound message.")
|
||||
|
||||
async def _on_text(self, update, context) -> None:
|
||||
"""Handle regular text messages."""
|
||||
if not self._check_user(update.effective_user.id):
|
||||
return
|
||||
|
||||
text = update.message.text.strip()
|
||||
if not text:
|
||||
return
|
||||
|
||||
chat_id = str(update.effective_chat.id)
|
||||
user_id = str(update.effective_user.id)
|
||||
msg_id = str(update.message.message_id)
|
||||
|
||||
# topic_id determines which DeerFlow thread the message maps to.
|
||||
# In private chats, use None so that all messages share a single
|
||||
# thread (the store key becomes "channel:chat_id").
|
||||
# In group chats, use the reply-to message id or the current
|
||||
# message id to keep separate conversation threads.
|
||||
if update.effective_chat.type == "private":
|
||||
topic_id = None
|
||||
else:
|
||||
reply_to = update.message.reply_to_message
|
||||
if reply_to:
|
||||
topic_id = str(reply_to.message_id)
|
||||
else:
|
||||
topic_id = msg_id
|
||||
|
||||
inbound = self._make_inbound(
|
||||
chat_id=chat_id,
|
||||
user_id=user_id,
|
||||
text=text,
|
||||
msg_type=InboundMessageType.CHAT,
|
||||
thread_ts=msg_id,
|
||||
)
|
||||
inbound.topic_id = topic_id
|
||||
|
||||
if self._main_loop and self._main_loop.is_running():
|
||||
fut = asyncio.run_coroutine_threadsafe(self._process_incoming_with_reply(chat_id, update.message.message_id, inbound), self._main_loop)
|
||||
fut.add_done_callback(lambda f: self._log_future_error(f, "process_incoming_with_reply", update.message.message_id))
|
||||
else:
|
||||
logger.warning("[Telegram] Main loop not running. Cannot publish inbound message.")
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,398 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import hashlib
|
||||
import logging
|
||||
from collections.abc import Awaitable, Callable
|
||||
from typing import Any, cast
|
||||
|
||||
from app.channels.base import Channel
|
||||
from app.channels.message_bus import (
|
||||
InboundMessageType,
|
||||
MessageBus,
|
||||
OutboundMessage,
|
||||
ResolvedAttachment,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class WeComChannel(Channel):
|
||||
def __init__(self, bus: MessageBus, config: dict[str, Any]) -> None:
|
||||
super().__init__(name="wecom", bus=bus, config=config)
|
||||
self._bot_id: str | None = None
|
||||
self._bot_secret: str | None = None
|
||||
self._ws_client = None
|
||||
self._ws_task: asyncio.Task | None = None
|
||||
self._ws_frames: dict[str, dict[str, Any]] = {}
|
||||
self._ws_stream_ids: dict[str, str] = {}
|
||||
self._working_message = "Working on it..."
|
||||
|
||||
@property
|
||||
def supports_streaming(self) -> bool:
|
||||
return True
|
||||
|
||||
def _clear_ws_context(self, thread_ts: str | None) -> None:
|
||||
if not thread_ts:
|
||||
return
|
||||
self._ws_frames.pop(thread_ts, None)
|
||||
self._ws_stream_ids.pop(thread_ts, None)
|
||||
|
||||
async def _send_ws_upload_command(self, req_id: str, body: dict[str, Any], cmd: str) -> dict[str, Any]:
|
||||
if not self._ws_client:
|
||||
raise RuntimeError("WeCom WebSocket client is not available")
|
||||
|
||||
ws_manager = getattr(self._ws_client, "_ws_manager", None)
|
||||
send_reply = getattr(ws_manager, "send_reply", None)
|
||||
if not callable(send_reply):
|
||||
raise RuntimeError("Installed wecom-aibot-python-sdk does not expose the WebSocket media upload API expected by DeerFlow. Use wecom-aibot-python-sdk==0.1.6 or update the adapter.")
|
||||
|
||||
send_reply_async = cast(Callable[[str, dict[str, Any], str], Awaitable[dict[str, Any]]], send_reply)
|
||||
return await send_reply_async(req_id, body, cmd)
|
||||
|
||||
async def start(self) -> None:
|
||||
if self._running:
|
||||
return
|
||||
|
||||
bot_id = self.config.get("bot_id")
|
||||
bot_secret = self.config.get("bot_secret")
|
||||
working_message = self.config.get("working_message")
|
||||
|
||||
self._bot_id = bot_id if isinstance(bot_id, str) and bot_id else None
|
||||
self._bot_secret = bot_secret if isinstance(bot_secret, str) and bot_secret else None
|
||||
self._working_message = working_message if isinstance(working_message, str) and working_message else "Working on it..."
|
||||
|
||||
if not self._bot_id or not self._bot_secret:
|
||||
logger.error("WeCom channel requires bot_id and bot_secret")
|
||||
return
|
||||
|
||||
try:
|
||||
from aibot import WSClient, WSClientOptions
|
||||
except ImportError:
|
||||
logger.error("wecom-aibot-python-sdk is not installed. Install it with: uv add wecom-aibot-python-sdk")
|
||||
return
|
||||
else:
|
||||
self._ws_client = WSClient(WSClientOptions(bot_id=self._bot_id, secret=self._bot_secret, logger=logger))
|
||||
self._ws_client.on("message.text", self._on_ws_text)
|
||||
self._ws_client.on("message.mixed", self._on_ws_mixed)
|
||||
self._ws_client.on("message.image", self._on_ws_image)
|
||||
self._ws_client.on("message.file", self._on_ws_file)
|
||||
self._ws_task = asyncio.create_task(self._ws_client.connect())
|
||||
|
||||
self._running = True
|
||||
self.bus.subscribe_outbound(self._on_outbound)
|
||||
logger.info("WeCom channel started")
|
||||
|
||||
async def stop(self) -> None:
|
||||
self._running = False
|
||||
self.bus.unsubscribe_outbound(self._on_outbound)
|
||||
if self._ws_task:
|
||||
try:
|
||||
self._ws_task.cancel()
|
||||
except Exception:
|
||||
pass
|
||||
self._ws_task = None
|
||||
if self._ws_client:
|
||||
try:
|
||||
self._ws_client.disconnect()
|
||||
except Exception:
|
||||
pass
|
||||
self._ws_client = None
|
||||
self._ws_frames.clear()
|
||||
self._ws_stream_ids.clear()
|
||||
logger.info("WeCom channel stopped")
|
||||
|
||||
async def send(self, msg: OutboundMessage, *, _max_retries: int = 3) -> None:
|
||||
if self._ws_client:
|
||||
await self._send_ws(msg, _max_retries=_max_retries)
|
||||
return
|
||||
logger.warning("[WeCom] send called but WebSocket client is not available")
|
||||
|
||||
async def _on_outbound(self, msg: OutboundMessage) -> None:
|
||||
if msg.channel_name != self.name:
|
||||
return
|
||||
|
||||
try:
|
||||
await self.send(msg)
|
||||
except Exception:
|
||||
logger.exception("Failed to send outbound message on channel %s", self.name)
|
||||
if msg.is_final:
|
||||
self._clear_ws_context(msg.thread_ts)
|
||||
return
|
||||
|
||||
for attachment in msg.attachments:
|
||||
try:
|
||||
success = await self.send_file(msg, attachment)
|
||||
if not success:
|
||||
logger.warning("[%s] file upload skipped for %s", self.name, attachment.filename)
|
||||
except Exception:
|
||||
logger.exception("[%s] failed to upload file %s", self.name, attachment.filename)
|
||||
|
||||
if msg.is_final:
|
||||
self._clear_ws_context(msg.thread_ts)
|
||||
|
||||
async def send_file(self, msg: OutboundMessage, attachment: ResolvedAttachment) -> bool:
|
||||
if not msg.is_final:
|
||||
return True
|
||||
if not self._ws_client:
|
||||
return False
|
||||
if not msg.thread_ts:
|
||||
return False
|
||||
frame = self._ws_frames.get(msg.thread_ts)
|
||||
if not frame:
|
||||
return False
|
||||
|
||||
media_type = "image" if attachment.is_image else "file"
|
||||
size_limit = 2 * 1024 * 1024 if attachment.is_image else 20 * 1024 * 1024
|
||||
if attachment.size > size_limit:
|
||||
logger.warning(
|
||||
"[WeCom] %s too large (%d bytes), skipping: %s",
|
||||
media_type,
|
||||
attachment.size,
|
||||
attachment.filename,
|
||||
)
|
||||
return False
|
||||
|
||||
try:
|
||||
media_id = await self._upload_media_ws(
|
||||
media_type=media_type,
|
||||
filename=attachment.filename,
|
||||
path=str(attachment.actual_path),
|
||||
size=attachment.size,
|
||||
)
|
||||
if not media_id:
|
||||
return False
|
||||
|
||||
body = {media_type: {"media_id": media_id}, "msgtype": media_type}
|
||||
await self._ws_client.reply(frame, body)
|
||||
logger.debug("[WeCom] %s sent via ws: %s", media_type, attachment.filename)
|
||||
return True
|
||||
except Exception:
|
||||
logger.exception("[WeCom] failed to upload/send file via ws: %s", attachment.filename)
|
||||
return False
|
||||
|
||||
async def _on_ws_text(self, frame: dict[str, Any]) -> None:
|
||||
body = frame.get("body", {}) or {}
|
||||
text = ((body.get("text") or {}).get("content") or "").strip()
|
||||
quote = body.get("quote", {}).get("text", {}).get("content", "").strip()
|
||||
if not text and not quote:
|
||||
return
|
||||
await self._publish_ws_inbound(frame, text + (f"\nQuote message: {quote}" if quote else ""))
|
||||
|
||||
async def _on_ws_mixed(self, frame: dict[str, Any]) -> None:
|
||||
body = frame.get("body", {}) or {}
|
||||
mixed = body.get("mixed") or {}
|
||||
items = mixed.get("msg_item") or []
|
||||
parts: list[str] = []
|
||||
files: list[dict[str, Any]] = []
|
||||
for item in items:
|
||||
item_type = (item or {}).get("msgtype")
|
||||
if item_type == "text":
|
||||
content = (((item or {}).get("text") or {}).get("content") or "").strip()
|
||||
if content:
|
||||
parts.append(content)
|
||||
elif item_type in ("image", "file"):
|
||||
payload = (item or {}).get(item_type) or {}
|
||||
url = payload.get("url")
|
||||
aeskey = payload.get("aeskey")
|
||||
if isinstance(url, str) and url:
|
||||
files.append(
|
||||
{
|
||||
"type": item_type,
|
||||
"url": url,
|
||||
"aeskey": (aeskey if isinstance(aeskey, str) and aeskey else None),
|
||||
}
|
||||
)
|
||||
text = "\n\n".join(parts).strip()
|
||||
if not text and not files:
|
||||
return
|
||||
if not text:
|
||||
text = "（receive image/file）"
|
||||
await self._publish_ws_inbound(frame, text, files=files)
|
||||
|
||||
async def _on_ws_image(self, frame: dict[str, Any]) -> None:
|
||||
body = frame.get("body", {}) or {}
|
||||
image = body.get("image") or {}
|
||||
url = image.get("url")
|
||||
aeskey = image.get("aeskey")
|
||||
if not isinstance(url, str) or not url:
|
||||
return
|
||||
await self._publish_ws_inbound(
|
||||
frame,
|
||||
"（receive image ）",
|
||||
files=[
|
||||
{
|
||||
"type": "image",
|
||||
"url": url,
|
||||
"aeskey": aeskey if isinstance(aeskey, str) and aeskey else None,
|
||||
}
|
||||
],
|
||||
)
|
||||
|
||||
async def _on_ws_file(self, frame: dict[str, Any]) -> None:
|
||||
body = frame.get("body", {}) or {}
|
||||
file_obj = body.get("file") or {}
|
||||
url = file_obj.get("url")
|
||||
aeskey = file_obj.get("aeskey")
|
||||
if not isinstance(url, str) or not url:
|
||||
return
|
||||
await self._publish_ws_inbound(
|
||||
frame,
|
||||
"（receive file）",
|
||||
files=[
|
||||
{
|
||||
"type": "file",
|
||||
"url": url,
|
||||
"aeskey": aeskey if isinstance(aeskey, str) and aeskey else None,
|
||||
}
|
||||
],
|
||||
)
|
||||
|
||||
async def _publish_ws_inbound(
|
||||
self,
|
||||
frame: dict[str, Any],
|
||||
text: str,
|
||||
*,
|
||||
files: list[dict[str, Any]] | None = None,
|
||||
) -> None:
|
||||
if not self._ws_client:
|
||||
return
|
||||
try:
|
||||
from aibot import generate_req_id
|
||||
except Exception:
|
||||
return
|
||||
|
||||
body = frame.get("body", {}) or {}
|
||||
msg_id = body.get("msgid")
|
||||
if not msg_id:
|
||||
return
|
||||
|
||||
user_id = (body.get("from") or {}).get("userid")
|
||||
|
||||
inbound_type = InboundMessageType.COMMAND if text.startswith("/") else InboundMessageType.CHAT
|
||||
inbound = self._make_inbound(
|
||||
chat_id=user_id, # keep user's conversation in memory
|
||||
user_id=user_id,
|
||||
text=text,
|
||||
msg_type=inbound_type,
|
||||
thread_ts=msg_id,
|
||||
files=files or [],
|
||||
metadata={"aibotid": body.get("aibotid"), "chattype": body.get("chattype")},
|
||||
)
|
||||
inbound.topic_id = user_id # keep the same thread
|
||||
|
||||
stream_id = generate_req_id("stream")
|
||||
self._ws_frames[msg_id] = frame
|
||||
self._ws_stream_ids[msg_id] = stream_id
|
||||
|
||||
try:
|
||||
await self._ws_client.reply_stream(frame, stream_id, self._working_message, False)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
await self.bus.publish_inbound(inbound)
|
||||
|
||||
async def _send_ws(self, msg: OutboundMessage, *, _max_retries: int = 3) -> None:
|
||||
if not self._ws_client:
|
||||
return
|
||||
try:
|
||||
from aibot import generate_req_id
|
||||
except Exception:
|
||||
generate_req_id = None
|
||||
|
||||
if msg.thread_ts and msg.thread_ts in self._ws_frames:
|
||||
frame = self._ws_frames[msg.thread_ts]
|
||||
stream_id = self._ws_stream_ids.get(msg.thread_ts)
|
||||
if not stream_id and generate_req_id:
|
||||
stream_id = generate_req_id("stream")
|
||||
self._ws_stream_ids[msg.thread_ts] = stream_id
|
||||
if not stream_id:
|
||||
return
|
||||
|
||||
last_exc: Exception | None = None
|
||||
for attempt in range(_max_retries):
|
||||
try:
|
||||
await self._ws_client.reply_stream(frame, stream_id, msg.text, bool(msg.is_final))
|
||||
return
|
||||
except Exception as exc:
|
||||
last_exc = exc
|
||||
if attempt < _max_retries - 1:
|
||||
await asyncio.sleep(2**attempt)
|
||||
if last_exc:
|
||||
raise last_exc
|
||||
|
||||
body = {"msgtype": "markdown", "markdown": {"content": msg.text}}
|
||||
last_exc = None
|
||||
for attempt in range(_max_retries):
|
||||
try:
|
||||
await self._ws_client.send_message(msg.chat_id, body)
|
||||
return
|
||||
except Exception as exc:
|
||||
last_exc = exc
|
||||
if attempt < _max_retries - 1:
|
||||
await asyncio.sleep(2**attempt)
|
||||
if last_exc:
|
||||
raise last_exc
|
||||
|
||||
async def _upload_media_ws(
|
||||
self,
|
||||
*,
|
||||
media_type: str,
|
||||
filename: str,
|
||||
path: str,
|
||||
size: int,
|
||||
) -> str | None:
|
||||
if not self._ws_client:
|
||||
return None
|
||||
try:
|
||||
from aibot import generate_req_id
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
chunk_size = 512 * 1024
|
||||
total_chunks = (size + chunk_size - 1) // chunk_size
|
||||
if total_chunks < 1 or total_chunks > 100:
|
||||
logger.warning("[WeCom] invalid total_chunks=%d for %s", total_chunks, filename)
|
||||
return None
|
||||
|
||||
md5_hasher = hashlib.md5()
|
||||
with open(path, "rb") as f:
|
||||
for chunk in iter(lambda: f.read(1024 * 1024), b""):
|
||||
md5_hasher.update(chunk)
|
||||
md5 = md5_hasher.hexdigest()
|
||||
|
||||
init_req_id = generate_req_id("aibot_upload_media_init")
|
||||
init_body = {
|
||||
"type": media_type,
|
||||
"filename": filename,
|
||||
"total_size": int(size),
|
||||
"total_chunks": int(total_chunks),
|
||||
"md5": md5,
|
||||
}
|
||||
init_ack = await self._send_ws_upload_command(init_req_id, init_body, "aibot_upload_media_init")
|
||||
upload_id = (init_ack.get("body") or {}).get("upload_id")
|
||||
if not upload_id:
|
||||
logger.warning("[WeCom] upload init returned no upload_id: %s", init_ack)
|
||||
return None
|
||||
|
||||
with open(path, "rb") as f:
|
||||
for idx in range(total_chunks):
|
||||
data = f.read(chunk_size)
|
||||
if not data:
|
||||
break
|
||||
chunk_req_id = generate_req_id("aibot_upload_media_chunk")
|
||||
chunk_body = {
|
||||
"upload_id": upload_id,
|
||||
"chunk_index": int(idx),
|
||||
"base64_data": base64.b64encode(data).decode("utf-8"),
|
||||
}
|
||||
await self._send_ws_upload_command(chunk_req_id, chunk_body, "aibot_upload_media_chunk")
|
||||
|
||||
finish_req_id = generate_req_id("aibot_upload_media_finish")
|
||||
finish_ack = await self._send_ws_upload_command(finish_req_id, {"upload_id": upload_id}, "aibot_upload_media_finish")
|
||||
media_id = (finish_ack.get("body") or {}).get("media_id")
|
||||
if not media_id:
|
||||
logger.warning("[WeCom] upload finish returned no media_id: %s", finish_ack)
|
||||
return None
|
||||
return media_id
|
||||
@@ -1,4 +0,0 @@
|
||||
from .app import app, create_app
|
||||
from .config import GatewayConfig, get_gateway_config
|
||||
|
||||
__all__ = ["app", "create_app", "GatewayConfig", "get_gateway_config"]
|
||||
@@ -1,385 +0,0 @@
|
||||
import asyncio
|
||||
import logging
|
||||
from collections.abc import AsyncGenerator
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
|
||||
from app.gateway.auth_middleware import AuthMiddleware
|
||||
from app.gateway.config import get_gateway_config
|
||||
from app.gateway.csrf_middleware import CSRFMiddleware, get_configured_cors_origins
|
||||
from app.gateway.deps import langgraph_runtime
|
||||
from app.gateway.routers import (
|
||||
agents,
|
||||
artifacts,
|
||||
assistants_compat,
|
||||
auth,
|
||||
channels,
|
||||
feedback,
|
||||
mcp,
|
||||
memory,
|
||||
models,
|
||||
runs,
|
||||
skills,
|
||||
suggestions,
|
||||
thread_runs,
|
||||
threads,
|
||||
uploads,
|
||||
)
|
||||
from deerflow.config import app_config as deerflow_app_config
|
||||
from deerflow.config.app_config import apply_logging_level
|
||||
|
||||
AppConfig = deerflow_app_config.AppConfig
|
||||
get_app_config = deerflow_app_config.get_app_config
|
||||
|
||||
# Default logging; lifespan overrides from config.yaml log_level.
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
|
||||
datefmt="%Y-%m-%d %H:%M:%S",
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Upper bound (seconds) each lifespan shutdown hook is allowed to run.
|
||||
# Bounds worker exit time so uvicorn's reload supervisor does not keep
|
||||
# firing signals into a worker that is stuck waiting for shutdown cleanup.
|
||||
_SHUTDOWN_HOOK_TIMEOUT_SECONDS = 5.0
|
||||
|
||||
|
||||
async def _ensure_admin_user(app: FastAPI) -> None:
|
||||
"""Startup hook: handle first boot and migrate orphan threads otherwise.
|
||||
|
||||
After admin creation, migrate orphan threads from the LangGraph
|
||||
store (metadata.user_id unset) to the admin account. This is the
|
||||
"no-auth → with-auth" upgrade path: users who ran DeerFlow without
|
||||
authentication have existing LangGraph thread data that needs an
|
||||
owner assigned.
|
||||
First boot (no admin exists):
|
||||
- Does NOT create any user accounts automatically.
|
||||
- The operator must visit ``/setup`` to create the first admin.
|
||||
|
||||
Subsequent boots (admin already exists):
|
||||
- Runs the one-time "no-auth → with-auth" orphan thread migration for
|
||||
existing LangGraph thread metadata that has no user_id.
|
||||
|
||||
No SQL persistence migration is needed: the four user_id columns
|
||||
(threads_meta, runs, run_events, feedback) only come into existence
|
||||
alongside the auth module via create_all, so freshly created tables
|
||||
never contain NULL-owner rows.
|
||||
"""
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.gateway.deps import get_local_provider
|
||||
from deerflow.persistence.engine import get_session_factory
|
||||
from deerflow.persistence.user.model import UserRow
|
||||
|
||||
try:
|
||||
provider = get_local_provider()
|
||||
except RuntimeError:
|
||||
# Auth persistence may not be initialized in some test/boot paths.
|
||||
# Skip admin migration work rather than failing gateway startup.
|
||||
logger.warning("Auth persistence not ready; skipping admin bootstrap check")
|
||||
return
|
||||
|
||||
sf = get_session_factory()
|
||||
if sf is None:
|
||||
return
|
||||
|
||||
admin_count = await provider.count_admin_users()
|
||||
|
||||
if admin_count == 0:
|
||||
logger.info("=" * 60)
|
||||
logger.info(" First boot detected — no admin account exists.")
|
||||
logger.info(" Visit /setup to complete admin account creation.")
|
||||
logger.info("=" * 60)
|
||||
return
|
||||
|
||||
# Admin already exists — run orphan thread migration for any
|
||||
# LangGraph thread metadata that pre-dates the auth module.
|
||||
async with sf() as session:
|
||||
stmt = select(UserRow).where(UserRow.system_role == "admin").limit(1)
|
||||
row = (await session.execute(stmt)).scalar_one_or_none()
|
||||
|
||||
if row is None:
|
||||
return # Should not happen (admin_count > 0 above), but be safe.
|
||||
|
||||
admin_id = str(row.id)
|
||||
|
||||
# LangGraph store orphan migration — non-fatal.
|
||||
# This covers the "no-auth → with-auth" upgrade path for users
|
||||
# whose existing LangGraph thread metadata has no user_id set.
|
||||
store = getattr(app.state, "store", None)
|
||||
if store is not None:
|
||||
try:
|
||||
migrated = await _migrate_orphaned_threads(store, admin_id)
|
||||
if migrated:
|
||||
logger.info("Migrated %d orphan LangGraph thread(s) to admin", migrated)
|
||||
except Exception:
|
||||
logger.exception("LangGraph thread migration failed (non-fatal)")
|
||||
|
||||
|
||||
async def _iter_store_items(store, namespace, *, page_size: int = 500):
|
||||
"""Paginated async iterator over a LangGraph store namespace.
|
||||
|
||||
Replaces the old hardcoded ``limit=1000`` call with a cursor-style
|
||||
loop so that environments with more than one page of orphans do
|
||||
not silently lose data. Terminates when a page is empty OR when a
|
||||
short page arrives (indicating the last page).
|
||||
"""
|
||||
offset = 0
|
||||
while True:
|
||||
batch = await store.asearch(namespace, limit=page_size, offset=offset)
|
||||
if not batch:
|
||||
return
|
||||
for item in batch:
|
||||
yield item
|
||||
if len(batch) < page_size:
|
||||
return
|
||||
offset += page_size
|
||||
|
||||
|
||||
async def _migrate_orphaned_threads(store, admin_user_id: str) -> int:
|
||||
"""Migrate LangGraph store threads with no user_id to the given admin.
|
||||
|
||||
Uses cursor pagination so all orphans are migrated regardless of
|
||||
count. Returns the number of rows migrated.
|
||||
"""
|
||||
migrated = 0
|
||||
async for item in _iter_store_items(store, ("threads",)):
|
||||
metadata = item.value.get("metadata", {})
|
||||
if not metadata.get("user_id"):
|
||||
metadata["user_id"] = admin_user_id
|
||||
item.value["metadata"] = metadata
|
||||
await store.aput(("threads",), item.key, item.value)
|
||||
migrated += 1
|
||||
return migrated
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
|
||||
"""Application lifespan handler."""
|
||||
|
||||
# Load config and check necessary environment variables at startup
|
||||
try:
|
||||
app.state.config = get_app_config()
|
||||
apply_logging_level(app.state.config.log_level)
|
||||
logger.info("Configuration loaded successfully")
|
||||
except Exception as e:
|
||||
error_msg = f"Failed to load configuration during gateway startup: {e}"
|
||||
logger.exception(error_msg)
|
||||
raise RuntimeError(error_msg) from e
|
||||
config = get_gateway_config()
|
||||
logger.info(f"Starting API Gateway on {config.host}:{config.port}")
|
||||
|
||||
# Initialize LangGraph runtime components (StreamBridge, RunManager, checkpointer, store)
|
||||
async with langgraph_runtime(app):
|
||||
logger.info("LangGraph runtime initialised")
|
||||
|
||||
# Check admin bootstrap state and migrate orphan threads after admin exists.
|
||||
# Must run AFTER langgraph_runtime so app.state.store is available for thread migration
|
||||
await _ensure_admin_user(app)
|
||||
|
||||
# Start IM channel service if any channels are configured
|
||||
try:
|
||||
from app.channels.service import start_channel_service
|
||||
|
||||
channel_service = await start_channel_service(app.state.config)
|
||||
logger.info("Channel service started: %s", channel_service.get_status())
|
||||
except Exception:
|
||||
logger.exception("No IM channels configured or channel service failed to start")
|
||||
|
||||
yield
|
||||
|
||||
# Stop channel service on shutdown (bounded to prevent worker hang)
|
||||
try:
|
||||
from app.channels.service import stop_channel_service
|
||||
|
||||
await asyncio.wait_for(
|
||||
stop_channel_service(),
|
||||
timeout=_SHUTDOWN_HOOK_TIMEOUT_SECONDS,
|
||||
)
|
||||
except TimeoutError:
|
||||
logger.warning(
|
||||
"Channel service shutdown exceeded %.1fs; proceeding with worker exit.",
|
||||
_SHUTDOWN_HOOK_TIMEOUT_SECONDS,
|
||||
)
|
||||
except Exception:
|
||||
logger.exception("Failed to stop channel service")
|
||||
|
||||
logger.info("Shutting down API Gateway")
|
||||
|
||||
|
||||
def create_app() -> FastAPI:
|
||||
"""Create and configure the FastAPI application.
|
||||
|
||||
Returns:
|
||||
Configured FastAPI application instance.
|
||||
"""
|
||||
config = get_gateway_config()
|
||||
docs_url = "/docs" if config.enable_docs else None
|
||||
redoc_url = "/redoc" if config.enable_docs else None
|
||||
openapi_url = "/openapi.json" if config.enable_docs else None
|
||||
|
||||
app = FastAPI(
|
||||
title="DeerFlow API Gateway",
|
||||
description="""
|
||||
## DeerFlow API Gateway
|
||||
|
||||
API Gateway for DeerFlow - A LangGraph-based AI agent backend with sandbox execution capabilities.
|
||||
|
||||
### Features
|
||||
|
||||
- **Models Management**: Query and retrieve available AI models
|
||||
- **MCP Configuration**: Manage Model Context Protocol (MCP) server configurations
|
||||
- **Memory Management**: Access and manage global memory data for personalized conversations
|
||||
- **Skills Management**: Query and manage skills and their enabled status
|
||||
- **Artifacts**: Access thread artifacts and generated files
|
||||
- **Health Monitoring**: System health check endpoints
|
||||
|
||||
### Architecture
|
||||
|
||||
LangGraph-compatible requests are routed through nginx to this gateway.
|
||||
This gateway provides runtime endpoints for agent runs plus custom endpoints for models, MCP configuration, skills, and artifacts.
|
||||
""",
|
||||
version="0.1.0",
|
||||
lifespan=lifespan,
|
||||
docs_url=docs_url,
|
||||
redoc_url=redoc_url,
|
||||
openapi_url=openapi_url,
|
||||
openapi_tags=[
|
||||
{
|
||||
"name": "models",
|
||||
"description": "Operations for querying available AI models and their configurations",
|
||||
},
|
||||
{
|
||||
"name": "mcp",
|
||||
"description": "Manage Model Context Protocol (MCP) server configurations",
|
||||
},
|
||||
{
|
||||
"name": "memory",
|
||||
"description": "Access and manage global memory data for personalized conversations",
|
||||
},
|
||||
{
|
||||
"name": "skills",
|
||||
"description": "Manage skills and their configurations",
|
||||
},
|
||||
{
|
||||
"name": "artifacts",
|
||||
"description": "Access and download thread artifacts and generated files",
|
||||
},
|
||||
{
|
||||
"name": "uploads",
|
||||
"description": "Upload and manage user files for threads",
|
||||
},
|
||||
{
|
||||
"name": "threads",
|
||||
"description": "Manage DeerFlow thread-local filesystem data",
|
||||
},
|
||||
{
|
||||
"name": "agents",
|
||||
"description": "Create and manage custom agents with per-agent config and prompts",
|
||||
},
|
||||
{
|
||||
"name": "suggestions",
|
||||
"description": "Generate follow-up question suggestions for conversations",
|
||||
},
|
||||
{
|
||||
"name": "channels",
|
||||
"description": "Manage IM channel integrations (Feishu, Slack, Telegram)",
|
||||
},
|
||||
{
|
||||
"name": "assistants-compat",
|
||||
"description": "LangGraph Platform-compatible assistants API (stub)",
|
||||
},
|
||||
{
|
||||
"name": "runs",
|
||||
"description": "LangGraph Platform-compatible runs lifecycle (create, stream, cancel)",
|
||||
},
|
||||
{
|
||||
"name": "health",
|
||||
"description": "Health check and system status endpoints",
|
||||
},
|
||||
],
|
||||
)
|
||||
|
||||
# Auth: reject unauthenticated requests to non-public paths (fail-closed safety net)
|
||||
app.add_middleware(AuthMiddleware)
|
||||
|
||||
# CSRF: Double Submit Cookie pattern for state-changing requests
|
||||
app.add_middleware(CSRFMiddleware)
|
||||
|
||||
# CORS: the unified nginx endpoint is same-origin by default. Split-origin
|
||||
# browser clients must opt in with this explicit Gateway allowlist so CORS
|
||||
# and CSRF origin checks share the same source of truth.
|
||||
cors_origins = sorted(get_configured_cors_origins())
|
||||
if cors_origins:
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=cors_origins,
|
||||
allow_credentials=True,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
# Include routers
|
||||
# Models API is mounted at /api/models
|
||||
app.include_router(models.router)
|
||||
|
||||
# MCP API is mounted at /api/mcp
|
||||
app.include_router(mcp.router)
|
||||
|
||||
# Memory API is mounted at /api/memory
|
||||
app.include_router(memory.router)
|
||||
|
||||
# Skills API is mounted at /api/skills
|
||||
app.include_router(skills.router)
|
||||
|
||||
# Artifacts API is mounted at /api/threads/{thread_id}/artifacts
|
||||
app.include_router(artifacts.router)
|
||||
|
||||
# Uploads API is mounted at /api/threads/{thread_id}/uploads
|
||||
app.include_router(uploads.router)
|
||||
|
||||
# Thread cleanup API is mounted at /api/threads/{thread_id}
|
||||
app.include_router(threads.router)
|
||||
|
||||
# Agents API is mounted at /api/agents
|
||||
app.include_router(agents.router)
|
||||
|
||||
# Suggestions API is mounted at /api/threads/{thread_id}/suggestions
|
||||
app.include_router(suggestions.router)
|
||||
|
||||
# Channels API is mounted at /api/channels
|
||||
app.include_router(channels.router)
|
||||
|
||||
# Assistants compatibility API (LangGraph Platform stub)
|
||||
app.include_router(assistants_compat.router)
|
||||
|
||||
# Auth API is mounted at /api/v1/auth
|
||||
app.include_router(auth.router)
|
||||
|
||||
# Feedback API is mounted at /api/threads/{thread_id}/runs/{run_id}/feedback
|
||||
app.include_router(feedback.router)
|
||||
|
||||
# Thread Runs API (LangGraph Platform-compatible runs lifecycle)
|
||||
app.include_router(thread_runs.router)
|
||||
|
||||
# Stateless Runs API (stream/wait without a pre-existing thread)
|
||||
app.include_router(runs.router)
|
||||
|
||||
@app.get("/health", tags=["health"])
|
||||
async def health_check() -> dict[str, str]:
|
||||
"""Health check endpoint.
|
||||
|
||||
Returns:
|
||||
Service health status information.
|
||||
"""
|
||||
return {"status": "healthy", "service": "deer-flow-gateway"}
|
||||
|
||||
return app
|
||||
|
||||
|
||||
# Create app instance for uvicorn
|
||||
app = create_app()
|
||||
@@ -1,42 +0,0 @@
|
||||
"""Authentication module for DeerFlow.
|
||||
|
||||
This module provides:
|
||||
- JWT-based authentication
|
||||
- Provider Factory pattern for extensible auth methods
|
||||
- UserRepository interface for storage backends (SQLite)
|
||||
"""
|
||||
|
||||
from app.gateway.auth.config import AuthConfig, get_auth_config, set_auth_config
|
||||
from app.gateway.auth.errors import AuthErrorCode, AuthErrorResponse, TokenError
|
||||
from app.gateway.auth.jwt import TokenPayload, create_access_token, decode_token
|
||||
from app.gateway.auth.local_provider import LocalAuthProvider
|
||||
from app.gateway.auth.models import User, UserResponse
|
||||
from app.gateway.auth.password import hash_password, verify_password
|
||||
from app.gateway.auth.providers import AuthProvider
|
||||
from app.gateway.auth.repositories.base import UserRepository
|
||||
|
||||
__all__ = [
|
||||
# Config
|
||||
"AuthConfig",
|
||||
"get_auth_config",
|
||||
"set_auth_config",
|
||||
# Errors
|
||||
"AuthErrorCode",
|
||||
"AuthErrorResponse",
|
||||
"TokenError",
|
||||
# JWT
|
||||
"TokenPayload",
|
||||
"create_access_token",
|
||||
"decode_token",
|
||||
# Password
|
||||
"hash_password",
|
||||
"verify_password",
|
||||
# Models
|
||||
"User",
|
||||
"UserResponse",
|
||||
# Providers
|
||||
"AuthProvider",
|
||||
"LocalAuthProvider",
|
||||
# Repository
|
||||
"UserRepository",
|
||||
]
|
||||
@@ -1,57 +0,0 @@
|
||||
"""Authentication configuration for DeerFlow."""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import secrets
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class AuthConfig(BaseModel):
|
||||
"""JWT and auth-related configuration. Parsed once at startup.
|
||||
|
||||
Note: the ``users`` table now lives in the shared persistence
|
||||
database managed by ``deerflow.persistence.engine``. The old
|
||||
``users_db_path`` config key has been removed — user storage is
|
||||
configured through ``config.database`` like every other table.
|
||||
"""
|
||||
|
||||
jwt_secret: str = Field(
|
||||
...,
|
||||
description="Secret key for JWT signing. MUST be set via AUTH_JWT_SECRET.",
|
||||
)
|
||||
token_expiry_days: int = Field(default=7, ge=1, le=30)
|
||||
oauth_github_client_id: str | None = Field(default=None)
|
||||
oauth_github_client_secret: str | None = Field(default=None)
|
||||
|
||||
|
||||
_auth_config: AuthConfig | None = None
|
||||
|
||||
|
||||
def get_auth_config() -> AuthConfig:
|
||||
"""Get the global AuthConfig instance. Parses from env on first call."""
|
||||
global _auth_config
|
||||
if _auth_config is None:
|
||||
from dotenv import load_dotenv
|
||||
|
||||
load_dotenv()
|
||||
jwt_secret = os.environ.get("AUTH_JWT_SECRET")
|
||||
if not jwt_secret:
|
||||
jwt_secret = secrets.token_urlsafe(32)
|
||||
os.environ["AUTH_JWT_SECRET"] = jwt_secret
|
||||
logger.warning(
|
||||
"⚠ AUTH_JWT_SECRET is not set — using an auto-generated ephemeral secret. "
|
||||
"Sessions will be invalidated on restart. "
|
||||
"For production, add AUTH_JWT_SECRET to your .env file: "
|
||||
'python -c "import secrets; print(secrets.token_urlsafe(32))"'
|
||||
)
|
||||
_auth_config = AuthConfig(jwt_secret=jwt_secret)
|
||||
return _auth_config
|
||||
|
||||
|
||||
def set_auth_config(config: AuthConfig) -> None:
|
||||
"""Set the global AuthConfig instance (for testing)."""
|
||||
global _auth_config
|
||||
_auth_config = config
|
||||
@@ -1,48 +0,0 @@
|
||||
"""Write initial admin credentials to a restricted file instead of logs.
|
||||
|
||||
Logging secrets to stdout/stderr is a well-known CodeQL finding
|
||||
(py/clear-text-logging-sensitive-data) — in production those logs
|
||||
get collected into ELK/Splunk/etc and become a secret sprawl
|
||||
source. This helper writes the credential to a 0600 file that only
|
||||
the process user can read, and returns the path so the caller can
|
||||
log **the path** (not the password) for the operator to pick up.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from deerflow.config.paths import get_paths
|
||||
|
||||
_CREDENTIAL_FILENAME = "admin_initial_credentials.txt"
|
||||
|
||||
|
||||
def write_initial_credentials(email: str, password: str, *, label: str = "initial") -> Path:
|
||||
"""Write the admin email + password to ``{base_dir}/admin_initial_credentials.txt``.
|
||||
|
||||
The file is created **atomically** with mode 0600 via ``os.open``
|
||||
so the password is never world-readable, even for the single syscall
|
||||
window between ``write_text`` and ``chmod``.
|
||||
|
||||
``label`` distinguishes "initial" (fresh creation) from "reset"
|
||||
(password reset) in the file header so an operator picking up the
|
||||
file after a restart can tell which event produced it.
|
||||
|
||||
Returns the absolute :class:`Path` to the file.
|
||||
"""
|
||||
target = get_paths().base_dir / _CREDENTIAL_FILENAME
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
content = (
|
||||
f"# DeerFlow admin {label} credentials\n# This file is generated on first boot or password reset.\n# Change the password after login via Settings -> Account,\n# then delete this file.\n#\nemail: {email}\npassword: {password}\n"
|
||||
)
|
||||
|
||||
# Atomic 0600 create-or-truncate. O_TRUNC (not O_EXCL) so the
|
||||
# reset-password path can rewrite an existing file without a
|
||||
# separate unlink-then-create dance.
|
||||
fd = os.open(target, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as fh:
|
||||
fh.write(content)
|
||||
|
||||
return target.resolve()
|
||||
@@ -1,45 +0,0 @@
|
||||
"""Typed error definitions for auth module.
|
||||
|
||||
AuthErrorCode: exhaustive enum of all auth failure conditions.
|
||||
TokenError: exhaustive enum of JWT decode failures.
|
||||
AuthErrorResponse: structured error payload for HTTP responses.
|
||||
"""
|
||||
|
||||
from enum import StrEnum
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
|
||||
class AuthErrorCode(StrEnum):
|
||||
"""Exhaustive list of auth error conditions."""
|
||||
|
||||
INVALID_CREDENTIALS = "invalid_credentials"
|
||||
TOKEN_EXPIRED = "token_expired"
|
||||
TOKEN_INVALID = "token_invalid"
|
||||
USER_NOT_FOUND = "user_not_found"
|
||||
EMAIL_ALREADY_EXISTS = "email_already_exists"
|
||||
PROVIDER_NOT_FOUND = "provider_not_found"
|
||||
NOT_AUTHENTICATED = "not_authenticated"
|
||||
SYSTEM_ALREADY_INITIALIZED = "system_already_initialized"
|
||||
|
||||
|
||||
class TokenError(StrEnum):
|
||||
"""Exhaustive list of JWT decode failure reasons."""
|
||||
|
||||
EXPIRED = "expired"
|
||||
INVALID_SIGNATURE = "invalid_signature"
|
||||
MALFORMED = "malformed"
|
||||
|
||||
|
||||
class AuthErrorResponse(BaseModel):
|
||||
"""Structured error response — replaces bare `detail` strings."""
|
||||
|
||||
code: AuthErrorCode
|
||||
message: str
|
||||
|
||||
|
||||
def token_error_to_code(err: TokenError) -> AuthErrorCode:
|
||||
"""Map TokenError to AuthErrorCode — single source of truth."""
|
||||
if err == TokenError.EXPIRED:
|
||||
return AuthErrorCode.TOKEN_EXPIRED
|
||||
return AuthErrorCode.TOKEN_INVALID
|
||||
@@ -1,55 +0,0 @@
|
||||
"""JWT token creation and verification."""
|
||||
|
||||
from datetime import UTC, datetime, timedelta
|
||||
|
||||
import jwt
|
||||
from pydantic import BaseModel
|
||||
|
||||
from app.gateway.auth.config import get_auth_config
|
||||
from app.gateway.auth.errors import TokenError
|
||||
|
||||
|
||||
class TokenPayload(BaseModel):
|
||||
"""JWT token payload."""
|
||||
|
||||
sub: str # user_id
|
||||
exp: datetime
|
||||
iat: datetime | None = None
|
||||
ver: int = 0 # token_version — must match User.token_version
|
||||
|
||||
|
||||
def create_access_token(user_id: str, expires_delta: timedelta | None = None, token_version: int = 0) -> str:
|
||||
"""Create a JWT access token.
|
||||
|
||||
Args:
|
||||
user_id: The user's UUID as string
|
||||
expires_delta: Optional custom expiry, defaults to 7 days
|
||||
token_version: User's current token_version for invalidation
|
||||
|
||||
Returns:
|
||||
Encoded JWT string
|
||||
"""
|
||||
config = get_auth_config()
|
||||
expiry = expires_delta or timedelta(days=config.token_expiry_days)
|
||||
|
||||
now = datetime.now(UTC)
|
||||
payload = {"sub": user_id, "exp": now + expiry, "iat": now, "ver": token_version}
|
||||
return jwt.encode(payload, config.jwt_secret, algorithm="HS256")
|
||||
|
||||
|
||||
def decode_token(token: str) -> TokenPayload | TokenError:
|
||||
"""Decode and validate a JWT token.
|
||||
|
||||
Returns:
|
||||
TokenPayload if valid, or a specific TokenError variant.
|
||||
"""
|
||||
config = get_auth_config()
|
||||
try:
|
||||
payload = jwt.decode(token, config.jwt_secret, algorithms=["HS256"])
|
||||
return TokenPayload(**payload)
|
||||
except jwt.ExpiredSignatureError:
|
||||
return TokenError.EXPIRED
|
||||
except jwt.InvalidSignatureError:
|
||||
return TokenError.INVALID_SIGNATURE
|
||||
except jwt.PyJWTError:
|
||||
return TokenError.MALFORMED
|
||||
@@ -1,104 +0,0 @@
|
||||
"""Local email/password authentication provider."""
|
||||
|
||||
import logging
|
||||
|
||||
from app.gateway.auth.models import User
|
||||
from app.gateway.auth.password import hash_password_async, needs_rehash, verify_password_async
|
||||
from app.gateway.auth.providers import AuthProvider
|
||||
from app.gateway.auth.repositories.base import UserRepository
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class LocalAuthProvider(AuthProvider):
|
||||
"""Email/password authentication provider using local database."""
|
||||
|
||||
def __init__(self, repository: UserRepository):
|
||||
"""Initialize with a UserRepository.
|
||||
|
||||
Args:
|
||||
repository: UserRepository implementation (SQLite)
|
||||
"""
|
||||
self._repo = repository
|
||||
|
||||
async def authenticate(self, credentials: dict) -> User | None:
|
||||
"""Authenticate with email and password.
|
||||
|
||||
Args:
|
||||
credentials: dict with 'email' and 'password' keys
|
||||
|
||||
Returns:
|
||||
User if authentication succeeds, None otherwise
|
||||
"""
|
||||
email = credentials.get("email")
|
||||
password = credentials.get("password")
|
||||
|
||||
if not email or not password:
|
||||
return None
|
||||
|
||||
user = await self._repo.get_user_by_email(email)
|
||||
if user is None:
|
||||
return None
|
||||
|
||||
if user.password_hash is None:
|
||||
# OAuth user without local password
|
||||
return None
|
||||
|
||||
if not await verify_password_async(password, user.password_hash):
|
||||
return None
|
||||
|
||||
if needs_rehash(user.password_hash):
|
||||
try:
|
||||
user.password_hash = await hash_password_async(password)
|
||||
await self._repo.update_user(user)
|
||||
except Exception:
|
||||
# Rehash is an opportunistic upgrade; a transient DB error must not
|
||||
# prevent an otherwise-valid login from succeeding.
|
||||
logger.warning("Failed to rehash password for user %s; login will still succeed", user.email, exc_info=True)
|
||||
|
||||
return user
|
||||
|
||||
async def get_user(self, user_id: str) -> User | None:
|
||||
"""Get user by ID."""
|
||||
return await self._repo.get_user_by_id(user_id)
|
||||
|
||||
async def create_user(self, email: str, password: str | None = None, system_role: str = "user", needs_setup: bool = False) -> User:
|
||||
"""Create a new local user.
|
||||
|
||||
Args:
|
||||
email: User email address
|
||||
password: Plain text password (will be hashed)
|
||||
system_role: Role to assign ("admin" or "user")
|
||||
needs_setup: If True, user must complete setup on first login
|
||||
|
||||
Returns:
|
||||
Created User instance
|
||||
"""
|
||||
password_hash = await hash_password_async(password) if password else None
|
||||
user = User(
|
||||
email=email,
|
||||
password_hash=password_hash,
|
||||
system_role=system_role,
|
||||
needs_setup=needs_setup,
|
||||
)
|
||||
return await self._repo.create_user(user)
|
||||
|
||||
async def get_user_by_oauth(self, provider: str, oauth_id: str) -> User | None:
|
||||
"""Get user by OAuth provider and ID."""
|
||||
return await self._repo.get_user_by_oauth(provider, oauth_id)
|
||||
|
||||
async def count_users(self) -> int:
|
||||
"""Return total number of registered users."""
|
||||
return await self._repo.count_users()
|
||||
|
||||
async def count_admin_users(self) -> int:
|
||||
"""Return number of admin users."""
|
||||
return await self._repo.count_admin_users()
|
||||
|
||||
async def update_user(self, user: User) -> User:
|
||||
"""Update an existing user."""
|
||||
return await self._repo.update_user(user)
|
||||
|
||||
async def get_user_by_email(self, email: str) -> User | None:
|
||||
"""Get user by email."""
|
||||
return await self._repo.get_user_by_email(email)
|
||||
@@ -1,41 +0,0 @@
|
||||
"""User Pydantic models for authentication."""
|
||||
|
||||
from datetime import UTC, datetime
|
||||
from typing import Literal
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, EmailStr, Field
|
||||
|
||||
|
||||
def _utc_now() -> datetime:
|
||||
"""Return current UTC time (timezone-aware)."""
|
||||
return datetime.now(UTC)
|
||||
|
||||
|
||||
class User(BaseModel):
|
||||
"""Internal user representation."""
|
||||
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: UUID = Field(default_factory=uuid4, description="Primary key")
|
||||
email: EmailStr = Field(..., description="Unique email address")
|
||||
password_hash: str | None = Field(None, description="bcrypt hash, nullable for OAuth users")
|
||||
system_role: Literal["admin", "user"] = Field(default="user")
|
||||
created_at: datetime = Field(default_factory=_utc_now)
|
||||
|
||||
# OAuth linkage (optional)
|
||||
oauth_provider: str | None = Field(None, description="e.g. 'github', 'google'")
|
||||
oauth_id: str | None = Field(None, description="User ID from OAuth provider")
|
||||
|
||||
# Auth lifecycle
|
||||
needs_setup: bool = Field(default=False, description="True when a reset account must complete setup")
|
||||
token_version: int = Field(default=0, description="Incremented on password change to invalidate old JWTs")
|
||||
|
||||
|
||||
class UserResponse(BaseModel):
|
||||
"""Response model for user info endpoint."""
|
||||
|
||||
id: str
|
||||
email: str
|
||||
system_role: Literal["admin", "user"]
|
||||
needs_setup: bool = False
|
||||
@@ -1,81 +0,0 @@
|
||||
"""Password hashing utilities with versioned hash format.
|
||||
|
||||
Hash format: ``$dfv<N>$<bcrypt_hash>`` where ``<N>`` is the version.
|
||||
|
||||
- **v1** (legacy): ``bcrypt(password)`` — plain bcrypt, susceptible to
|
||||
72-byte silent truncation.
|
||||
- **v2** (current): ``bcrypt(b64(sha256(password)))`` — SHA-256 pre-hash
|
||||
avoids the 72-byte truncation limit so the full password contributes
|
||||
to the hash.
|
||||
|
||||
Verification auto-detects the version and falls back to v1 for hashes
|
||||
without a prefix, so existing deployments upgrade transparently on next
|
||||
login.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import hashlib
|
||||
|
||||
import bcrypt
|
||||
|
||||
_CURRENT_VERSION = 2
|
||||
_PREFIX_V2 = "$dfv2$"
|
||||
_PREFIX_V1 = "$dfv1$"
|
||||
|
||||
|
||||
def _pre_hash_v2(password: str) -> bytes:
|
||||
"""SHA-256 pre-hash to bypass bcrypt's 72-byte limit."""
|
||||
return base64.b64encode(hashlib.sha256(password.encode("utf-8")).digest())
|
||||
|
||||
|
||||
def hash_password(password: str) -> str:
|
||||
"""Hash a password (current version: v2 — SHA-256 + bcrypt)."""
|
||||
raw = bcrypt.hashpw(_pre_hash_v2(password), bcrypt.gensalt()).decode("utf-8")
|
||||
return f"{_PREFIX_V2}{raw}"
|
||||
|
||||
|
||||
def verify_password(plain_password: str, hashed_password: str) -> bool:
|
||||
"""Verify a password, auto-detecting the hash version.
|
||||
|
||||
Accepts v2 (``$dfv2$…``), v1 (``$dfv1$…``), and bare bcrypt hashes
|
||||
(treated as v1 for backward compatibility with pre-versioning data).
|
||||
"""
|
||||
try:
|
||||
if hashed_password.startswith(_PREFIX_V2):
|
||||
bcrypt_hash = hashed_password[len(_PREFIX_V2) :]
|
||||
return bcrypt.checkpw(_pre_hash_v2(plain_password), bcrypt_hash.encode("utf-8"))
|
||||
|
||||
if hashed_password.startswith(_PREFIX_V1):
|
||||
bcrypt_hash = hashed_password[len(_PREFIX_V1) :]
|
||||
else:
|
||||
bcrypt_hash = hashed_password
|
||||
|
||||
return bcrypt.checkpw(plain_password.encode("utf-8"), bcrypt_hash.encode("utf-8"))
|
||||
except ValueError:
|
||||
# bcrypt raises ValueError for malformed or corrupt hashes (e.g., invalid salt).
|
||||
# Fail closed rather than crashing the request.
|
||||
return False
|
||||
|
||||
|
||||
def needs_rehash(hashed_password: str) -> bool:
|
||||
"""Return True if the hash uses an older version and should be rehashed."""
|
||||
return not hashed_password.startswith(_PREFIX_V2)
|
||||
|
||||
|
||||
async def hash_password_async(password: str) -> str:
|
||||
"""Hash a password using bcrypt (non-blocking).
|
||||
|
||||
Wraps the blocking bcrypt operation in a thread pool to avoid
|
||||
blocking the event loop during password hashing.
|
||||
"""
|
||||
return await asyncio.to_thread(hash_password, password)
|
||||
|
||||
|
||||
async def verify_password_async(plain_password: str, hashed_password: str) -> bool:
|
||||
"""Verify a password against its hash (non-blocking).
|
||||
|
||||
Wraps the blocking bcrypt operation in a thread pool to avoid
|
||||
blocking the event loop during password verification.
|
||||
"""
|
||||
return await asyncio.to_thread(verify_password, plain_password, hashed_password)
|
||||
@@ -1,24 +0,0 @@
|
||||
"""Auth provider abstraction."""
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
|
||||
|
||||
class AuthProvider(ABC):
|
||||
"""Abstract base class for authentication providers."""
|
||||
|
||||
@abstractmethod
|
||||
async def authenticate(self, credentials: dict) -> "User | None":
|
||||
"""Authenticate user with given credentials.
|
||||
|
||||
Returns User if authentication succeeds, None otherwise.
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def get_user(self, user_id: str) -> "User | None":
|
||||
"""Retrieve user by ID."""
|
||||
raise NotImplementedError
|
||||
|
||||
|
||||
# Import User at runtime to avoid circular imports
|
||||
from app.gateway.auth.models import User # noqa: E402
|
||||
@@ -1,102 +0,0 @@
|
||||
"""User repository interface for abstracting database operations."""
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
|
||||
from app.gateway.auth.models import User
|
||||
|
||||
|
||||
class UserNotFoundError(LookupError):
|
||||
"""Raised when a user repository operation targets a non-existent row.
|
||||
|
||||
Subclass of :class:`LookupError` so callers that already catch
|
||||
``LookupError`` for "missing entity" can keep working unchanged,
|
||||
while specific call sites can pin to this class to distinguish
|
||||
"concurrent delete during update" from other lookups.
|
||||
"""
|
||||
|
||||
|
||||
class UserRepository(ABC):
|
||||
"""Abstract interface for user data storage.
|
||||
|
||||
Implement this interface to support different storage backends
|
||||
(SQLite)
|
||||
"""
|
||||
|
||||
@abstractmethod
|
||||
async def create_user(self, user: User) -> User:
|
||||
"""Create a new user.
|
||||
|
||||
Args:
|
||||
user: User object to create
|
||||
|
||||
Returns:
|
||||
Created User with ID assigned
|
||||
|
||||
Raises:
|
||||
ValueError: If email already exists
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def get_user_by_id(self, user_id: str) -> User | None:
|
||||
"""Get user by ID.
|
||||
|
||||
Args:
|
||||
user_id: User UUID as string
|
||||
|
||||
Returns:
|
||||
User if found, None otherwise
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def get_user_by_email(self, email: str) -> User | None:
|
||||
"""Get user by email.
|
||||
|
||||
Args:
|
||||
email: User email address
|
||||
|
||||
Returns:
|
||||
User if found, None otherwise
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def update_user(self, user: User) -> User:
|
||||
"""Update an existing user.
|
||||
|
||||
Args:
|
||||
user: User object with updated fields
|
||||
|
||||
Returns:
|
||||
Updated User
|
||||
|
||||
Raises:
|
||||
UserNotFoundError: If no row exists for ``user.id``. This is
|
||||
a hard failure (not a no-op) so callers cannot mistake a
|
||||
concurrent-delete race for a successful update.
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def count_users(self) -> int:
|
||||
"""Return total number of registered users."""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def count_admin_users(self) -> int:
|
||||
"""Return number of users with system_role == 'admin'."""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def get_user_by_oauth(self, provider: str, oauth_id: str) -> User | None:
|
||||
"""Get user by OAuth provider and ID.
|
||||
|
||||
Args:
|
||||
provider: OAuth provider name (e.g. 'github', 'google')
|
||||
oauth_id: User ID from the OAuth provider
|
||||
|
||||
Returns:
|
||||
User if found, None otherwise
|
||||
"""
|
||||
raise NotImplementedError
|
||||
@@ -1,127 +0,0 @@
|
||||
"""SQLAlchemy-backed UserRepository implementation.
|
||||
|
||||
Uses the shared async session factory from
|
||||
``deerflow.persistence.engine`` — the ``users`` table lives in the
|
||||
same database as ``threads_meta``, ``runs``, ``run_events``, and
|
||||
``feedback``.
|
||||
|
||||
Constructor takes the session factory directly (same pattern as the
|
||||
other four repositories in ``deerflow.persistence.*``). Callers
|
||||
construct this after ``init_engine_from_config()`` has run.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC
|
||||
from uuid import UUID
|
||||
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
|
||||
from app.gateway.auth.models import User
|
||||
from app.gateway.auth.repositories.base import UserNotFoundError, UserRepository
|
||||
from deerflow.persistence.user.model import UserRow
|
||||
|
||||
|
||||
class SQLiteUserRepository(UserRepository):
|
||||
"""Async user repository backed by the shared SQLAlchemy engine."""
|
||||
|
||||
def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:
|
||||
self._sf = session_factory
|
||||
|
||||
# ── Converters ────────────────────────────────────────────────────
|
||||
|
||||
@staticmethod
|
||||
def _row_to_user(row: UserRow) -> User:
|
||||
return User(
|
||||
id=UUID(row.id),
|
||||
email=row.email,
|
||||
password_hash=row.password_hash,
|
||||
system_role=row.system_role, # type: ignore[arg-type]
|
||||
# SQLite loses tzinfo on read; reattach UTC so downstream
|
||||
# code can compare timestamps reliably.
|
||||
created_at=row.created_at if row.created_at.tzinfo else row.created_at.replace(tzinfo=UTC),
|
||||
oauth_provider=row.oauth_provider,
|
||||
oauth_id=row.oauth_id,
|
||||
needs_setup=row.needs_setup,
|
||||
token_version=row.token_version,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _user_to_row(user: User) -> UserRow:
|
||||
return UserRow(
|
||||
id=str(user.id),
|
||||
email=user.email,
|
||||
password_hash=user.password_hash,
|
||||
system_role=user.system_role,
|
||||
created_at=user.created_at,
|
||||
oauth_provider=user.oauth_provider,
|
||||
oauth_id=user.oauth_id,
|
||||
needs_setup=user.needs_setup,
|
||||
token_version=user.token_version,
|
||||
)
|
||||
|
||||
# ── CRUD ──────────────────────────────────────────────────────────
|
||||
|
||||
async def create_user(self, user: User) -> User:
|
||||
"""Insert a new user. Raises ``ValueError`` on duplicate email."""
|
||||
row = self._user_to_row(user)
|
||||
async with self._sf() as session:
|
||||
session.add(row)
|
||||
try:
|
||||
await session.commit()
|
||||
except IntegrityError as exc:
|
||||
await session.rollback()
|
||||
raise ValueError(f"Email already registered: {user.email}") from exc
|
||||
return user
|
||||
|
||||
async def get_user_by_id(self, user_id: str) -> User | None:
|
||||
async with self._sf() as session:
|
||||
row = await session.get(UserRow, user_id)
|
||||
return self._row_to_user(row) if row is not None else None
|
||||
|
||||
async def get_user_by_email(self, email: str) -> User | None:
|
||||
stmt = select(UserRow).where(UserRow.email == email)
|
||||
async with self._sf() as session:
|
||||
result = await session.execute(stmt)
|
||||
row = result.scalar_one_or_none()
|
||||
return self._row_to_user(row) if row is not None else None
|
||||
|
||||
async def update_user(self, user: User) -> User:
|
||||
async with self._sf() as session:
|
||||
row = await session.get(UserRow, str(user.id))
|
||||
if row is None:
|
||||
# Hard fail on concurrent delete: callers (reset_admin,
|
||||
# password change handlers, _ensure_admin_user) all
|
||||
# fetched the user just before this call, so a missing
|
||||
# row here means the row vanished underneath us. Silent
|
||||
# success would let the caller log "password reset" for
|
||||
# a row that no longer exists.
|
||||
raise UserNotFoundError(f"User {user.id} no longer exists")
|
||||
row.email = user.email
|
||||
row.password_hash = user.password_hash
|
||||
row.system_role = user.system_role
|
||||
row.oauth_provider = user.oauth_provider
|
||||
row.oauth_id = user.oauth_id
|
||||
row.needs_setup = user.needs_setup
|
||||
row.token_version = user.token_version
|
||||
await session.commit()
|
||||
return user
|
||||
|
||||
async def count_users(self) -> int:
|
||||
stmt = select(func.count()).select_from(UserRow)
|
||||
async with self._sf() as session:
|
||||
return await session.scalar(stmt) or 0
|
||||
|
||||
async def count_admin_users(self) -> int:
|
||||
stmt = select(func.count()).select_from(UserRow).where(UserRow.system_role == "admin")
|
||||
async with self._sf() as session:
|
||||
return await session.scalar(stmt) or 0
|
||||
|
||||
async def get_user_by_oauth(self, provider: str, oauth_id: str) -> User | None:
|
||||
stmt = select(UserRow).where(UserRow.oauth_provider == provider, UserRow.oauth_id == oauth_id)
|
||||
async with self._sf() as session:
|
||||
result = await session.execute(stmt)
|
||||
row = result.scalar_one_or_none()
|
||||
return self._row_to_user(row) if row is not None else None
|
||||
@@ -1,91 +0,0 @@
|
||||
"""CLI tool to reset an admin password.
|
||||
|
||||
Usage:
|
||||
python -m app.gateway.auth.reset_admin
|
||||
python -m app.gateway.auth.reset_admin --email admin@example.com
|
||||
|
||||
Writes the new password to ``.deer-flow/admin_initial_credentials.txt``
|
||||
(mode 0600) instead of printing it, so CI / log aggregators never see
|
||||
the cleartext secret.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import secrets
|
||||
import sys
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.gateway.auth.credential_file import write_initial_credentials
|
||||
from app.gateway.auth.password import hash_password
|
||||
from app.gateway.auth.repositories.sqlite import SQLiteUserRepository
|
||||
from deerflow.persistence.user.model import UserRow
|
||||
|
||||
|
||||
async def _run(email: str | None) -> int:
|
||||
from deerflow.config import get_app_config
|
||||
from deerflow.persistence.engine import (
|
||||
close_engine,
|
||||
get_session_factory,
|
||||
init_engine_from_config,
|
||||
)
|
||||
|
||||
config = get_app_config()
|
||||
await init_engine_from_config(config.database)
|
||||
try:
|
||||
sf = get_session_factory()
|
||||
if sf is None:
|
||||
print("Error: persistence engine not available (check config.database).", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
repo = SQLiteUserRepository(sf)
|
||||
|
||||
if email:
|
||||
user = await repo.get_user_by_email(email)
|
||||
else:
|
||||
# Find first admin via direct SELECT — repository does not
|
||||
# expose a "first admin" helper and we do not want to add
|
||||
# one just for this CLI.
|
||||
async with sf() as session:
|
||||
stmt = select(UserRow).where(UserRow.system_role == "admin").limit(1)
|
||||
row = (await session.execute(stmt)).scalar_one_or_none()
|
||||
if row is None:
|
||||
user = None
|
||||
else:
|
||||
user = await repo.get_user_by_id(row.id)
|
||||
|
||||
if user is None:
|
||||
if email:
|
||||
print(f"Error: user '{email}' not found.", file=sys.stderr)
|
||||
else:
|
||||
print("Error: no admin user found.", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
new_password = secrets.token_urlsafe(16)
|
||||
user.password_hash = hash_password(new_password)
|
||||
user.token_version += 1
|
||||
user.needs_setup = True
|
||||
await repo.update_user(user)
|
||||
|
||||
cred_path = write_initial_credentials(user.email, new_password, label="reset")
|
||||
print(f"Password reset for: {user.email}")
|
||||
print(f"Credentials written to: {cred_path} (mode 0600)")
|
||||
print("Next login will require setup (new email + password).")
|
||||
return 0
|
||||
finally:
|
||||
await close_engine()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description="Reset admin password")
|
||||
parser.add_argument("--email", help="Admin email (default: first admin found)")
|
||||
args = parser.parse_args()
|
||||
|
||||
exit_code = asyncio.run(_run(args.email))
|
||||
sys.exit(exit_code)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,126 +0,0 @@
|
||||
"""Global authentication middleware — fail-closed safety net.
|
||||
|
||||
Rejects unauthenticated requests to non-public paths with 401. When a
|
||||
request passes the cookie check, resolves the JWT payload to a real
|
||||
``User`` object and stamps it into both ``request.state.user`` and the
|
||||
``deerflow.runtime.user_context`` contextvar so that repository-layer
|
||||
owner filtering works automatically via the sentinel pattern.
|
||||
|
||||
Fine-grained permission checks remain in authz.py decorators.
|
||||
"""
|
||||
|
||||
from collections.abc import Callable
|
||||
|
||||
from fastapi import HTTPException, Request, Response
|
||||
from starlette.middleware.base import BaseHTTPMiddleware
|
||||
from starlette.responses import JSONResponse
|
||||
from starlette.types import ASGIApp
|
||||
|
||||
from app.gateway.auth.errors import AuthErrorCode, AuthErrorResponse
|
||||
from app.gateway.authz import _ALL_PERMISSIONS, AuthContext
|
||||
from app.gateway.internal_auth import INTERNAL_AUTH_HEADER_NAME, get_internal_user, is_valid_internal_auth_token
|
||||
from deerflow.runtime.user_context import reset_current_user, set_current_user
|
||||
|
||||
# Paths that never require authentication.
|
||||
_PUBLIC_PATH_PREFIXES: tuple[str, ...] = (
|
||||
"/health",
|
||||
"/docs",
|
||||
"/redoc",
|
||||
"/openapi.json",
|
||||
)
|
||||
|
||||
# Exact auth paths that are public (login/register/status check).
|
||||
# /api/v1/auth/me, /api/v1/auth/change-password etc. are NOT public.
|
||||
_PUBLIC_EXACT_PATHS: frozenset[str] = frozenset(
|
||||
{
|
||||
"/api/v1/auth/login/local",
|
||||
"/api/v1/auth/register",
|
||||
"/api/v1/auth/logout",
|
||||
"/api/v1/auth/setup-status",
|
||||
"/api/v1/auth/initialize",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _is_public(path: str) -> bool:
|
||||
stripped = path.rstrip("/")
|
||||
if stripped in _PUBLIC_EXACT_PATHS:
|
||||
return True
|
||||
return any(path.startswith(prefix) for prefix in _PUBLIC_PATH_PREFIXES)
|
||||
|
||||
|
||||
class AuthMiddleware(BaseHTTPMiddleware):
|
||||
"""Strict auth gate: reject requests without a valid session.
|
||||
|
||||
Two-stage check for non-public paths:
|
||||
|
||||
1. Cookie presence — return 401 NOT_AUTHENTICATED if missing
|
||||
2. JWT validation via ``get_optional_user_from_request`` — return 401
|
||||
TOKEN_INVALID if the token is absent, malformed, expired, or the
|
||||
signed user does not exist / is stale
|
||||
|
||||
On success, stamps ``request.state.user`` and the
|
||||
``deerflow.runtime.user_context`` contextvar so that repository-layer
|
||||
owner filters work downstream without every route needing a
|
||||
``@require_auth`` decorator. Routes that need per-resource
|
||||
authorization (e.g. "user A cannot read user B's thread by guessing
|
||||
the URL") should additionally use ``@require_permission(...,
|
||||
owner_check=True)`` for explicit enforcement — but authentication
|
||||
itself is fully handled here.
|
||||
"""
|
||||
|
||||
def __init__(self, app: ASGIApp) -> None:
|
||||
super().__init__(app)
|
||||
|
||||
async def dispatch(self, request: Request, call_next: Callable) -> Response:
|
||||
if _is_public(request.url.path):
|
||||
return await call_next(request)
|
||||
|
||||
internal_user = None
|
||||
if is_valid_internal_auth_token(request.headers.get(INTERNAL_AUTH_HEADER_NAME)):
|
||||
internal_user = get_internal_user()
|
||||
|
||||
# Non-public path: require session cookie
|
||||
if internal_user is None and not request.cookies.get("access_token"):
|
||||
return JSONResponse(
|
||||
status_code=401,
|
||||
content={
|
||||
"detail": AuthErrorResponse(
|
||||
code=AuthErrorCode.NOT_AUTHENTICATED,
|
||||
message="Authentication required",
|
||||
).model_dump()
|
||||
},
|
||||
)
|
||||
|
||||
# Strict JWT validation: reject junk/expired tokens with 401
|
||||
# right here instead of silently passing through. This closes
|
||||
# the "junk cookie bypass" gap (AUTH_TEST_PLAN test 7.5.8):
|
||||
# without this, non-isolation routes like /api/models would
|
||||
# accept any cookie-shaped string as authentication.
|
||||
#
|
||||
# We call the *strict* resolver so that fine-grained error
|
||||
# codes (token_expired, token_invalid, user_not_found, …)
|
||||
# propagate from AuthErrorCode, not get flattened into one
|
||||
# generic code. BaseHTTPMiddleware doesn't let HTTPException
|
||||
# bubble up, so we catch and render it as JSONResponse here.
|
||||
from app.gateway.deps import get_current_user_from_request
|
||||
|
||||
if internal_user is not None:
|
||||
user = internal_user
|
||||
else:
|
||||
try:
|
||||
user = await get_current_user_from_request(request)
|
||||
except HTTPException as exc:
|
||||
return JSONResponse(status_code=exc.status_code, content={"detail": exc.detail})
|
||||
|
||||
# Stamp both request.state.user (for the contextvar pattern)
|
||||
# and request.state.auth (so @require_permission's "auth is
|
||||
# None" branch short-circuits instead of running the entire
|
||||
# JWT-decode + DB-lookup pipeline a second time per request).
|
||||
request.state.user = user
|
||||
request.state.auth = AuthContext(user=user, permissions=_ALL_PERMISSIONS)
|
||||
token = set_current_user(user)
|
||||
try:
|
||||
return await call_next(request)
|
||||
finally:
|
||||
reset_current_user(token)
|
||||
@@ -1,301 +0,0 @@
|
||||
"""Authorization decorators and context for DeerFlow.
|
||||
|
||||
Inspired by LangGraph Auth system: https://github.com/langchain-ai/langgraph/blob/main/libs/sdk-py/langgraph_sdk/auth/__init__.py
|
||||
|
||||
**Usage:**
|
||||
|
||||
1. Use ``@require_auth`` on routes that need authentication
|
||||
2. Use ``@require_permission("resource", "action", filter_key=...)`` for permission checks
|
||||
3. The decorator chain processes from bottom to top
|
||||
|
||||
**Example:**
|
||||
|
||||
@router.get("/{thread_id}")
|
||||
@require_auth
|
||||
@require_permission("threads", "read", owner_check=True)
|
||||
async def get_thread(thread_id: str, request: Request):
|
||||
# User is authenticated and has threads:read permission
|
||||
...
|
||||
|
||||
**Permission Model:**
|
||||
|
||||
- threads:read - View thread
|
||||
- threads:write - Create/update thread
|
||||
- threads:delete - Delete thread
|
||||
- runs:create - Run agent
|
||||
- runs:read - View run
|
||||
- runs:cancel - Cancel run
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import functools
|
||||
import inspect
|
||||
from collections.abc import Callable
|
||||
from types import SimpleNamespace
|
||||
from typing import TYPE_CHECKING, Any, ParamSpec, TypeVar
|
||||
|
||||
from fastapi import HTTPException, Request
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from app.gateway.auth.models import User
|
||||
|
||||
P = ParamSpec("P")
|
||||
T = TypeVar("T")
|
||||
|
||||
|
||||
# Permission constants
|
||||
class Permissions:
|
||||
"""Permission constants for resource:action format."""
|
||||
|
||||
# Threads
|
||||
THREADS_READ = "threads:read"
|
||||
THREADS_WRITE = "threads:write"
|
||||
THREADS_DELETE = "threads:delete"
|
||||
|
||||
# Runs
|
||||
RUNS_CREATE = "runs:create"
|
||||
RUNS_READ = "runs:read"
|
||||
RUNS_CANCEL = "runs:cancel"
|
||||
|
||||
|
||||
class AuthContext:
|
||||
"""Authentication context for the current request.
|
||||
|
||||
Stored in request.state.auth after require_auth decoration.
|
||||
|
||||
Attributes:
|
||||
user: The authenticated user, or None if anonymous
|
||||
permissions: List of permission strings (e.g., "threads:read")
|
||||
"""
|
||||
|
||||
__slots__ = ("user", "permissions")
|
||||
|
||||
def __init__(self, user: User | None = None, permissions: list[str] | None = None):
|
||||
self.user = user
|
||||
self.permissions = permissions or []
|
||||
|
||||
@property
|
||||
def is_authenticated(self) -> bool:
|
||||
"""Check if user is authenticated."""
|
||||
return self.user is not None
|
||||
|
||||
def has_permission(self, resource: str, action: str) -> bool:
|
||||
"""Check if context has permission for resource:action.
|
||||
|
||||
Args:
|
||||
resource: Resource name (e.g., "threads")
|
||||
action: Action name (e.g., "read")
|
||||
|
||||
Returns:
|
||||
True if user has permission
|
||||
"""
|
||||
permission = f"{resource}:{action}"
|
||||
return permission in self.permissions
|
||||
|
||||
def require_user(self) -> User:
|
||||
"""Get user or raise 401.
|
||||
|
||||
Raises:
|
||||
HTTPException 401 if not authenticated
|
||||
"""
|
||||
if not self.user:
|
||||
raise HTTPException(status_code=401, detail="Authentication required")
|
||||
return self.user
|
||||
|
||||
|
||||
def get_auth_context(request: Request) -> AuthContext | None:
|
||||
"""Get AuthContext from request state."""
|
||||
return getattr(request.state, "auth", None)
|
||||
|
||||
|
||||
_ALL_PERMISSIONS: list[str] = [
|
||||
Permissions.THREADS_READ,
|
||||
Permissions.THREADS_WRITE,
|
||||
Permissions.THREADS_DELETE,
|
||||
Permissions.RUNS_CREATE,
|
||||
Permissions.RUNS_READ,
|
||||
Permissions.RUNS_CANCEL,
|
||||
]
|
||||
|
||||
|
||||
def _make_test_request_stub() -> Any:
|
||||
"""Create a minimal request-like object for direct unit calls.
|
||||
|
||||
Used when decorated route handlers are invoked without FastAPI's
|
||||
request injection. Includes fields accessed by auth helpers.
|
||||
"""
|
||||
return SimpleNamespace(state=SimpleNamespace(), cookies={}, _deerflow_test_bypass_auth=True)
|
||||
|
||||
|
||||
async def _authenticate(request: Request) -> AuthContext:
|
||||
"""Authenticate request and return AuthContext.
|
||||
|
||||
Delegates to deps.get_optional_user_from_request() for the JWT→User pipeline.
|
||||
Returns AuthContext with user=None for anonymous requests.
|
||||
"""
|
||||
from app.gateway.deps import get_optional_user_from_request
|
||||
|
||||
user = await get_optional_user_from_request(request)
|
||||
if user is None:
|
||||
return AuthContext(user=None, permissions=[])
|
||||
|
||||
# In future, permissions could be stored in user record
|
||||
return AuthContext(user=user, permissions=_ALL_PERMISSIONS)
|
||||
|
||||
|
||||
def require_auth[**P, T](func: Callable[P, T]) -> Callable[P, T]:
|
||||
"""Decorator that authenticates the request and enforces authentication.
|
||||
|
||||
Independently raises HTTP 401 for unauthenticated requests, regardless of
|
||||
whether ``AuthMiddleware`` is present in the ASGI stack. Sets the resolved
|
||||
``AuthContext`` on ``request.state.auth`` for downstream handlers.
|
||||
|
||||
Must be placed ABOVE other decorators (executes after them).
|
||||
|
||||
Usage:
|
||||
@router.get("/{thread_id}")
|
||||
@require_auth # Bottom decorator (executes first after permission check)
|
||||
@require_permission("threads", "read")
|
||||
async def get_thread(thread_id: str, request: Request):
|
||||
auth: AuthContext = request.state.auth
|
||||
...
|
||||
|
||||
Raises:
|
||||
HTTPException: 401 if the request is unauthenticated.
|
||||
ValueError: If 'request' parameter is missing.
|
||||
"""
|
||||
|
||||
@functools.wraps(func)
|
||||
async def wrapper(*args: Any, **kwargs: Any) -> Any:
|
||||
request = kwargs.get("request")
|
||||
if request is None:
|
||||
# Unit tests may call decorated handlers directly without a
|
||||
# FastAPI Request object. Inject a minimal request stub when
|
||||
# the wrapped function declares `request`.
|
||||
if "request" in inspect.signature(func).parameters:
|
||||
kwargs["request"] = _make_test_request_stub()
|
||||
else:
|
||||
raise ValueError("require_auth decorator requires 'request' parameter")
|
||||
request = kwargs["request"]
|
||||
|
||||
if getattr(request, "_deerflow_test_bypass_auth", False):
|
||||
return await func(*args, **kwargs)
|
||||
|
||||
# Authenticate and set context
|
||||
auth_context = await _authenticate(request)
|
||||
request.state.auth = auth_context
|
||||
|
||||
if not auth_context.is_authenticated:
|
||||
raise HTTPException(status_code=401, detail="Authentication required")
|
||||
|
||||
return await func(*args, **kwargs)
|
||||
|
||||
return wrapper
|
||||
|
||||
|
||||
def require_permission(
|
||||
resource: str,
|
||||
action: str,
|
||||
owner_check: bool = False,
|
||||
require_existing: bool = False,
|
||||
) -> Callable[[Callable[P, T]], Callable[P, T]]:
|
||||
"""Decorator that checks permission for resource:action.
|
||||
|
||||
Must be used AFTER @require_auth.
|
||||
|
||||
Args:
|
||||
resource: Resource name (e.g., "threads", "runs")
|
||||
action: Action name (e.g., "read", "write", "delete")
|
||||
owner_check: If True, validates that the current user owns the resource.
|
||||
Requires 'thread_id' path parameter and performs ownership check.
|
||||
require_existing: Only meaningful with ``owner_check=True``. If True, a
|
||||
missing ``threads_meta`` row counts as a denial (404)
|
||||
instead of "untracked legacy thread, allow". Use on
|
||||
**destructive / mutating** routes (DELETE, PATCH,
|
||||
state-update) so a deleted thread can't be re-targeted
|
||||
by another user via the missing-row code path.
|
||||
|
||||
Usage:
|
||||
# Read-style: legacy untracked threads are allowed
|
||||
@require_permission("threads", "read", owner_check=True)
|
||||
async def get_thread(thread_id: str, request: Request):
|
||||
...
|
||||
|
||||
# Destructive: thread row MUST exist and be owned by caller
|
||||
@require_permission("threads", "delete", owner_check=True, require_existing=True)
|
||||
async def delete_thread(thread_id: str, request: Request):
|
||||
...
|
||||
|
||||
Raises:
|
||||
HTTPException 401: If authentication required but user is anonymous
|
||||
HTTPException 403: If user lacks permission
|
||||
HTTPException 404: If owner_check=True but user doesn't own the thread
|
||||
ValueError: If owner_check=True but 'thread_id' parameter is missing
|
||||
"""
|
||||
|
||||
def decorator(func: Callable[P, T]) -> Callable[P, T]:
|
||||
@functools.wraps(func)
|
||||
async def wrapper(*args: Any, **kwargs: Any) -> Any:
|
||||
request = kwargs.get("request")
|
||||
if request is None:
|
||||
# Unit tests may call decorated route handlers directly without
|
||||
# constructing a FastAPI Request object. Inject a minimal stub
|
||||
# when the wrapped function declares `request`.
|
||||
if "request" in inspect.signature(func).parameters:
|
||||
kwargs["request"] = _make_test_request_stub()
|
||||
else:
|
||||
return await func(*args, **kwargs)
|
||||
request = kwargs["request"]
|
||||
|
||||
if getattr(request, "_deerflow_test_bypass_auth", False):
|
||||
return await func(*args, **kwargs)
|
||||
|
||||
auth: AuthContext = getattr(request.state, "auth", None)
|
||||
if auth is None:
|
||||
auth = await _authenticate(request)
|
||||
request.state.auth = auth
|
||||
|
||||
if not auth.is_authenticated:
|
||||
raise HTTPException(status_code=401, detail="Authentication required")
|
||||
|
||||
# Check permission
|
||||
if not auth.has_permission(resource, action):
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail=f"Permission denied: {resource}:{action}",
|
||||
)
|
||||
|
||||
# Owner check for thread-specific resources.
|
||||
#
|
||||
# 2.0-rc moved thread metadata into the SQL persistence layer
|
||||
# (``threads_meta`` table). We verify ownership via
|
||||
# ``ThreadMetaStore.check_access``: it returns True for
|
||||
# missing rows (untracked legacy thread) and for rows whose
|
||||
# ``user_id`` is NULL (shared / pre-auth data), so this is
|
||||
# strict-deny rather than strict-allow — only an *existing*
|
||||
# row with a *different* user_id triggers 404.
|
||||
if owner_check:
|
||||
thread_id = kwargs.get("thread_id")
|
||||
if thread_id is None:
|
||||
raise ValueError("require_permission with owner_check=True requires 'thread_id' parameter")
|
||||
|
||||
from app.gateway.deps import get_thread_store
|
||||
|
||||
thread_store = get_thread_store(request)
|
||||
allowed = await thread_store.check_access(
|
||||
thread_id,
|
||||
str(auth.user.id),
|
||||
require_existing=require_existing,
|
||||
)
|
||||
if not allowed:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"Thread {thread_id} not found",
|
||||
)
|
||||
|
||||
return await func(*args, **kwargs)
|
||||
|
||||
return wrapper
|
||||
|
||||
return decorator
|
||||
@@ -1,26 +0,0 @@
|
||||
import os
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class GatewayConfig(BaseModel):
|
||||
"""Configuration for the API Gateway."""
|
||||
|
||||
host: str = Field(default="0.0.0.0", description="Host to bind the gateway server")
|
||||
port: int = Field(default=8001, description="Port to bind the gateway server")
|
||||
enable_docs: bool = Field(default=True, description="Enable Swagger/ReDoc/OpenAPI endpoints")
|
||||
|
||||
|
||||
_gateway_config: GatewayConfig | None = None
|
||||
|
||||
|
||||
def get_gateway_config() -> GatewayConfig:
|
||||
"""Get gateway config, loading from environment if available."""
|
||||
global _gateway_config
|
||||
if _gateway_config is None:
|
||||
_gateway_config = GatewayConfig(
|
||||
host=os.getenv("GATEWAY_HOST", "0.0.0.0"),
|
||||
port=int(os.getenv("GATEWAY_PORT", "8001")),
|
||||
enable_docs=os.getenv("GATEWAY_ENABLE_DOCS", "true").lower() == "true",
|
||||
)
|
||||
return _gateway_config
|
||||
@@ -1,229 +0,0 @@
|
||||
"""CSRF protection middleware for FastAPI.
|
||||
|
||||
Per RFC-001:
|
||||
State-changing operations require CSRF protection.
|
||||
"""
|
||||
|
||||
import os
|
||||
import secrets
|
||||
from collections.abc import Awaitable, Callable
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from fastapi import Request, Response
|
||||
from starlette.middleware.base import BaseHTTPMiddleware
|
||||
from starlette.responses import JSONResponse
|
||||
from starlette.types import ASGIApp
|
||||
|
||||
CSRF_COOKIE_NAME = "csrf_token"
|
||||
CSRF_HEADER_NAME = "X-CSRF-Token"
|
||||
CSRF_TOKEN_LENGTH = 64 # bytes
|
||||
|
||||
|
||||
def is_secure_request(request: Request) -> bool:
|
||||
"""Detect whether the original client request was made over HTTPS."""
|
||||
return _request_scheme(request) == "https"
|
||||
|
||||
|
||||
def generate_csrf_token() -> str:
|
||||
"""Generate a secure random CSRF token."""
|
||||
return secrets.token_urlsafe(CSRF_TOKEN_LENGTH)
|
||||
|
||||
|
||||
def should_check_csrf(request: Request) -> bool:
|
||||
"""Determine if a request needs CSRF validation.
|
||||
|
||||
CSRF is checked for state-changing methods (POST, PUT, DELETE, PATCH).
|
||||
GET, HEAD, OPTIONS, and TRACE are exempt per RFC 7231.
|
||||
"""
|
||||
if request.method not in ("POST", "PUT", "DELETE", "PATCH"):
|
||||
return False
|
||||
|
||||
path = request.url.path.rstrip("/")
|
||||
# Exempt /api/v1/auth/me endpoint
|
||||
if path == "/api/v1/auth/me":
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
_AUTH_EXEMPT_PATHS: frozenset[str] = frozenset(
|
||||
{
|
||||
"/api/v1/auth/login/local",
|
||||
"/api/v1/auth/logout",
|
||||
"/api/v1/auth/register",
|
||||
"/api/v1/auth/initialize",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def is_auth_endpoint(request: Request) -> bool:
|
||||
"""Check if the request is to an auth endpoint.
|
||||
|
||||
Auth endpoints don't need CSRF validation on first call (no token).
|
||||
"""
|
||||
return request.url.path.rstrip("/") in _AUTH_EXEMPT_PATHS
|
||||
|
||||
|
||||
def _host_with_optional_port(hostname: str, port: int | None, scheme: str) -> str:
|
||||
"""Return normalized host[:port], omitting default ports."""
|
||||
host = hostname.lower()
|
||||
if ":" in host and not host.startswith("["):
|
||||
host = f"[{host}]"
|
||||
|
||||
if port is None or (scheme == "http" and port == 80) or (scheme == "https" and port == 443):
|
||||
return host
|
||||
return f"{host}:{port}"
|
||||
|
||||
|
||||
def _normalize_origin(origin: str) -> str | None:
|
||||
"""Return a normalized scheme://host[:port] origin, or None for invalid input."""
|
||||
try:
|
||||
parsed = urlsplit(origin.strip())
|
||||
port = parsed.port
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
scheme = parsed.scheme.lower()
|
||||
if scheme not in {"http", "https"} or not parsed.hostname:
|
||||
return None
|
||||
|
||||
# Browser Origin is only scheme/host/port. Reject URL-shaped or credentialed values.
|
||||
if parsed.username or parsed.password or parsed.path or parsed.query or parsed.fragment:
|
||||
return None
|
||||
|
||||
return f"{scheme}://{_host_with_optional_port(parsed.hostname, port, scheme)}"
|
||||
|
||||
|
||||
def _configured_cors_origins() -> set[str]:
|
||||
"""Return explicit configured browser origins that may call auth routes."""
|
||||
origins = set()
|
||||
for raw_origin in os.environ.get("GATEWAY_CORS_ORIGINS", "").split(","):
|
||||
origin = raw_origin.strip()
|
||||
if not origin or origin == "*":
|
||||
continue
|
||||
normalized = _normalize_origin(origin)
|
||||
if normalized:
|
||||
origins.add(normalized)
|
||||
return origins
|
||||
|
||||
|
||||
def get_configured_cors_origins() -> set[str]:
|
||||
"""Return normalized explicit browser origins from GATEWAY_CORS_ORIGINS."""
|
||||
return _configured_cors_origins()
|
||||
|
||||
|
||||
def _first_header_value(value: str | None) -> str | None:
|
||||
"""Return the first value from a comma-separated proxy header."""
|
||||
if not value:
|
||||
return None
|
||||
first = value.split(",", 1)[0].strip()
|
||||
return first or None
|
||||
|
||||
|
||||
def _forwarded_param(request: Request, name: str) -> str | None:
|
||||
"""Extract a parameter from the first RFC 7239 Forwarded header entry."""
|
||||
forwarded = _first_header_value(request.headers.get("forwarded"))
|
||||
if not forwarded:
|
||||
return None
|
||||
|
||||
for part in forwarded.split(";"):
|
||||
key, sep, value = part.strip().partition("=")
|
||||
if sep and key.lower() == name:
|
||||
return value.strip().strip('"') or None
|
||||
return None
|
||||
|
||||
|
||||
def _request_scheme(request: Request) -> str:
|
||||
"""Resolve the original request scheme from trusted proxy headers."""
|
||||
scheme = _forwarded_param(request, "proto") or _first_header_value(request.headers.get("x-forwarded-proto")) or request.url.scheme
|
||||
return scheme.lower()
|
||||
|
||||
|
||||
def _request_origin(request: Request) -> str | None:
|
||||
"""Build the origin for the URL the browser is targeting."""
|
||||
scheme = _request_scheme(request)
|
||||
host = _forwarded_param(request, "host") or _first_header_value(request.headers.get("x-forwarded-host")) or request.headers.get("host") or request.url.netloc
|
||||
|
||||
forwarded_port = _first_header_value(request.headers.get("x-forwarded-port"))
|
||||
if forwarded_port and ":" not in host.rsplit("]", 1)[-1]:
|
||||
host = f"{host}:{forwarded_port}"
|
||||
|
||||
return _normalize_origin(f"{scheme}://{host}")
|
||||
|
||||
|
||||
def is_allowed_auth_origin(request: Request) -> bool:
|
||||
"""Allow auth POSTs only from the same origin or explicit configured origins.
|
||||
|
||||
Login/register/initialize are exempt from the double-submit token because
|
||||
first-time browser clients do not have a CSRF token yet. They still create
|
||||
a session cookie, so browser requests with a hostile Origin header must be
|
||||
rejected to prevent login CSRF / session fixation. Requests without Origin
|
||||
are allowed for non-browser clients such as curl and mobile integrations.
|
||||
"""
|
||||
origin = request.headers.get("origin")
|
||||
if not origin:
|
||||
return True
|
||||
|
||||
normalized_origin = _normalize_origin(origin)
|
||||
if normalized_origin is None:
|
||||
return False
|
||||
|
||||
request_origin = _request_origin(request)
|
||||
return normalized_origin in _configured_cors_origins() or (request_origin is not None and normalized_origin == request_origin)
|
||||
|
||||
|
||||
class CSRFMiddleware(BaseHTTPMiddleware):
|
||||
"""Middleware that implements CSRF protection using Double Submit Cookie pattern."""
|
||||
|
||||
def __init__(self, app: ASGIApp) -> None:
|
||||
super().__init__(app)
|
||||
|
||||
async def dispatch(self, request: Request, call_next: Callable[[Request], Awaitable[Response]]) -> Response:
|
||||
_is_auth = is_auth_endpoint(request)
|
||||
|
||||
if should_check_csrf(request) and _is_auth and not is_allowed_auth_origin(request):
|
||||
return JSONResponse(
|
||||
status_code=403,
|
||||
content={"detail": "Cross-site auth request denied."},
|
||||
)
|
||||
|
||||
if should_check_csrf(request) and not _is_auth:
|
||||
cookie_token = request.cookies.get(CSRF_COOKIE_NAME)
|
||||
header_token = request.headers.get(CSRF_HEADER_NAME)
|
||||
|
||||
if not cookie_token or not header_token:
|
||||
return JSONResponse(
|
||||
status_code=403,
|
||||
content={"detail": "CSRF token missing. Include X-CSRF-Token header."},
|
||||
)
|
||||
|
||||
if not secrets.compare_digest(cookie_token, header_token):
|
||||
return JSONResponse(
|
||||
status_code=403,
|
||||
content={"detail": "CSRF token mismatch."},
|
||||
)
|
||||
|
||||
response = await call_next(request)
|
||||
|
||||
# For auth endpoints that set up session, also set CSRF cookie
|
||||
if _is_auth and request.method == "POST":
|
||||
# Generate a new CSRF token for the session
|
||||
csrf_token = generate_csrf_token()
|
||||
is_https = is_secure_request(request)
|
||||
response.set_cookie(
|
||||
key=CSRF_COOKIE_NAME,
|
||||
value=csrf_token,
|
||||
httponly=False, # Must be JS-readable for Double Submit Cookie pattern
|
||||
secure=is_https,
|
||||
samesite="strict",
|
||||
)
|
||||
|
||||
return response
|
||||
|
||||
|
||||
def get_csrf_token(request: Request) -> str | None:
|
||||
"""Get the CSRF token from the current request's cookies.
|
||||
|
||||
This is useful for server-side rendering where you need to embed
|
||||
token in forms or headers.
|
||||
"""
|
||||
return request.cookies.get(CSRF_COOKIE_NAME)
|
||||
@@ -1,245 +0,0 @@
|
||||
"""Centralized accessors for singleton objects stored on ``app.state``.
|
||||
|
||||
**Getters** (used by routers): raise 503 when a required dependency is
|
||||
missing, except ``get_store`` which returns ``None``.
|
||||
|
||||
Initialization is handled directly in ``app.py`` via :class:`AsyncExitStack`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import AsyncGenerator, Callable
|
||||
from contextlib import AsyncExitStack, asynccontextmanager
|
||||
from typing import TYPE_CHECKING, TypeVar, cast
|
||||
|
||||
from fastapi import FastAPI, HTTPException, Request
|
||||
from langgraph.types import Checkpointer
|
||||
|
||||
from deerflow.config.app_config import AppConfig
|
||||
from deerflow.persistence.feedback import FeedbackRepository
|
||||
from deerflow.runtime import RunContext, RunManager, StreamBridge
|
||||
from deerflow.runtime.events.store.base import RunEventStore
|
||||
from deerflow.runtime.runs.store.base import RunStore
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from app.gateway.auth.local_provider import LocalAuthProvider
|
||||
from app.gateway.auth.repositories.sqlite import SQLiteUserRepository
|
||||
from deerflow.persistence.thread_meta.base import ThreadMetaStore
|
||||
|
||||
|
||||
T = TypeVar("T")
|
||||
|
||||
|
||||
def get_config(request: Request) -> AppConfig:
|
||||
"""Return the app-scoped ``AppConfig`` stored on ``app.state``."""
|
||||
config = getattr(request.app.state, "config", None)
|
||||
if config is None:
|
||||
raise HTTPException(status_code=503, detail="Configuration not available")
|
||||
return config
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
|
||||
"""Bootstrap and tear down all LangGraph runtime singletons.
|
||||
|
||||
Usage in ``app.py``::
|
||||
|
||||
async with langgraph_runtime(app):
|
||||
yield
|
||||
"""
|
||||
from deerflow.persistence.engine import close_engine, get_session_factory, init_engine_from_config
|
||||
from deerflow.runtime import make_store, make_stream_bridge
|
||||
from deerflow.runtime.checkpointer.async_provider import make_checkpointer
|
||||
from deerflow.runtime.events.store import make_run_event_store
|
||||
|
||||
async with AsyncExitStack() as stack:
|
||||
config = getattr(app.state, "config", None)
|
||||
if config is None:
|
||||
raise RuntimeError("langgraph_runtime() requires app.state.config to be initialized")
|
||||
|
||||
app.state.stream_bridge = await stack.enter_async_context(make_stream_bridge(config))
|
||||
|
||||
# Initialize persistence engine BEFORE checkpointer so that
|
||||
# auto-create-database logic runs first (postgres backend).
|
||||
await init_engine_from_config(config.database)
|
||||
|
||||
app.state.checkpointer = await stack.enter_async_context(make_checkpointer(config))
|
||||
app.state.store = await stack.enter_async_context(make_store(config))
|
||||
|
||||
# Initialize repositories — one get_session_factory() call for all.
|
||||
sf = get_session_factory()
|
||||
if sf is not None:
|
||||
from deerflow.persistence.feedback import FeedbackRepository
|
||||
from deerflow.persistence.run import RunRepository
|
||||
|
||||
app.state.run_store = RunRepository(sf)
|
||||
app.state.feedback_repo = FeedbackRepository(sf)
|
||||
else:
|
||||
from deerflow.runtime.runs.store.memory import MemoryRunStore
|
||||
|
||||
app.state.run_store = MemoryRunStore()
|
||||
app.state.feedback_repo = None
|
||||
|
||||
from deerflow.persistence.thread_meta import make_thread_store
|
||||
|
||||
app.state.thread_store = make_thread_store(sf, app.state.store)
|
||||
|
||||
# Run event store (has its own factory with config-driven backend selection)
|
||||
run_events_config = getattr(config, "run_events", None)
|
||||
app.state.run_event_store = make_run_event_store(run_events_config)
|
||||
|
||||
# RunManager with store backing for persistence
|
||||
app.state.run_manager = RunManager(store=app.state.run_store)
|
||||
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
await close_engine()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Getters – called by routers per-request
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _require(attr: str, label: str) -> Callable[[Request], T]:
|
||||
"""Create a FastAPI dependency that returns ``app.state.<attr>`` or 503."""
|
||||
|
||||
def dep(request: Request) -> T:
|
||||
val = getattr(request.app.state, attr, None)
|
||||
if val is None:
|
||||
raise HTTPException(status_code=503, detail=f"{label} not available")
|
||||
return cast(T, val)
|
||||
|
||||
dep.__name__ = dep.__qualname__ = f"get_{attr}"
|
||||
return dep
|
||||
|
||||
|
||||
get_stream_bridge: Callable[[Request], StreamBridge] = _require("stream_bridge", "Stream bridge")
|
||||
get_run_manager: Callable[[Request], RunManager] = _require("run_manager", "Run manager")
|
||||
get_checkpointer: Callable[[Request], Checkpointer] = _require("checkpointer", "Checkpointer")
|
||||
get_run_event_store: Callable[[Request], RunEventStore] = _require("run_event_store", "Run event store")
|
||||
get_feedback_repo: Callable[[Request], FeedbackRepository] = _require("feedback_repo", "Feedback")
|
||||
get_run_store: Callable[[Request], RunStore] = _require("run_store", "Run store")
|
||||
|
||||
|
||||
def get_store(request: Request):
|
||||
"""Return the global store (may be ``None`` if not configured)."""
|
||||
return getattr(request.app.state, "store", None)
|
||||
|
||||
|
||||
def get_thread_store(request: Request) -> ThreadMetaStore:
|
||||
"""Return the thread metadata store (SQL or memory-backed)."""
|
||||
val = getattr(request.app.state, "thread_store", None)
|
||||
if val is None:
|
||||
raise HTTPException(status_code=503, detail="Thread metadata store not available")
|
||||
return val
|
||||
|
||||
|
||||
def get_run_context(request: Request) -> RunContext:
|
||||
"""Build a :class:`RunContext` from ``app.state`` singletons.
|
||||
|
||||
Returns a *base* context with infrastructure dependencies.
|
||||
"""
|
||||
config = get_config(request)
|
||||
return RunContext(
|
||||
checkpointer=get_checkpointer(request),
|
||||
store=get_store(request),
|
||||
event_store=get_run_event_store(request),
|
||||
run_events_config=getattr(config, "run_events", None),
|
||||
thread_store=get_thread_store(request),
|
||||
app_config=config,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Auth helpers (used by authz.py and auth middleware)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Cached singletons to avoid repeated instantiation per request
|
||||
_cached_local_provider: LocalAuthProvider | None = None
|
||||
_cached_repo: SQLiteUserRepository | None = None
|
||||
|
||||
|
||||
def get_local_provider() -> LocalAuthProvider:
|
||||
"""Get or create the cached LocalAuthProvider singleton.
|
||||
|
||||
Must be called after ``init_engine_from_config()`` — the shared
|
||||
session factory is required to construct the user repository.
|
||||
"""
|
||||
global _cached_local_provider, _cached_repo
|
||||
if _cached_repo is None:
|
||||
from app.gateway.auth.repositories.sqlite import SQLiteUserRepository
|
||||
from deerflow.persistence.engine import get_session_factory
|
||||
|
||||
sf = get_session_factory()
|
||||
if sf is None:
|
||||
raise RuntimeError("get_local_provider() called before init_engine_from_config(); cannot access users table")
|
||||
_cached_repo = SQLiteUserRepository(sf)
|
||||
if _cached_local_provider is None:
|
||||
from app.gateway.auth.local_provider import LocalAuthProvider
|
||||
|
||||
_cached_local_provider = LocalAuthProvider(repository=_cached_repo)
|
||||
return _cached_local_provider
|
||||
|
||||
|
||||
async def get_current_user_from_request(request: Request):
|
||||
"""Get the current authenticated user from the request cookie.
|
||||
|
||||
Raises HTTPException 401 if not authenticated.
|
||||
"""
|
||||
from app.gateway.auth import decode_token
|
||||
from app.gateway.auth.errors import AuthErrorCode, AuthErrorResponse, TokenError, token_error_to_code
|
||||
|
||||
access_token = request.cookies.get("access_token")
|
||||
if not access_token:
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail=AuthErrorResponse(code=AuthErrorCode.NOT_AUTHENTICATED, message="Not authenticated").model_dump(),
|
||||
)
|
||||
|
||||
payload = decode_token(access_token)
|
||||
if isinstance(payload, TokenError):
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail=AuthErrorResponse(code=token_error_to_code(payload), message=f"Token error: {payload.value}").model_dump(),
|
||||
)
|
||||
|
||||
provider = get_local_provider()
|
||||
user = await provider.get_user(payload.sub)
|
||||
if user is None:
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail=AuthErrorResponse(code=AuthErrorCode.USER_NOT_FOUND, message="User not found").model_dump(),
|
||||
)
|
||||
|
||||
# Token version mismatch → password was changed, token is stale
|
||||
if user.token_version != payload.ver:
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail=AuthErrorResponse(code=AuthErrorCode.TOKEN_INVALID, message="Token revoked (password changed)").model_dump(),
|
||||
)
|
||||
|
||||
return user
|
||||
|
||||
|
||||
async def get_optional_user_from_request(request: Request):
|
||||
"""Get optional authenticated user from request.
|
||||
|
||||
Returns None if not authenticated.
|
||||
"""
|
||||
try:
|
||||
return await get_current_user_from_request(request)
|
||||
except HTTPException:
|
||||
return None
|
||||
|
||||
|
||||
async def get_current_user(request: Request) -> str | None:
|
||||
"""Extract user_id from request cookie, or None if not authenticated.
|
||||
|
||||
Thin adapter that returns the string id for callers that only need
|
||||
identification (e.g., ``feedback.py``). Full-user callers should use
|
||||
``get_current_user_from_request`` or ``get_optional_user_from_request``.
|
||||
"""
|
||||
user = await get_optional_user_from_request(request)
|
||||
return str(user.id) if user else None
|
||||
@@ -1,26 +0,0 @@
|
||||
"""Process-local authentication for Gateway internal callers."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import secrets
|
||||
from types import SimpleNamespace
|
||||
|
||||
from deerflow.runtime.user_context import DEFAULT_USER_ID
|
||||
|
||||
INTERNAL_AUTH_HEADER_NAME = "X-DeerFlow-Internal-Token"
|
||||
_INTERNAL_AUTH_TOKEN = secrets.token_urlsafe(32)
|
||||
|
||||
|
||||
def create_internal_auth_headers() -> dict[str, str]:
|
||||
"""Return headers that authenticate same-process Gateway internal calls."""
|
||||
return {INTERNAL_AUTH_HEADER_NAME: _INTERNAL_AUTH_TOKEN}
|
||||
|
||||
|
||||
def is_valid_internal_auth_token(token: str | None) -> bool:
|
||||
"""Return True when *token* matches the process-local internal token."""
|
||||
return bool(token) and secrets.compare_digest(token, _INTERNAL_AUTH_TOKEN)
|
||||
|
||||
|
||||
def get_internal_user():
|
||||
"""Return the synthetic user used for trusted internal channel calls."""
|
||||
return SimpleNamespace(id=DEFAULT_USER_ID, system_role="internal")
|
||||
@@ -1,110 +0,0 @@
|
||||
"""LangGraph compatibility auth handler — shares JWT logic with Gateway.
|
||||
|
||||
The default DeerFlow runtime is embedded in the FastAPI Gateway; scripts and
|
||||
Docker deployments do not load this module. It is retained for LangGraph
|
||||
tooling, Studio, or direct LangGraph Server compatibility through
|
||||
``langgraph.json``'s ``auth.path``.
|
||||
|
||||
When that compatibility path is used, this module reuses the same JWT and CSRF
|
||||
rules as Gateway so both modes validate sessions consistently.
|
||||
|
||||
Two layers:
|
||||
1. @auth.authenticate — validates JWT cookie, extracts user_id,
|
||||
and enforces CSRF on state-changing methods (POST/PUT/DELETE/PATCH)
|
||||
2. @auth.on — returns metadata filter so each user only sees own threads
|
||||
"""
|
||||
|
||||
import secrets
|
||||
|
||||
from langgraph_sdk import Auth
|
||||
|
||||
from app.gateway.auth.errors import TokenError
|
||||
from app.gateway.auth.jwt import decode_token
|
||||
from app.gateway.deps import get_local_provider
|
||||
|
||||
auth = Auth()
|
||||
|
||||
# Methods that require CSRF validation (state-changing per RFC 7231).
|
||||
_CSRF_METHODS = frozenset({"POST", "PUT", "DELETE", "PATCH"})
|
||||
|
||||
|
||||
def _check_csrf(request) -> None:
|
||||
"""Enforce Double Submit Cookie CSRF check for state-changing requests.
|
||||
|
||||
Mirrors Gateway's CSRFMiddleware logic so that LangGraph routes
|
||||
proxied directly by nginx have the same CSRF protection.
|
||||
"""
|
||||
method = getattr(request, "method", "") or ""
|
||||
if method.upper() not in _CSRF_METHODS:
|
||||
return
|
||||
|
||||
cookie_token = request.cookies.get("csrf_token")
|
||||
header_token = request.headers.get("x-csrf-token")
|
||||
|
||||
if not cookie_token or not header_token:
|
||||
raise Auth.exceptions.HTTPException(
|
||||
status_code=403,
|
||||
detail="CSRF token missing. Include X-CSRF-Token header.",
|
||||
)
|
||||
|
||||
if not secrets.compare_digest(cookie_token, header_token):
|
||||
raise Auth.exceptions.HTTPException(
|
||||
status_code=403,
|
||||
detail="CSRF token mismatch.",
|
||||
)
|
||||
|
||||
|
||||
@auth.authenticate
|
||||
async def authenticate(request):
|
||||
"""Validate the session cookie, decode JWT, and check token_version.
|
||||
|
||||
Same validation chain as Gateway's get_current_user_from_request:
|
||||
cookie → decode JWT → DB lookup → token_version match
|
||||
Also enforces CSRF on state-changing methods.
|
||||
"""
|
||||
# CSRF check before authentication so forged cross-site requests
|
||||
# are rejected early, even if the cookie carries a valid JWT.
|
||||
_check_csrf(request)
|
||||
|
||||
token = request.cookies.get("access_token")
|
||||
if not token:
|
||||
raise Auth.exceptions.HTTPException(
|
||||
status_code=401,
|
||||
detail="Not authenticated",
|
||||
)
|
||||
|
||||
payload = decode_token(token)
|
||||
if isinstance(payload, TokenError):
|
||||
raise Auth.exceptions.HTTPException(
|
||||
status_code=401,
|
||||
detail="Invalid token",
|
||||
)
|
||||
|
||||
user = await get_local_provider().get_user(payload.sub)
|
||||
if user is None:
|
||||
raise Auth.exceptions.HTTPException(
|
||||
status_code=401,
|
||||
detail="User not found",
|
||||
)
|
||||
if user.token_version != payload.ver:
|
||||
raise Auth.exceptions.HTTPException(
|
||||
status_code=401,
|
||||
detail="Token revoked (password changed)",
|
||||
)
|
||||
|
||||
return payload.sub
|
||||
|
||||
|
||||
@auth.on
|
||||
async def add_owner_filter(ctx: Auth.types.AuthContext, value: dict):
|
||||
"""Inject user_id metadata on writes; filter by user_id on reads.
|
||||
|
||||
Gateway stores thread ownership as ``metadata.user_id``.
|
||||
This handler ensures LangGraph Server enforces the same isolation.
|
||||
"""
|
||||
# On create/update: stamp user_id into metadata
|
||||
metadata = value.setdefault("metadata", {})
|
||||
metadata["user_id"] = ctx.user.identity
|
||||
|
||||
# Return filter dict — LangGraph applies it to search/read/delete
|
||||
return {"user_id": ctx.user.identity}
|
||||
@@ -1,29 +0,0 @@
|
||||
"""Shared path resolution for thread virtual paths (e.g. mnt/user-data/outputs/...)."""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi import HTTPException
|
||||
|
||||
from deerflow.config.paths import get_paths
|
||||
from deerflow.runtime.user_context import get_effective_user_id
|
||||
|
||||
|
||||
def resolve_thread_virtual_path(thread_id: str, virtual_path: str) -> Path:
|
||||
"""Resolve a virtual path to the actual filesystem path under thread user-data.
|
||||
|
||||
Args:
|
||||
thread_id: The thread ID.
|
||||
virtual_path: The virtual path as seen inside the sandbox
|
||||
(e.g., /mnt/user-data/outputs/file.txt).
|
||||
|
||||
Returns:
|
||||
The resolved filesystem path.
|
||||
|
||||
Raises:
|
||||
HTTPException: If the path is invalid or outside allowed directories.
|
||||
"""
|
||||
try:
|
||||
return get_paths().resolve_virtual_path(thread_id, virtual_path, user_id=get_effective_user_id())
|
||||
except ValueError as e:
|
||||
status = 403 if "traversal" in str(e) else 400
|
||||
raise HTTPException(status_code=status, detail=str(e))
|
||||
@@ -1,3 +0,0 @@
|
||||
from . import artifacts, assistants_compat, mcp, models, skills, suggestions, thread_runs, threads, uploads
|
||||
|
||||
__all__ = ["artifacts", "assistants_compat", "mcp", "models", "skills", "suggestions", "threads", "thread_runs", "uploads"]
|
||||
@@ -1,446 +0,0 @@
|
||||
"""CRUD API for custom agents."""
|
||||
|
||||
import logging
|
||||
import re
|
||||
import shutil
|
||||
|
||||
import yaml
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from deerflow.config.agents_api_config import get_agents_api_config
|
||||
from deerflow.config.agents_config import AgentConfig, list_custom_agents, load_agent_config, load_agent_soul
|
||||
from deerflow.config.paths import get_paths
|
||||
from deerflow.runtime.user_context import get_effective_user_id
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/api", tags=["agents"])
|
||||
|
||||
AGENT_NAME_PATTERN = re.compile(r"^[A-Za-z0-9-]+$")
|
||||
|
||||
|
||||
class AgentResponse(BaseModel):
|
||||
"""Response model for a custom agent."""
|
||||
|
||||
name: str = Field(..., description="Agent name (hyphen-case)")
|
||||
description: str = Field(default="", description="Agent description")
|
||||
model: str | None = Field(default=None, description="Optional model override")
|
||||
tool_groups: list[str] | None = Field(default=None, description="Optional tool group whitelist")
|
||||
skills: list[str] | None = Field(default=None, description="Optional skill whitelist (None=all, []=none)")
|
||||
soul: str | None = Field(default=None, description="SOUL.md content")
|
||||
|
||||
|
||||
class AgentsListResponse(BaseModel):
|
||||
"""Response model for listing all custom agents."""
|
||||
|
||||
agents: list[AgentResponse]
|
||||
|
||||
|
||||
class AgentCreateRequest(BaseModel):
|
||||
"""Request body for creating a custom agent."""
|
||||
|
||||
name: str = Field(..., description="Agent name (must match ^[A-Za-z0-9-]+$, stored as lowercase)")
|
||||
description: str = Field(default="", description="Agent description")
|
||||
model: str | None = Field(default=None, description="Optional model override")
|
||||
tool_groups: list[str] | None = Field(default=None, description="Optional tool group whitelist")
|
||||
skills: list[str] | None = Field(default=None, description="Optional skill whitelist (None=all enabled, []=none)")
|
||||
soul: str = Field(default="", description="SOUL.md content — agent personality and behavioral guardrails")
|
||||
|
||||
|
||||
class AgentUpdateRequest(BaseModel):
|
||||
"""Request body for updating a custom agent."""
|
||||
|
||||
description: str | None = Field(default=None, description="Updated description")
|
||||
model: str | None = Field(default=None, description="Updated model override")
|
||||
tool_groups: list[str] | None = Field(default=None, description="Updated tool group whitelist")
|
||||
skills: list[str] | None = Field(default=None, description="Updated skill whitelist (None=all, []=none)")
|
||||
soul: str | None = Field(default=None, description="Updated SOUL.md content")
|
||||
|
||||
|
||||
def _validate_agent_name(name: str) -> None:
|
||||
"""Validate agent name against allowed pattern.
|
||||
|
||||
Args:
|
||||
name: The agent name to validate.
|
||||
|
||||
Raises:
|
||||
HTTPException: 422 if the name is invalid.
|
||||
"""
|
||||
if not AGENT_NAME_PATTERN.match(name):
|
||||
raise HTTPException(
|
||||
status_code=422,
|
||||
detail=f"Invalid agent name '{name}'. Must match ^[A-Za-z0-9-]+$ (letters, digits, and hyphens only).",
|
||||
)
|
||||
|
||||
|
||||
def _normalize_agent_name(name: str) -> str:
|
||||
"""Normalize agent name to lowercase for filesystem storage."""
|
||||
return name.lower()
|
||||
|
||||
|
||||
def _require_agents_api_enabled() -> None:
|
||||
"""Reject access unless the custom-agent management API is explicitly enabled."""
|
||||
if not get_agents_api_config().enabled:
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail=("Custom-agent management API is disabled. Set agents_api.enabled=true to expose agent and user-profile routes over HTTP."),
|
||||
)
|
||||
|
||||
|
||||
def _agent_config_to_response(agent_cfg: AgentConfig, include_soul: bool = False, *, user_id: str | None = None) -> AgentResponse:
|
||||
"""Convert AgentConfig to AgentResponse."""
|
||||
soul: str | None = None
|
||||
if include_soul:
|
||||
soul = load_agent_soul(agent_cfg.name, user_id=user_id) or ""
|
||||
|
||||
return AgentResponse(
|
||||
name=agent_cfg.name,
|
||||
description=agent_cfg.description,
|
||||
model=agent_cfg.model,
|
||||
tool_groups=agent_cfg.tool_groups,
|
||||
skills=agent_cfg.skills,
|
||||
soul=soul,
|
||||
)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/agents",
|
||||
response_model=AgentsListResponse,
|
||||
summary="List Custom Agents",
|
||||
description="List all custom agents available in the agents directory, including their soul content.",
|
||||
)
|
||||
async def list_agents() -> AgentsListResponse:
|
||||
"""List all custom agents.
|
||||
|
||||
Returns:
|
||||
List of all custom agents with their metadata and soul content.
|
||||
"""
|
||||
_require_agents_api_enabled()
|
||||
|
||||
user_id = get_effective_user_id()
|
||||
try:
|
||||
agents = list_custom_agents(user_id=user_id)
|
||||
return AgentsListResponse(agents=[_agent_config_to_response(a, include_soul=True, user_id=user_id) for a in agents])
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to list agents: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to list agents: {str(e)}")
|
||||
|
||||
|
||||
@router.get(
|
||||
"/agents/check",
|
||||
summary="Check Agent Name",
|
||||
description="Validate an agent name and check if it is available (case-insensitive).",
|
||||
)
|
||||
async def check_agent_name(name: str) -> dict:
|
||||
"""Check whether an agent name is valid and not yet taken.
|
||||
|
||||
Args:
|
||||
name: The agent name to check.
|
||||
|
||||
Returns:
|
||||
``{"available": true/false, "name": "<normalized>"}``
|
||||
|
||||
Raises:
|
||||
HTTPException: 422 if the name is invalid.
|
||||
"""
|
||||
_require_agents_api_enabled()
|
||||
_validate_agent_name(name)
|
||||
normalized = _normalize_agent_name(name)
|
||||
user_id = get_effective_user_id()
|
||||
paths = get_paths()
|
||||
# Treat the name as taken if either the per-user path or the legacy shared
|
||||
# path holds an agent — picking a name that collides with an unmigrated
|
||||
# legacy agent would shadow the legacy entry once migration runs.
|
||||
available = not paths.user_agent_dir(user_id, normalized).exists() and not paths.agent_dir(normalized).exists()
|
||||
return {"available": available, "name": normalized}
|
||||
|
||||
|
||||
@router.get(
|
||||
"/agents/{name}",
|
||||
response_model=AgentResponse,
|
||||
summary="Get Custom Agent",
|
||||
description="Retrieve details and SOUL.md content for a specific custom agent.",
|
||||
)
|
||||
async def get_agent(name: str) -> AgentResponse:
|
||||
"""Get a specific custom agent by name.
|
||||
|
||||
Args:
|
||||
name: The agent name.
|
||||
|
||||
Returns:
|
||||
Agent details including SOUL.md content.
|
||||
|
||||
Raises:
|
||||
HTTPException: 404 if agent not found.
|
||||
"""
|
||||
_require_agents_api_enabled()
|
||||
_validate_agent_name(name)
|
||||
name = _normalize_agent_name(name)
|
||||
user_id = get_effective_user_id()
|
||||
|
||||
try:
|
||||
agent_cfg = load_agent_config(name, user_id=user_id)
|
||||
return _agent_config_to_response(agent_cfg, include_soul=True, user_id=user_id)
|
||||
except FileNotFoundError:
|
||||
raise HTTPException(status_code=404, detail=f"Agent '{name}' not found")
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to get agent '{name}': {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to get agent: {str(e)}")
|
||||
|
||||
|
||||
@router.post(
|
||||
"/agents",
|
||||
response_model=AgentResponse,
|
||||
status_code=201,
|
||||
summary="Create Custom Agent",
|
||||
description="Create a new custom agent with its config and SOUL.md.",
|
||||
)
|
||||
async def create_agent_endpoint(request: AgentCreateRequest) -> AgentResponse:
|
||||
"""Create a new custom agent.
|
||||
|
||||
Args:
|
||||
request: The agent creation request.
|
||||
|
||||
Returns:
|
||||
The created agent details.
|
||||
|
||||
Raises:
|
||||
HTTPException: 409 if agent already exists, 422 if name is invalid.
|
||||
"""
|
||||
_require_agents_api_enabled()
|
||||
_validate_agent_name(request.name)
|
||||
normalized_name = _normalize_agent_name(request.name)
|
||||
user_id = get_effective_user_id()
|
||||
paths = get_paths()
|
||||
|
||||
agent_dir = paths.user_agent_dir(user_id, normalized_name)
|
||||
legacy_dir = paths.agent_dir(normalized_name)
|
||||
|
||||
if agent_dir.exists() or legacy_dir.exists():
|
||||
raise HTTPException(status_code=409, detail=f"Agent '{normalized_name}' already exists")
|
||||
|
||||
try:
|
||||
agent_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Write config.yaml
|
||||
config_data: dict = {"name": normalized_name}
|
||||
if request.description:
|
||||
config_data["description"] = request.description
|
||||
if request.model is not None:
|
||||
config_data["model"] = request.model
|
||||
if request.tool_groups is not None:
|
||||
config_data["tool_groups"] = request.tool_groups
|
||||
if request.skills is not None:
|
||||
config_data["skills"] = request.skills
|
||||
|
||||
config_file = agent_dir / "config.yaml"
|
||||
with open(config_file, "w", encoding="utf-8") as f:
|
||||
yaml.dump(config_data, f, default_flow_style=False, allow_unicode=True)
|
||||
|
||||
# Write SOUL.md
|
||||
soul_file = agent_dir / "SOUL.md"
|
||||
soul_file.write_text(request.soul, encoding="utf-8")
|
||||
|
||||
logger.info(f"Created agent '{normalized_name}' at {agent_dir}")
|
||||
|
||||
agent_cfg = load_agent_config(normalized_name, user_id=user_id)
|
||||
return _agent_config_to_response(agent_cfg, include_soul=True, user_id=user_id)
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
# Clean up on failure
|
||||
if agent_dir.exists():
|
||||
shutil.rmtree(agent_dir)
|
||||
logger.error(f"Failed to create agent '{request.name}': {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to create agent: {str(e)}")
|
||||
|
||||
|
||||
@router.put(
|
||||
"/agents/{name}",
|
||||
response_model=AgentResponse,
|
||||
summary="Update Custom Agent",
|
||||
description="Update an existing custom agent's config and/or SOUL.md.",
|
||||
)
|
||||
async def update_agent(name: str, request: AgentUpdateRequest) -> AgentResponse:
|
||||
"""Update an existing custom agent.
|
||||
|
||||
Args:
|
||||
name: The agent name.
|
||||
request: The update request (all fields optional).
|
||||
|
||||
Returns:
|
||||
The updated agent details.
|
||||
|
||||
Raises:
|
||||
HTTPException: 404 if agent not found.
|
||||
"""
|
||||
_require_agents_api_enabled()
|
||||
_validate_agent_name(name)
|
||||
name = _normalize_agent_name(name)
|
||||
user_id = get_effective_user_id()
|
||||
|
||||
try:
|
||||
agent_cfg = load_agent_config(name, user_id=user_id)
|
||||
except FileNotFoundError:
|
||||
raise HTTPException(status_code=404, detail=f"Agent '{name}' not found")
|
||||
|
||||
paths = get_paths()
|
||||
agent_dir = paths.user_agent_dir(user_id, name)
|
||||
if not agent_dir.exists() and paths.agent_dir(name).exists():
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail=(f"Agent '{name}' only exists in the legacy shared layout and is not scoped to a user. Run scripts/migrate_user_isolation.py to move legacy agents into the per-user layout before updating."),
|
||||
)
|
||||
|
||||
try:
|
||||
# Update config if any config fields changed
|
||||
# Use model_fields_set to distinguish "field omitted" from "explicitly set to null".
|
||||
# This is critical for skills where None means "inherit all" (not "don't change").
|
||||
fields_set = request.model_fields_set
|
||||
config_changed = bool(fields_set & {"description", "model", "tool_groups", "skills"})
|
||||
|
||||
if config_changed:
|
||||
updated: dict = {
|
||||
"name": agent_cfg.name,
|
||||
"description": request.description if "description" in fields_set else agent_cfg.description,
|
||||
}
|
||||
new_model = request.model if "model" in fields_set else agent_cfg.model
|
||||
if new_model is not None:
|
||||
updated["model"] = new_model
|
||||
|
||||
new_tool_groups = request.tool_groups if "tool_groups" in fields_set else agent_cfg.tool_groups
|
||||
if new_tool_groups is not None:
|
||||
updated["tool_groups"] = new_tool_groups
|
||||
|
||||
# skills: None = inherit all, [] = no skills, ["a","b"] = whitelist
|
||||
if "skills" in fields_set:
|
||||
new_skills = request.skills
|
||||
else:
|
||||
new_skills = agent_cfg.skills
|
||||
if new_skills is not None:
|
||||
updated["skills"] = new_skills
|
||||
|
||||
config_file = agent_dir / "config.yaml"
|
||||
with open(config_file, "w", encoding="utf-8") as f:
|
||||
yaml.dump(updated, f, default_flow_style=False, allow_unicode=True)
|
||||
|
||||
# Update SOUL.md if provided
|
||||
if request.soul is not None:
|
||||
soul_path = agent_dir / "SOUL.md"
|
||||
soul_path.write_text(request.soul, encoding="utf-8")
|
||||
|
||||
logger.info(f"Updated agent '{name}'")
|
||||
|
||||
refreshed_cfg = load_agent_config(name, user_id=user_id)
|
||||
return _agent_config_to_response(refreshed_cfg, include_soul=True, user_id=user_id)
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to update agent '{name}': {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to update agent: {str(e)}")
|
||||
|
||||
|
||||
class UserProfileResponse(BaseModel):
|
||||
"""Response model for the global user profile (USER.md)."""
|
||||
|
||||
content: str | None = Field(default=None, description="USER.md content, or null if not yet created")
|
||||
|
||||
|
||||
class UserProfileUpdateRequest(BaseModel):
|
||||
"""Request body for setting the global user profile."""
|
||||
|
||||
content: str = Field(default="", description="USER.md content — describes the user's background and preferences")
|
||||
|
||||
|
||||
@router.get(
|
||||
"/user-profile",
|
||||
response_model=UserProfileResponse,
|
||||
summary="Get User Profile",
|
||||
description="Read the global USER.md file that is injected into all custom agents.",
|
||||
)
|
||||
async def get_user_profile() -> UserProfileResponse:
|
||||
"""Return the current USER.md content.
|
||||
|
||||
Returns:
|
||||
UserProfileResponse with content=None if USER.md does not exist yet.
|
||||
"""
|
||||
_require_agents_api_enabled()
|
||||
|
||||
try:
|
||||
user_md_path = get_paths().user_md_file
|
||||
if not user_md_path.exists():
|
||||
return UserProfileResponse(content=None)
|
||||
raw = user_md_path.read_text(encoding="utf-8").strip()
|
||||
return UserProfileResponse(content=raw or None)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to read user profile: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to read user profile: {str(e)}")
|
||||
|
||||
|
||||
@router.put(
|
||||
"/user-profile",
|
||||
response_model=UserProfileResponse,
|
||||
summary="Update User Profile",
|
||||
description="Write the global USER.md file that is injected into all custom agents.",
|
||||
)
|
||||
async def update_user_profile(request: UserProfileUpdateRequest) -> UserProfileResponse:
|
||||
"""Create or overwrite the global USER.md.
|
||||
|
||||
Args:
|
||||
request: The update request with the new USER.md content.
|
||||
|
||||
Returns:
|
||||
UserProfileResponse with the saved content.
|
||||
"""
|
||||
_require_agents_api_enabled()
|
||||
|
||||
try:
|
||||
paths = get_paths()
|
||||
paths.base_dir.mkdir(parents=True, exist_ok=True)
|
||||
paths.user_md_file.write_text(request.content, encoding="utf-8")
|
||||
logger.info(f"Updated USER.md at {paths.user_md_file}")
|
||||
return UserProfileResponse(content=request.content or None)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to update user profile: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to update user profile: {str(e)}")
|
||||
|
||||
|
||||
@router.delete(
|
||||
"/agents/{name}",
|
||||
status_code=204,
|
||||
summary="Delete Custom Agent",
|
||||
description="Delete a custom agent and all its files (config, SOUL.md, memory).",
|
||||
)
|
||||
async def delete_agent(name: str) -> None:
|
||||
"""Delete a custom agent.
|
||||
|
||||
Args:
|
||||
name: The agent name.
|
||||
|
||||
Raises:
|
||||
HTTPException: 404 if no per-user copy exists; 409 if only a legacy
|
||||
shared copy exists (suggesting the migration script).
|
||||
"""
|
||||
_require_agents_api_enabled()
|
||||
_validate_agent_name(name)
|
||||
name = _normalize_agent_name(name)
|
||||
user_id = get_effective_user_id()
|
||||
paths = get_paths()
|
||||
agent_dir = paths.user_agent_dir(user_id, name)
|
||||
|
||||
if not agent_dir.exists():
|
||||
if paths.agent_dir(name).exists():
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail=(f"Agent '{name}' only exists in the legacy shared layout and is not scoped to a user. Run scripts/migrate_user_isolation.py to move legacy agents into the per-user layout before deleting."),
|
||||
)
|
||||
raise HTTPException(status_code=404, detail=f"Agent '{name}' not found")
|
||||
|
||||
try:
|
||||
shutil.rmtree(agent_dir)
|
||||
logger.info(f"Deleted agent '{name}' from {agent_dir}")
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to delete agent '{name}': {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to delete agent: {str(e)}")
|
||||
@@ -1,183 +0,0 @@
|
||||
import logging
|
||||
import mimetypes
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
from urllib.parse import quote
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Request
|
||||
from fastapi.responses import FileResponse, PlainTextResponse, Response
|
||||
|
||||
from app.gateway.authz import require_permission
|
||||
from app.gateway.path_utils import resolve_thread_virtual_path
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api", tags=["artifacts"])
|
||||
|
||||
ACTIVE_CONTENT_MIME_TYPES = {
|
||||
"text/html",
|
||||
"application/xhtml+xml",
|
||||
"image/svg+xml",
|
||||
}
|
||||
|
||||
|
||||
def _build_content_disposition(disposition_type: str, filename: str) -> str:
|
||||
"""Build an RFC 5987 encoded Content-Disposition header value."""
|
||||
return f"{disposition_type}; filename*=UTF-8''{quote(filename)}"
|
||||
|
||||
|
||||
def _build_attachment_headers(filename: str, extra_headers: dict[str, str] | None = None) -> dict[str, str]:
|
||||
headers = {"Content-Disposition": _build_content_disposition("attachment", filename)}
|
||||
if extra_headers:
|
||||
headers.update(extra_headers)
|
||||
return headers
|
||||
|
||||
|
||||
def is_text_file_by_content(path: Path, sample_size: int = 8192) -> bool:
|
||||
"""Check if file is text by examining content for null bytes."""
|
||||
try:
|
||||
with open(path, "rb") as f:
|
||||
chunk = f.read(sample_size)
|
||||
# Text files shouldn't contain null bytes
|
||||
return b"\x00" not in chunk
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def _extract_file_from_skill_archive(zip_path: Path, internal_path: str) -> bytes | None:
|
||||
"""Extract a file from a .skill ZIP archive.
|
||||
|
||||
Args:
|
||||
zip_path: Path to the .skill file (ZIP archive).
|
||||
internal_path: Path to the file inside the archive (e.g., "SKILL.md").
|
||||
|
||||
Returns:
|
||||
The file content as bytes, or None if not found.
|
||||
"""
|
||||
if not zipfile.is_zipfile(zip_path):
|
||||
return None
|
||||
|
||||
try:
|
||||
with zipfile.ZipFile(zip_path, "r") as zip_ref:
|
||||
# List all files in the archive
|
||||
namelist = zip_ref.namelist()
|
||||
|
||||
# Try direct path first
|
||||
if internal_path in namelist:
|
||||
return zip_ref.read(internal_path)
|
||||
|
||||
# Try with any top-level directory prefix (e.g., "skill-name/SKILL.md")
|
||||
for name in namelist:
|
||||
if name.endswith("/" + internal_path) or name == internal_path:
|
||||
return zip_ref.read(name)
|
||||
|
||||
# Not found
|
||||
return None
|
||||
except (zipfile.BadZipFile, KeyError):
|
||||
return None
|
||||
|
||||
|
||||
@router.get(
|
||||
"/threads/{thread_id}/artifacts/{path:path}",
|
||||
summary="Get Artifact File",
|
||||
description="Retrieve an artifact file generated by the AI agent. Text and binary files can be viewed inline, while active web content is always downloaded.",
|
||||
)
|
||||
@require_permission("threads", "read", owner_check=True)
|
||||
async def get_artifact(thread_id: str, path: str, request: Request, download: bool = False) -> Response:
|
||||
"""Get an artifact file by its path.
|
||||
|
||||
The endpoint automatically detects file types and returns appropriate content types.
|
||||
Use the `download` query parameter to force file download for non-active content.
|
||||
|
||||
Args:
|
||||
thread_id: The thread ID.
|
||||
path: The artifact path with virtual prefix (e.g., mnt/user-data/outputs/file.txt).
|
||||
request: FastAPI request object (automatically injected).
|
||||
|
||||
Returns:
|
||||
The file content as a FileResponse with appropriate content type:
|
||||
- Active content (HTML/XHTML/SVG): Served as download attachment
|
||||
- Text files: Plain text with proper MIME type
|
||||
- Binary files: Inline display with download option
|
||||
|
||||
Raises:
|
||||
HTTPException:
|
||||
- 400 if path is invalid or not a file
|
||||
- 403 if access denied (path traversal detected)
|
||||
- 404 if file not found
|
||||
|
||||
Query Parameters:
|
||||
download (bool): If true, forces attachment download for file types that are
|
||||
otherwise returned inline or as plain text. Active HTML/XHTML/SVG content
|
||||
is always downloaded regardless of this flag.
|
||||
|
||||
Example:
|
||||
- Get text file inline: `/api/threads/abc123/artifacts/mnt/user-data/outputs/notes.txt`
|
||||
- Download file: `/api/threads/abc123/artifacts/mnt/user-data/outputs/data.csv?download=true`
|
||||
- Active web content such as `.html`, `.xhtml`, and `.svg` artifacts is always downloaded
|
||||
"""
|
||||
# Check if this is a request for a file inside a .skill archive (e.g., xxx.skill/SKILL.md)
|
||||
if ".skill/" in path:
|
||||
# Split the path at ".skill/" to get the ZIP file path and internal path
|
||||
skill_marker = ".skill/"
|
||||
marker_pos = path.find(skill_marker)
|
||||
skill_file_path = path[: marker_pos + len(".skill")] # e.g., "mnt/user-data/outputs/my-skill.skill"
|
||||
internal_path = path[marker_pos + len(skill_marker) :] # e.g., "SKILL.md"
|
||||
|
||||
actual_skill_path = resolve_thread_virtual_path(thread_id, skill_file_path)
|
||||
|
||||
if not actual_skill_path.exists():
|
||||
raise HTTPException(status_code=404, detail=f"Skill file not found: {skill_file_path}")
|
||||
|
||||
if not actual_skill_path.is_file():
|
||||
raise HTTPException(status_code=400, detail=f"Path is not a file: {skill_file_path}")
|
||||
|
||||
# Extract the file from the .skill archive
|
||||
content = _extract_file_from_skill_archive(actual_skill_path, internal_path)
|
||||
if content is None:
|
||||
raise HTTPException(status_code=404, detail=f"File '{internal_path}' not found in skill archive")
|
||||
|
||||
# Determine MIME type based on the internal file
|
||||
mime_type, _ = mimetypes.guess_type(internal_path)
|
||||
# Add cache headers to avoid repeated ZIP extraction (cache for 5 minutes)
|
||||
cache_headers = {"Cache-Control": "private, max-age=300"}
|
||||
download_name = Path(internal_path).name or actual_skill_path.stem
|
||||
if download or mime_type in ACTIVE_CONTENT_MIME_TYPES:
|
||||
return Response(content=content, media_type=mime_type or "application/octet-stream", headers=_build_attachment_headers(download_name, cache_headers))
|
||||
|
||||
if mime_type and mime_type.startswith("text/"):
|
||||
return PlainTextResponse(content=content.decode("utf-8"), media_type=mime_type, headers=cache_headers)
|
||||
|
||||
# Default to plain text for unknown types that look like text
|
||||
try:
|
||||
return PlainTextResponse(content=content.decode("utf-8"), media_type="text/plain", headers=cache_headers)
|
||||
except UnicodeDecodeError:
|
||||
return Response(content=content, media_type=mime_type or "application/octet-stream", headers=cache_headers)
|
||||
|
||||
actual_path = resolve_thread_virtual_path(thread_id, path)
|
||||
|
||||
logger.info(f"Resolving artifact path: thread_id={thread_id}, requested_path={path}, actual_path={actual_path}")
|
||||
|
||||
if not actual_path.exists():
|
||||
raise HTTPException(status_code=404, detail=f"Artifact not found: {path}")
|
||||
|
||||
if not actual_path.is_file():
|
||||
raise HTTPException(status_code=400, detail=f"Path is not a file: {path}")
|
||||
|
||||
mime_type, _ = mimetypes.guess_type(actual_path)
|
||||
|
||||
if download:
|
||||
return FileResponse(path=actual_path, filename=actual_path.name, media_type=mime_type, headers=_build_attachment_headers(actual_path.name))
|
||||
|
||||
# Always force download for active content types to prevent script execution
|
||||
# in the application origin when users open generated artifacts.
|
||||
if mime_type in ACTIVE_CONTENT_MIME_TYPES:
|
||||
return FileResponse(path=actual_path, filename=actual_path.name, media_type=mime_type, headers=_build_attachment_headers(actual_path.name))
|
||||
|
||||
if mime_type and mime_type.startswith("text/"):
|
||||
return PlainTextResponse(content=actual_path.read_text(encoding="utf-8"), media_type=mime_type)
|
||||
|
||||
if is_text_file_by_content(actual_path):
|
||||
return PlainTextResponse(content=actual_path.read_text(encoding="utf-8"), media_type=mime_type)
|
||||
|
||||
return Response(content=actual_path.read_bytes(), media_type=mime_type, headers={"Content-Disposition": _build_content_disposition("inline", actual_path.name)})
|
||||
@@ -1,149 +0,0 @@
|
||||
"""Assistants compatibility endpoints.
|
||||
|
||||
Provides LangGraph Platform-compatible assistants API backed by the
|
||||
``langgraph.json`` graph registry and ``config.yaml`` agent definitions.
|
||||
|
||||
This is a minimal stub that satisfies the ``useStream`` React hook's
|
||||
initialization requirements (``assistants.search()`` and ``assistants.get()``).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/api/assistants", tags=["assistants-compat"])
|
||||
|
||||
|
||||
class AssistantResponse(BaseModel):
|
||||
assistant_id: str
|
||||
graph_id: str
|
||||
name: str
|
||||
config: dict[str, Any] = Field(default_factory=dict)
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
description: str | None = None
|
||||
created_at: str = ""
|
||||
updated_at: str = ""
|
||||
version: int = 1
|
||||
|
||||
|
||||
class AssistantSearchRequest(BaseModel):
|
||||
graph_id: str | None = None
|
||||
name: str | None = None
|
||||
metadata: dict[str, Any] | None = None
|
||||
limit: int = 10
|
||||
offset: int = 0
|
||||
|
||||
|
||||
def _get_default_assistant() -> AssistantResponse:
|
||||
"""Return the default lead_agent assistant."""
|
||||
now = datetime.now(UTC).isoformat()
|
||||
return AssistantResponse(
|
||||
assistant_id="lead_agent",
|
||||
graph_id="lead_agent",
|
||||
name="lead_agent",
|
||||
config={},
|
||||
metadata={"created_by": "system"},
|
||||
description="DeerFlow lead agent",
|
||||
created_at=now,
|
||||
updated_at=now,
|
||||
version=1,
|
||||
)
|
||||
|
||||
|
||||
def _list_assistants() -> list[AssistantResponse]:
|
||||
"""List all available assistants from config."""
|
||||
assistants = [_get_default_assistant()]
|
||||
|
||||
# Also include custom agents from config.yaml agents directory
|
||||
try:
|
||||
from deerflow.config.agents_config import list_custom_agents
|
||||
|
||||
for agent_cfg in list_custom_agents():
|
||||
now = datetime.now(UTC).isoformat()
|
||||
assistants.append(
|
||||
AssistantResponse(
|
||||
assistant_id=agent_cfg.name,
|
||||
graph_id="lead_agent", # All agents use the same graph
|
||||
name=agent_cfg.name,
|
||||
config={},
|
||||
metadata={"created_by": "user"},
|
||||
description=agent_cfg.description or "",
|
||||
created_at=now,
|
||||
updated_at=now,
|
||||
version=1,
|
||||
)
|
||||
)
|
||||
except Exception:
|
||||
logger.debug("Could not load custom agents for assistants list")
|
||||
|
||||
return assistants
|
||||
|
||||
|
||||
@router.post("/search", response_model=list[AssistantResponse])
|
||||
async def search_assistants(body: AssistantSearchRequest | None = None) -> list[AssistantResponse]:
|
||||
"""Search assistants.
|
||||
|
||||
Returns all registered assistants (lead_agent + custom agents from config).
|
||||
"""
|
||||
assistants = _list_assistants()
|
||||
|
||||
if body and body.graph_id:
|
||||
assistants = [a for a in assistants if a.graph_id == body.graph_id]
|
||||
if body and body.name:
|
||||
assistants = [a for a in assistants if body.name.lower() in a.name.lower()]
|
||||
|
||||
offset = body.offset if body else 0
|
||||
limit = body.limit if body else 10
|
||||
return assistants[offset : offset + limit]
|
||||
|
||||
|
||||
@router.get("/{assistant_id}", response_model=AssistantResponse)
|
||||
async def get_assistant_compat(assistant_id: str) -> AssistantResponse:
|
||||
"""Get an assistant by ID."""
|
||||
for a in _list_assistants():
|
||||
if a.assistant_id == assistant_id:
|
||||
return a
|
||||
raise HTTPException(status_code=404, detail=f"Assistant {assistant_id} not found")
|
||||
|
||||
|
||||
@router.get("/{assistant_id}/graph")
|
||||
async def get_assistant_graph(assistant_id: str) -> dict:
|
||||
"""Get the graph structure for an assistant.
|
||||
|
||||
Returns a minimal graph description. Full graph introspection is
|
||||
not supported in the Gateway — this stub satisfies SDK validation.
|
||||
"""
|
||||
found = any(a.assistant_id == assistant_id for a in _list_assistants())
|
||||
if not found:
|
||||
raise HTTPException(status_code=404, detail=f"Assistant {assistant_id} not found")
|
||||
|
||||
return {
|
||||
"graph_id": "lead_agent",
|
||||
"nodes": [],
|
||||
"edges": [],
|
||||
}
|
||||
|
||||
|
||||
@router.get("/{assistant_id}/schemas")
|
||||
async def get_assistant_schemas(assistant_id: str) -> dict:
|
||||
"""Get JSON schemas for an assistant's input/output/state.
|
||||
|
||||
Returns empty schemas — full introspection not supported in Gateway.
|
||||
"""
|
||||
found = any(a.assistant_id == assistant_id for a in _list_assistants())
|
||||
if not found:
|
||||
raise HTTPException(status_code=404, detail=f"Assistant {assistant_id} not found")
|
||||
|
||||
return {
|
||||
"graph_id": "lead_agent",
|
||||
"input_schema": {},
|
||||
"output_schema": {},
|
||||
"state_schema": {},
|
||||
"config_schema": {},
|
||||
}
|
||||
@@ -1,493 +0,0 @@
|
||||
"""Authentication endpoints."""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
from ipaddress import ip_address, ip_network
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request, Response, status
|
||||
from fastapi.security import OAuth2PasswordRequestForm
|
||||
from pydantic import BaseModel, EmailStr, Field, field_validator
|
||||
|
||||
from app.gateway.auth import (
|
||||
UserResponse,
|
||||
create_access_token,
|
||||
)
|
||||
from app.gateway.auth.config import get_auth_config
|
||||
from app.gateway.auth.errors import AuthErrorCode, AuthErrorResponse
|
||||
from app.gateway.csrf_middleware import is_secure_request
|
||||
from app.gateway.deps import get_current_user_from_request, get_local_provider
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/v1/auth", tags=["auth"])
|
||||
|
||||
|
||||
# ── Request/Response Models ──────────────────────────────────────────────
|
||||
|
||||
|
||||
class LoginResponse(BaseModel):
|
||||
"""Response model for login — token only lives in HttpOnly cookie."""
|
||||
|
||||
expires_in: int # seconds
|
||||
needs_setup: bool = False
|
||||
|
||||
|
||||
# Top common-password blocklist. Drawn from the public SecLists "10k worst
|
||||
# passwords" set, lowercased + length>=8 only (shorter ones already fail
|
||||
# the min_length check). Kept tight on purpose: this is the **lower bound**
|
||||
# defense, not a full HIBP / passlib check, and runs in-process per request.
|
||||
_COMMON_PASSWORDS: frozenset[str] = frozenset(
|
||||
{
|
||||
"password",
|
||||
"password1",
|
||||
"password12",
|
||||
"password123",
|
||||
"password1234",
|
||||
"12345678",
|
||||
"123456789",
|
||||
"1234567890",
|
||||
"qwerty12",
|
||||
"qwertyui",
|
||||
"qwerty123",
|
||||
"abc12345",
|
||||
"abcd1234",
|
||||
"iloveyou",
|
||||
"letmein1",
|
||||
"welcome1",
|
||||
"welcome123",
|
||||
"admin123",
|
||||
"administrator",
|
||||
"passw0rd",
|
||||
"p@ssw0rd",
|
||||
"monkey12",
|
||||
"trustno1",
|
||||
"sunshine",
|
||||
"princess",
|
||||
"football",
|
||||
"baseball",
|
||||
"superman",
|
||||
"batman123",
|
||||
"starwars",
|
||||
"dragon123",
|
||||
"master123",
|
||||
"shadow12",
|
||||
"michael1",
|
||||
"jennifer",
|
||||
"computer",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _password_is_common(password: str) -> bool:
|
||||
"""Case-insensitive blocklist check.
|
||||
|
||||
Lowercases the input so trivial mutations like ``Password`` /
|
||||
``PASSWORD`` are also rejected. Does not normalize digit substitutions
|
||||
(``p@ssw0rd`` is included as a literal entry instead) — keeping the
|
||||
rule cheap and predictable.
|
||||
"""
|
||||
return password.lower() in _COMMON_PASSWORDS
|
||||
|
||||
|
||||
def _validate_strong_password(value: str) -> str:
|
||||
"""Pydantic field-validator body shared by Register + ChangePassword.
|
||||
|
||||
Constraint = function, not type-level mixin. The two request models
|
||||
have no "is-a" relationship; they only share the password-strength
|
||||
rule. Lifting it into a free function lets each model bind it via
|
||||
``@field_validator(field_name)`` without inheritance gymnastics.
|
||||
"""
|
||||
if _password_is_common(value):
|
||||
raise ValueError("Password is too common; choose a stronger password.")
|
||||
return value
|
||||
|
||||
|
||||
class RegisterRequest(BaseModel):
|
||||
"""Request model for user registration."""
|
||||
|
||||
email: EmailStr
|
||||
password: str = Field(..., min_length=8)
|
||||
|
||||
_strong_password = field_validator("password")(classmethod(lambda cls, v: _validate_strong_password(v)))
|
||||
|
||||
|
||||
class ChangePasswordRequest(BaseModel):
|
||||
"""Request model for password change (also handles setup flow)."""
|
||||
|
||||
current_password: str
|
||||
new_password: str = Field(..., min_length=8)
|
||||
new_email: EmailStr | None = None
|
||||
|
||||
_strong_password = field_validator("new_password")(classmethod(lambda cls, v: _validate_strong_password(v)))
|
||||
|
||||
|
||||
class MessageResponse(BaseModel):
|
||||
"""Generic message response."""
|
||||
|
||||
message: str
|
||||
|
||||
|
||||
# ── Helpers ───────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _set_session_cookie(response: Response, token: str, request: Request) -> None:
|
||||
"""Set the access_token HttpOnly cookie on the response."""
|
||||
config = get_auth_config()
|
||||
is_https = is_secure_request(request)
|
||||
response.set_cookie(
|
||||
key="access_token",
|
||||
value=token,
|
||||
httponly=True,
|
||||
secure=is_https,
|
||||
samesite="lax",
|
||||
max_age=config.token_expiry_days * 24 * 3600 if is_https else None,
|
||||
)
|
||||
|
||||
|
||||
# ── Rate Limiting ────────────────────────────────────────────────────────
|
||||
# In-process dict — not shared across workers.
|
||||
#
|
||||
# **Limitation**: with multi-worker deployments (e.g., gunicorn -w N), each
|
||||
# worker maintains its own lockout table, so an attacker effectively gets
|
||||
# N × _MAX_LOGIN_ATTEMPTS guesses before being locked out everywhere. For
|
||||
# production multi-worker setups, replace this with a shared store (Redis,
|
||||
# database-backed counter) to enforce a true per-IP limit.
|
||||
|
||||
_MAX_LOGIN_ATTEMPTS = 5
|
||||
_LOCKOUT_SECONDS = 300 # 5 minutes
|
||||
|
||||
# ip → (fail_count, lock_until_timestamp)
|
||||
_login_attempts: dict[str, tuple[int, float]] = {}
|
||||
|
||||
|
||||
def _trusted_proxies() -> list:
|
||||
"""Parse ``AUTH_TRUSTED_PROXIES`` env var into a list of ip_network objects.
|
||||
|
||||
Comma-separated CIDR or single-IP entries. Empty / unset = no proxy is
|
||||
trusted (direct mode). Invalid entries are skipped with a logger warning.
|
||||
Read live so env-var overrides take effect immediately and tests can
|
||||
``monkeypatch.setenv`` without poking a module-level cache.
|
||||
"""
|
||||
raw = os.getenv("AUTH_TRUSTED_PROXIES", "").strip()
|
||||
if not raw:
|
||||
return []
|
||||
nets = []
|
||||
for entry in raw.split(","):
|
||||
entry = entry.strip()
|
||||
if not entry:
|
||||
continue
|
||||
try:
|
||||
nets.append(ip_network(entry, strict=False))
|
||||
except ValueError:
|
||||
logger.warning("AUTH_TRUSTED_PROXIES: ignoring invalid entry %r", entry)
|
||||
return nets
|
||||
|
||||
|
||||
def _get_client_ip(request: Request) -> str:
|
||||
"""Extract the real client IP for rate limiting.
|
||||
|
||||
Trust model:
|
||||
|
||||
- The TCP peer (``request.client.host``) is always the baseline. It is
|
||||
whatever the kernel reports as the connecting socket — unforgeable
|
||||
by the client itself.
|
||||
- ``X-Real-IP`` is **only** honored if the TCP peer is in the
|
||||
``AUTH_TRUSTED_PROXIES`` allowlist (set via env var, comma-separated
|
||||
CIDR or single IPs). When set, the gateway is assumed to be behind a
|
||||
reverse proxy (nginx, Cloudflare, ALB, …) that overwrites
|
||||
``X-Real-IP`` with the original client address.
|
||||
- With no ``AUTH_TRUSTED_PROXIES`` set, ``X-Real-IP`` is silently
|
||||
ignored — closing the bypass where any client could rotate the
|
||||
header to dodge per-IP rate limits in dev / direct-gateway mode.
|
||||
|
||||
``X-Forwarded-For`` is intentionally NOT used because it is naturally
|
||||
client-controlled at the *first* hop and the trust chain is harder to
|
||||
audit per-request.
|
||||
"""
|
||||
peer_host = request.client.host if request.client else None
|
||||
|
||||
trusted = _trusted_proxies()
|
||||
if trusted and peer_host:
|
||||
try:
|
||||
peer_ip = ip_address(peer_host)
|
||||
if any(peer_ip in net for net in trusted):
|
||||
real_ip = request.headers.get("x-real-ip", "").strip()
|
||||
if real_ip:
|
||||
return real_ip
|
||||
except ValueError:
|
||||
# peer_host wasn't a parseable IP (e.g. "unknown") — fall through
|
||||
pass
|
||||
|
||||
return peer_host or "unknown"
|
||||
|
||||
|
||||
def _check_rate_limit(ip: str) -> None:
|
||||
"""Raise 429 if the IP is currently locked out."""
|
||||
record = _login_attempts.get(ip)
|
||||
if record is None:
|
||||
return
|
||||
fail_count, lock_until = record
|
||||
if fail_count >= _MAX_LOGIN_ATTEMPTS:
|
||||
if time.time() < lock_until:
|
||||
raise HTTPException(
|
||||
status_code=429,
|
||||
detail="Too many login attempts. Try again later.",
|
||||
)
|
||||
del _login_attempts[ip]
|
||||
|
||||
|
||||
_MAX_TRACKED_IPS = 10000
|
||||
|
||||
|
||||
def _record_login_failure(ip: str) -> None:
|
||||
"""Record a failed login attempt for the given IP."""
|
||||
# Evict expired lockouts when dict grows too large
|
||||
if len(_login_attempts) >= _MAX_TRACKED_IPS:
|
||||
now = time.time()
|
||||
expired = [k for k, (c, t) in _login_attempts.items() if c >= _MAX_LOGIN_ATTEMPTS and now >= t]
|
||||
for k in expired:
|
||||
del _login_attempts[k]
|
||||
# If still too large, evict cheapest-to-lose half: below-threshold
|
||||
# IPs (lock_until=0.0) sort first, then earliest-expiring lockouts.
|
||||
if len(_login_attempts) >= _MAX_TRACKED_IPS:
|
||||
by_time = sorted(_login_attempts.items(), key=lambda kv: kv[1][1])
|
||||
for k, _ in by_time[: len(by_time) // 2]:
|
||||
del _login_attempts[k]
|
||||
|
||||
record = _login_attempts.get(ip)
|
||||
if record is None:
|
||||
_login_attempts[ip] = (1, 0.0)
|
||||
else:
|
||||
new_count = record[0] + 1
|
||||
lock_until = time.time() + _LOCKOUT_SECONDS if new_count >= _MAX_LOGIN_ATTEMPTS else 0.0
|
||||
_login_attempts[ip] = (new_count, lock_until)
|
||||
|
||||
|
||||
def _record_login_success(ip: str) -> None:
|
||||
"""Clear failure counter for the given IP on successful login."""
|
||||
_login_attempts.pop(ip, None)
|
||||
|
||||
|
||||
# ── Endpoints ─────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.post("/login/local", response_model=LoginResponse)
|
||||
async def login_local(
|
||||
request: Request,
|
||||
response: Response,
|
||||
form_data: OAuth2PasswordRequestForm = Depends(),
|
||||
):
|
||||
"""Local email/password login."""
|
||||
client_ip = _get_client_ip(request)
|
||||
_check_rate_limit(client_ip)
|
||||
|
||||
user = await get_local_provider().authenticate({"email": form_data.username, "password": form_data.password})
|
||||
|
||||
if user is None:
|
||||
_record_login_failure(client_ip)
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail=AuthErrorResponse(code=AuthErrorCode.INVALID_CREDENTIALS, message="Incorrect email or password").model_dump(),
|
||||
)
|
||||
|
||||
_record_login_success(client_ip)
|
||||
token = create_access_token(str(user.id), token_version=user.token_version)
|
||||
_set_session_cookie(response, token, request)
|
||||
|
||||
return LoginResponse(
|
||||
expires_in=get_auth_config().token_expiry_days * 24 * 3600,
|
||||
needs_setup=user.needs_setup,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/register", response_model=UserResponse, status_code=status.HTTP_201_CREATED)
|
||||
async def register(request: Request, response: Response, body: RegisterRequest):
|
||||
"""Register a new user account (always 'user' role).
|
||||
|
||||
The first admin is created explicitly through /initialize. This endpoint creates regular users.
|
||||
Auto-login by setting the session cookie.
|
||||
"""
|
||||
try:
|
||||
user = await get_local_provider().create_user(email=body.email, password=body.password, system_role="user")
|
||||
except ValueError:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail=AuthErrorResponse(code=AuthErrorCode.EMAIL_ALREADY_EXISTS, message="Email already registered").model_dump(),
|
||||
)
|
||||
|
||||
token = create_access_token(str(user.id), token_version=user.token_version)
|
||||
_set_session_cookie(response, token, request)
|
||||
|
||||
return UserResponse(id=str(user.id), email=user.email, system_role=user.system_role)
|
||||
|
||||
|
||||
@router.post("/logout", response_model=MessageResponse)
|
||||
async def logout(request: Request, response: Response):
|
||||
"""Logout current user by clearing the cookie."""
|
||||
response.delete_cookie(key="access_token", secure=is_secure_request(request), samesite="lax")
|
||||
return MessageResponse(message="Successfully logged out")
|
||||
|
||||
|
||||
@router.post("/change-password", response_model=MessageResponse)
|
||||
async def change_password(request: Request, response: Response, body: ChangePasswordRequest):
|
||||
"""Change password for the currently authenticated user.
|
||||
|
||||
Also handles the first-boot setup flow:
|
||||
- If new_email is provided, updates email (checks uniqueness)
|
||||
- If user.needs_setup is True and new_email is given, clears needs_setup
|
||||
- Always increments token_version to invalidate old sessions
|
||||
- Re-issues session cookie with new token_version
|
||||
"""
|
||||
from app.gateway.auth.password import hash_password_async, verify_password_async
|
||||
|
||||
user = await get_current_user_from_request(request)
|
||||
|
||||
if user.password_hash is None:
|
||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=AuthErrorResponse(code=AuthErrorCode.INVALID_CREDENTIALS, message="OAuth users cannot change password").model_dump())
|
||||
|
||||
if not await verify_password_async(body.current_password, user.password_hash):
|
||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=AuthErrorResponse(code=AuthErrorCode.INVALID_CREDENTIALS, message="Current password is incorrect").model_dump())
|
||||
|
||||
provider = get_local_provider()
|
||||
|
||||
# Update email if provided
|
||||
if body.new_email is not None:
|
||||
existing = await provider.get_user_by_email(body.new_email)
|
||||
if existing and str(existing.id) != str(user.id):
|
||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=AuthErrorResponse(code=AuthErrorCode.EMAIL_ALREADY_EXISTS, message="Email already in use").model_dump())
|
||||
user.email = body.new_email
|
||||
|
||||
# Update password + bump version
|
||||
user.password_hash = await hash_password_async(body.new_password)
|
||||
user.token_version += 1
|
||||
|
||||
# Clear setup flag if this is the setup flow
|
||||
if user.needs_setup and body.new_email is not None:
|
||||
user.needs_setup = False
|
||||
|
||||
await provider.update_user(user)
|
||||
|
||||
# Re-issue cookie with new token_version
|
||||
token = create_access_token(str(user.id), token_version=user.token_version)
|
||||
_set_session_cookie(response, token, request)
|
||||
|
||||
return MessageResponse(message="Password changed successfully")
|
||||
|
||||
|
||||
@router.get("/me", response_model=UserResponse)
|
||||
async def get_me(request: Request):
|
||||
"""Get current authenticated user info."""
|
||||
user = await get_current_user_from_request(request)
|
||||
return UserResponse(id=str(user.id), email=user.email, system_role=user.system_role, needs_setup=user.needs_setup)
|
||||
|
||||
|
||||
_SETUP_STATUS_COOLDOWN: dict[str, float] = {}
|
||||
_SETUP_STATUS_COOLDOWN_SECONDS = 60
|
||||
_MAX_TRACKED_SETUP_STATUS_IPS = 10000
|
||||
|
||||
|
||||
@router.get("/setup-status")
|
||||
async def setup_status(request: Request):
|
||||
"""Check if an admin account exists. Returns needs_setup=True when no admin exists."""
|
||||
client_ip = _get_client_ip(request)
|
||||
now = time.time()
|
||||
last_check = _SETUP_STATUS_COOLDOWN.get(client_ip, 0)
|
||||
elapsed = now - last_check
|
||||
if elapsed < _SETUP_STATUS_COOLDOWN_SECONDS:
|
||||
retry_after = max(1, int(_SETUP_STATUS_COOLDOWN_SECONDS - elapsed))
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_429_TOO_MANY_REQUESTS,
|
||||
detail="Setup status check is rate limited",
|
||||
headers={"Retry-After": str(retry_after)},
|
||||
)
|
||||
# Evict stale entries when dict grows too large to bound memory usage.
|
||||
if len(_SETUP_STATUS_COOLDOWN) >= _MAX_TRACKED_SETUP_STATUS_IPS:
|
||||
cutoff = now - _SETUP_STATUS_COOLDOWN_SECONDS
|
||||
stale = [k for k, t in _SETUP_STATUS_COOLDOWN.items() if t < cutoff]
|
||||
for k in stale:
|
||||
del _SETUP_STATUS_COOLDOWN[k]
|
||||
# If still too large after evicting expired entries, remove oldest half.
|
||||
if len(_SETUP_STATUS_COOLDOWN) >= _MAX_TRACKED_SETUP_STATUS_IPS:
|
||||
by_time = sorted(_SETUP_STATUS_COOLDOWN.items(), key=lambda kv: kv[1])
|
||||
for k, _ in by_time[: len(by_time) // 2]:
|
||||
del _SETUP_STATUS_COOLDOWN[k]
|
||||
_SETUP_STATUS_COOLDOWN[client_ip] = now
|
||||
admin_count = await get_local_provider().count_admin_users()
|
||||
return {"needs_setup": admin_count == 0}
|
||||
|
||||
|
||||
class InitializeAdminRequest(BaseModel):
|
||||
"""Request model for first-boot admin account creation."""
|
||||
|
||||
email: EmailStr
|
||||
password: str = Field(..., min_length=8)
|
||||
|
||||
_strong_password = field_validator("password")(classmethod(lambda cls, v: _validate_strong_password(v)))
|
||||
|
||||
|
||||
@router.post("/initialize", response_model=UserResponse, status_code=status.HTTP_201_CREATED)
|
||||
async def initialize_admin(request: Request, response: Response, body: InitializeAdminRequest):
|
||||
"""Create the first admin account on initial system setup.
|
||||
|
||||
Only callable when no admin exists. Returns 409 Conflict if an admin
|
||||
already exists.
|
||||
|
||||
On success, the admin account is created with ``needs_setup=False`` and
|
||||
the session cookie is set.
|
||||
"""
|
||||
admin_count = await get_local_provider().count_admin_users()
|
||||
if admin_count > 0:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_409_CONFLICT,
|
||||
detail=AuthErrorResponse(code=AuthErrorCode.SYSTEM_ALREADY_INITIALIZED, message="System already initialized").model_dump(),
|
||||
)
|
||||
|
||||
try:
|
||||
user = await get_local_provider().create_user(email=body.email, password=body.password, system_role="admin", needs_setup=False)
|
||||
except ValueError:
|
||||
# DB unique-constraint race: another concurrent request beat us.
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_409_CONFLICT,
|
||||
detail=AuthErrorResponse(code=AuthErrorCode.SYSTEM_ALREADY_INITIALIZED, message="System already initialized").model_dump(),
|
||||
)
|
||||
|
||||
token = create_access_token(str(user.id), token_version=user.token_version)
|
||||
_set_session_cookie(response, token, request)
|
||||
|
||||
return UserResponse(id=str(user.id), email=user.email, system_role=user.system_role)
|
||||
|
||||
|
||||
# ── OAuth Endpoints (Future/Placeholder) ─────────────────────────────────
|
||||
|
||||
|
||||
@router.get("/oauth/{provider}")
|
||||
async def oauth_login(provider: str):
|
||||
"""Initiate OAuth login flow.
|
||||
|
||||
Redirects to the OAuth provider's authorization URL.
|
||||
Currently a placeholder - requires OAuth provider implementation.
|
||||
"""
|
||||
if provider not in ["github", "google"]:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail=f"Unsupported OAuth provider: {provider}",
|
||||
)
|
||||
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_501_NOT_IMPLEMENTED,
|
||||
detail="OAuth login not yet implemented",
|
||||
)
|
||||
|
||||
|
||||
@router.get("/callback/{provider}")
|
||||
async def oauth_callback(provider: str, code: str, state: str):
|
||||
"""OAuth callback endpoint.
|
||||
|
||||
Handles the OAuth provider's callback after user authorization.
|
||||
Currently a placeholder.
|
||||
"""
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_501_NOT_IMPLEMENTED,
|
||||
detail="OAuth callback not yet implemented",
|
||||
)
|
||||
@@ -1,52 +0,0 @@
|
||||
"""Gateway router for IM channel management."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from pydantic import BaseModel
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/channels", tags=["channels"])
|
||||
|
||||
|
||||
class ChannelStatusResponse(BaseModel):
|
||||
service_running: bool
|
||||
channels: dict[str, dict]
|
||||
|
||||
|
||||
class ChannelRestartResponse(BaseModel):
|
||||
success: bool
|
||||
message: str
|
||||
|
||||
|
||||
@router.get("/", response_model=ChannelStatusResponse)
|
||||
async def get_channels_status() -> ChannelStatusResponse:
|
||||
"""Get the status of all IM channels."""
|
||||
from app.channels.service import get_channel_service
|
||||
|
||||
service = get_channel_service()
|
||||
if service is None:
|
||||
return ChannelStatusResponse(service_running=False, channels={})
|
||||
status = service.get_status()
|
||||
return ChannelStatusResponse(**status)
|
||||
|
||||
|
||||
@router.post("/{name}/restart", response_model=ChannelRestartResponse)
|
||||
async def restart_channel(name: str) -> ChannelRestartResponse:
|
||||
"""Restart a specific IM channel."""
|
||||
from app.channels.service import get_channel_service
|
||||
|
||||
service = get_channel_service()
|
||||
if service is None:
|
||||
raise HTTPException(status_code=503, detail="Channel service is not running")
|
||||
|
||||
success = await service.restart_channel(name)
|
||||
if success:
|
||||
logger.info("Channel %s restarted successfully", name)
|
||||
return ChannelRestartResponse(success=True, message=f"Channel {name} restarted successfully")
|
||||
else:
|
||||
logger.warning("Failed to restart channel %s", name)
|
||||
return ChannelRestartResponse(success=False, message=f"Failed to restart channel {name}")
|
||||
@@ -1,188 +0,0 @@
|
||||
"""Feedback endpoints — create, list, stats, delete.
|
||||
|
||||
Allows users to submit thumbs-up/down feedback on runs,
|
||||
optionally scoped to a specific message.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Request
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.gateway.authz import require_permission
|
||||
from app.gateway.deps import get_current_user, get_feedback_repo, get_run_store
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/api/threads", tags=["feedback"])
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Request / response models
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class FeedbackCreateRequest(BaseModel):
|
||||
rating: int = Field(..., description="Feedback rating: +1 (positive) or -1 (negative)")
|
||||
comment: str | None = Field(default=None, description="Optional text feedback")
|
||||
message_id: str | None = Field(default=None, description="Optional: scope feedback to a specific message")
|
||||
|
||||
|
||||
class FeedbackUpsertRequest(BaseModel):
|
||||
rating: int = Field(..., description="Feedback rating: +1 (positive) or -1 (negative)")
|
||||
comment: str | None = Field(default=None, description="Optional text feedback")
|
||||
|
||||
|
||||
class FeedbackResponse(BaseModel):
|
||||
feedback_id: str
|
||||
run_id: str
|
||||
thread_id: str
|
||||
user_id: str | None = None
|
||||
message_id: str | None = None
|
||||
rating: int
|
||||
comment: str | None = None
|
||||
created_at: str = ""
|
||||
|
||||
|
||||
class FeedbackStatsResponse(BaseModel):
|
||||
run_id: str
|
||||
total: int = 0
|
||||
positive: int = 0
|
||||
negative: int = 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Endpoints
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@router.put("/{thread_id}/runs/{run_id}/feedback", response_model=FeedbackResponse)
|
||||
@require_permission("threads", "write", owner_check=True, require_existing=True)
|
||||
async def upsert_feedback(
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
body: FeedbackUpsertRequest,
|
||||
request: Request,
|
||||
) -> dict[str, Any]:
|
||||
"""Create or update feedback for a run (idempotent)."""
|
||||
if body.rating not in (1, -1):
|
||||
raise HTTPException(status_code=400, detail="rating must be +1 or -1")
|
||||
|
||||
user_id = await get_current_user(request)
|
||||
|
||||
run_store = get_run_store(request)
|
||||
run = await run_store.get(run_id)
|
||||
if run is None:
|
||||
raise HTTPException(status_code=404, detail=f"Run {run_id} not found")
|
||||
if run.get("thread_id") != thread_id:
|
||||
raise HTTPException(status_code=404, detail=f"Run {run_id} not found in thread {thread_id}")
|
||||
|
||||
feedback_repo = get_feedback_repo(request)
|
||||
return await feedback_repo.upsert(
|
||||
run_id=run_id,
|
||||
thread_id=thread_id,
|
||||
rating=body.rating,
|
||||
user_id=user_id,
|
||||
comment=body.comment,
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/{thread_id}/runs/{run_id}/feedback")
|
||||
@require_permission("threads", "delete", owner_check=True, require_existing=True)
|
||||
async def delete_run_feedback(
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
request: Request,
|
||||
) -> dict[str, bool]:
|
||||
"""Delete the current user's feedback for a run."""
|
||||
user_id = await get_current_user(request)
|
||||
feedback_repo = get_feedback_repo(request)
|
||||
deleted = await feedback_repo.delete_by_run(
|
||||
thread_id=thread_id,
|
||||
run_id=run_id,
|
||||
user_id=user_id,
|
||||
)
|
||||
if not deleted:
|
||||
raise HTTPException(status_code=404, detail="No feedback found for this run")
|
||||
return {"success": True}
|
||||
|
||||
|
||||
@router.post("/{thread_id}/runs/{run_id}/feedback", response_model=FeedbackResponse)
|
||||
@require_permission("threads", "write", owner_check=True, require_existing=True)
|
||||
async def create_feedback(
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
body: FeedbackCreateRequest,
|
||||
request: Request,
|
||||
) -> dict[str, Any]:
|
||||
"""Submit feedback (thumbs-up/down) for a run."""
|
||||
if body.rating not in (1, -1):
|
||||
raise HTTPException(status_code=400, detail="rating must be +1 or -1")
|
||||
|
||||
user_id = await get_current_user(request)
|
||||
|
||||
# Validate run exists and belongs to thread
|
||||
run_store = get_run_store(request)
|
||||
run = await run_store.get(run_id)
|
||||
if run is None:
|
||||
raise HTTPException(status_code=404, detail=f"Run {run_id} not found")
|
||||
if run.get("thread_id") != thread_id:
|
||||
raise HTTPException(status_code=404, detail=f"Run {run_id} not found in thread {thread_id}")
|
||||
|
||||
feedback_repo = get_feedback_repo(request)
|
||||
return await feedback_repo.create(
|
||||
run_id=run_id,
|
||||
thread_id=thread_id,
|
||||
rating=body.rating,
|
||||
user_id=user_id,
|
||||
message_id=body.message_id,
|
||||
comment=body.comment,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{thread_id}/runs/{run_id}/feedback", response_model=list[FeedbackResponse])
|
||||
@require_permission("threads", "read", owner_check=True)
|
||||
async def list_feedback(
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
request: Request,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""List all feedback for a run."""
|
||||
feedback_repo = get_feedback_repo(request)
|
||||
return await feedback_repo.list_by_run(thread_id, run_id)
|
||||
|
||||
|
||||
@router.get("/{thread_id}/runs/{run_id}/feedback/stats", response_model=FeedbackStatsResponse)
|
||||
@require_permission("threads", "read", owner_check=True)
|
||||
async def feedback_stats(
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
request: Request,
|
||||
) -> dict[str, Any]:
|
||||
"""Get aggregated feedback stats (positive/negative counts) for a run."""
|
||||
feedback_repo = get_feedback_repo(request)
|
||||
return await feedback_repo.aggregate_by_run(thread_id, run_id)
|
||||
|
||||
|
||||
@router.delete("/{thread_id}/runs/{run_id}/feedback/{feedback_id}")
|
||||
@require_permission("threads", "delete", owner_check=True, require_existing=True)
|
||||
async def delete_feedback(
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
feedback_id: str,
|
||||
request: Request,
|
||||
) -> dict[str, bool]:
|
||||
"""Delete a feedback record."""
|
||||
feedback_repo = get_feedback_repo(request)
|
||||
# Verify feedback belongs to the specified thread/run before deleting
|
||||
existing = await feedback_repo.get(feedback_id)
|
||||
if existing is None:
|
||||
raise HTTPException(status_code=404, detail=f"Feedback {feedback_id} not found")
|
||||
if existing.get("thread_id") != thread_id or existing.get("run_id") != run_id:
|
||||
raise HTTPException(status_code=404, detail=f"Feedback {feedback_id} not found in run {run_id}")
|
||||
deleted = await feedback_repo.delete(feedback_id)
|
||||
if not deleted:
|
||||
raise HTTPException(status_code=404, detail=f"Feedback {feedback_id} not found")
|
||||
return {"success": True}
|
||||
@@ -1,169 +0,0 @@
|
||||
import json
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import Literal
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from deerflow.config.extensions_config import ExtensionsConfig, get_extensions_config, reload_extensions_config
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/api", tags=["mcp"])
|
||||
|
||||
|
||||
class McpOAuthConfigResponse(BaseModel):
|
||||
"""OAuth configuration for an MCP server."""
|
||||
|
||||
enabled: bool = Field(default=True, description="Whether OAuth token injection is enabled")
|
||||
token_url: str = Field(default="", description="OAuth token endpoint URL")
|
||||
grant_type: Literal["client_credentials", "refresh_token"] = Field(default="client_credentials", description="OAuth grant type")
|
||||
client_id: str | None = Field(default=None, description="OAuth client ID")
|
||||
client_secret: str | None = Field(default=None, description="OAuth client secret")
|
||||
refresh_token: str | None = Field(default=None, description="OAuth refresh token")
|
||||
scope: str | None = Field(default=None, description="OAuth scope")
|
||||
audience: str | None = Field(default=None, description="OAuth audience")
|
||||
token_field: str = Field(default="access_token", description="Token response field containing access token")
|
||||
token_type_field: str = Field(default="token_type", description="Token response field containing token type")
|
||||
expires_in_field: str = Field(default="expires_in", description="Token response field containing expires-in seconds")
|
||||
default_token_type: str = Field(default="Bearer", description="Default token type when response omits token_type")
|
||||
refresh_skew_seconds: int = Field(default=60, description="Refresh this many seconds before expiry")
|
||||
extra_token_params: dict[str, str] = Field(default_factory=dict, description="Additional form params sent to token endpoint")
|
||||
|
||||
|
||||
class McpServerConfigResponse(BaseModel):
|
||||
"""Response model for MCP server configuration."""
|
||||
|
||||
enabled: bool = Field(default=True, description="Whether this MCP server is enabled")
|
||||
type: str = Field(default="stdio", description="Transport type: 'stdio', 'sse', or 'http'")
|
||||
command: str | None = Field(default=None, description="Command to execute to start the MCP server (for stdio type)")
|
||||
args: list[str] = Field(default_factory=list, description="Arguments to pass to the command (for stdio type)")
|
||||
env: dict[str, str] = Field(default_factory=dict, description="Environment variables for the MCP server")
|
||||
url: str | None = Field(default=None, description="URL of the MCP server (for sse or http type)")
|
||||
headers: dict[str, str] = Field(default_factory=dict, description="HTTP headers to send (for sse or http type)")
|
||||
oauth: McpOAuthConfigResponse | None = Field(default=None, description="OAuth configuration for MCP HTTP/SSE servers")
|
||||
description: str = Field(default="", description="Human-readable description of what this MCP server provides")
|
||||
|
||||
|
||||
class McpConfigResponse(BaseModel):
|
||||
"""Response model for MCP configuration."""
|
||||
|
||||
mcp_servers: dict[str, McpServerConfigResponse] = Field(
|
||||
default_factory=dict,
|
||||
description="Map of MCP server name to configuration",
|
||||
)
|
||||
|
||||
|
||||
class McpConfigUpdateRequest(BaseModel):
|
||||
"""Request model for updating MCP configuration."""
|
||||
|
||||
mcp_servers: dict[str, McpServerConfigResponse] = Field(
|
||||
...,
|
||||
description="Map of MCP server name to configuration",
|
||||
)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/mcp/config",
|
||||
response_model=McpConfigResponse,
|
||||
summary="Get MCP Configuration",
|
||||
description="Retrieve the current Model Context Protocol (MCP) server configurations.",
|
||||
)
|
||||
async def get_mcp_configuration() -> McpConfigResponse:
|
||||
"""Get the current MCP configuration.
|
||||
|
||||
Returns:
|
||||
The current MCP configuration with all servers.
|
||||
|
||||
Example:
|
||||
```json
|
||||
{
|
||||
"mcp_servers": {
|
||||
"github": {
|
||||
"enabled": true,
|
||||
"command": "npx",
|
||||
"args": ["-y", "@modelcontextprotocol/server-github"],
|
||||
"env": {"GITHUB_TOKEN": "ghp_xxx"},
|
||||
"description": "GitHub MCP server for repository operations"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
"""
|
||||
config = get_extensions_config()
|
||||
|
||||
return McpConfigResponse(mcp_servers={name: McpServerConfigResponse(**server.model_dump()) for name, server in config.mcp_servers.items()})
|
||||
|
||||
|
||||
@router.put(
|
||||
"/mcp/config",
|
||||
response_model=McpConfigResponse,
|
||||
summary="Update MCP Configuration",
|
||||
description="Update Model Context Protocol (MCP) server configurations and save to file.",
|
||||
)
|
||||
async def update_mcp_configuration(request: McpConfigUpdateRequest) -> McpConfigResponse:
|
||||
"""Update the MCP configuration.
|
||||
|
||||
This will:
|
||||
1. Save the new configuration to the mcp_config.json file
|
||||
2. Reload the configuration cache
|
||||
3. Reset MCP tools cache to trigger reinitialization
|
||||
|
||||
Args:
|
||||
request: The new MCP configuration to save.
|
||||
|
||||
Returns:
|
||||
The updated MCP configuration.
|
||||
|
||||
Raises:
|
||||
HTTPException: 500 if the configuration file cannot be written.
|
||||
|
||||
Example Request:
|
||||
```json
|
||||
{
|
||||
"mcp_servers": {
|
||||
"github": {
|
||||
"enabled": true,
|
||||
"command": "npx",
|
||||
"args": ["-y", "@modelcontextprotocol/server-github"],
|
||||
"env": {"GITHUB_TOKEN": "$GITHUB_TOKEN"},
|
||||
"description": "GitHub MCP server for repository operations"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
"""
|
||||
try:
|
||||
# Get the current config path (or determine where to save it)
|
||||
config_path = ExtensionsConfig.resolve_config_path()
|
||||
|
||||
# If no config file exists, create one in the parent directory (project root)
|
||||
if config_path is None:
|
||||
config_path = Path.cwd().parent / "extensions_config.json"
|
||||
logger.info(f"No existing extensions config found. Creating new config at: {config_path}")
|
||||
|
||||
# Load current config to preserve skills configuration
|
||||
current_config = get_extensions_config()
|
||||
|
||||
# Convert request to dict format for JSON serialization
|
||||
config_data = {
|
||||
"mcpServers": {name: server.model_dump() for name, server in request.mcp_servers.items()},
|
||||
"skills": {name: {"enabled": skill.enabled} for name, skill in current_config.skills.items()},
|
||||
}
|
||||
|
||||
# Write the configuration to file
|
||||
with open(config_path, "w", encoding="utf-8") as f:
|
||||
json.dump(config_data, f, indent=2)
|
||||
|
||||
logger.info(f"MCP configuration updated and saved to: {config_path}")
|
||||
|
||||
# NOTE: No need to reload/reset cache here - LangGraph Server (separate process)
|
||||
# will detect config file changes via mtime and reinitialize MCP tools automatically
|
||||
|
||||
# Reload the configuration and update the global cache
|
||||
reloaded_config = reload_extensions_config()
|
||||
return McpConfigResponse(mcp_servers={name: McpServerConfigResponse(**server.model_dump()) for name, server in reloaded_config.mcp_servers.items()})
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to update MCP configuration: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to update MCP configuration: {str(e)}")
|
||||
@@ -1,356 +0,0 @@
|
||||
"""Memory API router for retrieving and managing global memory data."""
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from deerflow.agents.memory.updater import (
|
||||
clear_memory_data,
|
||||
create_memory_fact,
|
||||
delete_memory_fact,
|
||||
get_memory_data,
|
||||
import_memory_data,
|
||||
reload_memory_data,
|
||||
update_memory_fact,
|
||||
)
|
||||
from deerflow.config.memory_config import get_memory_config
|
||||
from deerflow.runtime.user_context import get_effective_user_id
|
||||
|
||||
router = APIRouter(prefix="/api", tags=["memory"])
|
||||
|
||||
|
||||
class ContextSection(BaseModel):
|
||||
"""Model for context sections (user and history)."""
|
||||
|
||||
summary: str = Field(default="", description="Summary content")
|
||||
updatedAt: str = Field(default="", description="Last update timestamp")
|
||||
|
||||
|
||||
class UserContext(BaseModel):
|
||||
"""Model for user context."""
|
||||
|
||||
workContext: ContextSection = Field(default_factory=ContextSection)
|
||||
personalContext: ContextSection = Field(default_factory=ContextSection)
|
||||
topOfMind: ContextSection = Field(default_factory=ContextSection)
|
||||
|
||||
|
||||
class HistoryContext(BaseModel):
|
||||
"""Model for history context."""
|
||||
|
||||
recentMonths: ContextSection = Field(default_factory=ContextSection)
|
||||
earlierContext: ContextSection = Field(default_factory=ContextSection)
|
||||
longTermBackground: ContextSection = Field(default_factory=ContextSection)
|
||||
|
||||
|
||||
class Fact(BaseModel):
|
||||
"""Model for a memory fact."""
|
||||
|
||||
id: str = Field(..., description="Unique identifier for the fact")
|
||||
content: str = Field(..., description="Fact content")
|
||||
category: str = Field(default="context", description="Fact category")
|
||||
confidence: float = Field(default=0.5, description="Confidence score (0-1)")
|
||||
createdAt: str = Field(default="", description="Creation timestamp")
|
||||
source: str = Field(default="unknown", description="Source thread ID")
|
||||
sourceError: str | None = Field(default=None, description="Optional description of the prior mistake or wrong approach")
|
||||
|
||||
|
||||
class MemoryResponse(BaseModel):
|
||||
"""Response model for memory data."""
|
||||
|
||||
version: str = Field(default="1.0", description="Memory schema version")
|
||||
lastUpdated: str = Field(default="", description="Last update timestamp")
|
||||
user: UserContext = Field(default_factory=UserContext)
|
||||
history: HistoryContext = Field(default_factory=HistoryContext)
|
||||
facts: list[Fact] = Field(default_factory=list)
|
||||
|
||||
|
||||
def _map_memory_fact_value_error(exc: ValueError) -> HTTPException:
|
||||
"""Convert updater validation errors into stable API responses."""
|
||||
if exc.args and exc.args[0] == "confidence":
|
||||
detail = "Invalid confidence value; must be between 0 and 1."
|
||||
else:
|
||||
detail = "Memory fact content cannot be empty."
|
||||
return HTTPException(status_code=400, detail=detail)
|
||||
|
||||
|
||||
class FactCreateRequest(BaseModel):
|
||||
"""Request model for creating a memory fact."""
|
||||
|
||||
content: str = Field(..., min_length=1, description="Fact content")
|
||||
category: str = Field(default="context", description="Fact category")
|
||||
confidence: float = Field(default=0.5, ge=0.0, le=1.0, description="Confidence score (0-1)")
|
||||
|
||||
|
||||
class FactPatchRequest(BaseModel):
|
||||
"""PATCH request model that preserves existing values for omitted fields."""
|
||||
|
||||
content: str | None = Field(default=None, min_length=1, description="Fact content")
|
||||
category: str | None = Field(default=None, description="Fact category")
|
||||
confidence: float | None = Field(default=None, ge=0.0, le=1.0, description="Confidence score (0-1)")
|
||||
|
||||
|
||||
class MemoryConfigResponse(BaseModel):
|
||||
"""Response model for memory configuration."""
|
||||
|
||||
enabled: bool = Field(..., description="Whether memory is enabled")
|
||||
storage_path: str = Field(..., description="Path to memory storage file")
|
||||
debounce_seconds: int = Field(..., description="Debounce time for memory updates")
|
||||
max_facts: int = Field(..., description="Maximum number of facts to store")
|
||||
fact_confidence_threshold: float = Field(..., description="Minimum confidence threshold for facts")
|
||||
injection_enabled: bool = Field(..., description="Whether memory injection is enabled")
|
||||
max_injection_tokens: int = Field(..., description="Maximum tokens for memory injection")
|
||||
|
||||
|
||||
class MemoryStatusResponse(BaseModel):
|
||||
"""Response model for memory status."""
|
||||
|
||||
config: MemoryConfigResponse
|
||||
data: MemoryResponse
|
||||
|
||||
|
||||
@router.get(
|
||||
"/memory",
|
||||
response_model=MemoryResponse,
|
||||
response_model_exclude_none=True,
|
||||
summary="Get Memory Data",
|
||||
description="Retrieve the current global memory data including user context, history, and facts.",
|
||||
)
|
||||
async def get_memory() -> MemoryResponse:
|
||||
"""Get the current global memory data.
|
||||
|
||||
Returns:
|
||||
The current memory data with user context, history, and facts.
|
||||
|
||||
Example Response:
|
||||
```json
|
||||
{
|
||||
"version": "1.0",
|
||||
"lastUpdated": "2024-01-15T10:30:00Z",
|
||||
"user": {
|
||||
"workContext": {"summary": "Working on DeerFlow project", "updatedAt": "..."},
|
||||
"personalContext": {"summary": "Prefers concise responses", "updatedAt": "..."},
|
||||
"topOfMind": {"summary": "Building memory API", "updatedAt": "..."}
|
||||
},
|
||||
"history": {
|
||||
"recentMonths": {"summary": "Recent development activities", "updatedAt": "..."},
|
||||
"earlierContext": {"summary": "", "updatedAt": ""},
|
||||
"longTermBackground": {"summary": "", "updatedAt": ""}
|
||||
},
|
||||
"facts": [
|
||||
{
|
||||
"id": "fact_abc123",
|
||||
"content": "User prefers TypeScript over JavaScript",
|
||||
"category": "preference",
|
||||
"confidence": 0.9,
|
||||
"createdAt": "2024-01-15T10:30:00Z",
|
||||
"source": "thread_xyz"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
"""
|
||||
memory_data = get_memory_data(user_id=get_effective_user_id())
|
||||
return MemoryResponse(**memory_data)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/memory/reload",
|
||||
response_model=MemoryResponse,
|
||||
response_model_exclude_none=True,
|
||||
summary="Reload Memory Data",
|
||||
description="Reload memory data from the storage file, refreshing the in-memory cache.",
|
||||
)
|
||||
async def reload_memory() -> MemoryResponse:
|
||||
"""Reload memory data from file.
|
||||
|
||||
This forces a reload of the memory data from the storage file,
|
||||
useful when the file has been modified externally.
|
||||
|
||||
Returns:
|
||||
The reloaded memory data.
|
||||
"""
|
||||
memory_data = reload_memory_data(user_id=get_effective_user_id())
|
||||
return MemoryResponse(**memory_data)
|
||||
|
||||
|
||||
@router.delete(
|
||||
"/memory",
|
||||
response_model=MemoryResponse,
|
||||
response_model_exclude_none=True,
|
||||
summary="Clear All Memory Data",
|
||||
description="Delete all saved memory data and reset the memory structure to an empty state.",
|
||||
)
|
||||
async def clear_memory() -> MemoryResponse:
|
||||
"""Clear all persisted memory data."""
|
||||
try:
|
||||
memory_data = clear_memory_data(user_id=get_effective_user_id())
|
||||
except OSError as exc:
|
||||
raise HTTPException(status_code=500, detail="Failed to clear memory data.") from exc
|
||||
|
||||
return MemoryResponse(**memory_data)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/memory/facts",
|
||||
response_model=MemoryResponse,
|
||||
response_model_exclude_none=True,
|
||||
summary="Create Memory Fact",
|
||||
description="Create a single saved memory fact manually.",
|
||||
)
|
||||
async def create_memory_fact_endpoint(request: FactCreateRequest) -> MemoryResponse:
|
||||
"""Create a single fact manually."""
|
||||
try:
|
||||
memory_data = create_memory_fact(
|
||||
content=request.content,
|
||||
category=request.category,
|
||||
confidence=request.confidence,
|
||||
user_id=get_effective_user_id(),
|
||||
)
|
||||
except ValueError as exc:
|
||||
raise _map_memory_fact_value_error(exc) from exc
|
||||
except OSError as exc:
|
||||
raise HTTPException(status_code=500, detail="Failed to create memory fact.") from exc
|
||||
|
||||
return MemoryResponse(**memory_data)
|
||||
|
||||
|
||||
@router.delete(
|
||||
"/memory/facts/{fact_id}",
|
||||
response_model=MemoryResponse,
|
||||
response_model_exclude_none=True,
|
||||
summary="Delete Memory Fact",
|
||||
description="Delete a single saved memory fact by its fact id.",
|
||||
)
|
||||
async def delete_memory_fact_endpoint(fact_id: str) -> MemoryResponse:
|
||||
"""Delete a single fact from memory by fact id."""
|
||||
try:
|
||||
memory_data = delete_memory_fact(fact_id, user_id=get_effective_user_id())
|
||||
except KeyError as exc:
|
||||
raise HTTPException(status_code=404, detail=f"Memory fact '{fact_id}' not found.") from exc
|
||||
except OSError as exc:
|
||||
raise HTTPException(status_code=500, detail="Failed to delete memory fact.") from exc
|
||||
|
||||
return MemoryResponse(**memory_data)
|
||||
|
||||
|
||||
@router.patch(
|
||||
"/memory/facts/{fact_id}",
|
||||
response_model=MemoryResponse,
|
||||
response_model_exclude_none=True,
|
||||
summary="Patch Memory Fact",
|
||||
description="Partially update a single saved memory fact by its fact id while preserving omitted fields.",
|
||||
)
|
||||
async def update_memory_fact_endpoint(fact_id: str, request: FactPatchRequest) -> MemoryResponse:
|
||||
"""Partially update a single fact manually."""
|
||||
try:
|
||||
memory_data = update_memory_fact(
|
||||
fact_id=fact_id,
|
||||
content=request.content,
|
||||
category=request.category,
|
||||
confidence=request.confidence,
|
||||
user_id=get_effective_user_id(),
|
||||
)
|
||||
except ValueError as exc:
|
||||
raise _map_memory_fact_value_error(exc) from exc
|
||||
except KeyError as exc:
|
||||
raise HTTPException(status_code=404, detail=f"Memory fact '{fact_id}' not found.") from exc
|
||||
except OSError as exc:
|
||||
raise HTTPException(status_code=500, detail="Failed to update memory fact.") from exc
|
||||
|
||||
return MemoryResponse(**memory_data)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/memory/export",
|
||||
response_model=MemoryResponse,
|
||||
response_model_exclude_none=True,
|
||||
summary="Export Memory Data",
|
||||
description="Export the current global memory data as JSON for backup or transfer.",
|
||||
)
|
||||
async def export_memory() -> MemoryResponse:
|
||||
"""Export the current memory data."""
|
||||
memory_data = get_memory_data(user_id=get_effective_user_id())
|
||||
return MemoryResponse(**memory_data)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/memory/import",
|
||||
response_model=MemoryResponse,
|
||||
response_model_exclude_none=True,
|
||||
summary="Import Memory Data",
|
||||
description="Import and overwrite the current global memory data from a JSON payload.",
|
||||
)
|
||||
async def import_memory(request: MemoryResponse) -> MemoryResponse:
|
||||
"""Import and persist memory data."""
|
||||
try:
|
||||
memory_data = import_memory_data(request.model_dump(), user_id=get_effective_user_id())
|
||||
except OSError as exc:
|
||||
raise HTTPException(status_code=500, detail="Failed to import memory data.") from exc
|
||||
|
||||
return MemoryResponse(**memory_data)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/memory/config",
|
||||
response_model=MemoryConfigResponse,
|
||||
summary="Get Memory Configuration",
|
||||
description="Retrieve the current memory system configuration.",
|
||||
)
|
||||
async def get_memory_config_endpoint() -> MemoryConfigResponse:
|
||||
"""Get the memory system configuration.
|
||||
|
||||
Returns:
|
||||
The current memory configuration settings.
|
||||
|
||||
Example Response:
|
||||
```json
|
||||
{
|
||||
"enabled": true,
|
||||
"storage_path": ".deer-flow/memory.json",
|
||||
"debounce_seconds": 30,
|
||||
"max_facts": 100,
|
||||
"fact_confidence_threshold": 0.7,
|
||||
"injection_enabled": true,
|
||||
"max_injection_tokens": 2000
|
||||
}
|
||||
```
|
||||
"""
|
||||
config = get_memory_config()
|
||||
return MemoryConfigResponse(
|
||||
enabled=config.enabled,
|
||||
storage_path=config.storage_path,
|
||||
debounce_seconds=config.debounce_seconds,
|
||||
max_facts=config.max_facts,
|
||||
fact_confidence_threshold=config.fact_confidence_threshold,
|
||||
injection_enabled=config.injection_enabled,
|
||||
max_injection_tokens=config.max_injection_tokens,
|
||||
)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/memory/status",
|
||||
response_model=MemoryStatusResponse,
|
||||
response_model_exclude_none=True,
|
||||
summary="Get Memory Status",
|
||||
description="Retrieve both memory configuration and current data in a single request.",
|
||||
)
|
||||
async def get_memory_status() -> MemoryStatusResponse:
|
||||
"""Get the memory system status including configuration and data.
|
||||
|
||||
Returns:
|
||||
Combined memory configuration and current data.
|
||||
"""
|
||||
config = get_memory_config()
|
||||
memory_data = get_memory_data(user_id=get_effective_user_id())
|
||||
|
||||
return MemoryStatusResponse(
|
||||
config=MemoryConfigResponse(
|
||||
enabled=config.enabled,
|
||||
storage_path=config.storage_path,
|
||||
debounce_seconds=config.debounce_seconds,
|
||||
max_facts=config.max_facts,
|
||||
fact_confidence_threshold=config.fact_confidence_threshold,
|
||||
injection_enabled=config.injection_enabled,
|
||||
max_injection_tokens=config.max_injection_tokens,
|
||||
),
|
||||
data=MemoryResponse(**memory_data),
|
||||
)
|
||||
@@ -1,132 +0,0 @@
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.gateway.deps import get_config
|
||||
from deerflow.config.app_config import AppConfig
|
||||
|
||||
router = APIRouter(prefix="/api", tags=["models"])
|
||||
|
||||
|
||||
class ModelResponse(BaseModel):
|
||||
"""Response model for model information."""
|
||||
|
||||
name: str = Field(..., description="Unique identifier for the model")
|
||||
model: str = Field(..., description="Actual provider model identifier")
|
||||
display_name: str | None = Field(None, description="Human-readable name")
|
||||
description: str | None = Field(None, description="Model description")
|
||||
supports_thinking: bool = Field(default=False, description="Whether model supports thinking mode")
|
||||
supports_reasoning_effort: bool = Field(default=False, description="Whether model supports reasoning effort")
|
||||
|
||||
|
||||
class TokenUsageResponse(BaseModel):
|
||||
"""Token usage display configuration."""
|
||||
|
||||
enabled: bool = Field(default=False, description="Whether token usage display is enabled")
|
||||
|
||||
|
||||
class ModelsListResponse(BaseModel):
|
||||
"""Response model for listing all models."""
|
||||
|
||||
models: list[ModelResponse]
|
||||
token_usage: TokenUsageResponse
|
||||
|
||||
|
||||
@router.get(
|
||||
"/models",
|
||||
response_model=ModelsListResponse,
|
||||
summary="List All Models",
|
||||
description="Retrieve a list of all available AI models configured in the system.",
|
||||
)
|
||||
async def list_models(config: AppConfig = Depends(get_config)) -> ModelsListResponse:
|
||||
"""List all available models from configuration.
|
||||
|
||||
Returns model information suitable for frontend display,
|
||||
excluding sensitive fields like API keys and internal configuration.
|
||||
|
||||
Returns:
|
||||
A list of all configured models with their metadata and token usage display settings.
|
||||
|
||||
Example Response:
|
||||
```json
|
||||
{
|
||||
"models": [
|
||||
{
|
||||
"name": "gpt-4",
|
||||
"model": "gpt-4",
|
||||
"display_name": "GPT-4",
|
||||
"description": "OpenAI GPT-4 model",
|
||||
"supports_thinking": false,
|
||||
"supports_reasoning_effort": false
|
||||
},
|
||||
{
|
||||
"name": "claude-3-opus",
|
||||
"model": "claude-3-opus",
|
||||
"display_name": "Claude 3 Opus",
|
||||
"description": "Anthropic Claude 3 Opus model",
|
||||
"supports_thinking": true,
|
||||
"supports_reasoning_effort": false
|
||||
}
|
||||
],
|
||||
"token_usage": {
|
||||
"enabled": true
|
||||
}
|
||||
}
|
||||
```
|
||||
"""
|
||||
models = [
|
||||
ModelResponse(
|
||||
name=model.name,
|
||||
model=model.model,
|
||||
display_name=model.display_name,
|
||||
description=model.description,
|
||||
supports_thinking=model.supports_thinking,
|
||||
supports_reasoning_effort=model.supports_reasoning_effort,
|
||||
)
|
||||
for model in config.models
|
||||
]
|
||||
return ModelsListResponse(
|
||||
models=models,
|
||||
token_usage=TokenUsageResponse(enabled=config.token_usage.enabled),
|
||||
)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/models/{model_name}",
|
||||
response_model=ModelResponse,
|
||||
summary="Get Model Details",
|
||||
description="Retrieve detailed information about a specific AI model by its name.",
|
||||
)
|
||||
async def get_model(model_name: str, config: AppConfig = Depends(get_config)) -> ModelResponse:
|
||||
"""Get a specific model by name.
|
||||
|
||||
Args:
|
||||
model_name: The unique name of the model to retrieve.
|
||||
|
||||
Returns:
|
||||
Model information if found.
|
||||
|
||||
Raises:
|
||||
HTTPException: 404 if model not found.
|
||||
|
||||
Example Response:
|
||||
```json
|
||||
{
|
||||
"name": "gpt-4",
|
||||
"display_name": "GPT-4",
|
||||
"description": "OpenAI GPT-4 model",
|
||||
"supports_thinking": false
|
||||
}
|
||||
```
|
||||
"""
|
||||
model = config.get_model_config(model_name)
|
||||
if model is None:
|
||||
raise HTTPException(status_code=404, detail=f"Model '{model_name}' not found")
|
||||
|
||||
return ModelResponse(
|
||||
name=model.name,
|
||||
model=model.model,
|
||||
display_name=model.display_name,
|
||||
description=model.description,
|
||||
supports_thinking=model.supports_thinking,
|
||||
supports_reasoning_effort=model.supports_reasoning_effort,
|
||||
)
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user