mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-06-10 01:15:58 +00:00
Compare commits
3 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 9c944fe698 | |||
| b62ac7672a | |||
| d71b452a34 |
@@ -59,7 +59,7 @@ smoke-test/
|
||||
2. **Check pnpm** - Package manager
|
||||
3. **Check uv** - Python package manager
|
||||
4. **Check nginx** - Reverse proxy
|
||||
5. **Check required ports** - Confirm that ports 2026, 3000, and 8001 are not occupied
|
||||
5. **Check required ports** - Confirm that ports 2026, 3000, 8001, and 2024 are not occupied
|
||||
|
||||
**Docker mode environment check** (if Docker is selected):
|
||||
1. **Check whether Docker is installed** - Run `docker --version`
|
||||
@@ -93,17 +93,17 @@ smoke-test/
|
||||
### Phase 5: Service Health Check
|
||||
|
||||
**Local mode health check**:
|
||||
1. **Check process status** - Confirm that Gateway, Frontend, and Nginx processes are all running
|
||||
1. **Check process status** - Confirm that LangGraph, Gateway, Frontend, and Nginx processes are all running
|
||||
2. **Check frontend service** - Visit `http://localhost:2026` and verify that the page loads
|
||||
3. **Check API Gateway** - Verify the `http://localhost:2026/health` endpoint
|
||||
4. **Check LangGraph-compatible API** - Verify the `/api/langgraph/*` route exposed by Gateway
|
||||
4. **Check LangGraph service** - Verify the availability of relevant endpoints
|
||||
5. **Frontend route smoke check** - Run `bash .agent/skills/smoke-test/scripts/frontend_check.sh` to verify key routes under `/workspace`
|
||||
|
||||
**Docker mode health check** (when using Docker):
|
||||
1. **Check container status** - Run `docker ps` and confirm that all containers are running
|
||||
2. **Check frontend service** - Visit `http://localhost:2026` and verify that the page loads
|
||||
3. **Check API Gateway** - Verify the `http://localhost:2026/health` endpoint
|
||||
4. **Check LangGraph-compatible API** - Verify the `/api/langgraph/*` route exposed by Gateway
|
||||
4. **Check LangGraph service** - Verify the availability of relevant endpoints
|
||||
5. **Frontend route smoke check** - Run `bash .agent/skills/smoke-test/scripts/frontend_check.sh` to verify key routes under `/workspace`
|
||||
|
||||
### Optional Functional Verification
|
||||
@@ -135,7 +135,7 @@ smoke-test/
|
||||
|
||||
The following warnings can appear during smoke testing and do not block a successful result:
|
||||
- Feishu/Lark SSL errors in Gateway logs (certificate verification failure) can be ignored if that channel is not enabled
|
||||
- Warnings in Gateway logs about missing methods in the custom checkpointer, such as `adelete_for_runs` or `aprune`, do not affect the core functionality
|
||||
- Warnings in LangGraph logs about missing methods in the custom checkpointer, such as `adelete_for_runs` or `aprune`, do not affect the core functionality
|
||||
|
||||
## Key Tools
|
||||
|
||||
|
||||
@@ -138,6 +138,7 @@ This document describes the detailed operating steps for each phase of the DeerF
|
||||
lsof -i :2026 # Main port
|
||||
lsof -i :3000 # Frontend
|
||||
lsof -i :8001 # Gateway
|
||||
lsof -i :2024 # LangGraph
|
||||
```
|
||||
|
||||
**Success Criteria**: All ports are free, or they are occupied only by DeerFlow-related processes.
|
||||
@@ -257,7 +258,7 @@ This document describes the detailed operating steps for each phase of the DeerF
|
||||
**Steps**:
|
||||
1. Run `make dev-daemon` (background mode)
|
||||
|
||||
**Description**: This command starts all services (Gateway embedded runtime, Frontend, Nginx).
|
||||
**Description**: This command starts all services (LangGraph, Gateway, Frontend, Nginx).
|
||||
|
||||
**Notes**:
|
||||
- `make dev` runs in the foreground and stops with Ctrl+C
|
||||
@@ -271,6 +272,7 @@ This document describes the detailed operating steps for each phase of the DeerF
|
||||
**Steps**:
|
||||
1. Wait 90-120 seconds for all services to start completely
|
||||
2. You can monitor startup progress by checking these log files:
|
||||
- `logs/langgraph.log`
|
||||
- `logs/gateway.log`
|
||||
- `logs/frontend.log`
|
||||
- `logs/nginx.log`
|
||||
@@ -314,10 +316,11 @@ This document describes the detailed operating steps for each phase of the DeerF
|
||||
**Steps**:
|
||||
1. Run the following command to check processes:
|
||||
```bash
|
||||
ps aux | grep -E "(uvicorn|next|nginx)" | grep -v grep
|
||||
ps aux | grep -E "(langgraph|uvicorn|next|nginx)" | grep -v grep
|
||||
```
|
||||
|
||||
**Success Criteria**: Confirm that the following processes are running:
|
||||
- LangGraph (`langgraph dev`)
|
||||
- Gateway (`uvicorn app.gateway.app:app`)
|
||||
- Frontend (`next dev` or `next start`)
|
||||
- Nginx (`nginx`)
|
||||
@@ -353,11 +356,10 @@ curl http://localhost:2026/health
|
||||
|
||||
---
|
||||
|
||||
#### 5.1.4 Check LangGraph-compatible API
|
||||
#### 5.1.4 Check LangGraph Service
|
||||
|
||||
**Steps**:
|
||||
1. Visit `http://localhost:2026/api/langgraph/assistants/lead_agent` to verify Gateway's LangGraph-compatible API route is reachable.
|
||||
2. A `401` response is acceptable when authentication is enabled and no session cookie is provided.
|
||||
1. Visit relevant LangGraph endpoints to verify availability
|
||||
|
||||
---
|
||||
|
||||
@@ -371,6 +373,7 @@ curl http://localhost:2026/health
|
||||
- `deer-flow-nginx`
|
||||
- `deer-flow-frontend`
|
||||
- `deer-flow-gateway`
|
||||
- `deer-flow-langgraph` (if not in gateway mode)
|
||||
|
||||
---
|
||||
|
||||
@@ -403,11 +406,10 @@ curl http://localhost:2026/health
|
||||
|
||||
---
|
||||
|
||||
#### 5.2.4 Check LangGraph-compatible API
|
||||
#### 5.2.4 Check LangGraph Service
|
||||
|
||||
**Steps**:
|
||||
1. Visit `http://localhost:2026/api/langgraph/assistants/lead_agent` to verify Gateway's LangGraph-compatible API route is reachable.
|
||||
2. A `401` response is acceptable when authentication is enabled and no session cookie is provided.
|
||||
1. Visit relevant LangGraph endpoints to verify availability
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -254,6 +254,7 @@ Processes exit quickly after running `make dev-daemon`.
|
||||
**Solutions**:
|
||||
1. Check log files:
|
||||
```bash
|
||||
tail -f logs/langgraph.log
|
||||
tail -f logs/gateway.log
|
||||
tail -f logs/frontend.log
|
||||
tail -f logs/nginx.log
|
||||
@@ -366,7 +367,24 @@ Errors appear in `gateway.log`.
|
||||
uv sync
|
||||
```
|
||||
|
||||
4. Confirm that the Gateway process is running normally.
|
||||
4. Confirm that the LangGraph service is running normally (if not in gateway mode)
|
||||
|
||||
---
|
||||
|
||||
### Issue: LangGraph Fails to Start
|
||||
|
||||
**Symptoms**:
|
||||
Errors appear in `langgraph.log`.
|
||||
|
||||
**Solutions**:
|
||||
1. Check LangGraph logs:
|
||||
```bash
|
||||
tail -f logs/langgraph.log
|
||||
```
|
||||
|
||||
2. Check config.yaml
|
||||
3. Check whether Python dependencies are complete
|
||||
4. Confirm that port 2024 is not occupied
|
||||
|
||||
---
|
||||
|
||||
@@ -501,7 +519,7 @@ Accessing `/health` returns an error or times out.
|
||||
|
||||
2. Confirm that config.yaml exists and has valid formatting
|
||||
3. Check whether Python dependencies are complete
|
||||
4. Confirm that the Gateway process is running normally.
|
||||
4. Confirm that the LangGraph service is running normally
|
||||
|
||||
**Solutions** (Docker mode):
|
||||
1. Check gateway container logs:
|
||||
@@ -511,7 +529,7 @@ Accessing `/health` returns an error or times out.
|
||||
|
||||
2. Confirm that config.yaml is mounted correctly
|
||||
3. Check whether Python dependencies are complete
|
||||
4. Confirm that the Gateway process is running normally.
|
||||
4. Confirm that the LangGraph service is running normally
|
||||
|
||||
---
|
||||
|
||||
@@ -521,7 +539,7 @@ Accessing `/health` returns an error or times out.
|
||||
|
||||
#### View All Service Processes
|
||||
```bash
|
||||
ps aux | grep -E "(uvicorn|next|nginx)" | grep -v grep
|
||||
ps aux | grep -E "(langgraph|uvicorn|next|nginx)" | grep -v grep
|
||||
```
|
||||
|
||||
#### View Service Logs
|
||||
@@ -530,6 +548,7 @@ ps aux | grep -E "(uvicorn|next|nginx)" | grep -v grep
|
||||
tail -f logs/*.log
|
||||
|
||||
# View specific service logs
|
||||
tail -f logs/langgraph.log
|
||||
tail -f logs/gateway.log
|
||||
tail -f logs/frontend.log
|
||||
tail -f logs/nginx.log
|
||||
|
||||
@@ -65,7 +65,7 @@ if ! command -v lsof >/dev/null 2>&1; then
|
||||
echo " Install lsof and rerun this check"
|
||||
all_passed=false
|
||||
else
|
||||
for port in 2026 3000 8001; do
|
||||
for port in 2026 3000 8001 2024; do
|
||||
if lsof -i :$port >/dev/null 2>&1; then
|
||||
echo "⚠ Port $port is already in use:"
|
||||
lsof -i :$port | head -2
|
||||
|
||||
@@ -54,6 +54,7 @@ echo "=========================================="
|
||||
echo ""
|
||||
echo "🌐 Access URL: http://localhost:2026"
|
||||
echo "📋 View logs:"
|
||||
echo " - logs/langgraph.log"
|
||||
echo " - logs/gateway.log"
|
||||
echo " - logs/frontend.log"
|
||||
echo " - logs/nginx.log"
|
||||
|
||||
@@ -76,11 +76,12 @@ if [ "$mode" = "docker" ]; then
|
||||
all_passed=false
|
||||
fi
|
||||
else
|
||||
summary_hint="logs/{gateway,frontend,nginx}.log"
|
||||
summary_hint="logs/{langgraph,gateway,frontend,nginx}.log"
|
||||
print_step "1. Checking local service ports..."
|
||||
check_listen_port "Nginx" 2026
|
||||
check_listen_port "Frontend" 3000
|
||||
check_listen_port "Gateway" 8001
|
||||
check_listen_port "LangGraph" 2024
|
||||
fi
|
||||
echo ""
|
||||
|
||||
@@ -103,8 +104,8 @@ else
|
||||
fi
|
||||
echo ""
|
||||
|
||||
echo "5. Checking LangGraph-compatible Gateway API..."
|
||||
check_http_status "LangGraph-compatible Gateway API" "http://localhost:2026/api/langgraph/assistants/lead_agent" "200|401"
|
||||
echo "5. Checking LangGraph service..."
|
||||
check_http_status "LangGraph service" "http://localhost:2024/" "200|301|302|307|308|404"
|
||||
echo ""
|
||||
|
||||
echo "=========================================="
|
||||
|
||||
@@ -78,7 +78,7 @@
|
||||
- [x] Container status - {{status_containers}}
|
||||
- [x] Frontend service - {{status_frontend}}
|
||||
- [x] API Gateway - {{status_api_gateway}}
|
||||
- [x] LangGraph-compatible Gateway API - {{status_langgraph}}
|
||||
- [x] LangGraph service - {{status_langgraph}}
|
||||
|
||||
**Phase Status**: {{stage5_status}}
|
||||
|
||||
@@ -147,6 +147,7 @@ Commit Message: {{git_commit_message}}
|
||||
| deer-flow-nginx | {{nginx_status}} | {{nginx_uptime}} |
|
||||
| deer-flow-frontend | {{frontend_status}} | {{frontend_uptime}} |
|
||||
| deer-flow-gateway | {{gateway_status}} | {{gateway_uptime}} |
|
||||
| deer-flow-langgraph | {{langgraph_status}} | {{langgraph_uptime}} |
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -80,7 +80,7 @@
|
||||
- [x] Process status - {{status_processes}}
|
||||
- [x] Frontend service - {{status_frontend}}
|
||||
- [x] API Gateway - {{status_api_gateway}}
|
||||
- [x] LangGraph-compatible Gateway API - {{status_langgraph}}
|
||||
- [x] LangGraph service - {{status_langgraph}}
|
||||
|
||||
**Phase Status**: {{stage5_status}}
|
||||
|
||||
@@ -152,7 +152,7 @@ Commit Message: {{git_commit_message}}
|
||||
| Nginx | {{nginx_status}} | {{nginx_endpoint}} |
|
||||
| Frontend | {{frontend_status}} | {{frontend_endpoint}} |
|
||||
| Gateway | {{gateway_status}} | {{gateway_endpoint}} |
|
||||
| Gateway LangGraph API | {{langgraph_status}} | {{langgraph_endpoint}} |
|
||||
| LangGraph | {{langgraph_status}} | {{langgraph_endpoint}} |
|
||||
|
||||
---
|
||||
|
||||
@@ -166,7 +166,7 @@ Commit Message: {{git_commit_message}}
|
||||
|
||||
### If the Test Fails
|
||||
1. [ ] Review references/troubleshooting.md for common solutions
|
||||
2. [ ] Check local logs: `logs/{gateway,frontend,nginx}.log`
|
||||
2. [ ] Check local logs: `logs/{langgraph,gateway,frontend,nginx}.log`
|
||||
3. [ ] Verify configuration file format and content
|
||||
4. [ ] If needed, fully reset the environment: `make stop && make clean && make install && make dev-daemon`
|
||||
|
||||
|
||||
+2
-33
@@ -1,6 +1,3 @@
|
||||
# Serper API Key (Google Search) - https://serper.dev
|
||||
SERPER_API_KEY=your-serper-api-key
|
||||
|
||||
# TAVILY API Key
|
||||
TAVILY_API_KEY=your-tavily-api-key
|
||||
|
||||
@@ -9,9 +6,8 @@ JINA_API_KEY=your-jina-api-key
|
||||
|
||||
# InfoQuest API Key
|
||||
INFOQUEST_API_KEY=your-infoquest-api-key
|
||||
# Browser CORS allowlist for split-origin or port-forwarded deployments (comma-separated exact origins).
|
||||
# Leave unset when using the unified nginx endpoint, e.g. http://localhost:2026.
|
||||
# GATEWAY_CORS_ORIGINS=http://localhost:3000,http://127.0.0.1:3000
|
||||
# CORS Origins (comma-separated) - e.g., http://localhost:3000,http://localhost:3001
|
||||
# CORS_ORIGINS=http://localhost:3000
|
||||
|
||||
# Optional:
|
||||
# FIRECRAWL_API_KEY=your-firecrawl-api-key
|
||||
@@ -21,7 +17,6 @@ INFOQUEST_API_KEY=your-infoquest-api-key
|
||||
# DEEPSEEK_API_KEY=your-deepseek-api-key
|
||||
# NOVITA_API_KEY=your-novita-api-key # OpenAI-compatible, see https://novita.ai
|
||||
# MINIMAX_API_KEY=your-minimax-api-key # OpenAI-compatible, see https://platform.minimax.io
|
||||
# STEPFUN_API_KEY=your-stepfun-api-key # OpenAI-compatible, see https://platform.stepfun.com
|
||||
# VLLM_API_KEY=your-vllm-api-key # OpenAI-compatible
|
||||
# FEISHU_APP_ID=your-feishu-app-id
|
||||
# FEISHU_APP_SECRET=your-feishu-app-secret
|
||||
@@ -29,7 +24,6 @@ INFOQUEST_API_KEY=your-infoquest-api-key
|
||||
# SLACK_BOT_TOKEN=your-slack-bot-token
|
||||
# SLACK_APP_TOKEN=your-slack-app-token
|
||||
# TELEGRAM_BOT_TOKEN=your-telegram-bot-token
|
||||
# DISCORD_BOT_TOKEN=your-discord-bot-token
|
||||
|
||||
# Enable LangSmith to monitor and debug your LLM calls, agent runs, and tool executions.
|
||||
# LANGSMITH_TRACING=true
|
||||
@@ -39,30 +33,5 @@ INFOQUEST_API_KEY=your-infoquest-api-key
|
||||
|
||||
# GitHub API Token
|
||||
# GITHUB_TOKEN=your-github-token
|
||||
|
||||
# Database (only needed when config.yaml has database.backend: postgres)
|
||||
# DATABASE_URL=postgresql://deerflow:password@localhost:5432/deerflow
|
||||
#
|
||||
# WECOM_BOT_ID=your-wecom-bot-id
|
||||
# WECOM_BOT_SECRET=your-wecom-bot-secret
|
||||
# DINGTALK_CLIENT_ID=your-dingtalk-client-id
|
||||
# DINGTALK_CLIENT_SECRET=your-dingtalk-client-secret
|
||||
|
||||
# Set to "false" to disable Swagger UI, ReDoc, and OpenAPI schema in production
|
||||
# GATEWAY_ENABLE_DOCS=false
|
||||
|
||||
# Shared internal Gateway auth token for multi-worker deployments.
|
||||
# `make up` generates and persists this automatically; set it manually only
|
||||
# when you run Gateway workers outside the bundled deploy script.
|
||||
# DEER_FLOW_INTERNAL_AUTH_TOKEN=your-shared-internal-token
|
||||
|
||||
# ── Frontend SSR → Gateway wiring ─────────────────────────────────────────────
|
||||
# The Next.js server uses these to reach the Gateway during SSR (auth checks,
|
||||
# /api/* rewrites). They default to localhost values that match `make dev` and
|
||||
# `make start`, so most local users do not need to set them.
|
||||
#
|
||||
# Override only when the Gateway is not on localhost:8001 (e.g. when the
|
||||
# frontend and gateway run on different hosts, in containers with a service
|
||||
# alias, or behind a different port). docker-compose already sets these.
|
||||
# DEER_FLOW_INTERNAL_GATEWAY_BASE_URL=http://localhost:8001
|
||||
# DEER_FLOW_TRUSTED_ORIGINS=http://localhost:3000,http://localhost:2026
|
||||
|
||||
@@ -1,159 +0,0 @@
|
||||
name: 🐛 Bug report
|
||||
description: Report something that isn't working so maintainers can reproduce and fix it.
|
||||
title: "[bug] "
|
||||
labels: ["bug"]
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
Thanks for taking the time to file a bug. A clear, reproducible report is the
|
||||
single biggest factor in how fast it gets fixed.
|
||||
|
||||
Please fill in every required field — especially **reproduction steps** and **logs**.
|
||||
|
||||
- type: checkboxes
|
||||
id: preflight
|
||||
attributes:
|
||||
label: Before you start
|
||||
options:
|
||||
- label: I searched [existing issues](https://github.com/bytedance/deer-flow/issues?q=is%3Aissue) and this is not a duplicate.
|
||||
required: true
|
||||
- label: I can reproduce this on the latest `main`.
|
||||
required: false
|
||||
|
||||
- type: input
|
||||
id: summary
|
||||
attributes:
|
||||
label: Problem summary
|
||||
description: One sentence describing the bug.
|
||||
placeholder: e.g. make dev fails to start the gateway service
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: dropdown
|
||||
id: areas
|
||||
attributes:
|
||||
label: Affected area(s)
|
||||
description: Which part of DeerFlow does this touch? Select all that apply.
|
||||
multiple: true
|
||||
options:
|
||||
- Frontend (UI / Next.js)
|
||||
- Backend API (gateway / endpoints / SSE)
|
||||
- Agents / LangGraph (graph, prompts, langgraph.json)
|
||||
- Sandbox / Docker
|
||||
- Skills
|
||||
- MCP
|
||||
- Config / setup (make, config.yaml, env)
|
||||
- Docs
|
||||
- Not sure
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: actual
|
||||
attributes:
|
||||
label: What happened?
|
||||
description: The actual behavior. Include the key error lines verbatim.
|
||||
placeholder: When I do X, I expected Y but I got Z.
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: expected
|
||||
attributes:
|
||||
label: Expected behavior
|
||||
placeholder: What did you expect to happen instead?
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: reproduce
|
||||
attributes:
|
||||
label: Steps to reproduce
|
||||
description: Exact commands and sequence. Minimal steps that reliably reproduce the problem.
|
||||
placeholder: |
|
||||
1. make check
|
||||
2. make install
|
||||
3. make dev
|
||||
4. ...
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: logs
|
||||
attributes:
|
||||
label: Relevant logs
|
||||
description: Paste key lines from logs (for example `logs/gateway.log`, `logs/frontend.log`). Redact secrets.
|
||||
render: shell
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: dropdown
|
||||
id: run_mode
|
||||
attributes:
|
||||
label: How are you running DeerFlow?
|
||||
options:
|
||||
- Local (make dev)
|
||||
- Docker (make docker-start)
|
||||
- CI
|
||||
- Other
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: dropdown
|
||||
id: os
|
||||
attributes:
|
||||
label: Operating system
|
||||
options:
|
||||
- macOS
|
||||
- Linux
|
||||
- Windows
|
||||
- Other
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: input
|
||||
id: platform_details
|
||||
attributes:
|
||||
label: Platform details
|
||||
description: Architecture and shell, if relevant.
|
||||
placeholder: e.g. arm64, zsh
|
||||
|
||||
- type: input
|
||||
id: python_version
|
||||
attributes:
|
||||
label: Python version
|
||||
placeholder: e.g. Python 3.12.9
|
||||
|
||||
- type: input
|
||||
id: node_version
|
||||
attributes:
|
||||
label: Node.js version
|
||||
placeholder: e.g. v22.11.0
|
||||
|
||||
- type: input
|
||||
id: pnpm_version
|
||||
attributes:
|
||||
label: pnpm version
|
||||
placeholder: e.g. 10.26.2
|
||||
|
||||
- type: input
|
||||
id: uv_version
|
||||
attributes:
|
||||
label: uv version
|
||||
placeholder: e.g. 0.7.20
|
||||
|
||||
- type: textarea
|
||||
id: git_info
|
||||
attributes:
|
||||
label: Git state
|
||||
description: Output of `git branch --show-current` and the latest commit SHA.
|
||||
placeholder: |
|
||||
branch: feature/my-branch
|
||||
commit: abcdef1
|
||||
|
||||
- type: textarea
|
||||
id: additional
|
||||
attributes:
|
||||
label: Additional context
|
||||
description: Screenshots, related issues, config snippets (redacted), or anything else that helps triage.
|
||||
@@ -1,11 +0,0 @@
|
||||
blank_issues_enabled: false
|
||||
contact_links:
|
||||
- name: 💬 Questions & usage help
|
||||
url: https://github.com/bytedance/deer-flow/discussions/categories/q-a
|
||||
about: "How do I use X? Why does Y behave like that? Ask in Discussions — it gets answered faster and stays searchable."
|
||||
- name: 💡 Ideas & proposals
|
||||
url: https://github.com/bytedance/deer-flow/discussions/categories/ideas
|
||||
about: Have a half-formed idea? Float it in Discussions before opening a formal feature request.
|
||||
- name: 🔒 Report a security vulnerability
|
||||
url: https://github.com/bytedance/deer-flow/security/policy
|
||||
about: Do not open a public issue for security problems. Follow the security policy instead.
|
||||
@@ -1,67 +0,0 @@
|
||||
name: 💡 Feature request
|
||||
description: Propose a new capability or an improvement to an existing one.
|
||||
title: "[feat] "
|
||||
labels: ["enhancement"]
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
Thanks for the suggestion. For non-trivial features, please open a
|
||||
[Discussion](https://github.com/bytedance/deer-flow/discussions/categories/ideas)
|
||||
first to align on scope before writing code.
|
||||
|
||||
- type: checkboxes
|
||||
id: preflight
|
||||
attributes:
|
||||
label: Before you start
|
||||
options:
|
||||
- label: I searched [existing issues](https://github.com/bytedance/deer-flow/issues?q=is%3Aissue) and this is not a duplicate.
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: problem
|
||||
attributes:
|
||||
label: Problem / motivation
|
||||
description: What problem does this solve? What is painful today, or what does it unblock?
|
||||
placeholder: "I'm always frustrated when ..."
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: solution
|
||||
attributes:
|
||||
label: Proposed solution
|
||||
description: Describe the change from a user's / caller's perspective.
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: dropdown
|
||||
id: areas
|
||||
attributes:
|
||||
label: Affected area(s)
|
||||
description: Which part of DeerFlow would this touch? Select all that apply.
|
||||
multiple: true
|
||||
options:
|
||||
- Frontend (UI / Next.js)
|
||||
- Backend API (gateway / endpoints / SSE)
|
||||
- Agents / LangGraph (graph, prompts, langgraph.json)
|
||||
- Sandbox / Docker
|
||||
- Skills
|
||||
- MCP
|
||||
- Config / setup
|
||||
- Docs
|
||||
- Not sure
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: alternatives
|
||||
attributes:
|
||||
label: Alternatives considered
|
||||
description: Other approaches you weighed and why you discarded them.
|
||||
|
||||
- type: textarea
|
||||
id: additional
|
||||
attributes:
|
||||
label: Additional context
|
||||
description: Mockups, links, related issues, or anything else that helps.
|
||||
@@ -0,0 +1,128 @@
|
||||
name: Runtime Information
|
||||
description: Report runtime/environment details to help reproduce an issue.
|
||||
title: "[runtime] "
|
||||
labels:
|
||||
- needs-triage
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
Thanks for sharing runtime details.
|
||||
Complete this form so maintainers can quickly reproduce and diagnose the problem.
|
||||
|
||||
- type: input
|
||||
id: summary
|
||||
attributes:
|
||||
label: Problem summary
|
||||
description: Short summary of the issue.
|
||||
placeholder: e.g. make dev fails to start gateway service
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: expected
|
||||
attributes:
|
||||
label: Expected behavior
|
||||
placeholder: What did you expect to happen?
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: actual
|
||||
attributes:
|
||||
label: Actual behavior
|
||||
placeholder: What happened instead? Include key error lines.
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: dropdown
|
||||
id: os
|
||||
attributes:
|
||||
label: Operating system
|
||||
options:
|
||||
- macOS
|
||||
- Linux
|
||||
- Windows
|
||||
- Other
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: input
|
||||
id: platform_details
|
||||
attributes:
|
||||
label: Platform details
|
||||
description: Add architecture and shell if relevant.
|
||||
placeholder: e.g. arm64, zsh
|
||||
|
||||
- type: input
|
||||
id: python_version
|
||||
attributes:
|
||||
label: Python version
|
||||
placeholder: e.g. Python 3.12.9
|
||||
|
||||
- type: input
|
||||
id: node_version
|
||||
attributes:
|
||||
label: Node.js version
|
||||
placeholder: e.g. v23.11.0
|
||||
|
||||
- type: input
|
||||
id: pnpm_version
|
||||
attributes:
|
||||
label: pnpm version
|
||||
placeholder: e.g. 10.26.2
|
||||
|
||||
- type: input
|
||||
id: uv_version
|
||||
attributes:
|
||||
label: uv version
|
||||
placeholder: e.g. 0.7.20
|
||||
|
||||
- type: dropdown
|
||||
id: run_mode
|
||||
attributes:
|
||||
label: How are you running DeerFlow?
|
||||
options:
|
||||
- Local (make dev)
|
||||
- Docker (make docker-dev)
|
||||
- CI
|
||||
- Other
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: reproduce
|
||||
attributes:
|
||||
label: Reproduction steps
|
||||
description: Provide exact commands and sequence.
|
||||
placeholder: |
|
||||
1. make check
|
||||
2. make install
|
||||
3. make dev
|
||||
4. ...
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: logs
|
||||
attributes:
|
||||
label: Relevant logs
|
||||
description: Paste key lines from logs (for example logs/gateway.log, logs/frontend.log).
|
||||
render: shell
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: git_info
|
||||
attributes:
|
||||
label: Git state
|
||||
description: Share output of git branch and latest commit SHA.
|
||||
placeholder: |
|
||||
branch: feature/my-branch
|
||||
commit: abcdef1
|
||||
|
||||
- type: textarea
|
||||
id: additional
|
||||
attributes:
|
||||
label: Additional context
|
||||
description: Add anything else that might help triage.
|
||||
@@ -1,119 +0,0 @@
|
||||
# Declarative label source of truth for DeerFlow.
|
||||
#
|
||||
# This file is the single source of truth for repository labels used by the
|
||||
# auto-labeling workflows (.github/workflows/pr-labeler.yml, pr-triage.yml,
|
||||
# issue-triage.yml). Auto-labelers can only apply labels that already exist,
|
||||
# so every label referenced by a workflow MUST be declared here.
|
||||
#
|
||||
# Apply with: uv run --with pyyaml python scripts/sync_labels.py [--repo OWNER/NAME]
|
||||
# CI keeps it in sync via .github/workflows/label-sync.yml (runs on changes here).
|
||||
#
|
||||
# Sync is additive/update-only: it creates or updates the labels listed below
|
||||
# and never deletes labels that are not listed.
|
||||
#
|
||||
# Color = 6-digit hex without the leading '#'.
|
||||
|
||||
labels:
|
||||
# ── Type ─────────────────────────────────────────────────────────────────
|
||||
# Mostly GitHub defaults; declared here so colors/descriptions stay stable
|
||||
# and so issue templates can rely on them existing.
|
||||
- name: bug
|
||||
color: d73a4a
|
||||
description: Something isn't working
|
||||
- name: enhancement
|
||||
color: a2eeef
|
||||
description: New feature or request
|
||||
- name: documentation
|
||||
color: 0075ca
|
||||
description: Improvements or additions to documentation
|
||||
- name: question
|
||||
color: d876e3
|
||||
description: Further information is requested
|
||||
|
||||
# ── Area (auto, by changed paths — see .github/labeler.yml) ───────────────
|
||||
# Mirrors the "Surface area" section of the pull request template.
|
||||
- name: "area:frontend"
|
||||
color: c5def5
|
||||
description: Next.js frontend under frontend/
|
||||
- name: "area:backend"
|
||||
color: c5def5
|
||||
description: Gateway / runtime / core backend under backend/
|
||||
- name: "area:agents"
|
||||
color: c5def5
|
||||
description: Agents, subagents, graph wiring, prompts, langgraph.json
|
||||
- name: "area:sandbox"
|
||||
color: c5def5
|
||||
description: Sandboxed execution and docker/
|
||||
- name: "area:skills"
|
||||
color: c5def5
|
||||
description: Skills under skills/ or the skills harness
|
||||
- name: "area:mcp"
|
||||
color: c5def5
|
||||
description: Model Context Protocol integration
|
||||
- name: "area:ci"
|
||||
color: c5def5
|
||||
description: GitHub Actions, CI config, repo tooling
|
||||
- name: "area:docs"
|
||||
color: c5def5
|
||||
description: Documentation and Markdown only
|
||||
- name: "area:deps"
|
||||
color: c5def5
|
||||
description: Dependency manifests / lockfiles
|
||||
|
||||
# ── Size (auto, by additions + deletions — see pr-triage.yml) ─────────────
|
||||
- name: "size/XS"
|
||||
color: "009900"
|
||||
description: PR changes < 20 lines
|
||||
- name: "size/S"
|
||||
color: 77bb00
|
||||
description: PR changes 20-100 lines
|
||||
- name: "size/M"
|
||||
color: eebb00
|
||||
description: PR changes 100-300 lines
|
||||
- name: "size/L"
|
||||
color: ee9900
|
||||
description: PR changes 300-700 lines
|
||||
- name: "size/XL"
|
||||
color: ee5500
|
||||
description: PR changes 700+ lines
|
||||
|
||||
# ── Risk (auto, by changed paths — see pr-triage.yml) ─────────────────────
|
||||
- name: "risk:low"
|
||||
color: 0e8a16
|
||||
description: "Low risk: docs / i18n / assets only"
|
||||
- name: "risk:medium"
|
||||
color: fbca04
|
||||
description: "Medium risk: regular code changes"
|
||||
- name: "risk:high"
|
||||
color: b60205
|
||||
description: "High risk: backend API, agents, sandbox, auth, deps, CI"
|
||||
|
||||
# ── Priority (manual) ─────────────────────────────────────────────────────
|
||||
- name: P0
|
||||
color: b60205
|
||||
description: Critical priority
|
||||
- name: P1
|
||||
color: d93f0b
|
||||
description: Major priority
|
||||
- name: P2
|
||||
color: e99695
|
||||
description: Normal priority
|
||||
|
||||
# ── Status (auto + manual) ────────────────────────────────────────────────
|
||||
- name: needs-triage
|
||||
color: fef2c0
|
||||
description: Awaiting maintainer triage
|
||||
- name: needs-validation
|
||||
color: d4c5f9
|
||||
description: Touches front/back contract surface; needs real-path validation
|
||||
- name: skip-validation
|
||||
color: cccccc
|
||||
description: "Maintainer override: do not auto-add needs-validation on this PR"
|
||||
- name: reviewing
|
||||
color: 5319e7
|
||||
description: A maintainer is reviewing this PR
|
||||
|
||||
# ── Contributor ───────────────────────────────────────────────────────────
|
||||
- name: first-time-contributor
|
||||
color: c2e0c6
|
||||
description: First contribution to this repository — be welcoming
|
||||
@@ -1,75 +0,0 @@
|
||||
<!-- Reference a related issue with #123. Use Fixes / Closes / Resolves to
|
||||
auto-close it on merge. Delete this line if the PR doesn't reference an issue. -->
|
||||
Fixes #
|
||||
|
||||
## Why
|
||||
|
||||
<!-- Why are you opening this PR? Cover two things:
|
||||
- The trigger — what made you write this? A bug you hit, a feature you need,
|
||||
tech debt, or a prod issue?
|
||||
- The pain being addressed — user-facing problem, or what it unblocks.
|
||||
For non-trivial features, please open an issue/discussion first to align on
|
||||
scope before writing code. -->
|
||||
|
||||
|
||||
## What changed
|
||||
|
||||
<!-- Describe the change from a user's / caller's perspective, not as a code diff. e.g.:
|
||||
- "Settings now has a 'Custom endpoint' field, off by default"
|
||||
- "Backend /api/chat gains a `stream` flag, defaults to false"
|
||||
- "Default model changed from X to Y — existing users notice on first run" -->
|
||||
|
||||
|
||||
## Surface area
|
||||
|
||||
<!-- Check every box that applies. Reviewers use this to scope the review. -->
|
||||
|
||||
- [ ] **Frontend UI** — page / component / setting / interaction under `frontend/`
|
||||
- [ ] **Backend API** — endpoint / SSE event / request-response shape under `backend/app`
|
||||
- [ ] **Agents / LangGraph** — agent node, graph wiring, `langgraph.json`, or prompt change
|
||||
- [ ] **Sandbox** — `docker/` or sandboxed execution
|
||||
- [ ] **Skills** — change under `skills/`
|
||||
- [ ] **Dependencies** — new/upgraded entry in `backend/pyproject.toml` or `frontend/package.json` (say what it buys us)
|
||||
- [ ] **Default behavior change** — changes existing behavior without the user opting in (default model, default setting, data shape)
|
||||
- [ ] **Docs / tests / CI only** — no runtime behavior change
|
||||
|
||||
|
||||
## Screenshots / Recording
|
||||
|
||||
<!-- If you checked "Frontend UI", attach screenshots showing the entry point —
|
||||
where users discover the change — not just the feature in isolation.
|
||||
Before/after is best for behavior changes. Short GIFs welcome. -->
|
||||
|
||||
|
||||
## Bug fix verification
|
||||
|
||||
<!-- Skip (delete) this section if this PR is not a bug fix.
|
||||
|
||||
Bugs should be encoded as a failing test that goes red before the fix.
|
||||
Confirm:
|
||||
- Test path that reproduces the bug:
|
||||
- Did it go red on `main` and green on this branch? (yes / no)
|
||||
- If a red test wasn't cheap to write, explain why and what you did instead. -->
|
||||
|
||||
|
||||
## Validation
|
||||
|
||||
<!-- What you actually ran. Run at least the checks for the area you changed:
|
||||
Backend: cd backend && make lint && make test
|
||||
Frontend: cd frontend && pnpm format && pnpm lint && pnpm typecheck && BETTER_AUTH_SECRET=local-dev-secret pnpm build && make test
|
||||
Frontend E2E (if you touched frontend/): cd frontend && make test-e2e -->
|
||||
|
||||
|
||||
## AI assistance
|
||||
|
||||
<!-- DeerFlow is an AI project — most PRs here use AI coding tools, and that's
|
||||
welcome. Disclosing it just helps reviewers calibrate how closely to read the
|
||||
diff. Please fill all three; don't delete the section. -->
|
||||
|
||||
**Tool(s) used:** <!-- e.g. Claude Code, Cursor, GitHub Copilot, Codex, Windsurf, or "none" -->
|
||||
|
||||
**How you used it:** <!-- e.g. "generated the module from a spec", "autocomplete only",
|
||||
"AI wrote tests, I wrote the impl". A prompt or conversation link is great too. -->
|
||||
|
||||
- [ ] I've read and understand every line of this change and take responsibility for it — it's not unreviewed AI output.
|
||||
|
||||
@@ -1,46 +0,0 @@
|
||||
name: Backend Blocking IO
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: ["main"]
|
||||
paths:
|
||||
- "backend/**"
|
||||
- ".github/workflows/backend-blocking-io-tests.yml"
|
||||
pull_request:
|
||||
types: [opened, synchronize, reopened, ready_for_review]
|
||||
paths:
|
||||
- "backend/**"
|
||||
- ".github/workflows/backend-blocking-io-tests.yml"
|
||||
|
||||
concurrency:
|
||||
group: blocking-io-${{ github.event.pull_request.number || github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
backend-blocking-io:
|
||||
if: github.event_name != 'pull_request' || github.event.pull_request.draft == false
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.12"
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v3
|
||||
|
||||
- name: Install backend dependencies
|
||||
working-directory: backend
|
||||
run: uv sync --group dev
|
||||
|
||||
- name: Run blocking IO regression tests
|
||||
working-directory: backend
|
||||
run: make test-blocking-io
|
||||
@@ -1,101 +0,0 @@
|
||||
name: Publish Containers
|
||||
|
||||
on:
|
||||
push:
|
||||
tags:
|
||||
- "v*"
|
||||
|
||||
jobs:
|
||||
|
||||
backend-container:
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
attestations: write
|
||||
id-token: write
|
||||
env:
|
||||
REGISTRY: ghcr.io
|
||||
IMAGE_NAME: ${{ github.repository }}-backend
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v6
|
||||
- name: Log in to the Container registry
|
||||
uses: docker/login-action@74a5d142397b4f367a81961eba4e8cd7edddf772 #v3.4.0
|
||||
with:
|
||||
registry: ${{ env.REGISTRY }}
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
- name: Extract metadata (tags, labels) for Docker
|
||||
id: meta
|
||||
uses: docker/metadata-action@902fa8ec7d6ecbf8d84d538b9b233a880e428804 #v5.7.0
|
||||
with:
|
||||
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
|
||||
tags: |
|
||||
type=ref,event=tag
|
||||
type=ref,event=branch
|
||||
type=sha
|
||||
type=raw,value=latest,enable={{is_default_branch}}
|
||||
- name: Build and push Docker image
|
||||
id: push
|
||||
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 #v6.18.0
|
||||
with:
|
||||
context: .
|
||||
file: backend/Dockerfile
|
||||
push: true
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
|
||||
- name: Generate artifact attestation
|
||||
uses: actions/attest-build-provenance@v2
|
||||
with:
|
||||
subject-name: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME}}
|
||||
subject-digest: ${{ steps.push.outputs.digest }}
|
||||
push-to-registry: true
|
||||
|
||||
frontend-container:
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
attestations: write
|
||||
id-token: write
|
||||
env:
|
||||
REGISTRY: ghcr.io
|
||||
IMAGE_NAME: ${{ github.repository }}-frontend
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v6
|
||||
- name: Log in to the Container registry
|
||||
uses: docker/login-action@74a5d142397b4f367a81961eba4e8cd7edddf772 #v3.4.0
|
||||
with:
|
||||
registry: ${{ env.REGISTRY }}
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
- name: Extract metadata (tags, labels) for Docker
|
||||
id: meta
|
||||
uses: docker/metadata-action@902fa8ec7d6ecbf8d84d538b9b233a880e428804 #v5.7.0
|
||||
with:
|
||||
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
|
||||
tags: |
|
||||
type=ref,event=tag
|
||||
type=ref,event=branch
|
||||
type=sha
|
||||
type=raw,value=latest,enable={{is_default_branch}}
|
||||
- name: Build and push Docker image
|
||||
id: push
|
||||
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 #v6.18.0
|
||||
with:
|
||||
context: .
|
||||
file: frontend/Dockerfile
|
||||
push: true
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
|
||||
- name: Generate artifact attestation
|
||||
uses: actions/attest-build-provenance@v2
|
||||
with:
|
||||
subject-name: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME}}
|
||||
subject-digest: ${{ steps.push.outputs.digest }}
|
||||
push-to-registry: true
|
||||
@@ -1,63 +0,0 @@
|
||||
name: E2E Tests
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [ 'main' ]
|
||||
paths:
|
||||
- 'frontend/**'
|
||||
- '.github/workflows/e2e-tests.yml'
|
||||
pull_request:
|
||||
types: [opened, synchronize, reopened, ready_for_review]
|
||||
paths:
|
||||
- 'frontend/**'
|
||||
- '.github/workflows/e2e-tests.yml'
|
||||
|
||||
concurrency:
|
||||
group: e2e-tests-${{ github.event.pull_request.number || github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
e2e-tests:
|
||||
if: ${{ github.event_name != 'pull_request' || github.event.pull_request.draft == false }}
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '22'
|
||||
|
||||
- name: Enable Corepack
|
||||
run: corepack enable
|
||||
|
||||
- name: Use pinned pnpm version
|
||||
run: corepack prepare pnpm@10.26.2 --activate
|
||||
|
||||
- name: Install frontend dependencies
|
||||
working-directory: frontend
|
||||
run: pnpm install --frozen-lockfile
|
||||
|
||||
- name: Install Playwright Chromium
|
||||
working-directory: frontend
|
||||
run: npx playwright install chromium --with-deps
|
||||
|
||||
- name: Run E2E tests
|
||||
working-directory: frontend
|
||||
run: pnpm exec playwright test
|
||||
env:
|
||||
SKIP_ENV_VALIDATION: '1'
|
||||
|
||||
- name: Upload Playwright report
|
||||
uses: actions/upload-artifact@v4
|
||||
if: ${{ !cancelled() }}
|
||||
with:
|
||||
name: playwright-report
|
||||
path: frontend/playwright-report/
|
||||
retention-days: 7
|
||||
@@ -1,43 +0,0 @@
|
||||
name: Frontend Unit Tests
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [ 'main' ]
|
||||
pull_request:
|
||||
types: [opened, synchronize, reopened, ready_for_review]
|
||||
|
||||
concurrency:
|
||||
group: frontend-unit-tests-${{ github.event.pull_request.number || github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
frontend-unit-tests:
|
||||
if: github.event.pull_request.draft == false
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '22'
|
||||
|
||||
- name: Enable Corepack
|
||||
run: corepack enable
|
||||
|
||||
- name: Use pinned pnpm version
|
||||
run: corepack prepare pnpm@10.26.2 --activate
|
||||
|
||||
- name: Install frontend dependencies
|
||||
working-directory: frontend
|
||||
run: pnpm install --frozen-lockfile
|
||||
|
||||
- name: Run unit tests of frontend
|
||||
working-directory: frontend
|
||||
run: make test
|
||||
@@ -1,38 +0,0 @@
|
||||
name: Label Sync
|
||||
|
||||
# Keeps repository labels in sync with the declarative source of truth
|
||||
# (.github/labels.yml). Runs whenever that file changes on main, and can be
|
||||
# triggered manually. Additive/update-only — never deletes labels.
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
- ".github/labels.yml"
|
||||
- "scripts/sync_labels.py"
|
||||
- ".github/workflows/label-sync.yml"
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
|
||||
concurrency:
|
||||
group: label-sync
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
sync:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v7
|
||||
|
||||
- name: Sync labels
|
||||
run: uv run --with pyyaml python scripts/sync_labels.py
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
GH_REPO: ${{ github.repository }}
|
||||
@@ -1,108 +0,0 @@
|
||||
name: Replay E2E (front-back contract)
|
||||
|
||||
# Guards the front-back contract via record/replay (no API key in CI):
|
||||
# Layer 1 — backend golden: replay a recorded trace through the real gateway,
|
||||
# assert the SSE event sequence matches the committed golden.
|
||||
# Layer 2 — full-stack render: real Next.js frontend + real gateway (replay
|
||||
# model) + Chromium; assert the replayed turns render in the browser.
|
||||
# Triggered by changes on EITHER side of the contract so a backend change can no
|
||||
# longer pass without the frontend-facing checks running.
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: ["main"]
|
||||
paths:
|
||||
- "frontend/**"
|
||||
- "backend/app/gateway/**"
|
||||
- "backend/packages/harness/**"
|
||||
- "backend/tests/fixtures/replay/**"
|
||||
- "backend/tests/replay_provider.py"
|
||||
- "backend/tests/_replay_fixture.py"
|
||||
- "backend/tests/seed_runs_router.py"
|
||||
- "backend/tests/test_replay_golden.py"
|
||||
- "backend/scripts/run_replay_gateway.py"
|
||||
- ".github/workflows/replay-e2e.yml"
|
||||
pull_request:
|
||||
types: [opened, synchronize, reopened, ready_for_review]
|
||||
paths:
|
||||
- "frontend/**"
|
||||
- "backend/app/gateway/**"
|
||||
- "backend/packages/harness/**"
|
||||
- "backend/tests/fixtures/replay/**"
|
||||
- "backend/tests/replay_provider.py"
|
||||
- "backend/tests/_replay_fixture.py"
|
||||
- "backend/tests/seed_runs_router.py"
|
||||
- "backend/tests/test_replay_golden.py"
|
||||
- "backend/scripts/run_replay_gateway.py"
|
||||
- ".github/workflows/replay-e2e.yml"
|
||||
|
||||
concurrency:
|
||||
group: replay-e2e-${{ github.event.pull_request.number || github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
backend-replay-golden:
|
||||
name: Layer 1 — backend golden (no API key)
|
||||
if: github.event_name != 'pull_request' || github.event.pull_request.draft == false
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.12"
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v7
|
||||
- name: Install backend dependencies
|
||||
working-directory: backend
|
||||
run: uv sync --group dev
|
||||
- name: Replay golden (backend SSE contract)
|
||||
working-directory: backend
|
||||
run: PYTHONPATH=. uv run pytest tests/test_replay_golden.py -v
|
||||
|
||||
fullstack-replay-render:
|
||||
name: Layer 2 — full-stack render (no API key)
|
||||
if: github.event_name != 'pull_request' || github.event.pull_request.draft == false
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 25
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.12"
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v7
|
||||
- name: Install backend dependencies (replay gateway)
|
||||
working-directory: backend
|
||||
run: uv sync --group dev
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: "22"
|
||||
- name: Enable Corepack
|
||||
run: corepack enable
|
||||
- name: Use pinned pnpm version
|
||||
run: corepack prepare pnpm@10.26.2 --activate
|
||||
- name: Install frontend dependencies
|
||||
working-directory: frontend
|
||||
run: pnpm install --frozen-lockfile
|
||||
- name: Install Playwright Chromium
|
||||
working-directory: frontend
|
||||
run: npx playwright install chromium --with-deps
|
||||
- name: Full-stack replay render (DOM assertions are the gate)
|
||||
working-directory: frontend
|
||||
run: pnpm exec playwright test -c playwright.real-backend.config.ts
|
||||
- name: Upload report + render artifact
|
||||
uses: actions/upload-artifact@v4
|
||||
if: ${{ !cancelled() }}
|
||||
with:
|
||||
name: replay-render
|
||||
path: |
|
||||
frontend/playwright-report/
|
||||
frontend/test-results/
|
||||
retention-days: 7
|
||||
@@ -1,223 +0,0 @@
|
||||
name: Triage
|
||||
|
||||
# One workflow for all event-driven PR/issue labeling. Replaces the former
|
||||
# pr-labeler / pr-triage / issue-triage workflows (and drops actions/labeler).
|
||||
#
|
||||
# Design notes:
|
||||
# * All jobs are pure-metadata: they read changed-file lists / PR fields / the
|
||||
# review payload via the API and write labels. PR code is NEVER checked out
|
||||
# or executed, so pull_request_target is safe here.
|
||||
# * Each job only reconciles labels in namespaces IT owns
|
||||
# (area:* / size/* / risk:* / needs-validation). It never touches labels
|
||||
# applied by maintainers or other tools (bug, priority, etc.). first-time-
|
||||
# contributor and reviewing are add-only.
|
||||
# * State is read LIVE (listFiles + listLabelsOnIssue) at run time, not from
|
||||
# the (stale) event payload, so rapid synchronize events converge instead
|
||||
# of thrashing.
|
||||
|
||||
on:
|
||||
pull_request_target:
|
||||
types: [opened, synchronize, reopened, ready_for_review]
|
||||
pull_request_review:
|
||||
types: [submitted]
|
||||
issues:
|
||||
types: [opened]
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: write
|
||||
issues: write
|
||||
|
||||
jobs:
|
||||
# ── PR: area / size / risk / needs-validation / first-time ─────────────────
|
||||
pr-labels:
|
||||
if: github.event_name == 'pull_request_target' && github.event.pull_request.draft == false
|
||||
runs-on: ubuntu-latest
|
||||
concurrency:
|
||||
group: triage-pr-${{ github.event.pull_request.number }}
|
||||
cancel-in-progress: true
|
||||
steps:
|
||||
- name: Apply PR labels from live state
|
||||
uses: actions/github-script@v8
|
||||
with:
|
||||
script: |
|
||||
const pr = context.payload.pull_request;
|
||||
const { owner, repo } = context.repo;
|
||||
const num = pr.number;
|
||||
|
||||
// ---- live changed files ----
|
||||
const files = await github.paginate(github.rest.pulls.listFiles, {
|
||||
owner, repo, pull_number: num, per_page: 100,
|
||||
});
|
||||
const paths = files.map(f => f.filename);
|
||||
const m = (re) => paths.some(p => re.test(p));
|
||||
|
||||
// ---- area: replaces .github/labeler.yml (path -> area) ----
|
||||
const AREA_RULES = [
|
||||
['area:frontend', [/^frontend\//]],
|
||||
['area:backend', [/^backend\/app\//, /^backend\/packages\/harness\/deerflow\/(runtime|persistence|config|tools|guardrails|tracing|models|utils|uploads)\//]],
|
||||
['area:agents', [/^backend\/packages\/harness\/deerflow\/(agents|subagents|reflection)\//, /(^|\/)langgraph\.json$/, /^backend\/.*\/prompts\//]],
|
||||
['area:sandbox', [/^docker\//, /^backend\/packages\/harness\/deerflow\/sandbox\//, /(^|\/)Dockerfile$/]],
|
||||
['area:skills', [/^skills\//, /^backend\/packages\/harness\/deerflow\/skills\//, /^frontend\/src\/core\/skills\//]],
|
||||
['area:mcp', [/^backend\/packages\/harness\/deerflow\/mcp\//, /^frontend\/src\/core\/mcp\//]],
|
||||
['area:ci', [/^\.github\//, /^scripts\//]],
|
||||
['area:docs', [/^docs\//, /\.mdx?$/]],
|
||||
['area:deps', [/(^|\/)(pyproject\.toml|uv\.lock|package\.json|pnpm-lock\.yaml)$/]],
|
||||
];
|
||||
const areaLabels = AREA_RULES
|
||||
.filter(([, res]) => res.some(re => m(re)))
|
||||
.map(([label]) => label);
|
||||
|
||||
// ---- size: additions+deletions, excluding lockfiles/snapshots ----
|
||||
const EXCLUDE_SIZE = /(^|\/)(uv\.lock|pnpm-lock\.yaml|package-lock\.json)$|\.snap$/;
|
||||
const churn = files
|
||||
.filter(f => !EXCLUDE_SIZE.test(f.filename))
|
||||
.reduce((s, f) => s + (f.additions || 0) + (f.deletions || 0), 0);
|
||||
const sizeLabel =
|
||||
churn < 20 ? 'size/XS' :
|
||||
churn < 100 ? 'size/S' :
|
||||
churn < 300 ? 'size/M' :
|
||||
churn < 700 ? 'size/L' : 'size/XL';
|
||||
|
||||
// ---- risk ----
|
||||
const docsOnly = paths.length > 0 && paths.every(p =>
|
||||
/\.(md|mdx|txt)$/i.test(p) || p.startsWith('docs/') ||
|
||||
/\.(png|jpe?g|gif|svg|webp|ico)$/i.test(p));
|
||||
const highRisk =
|
||||
m(/^backend\/app\/gateway\//) ||
|
||||
m(/^backend\/packages\/harness\/deerflow\/(agents|subagents|sandbox)\//) ||
|
||||
m(/(^|\/)langgraph\.json$/) ||
|
||||
m(/(^|\/)(auth|authz|security)/i) ||
|
||||
m(/(pyproject\.toml|uv\.lock|package\.json|pnpm-lock\.yaml)$/) ||
|
||||
m(/^docker\//) ||
|
||||
m(/^\.github\/workflows\//);
|
||||
const riskLabel = docsOnly ? 'risk:low' : (highRisk ? 'risk:high' : 'risk:medium');
|
||||
|
||||
// ---- needs-validation: front/back contract surface ----
|
||||
const contract =
|
||||
m(/^backend\/app\/gateway\//) ||
|
||||
m(/^backend\/packages\/harness\/deerflow\/(agents|subagents)\//) ||
|
||||
m(/(^|\/)langgraph\.json$/) ||
|
||||
m(/^frontend\/src\/core\/(api|threads|messages)\//);
|
||||
|
||||
// ---- live current labels (NOT the stale event payload) ----
|
||||
const current = (await github.paginate(github.rest.issues.listLabelsOnIssue, {
|
||||
owner, repo, issue_number: num, per_page: 100,
|
||||
})).map(l => l.name);
|
||||
const hasSkip = current.includes('skip-validation');
|
||||
|
||||
// Reconcile ONLY namespaces we own; never touch others.
|
||||
const owned = (n) =>
|
||||
n.startsWith('area:') || n.startsWith('size/') ||
|
||||
n.startsWith('risk:') || n === 'needs-validation';
|
||||
const desired = new Set([...areaLabels, sizeLabel, riskLabel]);
|
||||
if (contract && !hasSkip) desired.add('needs-validation');
|
||||
|
||||
const toRemove = current.filter(n => owned(n) && !desired.has(n));
|
||||
const toAdd = [...desired].filter(n => !current.includes(n));
|
||||
|
||||
// first-time-contributor: add-only, on opened, real users only.
|
||||
if (context.payload.action === 'opened' &&
|
||||
pr.user.type === 'User' &&
|
||||
['FIRST_TIME_CONTRIBUTOR', 'FIRST_TIMER'].includes(pr.author_association) &&
|
||||
!current.includes('first-time-contributor')) {
|
||||
toAdd.push('first-time-contributor');
|
||||
}
|
||||
|
||||
for (const name of toRemove) {
|
||||
try {
|
||||
await github.rest.issues.removeLabel({ owner, repo, issue_number: num, name });
|
||||
} catch (e) {
|
||||
if (e.status !== 404) throw e;
|
||||
}
|
||||
}
|
||||
if (toAdd.length) {
|
||||
await github.rest.issues.addLabels({ owner, repo, issue_number: num, labels: toAdd });
|
||||
}
|
||||
core.info(`area=[${areaLabels.join(',')}] ${sizeLabel} ${riskLabel} churn=${churn} ` +
|
||||
`validation=${desired.has('needs-validation')} ` +
|
||||
`(+${toAdd.join(',') || '-'} / -${toRemove.join(',') || '-'})`);
|
||||
|
||||
# ── PR: reviewing label on a maintainer's human review ─────────────────────
|
||||
reviewing:
|
||||
if: github.event_name == 'pull_request_review'
|
||||
runs-on: ubuntu-latest
|
||||
concurrency:
|
||||
group: triage-review-${{ github.event.pull_request.number }}
|
||||
cancel-in-progress: false
|
||||
steps:
|
||||
- name: Add reviewing label for maintainer reviews
|
||||
uses: actions/github-script@v8
|
||||
with:
|
||||
script: |
|
||||
const { owner, repo } = context.repo;
|
||||
const num = context.payload.pull_request.number;
|
||||
const review = context.payload.review;
|
||||
const assoc = review.author_association; // payload field; no API call
|
||||
const type = review.user && review.user.type;
|
||||
|
||||
// author_association is NONE for every automated reviewer
|
||||
// (Copilot, CodeRabbit, Codex, Sourcery, ...), so this allowlist
|
||||
// drops them all without a denylist — and never calls the
|
||||
// collaborators API that 404s on "Copilot is not a user".
|
||||
// user.type === 'User' guards the rare bot-added-as-collaborator case.
|
||||
if (!['OWNER', 'MEMBER', 'COLLABORATOR'].includes(assoc) || type !== 'User') {
|
||||
core.info(`reviewer ${review.user && review.user.login} assoc=${assoc} type=${type}; skipping.`);
|
||||
return;
|
||||
}
|
||||
|
||||
const labels = (await github.paginate(github.rest.issues.listLabelsOnIssue, {
|
||||
owner, repo, issue_number: num, per_page: 100,
|
||||
})).map(l => l.name);
|
||||
if (labels.includes('reviewing')) {
|
||||
core.info('Already labeled reviewing; skipping.');
|
||||
return;
|
||||
}
|
||||
try {
|
||||
await github.rest.issues.addLabels({
|
||||
owner, repo, issue_number: num, labels: ['reviewing'],
|
||||
});
|
||||
core.info('Added "reviewing".');
|
||||
} catch (e) {
|
||||
if (e.status === 403) core.info('No permission to label (expected on some fork PRs).');
|
||||
else throw e;
|
||||
}
|
||||
|
||||
# ── Issue: needs-triage on every new issue ────────────────────────────────
|
||||
issue-triage:
|
||||
if: github.event_name == 'issues'
|
||||
runs-on: ubuntu-latest
|
||||
concurrency:
|
||||
group: triage-issue-${{ github.event.issue.number }}
|
||||
cancel-in-progress: false
|
||||
steps:
|
||||
- name: Add needs-triage label
|
||||
uses: actions/github-script@v8
|
||||
with:
|
||||
script: |
|
||||
const { owner, repo } = context.repo;
|
||||
const issue_number = context.payload.issue.number;
|
||||
|
||||
// Read live labels (not the event payload) so labels added at creation
|
||||
// time via the API or by another automation are seen — consistent with
|
||||
// the live-state reads in the PR jobs above.
|
||||
const current = (await github.paginate(github.rest.issues.listLabelsOnIssue, {
|
||||
owner, repo, issue_number, per_page: 100,
|
||||
})).map(l => l.name);
|
||||
if (current.includes('needs-triage')) {
|
||||
core.info('Issue already has needs-triage; nothing to do.');
|
||||
return;
|
||||
}
|
||||
// Self-heal: create the label if it does not exist yet.
|
||||
try {
|
||||
await github.rest.issues.createLabel({
|
||||
owner, repo, name: 'needs-triage', color: 'fef2c0',
|
||||
description: 'Awaiting maintainer triage',
|
||||
});
|
||||
} catch (e) {
|
||||
if (e.status !== 422) throw e; // 422 = already exists
|
||||
}
|
||||
await github.rest.issues.addLabels({
|
||||
owner, repo, issue_number, labels: ['needs-triage'],
|
||||
});
|
||||
core.info(`Added needs-triage to #${issue_number}.`);
|
||||
@@ -40,7 +40,6 @@ coverage/
|
||||
skills/custom/*
|
||||
logs/
|
||||
log/
|
||||
debug.log
|
||||
|
||||
# Local git hooks (keep only on this machine, do not push)
|
||||
.githooks/
|
||||
@@ -56,7 +55,5 @@ web/
|
||||
backend/Dockerfile.langgraph
|
||||
config.yaml.bak
|
||||
.playwright-mcp
|
||||
/frontend/test-results/
|
||||
/frontend/playwright-report/
|
||||
.gstack/
|
||||
.worktrees
|
||||
|
||||
@@ -1,33 +0,0 @@
|
||||
repos:
|
||||
# Backend: ruff lint + format via uv (uses the same ruff version as backend deps)
|
||||
- repo: local
|
||||
hooks:
|
||||
- id: ruff
|
||||
name: ruff lint
|
||||
entry: bash -c 'cd backend && uv run ruff check --fix "${@/#backend\//}"' --
|
||||
language: system
|
||||
types_or: [python]
|
||||
files: ^backend/
|
||||
- id: ruff-format
|
||||
name: ruff format
|
||||
entry: bash -c 'cd backend && uv run ruff format "${@/#backend\//}"' --
|
||||
language: system
|
||||
types_or: [python]
|
||||
files: ^backend/
|
||||
|
||||
# Frontend: eslint + prettier (must run from frontend/ for node_modules resolution)
|
||||
- repo: local
|
||||
hooks:
|
||||
- id: frontend-eslint
|
||||
name: eslint (frontend)
|
||||
entry: bash -c 'cd frontend && npx eslint --fix "${@/#frontend\//}"' --
|
||||
language: system
|
||||
types_or: [javascript, tsx, ts]
|
||||
files: ^frontend/
|
||||
|
||||
- id: frontend-prettier
|
||||
name: prettier (frontend)
|
||||
entry: bash -c 'cd frontend && npx prettier --write "${@/#frontend\//}"' --
|
||||
language: system
|
||||
files: ^frontend/
|
||||
types_or: [javascript, tsx, ts, json, css]
|
||||
+25
-39
@@ -46,12 +46,12 @@ Docker provides a consistent, isolated environment with all dependencies pre-con
|
||||
All services will start with hot-reload enabled:
|
||||
- Frontend changes are automatically reloaded
|
||||
- Backend changes trigger automatic restart
|
||||
- Gateway-hosted LangGraph-compatible runtime supports hot-reload
|
||||
- LangGraph server supports hot-reload
|
||||
|
||||
4. **Access the application**:
|
||||
- Web Interface: http://localhost:2026
|
||||
- API Gateway: http://localhost:2026/api/*
|
||||
- LangGraph-compatible API: http://localhost:2026/api/langgraph/*
|
||||
- LangGraph: http://localhost:2026/api/langgraph/*
|
||||
|
||||
#### Docker Commands
|
||||
|
||||
@@ -94,7 +94,7 @@ Use these as practical starting points for development and review environments:
|
||||
If `make docker-init`, `make docker-start`, or `make docker-stop` fails on Linux with an error like below, your current user likely does not have permission to access the Docker daemon socket:
|
||||
|
||||
```text
|
||||
unable to get image 'deer-flow-gateway': permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock
|
||||
unable to get image 'deer-flow-dev-langgraph': permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock
|
||||
```
|
||||
|
||||
Recommended fix: add your current user to the `docker` group so Docker commands work without `sudo`.
|
||||
@@ -131,7 +131,8 @@ Host Machine
|
||||
Docker Compose (deer-flow-dev)
|
||||
├→ nginx (port 2026) ← Reverse proxy
|
||||
├→ web (port 3000) ← Frontend with hot-reload
|
||||
├→ gateway (port 8001) ← Gateway API + LangGraph-compatible runtime with hot-reload
|
||||
├→ api (port 8001) ← Gateway API with hot-reload
|
||||
├→ langgraph (port 2024) ← LangGraph server with hot-reload
|
||||
└→ provisioner (optional, port 8002) ← Started only in provisioner/K8s sandbox mode
|
||||
```
|
||||
|
||||
@@ -165,7 +166,7 @@ Required tools:
|
||||
|
||||
1. **Configure the application** (same as Docker setup above)
|
||||
|
||||
2. **Install dependencies** (this also sets up pre-commit hooks):
|
||||
2. **Install dependencies**:
|
||||
```bash
|
||||
make install
|
||||
```
|
||||
@@ -183,13 +184,17 @@ Required tools:
|
||||
|
||||
If you need to start services individually:
|
||||
|
||||
1. **Start backend service**:
|
||||
1. **Start backend services**:
|
||||
```bash
|
||||
# Terminal 1: Start Gateway API + embedded agent runtime (port 8001)
|
||||
# Terminal 1: Start LangGraph Server (port 2024)
|
||||
cd backend
|
||||
make dev
|
||||
|
||||
# Terminal 2: Start Frontend (port 3000)
|
||||
# Terminal 2: Start Gateway API (port 8001)
|
||||
cd backend
|
||||
make gateway
|
||||
|
||||
# Terminal 3: Start Frontend (port 3000)
|
||||
cd frontend
|
||||
pnpm dev
|
||||
```
|
||||
@@ -207,10 +212,10 @@ If you need to start services individually:
|
||||
|
||||
The nginx configuration provides:
|
||||
- Unified entry point on port 2026
|
||||
- Rewrites `/api/langgraph/*` to Gateway's LangGraph-compatible API (8001)
|
||||
- Routes `/api/langgraph/*` to LangGraph Server (2024)
|
||||
- Routes other `/api/*` endpoints to Gateway API (8001)
|
||||
- Routes non-API requests to Frontend (3000)
|
||||
- Same-origin API routing; split-origin or port-forwarded browser clients should use the Gateway `GATEWAY_CORS_ORIGINS` allowlist
|
||||
- Centralized CORS handling
|
||||
- SSE/streaming support for real-time agent responses
|
||||
- Optimized timeouts for long-running operations
|
||||
|
||||
@@ -230,8 +235,8 @@ deer-flow/
|
||||
│ └── nginx.local.conf # Nginx config for local dev
|
||||
├── backend/ # Backend application
|
||||
│ ├── src/
|
||||
│ │ ├── gateway/ # Gateway API and LangGraph-compatible runtime (port 8001)
|
||||
│ │ ├── agents/ # LangGraph agent runtime used by Gateway
|
||||
│ │ ├── gateway/ # Gateway API (port 8001)
|
||||
│ │ ├── agents/ # LangGraph agents (port 2024)
|
||||
│ │ ├── mcp/ # Model Context Protocol integration
|
||||
│ │ ├── skills/ # Skills system
|
||||
│ │ └── sandbox/ # Sandbox execution
|
||||
@@ -251,7 +256,8 @@ Browser
|
||||
↓
|
||||
Nginx (port 2026) ← Unified entry point
|
||||
├→ Frontend (port 3000) ← / (non-API requests)
|
||||
└→ Gateway API (port 8001) ← /api/* and /api/langgraph/* (LangGraph-compatible agent interactions)
|
||||
├→ Gateway API (port 8001) ← /api/models, /api/mcp, /api/skills, /api/threads/*/artifacts
|
||||
└→ LangGraph Server (port 2024) ← /api/langgraph/* (agent interactions)
|
||||
```
|
||||
|
||||
## Development Workflow
|
||||
@@ -287,44 +293,24 @@ Nginx (port 2026) ← Unified entry point
|
||||
git push origin feature/your-feature-name
|
||||
```
|
||||
|
||||
## AI assistance disclosure
|
||||
|
||||
DeerFlow is an AI project and we welcome AI-assisted contributions. To help
|
||||
reviewers calibrate how closely to read a change, **every pull request must
|
||||
complete the "AI assistance" section of the
|
||||
[PR template](.github/pull_request_template.md)**:
|
||||
|
||||
- which tool(s) you used (or `none`),
|
||||
- how you used them, and
|
||||
- a confirmation that a human has read, understands, and takes responsibility
|
||||
for the change.
|
||||
|
||||
Please don't delete the section. PRs that ignore it may be asked to fill it in
|
||||
before review.
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
# Backend tests
|
||||
cd backend
|
||||
make test
|
||||
uv run pytest
|
||||
|
||||
# Frontend unit tests
|
||||
# Frontend checks
|
||||
cd frontend
|
||||
make test
|
||||
|
||||
# Frontend E2E tests (requires Chromium; builds and auto-starts the Next.js production server)
|
||||
cd frontend
|
||||
make test-e2e
|
||||
pnpm check
|
||||
```
|
||||
|
||||
### PR Regression Checks
|
||||
|
||||
Every pull request triggers the following CI workflows:
|
||||
Every pull request runs the backend regression workflow at [.github/workflows/backend-unit-tests.yml](.github/workflows/backend-unit-tests.yml), including:
|
||||
|
||||
- **Backend unit tests** — [.github/workflows/backend-unit-tests.yml](.github/workflows/backend-unit-tests.yml)
|
||||
- **Frontend unit tests** — [.github/workflows/frontend-unit-tests.yml](.github/workflows/frontend-unit-tests.yml)
|
||||
- **Frontend E2E tests** — [.github/workflows/e2e-tests.yml](.github/workflows/e2e-tests.yml) (triggered only when `frontend/` files change)
|
||||
- `tests/test_provisioner_kubeconfig.py`
|
||||
- `tests/test_docker_sandbox_mode_detection.py`
|
||||
|
||||
## Code Style
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# DeerFlow - Unified Development Environment
|
||||
|
||||
.PHONY: help config config-upgrade check install setup doctor detect-thread-boundaries detect-blocking-io dev dev-daemon start start-daemon stop up down clean docker-init docker-start docker-stop docker-logs docker-logs-frontend docker-logs-gateway
|
||||
.PHONY: help config config-upgrade check install setup doctor dev dev-pro dev-daemon dev-daemon-pro start start-pro start-daemon start-daemon-pro stop up up-pro down clean docker-init docker-start docker-start-pro docker-stop docker-logs docker-logs-frontend docker-logs-gateway
|
||||
|
||||
BASH ?= bash
|
||||
BACKEND_UV_RUN = cd backend && uv run
|
||||
@@ -23,24 +23,28 @@ help:
|
||||
@echo " make config - Generate local config files (aborts if config already exists)"
|
||||
@echo " make config-upgrade - Merge new fields from config.example.yaml into config.yaml"
|
||||
@echo " make check - Check if all required tools are installed"
|
||||
@echo " make detect-thread-boundaries - Inventory async/thread boundary points"
|
||||
@echo " make detect-blocking-io - Inventory blocking IO that may block the backend event loop"
|
||||
@echo " make install - Install all dependencies (frontend + backend + pre-commit hooks)"
|
||||
@echo " make install - Install all dependencies (frontend + backend)"
|
||||
@echo " make setup-sandbox - Pre-pull sandbox container image (recommended)"
|
||||
@echo " make dev - Start all services in development mode (with hot-reloading)"
|
||||
@echo " make dev-pro - Start in dev + Gateway mode (experimental, no LangGraph server)"
|
||||
@echo " make dev-daemon - Start dev services in background (daemon mode)"
|
||||
@echo " make dev-daemon-pro - Start dev daemon + Gateway mode (experimental)"
|
||||
@echo " make start - Start all services in production mode (optimized, no hot-reloading)"
|
||||
@echo " make start-pro - Start in prod + Gateway mode (experimental)"
|
||||
@echo " make start-daemon - Start prod services in background (daemon mode)"
|
||||
@echo " make start-daemon-pro - Start prod daemon + Gateway mode (experimental)"
|
||||
@echo " make stop - Stop all running services"
|
||||
@echo " make clean - Clean up processes and temporary files"
|
||||
@echo ""
|
||||
@echo "Docker Production Commands:"
|
||||
@echo " make up - Build and start production Docker services (localhost:2026)"
|
||||
@echo " make up-pro - Build and start production Docker in Gateway mode (experimental)"
|
||||
@echo " make down - Stop and remove production Docker containers"
|
||||
@echo ""
|
||||
@echo "Docker Development Commands:"
|
||||
@echo " make docker-init - Pull the sandbox image"
|
||||
@echo " make docker-start - Start Docker services (mode-aware from config.yaml, localhost:2026)"
|
||||
@echo " make docker-start-pro - Start Docker in Gateway mode (experimental, no LangGraph container)"
|
||||
@echo " make docker-stop - Stop Docker development services"
|
||||
@echo " make docker-logs - View Docker development logs"
|
||||
@echo " make docker-logs-frontend - View Docker frontend logs"
|
||||
@@ -53,12 +57,6 @@ setup:
|
||||
doctor:
|
||||
@$(BACKEND_UV_RUN) python ../scripts/doctor.py
|
||||
|
||||
detect-thread-boundaries:
|
||||
@$(PYTHON) ./scripts/detect_thread_boundaries.py
|
||||
|
||||
detect-blocking-io:
|
||||
@$(MAKE) -C backend detect-blocking-io
|
||||
|
||||
config:
|
||||
@$(PYTHON) ./scripts/configure.py
|
||||
|
||||
@@ -75,8 +73,6 @@ install:
|
||||
@cd backend && uv sync
|
||||
@echo "Installing frontend dependencies..."
|
||||
@cd frontend && pnpm install
|
||||
@echo "Installing pre-commit hooks..."
|
||||
@$(BACKEND_UV_RUN) --with pre-commit pre-commit install
|
||||
@echo "✓ All dependencies installed"
|
||||
@echo ""
|
||||
@echo "=========================================="
|
||||
@@ -89,28 +85,77 @@ install:
|
||||
|
||||
# Pre-pull sandbox Docker image (optional but recommended)
|
||||
setup-sandbox:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/setup-sandbox.sh
|
||||
@echo "=========================================="
|
||||
@echo " Pre-pulling Sandbox Container Image"
|
||||
@echo "=========================================="
|
||||
@echo ""
|
||||
@IMAGE=$$(grep -A 20 "# sandbox:" config.yaml 2>/dev/null | grep "image:" | awk '{print $$2}' | head -1); \
|
||||
if [ -z "$$IMAGE" ]; then \
|
||||
IMAGE="enterprise-public-cn-beijing.cr.volces.com/vefaas-public/all-in-one-sandbox:latest"; \
|
||||
echo "Using default image: $$IMAGE"; \
|
||||
else \
|
||||
echo "Using configured image: $$IMAGE"; \
|
||||
fi; \
|
||||
echo ""; \
|
||||
if command -v container >/dev/null 2>&1 && [ "$$(uname)" = "Darwin" ]; then \
|
||||
echo "Detected Apple Container on macOS, pulling image..."; \
|
||||
container pull "$$IMAGE" || echo "⚠ Apple Container pull failed, will try Docker"; \
|
||||
fi; \
|
||||
if command -v docker >/dev/null 2>&1; then \
|
||||
echo "Pulling image using Docker..."; \
|
||||
if docker pull "$$IMAGE"; then \
|
||||
echo ""; \
|
||||
echo "✓ Sandbox image pulled successfully"; \
|
||||
else \
|
||||
echo ""; \
|
||||
echo "⚠ Failed to pull sandbox image (this is OK for local sandbox mode)"; \
|
||||
fi; \
|
||||
else \
|
||||
echo "✗ Neither Docker nor Apple Container is available"; \
|
||||
echo " Please install Docker: https://docs.docker.com/get-docker/"; \
|
||||
exit 1; \
|
||||
fi
|
||||
|
||||
# Start all services in development mode (with hot-reloading)
|
||||
dev:
|
||||
@$(PYTHON) ./scripts/check.py
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --dev
|
||||
|
||||
# Start all services in dev + Gateway mode (experimental: agent runtime embedded in Gateway)
|
||||
dev-pro:
|
||||
@$(PYTHON) ./scripts/check.py
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --dev --gateway
|
||||
|
||||
# Start all services in production mode (with optimizations)
|
||||
start:
|
||||
@$(PYTHON) ./scripts/check.py
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --prod
|
||||
|
||||
# Start all services in prod + Gateway mode (experimental)
|
||||
start-pro:
|
||||
@$(PYTHON) ./scripts/check.py
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --prod --gateway
|
||||
|
||||
# Start all services in daemon mode (background)
|
||||
dev-daemon:
|
||||
@$(PYTHON) ./scripts/check.py
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --dev --daemon
|
||||
|
||||
# Start daemon + Gateway mode (experimental)
|
||||
dev-daemon-pro:
|
||||
@$(PYTHON) ./scripts/check.py
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --dev --gateway --daemon
|
||||
|
||||
# Start prod services in daemon mode (background)
|
||||
start-daemon:
|
||||
@$(PYTHON) ./scripts/check.py
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --prod --daemon
|
||||
|
||||
# Start prod daemon + Gateway mode (experimental)
|
||||
start-daemon-pro:
|
||||
@$(PYTHON) ./scripts/check.py
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --prod --gateway --daemon
|
||||
|
||||
# Stop all services
|
||||
stop:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/serve.sh --stop
|
||||
@@ -119,6 +164,7 @@ stop:
|
||||
clean: stop
|
||||
@echo "Cleaning up..."
|
||||
@-rm -rf backend/.deer-flow 2>/dev/null || true
|
||||
@-rm -rf backend/.langgraph_api 2>/dev/null || true
|
||||
@-rm -rf logs/*.log 2>/dev/null || true
|
||||
@echo "✓ Cleanup complete"
|
||||
|
||||
@@ -134,6 +180,10 @@ docker-init:
|
||||
docker-start:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/docker.sh start
|
||||
|
||||
# Start Docker in Gateway mode (experimental)
|
||||
docker-start-pro:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/docker.sh start --gateway
|
||||
|
||||
# Stop Docker development environment
|
||||
docker-stop:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/docker.sh stop
|
||||
@@ -156,6 +206,10 @@ docker-logs-gateway:
|
||||
up:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/deploy.sh
|
||||
|
||||
# Build and start production services in Gateway mode
|
||||
up-pro:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/deploy.sh --gateway
|
||||
|
||||
# Stop and remove production containers
|
||||
down:
|
||||
@$(RUN_WITH_GIT_BASH) ./scripts/deploy.sh down
|
||||
|
||||
@@ -243,9 +243,10 @@ make up # Build images and start all production services
|
||||
make down # Stop and remove containers
|
||||
```
|
||||
|
||||
Access: http://localhost:2026
|
||||
> [!NOTE]
|
||||
> The LangGraph agent server currently runs via `langgraph dev` (the open-source CLI server).
|
||||
|
||||
The unified nginx endpoint is same-origin by default and does not emit browser CORS headers. If you run a split-origin or port-forwarded browser client, set `GATEWAY_CORS_ORIGINS` to comma-separated exact origins such as `http://localhost:3000`; the Gateway then applies the CORS allowlist and matching CSRF origin checks.
|
||||
Access: http://localhost:2026
|
||||
|
||||
See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed Docker development guide.
|
||||
|
||||
@@ -253,7 +254,7 @@ See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed Docker development guide.
|
||||
|
||||
If you prefer running services locally:
|
||||
|
||||
Prerequisite: complete the "Configuration" steps above first (`make setup`). `make dev` requires a valid `config.yaml` in the project root. Set `DEER_FLOW_PROJECT_ROOT` to define that root explicitly, or `DEER_FLOW_CONFIG_PATH` to point at a specific config file. Runtime state defaults to `.deer-flow` under the project root and can be moved with `DEER_FLOW_HOME`; skills default to `skills/` under the project root and can be moved with `DEER_FLOW_SKILLS_PATH`. Run `make doctor` to verify your setup before starting.
|
||||
Prerequisite: complete the "Configuration" steps above first (`make setup`). `make dev` requires a valid `config.yaml` in the project root (can be overridden via `DEER_FLOW_CONFIG_PATH`). Run `make doctor` to verify your setup before starting.
|
||||
On Windows, run the local development flow from Git Bash. Native `cmd.exe` and PowerShell shells are not supported for the bash-based service scripts, and WSL is not guaranteed because some scripts rely on Git for Windows utilities such as `cygpath`.
|
||||
|
||||
1. **Check prerequisites**:
|
||||
@@ -263,7 +264,7 @@ On Windows, run the local development flow from Git Bash. Native `cmd.exe` and P
|
||||
|
||||
2. **Install dependencies**:
|
||||
```bash
|
||||
make install # Install backend + frontend dependencies + pre-commit hooks
|
||||
make install # Install backend + frontend dependencies
|
||||
```
|
||||
|
||||
3. **(Optional) Pre-pull sandbox image**:
|
||||
@@ -288,31 +289,53 @@ On Windows, run the local development flow from Git Bash. Native `cmd.exe` and P
|
||||
|
||||
#### Startup Modes
|
||||
|
||||
DeerFlow runs the agent runtime inside the Gateway API. Development mode enables hot-reload; production mode uses a pre-built frontend.
|
||||
DeerFlow supports multiple startup modes across two dimensions:
|
||||
|
||||
- **Dev / Prod** — dev enables hot-reload; prod uses pre-built frontend
|
||||
- **Standard / Gateway** — standard uses a separate LangGraph server (4 processes); Gateway mode (experimental) embeds the agent runtime in the Gateway API (3 processes)
|
||||
|
||||
| | **Local Foreground** | **Local Daemon** | **Docker Dev** | **Docker Prod** |
|
||||
|---|---|---|---|---|
|
||||
| **Dev** | `./scripts/serve.sh --dev`<br/>`make dev` | `./scripts/serve.sh --dev --daemon`<br/>`make dev-daemon` | `./scripts/docker.sh start`<br/>`make docker-start` | — |
|
||||
| **Dev + Gateway** | `./scripts/serve.sh --dev --gateway`<br/>`make dev-pro` | `./scripts/serve.sh --dev --gateway --daemon`<br/>`make dev-daemon-pro` | `./scripts/docker.sh start --gateway`<br/>`make docker-start-pro` | — |
|
||||
| **Prod** | `./scripts/serve.sh --prod`<br/>`make start` | `./scripts/serve.sh --prod --daemon`<br/>`make start-daemon` | — | `./scripts/deploy.sh`<br/>`make up` |
|
||||
| **Prod + Gateway** | `./scripts/serve.sh --prod --gateway`<br/>`make start-pro` | `./scripts/serve.sh --prod --gateway --daemon`<br/>`make start-daemon-pro` | — | `./scripts/deploy.sh --gateway`<br/>`make up-pro` |
|
||||
|
||||
| Action | Local | Docker Dev | Docker Prod |
|
||||
|---|---|---|---|
|
||||
| **Stop** | `./scripts/serve.sh --stop`<br/>`make stop` | `./scripts/docker.sh stop`<br/>`make docker-stop` | `./scripts/deploy.sh down`<br/>`make down` |
|
||||
| **Restart** | `./scripts/serve.sh --restart [flags]` | `./scripts/docker.sh restart` | — |
|
||||
|
||||
Gateway owns `/api/langgraph/*` and translates those public LangGraph-compatible paths to its native `/api/*` routers behind nginx.
|
||||
> **Gateway mode** eliminates the LangGraph server process — the Gateway API handles agent execution directly via async tasks, managing its own concurrency.
|
||||
|
||||
#### Why Gateway Mode?
|
||||
|
||||
In standard mode, DeerFlow runs a dedicated [LangGraph Platform](https://langchain-ai.github.io/langgraph/) server alongside the Gateway API. This architecture works well but has trade-offs:
|
||||
|
||||
| | Standard Mode | Gateway Mode |
|
||||
|---|---|---|
|
||||
| **Architecture** | Gateway (REST API) + LangGraph (agent runtime) | Gateway embeds agent runtime |
|
||||
| **Concurrency** | `--n-jobs-per-worker` per worker (requires license) | `--workers` × async tasks (no per-worker cap) |
|
||||
| **Containers / Processes** | 4 (frontend, gateway, langgraph, nginx) | 3 (frontend, gateway, nginx) |
|
||||
| **Resource usage** | Higher (two Python runtimes) | Lower (single Python runtime) |
|
||||
| **LangGraph Platform license** | Required for production images | Not required |
|
||||
| **Cold start** | Slower (two services to initialize) | Faster |
|
||||
|
||||
Both modes are functionally equivalent — the same agents, tools, and skills work in either mode.
|
||||
|
||||
#### Docker Production Deployment
|
||||
|
||||
`deploy.sh` supports building and starting separately:
|
||||
`deploy.sh` supports building and starting separately. Images are mode-agnostic — runtime mode is selected at start time:
|
||||
|
||||
```bash
|
||||
# One-step (build + start)
|
||||
deploy.sh
|
||||
deploy.sh # standard mode (default)
|
||||
deploy.sh --gateway # gateway mode
|
||||
|
||||
# Two-step (build once, start later)
|
||||
# Two-step (build once, start with any mode)
|
||||
deploy.sh build # build all images
|
||||
deploy.sh start # start pre-built images
|
||||
deploy.sh start # start in standard mode
|
||||
deploy.sh start --gateway # start in gateway mode
|
||||
|
||||
# Stop
|
||||
deploy.sh down
|
||||
@@ -347,14 +370,13 @@ DeerFlow supports receiving tasks from messaging apps. Channels auto-start when
|
||||
| Feishu / Lark | WebSocket | Moderate |
|
||||
| WeChat | Tencent iLink (long-polling) | Moderate |
|
||||
| WeCom | WebSocket | Moderate |
|
||||
| DingTalk | Stream Push (WebSocket) | Moderate |
|
||||
|
||||
**Configuration in `config.yaml`:**
|
||||
|
||||
```yaml
|
||||
channels:
|
||||
# LangGraph-compatible Gateway API base URL (default: http://localhost:8001/api)
|
||||
langgraph_url: http://localhost:8001/api
|
||||
# LangGraph Server URL (default: http://localhost:2024)
|
||||
langgraph_url: http://localhost:2024
|
||||
# Gateway API URL (default: http://localhost:8001)
|
||||
gateway_url: http://localhost:8001
|
||||
|
||||
@@ -417,19 +439,11 @@ channels:
|
||||
context:
|
||||
thinking_enabled: true
|
||||
subagent_enabled: true
|
||||
|
||||
dingtalk:
|
||||
enabled: true
|
||||
client_id: $DINGTALK_CLIENT_ID # Client ID of your DingTalk application
|
||||
client_secret: $DINGTALK_CLIENT_SECRET # Client Secret of your DingTalk application
|
||||
allowed_users: [] # empty = allow all
|
||||
card_template_id: "" # Optional: AI Card template ID for streaming typewriter effect
|
||||
```
|
||||
|
||||
Notes:
|
||||
- `assistant_id: lead_agent` calls the default LangGraph assistant directly.
|
||||
- If `assistant_id` is set to a custom agent name, DeerFlow still routes through `lead_agent` and injects that value as `agent_name`, so the custom agent's SOUL/config takes effect for IM channels.
|
||||
- IM channel workers call Gateway's LangGraph-compatible API internally and automatically attach process-local internal auth plus the CSRF cookie/header pair required for thread and run creation.
|
||||
|
||||
Set the corresponding API keys in your `.env` file:
|
||||
|
||||
@@ -452,10 +466,6 @@ WECHAT_ILINK_BOT_ID=your_ilink_bot_id
|
||||
# WeCom
|
||||
WECOM_BOT_ID=your_bot_id
|
||||
WECOM_BOT_SECRET=your_bot_secret
|
||||
|
||||
# DingTalk
|
||||
DINGTALK_CLIENT_ID=your_client_id
|
||||
DINGTALK_CLIENT_SECRET=your_client_secret
|
||||
```
|
||||
|
||||
**Telegram Setup**
|
||||
@@ -494,15 +504,7 @@ DINGTALK_CLIENT_SECRET=your_client_secret
|
||||
4. Make sure backend dependencies include `wecom-aibot-python-sdk`. The channel uses a WebSocket long connection and does not require a public callback URL.
|
||||
5. The current integration supports inbound text, image, and file messages. Final images/files generated by the agent are also sent back to the WeCom conversation.
|
||||
|
||||
**DingTalk Setup**
|
||||
|
||||
1. Create a DingTalk application in the [DingTalk Developer Console](https://open.dingtalk.com/) and enable **Robot** capability.
|
||||
2. Set the message receiving mode to **Stream Mode** in the robot configuration page.
|
||||
3. Copy the `Client ID` and `Client Secret`, set `DINGTALK_CLIENT_ID` and `DINGTALK_CLIENT_SECRET` in `.env`, and enable the channel in `config.yaml`.
|
||||
4. *(Optional)* To enable streaming AI Card replies (typewriter effect), create an **AI Card** template on the [DingTalk Card Platform](https://open.dingtalk.com/document/dingstart/typewriter-effect-streaming-ai-card), then set `card_template_id` in `config.yaml` to the template ID. You also need to apply for the `Card.Streaming.Write` and `Card.Instance.Write` permissions.
|
||||
|
||||
|
||||
When DeerFlow runs in Docker Compose, IM channels execute inside the `gateway` container. In that case, do not point `channels.langgraph_url` or `channels.gateway_url` at `localhost`; use container service names such as `http://gateway:8001/api` and `http://gateway:8001`, or set `DEER_FLOW_CHANNELS_LANGGRAPH_URL` and `DEER_FLOW_CHANNELS_GATEWAY_URL`.
|
||||
When DeerFlow runs in Docker Compose, IM channels execute inside the `gateway` container. In that case, do not point `channels.langgraph_url` or `channels.gateway_url` at `localhost`; use container service names such as `http://langgraph:2024` and `http://gateway:8001`, or set `DEER_FLOW_CHANNELS_LANGGRAPH_URL` and `DEER_FLOW_CHANNELS_GATEWAY_URL`.
|
||||
|
||||
**Commands**
|
||||
|
||||
@@ -546,15 +548,6 @@ LANGFUSE_BASE_URL=https://cloud.langfuse.com
|
||||
|
||||
If you are using a self-hosted Langfuse instance, set `LANGFUSE_BASE_URL` to your deployment URL.
|
||||
|
||||
**Trace correlation fields.** Every agent run is annotated with Langfuse's reserved trace attributes so the Sessions and Users pages light up automatically:
|
||||
|
||||
- `session_id` = LangGraph `thread_id` — groups every trace of the same conversation
|
||||
- `user_id` = effective user from `get_effective_user_id()` (falls back to `default` in no-auth mode)
|
||||
- `trace_name` = assistant id (defaults to `lead-agent`)
|
||||
- `tags` = `[env:<DEER_FLOW_ENV>, model:<model_name>]` (omitted when not set)
|
||||
|
||||
These are injected into `RunnableConfig.metadata` at the graph invocation root for both the gateway path (`runtime/runs/worker.py::run_agent`) and the embedded path (`client.py::DeerFlowClient.stream`), so any LangChain-compatible callback can read them. Set `DEER_FLOW_ENV` (or `ENVIRONMENT`) to tag traces by deployment environment.
|
||||
|
||||
#### Using Both Providers
|
||||
|
||||
If both LangSmith and Langfuse are enabled, DeerFlow attaches both tracing callbacks and reports the same model activity to both systems.
|
||||
@@ -585,8 +578,6 @@ A standard Agent Skill is a structured capability module — a Markdown file tha
|
||||
|
||||
Skills are loaded progressively — only when the task needs them, not all at once. This keeps the context window lean and makes DeerFlow work well even with token-sensitive models.
|
||||
|
||||
Users can explicitly activate an enabled skill for a single turn by starting the request with `/skill-name`, for example `/data-analysis analyze uploads/foo.csv`. DeerFlow loads that skill's `SKILL.md` as hidden current-turn context while leaving the base prompt limited to skill metadata. Slash activation respects disabled skills, custom-agent skill whitelists, and existing channel commands such as `/new` and `/help`.
|
||||
|
||||
When you install `.skill` archives through the Gateway, DeerFlow accepts standard optional frontmatter metadata such as `version`, `author`, and `compatibility` instead of rejecting otherwise valid external skills.
|
||||
|
||||
Tools follow the same philosophy. DeerFlow comes with a core toolset — web search, web fetch, file operations, bash execution — and supports custom tools via MCP servers and Python functions. Swap anything. Add anything.
|
||||
@@ -639,7 +630,7 @@ See [`skills/public/claude-to-deerflow/SKILL.md`](skills/public/claude-to-deerfl
|
||||
|
||||
Complex tasks rarely fit in a single pass. DeerFlow decomposes them.
|
||||
|
||||
The lead agent can spawn sub-agents on the fly — each with its own scoped context, tools, and termination conditions. Sub-agents run in parallel when possible, report back structured results, and the lead agent synthesizes everything into a coherent output. When token usage tracking is enabled, completed sub-agent usage is attributed back to the dispatching step.
|
||||
The lead agent can spawn sub-agents on the fly — each with its own scoped context, tools, and termination conditions. Sub-agents run in parallel when possible, report back structured results, and the lead agent synthesizes everything into a coherent output.
|
||||
|
||||
This is how DeerFlow handles tasks that take minutes to hours: a research task might fan out into a dozen sub-agents, each exploring a different angle, then converge into a single report — or a website — or a slide deck with generated visuals. One harness, many hands.
|
||||
|
||||
@@ -667,8 +658,6 @@ This is the difference between a chatbot with tool access and an agent with an a
|
||||
|
||||
**Summarization**: Within a session, DeerFlow manages context aggressively — summarizing completed sub-tasks, offloading intermediate results to the filesystem, compressing what's no longer immediately relevant. This lets it stay sharp across long, multi-step tasks without blowing the context window.
|
||||
|
||||
**Strict Tool-Call Recovery**: When a provider or middleware interrupts a tool-call loop, DeerFlow now strips provider-level raw tool-call metadata on forced-stop assistant messages and injects placeholder tool results for dangling calls before the next model invocation. This keeps OpenAI-compatible reasoning models that strictly validate `tool_call_id` sequences from failing with malformed history errors.
|
||||
|
||||
### Long-Term Memory
|
||||
|
||||
Most agents forget everything the moment a conversation ends. DeerFlow remembers.
|
||||
@@ -742,12 +731,6 @@ DeerFlow has key high-privilege capabilities including **system command executio
|
||||
We welcome contributions! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, workflow, and guidelines.
|
||||
|
||||
Regression coverage includes Docker sandbox mode detection and provisioner kubeconfig-path handling tests in `backend/tests/`.
|
||||
Backend blocking-IO diagnostics are available from the repository root with
|
||||
`make detect-blocking-io`: it statically scans backend business code for
|
||||
blocking IO that may run on the backend event loop, prints a concise summary,
|
||||
and writes complete JSON findings to `.deer-flow/blocking-io-findings.json`.
|
||||
The JSON includes compact review records with `priority`, `location`,
|
||||
`blocking_call`, `event_loop_exposure`, `reason`, and `code`.
|
||||
Gateway artifact serving now forces active web content types (`text/html`, `application/xhtml+xml`, `image/svg+xml`) to download as attachments instead of inline rendering, reducing XSS risk for generated artifacts.
|
||||
|
||||
## License
|
||||
|
||||
+3
-22
@@ -228,7 +228,7 @@ make down # Stop and remove containers
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> Le runtime d'agent s'exécute actuellement dans la Gateway. nginx réécrit `/api/langgraph/*` vers l'API compatible LangGraph servie par la Gateway.
|
||||
> Le serveur d'agents LangGraph fonctionne actuellement via `langgraph dev` (le serveur CLI open source).
|
||||
|
||||
Accès : http://localhost:2026
|
||||
|
||||
@@ -290,14 +290,13 @@ DeerFlow peut recevoir des tâches depuis des applications de messagerie. Les ca
|
||||
| Telegram | Bot API (long-polling) | Facile |
|
||||
| Slack | Socket Mode | Modérée |
|
||||
| Feishu / Lark | WebSocket | Modérée |
|
||||
| DingTalk | Stream Push (WebSocket) | Modérée |
|
||||
|
||||
**Configuration dans `config.yaml` :**
|
||||
|
||||
```yaml
|
||||
channels:
|
||||
# LangGraph-compatible Gateway API base URL (default: http://localhost:8001/api)
|
||||
langgraph_url: http://localhost:8001/api
|
||||
# LangGraph Server URL (default: http://localhost:2024)
|
||||
langgraph_url: http://localhost:2024
|
||||
# Gateway API URL (default: http://localhost:8001)
|
||||
gateway_url: http://localhost:8001
|
||||
|
||||
@@ -342,13 +341,6 @@ channels:
|
||||
context:
|
||||
thinking_enabled: true
|
||||
subagent_enabled: true
|
||||
|
||||
dingtalk:
|
||||
enabled: true
|
||||
client_id: $DINGTALK_CLIENT_ID # ClientId depuis DingTalk Open Platform
|
||||
client_secret: $DINGTALK_CLIENT_SECRET # ClientSecret depuis DingTalk Open Platform
|
||||
allowed_users: [] # vide = tout le monde autorisé
|
||||
card_template_id: "" # Optionnel : ID de modèle AI Card pour l'effet machine à écrire en streaming
|
||||
```
|
||||
|
||||
Définissez les clés API correspondantes dans votre fichier `.env` :
|
||||
@@ -364,10 +356,6 @@ SLACK_APP_TOKEN=xapp-...
|
||||
# Feishu / Lark
|
||||
FEISHU_APP_ID=cli_xxxx
|
||||
FEISHU_APP_SECRET=your_app_secret
|
||||
|
||||
# DingTalk
|
||||
DINGTALK_CLIENT_ID=your_client_id
|
||||
DINGTALK_CLIENT_SECRET=your_client_secret
|
||||
```
|
||||
|
||||
**Configuration Telegram**
|
||||
@@ -390,13 +378,6 @@ DINGTALK_CLIENT_SECRET=your_client_secret
|
||||
3. Dans **Events**, abonnez-vous à `im.message.receive_v1` et sélectionnez le mode **Long Connection**.
|
||||
4. Copiez l'App ID et l'App Secret. Définissez `FEISHU_APP_ID` et `FEISHU_APP_SECRET` dans `.env` et activez le canal dans `config.yaml`.
|
||||
|
||||
**Configuration DingTalk**
|
||||
|
||||
1. Créez une application sur [DingTalk Open Platform](https://open.dingtalk.com/) et activez la capacité **Robot**.
|
||||
2. Dans la page de configuration du robot, définissez le mode de réception des messages sur **Stream**.
|
||||
3. Copiez le `Client ID` et le `Client Secret`. Définissez `DINGTALK_CLIENT_ID` et `DINGTALK_CLIENT_SECRET` dans `.env` et activez le canal dans `config.yaml`.
|
||||
4. *(Optionnel)* Pour activer les réponses en streaming AI Card (effet machine à écrire), créez un modèle **AI Card** sur la [plateforme de cartes DingTalk](https://open.dingtalk.com/document/dingstart/typewriter-effect-streaming-ai-card), puis définissez `card_template_id` dans `config.yaml` avec l'ID du modèle. Vous devez également demander les permissions `Card.Streaming.Write` et `Card.Instance.Write`.
|
||||
|
||||
**Commandes**
|
||||
|
||||
Une fois un canal connecté, vous pouvez interagir avec DeerFlow directement depuis le chat :
|
||||
|
||||
+3
-22
@@ -181,7 +181,7 @@ make down # コンテナを停止して削除
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> Agentランタイムは現在Gateway内で実行されます。`/api/langgraph/*`はnginxによってGatewayのLangGraph-compatible APIへ書き換えられます。
|
||||
> LangGraphエージェントサーバーは現在`langgraph dev`(オープンソースCLIサーバー)経由で実行されます。
|
||||
|
||||
アクセス: http://localhost:2026
|
||||
|
||||
@@ -243,14 +243,13 @@ DeerFlowはメッセージングアプリからのタスク受信をサポート
|
||||
| Telegram | Bot API(ロングポーリング) | 簡単 |
|
||||
| Slack | Socket Mode | 中程度 |
|
||||
| Feishu / Lark | WebSocket | 中程度 |
|
||||
| DingTalk | Stream Push(WebSocket) | 中程度 |
|
||||
|
||||
**`config.yaml`での設定:**
|
||||
|
||||
```yaml
|
||||
channels:
|
||||
# LangGraph-compatible Gateway API base URL(デフォルト: http://localhost:8001/api)
|
||||
langgraph_url: http://localhost:8001/api
|
||||
# LangGraphサーバーURL(デフォルト: http://localhost:2024)
|
||||
langgraph_url: http://localhost:2024
|
||||
# Gateway API URL(デフォルト: http://localhost:8001)
|
||||
gateway_url: http://localhost:8001
|
||||
|
||||
@@ -295,13 +294,6 @@ channels:
|
||||
context:
|
||||
thinking_enabled: true
|
||||
subagent_enabled: true
|
||||
|
||||
dingtalk:
|
||||
enabled: true
|
||||
client_id: $DINGTALK_CLIENT_ID # DingTalk Open PlatformのClientId
|
||||
client_secret: $DINGTALK_CLIENT_SECRET # DingTalk Open PlatformのClientSecret
|
||||
allowed_users: [] # 空 = 全員許可
|
||||
card_template_id: "" # オプション:ストリーミングタイプライター効果用のAIカードテンプレートID
|
||||
```
|
||||
|
||||
対応するAPIキーを`.env`ファイルに設定します:
|
||||
@@ -317,10 +309,6 @@ SLACK_APP_TOKEN=xapp-...
|
||||
# Feishu / Lark
|
||||
FEISHU_APP_ID=cli_xxxx
|
||||
FEISHU_APP_SECRET=your_app_secret
|
||||
|
||||
# DingTalk
|
||||
DINGTALK_CLIENT_ID=your_client_id
|
||||
DINGTALK_CLIENT_SECRET=your_client_secret
|
||||
```
|
||||
|
||||
**Telegramのセットアップ**
|
||||
@@ -343,13 +331,6 @@ DINGTALK_CLIENT_SECRET=your_client_secret
|
||||
3. **イベント**で`im.message.receive_v1`を購読し、**ロングコネクション**モードを選択。
|
||||
4. App IDとApp Secretをコピー。`.env`に`FEISHU_APP_ID`と`FEISHU_APP_SECRET`を設定し、`config.yaml`でチャネルを有効にします。
|
||||
|
||||
**DingTalkのセットアップ**
|
||||
|
||||
1. [DingTalk Open Platform](https://open.dingtalk.com/)でアプリを作成し、**ロボット**機能を有効化します。
|
||||
2. ロボット設定ページでメッセージ受信モードを**Streamモード**に設定します。
|
||||
3. `Client ID`と`Client Secret`をコピー。`.env`に`DINGTALK_CLIENT_ID`と`DINGTALK_CLIENT_SECRET`を設定し、`config.yaml`でチャネルを有効にします。
|
||||
4. *(オプション)* ストリーミングAIカード返信(タイプライター効果)を有効にするには、[DingTalkカードプラットフォーム](https://open.dingtalk.com/document/dingstart/typewriter-effect-streaming-ai-card)で**AIカード**テンプレートを作成し、`config.yaml`の`card_template_id`にテンプレートIDを設定します。`Card.Streaming.Write` および `Card.Instance.Write` 権限の申請も必要です。
|
||||
|
||||
**コマンド**
|
||||
|
||||
チャネル接続後、チャットから直接DeerFlowと対話できます:
|
||||
|
||||
@@ -256,7 +256,6 @@ DeerFlow принимает задачи прямо из мессенджеро
|
||||
| Telegram | Bot API (long-polling) | Просто |
|
||||
| Slack | Socket Mode | Средне |
|
||||
| Feishu / Lark | WebSocket | Средне |
|
||||
| DingTalk | Stream Push (WebSocket) | Средне |
|
||||
|
||||
**Конфигурация в `config.yaml`:**
|
||||
|
||||
@@ -279,13 +278,6 @@ channels:
|
||||
enabled: true
|
||||
bot_token: $TELEGRAM_BOT_TOKEN
|
||||
allowed_users: []
|
||||
|
||||
dingtalk:
|
||||
enabled: true
|
||||
client_id: $DINGTALK_CLIENT_ID # ClientId с DingTalk Open Platform
|
||||
client_secret: $DINGTALK_CLIENT_SECRET # ClientSecret с DingTalk Open Platform
|
||||
allowed_users: [] # пусто = разрешить всем
|
||||
card_template_id: "" # Опционально: ID шаблона AI Card для потокового эффекта печатной машинки
|
||||
```
|
||||
|
||||
**Настройка Telegram**
|
||||
@@ -293,13 +285,6 @@ channels:
|
||||
1. Напишите [@BotFather](https://t.me/BotFather), отправьте `/newbot` и скопируйте HTTP API-токен.
|
||||
2. Укажите `TELEGRAM_BOT_TOKEN` в `.env` и включите канал в `config.yaml`.
|
||||
|
||||
**Настройка DingTalk**
|
||||
|
||||
1. Создайте приложение на [DingTalk Open Platform](https://open.dingtalk.com/) и включите возможность **Робот**.
|
||||
2. На странице настроек робота установите режим приёма сообщений на **Stream**.
|
||||
3. Скопируйте `Client ID` и `Client Secret`. Укажите `DINGTALK_CLIENT_ID` и `DINGTALK_CLIENT_SECRET` в `.env` и включите канал в `config.yaml`.
|
||||
4. *(Опционально)* Для включения потоковых ответов AI Card (эффект печатной машинки) создайте шаблон **AI Card** на [платформе карточек DingTalk](https://open.dingtalk.com/document/dingstart/typewriter-effect-streaming-ai-card), затем укажите `card_template_id` в `config.yaml` с ID шаблона. Также необходимо запросить разрешения `Card.Streaming.Write` и `Card.Instance.Write`.
|
||||
|
||||
**Доступные команды**
|
||||
|
||||
| Команда | Описание |
|
||||
|
||||
+4
-23
@@ -184,7 +184,7 @@ make down # 停止并移除容器
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> 当前 Agent 运行时嵌入在 Gateway 中运行,`/api/langgraph/*` 会由 nginx 重写到 Gateway 的 LangGraph-compatible API。
|
||||
> 当前 LangGraph agent server 通过开源 CLI 服务 `langgraph dev` 运行。
|
||||
|
||||
访问地址:http://localhost:2026
|
||||
|
||||
@@ -194,7 +194,7 @@ make down # 停止并移除容器
|
||||
|
||||
如果你更希望直接在本地启动各个服务:
|
||||
|
||||
前提:先完成上面的“配置”步骤(`make config` 和模型 API key 配置)。`make dev` 需要有效配置文件,默认读取项目根目录下的 `config.yaml`。可以用 `DEER_FLOW_PROJECT_ROOT` 显式指定项目根目录,也可以用 `DEER_FLOW_CONFIG_PATH` 指向某个具体配置文件。运行期状态默认写到项目根目录下的 `.deer-flow`,可用 `DEER_FLOW_HOME` 覆盖;skills 默认读取项目根目录下的 `skills/`,可用 `DEER_FLOW_SKILLS_PATH` 覆盖。
|
||||
前提:先完成上面的“配置”步骤(`make config` 和模型 API key 配置)。`make dev` 需要有效配置文件,默认读取项目根目录下的 `config.yaml`,也可以通过 `DEER_FLOW_CONFIG_PATH` 覆盖。
|
||||
在 Windows 上,请使用 Git Bash 运行本地开发流程。基于 bash 的服务脚本不支持直接在原生 `cmd.exe` 或 PowerShell 中执行,且 WSL 也不保证可用,因为部分脚本依赖 Git for Windows 的 `cygpath` 等工具。
|
||||
|
||||
1. **检查依赖环境**:
|
||||
@@ -248,14 +248,13 @@ DeerFlow 支持从即时通讯应用接收任务。只要配置完成,对应
|
||||
| Slack | Socket Mode | 中等 |
|
||||
| Feishu / Lark | WebSocket | 中等 |
|
||||
| 企业微信智能机器人 | WebSocket | 中等 |
|
||||
| 钉钉 | Stream Push(WebSocket) | 中等 |
|
||||
|
||||
**`config.yaml` 中的配置示例:**
|
||||
|
||||
```yaml
|
||||
channels:
|
||||
# LangGraph-compatible Gateway API base URL(默认:http://localhost:8001/api)
|
||||
langgraph_url: http://localhost:8001/api
|
||||
# LangGraph Server URL(默认:http://localhost:2024)
|
||||
langgraph_url: http://localhost:2024
|
||||
# Gateway API URL(默认:http://localhost:8001)
|
||||
gateway_url: http://localhost:8001
|
||||
|
||||
@@ -305,13 +304,6 @@ channels:
|
||||
context:
|
||||
thinking_enabled: true
|
||||
subagent_enabled: true
|
||||
|
||||
dingtalk:
|
||||
enabled: true
|
||||
client_id: $DINGTALK_CLIENT_ID # 钉钉开放平台 ClientId
|
||||
client_secret: $DINGTALK_CLIENT_SECRET # 钉钉开放平台 ClientSecret
|
||||
allowed_users: [] # 留空表示允许所有人
|
||||
card_template_id: "" # 可选:AI 卡片模板 ID,用于流式打字机效果
|
||||
```
|
||||
|
||||
说明:
|
||||
@@ -335,10 +327,6 @@ FEISHU_APP_SECRET=your_app_secret
|
||||
# 企业微信智能机器人
|
||||
WECOM_BOT_ID=your_bot_id
|
||||
WECOM_BOT_SECRET=your_bot_secret
|
||||
|
||||
# 钉钉
|
||||
DINGTALK_CLIENT_ID=your_client_id
|
||||
DINGTALK_CLIENT_SECRET=your_client_secret
|
||||
```
|
||||
|
||||
**Telegram 配置**
|
||||
@@ -369,13 +357,6 @@ DINGTALK_CLIENT_SECRET=your_client_secret
|
||||
4. 安装后端依赖时确保包含 `wecom-aibot-python-sdk`,渠道会通过 WebSocket 长连接接收消息,无需公网回调地址。
|
||||
5. 当前支持文本、图片和文件入站消息;agent 生成的最终图片/文件也会回传到企业微信会话中。
|
||||
|
||||
**钉钉配置**
|
||||
|
||||
1. 在 [钉钉开放平台](https://open.dingtalk.com/) 创建应用,并启用 **机器人** 能力。
|
||||
2. 在机器人配置页面设置消息接收模式为 **Stream模式**。
|
||||
3. 复制 `Client ID` 和 `Client Secret`,在 `.env` 中设置 `DINGTALK_CLIENT_ID` 和 `DINGTALK_CLIENT_SECRET`,并在 `config.yaml` 中启用该渠道。
|
||||
4. *(可选)* 如需开启流式 AI 卡片回复(打字机效果),请在[钉钉卡片平台](https://open.dingtalk.com/document/dingstart/typewriter-effect-streaming-ai-card)创建 **AI 卡片**模板,然后在 `config.yaml` 中将 `card_template_id` 设为该模板 ID。同时需要申请 `Card.Streaming.Write` 和 `Card.Instance.Write` 权限。
|
||||
|
||||
**命令**
|
||||
|
||||
渠道连接完成后,你可以直接在聊天窗口里和 DeerFlow 交互:
|
||||
|
||||
@@ -24,10 +24,5 @@ config.yaml
|
||||
# Langgraph
|
||||
.langgraph_api
|
||||
|
||||
# Sandbox runtime working dir — pre-created and excluded from uvicorn reload
|
||||
# (scripts/serve.sh, docker/dev-entrypoint.sh). Anchored so it does not match
|
||||
# the source package backend/packages/harness/deerflow/sandbox/.
|
||||
/sandbox/
|
||||
|
||||
# Claude Code settings
|
||||
.claude/settings.local.json
|
||||
|
||||
+62
-148
@@ -7,13 +7,15 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
|
||||
DeerFlow is a LangGraph-based AI super agent system with a full-stack architecture. The backend provides a "super agent" with sandbox execution, persistent memory, subagent delegation, and extensible tool integration - all operating in per-thread isolated environments.
|
||||
|
||||
**Architecture**:
|
||||
- **Gateway API** (port 8001): REST API plus embedded LangGraph-compatible agent runtime
|
||||
- **LangGraph Server** (port 2024): Agent runtime and workflow execution
|
||||
- **Gateway API** (port 8001): REST API for models, MCP, skills, memory, artifacts, uploads, and local thread cleanup
|
||||
- **Frontend** (port 3000): Next.js web interface
|
||||
- **Nginx** (port 2026): Unified reverse proxy entry point
|
||||
- **Provisioner** (port 8002, optional in Docker dev): Started only when sandbox is configured for provisioner/Kubernetes mode
|
||||
|
||||
**Runtime**:
|
||||
- `make dev`, Docker dev, and production all run the agent runtime in Gateway via `RunManager` + `run_agent()` + `StreamBridge` (`packages/harness/deerflow/runtime/`). Nginx exposes that runtime at `/api/langgraph/*` and rewrites it to Gateway's native `/api/*` routers.
|
||||
**Runtime Modes**:
|
||||
- **Standard mode** (`make dev`): LangGraph Server handles agent execution as a separate process. 4 processes total.
|
||||
- **Gateway mode** (`make dev-pro`, experimental): Agent runtime embedded in Gateway via `RunManager` + `run_agent()` + `StreamBridge` (`packages/harness/deerflow/runtime/`). Service manages its own concurrency via async tasks. 3 processes total, no LangGraph Server.
|
||||
|
||||
**Project Structure**:
|
||||
```
|
||||
@@ -23,7 +25,7 @@ deer-flow/
|
||||
├── extensions_config.json # MCP servers and skills configuration
|
||||
├── backend/ # Backend application (this directory)
|
||||
│ ├── Makefile # Backend-only commands (dev, gateway, lint)
|
||||
│ ├── langgraph.json # LangGraph Studio graph configuration
|
||||
│ ├── langgraph.json # LangGraph server configuration
|
||||
│ ├── packages/
|
||||
│ │ └── harness/ # deerflow-harness package (import: deerflow.*)
|
||||
│ │ ├── pyproject.toml
|
||||
@@ -81,64 +83,26 @@ When making code changes, you MUST update the relevant documentation:
|
||||
```bash
|
||||
make check # Check system requirements
|
||||
make install # Install all dependencies (frontend + backend)
|
||||
make dev # Start all services (Gateway + Frontend + Nginx), with config.yaml preflight
|
||||
make start # Start production services locally
|
||||
make dev # Start all services (LangGraph + Gateway + Frontend + Nginx), with config.yaml preflight
|
||||
make dev-pro # Gateway mode (experimental): skip LangGraph, agent runtime embedded in Gateway
|
||||
make start-pro # Production + Gateway mode (experimental)
|
||||
make stop # Stop all services
|
||||
```
|
||||
|
||||
**Backend directory** (for backend development only):
|
||||
```bash
|
||||
make install # Install backend dependencies
|
||||
make dev # Run Gateway API with reload (port 8001)
|
||||
make dev # Run LangGraph server only (port 2024)
|
||||
make gateway # Run Gateway API only (port 8001)
|
||||
make test # Run all backend tests
|
||||
make test-blocking-io # Run strict Blockbuster runtime gate on tests/blocking_io/
|
||||
make lint # Lint with ruff
|
||||
make format # Format code with ruff
|
||||
```
|
||||
|
||||
The `detect-blocking-io` target parses `app/`, `packages/harness/deerflow/`,
|
||||
and `scripts/` with AST. By default it reports only blocking IO candidates that
|
||||
are inside async code, reachable from async code in the same file, or reachable
|
||||
from sync-only `AgentMiddleware` before/after hooks that LangGraph can execute
|
||||
on the async graph path. It prints a concise summary and writes complete JSON
|
||||
findings to `.deer-flow/blocking-io-findings.json` at the repository root
|
||||
(both `make detect-blocking-io` from the repo root and `cd backend && make
|
||||
detect-blocking-io` resolve to the same repo-root path). JSON findings include
|
||||
`priority`, `location`, `blocking_call`, `event_loop_exposure`, `reason`, and
|
||||
`code` for model-assisted or manual review. `priority` is a deterministic
|
||||
review ordering from operation type, not proof of a bug. Bare-name same-file
|
||||
calls are resolved by function name, so duplicate helper names in one file can
|
||||
conservatively over-report async reachability. It is intentionally
|
||||
informational and is not run from CI in this round.
|
||||
|
||||
Regression tests related to Docker/provisioner behavior:
|
||||
- `tests/test_docker_sandbox_mode_detection.py` (mode detection from `config.yaml`)
|
||||
- `tests/test_provisioner_kubeconfig.py` (kubeconfig file/directory handling)
|
||||
|
||||
Blocking-IO runtime gate (`tests/blocking_io/`):
|
||||
- Wraps every item under `tests/blocking_io/` with a strict Blockbuster
|
||||
context scoped to `app.*` and `deerflow.*` (see
|
||||
`tests/support/detectors/blocking_io_runtime.py`). Any sync blocking IO
|
||||
call whose stack passes through DeerFlow business code while running on
|
||||
the asyncio event loop raises `BlockingError` and fails the test.
|
||||
- Regression anchors live there: `test_skills_load.py` (locks the
|
||||
`asyncio.to_thread` offload around `LocalSkillStorage.load_skills`, fix
|
||||
for #1917); `test_sqlite_lifespan.py` (locks the offload around
|
||||
SQLite path resolution plus `ensure_sqlite_parent_dir`, fix for #1912);
|
||||
`test_jsonl_run_event_store.py` (locks `JsonlRunEventStore`'s async
|
||||
API offloading its file IO via `asyncio.to_thread`, fix #3084); and
|
||||
`test_uploads_middleware.py` (locks `UploadsMiddleware.abefore_agent`
|
||||
offloading the uploads-directory scan off the event loop).
|
||||
- `test_gate_smoke.py` is a meta-test asserting the gate actually catches
|
||||
unoffloaded blocking IO and that the `@pytest.mark.allow_blocking_io`
|
||||
opt-out works.
|
||||
- Coverage boundary: the gate only sees code that test execution actually
|
||||
touches. Static AST coverage is a separate concern (out of scope for
|
||||
this PR).
|
||||
- CI: runs on every PR via `.github/workflows/backend-blocking-io-tests.yml`,
|
||||
hard-fail.
|
||||
|
||||
Boundary check (harness → app import firewall):
|
||||
- `tests/test_harness_boundary.py` — ensures `packages/harness/deerflow/` never imports from `app.*`
|
||||
|
||||
@@ -151,7 +115,7 @@ CI runs these regression tests for every pull request via [.github/workflows/bac
|
||||
The backend is split into two layers with a strict dependency direction:
|
||||
|
||||
- **Harness** (`packages/harness/deerflow/`): Publishable agent framework package (`deerflow-harness`). Import prefix: `deerflow.*`. Contains agent orchestration, tools, sandbox, models, MCP, skills, config — everything needed to build and run agents.
|
||||
- **App** (`app/`): Unpublished application code. Import prefix: `app.*`. Contains the FastAPI Gateway API and IM channel integrations (Feishu, Slack, Telegram, DingTalk).
|
||||
- **App** (`app/`): Unpublished application code. Import prefix: `app.*`. Contains the FastAPI Gateway API and IM channel integrations (Feishu, Slack, Telegram).
|
||||
|
||||
**Dependency rule**: App imports deerflow, but deerflow never imports app. This boundary is enforced by `tests/test_harness_boundary.py` which runs in CI.
|
||||
|
||||
@@ -192,27 +156,20 @@ from deerflow.config import get_app_config
|
||||
|
||||
### Middleware Chain
|
||||
|
||||
Lead-agent middlewares are assembled in strict append order across `packages/harness/deerflow/agents/middlewares/tool_error_handling_middleware.py` (`build_lead_runtime_middlewares`) and `packages/harness/deerflow/agents/lead_agent/agent.py` (`build_middlewares`):
|
||||
Middlewares execute in strict order in `packages/harness/deerflow/agents/lead_agent/agent.py`:
|
||||
|
||||
1. **ThreadDataMiddleware** - Creates per-thread directories under the user's isolation scope (`backend/.deer-flow/users/{user_id}/threads/{thread_id}/user-data/{workspace,uploads,outputs}`); resolves `user_id` via `get_effective_user_id()` (falls back to `"default"` in no-auth mode); Web UI thread deletion now follows LangGraph thread removal with Gateway cleanup of the local thread directory
|
||||
1. **ThreadDataMiddleware** - Creates per-thread directories (`backend/.deer-flow/threads/{thread_id}/user-data/{workspace,uploads,outputs}`); Web UI thread deletion now follows LangGraph thread removal with Gateway cleanup of the local `.deer-flow/threads/{thread_id}` directory
|
||||
2. **UploadsMiddleware** - Tracks and injects newly uploaded files into conversation
|
||||
3. **SandboxMiddleware** - Acquires sandbox, stores `sandbox_id` in state
|
||||
4. **DanglingToolCallMiddleware** - Injects placeholder ToolMessages for AIMessage tool_calls that lack responses (e.g., due to user interruption), including raw provider tool-call payloads preserved only in `additional_kwargs["tool_calls"]`
|
||||
5. **LLMErrorHandlingMiddleware** - Normalizes provider/model invocation failures into recoverable assistant-facing errors before later middleware/tool stages run
|
||||
6. **GuardrailMiddleware** - Pre-tool-call authorization via pluggable `GuardrailProvider` protocol (optional, if `guardrails.enabled` in config). Evaluates each tool call and returns error ToolMessage on deny. Three provider options: built-in `AllowlistProvider` (zero deps), OAP policy providers (e.g. `aport-agent-guardrails`), or custom providers. See [docs/GUARDRAILS.md](docs/GUARDRAILS.md) for setup, usage, and how to implement a provider.
|
||||
7. **SandboxAuditMiddleware** - Audits sandboxed shell/file operations for security logging before tool execution continues
|
||||
8. **ToolErrorHandlingMiddleware** - Converts tool exceptions into error `ToolMessage`s so the run can continue instead of aborting
|
||||
9. **SkillActivationMiddleware** - Detects strict `/skill-name task` syntax on the latest real user message, resolves only enabled and runtime-allowed skills, reads `SKILL.md` from trusted skill storage, injects the skill body as hidden current-turn model context, and records a `middleware:skill_activation` audit event with skill name, category, path, and content hash
|
||||
10. **SummarizationMiddleware** - Context reduction when approaching token limits (optional, if enabled)
|
||||
11. **TodoListMiddleware** - Task tracking with `write_todos` tool (optional, if plan_mode)
|
||||
12. **TokenUsageMiddleware** - Records token usage metrics when token tracking is enabled (optional); subagent usage is cached by `tool_call_id` only while token usage is enabled and merged back into the dispatching AIMessage by message position rather than message id
|
||||
13. **TitleMiddleware** - Auto-generates thread title after first complete exchange and normalizes structured message content before prompting the title model
|
||||
14. **MemoryMiddleware** - Queues conversations for async memory update (filters to user + final AI responses)
|
||||
15. **ViewImageMiddleware** - Injects base64 image data before LLM call (conditional on vision support)
|
||||
16. **DeferredToolFilterMiddleware** - Hides deferred (MCP) tool schemas from the bound model using a build-time deferred-name set + catalog hash, reading per-thread promotions from `ThreadState.promoted` (hash-scoped, no ContextVar); a tool becomes bound on subsequent turns after `tool_search` returns its schema (optional, if `tool_search.enabled`)
|
||||
17. **SubagentLimitMiddleware** - Truncates excess `task` tool calls from model response to enforce `MAX_CONCURRENT_SUBAGENTS` limit (optional, if `subagent_enabled`)
|
||||
18. **LoopDetectionMiddleware** - Detects repeated tool-call loops; hard-stop responses clear both structured `tool_calls` and raw provider tool-call metadata before forcing a final text answer
|
||||
19. **ClarificationMiddleware** - Intercepts `ask_clarification` tool calls, interrupts via `Command(goto=END)` (must be last)
|
||||
4. **DanglingToolCallMiddleware** - Injects placeholder ToolMessages for AIMessage tool_calls that lack responses (e.g., due to user interruption)
|
||||
5. **GuardrailMiddleware** - Pre-tool-call authorization via pluggable `GuardrailProvider` protocol (optional, if `guardrails.enabled` in config). Evaluates each tool call and returns error ToolMessage on deny. Three provider options: built-in `AllowlistProvider` (zero deps), OAP policy providers (e.g. `aport-agent-guardrails`), or custom providers. See [docs/GUARDRAILS.md](docs/GUARDRAILS.md) for setup, usage, and how to implement a provider.
|
||||
6. **SummarizationMiddleware** - Context reduction when approaching token limits (optional, if enabled)
|
||||
7. **TodoListMiddleware** - Task tracking with `write_todos` tool (optional, if plan_mode)
|
||||
8. **TitleMiddleware** - Auto-generates thread title after first complete exchange and normalizes structured message content before prompting the title model
|
||||
9. **MemoryMiddleware** - Queues conversations for async memory update (filters to user + final AI responses)
|
||||
10. **ViewImageMiddleware** - Injects base64 image data before LLM call (conditional on vision support)
|
||||
11. **SubagentLimitMiddleware** - Truncates excess `task` tool calls from model response to enforce `MAX_CONCURRENT_SUBAGENTS` limit (optional, if subagent_enabled)
|
||||
12. **ClarificationMiddleware** - Intercepts `ask_clarification` tool calls, interrupts via `Command(goto=END)` (must be last)
|
||||
|
||||
### Configuration System
|
||||
|
||||
@@ -224,10 +181,6 @@ Setup: Copy `config.example.yaml` to `config.yaml` in the **project root** direc
|
||||
|
||||
**Config Caching**: `get_app_config()` caches the parsed config, but automatically reloads it when the resolved config path changes or the file's mtime increases. This keeps Gateway and LangGraph reads aligned with `config.yaml` edits without requiring a manual process restart.
|
||||
|
||||
**Config Hot-Reload Boundary**: Gateway dependencies route through `get_app_config()` on every request, so per-run fields like `models[*].max_tokens`, `summarization.*`, `title.*`, `memory.*`, `subagents.*`, `tools[*]`, and the agent system prompt pick up `config.yaml` edits on the next message. `AppConfig` is intentionally **not** cached on `app.state` — `lifespan()` keeps a local `startup_config` variable for one-shot bootstrap work and passes it to `langgraph_runtime(app, startup_config)`.
|
||||
|
||||
Infrastructure fields are **restart-required**. The authoritative list lives in `packages/harness/deerflow/config/reload_boundary.py::STARTUP_ONLY_FIELDS` and is mirrored by the standardised `"startup-only:"` prefix on the corresponding `Field(description=...)` in `AppConfig`, so IDE hover on those fields surfaces the reason inline (no need to context-switch into this table). Currently registered: `database`, `checkpointer`, `run_events`, `stream_bridge`, `sandbox`, `log_level`, `channels`. Adding a new restart-required field requires updating the registry; drift is pinned by `tests/test_reload_boundary.py`.
|
||||
|
||||
Configuration priority:
|
||||
1. Explicit `config_path` argument
|
||||
2. `DEER_FLOW_CONFIG_PATH` environment variable
|
||||
@@ -249,9 +202,7 @@ Configuration priority:
|
||||
|
||||
### Gateway API (`app/gateway/`)
|
||||
|
||||
FastAPI application on port 8001 with health check at `GET /health`. Set `GATEWAY_ENABLE_DOCS=false` to disable `/docs`, `/redoc`, and `/openapi.json` in production (default: enabled).
|
||||
|
||||
CORS is same-origin by default when requests enter through nginx on port 2026. Split-origin or port-forwarded browser clients must opt in with `GATEWAY_CORS_ORIGINS` (comma-separated exact origins); Gateway `CORSMiddleware` and `CSRFMiddleware` both read that variable so browser CORS and auth-origin checks stay aligned.
|
||||
FastAPI application on port 8001 with health check at `GET /health`.
|
||||
|
||||
**Routers**:
|
||||
|
||||
@@ -264,39 +215,29 @@ CORS is same-origin by default when requests enter through nginx on port 2026. S
|
||||
| **Uploads** (`/api/threads/{id}/uploads`) | `POST /` - upload files (auto-converts PDF/PPT/Excel/Word); `GET /list` - list; `DELETE /{filename}` - delete |
|
||||
| **Threads** (`/api/threads/{id}`) | `DELETE /` - remove DeerFlow-managed local thread data after LangGraph thread deletion; unexpected failures are logged server-side and return a generic 500 detail |
|
||||
| **Artifacts** (`/api/threads/{id}/artifacts`) | `GET /{path}` - serve artifacts; active content types (`text/html`, `application/xhtml+xml`, `image/svg+xml`) are always forced as download attachments to reduce XSS risk; `?download=true` still forces download for other file types |
|
||||
| **Suggestions** (`/api/threads/{id}/suggestions`) | `POST /` - generate follow-up questions; rich list/block model content is normalized and inline reasoning (`<think>...</think>`, including unclosed/truncated blocks from reasoning models like MiniMax-M3) is stripped before JSON parsing |
|
||||
| **Thread Runs** (`/api/threads/{id}/runs`) | `POST /` - create background run; `POST /stream` - create + SSE stream; `POST /wait` - create + block; `GET /` - list runs; `GET /{rid}` - run details; `POST /{rid}/cancel` - cancel; `GET /{rid}/join` - join SSE; `GET /{rid}/messages` - paginated messages `{data, has_more}`; `GET /{rid}/events` - full event stream; `GET /../messages` - thread messages with feedback; `GET /../token-usage` - aggregate tokens |
|
||||
| **Feedback** (`/api/threads/{id}/runs/{rid}/feedback`) | `PUT /` - upsert feedback; `DELETE /` - delete user feedback; `POST /` - create feedback; `GET /` - list feedback; `GET /stats` - aggregate stats; `DELETE /{fid}` - delete specific |
|
||||
| **Runs** (`/api/runs`) | `POST /stream` - stateless run + SSE; `POST /wait` - stateless run + block; `GET /{rid}/messages` - paginated messages by run_id `{data, has_more}` (cursor: `after_seq`/`before_seq`); `GET /{rid}/feedback` - list feedback by run_id |
|
||||
| **Suggestions** (`/api/threads/{id}/suggestions`) | `POST /` - generate follow-up questions; rich list/block model content is normalized before JSON parsing |
|
||||
|
||||
**RunManager / RunStore contract**:
|
||||
- `RunManager.get()` is async; direct callers must `await` it.
|
||||
- When a persistent `RunStore` is configured, `get()` and `list_by_thread()` hydrate historical runs from the store. In-memory records win for the same `run_id` so task, abort, and stream-control state stays attached to active local runs.
|
||||
- `cancel()` and `create_or_reject(..., multitask_strategy="interrupt"|"rollback")` persist interrupted status through `RunStore.update_status()`, matching normal `set_status()` transitions.
|
||||
- Store-only hydrated runs are readable history. If the current worker has no in-memory task/control state for that run, cancellation APIs can return 409 because this worker cannot stop the task.
|
||||
- `POST /wait` (both thread-scoped and `/api/runs/wait`) drains the stream bridge via `wait_for_run_completion()` instead of bare `await record.task`, so it honours the run's `on_disconnect` setting and cancels the background run on real client disconnect rather than returning a stale checkpoint (issue #3265).
|
||||
|
||||
Proxied through nginx: `/api/langgraph/*` → Gateway LangGraph-compatible runtime, all other `/api/*` → Gateway REST APIs.
|
||||
Proxied through nginx: `/api/langgraph/*` → LangGraph, all other `/api/*` → Gateway.
|
||||
|
||||
### Sandbox System (`packages/harness/deerflow/sandbox/`)
|
||||
|
||||
**Interface**: Abstract `Sandbox` with `execute_command`, `read_file`, `write_file`, `list_dir`
|
||||
**Provider Pattern**: `SandboxProvider` with `acquire`, `acquire_async`, `get`, `release` lifecycle. Async agent/tool paths call async sandbox lifecycle hooks so Docker sandbox creation, discovery, cross-process locking, readiness polling, and release stay off the event loop.
|
||||
**Provider Pattern**: `SandboxProvider` with `acquire`, `get`, `release` lifecycle
|
||||
**Implementations**:
|
||||
- `LocalSandboxProvider` - Local filesystem execution. `acquire(thread_id)` returns a per-thread `LocalSandbox` (id `local:{thread_id}`) whose `path_mappings` resolve `/mnt/user-data/{workspace,uploads,outputs}` and `/mnt/acp-workspace` to that thread's host directories, so the public `Sandbox` API honours the `/mnt/user-data` contract uniformly with AIO. `acquire()` / `acquire(None)` keeps the legacy generic singleton (id `local`) for callers without a thread context. Per-thread sandboxes are held in an LRU cache (default 256 entries) guarded by a `threading.Lock`.
|
||||
- `LocalSandboxProvider` - Singleton local filesystem execution with path mappings
|
||||
- `AioSandboxProvider` (`packages/harness/deerflow/community/`) - Docker-based isolation
|
||||
|
||||
**Virtual Path System**:
|
||||
- Agent sees: `/mnt/user-data/{workspace,uploads,outputs}`, `/mnt/skills`
|
||||
- Physical: `backend/.deer-flow/users/{user_id}/threads/{thread_id}/user-data/...`, `deer-flow/skills/`
|
||||
- Translation: `LocalSandboxProvider` builds per-thread `PathMapping`s for the user-data prefixes at acquire time; `tools.py` keeps `replace_virtual_path()` / `replace_virtual_paths_in_command()` as a defense-in-depth layer (and for path validation). AIO has the directories volume-mounted at the same virtual paths inside its container, so both implementations accept `/mnt/user-data/...` natively.
|
||||
- Detection: `is_local_sandbox()` accepts both `sandbox_id == "local"` (legacy / no-thread) and `sandbox_id.startswith("local:")` (per-thread)
|
||||
- Physical: `backend/.deer-flow/threads/{thread_id}/user-data/...`, `deer-flow/skills/`
|
||||
- Translation: `replace_virtual_path()` / `replace_virtual_paths_in_command()`
|
||||
- Detection: `is_local_sandbox()` checks `sandbox_id == "local"`
|
||||
|
||||
**Sandbox Tools** (in `packages/harness/deerflow/sandbox/tools.py`):
|
||||
- `bash` - Execute commands with path translation and error handling
|
||||
- `ls` - Directory listing (tree format, max 2 levels)
|
||||
- `read_file` - Read file contents with optional line range
|
||||
- `write_file` - Write/append to files, creates directories; overwrites by default and exposes the `append` argument in the model-facing schema for end-of-file writes
|
||||
- `write_file` - Write/append to files, creates directories
|
||||
- `str_replace` - Substring replacement (single or all occurrences); same-path serialization is scoped to `(sandbox.id, path)` so isolated sandboxes do not contend on identical virtual paths inside one process
|
||||
|
||||
### Subagent System (`packages/harness/deerflow/subagents/`)
|
||||
@@ -306,7 +247,6 @@ Proxied through nginx: `/api/langgraph/*` → Gateway LangGraph-compatible runti
|
||||
**Concurrency**: `MAX_CONCURRENT_SUBAGENTS = 3` enforced by `SubagentLimitMiddleware` (truncates excess tool calls in `after_model`), 15-minute timeout
|
||||
**Flow**: `task()` tool → `SubagentExecutor` → background thread → poll 5s → SSE events → result
|
||||
**Events**: `task_started`, `task_running`, `task_completed`/`task_failed`/`task_timed_out`
|
||||
**Deferred MCP tools** (if `tool_search.enabled`): `SubagentExecutor._build_initial_state` assembles deferral after policy filtering via the shared `assemble_deferred_tools` (fail-closed), appends the `tool_search` tool, injects the `<available-deferred-tools>` section into the subagent's `SystemMessage`, and threads the setup to `_create_agent`, which attaches `DeferredToolFilterMiddleware` through `build_subagent_runtime_middlewares(deferred_setup=...)`. Subagents thus withhold full MCP schemas until promotion, same as the lead agent; each task run gets a fresh `ThreadState` so promotion is isolated per run
|
||||
|
||||
### Tool System (`packages/harness/deerflow/tools/`)
|
||||
|
||||
@@ -317,10 +257,8 @@ Proxied through nginx: `/api/langgraph/*` → Gateway LangGraph-compatible runti
|
||||
- `present_files` - Make output files visible to user (only `/mnt/user-data/outputs`)
|
||||
- `ask_clarification` - Request clarification (intercepted by ClarificationMiddleware → interrupts)
|
||||
- `view_image` - Read image as base64 (added only if model supports vision)
|
||||
- `setup_agent` - Bootstrap-only: persist a brand-new custom agent's `SOUL.md` and `config.yaml`. Bound only when `is_bootstrap=True`.
|
||||
- `update_agent` - Custom-agent-only: persist self-updates to the current agent's `SOUL.md` / `config.yaml` from inside a normal chat (partial update + atomic write). Bound when `agent_name` is set and `is_bootstrap=False`.
|
||||
4. **Subagent tool** (if enabled):
|
||||
- `task` - Delegate to subagent (description, prompt, subagent_type)
|
||||
- `task` - Delegate to subagent (description, prompt, subagent_type, max_turns)
|
||||
|
||||
**Community tools** (`packages/harness/deerflow/community/`):
|
||||
- `tavily/` - Web search (5 results default) and web fetch (4KB limit)
|
||||
@@ -331,7 +269,7 @@ Proxied through nginx: `/api/langgraph/*` → Gateway LangGraph-compatible runti
|
||||
- `invoke_acp_agent` - Invokes external ACP-compatible agents from `config.yaml`
|
||||
- ACP launchers must be real ACP adapters. The standard `codex` CLI is not ACP-compatible by itself; configure a wrapper such as `npx -y @zed-industries/codex-acp` or an installed `codex-acp` binary
|
||||
- Missing ACP executables now return an actionable error message instead of a raw `[Errno 2]`
|
||||
- Each ACP agent uses a per-thread workspace at `{base_dir}/users/{user_id}/threads/{thread_id}/acp-workspace/`. The workspace is accessible to the lead agent via the virtual path `/mnt/acp-workspace/` (read-only). In docker sandbox mode, the directory is volume-mounted into the container at `/mnt/acp-workspace` (read-only); in local sandbox mode, path translation is handled by `tools.py`
|
||||
- Each ACP agent uses a per-thread workspace at `{base_dir}/threads/{thread_id}/acp-workspace/`. The workspace is accessible to the lead agent via the virtual path `/mnt/acp-workspace/` (read-only). In docker sandbox mode, the directory is volume-mounted into the container at `/mnt/acp-workspace` (read-only); in local sandbox mode, path translation is handled by `tools.py`
|
||||
- `image_search/` - Image search via DuckDuckGo
|
||||
|
||||
### MCP System (`packages/harness/deerflow/mcp/`)
|
||||
@@ -341,7 +279,7 @@ Proxied through nginx: `/api/langgraph/*` → Gateway LangGraph-compatible runti
|
||||
- **Cache invalidation**: Detects config file changes via mtime comparison
|
||||
- **Transports**: stdio (command-based), SSE, HTTP
|
||||
- **OAuth (HTTP/SSE)**: Supports token endpoint flows (`client_credentials`, `refresh_token`) with automatic token refresh + Authorization header injection
|
||||
- **Runtime updates**: Gateway API saves to extensions_config.json; the Gateway-embedded runtime detects changes via mtime
|
||||
- **Runtime updates**: Gateway API saves to extensions_config.json; LangGraph detects via mtime
|
||||
|
||||
### Skills System (`packages/harness/deerflow/skills/`)
|
||||
|
||||
@@ -349,7 +287,6 @@ Proxied through nginx: `/api/langgraph/*` → Gateway LangGraph-compatible runti
|
||||
- **Format**: Directory with `SKILL.md` (YAML frontmatter: name, description, license, allowed-tools)
|
||||
- **Loading**: `load_skills()` recursively scans `skills/{public,custom}` for `SKILL.md`, parses metadata, and reads enabled state from extensions_config.json
|
||||
- **Injection**: Enabled skills listed in agent system prompt with container paths
|
||||
- **Slash activation**: `/skill-name task` loads that enabled skill's `SKILL.md` for the current model call only. The resolver rejects leading whitespace, missing separators, reserved channel commands (`/new`, `/help`, `/bootstrap`, `/status`, `/models`, `/memory`), disabled skills, and skills outside a custom agent's whitelist.
|
||||
- **Installation**: `POST /api/skills/install` extracts .skill ZIP archive to custom/ directory
|
||||
|
||||
### Model Factory (`packages/harness/deerflow/models/factory.py`)
|
||||
@@ -369,10 +306,9 @@ Proxied through nginx: `/api/langgraph/*` → Gateway LangGraph-compatible runti
|
||||
|
||||
### IM Channels System (`app/channels/`)
|
||||
|
||||
Bridges external messaging platforms (Feishu, Slack, Telegram, DingTalk) to the DeerFlow agent via Gateway's LangGraph-compatible API.
|
||||
Bridges external messaging platforms (Feishu, Slack, Telegram) to the DeerFlow agent via the LangGraph Server.
|
||||
|
||||
|
||||
**Architecture**: Channels communicate with Gateway through the `langgraph-sdk` HTTP client (same as the frontend), ensuring threads are created and managed server-side. The internal SDK client injects process-local internal auth plus a matching CSRF cookie/header pair so Gateway accepts state-changing thread/run requests from channel workers without relying on browser session cookies.
|
||||
**Architecture**: Channels communicate with the LangGraph Server through `langgraph-sdk` HTTP client (same as the frontend), ensuring threads are created and managed server-side.
|
||||
|
||||
**Components**:
|
||||
- `message_bus.py` - Async pub/sub hub (`InboundMessage` → queue → dispatcher; `OutboundMessage` → callbacks → channels)
|
||||
@@ -380,52 +316,40 @@ Bridges external messaging platforms (Feishu, Slack, Telegram, DingTalk) to the
|
||||
- `manager.py` - Core dispatcher: creates threads via `client.threads.create()`, routes commands, keeps Slack/Telegram on `client.runs.wait()`, and uses `client.runs.stream(["messages-tuple", "values"])` for Feishu incremental outbound updates
|
||||
- `base.py` - Abstract `Channel` base class (start/stop/send lifecycle)
|
||||
- `service.py` - Manages lifecycle of all configured channels from `config.yaml`
|
||||
- `slack.py` / `feishu.py` / `telegram.py` / `dingtalk.py` - Platform-specific implementations (`feishu.py` tracks the running card `message_id` in memory and patches the same card in place; `dingtalk.py` optionally uses AI Card streaming for in-place updates when `card_template_id` is configured)
|
||||
- `slack.py` / `feishu.py` / `telegram.py` - Platform-specific implementations (`feishu.py` tracks the running card `message_id` in memory and patches the same card in place)
|
||||
|
||||
**Message Flow**:
|
||||
1. External platform -> Channel impl -> `MessageBus.publish_inbound()`
|
||||
2. `ChannelManager._dispatch_loop()` consumes from queue
|
||||
3. For chat: look up/create thread through Gateway's LangGraph-compatible API
|
||||
3. For chat: look up/create thread on LangGraph Server
|
||||
4. Feishu chat: `runs.stream()` → accumulate AI text → publish multiple outbound updates (`is_final=False`) → publish final outbound (`is_final=True`)
|
||||
5. Slack/Telegram chat: `runs.wait()` → extract final response → publish outbound
|
||||
6. Feishu channel sends one running reply card up front, then patches the same card for each outbound update (card JSON sets `config.update_multi=true` for Feishu's patch API requirement)
|
||||
7. DingTalk AI Card mode (when `card_template_id` configured): `runs.stream()` → create card with initial text → stream updates via `PUT /v1.0/card/streaming` → finalize on `is_final=True`. Falls back to `sampleMarkdown` if card creation or streaming fails
|
||||
8. For commands (`/new`, `/status`, `/models`, `/memory`, `/help`): handle locally or query Gateway API
|
||||
9. Outbound → channel callbacks → platform reply
|
||||
7. For commands (`/new`, `/status`, `/models`, `/memory`, `/help`): handle locally or query Gateway API
|
||||
8. Outbound → channel callbacks → platform reply
|
||||
|
||||
**Configuration** (`config.yaml` -> `channels`):
|
||||
- `langgraph_url` - LangGraph-compatible Gateway API base URL (default: `http://localhost:8001/api`)
|
||||
- `langgraph_url` - LangGraph Server URL (default: `http://localhost:2024`)
|
||||
- `gateway_url` - Gateway API URL for auxiliary commands (default: `http://localhost:8001`)
|
||||
- In Docker Compose, IM channels run inside the `gateway` container, so `localhost` points back to that container. Use `http://gateway:8001/api` for `langgraph_url` and `http://gateway:8001` for `gateway_url`, or set `DEER_FLOW_CHANNELS_LANGGRAPH_URL` / `DEER_FLOW_CHANNELS_GATEWAY_URL`.
|
||||
- Per-channel configs: `feishu` (app_id, app_secret), `slack` (bot_token, app_token), `telegram` (bot_token), `dingtalk` (client_id, client_secret, optional `card_template_id` for AI Card streaming)
|
||||
|
||||
- In Docker Compose, IM channels run inside the `gateway` container, so `localhost` points back to that container. Use `http://langgraph:2024` / `http://gateway:8001`, or set `DEER_FLOW_CHANNELS_LANGGRAPH_URL` / `DEER_FLOW_CHANNELS_GATEWAY_URL`.
|
||||
- Per-channel configs: `feishu` (app_id, app_secret), `slack` (bot_token, app_token), `telegram` (bot_token)
|
||||
|
||||
### Memory System (`packages/harness/deerflow/agents/memory/`)
|
||||
|
||||
**Components**:
|
||||
- `updater.py` - LLM-based memory updates with fact extraction, whitespace-normalized fact deduplication (trims leading/trailing whitespace before comparing), and atomic file I/O
|
||||
- `queue.py` - Debounced update queue (per-thread deduplication, configurable wait time); captures `user_id` at enqueue time so it survives the `threading.Timer` boundary
|
||||
- `queue.py` - Debounced update queue (per-thread deduplication, configurable wait time)
|
||||
- `prompt.py` - Prompt templates for memory updates
|
||||
- `storage.py` - File-based storage with per-user isolation; cache keyed by `(user_id, agent_name)` tuple
|
||||
|
||||
**Per-User Isolation**:
|
||||
- Memory is stored per-user at `{base_dir}/users/{user_id}/memory.json`
|
||||
- Per-agent per-user memory at `{base_dir}/users/{user_id}/agents/{agent_name}/memory.json`
|
||||
- Custom agent definitions (`SOUL.md` + `config.yaml`) are also per-user at `{base_dir}/users/{user_id}/agents/{agent_name}/`. The legacy shared layout `{base_dir}/agents/{agent_name}/` remains read-only fallback for unmigrated installations
|
||||
- `user_id` is resolved via `get_effective_user_id()` from `deerflow.runtime.user_context`
|
||||
- In no-auth mode, `user_id` defaults to `"default"` (constant `DEFAULT_USER_ID`)
|
||||
- Absolute `storage_path` in config opts out of per-user isolation
|
||||
- **Migration**: Run `PYTHONPATH=. python scripts/migrate_user_isolation.py` to move legacy `memory.json`, `threads/`, and `agents/` into per-user layout. Supports `--dry-run` (preview changes) and `--user-id USER_ID` (assign unowned legacy data to a user, defaults to `default`).
|
||||
|
||||
**Data Structure** (stored in `{base_dir}/users/{user_id}/memory.json`):
|
||||
**Data Structure** (stored in `backend/.deer-flow/memory.json`):
|
||||
- **User Context**: `workContext`, `personalContext`, `topOfMind` (1-3 sentence summaries)
|
||||
- **History**: `recentMonths`, `earlierContext`, `longTermBackground`
|
||||
- **Facts**: Discrete facts with `id`, `content`, `category` (preference/knowledge/context/behavior/goal), `confidence` (0-1), `createdAt`, `source`
|
||||
|
||||
**Workflow**:
|
||||
1. `MemoryMiddleware` filters messages (user inputs + final AI responses), captures `user_id` via `get_effective_user_id()`, and queues conversation with the captured `user_id`
|
||||
1. `MemoryMiddleware` filters messages (user inputs + final AI responses) and queues conversation
|
||||
2. Queue debounces (30s default), batches updates, deduplicates per-thread
|
||||
3. Background thread invokes LLM to extract context updates and facts, using the stored `user_id` (not the contextvar, which is unavailable on timer threads)
|
||||
3. Background thread invokes LLM to extract context updates and facts
|
||||
4. Applies updates atomically (temp file + rename) with cache invalidation, skipping duplicate fact content before append
|
||||
5. Next interaction injects top 15 facts + context into `<memory>` tags in system prompt
|
||||
|
||||
@@ -433,7 +357,7 @@ Focused regression coverage for the updater lives in `backend/tests/test_memory_
|
||||
|
||||
**Configuration** (`config.yaml` → `memory`):
|
||||
- `enabled` / `injection_enabled` - Master switches
|
||||
- `storage_path` - Path to memory.json (absolute path opts out of per-user isolation)
|
||||
- `storage_path` - Path to memory.json
|
||||
- `debounce_seconds` - Wait time before processing (default: 30)
|
||||
- `model_name` - LLM for updates (null = default model)
|
||||
- `max_facts` / `fact_confidence_threshold` - Fact storage limits (100 / 0.7)
|
||||
@@ -444,24 +368,6 @@ Focused regression coverage for the updater lives in `backend/tests/test_memory_
|
||||
- `resolve_variable(path)` - Import module and return variable (e.g., `module.path:variable_name`)
|
||||
- `resolve_class(path, base_class)` - Import and validate class against base class
|
||||
|
||||
### Tracing System (`packages/harness/deerflow/tracing/`)
|
||||
|
||||
LangSmith and Langfuse are both supported. The wiring lives in two layers:
|
||||
|
||||
- `factory.py::build_tracing_callbacks()` — returns the LangChain `CallbackHandler` list for the providers currently enabled via env vars (`LANGSMITH_TRACING`, `LANGFUSE_TRACING`, etc.). The handlers are attached at the **graph invocation root** for in-graph runs (`make_lead_agent` and `DeerFlowClient.stream` both append them to `config["callbacks"]` before invoking the graph) so a single run produces one trace with all node / LLM / tool calls as child spans. Standalone callers — anything that invokes a model outside such a graph (e.g. `MemoryUpdater`) — keep `create_chat_model`'s default `attach_tracing=True`, which falls back to model-level callback attachment.
|
||||
- `metadata.py::build_langfuse_trace_metadata()` — builds the Langfuse-reserved trace attributes for `RunnableConfig.metadata`. The Langfuse v4 `langchain.CallbackHandler` lifts these onto the root trace (see its `_parse_langfuse_trace_attributes`), but only when it sees `on_chain_start(parent_run_id=None)` — which is why the callbacks have to live at the graph root, not the model.
|
||||
|
||||
**Trace-attribute injection points**: both `runtime/runs/worker.py::run_agent` (gateway path) and `client.py::DeerFlowClient.stream` (embedded path) merge the metadata into `config["metadata"]` right before constructing the graph. Caller-supplied keys win via `setdefault`, so an external `session_id` override is preserved. Field mapping:
|
||||
|
||||
| Langfuse field | Source |
|
||||
|-----------------------|----------------------------------------------|
|
||||
| `langfuse_session_id` | LangGraph `thread_id` |
|
||||
| `langfuse_user_id` | `get_effective_user_id()` (`default` in no-auth) |
|
||||
| `langfuse_trace_name` | `RunRecord.assistant_id` / client `agent_name` (defaults to `lead-agent`) |
|
||||
| `langfuse_tags` | `env:<DEER_FLOW_ENV>` + `model:<model_name>` |
|
||||
|
||||
Returns `{}` when Langfuse is not in the enabled providers — LangSmith-only deployments are unaffected. Set `DEER_FLOW_ENV` (or `ENVIRONMENT`) to tag traces by deployment environment. Tests live in `tests/test_tracing_factory.py`, `tests/test_tracing_metadata.py`, `tests/test_worker_langfuse_metadata.py`, and `tests/test_client_langfuse_metadata.py`.
|
||||
|
||||
### Config Schema
|
||||
|
||||
**`config.yaml`** key sections:
|
||||
@@ -486,16 +392,16 @@ Both can be modified at runtime via Gateway API endpoints or `DeerFlowClient` me
|
||||
|
||||
`DeerFlowClient` provides direct in-process access to all DeerFlow capabilities without HTTP services. All return types align with the Gateway API response schemas, so consumer code works identically in HTTP and embedded modes.
|
||||
|
||||
**Architecture**: Imports the same `deerflow` modules that Gateway API uses. Shares the same config files and data directories. No FastAPI dependency.
|
||||
**Architecture**: Imports the same `deerflow` modules that LangGraph Server and Gateway API use. Shares the same config files and data directories. No FastAPI dependency.
|
||||
|
||||
**Agent Conversation**:
|
||||
**Agent Conversation** (replaces LangGraph Server):
|
||||
- `chat(message, thread_id)` — synchronous, accumulates streaming deltas per message-id and returns the final AI text
|
||||
- `stream(message, thread_id)` — subscribes to LangGraph `stream_mode=["values", "messages", "custom"]` and yields `StreamEvent`:
|
||||
- `"values"` — full state snapshot (title, messages, artifacts); AI text already delivered via `messages` mode is **not** re-synthesized here to avoid duplicate deliveries
|
||||
- `"messages-tuple"` — per-chunk update: for AI text this is a **delta** (concat per `id` to rebuild the full message); tool calls and tool results are emitted once each
|
||||
- `"custom"` — forwarded from `StreamWriter`
|
||||
- `"end"` — stream finished (carries cumulative `usage` counted once per message id)
|
||||
- Agent created lazily via `create_agent()` + `build_middlewares()`, same as `make_lead_agent`
|
||||
- Agent created lazily via `create_agent()` + `_build_middlewares()`, same as `make_lead_agent`
|
||||
- Supports `checkpointer` parameter for state persistence across turns
|
||||
- `reset_agent()` forces agent recreation (e.g. after memory or skill changes)
|
||||
- See [docs/STREAMING.md](docs/STREAMING.md) for the full design: why Gateway and DeerFlowClient are parallel paths, LangGraph's `stream_mode` semantics, the per-id dedup invariants, and regression testing strategy
|
||||
@@ -551,15 +457,20 @@ This starts all services and makes the application available at `http://localhos
|
||||
| | **Local Foreground** | **Local Daemon** | **Docker Dev** | **Docker Prod** |
|
||||
|---|---|---|---|---|
|
||||
| **Dev** | `./scripts/serve.sh --dev`<br/>`make dev` | `./scripts/serve.sh --dev --daemon`<br/>`make dev-daemon` | `./scripts/docker.sh start`<br/>`make docker-start` | — |
|
||||
| **Dev + Gateway** | `./scripts/serve.sh --dev --gateway`<br/>`make dev-pro` | `./scripts/serve.sh --dev --gateway --daemon`<br/>`make dev-daemon-pro` | `./scripts/docker.sh start --gateway`<br/>`make docker-start-pro` | — |
|
||||
| **Prod** | `./scripts/serve.sh --prod`<br/>`make start` | `./scripts/serve.sh --prod --daemon`<br/>`make start-daemon` | — | `./scripts/deploy.sh`<br/>`make up` |
|
||||
| **Prod + Gateway** | `./scripts/serve.sh --prod --gateway`<br/>`make start-pro` | `./scripts/serve.sh --prod --gateway --daemon`<br/>`make start-daemon-pro` | — | `./scripts/deploy.sh --gateway`<br/>`make up-pro` |
|
||||
|
||||
| Action | Local | Docker Dev | Docker Prod |
|
||||
|---|---|---|---|
|
||||
| **Stop** | `./scripts/serve.sh --stop`<br/>`make stop` | `./scripts/docker.sh stop`<br/>`make docker-stop` | `./scripts/deploy.sh down`<br/>`make down` |
|
||||
| **Restart** | `./scripts/serve.sh --restart [flags]` | `./scripts/docker.sh restart` | — |
|
||||
|
||||
Gateway mode embeds the agent runtime in Gateway, no LangGraph server.
|
||||
|
||||
**Nginx routing**:
|
||||
- `/api/langgraph/*` → Gateway embedded runtime (8001), rewritten to `/api/*`
|
||||
- Standard mode: `/api/langgraph/*` → LangGraph Server (2024)
|
||||
- Gateway mode: `/api/langgraph/*` → Gateway embedded runtime (8001) (via envsubst)
|
||||
- `/api/*` (other) → Gateway API (8001)
|
||||
- `/` (non-API) → Frontend (3000)
|
||||
|
||||
@@ -568,11 +479,15 @@ This starts all services and makes the application available at `http://localhos
|
||||
From the **backend** directory:
|
||||
|
||||
```bash
|
||||
# Gateway API
|
||||
# Terminal 1: LangGraph server
|
||||
make dev
|
||||
|
||||
# Terminal 2: Gateway API
|
||||
make gateway
|
||||
```
|
||||
|
||||
Direct access (without nginx):
|
||||
- LangGraph: `http://localhost:2024`
|
||||
- Gateway: `http://localhost:8001`
|
||||
|
||||
### Frontend Configuration
|
||||
@@ -593,7 +508,6 @@ Multi-file upload with automatic document conversion:
|
||||
- Rejects directory inputs before copying so uploads stay all-or-nothing
|
||||
- Reuses one conversion worker per request when called from an active event loop
|
||||
- Files stored in thread-isolated directories
|
||||
- Duplicate filenames in a single upload request are auto-renamed with `_N` suffixes so later files do not truncate earlier files
|
||||
- Agent receives uploaded file list via `UploadsMiddleware`
|
||||
|
||||
See [docs/FILE_UPLOAD.md](docs/FILE_UPLOAD.md) for details.
|
||||
|
||||
@@ -56,8 +56,11 @@ export OPENAI_API_KEY="your-api-key"
|
||||
### Run the Development Server
|
||||
|
||||
```bash
|
||||
# Gateway API + embedded agent runtime
|
||||
# Terminal 1: LangGraph server
|
||||
make dev
|
||||
|
||||
# Terminal 2: Gateway API
|
||||
make gateway
|
||||
```
|
||||
|
||||
## Project Structure
|
||||
|
||||
+4
-18
@@ -13,9 +13,6 @@ FROM python:3.12-slim-bookworm AS builder
|
||||
ARG NODE_MAJOR=22
|
||||
ARG APT_MIRROR
|
||||
ARG UV_INDEX_URL
|
||||
# Optional extras to install (e.g. "postgres" for PostgreSQL support)
|
||||
# Usage: docker build --build-arg UV_EXTRAS=postgres ...
|
||||
ARG UV_EXTRAS
|
||||
|
||||
# Optionally override apt mirror for restricted networks (e.g. APT_MIRROR=mirrors.aliyun.com)
|
||||
RUN if [ -n "${APT_MIRROR}" ]; then \
|
||||
@@ -46,15 +43,8 @@ WORKDIR /app
|
||||
COPY backend ./backend
|
||||
|
||||
# Install dependencies with cache mount
|
||||
# When UV_EXTRAS is set (e.g. "postgres"), installs optional dependencies.
|
||||
RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
sh -c "cd backend && UV_INDEX_URL=${UV_INDEX_URL:-https://pypi.org/simple} uv sync ${UV_EXTRAS:+--extra $UV_EXTRAS}"
|
||||
|
||||
# UTF-8 locale prevents UnicodeEncodeError on Chinese/emoji content in minimal
|
||||
# containers where locale configuration may be missing and the default encoding is not UTF-8.
|
||||
ENV LANG=C.UTF-8
|
||||
ENV LC_ALL=C.UTF-8
|
||||
ENV PYTHONIOENCODING=utf-8
|
||||
sh -c "cd backend && UV_INDEX_URL=${UV_INDEX_URL:-https://pypi.org/simple} uv sync"
|
||||
|
||||
# ── Stage 2: Dev ──────────────────────────────────────────────────────────────
|
||||
# Retains compiler toolchain from builder so startup-time `uv sync` can build
|
||||
@@ -64,7 +54,7 @@ FROM builder AS dev
|
||||
# Install Docker CLI (for DooD: allows starting sandbox containers via host Docker socket)
|
||||
COPY --from=docker:cli /usr/local/bin/docker /usr/local/bin/docker
|
||||
|
||||
EXPOSE 8001
|
||||
EXPOSE 8001 2024
|
||||
|
||||
CMD ["sh", "-c", "cd backend && PYTHONPATH=. uv run uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001"]
|
||||
|
||||
@@ -72,10 +62,6 @@ CMD ["sh", "-c", "cd backend && PYTHONPATH=. uv run uvicorn app.gateway.app:app
|
||||
# Clean image without build-essential — reduces size (~200 MB) and attack surface.
|
||||
FROM python:3.12-slim-bookworm
|
||||
|
||||
ENV LANG=C.UTF-8
|
||||
ENV LC_ALL=C.UTF-8
|
||||
ENV PYTHONIOENCODING=utf-8
|
||||
|
||||
# Copy Node.js runtime from builder (provides npx for MCP servers)
|
||||
COPY --from=builder /usr/bin/node /usr/bin/node
|
||||
COPY --from=builder /usr/lib/node_modules /usr/lib/node_modules
|
||||
@@ -94,8 +80,8 @@ WORKDIR /app
|
||||
# Copy backend with pre-built virtualenv from builder
|
||||
COPY --from=builder /app/backend ./backend
|
||||
|
||||
# Expose Gateway API port.
|
||||
EXPOSE 8001
|
||||
# Expose ports (gateway: 8001, langgraph: 2024)
|
||||
EXPOSE 8001 2024
|
||||
|
||||
# Default command (can be overridden in docker-compose)
|
||||
CMD ["sh", "-c", "cd backend && PYTHONPATH=. uv run --no-sync uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001"]
|
||||
|
||||
+3
-9
@@ -2,16 +2,13 @@ install:
|
||||
uv sync
|
||||
|
||||
dev:
|
||||
PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001 --reload
|
||||
uv run langgraph dev --no-browser --no-reload --n-jobs-per-worker 10
|
||||
|
||||
gateway:
|
||||
PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001
|
||||
PYTHONPATH=. uv run uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001
|
||||
|
||||
test:
|
||||
PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run pytest tests/ -v
|
||||
|
||||
test-blocking-io:
|
||||
PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run pytest tests/blocking_io -q --tb=short
|
||||
PYTHONPATH=. uv run pytest tests/ -v
|
||||
|
||||
lint:
|
||||
uvx ruff check .
|
||||
@@ -19,6 +16,3 @@ lint:
|
||||
|
||||
format:
|
||||
uvx ruff check . --fix && uvx ruff format .
|
||||
|
||||
detect-blocking-io:
|
||||
@PYTHONPATH=. PYTHONIOENCODING=utf-8 PYTHONUTF8=1 uv run python ../scripts/detect_blocking_io_static.py --output ../.deer-flow/blocking-io-findings.json
|
||||
|
||||
+33
-42
@@ -11,26 +11,31 @@ DeerFlow is a LangGraph-based AI super agent with sandbox execution, persistent
|
||||
│ Nginx (Port 2026) │
|
||||
│ Unified reverse proxy │
|
||||
└───────┬──────────────────┬───────────┘
|
||||
│
|
||||
/api/langgraph/* │ /api/* (other)
|
||||
rewritten to /api/* │
|
||||
▼
|
||||
┌────────────────────────────────────────┐
|
||||
│ Gateway API (8001) │
|
||||
│ FastAPI REST + agent runtime │
|
||||
│ │
|
||||
│ Models, MCP, Skills, Memory, Uploads, │
|
||||
│ Artifacts, Threads, Runs, Streaming │
|
||||
│ │
|
||||
│ ┌────────────────────────────────────┐ │
|
||||
│ │ Lead Agent │ │
|
||||
│ │ Middleware Chain, Tools, Subagents │ │
|
||||
│ └────────────────────────────────────┘ │
|
||||
└────────────────────────────────────────┘
|
||||
/api/langgraph/* │ │ /api/* (other)
|
||||
▼ ▼
|
||||
┌────────────────────┐ ┌────────────────────────┐
|
||||
│ LangGraph Server │ │ Gateway API (8001) │
|
||||
│ (Port 2024) │ │ FastAPI REST │
|
||||
│ │ │ │
|
||||
│ ┌────────────────┐ │ │ Models, MCP, Skills, │
|
||||
│ │ Lead Agent │ │ │ Memory, Uploads, │
|
||||
│ │ ┌──────────┐ │ │ │ Artifacts │
|
||||
│ │ │Middleware│ │ │ └────────────────────────┘
|
||||
│ │ │ Chain │ │ │
|
||||
│ │ └──────────┘ │ │
|
||||
│ │ ┌──────────┐ │ │
|
||||
│ │ │ Tools │ │ │
|
||||
│ │ └──────────┘ │ │
|
||||
│ │ ┌──────────┐ │ │
|
||||
│ │ │Subagents │ │ │
|
||||
│ │ └──────────┘ │ │
|
||||
│ └────────────────┘ │
|
||||
└────────────────────┘
|
||||
```
|
||||
|
||||
**Request Routing** (via Nginx):
|
||||
- `/api/langgraph/*` → Gateway LangGraph-compatible API - agent interactions, threads, streaming
|
||||
- `/api/langgraph/*` → LangGraph Server - agent interactions, threads, streaming
|
||||
- `/api/*` (other) → Gateway API - models, MCP, skills, memory, artifacts, uploads, thread-local cleanup
|
||||
- `/` (non-API) → Frontend - Next.js web interface
|
||||
|
||||
@@ -69,12 +74,12 @@ Middlewares execute in strict order, each handling a specific concern:
|
||||
Per-thread isolated execution with virtual path translation:
|
||||
|
||||
- **Abstract interface**: `execute_command`, `read_file`, `write_file`, `list_dir`
|
||||
- **Providers**: `LocalSandboxProvider` (filesystem) and `AioSandboxProvider` (Docker, in community/). Async runtime paths use async sandbox lifecycle hooks so startup, readiness polling, and release do not block the event loop.
|
||||
- **Providers**: `LocalSandboxProvider` (filesystem) and `AioSandboxProvider` (Docker, in community/)
|
||||
- **Virtual paths**: `/mnt/user-data/{workspace,uploads,outputs}` → thread-specific physical directories
|
||||
- **Skills path**: `/mnt/skills` → `deer-flow/skills/` directory
|
||||
- **Skills loading**: Recursively discovers nested `SKILL.md` files under `skills/{public,custom}` and preserves nested container paths
|
||||
- **File-write safety**: `str_replace` serializes read-modify-write per `(sandbox.id, path)` so isolated sandboxes keep concurrency even when virtual paths match
|
||||
- **Tools**: `bash`, `ls`, `read_file`, `write_file`, `str_replace` (`write_file` overwrites by default and exposes `append` for end-of-file writes; `bash` is disabled by default when using `LocalSandboxProvider`; use `AioSandboxProvider` for isolated shell access)
|
||||
- **Tools**: `bash`, `ls`, `read_file`, `write_file`, `str_replace` (`bash` is disabled by default when using `LocalSandboxProvider`; use `AioSandboxProvider` for isolated shell access)
|
||||
|
||||
### Subagent System
|
||||
|
||||
@@ -119,7 +124,7 @@ FastAPI application providing REST endpoints for frontend integration:
|
||||
| `POST /api/memory/reload` | Force memory reload |
|
||||
| `GET /api/memory/config` | Memory configuration |
|
||||
| `GET /api/memory/status` | Combined config + data |
|
||||
| `POST /api/threads/{id}/uploads` | Upload files (auto-converts PDF/PPT/Excel/Word to Markdown, rejects directory paths, auto-renames duplicate filenames in one request) |
|
||||
| `POST /api/threads/{id}/uploads` | Upload files (auto-converts PDF/PPT/Excel/Word to Markdown, rejects directory paths) |
|
||||
| `GET /api/threads/{id}/uploads/list` | List uploaded files |
|
||||
| `DELETE /api/threads/{id}` | Delete DeerFlow-managed local thread data after LangGraph thread deletion; unexpected failures are logged server-side and return a generic 500 detail |
|
||||
| `GET /api/threads/{id}/artifacts/{path}` | Serve generated artifacts |
|
||||
@@ -188,7 +193,7 @@ export OPENAI_API_KEY="your-api-key-here"
|
||||
**Full Application** (from project root):
|
||||
|
||||
```bash
|
||||
make dev # Starts Gateway + Frontend + Nginx
|
||||
make dev # Starts LangGraph + Gateway + Frontend + Nginx
|
||||
```
|
||||
|
||||
Access at: http://localhost:2026
|
||||
@@ -196,11 +201,14 @@ Access at: http://localhost:2026
|
||||
**Backend Only** (from backend directory):
|
||||
|
||||
```bash
|
||||
# Gateway API + embedded agent runtime
|
||||
# Terminal 1: LangGraph server
|
||||
make dev
|
||||
|
||||
# Terminal 2: Gateway API
|
||||
make gateway
|
||||
```
|
||||
|
||||
Direct access: Gateway at http://localhost:8001
|
||||
Direct access: LangGraph at http://localhost:2024, Gateway at http://localhost:8001
|
||||
|
||||
---
|
||||
|
||||
@@ -236,16 +244,12 @@ backend/
|
||||
│ └── utils/ # Utilities
|
||||
├── docs/ # Documentation
|
||||
├── tests/ # Test suite
|
||||
├── langgraph.json # LangGraph graph registry for tooling/Studio compatibility
|
||||
├── langgraph.json # LangGraph server configuration
|
||||
├── pyproject.toml # Python dependencies
|
||||
├── Makefile # Development commands
|
||||
└── Dockerfile # Container build
|
||||
```
|
||||
|
||||
`langgraph.json` is not the default service entrypoint. The scripts and Docker
|
||||
deployments run the Gateway embedded runtime; the file is kept for LangGraph
|
||||
tooling, Studio, or direct LangGraph Server compatibility.
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
@@ -358,11 +362,10 @@ If a provider is explicitly enabled but required credentials are missing, or the
|
||||
|
||||
```bash
|
||||
make install # Install dependencies
|
||||
make dev # Run Gateway API + embedded agent runtime (port 8001)
|
||||
make gateway # Run Gateway API without reload (port 8001)
|
||||
make dev # Run LangGraph server (port 2024)
|
||||
make gateway # Run Gateway API (port 8001)
|
||||
make lint # Run linter (ruff)
|
||||
make format # Format code (ruff)
|
||||
make detect-blocking-io # Inventory blocking IO that may block the backend event loop
|
||||
```
|
||||
|
||||
### Code Style
|
||||
@@ -379,18 +382,6 @@ make detect-blocking-io # Inventory blocking IO that may block the backend even
|
||||
uv run pytest
|
||||
```
|
||||
|
||||
`make detect-blocking-io` statically scans backend business code for blocking
|
||||
IO that may run on the backend event loop and is not test-coverage-bound. It
|
||||
prints a concise summary for human review and writes complete JSON findings to
|
||||
`.deer-flow/blocking-io-findings.json` at the repository root (regardless of
|
||||
whether the target is invoked from the repo root or from `backend/`). JSON
|
||||
findings include both broad IO category and review-oriented fields such as
|
||||
`priority`, `location`, `blocking_call`, `event_loop_exposure`, `reason`, and
|
||||
`code`. `priority` is a deterministic review ordering from the operation type,
|
||||
not proof of a bug. Bare-name same-file calls are resolved by function name,
|
||||
so duplicate helper names in one file can conservatively over-report async
|
||||
reachability.
|
||||
|
||||
---
|
||||
|
||||
## Technology Stack
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
Provides a pluggable channel system that connects external messaging platforms
|
||||
(Feishu/Lark, Slack, Telegram) to the DeerFlow agent via the ChannelManager,
|
||||
which uses ``langgraph-sdk`` to communicate with Gateway's LangGraph-compatible API.
|
||||
which uses ``langgraph-sdk`` to communicate with the underlying LangGraph Server.
|
||||
"""
|
||||
|
||||
from app.channels.base import Channel
|
||||
|
||||
@@ -31,10 +31,6 @@ class Channel(ABC):
|
||||
def is_running(self) -> bool:
|
||||
return self._running
|
||||
|
||||
@property
|
||||
def supports_streaming(self) -> bool:
|
||||
return False
|
||||
|
||||
# -- lifecycle ---------------------------------------------------------
|
||||
|
||||
@abstractmethod
|
||||
|
||||
@@ -18,10 +18,3 @@ KNOWN_CHANNEL_COMMANDS: frozenset[str] = frozenset(
|
||||
"/help",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def is_known_channel_command(text: str) -> bool:
|
||||
"""Return whether text starts with a registered channel control command."""
|
||||
if not text.startswith("/"):
|
||||
return False
|
||||
return text.split(maxsplit=1)[0].lower() in KNOWN_CHANNEL_COMMANDS
|
||||
|
||||
@@ -1,738 +0,0 @@
|
||||
"""DingTalk channel implementation."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
from app.channels.base import Channel
|
||||
from app.channels.commands import is_known_channel_command
|
||||
from app.channels.message_bus import InboundMessage, InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
DINGTALK_API_BASE = "https://api.dingtalk.com"
|
||||
|
||||
_TOKEN_REFRESH_MARGIN_SECONDS = 300
|
||||
|
||||
_CONVERSATION_TYPE_P2P = "1"
|
||||
_CONVERSATION_TYPE_GROUP = "2"
|
||||
|
||||
_MAX_UPLOAD_SIZE_BYTES = 20 * 1024 * 1024
|
||||
|
||||
|
||||
def _normalize_conversation_type(raw: Any) -> str:
|
||||
"""Normalize ``conversationType`` to ``"1"`` (P2P) or ``"2"`` (group).
|
||||
|
||||
Stream payloads may send int or string values.
|
||||
"""
|
||||
if raw is None:
|
||||
return _CONVERSATION_TYPE_P2P
|
||||
s = str(raw).strip()
|
||||
if s == _CONVERSATION_TYPE_GROUP:
|
||||
return _CONVERSATION_TYPE_GROUP
|
||||
return _CONVERSATION_TYPE_P2P
|
||||
|
||||
|
||||
def _normalize_allowed_users(allowed_users: Any) -> set[str]:
|
||||
if allowed_users is None:
|
||||
return set()
|
||||
if isinstance(allowed_users, str):
|
||||
values = [allowed_users]
|
||||
elif isinstance(allowed_users, (list, tuple, set)):
|
||||
values = allowed_users
|
||||
else:
|
||||
logger.warning(
|
||||
"DingTalk allowed_users should be a list of user IDs; treating %s as one string value",
|
||||
type(allowed_users).__name__,
|
||||
)
|
||||
values = [allowed_users]
|
||||
return {str(uid) for uid in values if str(uid)}
|
||||
|
||||
|
||||
def _is_dingtalk_command(text: str) -> bool:
|
||||
return is_known_channel_command(text)
|
||||
|
||||
|
||||
def _extract_text_from_rich_text(rich_text_list: list) -> str:
|
||||
parts: list[str] = []
|
||||
for item in rich_text_list:
|
||||
if isinstance(item, dict) and "text" in item:
|
||||
parts.append(item["text"])
|
||||
return " ".join(parts)
|
||||
|
||||
|
||||
_FENCED_CODE_BLOCK_RE = re.compile(r"```(\w*)\n(.*?)```", re.DOTALL)
|
||||
_INLINE_CODE_RE = re.compile(r"`([^`\n]+)`")
|
||||
_HORIZONTAL_RULE_RE = re.compile(r"^-{3,}$", re.MULTILINE)
|
||||
_TABLE_SEPARATOR_RE = re.compile(r"^\|[-:| ]+\|$", re.MULTILINE)
|
||||
|
||||
|
||||
def _convert_markdown_table(text: str) -> str:
|
||||
# DingTalk sampleMarkdown does not render pipe-delimited tables.
|
||||
lines = text.split("\n")
|
||||
result: list[str] = []
|
||||
i = 0
|
||||
while i < len(lines):
|
||||
line = lines[i]
|
||||
# Detect table: header row followed by separator row
|
||||
if i + 1 < len(lines) and line.strip().startswith("|") and _TABLE_SEPARATOR_RE.match(lines[i + 1].strip()):
|
||||
headers = [h.strip() for h in line.strip().strip("|").split("|")]
|
||||
i += 2 # skip header + separator
|
||||
while i < len(lines) and lines[i].strip().startswith("|"):
|
||||
cells = [c.strip() for c in lines[i].strip().strip("|").split("|")]
|
||||
for h, c in zip(headers, cells):
|
||||
result.append(f"> **{h}**: {c}")
|
||||
result.append("")
|
||||
i += 1
|
||||
else:
|
||||
result.append(line)
|
||||
i += 1
|
||||
return "\n".join(result)
|
||||
|
||||
|
||||
def _adapt_markdown_for_dingtalk(text: str) -> str:
|
||||
"""Adapt markdown for DingTalk's limited sampleMarkdown renderer."""
|
||||
|
||||
def _code_block_to_quote(match: re.Match) -> str:
|
||||
lang = match.group(1)
|
||||
code = match.group(2).rstrip("\n")
|
||||
prefix = f"> **{lang}**\n" if lang else ""
|
||||
quoted_lines = "\n".join(f"> {line}" for line in code.split("\n"))
|
||||
return f"{prefix}{quoted_lines}\n"
|
||||
|
||||
text = _FENCED_CODE_BLOCK_RE.sub(_code_block_to_quote, text)
|
||||
text = _INLINE_CODE_RE.sub(r"**\1**", text)
|
||||
text = _convert_markdown_table(text)
|
||||
text = _HORIZONTAL_RULE_RE.sub("───────────", text)
|
||||
return text
|
||||
|
||||
|
||||
class DingTalkChannel(Channel):
|
||||
"""DingTalk IM channel using Stream Push (WebSocket, no public IP needed)."""
|
||||
|
||||
def __init__(self, bus: MessageBus, config: dict[str, Any]) -> None:
|
||||
super().__init__(name="dingtalk", bus=bus, config=config)
|
||||
self._thread: threading.Thread | None = None
|
||||
self._main_loop: asyncio.AbstractEventLoop | None = None
|
||||
self._client_id: str = ""
|
||||
self._client_secret: str = ""
|
||||
self._allowed_users: set[str] = _normalize_allowed_users(config.get("allowed_users"))
|
||||
self._cached_token: str = ""
|
||||
self._token_expires_at: float = 0.0
|
||||
self._token_lock = asyncio.Lock()
|
||||
self._card_template_id: str = config.get("card_template_id", "")
|
||||
self._card_track_ids: dict[str, str] = {}
|
||||
self._dingtalk_client: Any = None
|
||||
self._stream_client: Any = None
|
||||
self._incoming_messages: dict[str, Any] = {}
|
||||
self._incoming_messages_lock = threading.Lock()
|
||||
self._card_repliers: dict[str, Any] = {}
|
||||
|
||||
@property
|
||||
def supports_streaming(self) -> bool:
|
||||
return bool(self._card_template_id)
|
||||
|
||||
async def start(self) -> None:
|
||||
if self._running:
|
||||
return
|
||||
|
||||
try:
|
||||
import dingtalk_stream # noqa: F401
|
||||
except ImportError:
|
||||
logger.error("dingtalk-stream is not installed. Install it with: uv add dingtalk-stream")
|
||||
return
|
||||
|
||||
client_id = self.config.get("client_id", "")
|
||||
client_secret = self.config.get("client_secret", "")
|
||||
|
||||
if not client_id or not client_secret:
|
||||
logger.error("DingTalk channel requires client_id and client_secret")
|
||||
return
|
||||
|
||||
self._client_id = client_id
|
||||
self._client_secret = client_secret
|
||||
self._main_loop = asyncio.get_running_loop()
|
||||
|
||||
if self._card_template_id:
|
||||
logger.info("[DingTalk] AI Card mode enabled (template=%s)", self._card_template_id)
|
||||
|
||||
self._running = True
|
||||
self.bus.subscribe_outbound(self._on_outbound)
|
||||
|
||||
self._thread = threading.Thread(
|
||||
target=self._run_stream,
|
||||
args=(client_id, client_secret),
|
||||
daemon=True,
|
||||
)
|
||||
self._thread.start()
|
||||
logger.info("DingTalk channel started")
|
||||
|
||||
async def stop(self) -> None:
|
||||
self._running = False
|
||||
self.bus.unsubscribe_outbound(self._on_outbound)
|
||||
|
||||
stream_client = self._stream_client
|
||||
if stream_client is not None:
|
||||
try:
|
||||
if hasattr(stream_client, "disconnect"):
|
||||
stream_client.disconnect()
|
||||
except Exception:
|
||||
logger.debug("[DingTalk] error disconnecting stream client", exc_info=True)
|
||||
|
||||
self._dingtalk_client = None
|
||||
self._stream_client = None
|
||||
with self._incoming_messages_lock:
|
||||
self._incoming_messages.clear()
|
||||
self._card_repliers.clear()
|
||||
self._card_track_ids.clear()
|
||||
if self._thread:
|
||||
self._thread.join(timeout=5)
|
||||
self._thread = None
|
||||
logger.info("DingTalk channel stopped")
|
||||
|
||||
def _resolve_routing(self, msg: OutboundMessage) -> tuple[str, str, str]:
|
||||
"""Return (conversation_type, sender_staff_id, conversation_id).
|
||||
|
||||
Uses msg.chat_id as the primary routing key; metadata as fallback.
|
||||
"""
|
||||
conversation_type = _normalize_conversation_type(msg.metadata.get("conversation_type"))
|
||||
sender_staff_id = msg.metadata.get("sender_staff_id", "")
|
||||
conversation_id = msg.metadata.get("conversation_id", "")
|
||||
if conversation_type == _CONVERSATION_TYPE_GROUP:
|
||||
conversation_id = msg.chat_id or conversation_id
|
||||
else:
|
||||
sender_staff_id = msg.chat_id or sender_staff_id
|
||||
return conversation_type, sender_staff_id, conversation_id
|
||||
|
||||
async def send(self, msg: OutboundMessage, *, _max_retries: int = 3) -> None:
|
||||
conversation_type, sender_staff_id, conversation_id = self._resolve_routing(msg)
|
||||
robot_code = self._client_id
|
||||
|
||||
# Card mode: stream update to existing AI card
|
||||
source_key = self._make_card_source_key_from_outbound(msg)
|
||||
out_track_id = self._card_track_ids.get(source_key)
|
||||
|
||||
# ``card_template_id`` enables ``runs.stream`` (non-final + final outbounds).
|
||||
# If card creation failed, skip non-final chunks to avoid duplicate messages.
|
||||
if self._card_template_id and not out_track_id and not msg.is_final:
|
||||
return
|
||||
|
||||
if out_track_id:
|
||||
try:
|
||||
await self._stream_update_card(
|
||||
out_track_id,
|
||||
msg.text,
|
||||
is_finalize=msg.is_final,
|
||||
)
|
||||
except Exception:
|
||||
logger.warning("[DingTalk] card stream failed, falling back to sampleMarkdown")
|
||||
if msg.is_final:
|
||||
self._card_track_ids.pop(source_key, None)
|
||||
self._card_repliers.pop(out_track_id, None)
|
||||
await self._send_markdown_fallback(robot_code, conversation_type, sender_staff_id, conversation_id, msg.text)
|
||||
return
|
||||
if msg.is_final:
|
||||
self._card_track_ids.pop(source_key, None)
|
||||
self._card_repliers.pop(out_track_id, None)
|
||||
return
|
||||
|
||||
# Non-card mode: send sampleMarkdown with retry
|
||||
last_exc: Exception | None = None
|
||||
for attempt in range(_max_retries):
|
||||
try:
|
||||
if conversation_type == _CONVERSATION_TYPE_GROUP:
|
||||
await self._send_group_message(robot_code, conversation_id, msg.text, at_user_ids=[sender_staff_id] if sender_staff_id else None)
|
||||
else:
|
||||
await self._send_p2p_message(robot_code, sender_staff_id, msg.text)
|
||||
return
|
||||
except Exception as exc:
|
||||
last_exc = exc
|
||||
if attempt < _max_retries - 1:
|
||||
delay = 2**attempt
|
||||
logger.warning(
|
||||
"[DingTalk] send failed (attempt %d/%d), retrying in %ds: %s",
|
||||
attempt + 1,
|
||||
_max_retries,
|
||||
delay,
|
||||
exc,
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
|
||||
logger.error("[DingTalk] send failed after %d attempts: %s", _max_retries, last_exc)
|
||||
if last_exc is None:
|
||||
raise RuntimeError("DingTalk send failed without an exception from any attempt")
|
||||
raise last_exc
|
||||
|
||||
async def _send_markdown_fallback(
|
||||
self,
|
||||
robot_code: str,
|
||||
conversation_type: str,
|
||||
sender_staff_id: str,
|
||||
conversation_id: str,
|
||||
text: str,
|
||||
) -> None:
|
||||
try:
|
||||
if conversation_type == _CONVERSATION_TYPE_GROUP:
|
||||
await self._send_group_message(robot_code, conversation_id, text)
|
||||
else:
|
||||
await self._send_p2p_message(robot_code, sender_staff_id, text)
|
||||
except Exception:
|
||||
logger.exception("[DingTalk] markdown fallback also failed")
|
||||
raise
|
||||
|
||||
async def send_file(self, msg: OutboundMessage, attachment: ResolvedAttachment) -> bool:
|
||||
if attachment.size > _MAX_UPLOAD_SIZE_BYTES:
|
||||
logger.warning("[DingTalk] file too large (%d bytes), skipping: %s", attachment.size, attachment.filename)
|
||||
return False
|
||||
|
||||
conversation_type, sender_staff_id, conversation_id = self._resolve_routing(msg)
|
||||
robot_code = self._client_id
|
||||
|
||||
try:
|
||||
media_id = await self._upload_media(attachment.actual_path, "image" if attachment.is_image else "file")
|
||||
if not media_id:
|
||||
return False
|
||||
|
||||
if attachment.is_image:
|
||||
msg_key = "sampleImageMsg"
|
||||
msg_param = json.dumps({"photoURL": media_id})
|
||||
else:
|
||||
msg_key = "sampleFile"
|
||||
msg_param = json.dumps(
|
||||
{
|
||||
"fileUrl": media_id,
|
||||
"fileName": attachment.filename,
|
||||
"fileSize": str(attachment.size),
|
||||
}
|
||||
)
|
||||
|
||||
token = await self._get_access_token()
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(30.0)) as client:
|
||||
if conversation_type == _CONVERSATION_TYPE_GROUP:
|
||||
response = await client.post(
|
||||
f"{DINGTALK_API_BASE}/v1.0/robot/groupMessages/send",
|
||||
headers=self._api_headers(token),
|
||||
json={
|
||||
"msgKey": msg_key,
|
||||
"msgParam": msg_param,
|
||||
"robotCode": robot_code,
|
||||
"openConversationId": conversation_id,
|
||||
},
|
||||
)
|
||||
else:
|
||||
response = await client.post(
|
||||
f"{DINGTALK_API_BASE}/v1.0/robot/oToMessages/batchSend",
|
||||
headers=self._api_headers(token),
|
||||
json={
|
||||
"msgKey": msg_key,
|
||||
"msgParam": msg_param,
|
||||
"robotCode": robot_code,
|
||||
"userIds": [sender_staff_id],
|
||||
},
|
||||
)
|
||||
response.raise_for_status()
|
||||
|
||||
logger.info("[DingTalk] file sent: %s", attachment.filename)
|
||||
return True
|
||||
except (httpx.HTTPError, OSError, ValueError, TypeError, AttributeError):
|
||||
logger.exception("[DingTalk] failed to send file: %s", attachment.filename)
|
||||
return False
|
||||
|
||||
# -- stream client (runs in dedicated thread) --------------------------
|
||||
|
||||
def _run_stream(self, client_id: str, client_secret: str) -> None:
|
||||
try:
|
||||
import dingtalk_stream
|
||||
|
||||
credential = dingtalk_stream.Credential(client_id, client_secret)
|
||||
client = dingtalk_stream.DingTalkStreamClient(credential)
|
||||
self._stream_client = client
|
||||
client.register_callback_handler(
|
||||
dingtalk_stream.chatbot.ChatbotMessage.TOPIC,
|
||||
_DingTalkMessageHandler(self),
|
||||
)
|
||||
client.start_forever()
|
||||
except Exception:
|
||||
if self._running:
|
||||
logger.exception("DingTalk Stream Push error")
|
||||
finally:
|
||||
self._stream_client = None
|
||||
|
||||
def _on_chatbot_message(self, message: Any) -> None:
|
||||
if not self._running:
|
||||
return
|
||||
try:
|
||||
sender_staff_id = message.sender_staff_id or ""
|
||||
conversation_type = _normalize_conversation_type(message.conversation_type)
|
||||
conversation_id = message.conversation_id or ""
|
||||
msg_id = message.message_id or ""
|
||||
sender_nick = message.sender_nick or ""
|
||||
|
||||
if self._allowed_users and sender_staff_id not in self._allowed_users:
|
||||
logger.debug("[DingTalk] ignoring message from non-allowed user: %s", sender_staff_id)
|
||||
return
|
||||
|
||||
text = self._extract_text(message)
|
||||
if not text:
|
||||
logger.info("[DingTalk] empty text, ignoring message")
|
||||
return
|
||||
|
||||
logger.info(
|
||||
"[DingTalk] parsed message: conv_type=%s, msg_id=%s, sender=%s(%s), text=%r",
|
||||
conversation_type,
|
||||
msg_id,
|
||||
sender_staff_id,
|
||||
sender_nick,
|
||||
text[:100],
|
||||
)
|
||||
|
||||
if _is_dingtalk_command(text):
|
||||
msg_type = InboundMessageType.COMMAND
|
||||
else:
|
||||
msg_type = InboundMessageType.CHAT
|
||||
|
||||
# P2P: topic_id=None (single thread per user, like Telegram private chat)
|
||||
# Group: topic_id=msg_id (each new message starts a new topic, like Feishu)
|
||||
topic_id: str | None = msg_id if conversation_type == _CONVERSATION_TYPE_GROUP else None
|
||||
|
||||
# chat_id uses conversation_id for groups, sender_staff_id for P2P
|
||||
chat_id = conversation_id if conversation_type == _CONVERSATION_TYPE_GROUP else sender_staff_id
|
||||
|
||||
inbound = self._make_inbound(
|
||||
chat_id=chat_id,
|
||||
user_id=sender_staff_id,
|
||||
text=text,
|
||||
msg_type=msg_type,
|
||||
thread_ts=msg_id,
|
||||
metadata={
|
||||
"conversation_type": conversation_type,
|
||||
"conversation_id": conversation_id,
|
||||
"sender_staff_id": sender_staff_id,
|
||||
"sender_nick": sender_nick,
|
||||
"message_id": msg_id,
|
||||
},
|
||||
)
|
||||
inbound.topic_id = topic_id
|
||||
|
||||
if self._card_template_id:
|
||||
source_key = self._make_card_source_key(inbound)
|
||||
with self._incoming_messages_lock:
|
||||
self._incoming_messages[source_key] = message
|
||||
|
||||
if self._main_loop and self._main_loop.is_running():
|
||||
logger.info("[DingTalk] publishing inbound message to bus (type=%s, msg_id=%s)", msg_type.value, msg_id)
|
||||
fut = asyncio.run_coroutine_threadsafe(
|
||||
self._prepare_inbound(chat_id, inbound),
|
||||
self._main_loop,
|
||||
)
|
||||
fut.add_done_callback(lambda f, mid=msg_id: self._log_future_error(f, "prepare_inbound", mid))
|
||||
else:
|
||||
logger.warning("[DingTalk] main loop not running, cannot publish inbound message")
|
||||
except Exception:
|
||||
logger.exception("[DingTalk] error processing chatbot message")
|
||||
|
||||
@staticmethod
|
||||
def _extract_text(message: Any) -> str:
|
||||
msg_type = message.message_type
|
||||
if msg_type == "text" and message.text:
|
||||
return message.text.content.strip()
|
||||
if msg_type == "richText" and message.rich_text_content:
|
||||
return _extract_text_from_rich_text(message.rich_text_content.rich_text_list).strip()
|
||||
return ""
|
||||
|
||||
async def _prepare_inbound(self, chat_id: str, inbound: InboundMessage) -> None:
|
||||
# Running reply must finish before publish_inbound so AI card tracks are
|
||||
# registered before the manager emits streaming outbounds.
|
||||
await self._send_running_reply(chat_id, inbound)
|
||||
await self.bus.publish_inbound(inbound)
|
||||
|
||||
async def _send_running_reply(self, chat_id: str, inbound: InboundMessage) -> None:
|
||||
conversation_type = inbound.metadata.get("conversation_type", _CONVERSATION_TYPE_P2P)
|
||||
sender_staff_id = inbound.metadata.get("sender_staff_id", "")
|
||||
conversation_id = inbound.metadata.get("conversation_id", "")
|
||||
text = "\u23f3 Working on it..."
|
||||
|
||||
try:
|
||||
if self._card_template_id:
|
||||
source_key = self._make_card_source_key(inbound)
|
||||
with self._incoming_messages_lock:
|
||||
chatbot_message = self._incoming_messages.pop(source_key, None)
|
||||
out_track_id = await self._create_and_deliver_card(
|
||||
text,
|
||||
chatbot_message=chatbot_message,
|
||||
)
|
||||
if out_track_id:
|
||||
self._card_track_ids[source_key] = out_track_id
|
||||
logger.info("[DingTalk] AI card running reply sent for chat=%s", chat_id)
|
||||
return
|
||||
|
||||
robot_code = self._client_id
|
||||
if conversation_type == _CONVERSATION_TYPE_GROUP:
|
||||
await self._send_text_message_to_group(robot_code, conversation_id, text)
|
||||
else:
|
||||
await self._send_text_message_to_user(robot_code, sender_staff_id, text)
|
||||
logger.info("[DingTalk] 'Working on it...' reply sent for chat=%s", chat_id)
|
||||
except Exception:
|
||||
logger.exception("[DingTalk] failed to send running reply for chat=%s", chat_id)
|
||||
|
||||
# -- DingTalk API helpers ----------------------------------------------
|
||||
|
||||
async def _get_access_token(self) -> str:
|
||||
if self._cached_token and time.monotonic() < self._token_expires_at:
|
||||
return self._cached_token
|
||||
async with self._token_lock:
|
||||
if self._cached_token and time.monotonic() < self._token_expires_at:
|
||||
return self._cached_token
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(10.0)) as client:
|
||||
response = await client.post(
|
||||
f"{DINGTALK_API_BASE}/v1.0/oauth2/accessToken",
|
||||
json={"appKey": self._client_id, "appSecret": self._client_secret}, # DingTalk API field names
|
||||
)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
|
||||
if not isinstance(data, dict):
|
||||
raise ValueError(f"DingTalk access token response must be a JSON object, got {type(data).__name__}")
|
||||
|
||||
access_token = data.get("accessToken")
|
||||
if not isinstance(access_token, str) or not access_token.strip():
|
||||
raise ValueError("DingTalk access token response did not contain a usable accessToken")
|
||||
|
||||
raw_expires_in = data.get("expireIn", 7200)
|
||||
try:
|
||||
expires_in = int(raw_expires_in)
|
||||
except (TypeError, ValueError):
|
||||
logger.warning("[DingTalk] invalid expireIn value %r, using default 7200s", raw_expires_in)
|
||||
expires_in = 7200
|
||||
|
||||
self._cached_token = access_token.strip()
|
||||
self._token_expires_at = time.monotonic() + expires_in - _TOKEN_REFRESH_MARGIN_SECONDS
|
||||
return self._cached_token
|
||||
|
||||
@staticmethod
|
||||
def _api_headers(token: str) -> dict[str, str]:
|
||||
return {
|
||||
"x-acs-dingtalk-access-token": token,
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
async def _send_text_message_to_user(self, robot_code: str, user_id: str, text: str) -> None:
|
||||
token = await self._get_access_token()
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(30.0)) as client:
|
||||
response = await client.post(
|
||||
f"{DINGTALK_API_BASE}/v1.0/robot/oToMessages/batchSend",
|
||||
headers=self._api_headers(token),
|
||||
json={
|
||||
"msgKey": "sampleText",
|
||||
"msgParam": json.dumps({"content": text}),
|
||||
"robotCode": robot_code,
|
||||
"userIds": [user_id],
|
||||
},
|
||||
)
|
||||
response.raise_for_status()
|
||||
|
||||
async def _send_text_message_to_group(self, robot_code: str, conversation_id: str, text: str) -> None:
|
||||
token = await self._get_access_token()
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(30.0)) as client:
|
||||
response = await client.post(
|
||||
f"{DINGTALK_API_BASE}/v1.0/robot/groupMessages/send",
|
||||
headers=self._api_headers(token),
|
||||
json={
|
||||
"msgKey": "sampleText",
|
||||
"msgParam": json.dumps({"content": text}),
|
||||
"robotCode": robot_code,
|
||||
"openConversationId": conversation_id,
|
||||
},
|
||||
)
|
||||
response.raise_for_status()
|
||||
|
||||
async def _send_p2p_message(self, robot_code: str, user_id: str, text: str) -> None:
|
||||
text = _adapt_markdown_for_dingtalk(text)
|
||||
token = await self._get_access_token()
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(30.0)) as client:
|
||||
response = await client.post(
|
||||
f"{DINGTALK_API_BASE}/v1.0/robot/oToMessages/batchSend",
|
||||
headers=self._api_headers(token),
|
||||
json={
|
||||
"msgKey": "sampleMarkdown",
|
||||
"msgParam": json.dumps({"title": "DeerFlow", "text": text}),
|
||||
"robotCode": robot_code,
|
||||
"userIds": [user_id],
|
||||
},
|
||||
)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
if data.get("processQueryKey"):
|
||||
logger.info("[DingTalk] P2P message sent to user=%s", user_id)
|
||||
else:
|
||||
logger.warning("[DingTalk] P2P send response: %s", data)
|
||||
|
||||
async def _send_group_message(
|
||||
self,
|
||||
robot_code: str,
|
||||
conversation_id: str,
|
||||
text: str,
|
||||
*,
|
||||
at_user_ids: list[str] | None = None, # noqa: ARG002
|
||||
) -> None:
|
||||
# at_user_ids accepted for call-site compatibility but not passed to the API
|
||||
# (sampleMarkdown does not support @mentions).
|
||||
text = _adapt_markdown_for_dingtalk(text)
|
||||
token = await self._get_access_token()
|
||||
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(30.0)) as client:
|
||||
response = await client.post(
|
||||
f"{DINGTALK_API_BASE}/v1.0/robot/groupMessages/send",
|
||||
headers=self._api_headers(token),
|
||||
json={
|
||||
"msgKey": "sampleMarkdown",
|
||||
"msgParam": json.dumps({"title": "DeerFlow", "text": text}),
|
||||
"robotCode": robot_code,
|
||||
"openConversationId": conversation_id,
|
||||
},
|
||||
)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
if data.get("processQueryKey"):
|
||||
logger.info("[DingTalk] group message sent to conversation=%s", conversation_id)
|
||||
else:
|
||||
logger.warning("[DingTalk] group send response: %s", data)
|
||||
|
||||
# -- AI Card streaming helpers -------------------------------------------
|
||||
|
||||
def _make_card_source_key(self, inbound: InboundMessage) -> str:
|
||||
m = inbound.metadata
|
||||
return f"{m.get('conversation_type', '')}:{m.get('sender_staff_id', '')}:{m.get('conversation_id', '')}:{m.get('message_id', '')}"
|
||||
|
||||
def _make_card_source_key_from_outbound(self, msg: OutboundMessage) -> str:
|
||||
m = msg.metadata
|
||||
correlation_id = m.get("message_id") or msg.thread_ts or ""
|
||||
return f"{m.get('conversation_type', '')}:{m.get('sender_staff_id', '')}:{m.get('conversation_id', '')}:{correlation_id}"
|
||||
|
||||
async def _create_and_deliver_card(
|
||||
self,
|
||||
initial_text: str,
|
||||
*,
|
||||
chatbot_message: Any = None,
|
||||
) -> str | None:
|
||||
if self._dingtalk_client is None or chatbot_message is None:
|
||||
logger.warning("[DingTalk] SDK client or chatbot_message unavailable, skipping AI card")
|
||||
return None
|
||||
|
||||
try:
|
||||
from dingtalk_stream.card_replier import AICardReplier
|
||||
except ImportError:
|
||||
logger.warning("[DingTalk] dingtalk-stream card_replier not available")
|
||||
return None
|
||||
|
||||
try:
|
||||
replier = AICardReplier(self._dingtalk_client, chatbot_message)
|
||||
card_instance_id = await replier.async_create_and_deliver_card(
|
||||
card_template_id=self._card_template_id,
|
||||
card_data={"content": initial_text},
|
||||
)
|
||||
if not card_instance_id:
|
||||
return None
|
||||
|
||||
self._card_repliers[card_instance_id] = replier
|
||||
logger.info("[DingTalk] AI card created: outTrackId=%s", card_instance_id)
|
||||
return card_instance_id
|
||||
except Exception:
|
||||
logger.exception("[DingTalk] failed to create AI card")
|
||||
return None
|
||||
|
||||
async def _stream_update_card(
|
||||
self,
|
||||
out_track_id: str,
|
||||
content: str,
|
||||
*,
|
||||
is_finalize: bool = False,
|
||||
is_error: bool = False,
|
||||
) -> None:
|
||||
replier = self._card_repliers.get(out_track_id)
|
||||
if not replier:
|
||||
raise RuntimeError(f"No AICardReplier found for track ID {out_track_id}")
|
||||
|
||||
await replier.async_streaming(
|
||||
card_instance_id=out_track_id,
|
||||
content_key="content",
|
||||
content_value=content,
|
||||
append=False,
|
||||
finished=is_finalize,
|
||||
failed=is_error,
|
||||
)
|
||||
|
||||
# -- media upload --------------------------------------------------------
|
||||
|
||||
async def _upload_media(self, file_path: str | Path, media_type: str) -> str | None:
|
||||
try:
|
||||
file_bytes = await asyncio.to_thread(Path(file_path).read_bytes)
|
||||
token = await self._get_access_token()
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(60.0)) as client:
|
||||
response = await client.post(
|
||||
f"{DINGTALK_API_BASE}/v1.0/files/upload",
|
||||
headers={"x-acs-dingtalk-access-token": token},
|
||||
files={"file": ("upload", file_bytes)},
|
||||
data={"type": media_type},
|
||||
)
|
||||
response.raise_for_status()
|
||||
try:
|
||||
payload = response.json()
|
||||
except json.JSONDecodeError:
|
||||
logger.exception("[DingTalk] failed to decode upload response JSON: %s", file_path)
|
||||
return None
|
||||
if not isinstance(payload, dict):
|
||||
logger.warning("[DingTalk] unexpected upload response type %s for %s", type(payload).__name__, file_path)
|
||||
return None
|
||||
return payload.get("mediaId")
|
||||
except (httpx.HTTPError, OSError):
|
||||
logger.exception("[DingTalk] failed to upload media: %s", file_path)
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def _log_future_error(fut: Any, name: str, msg_id: str) -> None:
|
||||
try:
|
||||
exc = fut.exception()
|
||||
if exc:
|
||||
logger.error("[DingTalk] %s failed for msg_id=%s: %s", name, msg_id, exc)
|
||||
except (asyncio.CancelledError, asyncio.InvalidStateError):
|
||||
pass
|
||||
|
||||
|
||||
class _DingTalkMessageHandler:
|
||||
"""Callback handler registered with dingtalk-stream."""
|
||||
|
||||
def __init__(self, channel: DingTalkChannel) -> None:
|
||||
self._channel = channel
|
||||
|
||||
def pre_start(self) -> None:
|
||||
if hasattr(self, "dingtalk_client") and self.dingtalk_client is not None:
|
||||
self._channel._dingtalk_client = self.dingtalk_client
|
||||
|
||||
async def raw_process(self, callback_message: Any) -> Any:
|
||||
import dingtalk_stream
|
||||
from dingtalk_stream.frames import Headers
|
||||
|
||||
code, message = await self.process(callback_message)
|
||||
ack_message = dingtalk_stream.AckMessage()
|
||||
ack_message.code = code
|
||||
ack_message.headers.message_id = callback_message.headers.message_id
|
||||
ack_message.headers.content_type = Headers.CONTENT_TYPE_APPLICATION_JSON
|
||||
ack_message.data = {"response": message}
|
||||
return ack_message
|
||||
|
||||
async def process(self, callback: Any) -> tuple[int, str]:
|
||||
import dingtalk_stream
|
||||
|
||||
incoming_message = dingtalk_stream.ChatbotMessage.from_dict(callback.data)
|
||||
self._channel._on_chatbot_message(incoming_message)
|
||||
return dingtalk_stream.AckMessage.STATUS_OK, "OK"
|
||||
@@ -1,554 +0,0 @@
|
||||
"""Discord channel integration using discord.py."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import threading
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from app.channels.base import Channel
|
||||
from app.channels.commands import is_known_channel_command
|
||||
from app.channels.message_bus import InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_DISCORD_MAX_MESSAGE_LEN = 2000
|
||||
|
||||
|
||||
class DiscordChannel(Channel):
|
||||
"""Discord bot channel.
|
||||
|
||||
Configuration keys (in ``config.yaml`` under ``channels.discord``):
|
||||
- ``bot_token``: Discord Bot token.
|
||||
- ``allowed_guilds``: (optional) List of allowed Discord guild IDs. Empty = allow all.
|
||||
- ``mention_only``: (optional) If true, only respond when the bot is mentioned.
|
||||
- ``allowed_channels``: (optional) List of channel IDs where messages are always accepted
|
||||
(even when mention_only is true). Use for channels where you want the bot to respond
|
||||
without mentions. Empty = mention_only applies everywhere.
|
||||
- ``thread_mode``: (optional) If true, group a channel conversation into a thread.
|
||||
Default: same as ``mention_only``.
|
||||
"""
|
||||
|
||||
def __init__(self, bus: MessageBus, config: dict[str, Any]) -> None:
|
||||
super().__init__(name="discord", bus=bus, config=config)
|
||||
self._bot_token = str(config.get("bot_token", "")).strip()
|
||||
self._allowed_guilds: set[int] = set()
|
||||
for guild_id in config.get("allowed_guilds", []):
|
||||
try:
|
||||
self._allowed_guilds.add(int(guild_id))
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
self._mention_only: bool = bool(config.get("mention_only", False))
|
||||
self._thread_mode: bool = config.get("thread_mode", self._mention_only)
|
||||
self._allowed_channels: set[str] = set()
|
||||
for channel_id in config.get("allowed_channels", []):
|
||||
self._allowed_channels.add(str(channel_id))
|
||||
|
||||
# Session tracking: channel_id -> Discord thread_id (in-memory, persisted to JSON).
|
||||
# Uses a dedicated JSON file separate from ChannelStore, which maps IM
|
||||
# conversations to DeerFlow thread IDs — a different concern.
|
||||
self._active_threads: dict[str, str] = {}
|
||||
# Reverse-lookup set for O(1) thread ID checks (avoids O(n) scan of _active_threads.values()).
|
||||
self._active_thread_ids: set[str] = set()
|
||||
# Lock protecting _active_threads and the JSON file from concurrent access.
|
||||
# _run_client (Discord loop thread) and the main thread both read/write.
|
||||
self._thread_store_lock = threading.Lock()
|
||||
store = config.get("channel_store")
|
||||
if store is not None:
|
||||
self._thread_store_path = store._path.parent / "discord_threads.json"
|
||||
else:
|
||||
self._thread_store_path = Path.home() / ".deer-flow" / "channels" / "discord_threads.json"
|
||||
|
||||
# Typing indicator management
|
||||
self._typing_tasks: dict[str, asyncio.Task] = {}
|
||||
|
||||
self._client = None
|
||||
self._thread: threading.Thread | None = None
|
||||
self._discord_loop: asyncio.AbstractEventLoop | None = None
|
||||
self._main_loop: asyncio.AbstractEventLoop | None = None
|
||||
self._discord_module = None
|
||||
|
||||
async def start(self) -> None:
|
||||
if self._running:
|
||||
return
|
||||
|
||||
try:
|
||||
import discord
|
||||
except ImportError:
|
||||
logger.error("discord.py is not installed. Install it with: uv add discord.py")
|
||||
return
|
||||
|
||||
if not self._bot_token:
|
||||
logger.error("Discord channel requires bot_token")
|
||||
return
|
||||
|
||||
intents = discord.Intents.default()
|
||||
intents.messages = True
|
||||
intents.guilds = True
|
||||
intents.message_content = True
|
||||
|
||||
client = discord.Client(
|
||||
intents=intents,
|
||||
allowed_mentions=discord.AllowedMentions.none(),
|
||||
)
|
||||
self._client = client
|
||||
self._discord_module = discord
|
||||
self._main_loop = asyncio.get_event_loop()
|
||||
|
||||
@client.event
|
||||
async def on_message(message) -> None:
|
||||
await self._on_message(message)
|
||||
|
||||
self._running = True
|
||||
self.bus.subscribe_outbound(self._on_outbound)
|
||||
|
||||
self._thread = threading.Thread(target=self._run_client, daemon=True)
|
||||
self._thread.start()
|
||||
self._load_active_threads()
|
||||
logger.info("Discord channel started")
|
||||
|
||||
def _load_active_threads(self) -> None:
|
||||
"""Restore Discord thread mappings from the dedicated JSON file on startup."""
|
||||
with self._thread_store_lock:
|
||||
try:
|
||||
if not self._thread_store_path.exists():
|
||||
logger.debug("[Discord] no thread mappings file at %s", self._thread_store_path)
|
||||
return
|
||||
data = json.loads(self._thread_store_path.read_text())
|
||||
self._active_threads.clear()
|
||||
self._active_thread_ids.clear()
|
||||
for channel_id, thread_id in data.items():
|
||||
self._active_threads[channel_id] = thread_id
|
||||
self._active_thread_ids.add(thread_id)
|
||||
if self._active_threads:
|
||||
logger.info("[Discord] restored %d thread mappings from %s", len(self._active_threads), self._thread_store_path)
|
||||
except Exception:
|
||||
logger.exception("[Discord] failed to load thread mappings")
|
||||
|
||||
def _save_thread(self, channel_id: str, thread_id: str) -> None:
|
||||
"""Persist a Discord thread mapping to the dedicated JSON file."""
|
||||
with self._thread_store_lock:
|
||||
try:
|
||||
data: dict[str, str] = {}
|
||||
if self._thread_store_path.exists():
|
||||
data = json.loads(self._thread_store_path.read_text())
|
||||
old_id = data.get(channel_id)
|
||||
data[channel_id] = thread_id
|
||||
# Update reverse-lookup set
|
||||
if old_id:
|
||||
self._active_thread_ids.discard(old_id)
|
||||
self._active_thread_ids.add(thread_id)
|
||||
self._thread_store_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
self._thread_store_path.write_text(json.dumps(data, indent=2))
|
||||
except Exception:
|
||||
logger.exception("[Discord] failed to save thread mapping for channel %s", channel_id)
|
||||
|
||||
async def stop(self) -> None:
|
||||
self._running = False
|
||||
self.bus.unsubscribe_outbound(self._on_outbound)
|
||||
|
||||
# Cancel all active typing indicator tasks
|
||||
for target_id, task in list(self._typing_tasks.items()):
|
||||
if not task.done():
|
||||
task.cancel()
|
||||
logger.debug("[Discord] cancelled typing task for target %s", target_id)
|
||||
self._typing_tasks.clear()
|
||||
|
||||
if self._client and self._discord_loop and self._discord_loop.is_running():
|
||||
close_future = asyncio.run_coroutine_threadsafe(self._client.close(), self._discord_loop)
|
||||
try:
|
||||
await asyncio.wait_for(asyncio.wrap_future(close_future), timeout=10)
|
||||
except TimeoutError:
|
||||
logger.warning("[Discord] client close timed out after 10s")
|
||||
except Exception:
|
||||
logger.exception("[Discord] error while closing client")
|
||||
|
||||
if self._thread:
|
||||
self._thread.join(timeout=10)
|
||||
self._thread = None
|
||||
|
||||
self._client = None
|
||||
self._discord_loop = None
|
||||
self._discord_module = None
|
||||
logger.info("Discord channel stopped")
|
||||
|
||||
async def send(self, msg: OutboundMessage) -> None:
|
||||
# Stop typing indicator once we're sending the response
|
||||
stop_future = asyncio.run_coroutine_threadsafe(self._stop_typing(msg.chat_id, msg.thread_ts), self._discord_loop)
|
||||
await asyncio.wrap_future(stop_future)
|
||||
|
||||
target = await self._resolve_target(msg)
|
||||
if target is None:
|
||||
logger.error("[Discord] target not found for chat_id=%s thread_ts=%s", msg.chat_id, msg.thread_ts)
|
||||
return
|
||||
|
||||
text = msg.text or ""
|
||||
for chunk in self._split_text(text):
|
||||
send_future = asyncio.run_coroutine_threadsafe(target.send(chunk), self._discord_loop)
|
||||
await asyncio.wrap_future(send_future)
|
||||
|
||||
async def send_file(self, msg: OutboundMessage, attachment: ResolvedAttachment) -> bool:
|
||||
stop_future = asyncio.run_coroutine_threadsafe(self._stop_typing(msg.chat_id, msg.thread_ts), self._discord_loop)
|
||||
await asyncio.wrap_future(stop_future)
|
||||
|
||||
target = await self._resolve_target(msg)
|
||||
if target is None:
|
||||
logger.error("[Discord] target not found for file upload chat_id=%s thread_ts=%s", msg.chat_id, msg.thread_ts)
|
||||
return False
|
||||
|
||||
if self._discord_module is None:
|
||||
return False
|
||||
|
||||
try:
|
||||
fp = open(str(attachment.actual_path), "rb") # noqa: SIM115
|
||||
file = self._discord_module.File(fp, filename=attachment.filename)
|
||||
send_future = asyncio.run_coroutine_threadsafe(target.send(file=file), self._discord_loop)
|
||||
await asyncio.wrap_future(send_future)
|
||||
logger.info("[Discord] file uploaded: %s", attachment.filename)
|
||||
return True
|
||||
except Exception:
|
||||
logger.exception("[Discord] failed to upload file: %s", attachment.filename)
|
||||
return False
|
||||
|
||||
async def _start_typing(self, channel, chat_id: str, thread_ts: str | None = None) -> None:
|
||||
"""Starts a loop to send periodic typing indicators."""
|
||||
target_id = thread_ts or chat_id
|
||||
if target_id in self._typing_tasks:
|
||||
return # Already typing for this target
|
||||
|
||||
async def _typing_loop():
|
||||
try:
|
||||
while True:
|
||||
try:
|
||||
await channel.trigger_typing()
|
||||
except Exception:
|
||||
pass
|
||||
await asyncio.sleep(10)
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
|
||||
task = asyncio.create_task(_typing_loop())
|
||||
self._typing_tasks[target_id] = task
|
||||
|
||||
async def _stop_typing(self, chat_id: str, thread_ts: str | None = None) -> None:
|
||||
"""Stops the typing loop for a specific target."""
|
||||
target_id = thread_ts or chat_id
|
||||
task = self._typing_tasks.pop(target_id, None)
|
||||
if task and not task.done():
|
||||
task.cancel()
|
||||
logger.debug("[Discord] stopped typing indicator for target %s", target_id)
|
||||
|
||||
async def _add_reaction(self, message) -> None:
|
||||
"""Add a checkmark reaction to acknowledge the message was received."""
|
||||
try:
|
||||
await message.add_reaction("✅")
|
||||
except Exception:
|
||||
logger.debug("[Discord] failed to add reaction to message %s", message.id, exc_info=True)
|
||||
|
||||
async def _on_message(self, message) -> None:
|
||||
if not self._running or not self._client:
|
||||
return
|
||||
|
||||
if message.author.bot:
|
||||
return
|
||||
|
||||
if self._client.user and message.author.id == self._client.user.id:
|
||||
return
|
||||
|
||||
guild = message.guild
|
||||
if self._allowed_guilds:
|
||||
if guild is None or guild.id not in self._allowed_guilds:
|
||||
return
|
||||
|
||||
text = (message.content or "").strip()
|
||||
if not text:
|
||||
return
|
||||
|
||||
if self._discord_module is None:
|
||||
return
|
||||
|
||||
# Determine whether the bot is mentioned in this message
|
||||
user = self._client.user if self._client else None
|
||||
if user:
|
||||
bot_mention = user.mention # <@ID>
|
||||
alt_mention = f"<@!{user.id}>" # <@!ID> (ping variant)
|
||||
standard_mention = f"<@{user.id}>"
|
||||
else:
|
||||
bot_mention = None
|
||||
alt_mention = None
|
||||
standard_mention = ""
|
||||
has_mention = (bot_mention and bot_mention in message.content) or (alt_mention and alt_mention in message.content) or (standard_mention and standard_mention in message.content)
|
||||
|
||||
# Strip mention from text for processing
|
||||
if has_mention:
|
||||
text = text.replace(bot_mention or "", "").replace(alt_mention or "", "").replace(standard_mention or "", "").strip()
|
||||
# Don't return early if text is empty — still process the mention (e.g., create thread)
|
||||
|
||||
# --- Determine thread/channel routing and typing target ---
|
||||
thread_id = None
|
||||
chat_id = None
|
||||
typing_target = None # The Discord object to type into
|
||||
|
||||
if isinstance(message.channel, self._discord_module.Thread):
|
||||
# --- Message already inside a thread ---
|
||||
thread_obj = message.channel
|
||||
thread_id = str(thread_obj.id)
|
||||
chat_id = str(thread_obj.parent_id or thread_obj.id)
|
||||
typing_target = thread_obj
|
||||
|
||||
# If this is a known active thread, process normally
|
||||
if thread_id in self._active_thread_ids:
|
||||
msg_type = InboundMessageType.COMMAND if is_known_channel_command(text) else InboundMessageType.CHAT
|
||||
inbound = self._make_inbound(
|
||||
chat_id=chat_id,
|
||||
user_id=str(message.author.id),
|
||||
text=text,
|
||||
msg_type=msg_type,
|
||||
thread_ts=thread_id,
|
||||
metadata={
|
||||
"guild_id": str(guild.id) if guild else None,
|
||||
"channel_id": str(message.channel.id),
|
||||
"message_id": str(message.id),
|
||||
},
|
||||
)
|
||||
inbound.topic_id = thread_id
|
||||
self._publish(inbound)
|
||||
# Start typing indicator in the thread
|
||||
if typing_target:
|
||||
asyncio.create_task(self._start_typing(typing_target, chat_id, thread_id))
|
||||
asyncio.create_task(self._add_reaction(message))
|
||||
return
|
||||
|
||||
# Thread not tracked (orphaned) — create new thread and handle below
|
||||
logger.debug("[Discord] message in orphaned thread %s, will create new thread", thread_id)
|
||||
thread_id = None
|
||||
typing_target = None
|
||||
|
||||
# At this point we're guaranteed to be in a channel, not a thread
|
||||
# (the Thread case is handled above). Apply mention_only for all
|
||||
# non-thread messages — no special case needed.
|
||||
channel_id = str(message.channel.id)
|
||||
|
||||
# Check if there's an active thread for this channel
|
||||
if channel_id in self._active_threads:
|
||||
# respect mention_only: if enabled, only process messages that mention the bot
|
||||
# (unless the channel is in allowed_channels)
|
||||
# Messages within a thread are always allowed through (continuation).
|
||||
# At this code point we know the message is in a channel, not a thread
|
||||
# (Thread case handled above), so always apply the check.
|
||||
if self._mention_only and not has_mention and channel_id not in self._allowed_channels:
|
||||
logger.debug("[Discord] skipping no-@ message in channel %s (not in thread)", channel_id)
|
||||
return
|
||||
# mention_only + fresh @ → create new thread instead of routing to existing one
|
||||
if self._mention_only and has_mention:
|
||||
thread_obj = await self._create_thread(message)
|
||||
if thread_obj is not None:
|
||||
target_thread_id = str(thread_obj.id)
|
||||
self._active_threads[channel_id] = target_thread_id
|
||||
self._save_thread(channel_id, target_thread_id)
|
||||
thread_id = target_thread_id
|
||||
chat_id = channel_id
|
||||
typing_target = thread_obj
|
||||
logger.info("[Discord] created new thread %s in channel %s on mention (replacing existing thread)", target_thread_id, channel_id)
|
||||
else:
|
||||
logger.info("[Discord] thread creation failed in channel %s, falling back to channel replies", channel_id)
|
||||
thread_id = channel_id
|
||||
chat_id = channel_id
|
||||
typing_target = message.channel
|
||||
else:
|
||||
# Existing session → route to the existing thread
|
||||
target_thread_id = self._active_threads[channel_id]
|
||||
logger.debug("[Discord] routing message in channel %s to existing thread %s", channel_id, target_thread_id)
|
||||
thread_id = target_thread_id
|
||||
chat_id = channel_id
|
||||
typing_target = await self._get_channel_or_thread(target_thread_id)
|
||||
elif self._mention_only and not has_mention and channel_id not in self._allowed_channels:
|
||||
# Not mentioned and not in an allowed channel → skip
|
||||
logger.debug("[Discord] skipping message without mention in channel %s", channel_id)
|
||||
return
|
||||
elif self._mention_only and has_mention:
|
||||
# First mention in this channel → create thread
|
||||
thread_obj = await self._create_thread(message)
|
||||
if thread_obj is not None:
|
||||
target_thread_id = str(thread_obj.id)
|
||||
self._active_threads[channel_id] = target_thread_id
|
||||
self._save_thread(channel_id, target_thread_id)
|
||||
thread_id = target_thread_id
|
||||
chat_id = channel_id
|
||||
typing_target = thread_obj # Type into the new thread
|
||||
logger.info("[Discord] created thread %s in channel %s for user %s", target_thread_id, channel_id, message.author.display_name)
|
||||
else:
|
||||
# Fallback: thread creation failed (disabled/permissions), reply in channel
|
||||
logger.info("[Discord] thread creation failed in channel %s, falling back to channel replies", channel_id)
|
||||
thread_id = channel_id
|
||||
chat_id = channel_id
|
||||
typing_target = message.channel # Type into the channel
|
||||
elif self._thread_mode:
|
||||
# thread_mode but mention_only is False → create thread anyway for conversation grouping
|
||||
thread_obj = await self._create_thread(message)
|
||||
if thread_obj is None:
|
||||
# Thread creation failed (disabled/permissions), fall back to channel replies
|
||||
logger.info("[Discord] thread creation failed in channel %s, falling back to channel replies", channel_id)
|
||||
thread_id = channel_id
|
||||
chat_id = channel_id
|
||||
typing_target = message.channel # Type into the channel
|
||||
else:
|
||||
target_thread_id = str(thread_obj.id)
|
||||
self._active_threads[channel_id] = target_thread_id
|
||||
self._save_thread(channel_id, target_thread_id)
|
||||
thread_id = target_thread_id
|
||||
chat_id = channel_id
|
||||
typing_target = thread_obj # Type into the new thread
|
||||
else:
|
||||
# No threading — reply directly in channel
|
||||
thread_id = channel_id
|
||||
chat_id = channel_id
|
||||
typing_target = message.channel # Type into the channel
|
||||
|
||||
msg_type = InboundMessageType.COMMAND if is_known_channel_command(text) else InboundMessageType.CHAT
|
||||
inbound = self._make_inbound(
|
||||
chat_id=chat_id,
|
||||
user_id=str(message.author.id),
|
||||
text=text,
|
||||
msg_type=msg_type,
|
||||
thread_ts=thread_id,
|
||||
metadata={
|
||||
"guild_id": str(guild.id) if guild else None,
|
||||
"channel_id": str(message.channel.id),
|
||||
"message_id": str(message.id),
|
||||
},
|
||||
)
|
||||
inbound.topic_id = thread_id
|
||||
|
||||
# Start typing indicator in the correct target (thread or channel)
|
||||
if typing_target:
|
||||
asyncio.create_task(self._start_typing(typing_target, chat_id, thread_id))
|
||||
|
||||
self._publish(inbound)
|
||||
asyncio.create_task(self._add_reaction(message))
|
||||
|
||||
def _publish(self, inbound) -> None:
|
||||
"""Publish an inbound message to the main event loop."""
|
||||
if self._main_loop and self._main_loop.is_running():
|
||||
future = asyncio.run_coroutine_threadsafe(self.bus.publish_inbound(inbound), self._main_loop)
|
||||
future.add_done_callback(lambda f: logger.exception("[Discord] publish_inbound failed", exc_info=f.exception()) if f.exception() else None)
|
||||
|
||||
def _run_client(self) -> None:
|
||||
self._discord_loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(self._discord_loop)
|
||||
try:
|
||||
self._discord_loop.run_until_complete(self._client.start(self._bot_token))
|
||||
except Exception:
|
||||
if self._running:
|
||||
logger.exception("Discord client error")
|
||||
finally:
|
||||
try:
|
||||
if self._client and not self._client.is_closed():
|
||||
self._discord_loop.run_until_complete(self._client.close())
|
||||
except Exception:
|
||||
logger.exception("Error during Discord shutdown")
|
||||
|
||||
async def _create_thread(self, message):
|
||||
try:
|
||||
if self._discord_module is None:
|
||||
return None
|
||||
|
||||
# Only TextChannel (type 0) and NewsChannel (type 10) support threads
|
||||
channel_type = message.channel.type
|
||||
if channel_type not in (
|
||||
self._discord_module.ChannelType.text,
|
||||
self._discord_module.ChannelType.news,
|
||||
):
|
||||
logger.info(
|
||||
"[Discord] channel type %s (%s) does not support threads",
|
||||
channel_type.value,
|
||||
channel_type.name,
|
||||
)
|
||||
return None
|
||||
|
||||
thread_name = f"deerflow-{message.author.display_name}-{message.id}"[:100]
|
||||
return await message.create_thread(name=thread_name)
|
||||
except self._discord_module.errors.HTTPException as exc:
|
||||
if exc.code == 50024:
|
||||
logger.info(
|
||||
"[Discord] cannot create thread in channel %s (error code 50024): %s",
|
||||
message.channel.id,
|
||||
channel_type.name if (channel_type := message.channel.type) else "unknown",
|
||||
)
|
||||
else:
|
||||
logger.exception(
|
||||
"[Discord] failed to create thread for message=%s (HTTPException %s)",
|
||||
message.id,
|
||||
exc.code,
|
||||
)
|
||||
return None
|
||||
except Exception:
|
||||
logger.exception("[Discord] failed to create thread for message=%s (threads may be disabled or missing permissions)", message.id)
|
||||
return None
|
||||
|
||||
async def _resolve_target(self, msg: OutboundMessage):
|
||||
if not self._client or not self._discord_loop:
|
||||
return None
|
||||
|
||||
target_ids: list[str] = []
|
||||
if msg.thread_ts:
|
||||
target_ids.append(msg.thread_ts)
|
||||
if msg.chat_id and msg.chat_id not in target_ids:
|
||||
target_ids.append(msg.chat_id)
|
||||
|
||||
for raw_id in target_ids:
|
||||
target = await self._get_channel_or_thread(raw_id)
|
||||
if target is not None:
|
||||
return target
|
||||
return None
|
||||
|
||||
async def _get_channel_or_thread(self, raw_id: str):
|
||||
if not self._client or not self._discord_loop:
|
||||
return None
|
||||
|
||||
try:
|
||||
target_id = int(raw_id)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
get_future = asyncio.run_coroutine_threadsafe(self._fetch_channel(target_id), self._discord_loop)
|
||||
try:
|
||||
return await asyncio.wrap_future(get_future)
|
||||
except Exception:
|
||||
logger.exception("[Discord] failed to resolve target id=%s", raw_id)
|
||||
return None
|
||||
|
||||
async def _fetch_channel(self, target_id: int):
|
||||
if not self._client:
|
||||
return None
|
||||
|
||||
channel = self._client.get_channel(target_id)
|
||||
if channel is not None:
|
||||
return channel
|
||||
|
||||
try:
|
||||
return await self._client.fetch_channel(target_id)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def _split_text(text: str) -> list[str]:
|
||||
if not text:
|
||||
return [""]
|
||||
|
||||
chunks: list[str] = []
|
||||
remaining = text
|
||||
while len(remaining) > _DISCORD_MAX_MESSAGE_LEN:
|
||||
split_at = remaining.rfind("\n", 0, _DISCORD_MAX_MESSAGE_LEN)
|
||||
if split_at <= 0:
|
||||
split_at = _DISCORD_MAX_MESSAGE_LEN
|
||||
chunks.append(remaining[:split_at])
|
||||
remaining = remaining[split_at:].lstrip("\n")
|
||||
|
||||
if remaining:
|
||||
chunks.append(remaining)
|
||||
|
||||
return chunks
|
||||
+15
-198
@@ -7,30 +7,21 @@ import json
|
||||
import logging
|
||||
import re
|
||||
import threading
|
||||
import time
|
||||
from typing import Any, Literal
|
||||
|
||||
from app.channels.base import Channel
|
||||
from app.channels.commands import is_known_channel_command
|
||||
from app.channels.message_bus import (
|
||||
PENDING_CLARIFICATION_METADATA_KEY,
|
||||
RESOLVED_FROM_PENDING_CLARIFICATION_METADATA_KEY,
|
||||
InboundMessage,
|
||||
InboundMessageType,
|
||||
MessageBus,
|
||||
OutboundMessage,
|
||||
ResolvedAttachment,
|
||||
)
|
||||
from app.channels.commands import KNOWN_CHANNEL_COMMANDS
|
||||
from app.channels.message_bus import InboundMessage, InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
|
||||
from deerflow.config.paths import VIRTUAL_PATH_PREFIX, get_paths
|
||||
from deerflow.runtime.user_context import get_effective_user_id
|
||||
from deerflow.sandbox.sandbox_provider import get_sandbox_provider
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
PENDING_CLARIFICATION_TTL_SECONDS = 30 * 60
|
||||
|
||||
|
||||
def _is_feishu_command(text: str) -> bool:
|
||||
return is_known_channel_command(text)
|
||||
if not text.startswith("/"):
|
||||
return False
|
||||
return text.split(maxsplit=1)[0].lower() in KNOWN_CHANNEL_COMMANDS
|
||||
|
||||
|
||||
class FeishuChannel(Channel):
|
||||
@@ -64,7 +55,6 @@ class FeishuChannel(Channel):
|
||||
self._background_tasks: set[asyncio.Task] = set()
|
||||
self._running_card_ids: dict[str, str] = {}
|
||||
self._running_card_tasks: dict[str, asyncio.Task] = {}
|
||||
self._pending_clarifications: dict[tuple[str, str], list[dict[str, Any]]] = {}
|
||||
self._CreateFileRequest = None
|
||||
self._CreateFileRequestBody = None
|
||||
self._CreateImageRequest = None
|
||||
@@ -72,20 +62,6 @@ class FeishuChannel(Channel):
|
||||
self._GetMessageResourceRequest = None
|
||||
self._thread_lock = threading.Lock()
|
||||
|
||||
@staticmethod
|
||||
def _non_empty_str(value: Any) -> str | None:
|
||||
if isinstance(value, str) and value.strip():
|
||||
return value.strip()
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def _pending_key(chat_id: str, user_id: str) -> tuple[str, str]:
|
||||
return (chat_id, user_id)
|
||||
|
||||
@property
|
||||
def supports_streaming(self) -> bool:
|
||||
return True
|
||||
|
||||
async def start(self) -> None:
|
||||
if self._running:
|
||||
return
|
||||
@@ -368,9 +344,8 @@ class FeishuChannel(Channel):
|
||||
return f"Failed to obtain the [{type}]"
|
||||
|
||||
paths = get_paths()
|
||||
user_id = get_effective_user_id()
|
||||
paths.ensure_thread_dirs(thread_id, user_id=user_id)
|
||||
uploads_dir = paths.sandbox_uploads_dir(thread_id, user_id=user_id).resolve()
|
||||
paths.ensure_thread_dirs(thread_id)
|
||||
uploads_dir = paths.sandbox_uploads_dir(thread_id).resolve()
|
||||
|
||||
ext = "png" if type == "image" else "bin"
|
||||
raw_filename = getattr(response, "file_name", "") or f"feishu_{file_key[-12:]}.{ext}"
|
||||
@@ -550,25 +525,18 @@ class FeishuChannel(Channel):
|
||||
"[Feishu] failed to patch running card %s, falling back to final reply",
|
||||
running_card_id,
|
||||
)
|
||||
fallback_card_id = await self._reply_card(source_message_id, msg.text)
|
||||
self._remember_thread_mapping(msg, source_message_id, fallback_card_id)
|
||||
self._remember_pending_clarification(msg, fallback_card_id)
|
||||
await self._reply_card(source_message_id, msg.text)
|
||||
else:
|
||||
self._remember_thread_mapping(msg, source_message_id, running_card_id)
|
||||
self._remember_pending_clarification(msg, running_card_id)
|
||||
logger.info("[Feishu] running card updated: source=%s card=%s", source_message_id, running_card_id)
|
||||
elif msg.is_final:
|
||||
final_card_id = await self._reply_card(source_message_id, msg.text)
|
||||
self._remember_thread_mapping(msg, source_message_id, final_card_id)
|
||||
self._remember_pending_clarification(msg, final_card_id)
|
||||
await self._reply_card(source_message_id, msg.text)
|
||||
elif awaited_running_card_task:
|
||||
logger.warning(
|
||||
"[Feishu] running card task finished without message_id for source=%s, skipping duplicate non-final creation",
|
||||
source_message_id,
|
||||
)
|
||||
else:
|
||||
created_card_id = await self._ensure_running_card(source_message_id, msg.text)
|
||||
self._remember_thread_mapping(msg, source_message_id, created_card_id)
|
||||
await self._ensure_running_card(source_message_id, msg.text)
|
||||
|
||||
if msg.is_final:
|
||||
self._running_card_ids.pop(source_message_id, None)
|
||||
@@ -579,129 +547,6 @@ class FeishuChannel(Channel):
|
||||
|
||||
# -- internal ----------------------------------------------------------
|
||||
|
||||
def _remember_thread_mapping(self, msg: OutboundMessage, *topic_ids: str | None) -> None:
|
||||
store = self.config.get("channel_store")
|
||||
if store is None or not msg.thread_id:
|
||||
return
|
||||
|
||||
metadata_topic_ids = [
|
||||
msg.metadata.get("message_id"),
|
||||
msg.metadata.get("root_id"),
|
||||
msg.metadata.get("parent_id"),
|
||||
msg.metadata.get("thread_id"),
|
||||
msg.metadata.get("topic_id"),
|
||||
]
|
||||
user_id = ""
|
||||
raw_user_id = msg.metadata.get("user_id")
|
||||
if isinstance(raw_user_id, str):
|
||||
user_id = raw_user_id
|
||||
|
||||
seen: set[str] = set()
|
||||
for topic_id in [*topic_ids, *metadata_topic_ids]:
|
||||
topic_id = self._non_empty_str(topic_id)
|
||||
if not topic_id or topic_id in seen:
|
||||
continue
|
||||
seen.add(topic_id)
|
||||
try:
|
||||
store.set_thread_id(
|
||||
self.name,
|
||||
msg.chat_id,
|
||||
msg.thread_id,
|
||||
topic_id=topic_id,
|
||||
user_id=user_id,
|
||||
)
|
||||
except Exception:
|
||||
logger.exception("[Feishu] failed to remember thread mapping for topic_id=%s", topic_id)
|
||||
|
||||
def _remember_pending_clarification(self, msg: OutboundMessage, card_message_id: str | None) -> None:
|
||||
if not msg.is_final or msg.metadata.get(PENDING_CLARIFICATION_METADATA_KEY) is not True:
|
||||
return
|
||||
|
||||
user_id = self._non_empty_str(msg.metadata.get("user_id"))
|
||||
topic_id = self._non_empty_str(msg.metadata.get("topic_id"))
|
||||
source_message_id = self._non_empty_str(msg.thread_ts) or self._non_empty_str(msg.metadata.get("message_id"))
|
||||
if not (user_id and topic_id and msg.thread_id and source_message_id and card_message_id):
|
||||
return
|
||||
|
||||
key = self._pending_key(msg.chat_id, user_id)
|
||||
pending = {
|
||||
"thread_id": msg.thread_id,
|
||||
"topic_id": topic_id,
|
||||
"source_message_id": source_message_id,
|
||||
"card_message_id": card_message_id,
|
||||
"created_at": time.time(),
|
||||
}
|
||||
with self._thread_lock:
|
||||
# Plain-message clarification continuity is a short-lived in-memory
|
||||
# hint; explicit Feishu replies are still covered by persisted
|
||||
# message-id mappings.
|
||||
self._pending_clarifications.setdefault(key, []).append(pending)
|
||||
logger.info(
|
||||
"[Feishu] pending clarification remembered: chat_id=%s user_id=%s topic_id=%s thread_id=%s",
|
||||
msg.chat_id,
|
||||
user_id,
|
||||
topic_id,
|
||||
msg.thread_id,
|
||||
)
|
||||
|
||||
def _consume_pending_clarification(self, chat_id: str, user_id: str) -> dict[str, Any] | None:
|
||||
key = self._pending_key(chat_id, user_id)
|
||||
with self._thread_lock:
|
||||
pending_items = self._pending_clarifications.get(key)
|
||||
if not pending_items:
|
||||
return None
|
||||
|
||||
now = time.time()
|
||||
while pending_items:
|
||||
pending = pending_items.pop(0)
|
||||
created_at = pending.get("created_at")
|
||||
if isinstance(created_at, (int, float)) and now - created_at <= PENDING_CLARIFICATION_TTL_SECONDS:
|
||||
if pending_items:
|
||||
self._pending_clarifications[key] = pending_items
|
||||
else:
|
||||
self._pending_clarifications.pop(key, None)
|
||||
return pending
|
||||
logger.info("[Feishu] pending clarification expired: chat_id=%s user_id=%s", chat_id, user_id)
|
||||
|
||||
self._pending_clarifications.pop(key, None)
|
||||
return None
|
||||
|
||||
def _ensure_pending_thread_mapping(self, chat_id: str, user_id: str, pending: dict[str, Any]) -> None:
|
||||
store = self.config.get("channel_store")
|
||||
topic_id = self._non_empty_str(pending.get("topic_id"))
|
||||
thread_id = self._non_empty_str(pending.get("thread_id"))
|
||||
if store is None or not topic_id or not thread_id:
|
||||
return
|
||||
try:
|
||||
store.set_thread_id(self.name, chat_id, thread_id, topic_id=topic_id, user_id=user_id)
|
||||
except Exception:
|
||||
logger.exception("[Feishu] failed to restore pending clarification mapping for topic_id=%s", topic_id)
|
||||
|
||||
def _resolve_topic_id(
|
||||
self,
|
||||
chat_id: str,
|
||||
msg_id: str,
|
||||
*,
|
||||
root_id: str | None,
|
||||
parent_id: str | None,
|
||||
thread_id: str | None,
|
||||
) -> tuple[str, bool]:
|
||||
store = self.config.get("channel_store")
|
||||
candidates = [root_id, parent_id, thread_id]
|
||||
|
||||
if store is not None:
|
||||
for candidate in candidates:
|
||||
candidate = self._non_empty_str(candidate)
|
||||
if not candidate:
|
||||
continue
|
||||
try:
|
||||
if store.get_thread_id(self.name, chat_id, topic_id=candidate):
|
||||
return candidate, True
|
||||
except Exception:
|
||||
logger.exception("[Feishu] failed to resolve stored topic mapping for topic_id=%s", candidate)
|
||||
|
||||
return root_id or msg_id, False
|
||||
|
||||
@staticmethod
|
||||
def _log_future_error(fut, name: str, msg_id: str) -> None:
|
||||
"""Callback for run_coroutine_threadsafe futures to surface errors."""
|
||||
@@ -742,9 +587,7 @@ class FeishuChannel(Channel):
|
||||
|
||||
# root_id is set when the message is a reply within a Feishu thread.
|
||||
# Use it as topic_id so all replies share the same DeerFlow thread.
|
||||
root_id = self._non_empty_str(getattr(message, "root_id", None))
|
||||
parent_id = self._non_empty_str(getattr(message, "parent_id", None))
|
||||
feishu_thread_id = self._non_empty_str(getattr(message, "thread_id", None))
|
||||
root_id = getattr(message, "root_id", None) or None
|
||||
|
||||
# Parse message content
|
||||
content = json.loads(message.content)
|
||||
@@ -805,12 +648,10 @@ class FeishuChannel(Channel):
|
||||
text = text.strip()
|
||||
|
||||
logger.info(
|
||||
"[Feishu] parsed message: chat_id=%s, msg_id=%s, root_id=%s, parent_id=%s, thread_id=%s, sender=%s, text=%r",
|
||||
"[Feishu] parsed message: chat_id=%s, msg_id=%s, root_id=%s, sender=%s, text=%r",
|
||||
chat_id,
|
||||
msg_id,
|
||||
root_id,
|
||||
parent_id,
|
||||
feishu_thread_id,
|
||||
sender_id,
|
||||
text[:100] if text else "",
|
||||
)
|
||||
@@ -826,24 +667,8 @@ class FeishuChannel(Channel):
|
||||
else:
|
||||
msg_type = InboundMessageType.CHAT
|
||||
|
||||
# Prefer any platform message id that already maps to a DeerFlow
|
||||
# thread. This keeps replies to bot clarification cards in the
|
||||
# original conversation even when Feishu reports the card as root.
|
||||
topic_id, resolved_from_stored_mapping = self._resolve_topic_id(
|
||||
chat_id,
|
||||
msg_id,
|
||||
root_id=root_id,
|
||||
parent_id=parent_id,
|
||||
thread_id=feishu_thread_id,
|
||||
)
|
||||
resolved_from_pending = False
|
||||
if msg_type == InboundMessageType.CHAT and not resolved_from_stored_mapping:
|
||||
pending = self._consume_pending_clarification(chat_id, sender_id)
|
||||
pending_topic_id = self._non_empty_str(pending.get("topic_id")) if pending else None
|
||||
if pending_topic_id:
|
||||
topic_id = pending_topic_id
|
||||
self._ensure_pending_thread_mapping(chat_id, sender_id, pending)
|
||||
resolved_from_pending = True
|
||||
# topic_id: use root_id for replies (same topic), msg_id for new messages (new topic)
|
||||
topic_id = root_id or msg_id
|
||||
|
||||
inbound = self._make_inbound(
|
||||
chat_id=chat_id,
|
||||
@@ -852,15 +677,7 @@ class FeishuChannel(Channel):
|
||||
msg_type=msg_type,
|
||||
thread_ts=msg_id,
|
||||
files=files_list,
|
||||
metadata={
|
||||
"message_id": msg_id,
|
||||
"root_id": root_id,
|
||||
"parent_id": parent_id,
|
||||
"thread_id": feishu_thread_id,
|
||||
"topic_id": topic_id,
|
||||
"user_id": sender_id,
|
||||
RESOLVED_FROM_PENDING_CLARIFICATION_METADATA_KEY: resolved_from_pending,
|
||||
},
|
||||
metadata={"message_id": msg_id, "root_id": root_id},
|
||||
)
|
||||
inbound.topic_id = topic_id
|
||||
|
||||
|
||||
+29
-290
@@ -1,4 +1,4 @@
|
||||
"""ChannelManager — consumes inbound messages and dispatches them to the DeerFlow agent via Gateway."""
|
||||
"""ChannelManager — consumes inbound messages and dispatches them to the DeerFlow agent via LangGraph Server."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -8,7 +8,6 @@ import mimetypes
|
||||
import re
|
||||
import time
|
||||
from collections.abc import Awaitable, Callable, Mapping
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
@@ -16,28 +15,12 @@ import httpx
|
||||
from langgraph_sdk.errors import ConflictError
|
||||
|
||||
from app.channels.commands import KNOWN_CHANNEL_COMMANDS
|
||||
from app.channels.message_bus import (
|
||||
PENDING_CLARIFICATION_METADATA_KEY,
|
||||
InboundMessage,
|
||||
InboundMessageType,
|
||||
MessageBus,
|
||||
OutboundMessage,
|
||||
ResolvedAttachment,
|
||||
)
|
||||
from app.channels.message_bus import InboundMessage, InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
|
||||
from app.channels.store import ChannelStore
|
||||
from app.gateway.csrf_middleware import CSRF_COOKIE_NAME, CSRF_HEADER_NAME, generate_csrf_token
|
||||
from app.gateway.internal_auth import create_internal_auth_headers
|
||||
from deerflow.config.agents_config import load_agent_config
|
||||
from deerflow.config.paths import make_safe_user_id
|
||||
from deerflow.runtime.user_context import get_effective_user_id
|
||||
from deerflow.skills.slash import parse_slash_skill_reference
|
||||
from deerflow.skills.storage import get_or_new_skill_storage
|
||||
from deerflow.skills.storage.skill_storage import SkillStorage
|
||||
from deerflow.utils.messages import ORIGINAL_USER_CONTENT_KEY
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
DEFAULT_LANGGRAPH_URL = "http://localhost:8001/api"
|
||||
DEFAULT_LANGGRAPH_URL = "http://localhost:2024"
|
||||
DEFAULT_GATEWAY_URL = "http://localhost:8001"
|
||||
DEFAULT_ASSISTANT_ID = "lead_agent"
|
||||
CUSTOM_AGENT_NAME_PATTERN = re.compile(r"^[A-Za-z0-9-]+$")
|
||||
@@ -52,8 +35,6 @@ STREAM_UPDATE_MIN_INTERVAL_SECONDS = 0.35
|
||||
THREAD_BUSY_MESSAGE = "This conversation is already processing another request. Please wait for it to finish and try again."
|
||||
|
||||
CHANNEL_CAPABILITIES = {
|
||||
"dingtalk": {"supports_streaming": False},
|
||||
"discord": {"supports_streaming": False},
|
||||
"feishu": {"supports_streaming": True},
|
||||
"slack": {"supports_streaming": False},
|
||||
"telegram": {"supports_streaming": False},
|
||||
@@ -63,13 +44,6 @@ CHANNEL_CAPABILITIES = {
|
||||
|
||||
InboundFileReader = Callable[[dict[str, Any], httpx.AsyncClient], Awaitable[bytes | None]]
|
||||
|
||||
_METADATA_DROP_KEYS = frozenset({"raw_message", "ref_msg"})
|
||||
|
||||
|
||||
def _slim_metadata(meta: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Return a shallow copy of *meta* with known-large keys removed."""
|
||||
return {k: v for k, v in meta.items() if k not in _METADATA_DROP_KEYS}
|
||||
|
||||
|
||||
INBOUND_FILE_READERS: dict[str, InboundFileReader] = {}
|
||||
|
||||
@@ -130,16 +104,6 @@ class InvalidChannelSessionConfigError(ValueError):
|
||||
"""Raised when IM channel session overrides contain invalid agent config."""
|
||||
|
||||
|
||||
class SlashSkillCommandResolutionError(RuntimeError):
|
||||
"""Raised when IM slash-skill command resolution cannot complete safely."""
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class _SlashSkillCommandResolution:
|
||||
route_to_chat: bool = False
|
||||
failure_message: str | None = None
|
||||
|
||||
|
||||
def _is_thread_busy_error(exc: BaseException | None) -> bool:
|
||||
if exc is None:
|
||||
return False
|
||||
@@ -179,6 +143,7 @@ def _extract_response_text(result: dict | list) -> str:
|
||||
Handles special cases:
|
||||
- Regular AI text responses
|
||||
- Clarification interrupts (``ask_clarification`` tool messages)
|
||||
- AI messages with tool_calls but no text content
|
||||
"""
|
||||
if isinstance(result, list):
|
||||
messages = result
|
||||
@@ -197,8 +162,6 @@ def _extract_response_text(result: dict | list) -> str:
|
||||
|
||||
# Stop at the last human message — anything before it is a previous turn
|
||||
if msg_type == "human":
|
||||
if _is_hidden_human_control_message(msg):
|
||||
continue
|
||||
break
|
||||
|
||||
# Check for tool messages from ask_clarification (interrupt case)
|
||||
@@ -226,54 +189,6 @@ def _extract_response_text(result: dict | list) -> str:
|
||||
return ""
|
||||
|
||||
|
||||
def _messages_from_result(result: dict | list) -> list[Any]:
|
||||
if isinstance(result, list):
|
||||
return result
|
||||
if isinstance(result, dict):
|
||||
messages = result.get("messages", [])
|
||||
if isinstance(messages, list):
|
||||
return messages
|
||||
return []
|
||||
|
||||
|
||||
def _current_turn_messages(result: dict | list) -> list[dict[str, Any]]:
|
||||
messages = _messages_from_result(result)
|
||||
current_turn: list[dict[str, Any]] = []
|
||||
for msg in reversed(messages):
|
||||
if not isinstance(msg, dict):
|
||||
continue
|
||||
if msg.get("type") == "human":
|
||||
break
|
||||
current_turn.append(msg)
|
||||
current_turn.reverse()
|
||||
return current_turn
|
||||
|
||||
|
||||
def _has_current_turn_clarification(result: dict | list) -> bool:
|
||||
"""Return True only when the current turn's final result is clarification."""
|
||||
for msg in reversed(_current_turn_messages(result)):
|
||||
msg_type = msg.get("type")
|
||||
if msg_type == "tool":
|
||||
return msg.get("name") == "ask_clarification"
|
||||
if msg_type == "ai":
|
||||
content = msg.get("content")
|
||||
if isinstance(content, str):
|
||||
if content:
|
||||
return False
|
||||
elif content:
|
||||
return False
|
||||
if msg.get("tool_calls"):
|
||||
return False
|
||||
return False
|
||||
|
||||
|
||||
def _response_metadata(base_metadata: dict[str, Any], *, pending_clarification: bool = False) -> dict[str, Any]:
|
||||
metadata = _slim_metadata(base_metadata)
|
||||
if pending_clarification:
|
||||
metadata[PENDING_CLARIFICATION_METADATA_KEY] = True
|
||||
return metadata
|
||||
|
||||
|
||||
def _extract_text_content(content: Any) -> str:
|
||||
"""Extract text from a streaming payload content field."""
|
||||
if isinstance(content, str):
|
||||
@@ -387,8 +302,6 @@ def _extract_artifacts(result: dict | list) -> list[str]:
|
||||
continue
|
||||
# Stop at the last human message — anything before it is a previous turn
|
||||
if msg.get("type") == "human":
|
||||
if _is_hidden_human_control_message(msg):
|
||||
continue
|
||||
break
|
||||
# Look for AI messages with present_files tool calls
|
||||
if msg.get("type") == "ai":
|
||||
@@ -401,18 +314,6 @@ def _extract_artifacts(result: dict | list) -> list[str]:
|
||||
return artifacts
|
||||
|
||||
|
||||
def _is_hidden_human_control_message(msg: Mapping[str, Any]) -> bool:
|
||||
"""Return whether a human message is an internal control message hidden from UI."""
|
||||
if msg.get("type") != "human":
|
||||
return False
|
||||
|
||||
additional_kwargs = msg.get("additional_kwargs")
|
||||
if not isinstance(additional_kwargs, Mapping):
|
||||
return False
|
||||
|
||||
return additional_kwargs.get("hide_from_ui") is True
|
||||
|
||||
|
||||
def _format_artifact_text(artifacts: list[str]) -> str:
|
||||
"""Format artifact paths into a human-readable text block listing filenames."""
|
||||
import posixpath
|
||||
@@ -426,46 +327,6 @@ def _format_artifact_text(artifacts: list[str]) -> str:
|
||||
_OUTPUTS_VIRTUAL_PREFIX = "/mnt/user-data/outputs/"
|
||||
|
||||
|
||||
def _unknown_command_reply(command: str | None = None) -> str:
|
||||
available = " | ".join(sorted(KNOWN_CHANNEL_COMMANDS))
|
||||
if command:
|
||||
return f"Unknown command: /{command}. Available commands: {available}"
|
||||
return f"Unknown command. Available commands: {available}"
|
||||
|
||||
|
||||
def _human_input_message(content: str, *, original_content: str | None = None) -> dict[str, Any]:
|
||||
message: dict[str, Any] = {"role": "human", "content": content}
|
||||
if original_content is not None and original_content != content:
|
||||
message["additional_kwargs"] = {ORIGINAL_USER_CONTENT_KEY: original_content}
|
||||
return message
|
||||
|
||||
|
||||
def _resolve_slash_skill_command(
|
||||
text: str,
|
||||
available_skills: set[str] | None = None,
|
||||
storage: SkillStorage | Callable[[], SkillStorage] | None = None,
|
||||
) -> _SlashSkillCommandResolution | None:
|
||||
reference = parse_slash_skill_reference(text)
|
||||
if reference is None:
|
||||
return None
|
||||
try:
|
||||
resolved_storage = storage() if callable(storage) else storage or get_or_new_skill_storage()
|
||||
skills = resolved_storage.load_skills(enabled_only=False)
|
||||
|
||||
skill = next((candidate for candidate in skills if candidate.name == reference.name), None)
|
||||
if skill is None:
|
||||
return None
|
||||
if not skill.enabled:
|
||||
return _SlashSkillCommandResolution(failure_message=f"Skill `/{reference.name}` is installed but disabled. Enable it before using slash activation.")
|
||||
if available_skills is not None and reference.name not in available_skills:
|
||||
return _SlashSkillCommandResolution(failure_message=f"Skill `/{reference.name}` is not available for this agent.")
|
||||
|
||||
return _SlashSkillCommandResolution(route_to_chat=True)
|
||||
except Exception as exc:
|
||||
logger.exception("[Manager] failed to resolve slash skill command")
|
||||
raise SlashSkillCommandResolutionError("Failed to resolve slash skill command. Please check the skill configuration.") from exc
|
||||
|
||||
|
||||
def _resolve_attachments(thread_id: str, artifacts: list[str]) -> list[ResolvedAttachment]:
|
||||
"""Resolve virtual artifact paths to host filesystem paths with metadata.
|
||||
|
||||
@@ -480,15 +341,14 @@ def _resolve_attachments(thread_id: str, artifacts: list[str]) -> list[ResolvedA
|
||||
|
||||
attachments: list[ResolvedAttachment] = []
|
||||
paths = get_paths()
|
||||
user_id = get_effective_user_id()
|
||||
outputs_dir = paths.sandbox_outputs_dir(thread_id, user_id=user_id).resolve()
|
||||
outputs_dir = paths.sandbox_outputs_dir(thread_id).resolve()
|
||||
for virtual_path in artifacts:
|
||||
# Security: only allow files from the agent outputs directory
|
||||
if not virtual_path.startswith(_OUTPUTS_VIRTUAL_PREFIX):
|
||||
logger.warning("[Manager] rejected non-outputs artifact path: %s", virtual_path)
|
||||
continue
|
||||
try:
|
||||
actual = paths.resolve_virtual_path(thread_id, virtual_path, user_id=user_id)
|
||||
actual = paths.resolve_virtual_path(thread_id, virtual_path)
|
||||
# Verify the resolved path is actually under the outputs directory
|
||||
# (guards against path-traversal even after prefix check)
|
||||
try:
|
||||
@@ -547,13 +407,7 @@ async def _ingest_inbound_files(thread_id: str, msg: InboundMessage) -> list[dic
|
||||
if not msg.files:
|
||||
return []
|
||||
|
||||
from deerflow.uploads.manager import (
|
||||
UnsafeUploadPathError,
|
||||
claim_unique_filename,
|
||||
ensure_uploads_dir,
|
||||
normalize_filename,
|
||||
write_upload_file_no_symlink,
|
||||
)
|
||||
from deerflow.uploads.manager import claim_unique_filename, ensure_uploads_dir, normalize_filename
|
||||
|
||||
uploads_dir = ensure_uploads_dir(thread_id)
|
||||
seen_names = {entry.name for entry in uploads_dir.iterdir() if entry.is_file()}
|
||||
@@ -604,10 +458,7 @@ async def _ingest_inbound_files(thread_id: str, msg: InboundMessage) -> list[dic
|
||||
|
||||
dest = uploads_dir / safe_name
|
||||
try:
|
||||
dest = write_upload_file_no_symlink(uploads_dir, safe_name, data)
|
||||
except UnsafeUploadPathError:
|
||||
logger.warning("[Manager] skipping inbound file with unsafe destination: %s", safe_name)
|
||||
continue
|
||||
dest.write_bytes(data)
|
||||
except Exception:
|
||||
logger.exception("[Manager] failed to write inbound file: %s", dest)
|
||||
continue
|
||||
@@ -655,7 +506,7 @@ class ChannelManager:
|
||||
"""Core dispatcher that bridges IM channels to the DeerFlow agent.
|
||||
|
||||
It reads from the MessageBus inbound queue, creates/reuses threads on
|
||||
Gateway's LangGraph-compatible API, sends messages via ``runs.wait``, and publishes
|
||||
the LangGraph Server, sends messages via ``runs.wait``, and publishes
|
||||
outbound responses back through the bus.
|
||||
"""
|
||||
|
||||
@@ -680,21 +531,12 @@ class ChannelManager:
|
||||
self._default_session = _as_dict(default_session)
|
||||
self._channel_sessions = dict(channel_sessions or {})
|
||||
self._client = None # lazy init — langgraph_sdk async client
|
||||
self._skill_storage: SkillStorage | None = None
|
||||
self._csrf_token = generate_csrf_token()
|
||||
self._semaphore: asyncio.Semaphore | None = None
|
||||
self._running = False
|
||||
self._task: asyncio.Task | None = None
|
||||
|
||||
@staticmethod
|
||||
def _channel_supports_streaming(channel_name: str) -> bool:
|
||||
from .service import get_channel_service
|
||||
|
||||
service = get_channel_service()
|
||||
if service:
|
||||
channel = service.get_channel(channel_name)
|
||||
if channel is not None:
|
||||
return channel.supports_streaming
|
||||
return CHANNEL_CAPABILITIES.get(channel_name, {}).get("supports_streaming", False)
|
||||
|
||||
def _resolve_session_layer(self, msg: InboundMessage) -> tuple[dict[str, Any], dict[str, Any]]:
|
||||
@@ -717,31 +559,12 @@ class ChannelManager:
|
||||
user_layer.get("config"),
|
||||
)
|
||||
|
||||
configurable = run_config.get("configurable")
|
||||
if isinstance(configurable, Mapping):
|
||||
configurable = dict(configurable)
|
||||
else:
|
||||
configurable = {}
|
||||
run_config["configurable"] = configurable
|
||||
# Pin channel-triggered runs to the root graph namespace so follow-up
|
||||
# turns continue from the same conversation checkpoint.
|
||||
configurable["checkpoint_ns"] = ""
|
||||
configurable["thread_id"] = thread_id
|
||||
|
||||
# ``user_id`` drives user-scoped filesystem buckets that only accept
|
||||
# ``[A-Za-z0-9_-]``, so normalize the channel id and keep the raw value
|
||||
# under ``channel_user_id`` for platform-facing lookups.
|
||||
run_context_identity: dict[str, Any] = {"thread_id": thread_id}
|
||||
if msg.user_id:
|
||||
run_context_identity["user_id"] = make_safe_user_id(msg.user_id)
|
||||
run_context_identity["channel_user_id"] = msg.user_id
|
||||
|
||||
run_context = _merge_dicts(
|
||||
DEFAULT_RUN_CONTEXT,
|
||||
self._default_session.get("context"),
|
||||
channel_layer.get("context"),
|
||||
user_layer.get("context"),
|
||||
run_context_identity,
|
||||
{"thread_id": thread_id},
|
||||
)
|
||||
|
||||
# Custom agents are implemented as lead_agent + agent_name context.
|
||||
@@ -753,21 +576,6 @@ class ChannelManager:
|
||||
|
||||
return assistant_id, run_config, run_context
|
||||
|
||||
def _resolve_available_skill_names(self, msg: InboundMessage) -> set[str] | None:
|
||||
thread_id = self.store.get_thread_id(msg.channel_name, msg.chat_id, topic_id=msg.topic_id) or ""
|
||||
_, _, run_context = self._resolve_run_params(msg, thread_id)
|
||||
if run_context.get("is_bootstrap"):
|
||||
return {"bootstrap"}
|
||||
|
||||
agent_name = run_context.get("agent_name")
|
||||
if not isinstance(agent_name, str) or not agent_name.strip():
|
||||
return None
|
||||
|
||||
agent_config = load_agent_config(_normalize_custom_agent_name(agent_name))
|
||||
if agent_config and agent_config.skills is not None:
|
||||
return set(agent_config.skills)
|
||||
return None
|
||||
|
||||
# -- LangGraph SDK client (lazy) ----------------------------------------
|
||||
|
||||
def _get_client(self):
|
||||
@@ -775,21 +583,9 @@ class ChannelManager:
|
||||
if self._client is None:
|
||||
from langgraph_sdk import get_client
|
||||
|
||||
self._client = get_client(
|
||||
url=self._langgraph_url,
|
||||
headers={
|
||||
**create_internal_auth_headers(),
|
||||
CSRF_HEADER_NAME: self._csrf_token,
|
||||
"Cookie": f"{CSRF_COOKIE_NAME}={self._csrf_token}",
|
||||
},
|
||||
)
|
||||
self._client = get_client(url=self._langgraph_url)
|
||||
return self._client
|
||||
|
||||
def _get_skill_storage(self) -> SkillStorage:
|
||||
if self._skill_storage is None:
|
||||
self._skill_storage = get_or_new_skill_storage()
|
||||
return self._skill_storage
|
||||
|
||||
# -- lifecycle ---------------------------------------------------------
|
||||
|
||||
async def start(self) -> None:
|
||||
@@ -859,14 +655,6 @@ class ChannelManager:
|
||||
exc,
|
||||
)
|
||||
await self._send_error(msg, str(exc))
|
||||
except SlashSkillCommandResolutionError as exc:
|
||||
logger.warning(
|
||||
"Slash skill command resolution failed for %s (chat=%s): %s",
|
||||
msg.channel_name,
|
||||
msg.chat_id,
|
||||
exc,
|
||||
)
|
||||
await self._send_error(msg, str(exc))
|
||||
except Exception:
|
||||
logger.exception(
|
||||
"Error handling message from %s (chat=%s)",
|
||||
@@ -878,7 +666,7 @@ class ChannelManager:
|
||||
# -- chat handling -----------------------------------------------------
|
||||
|
||||
async def _create_thread(self, client, msg: InboundMessage) -> str:
|
||||
"""Create a new thread through Gateway and store the mapping."""
|
||||
"""Create a new thread on the LangGraph Server and store the mapping."""
|
||||
thread = await client.threads.create()
|
||||
thread_id = thread["thread_id"]
|
||||
self.store.set_thread_id(
|
||||
@@ -888,7 +676,7 @@ class ChannelManager:
|
||||
topic_id=msg.topic_id,
|
||||
user_id=msg.user_id,
|
||||
)
|
||||
logger.info("[Manager] new thread created through Gateway: thread_id=%s for chat_id=%s topic_id=%s", thread_id, msg.chat_id, msg.topic_id)
|
||||
logger.info("[Manager] new thread created on LangGraph Server: thread_id=%s for chat_id=%s topic_id=%s", thread_id, msg.chat_id, msg.topic_id)
|
||||
return thread_id
|
||||
|
||||
async def _handle_chat(self, msg: InboundMessage, extra_context: dict[str, Any] | None = None) -> None:
|
||||
@@ -921,11 +709,9 @@ class ChannelManager:
|
||||
if extra_context:
|
||||
run_context.update(extra_context)
|
||||
|
||||
original_text = msg.text
|
||||
uploaded = await _ingest_inbound_files(thread_id, msg)
|
||||
if uploaded:
|
||||
msg.text = f"{_format_uploaded_files_block(uploaded)}\n\n{msg.text}".strip()
|
||||
human_message = _human_input_message(msg.text, original_content=original_text)
|
||||
|
||||
if self._channel_supports_streaming(msg.channel_name):
|
||||
await self._handle_streaming_chat(
|
||||
@@ -935,30 +721,19 @@ class ChannelManager:
|
||||
assistant_id,
|
||||
run_config,
|
||||
run_context,
|
||||
human_message,
|
||||
)
|
||||
return
|
||||
|
||||
logger.info("[Manager] invoking runs.wait(thread_id=%s, text=%r)", thread_id, msg.text[:100])
|
||||
try:
|
||||
result = await client.runs.wait(
|
||||
thread_id,
|
||||
assistant_id,
|
||||
input={"messages": [human_message]},
|
||||
input={"messages": [{"role": "human", "content": msg.text}]},
|
||||
config=run_config,
|
||||
context=run_context,
|
||||
multitask_strategy="reject",
|
||||
)
|
||||
except Exception as exc:
|
||||
if _is_thread_busy_error(exc):
|
||||
logger.warning("[Manager] thread busy (concurrent run rejected): thread_id=%s", thread_id)
|
||||
await self._send_error(msg, THREAD_BUSY_MESSAGE)
|
||||
return
|
||||
else:
|
||||
raise
|
||||
|
||||
response_text = _extract_response_text(result)
|
||||
pending_clarification = _has_current_turn_clarification(result)
|
||||
artifacts = _extract_artifacts(result)
|
||||
|
||||
logger.info(
|
||||
@@ -984,7 +759,6 @@ class ChannelManager:
|
||||
artifacts=artifacts,
|
||||
attachments=attachments,
|
||||
thread_ts=msg.thread_ts,
|
||||
metadata=_response_metadata(msg.metadata, pending_clarification=pending_clarification),
|
||||
)
|
||||
logger.info("[Manager] publishing outbound message to bus: channel=%s, chat_id=%s", msg.channel_name, msg.chat_id)
|
||||
await self.bus.publish_outbound(outbound)
|
||||
@@ -997,7 +771,6 @@ class ChannelManager:
|
||||
assistant_id: str,
|
||||
run_config: dict[str, Any],
|
||||
run_context: dict[str, Any],
|
||||
human_message: dict[str, Any],
|
||||
) -> None:
|
||||
logger.info("[Manager] invoking runs.stream(thread_id=%s, text=%r)", thread_id, msg.text[:100])
|
||||
|
||||
@@ -1013,7 +786,7 @@ class ChannelManager:
|
||||
async for chunk in client.runs.stream(
|
||||
thread_id,
|
||||
assistant_id,
|
||||
input={"messages": [human_message]},
|
||||
input={"messages": [{"role": "human", "content": msg.text}]},
|
||||
config=run_config,
|
||||
context=run_context,
|
||||
stream_mode=["messages-tuple", "values"],
|
||||
@@ -1047,7 +820,6 @@ class ChannelManager:
|
||||
text=latest_text,
|
||||
is_final=False,
|
||||
thread_ts=msg.thread_ts,
|
||||
metadata=_response_metadata(msg.metadata),
|
||||
)
|
||||
)
|
||||
last_published_text = latest_text
|
||||
@@ -1061,7 +833,6 @@ class ChannelManager:
|
||||
finally:
|
||||
result = last_values if last_values is not None else {"messages": [{"type": "ai", "content": latest_text}]}
|
||||
response_text = _extract_response_text(result)
|
||||
pending_clarification = _has_current_turn_clarification(result)
|
||||
artifacts = _extract_artifacts(result)
|
||||
response_text, attachments = _prepare_artifact_delivery(thread_id, response_text, artifacts)
|
||||
|
||||
@@ -1093,27 +864,17 @@ class ChannelManager:
|
||||
attachments=attachments,
|
||||
is_final=True,
|
||||
thread_ts=msg.thread_ts,
|
||||
metadata=_response_metadata(msg.metadata, pending_clarification=pending_clarification),
|
||||
)
|
||||
)
|
||||
|
||||
# -- command handling --------------------------------------------------
|
||||
|
||||
async def _handle_command(self, msg: InboundMessage) -> None:
|
||||
raw_text = msg.text
|
||||
text = raw_text.strip()
|
||||
text = msg.text.strip()
|
||||
parts = text.split(maxsplit=1)
|
||||
reply: str | None = None
|
||||
if not parts:
|
||||
command = None
|
||||
reply = _unknown_command_reply()
|
||||
else:
|
||||
command = parts[0].lower().removeprefix("/")
|
||||
command = parts[0].lower().lstrip("/")
|
||||
|
||||
if reply is None and not raw_text.startswith("/"):
|
||||
reply = _unknown_command_reply(command)
|
||||
|
||||
if reply is None and command == "bootstrap":
|
||||
if command == "bootstrap":
|
||||
from dataclasses import replace as _dc_replace
|
||||
|
||||
chat_text = parts[1] if len(parts) > 1 else "Initialize workspace"
|
||||
@@ -1121,8 +882,8 @@ class ChannelManager:
|
||||
await self._handle_chat(chat_msg, extra_context={"is_bootstrap": True})
|
||||
return
|
||||
|
||||
if reply is None and command == "new":
|
||||
# Create a new thread through Gateway
|
||||
if command == "new":
|
||||
# Create a new thread on the LangGraph Server
|
||||
client = self._get_client()
|
||||
thread = await client.threads.create()
|
||||
new_thread_id = thread["thread_id"]
|
||||
@@ -1134,14 +895,14 @@ class ChannelManager:
|
||||
user_id=msg.user_id,
|
||||
)
|
||||
reply = "New conversation started."
|
||||
elif reply is None and command == "status":
|
||||
elif command == "status":
|
||||
thread_id = self.store.get_thread_id(msg.channel_name, msg.chat_id, topic_id=msg.topic_id)
|
||||
reply = f"Active thread: {thread_id}" if thread_id else "No active conversation."
|
||||
elif reply is None and command == "models":
|
||||
elif command == "models":
|
||||
reply = await self._fetch_gateway("/api/models", "models")
|
||||
elif reply is None and command == "memory":
|
||||
elif command == "memory":
|
||||
reply = await self._fetch_gateway("/api/memory", "memory")
|
||||
elif reply is None and command == "help":
|
||||
elif command == "help":
|
||||
reply = (
|
||||
"Available commands:\n"
|
||||
"/bootstrap — Start a bootstrap session (enables agent setup)\n"
|
||||
@@ -1149,35 +910,18 @@ class ChannelManager:
|
||||
"/status — Show current thread info\n"
|
||||
"/models — List available models\n"
|
||||
"/memory — Show memory status\n"
|
||||
"/<skill-name> <task> — Activate an enabled skill for one turn\n"
|
||||
"/help — Show this help"
|
||||
)
|
||||
elif reply is None:
|
||||
slash_resolution = await asyncio.to_thread(
|
||||
lambda: _resolve_slash_skill_command(
|
||||
raw_text,
|
||||
self._resolve_available_skill_names(msg),
|
||||
self._get_skill_storage,
|
||||
)
|
||||
)
|
||||
if slash_resolution and slash_resolution.failure_message:
|
||||
reply = slash_resolution.failure_message
|
||||
elif slash_resolution and slash_resolution.route_to_chat:
|
||||
from dataclasses import replace as _dc_replace
|
||||
|
||||
chat_msg = _dc_replace(msg, msg_type=InboundMessageType.CHAT)
|
||||
await self._handle_chat(chat_msg)
|
||||
return
|
||||
else:
|
||||
reply = _unknown_command_reply(command)
|
||||
available = " | ".join(sorted(KNOWN_CHANNEL_COMMANDS))
|
||||
reply = f"Unknown command: /{command}. Available commands: {available}"
|
||||
|
||||
outbound = OutboundMessage(
|
||||
channel_name=msg.channel_name,
|
||||
chat_id=msg.chat_id,
|
||||
thread_id=self.store.get_thread_id(msg.channel_name, msg.chat_id, topic_id=msg.topic_id) or "",
|
||||
thread_id=self.store.get_thread_id(msg.channel_name, msg.chat_id) or "",
|
||||
text=reply,
|
||||
thread_ts=msg.thread_ts,
|
||||
metadata=_slim_metadata(msg.metadata),
|
||||
)
|
||||
await self.bus.publish_outbound(outbound)
|
||||
|
||||
@@ -1187,11 +931,7 @@ class ChannelManager:
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient() as http:
|
||||
resp = await http.get(
|
||||
f"{self._gateway_url}{path}",
|
||||
timeout=10,
|
||||
headers=create_internal_auth_headers(),
|
||||
)
|
||||
resp = await http.get(f"{self._gateway_url}{path}", timeout=10)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
except Exception:
|
||||
@@ -1212,9 +952,8 @@ class ChannelManager:
|
||||
outbound = OutboundMessage(
|
||||
channel_name=msg.channel_name,
|
||||
chat_id=msg.chat_id,
|
||||
thread_id=self.store.get_thread_id(msg.channel_name, msg.chat_id, topic_id=msg.topic_id) or "",
|
||||
thread_id=self.store.get_thread_id(msg.channel_name, msg.chat_id) or "",
|
||||
text=error_text,
|
||||
thread_ts=msg.thread_ts,
|
||||
metadata=_slim_metadata(msg.metadata),
|
||||
)
|
||||
await self.bus.publish_outbound(outbound)
|
||||
|
||||
@@ -13,9 +13,6 @@ from typing import Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
PENDING_CLARIFICATION_METADATA_KEY = "pending_clarification"
|
||||
RESOLVED_FROM_PENDING_CLARIFICATION_METADATA_KEY = "resolved_from_pending_clarification"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Message types
|
||||
|
||||
@@ -4,7 +4,7 @@ from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
from typing import TYPE_CHECKING, Any
|
||||
from typing import Any
|
||||
|
||||
from app.channels.base import Channel
|
||||
from app.channels.manager import DEFAULT_GATEWAY_URL, DEFAULT_LANGGRAPH_URL, ChannelManager
|
||||
@@ -13,13 +13,8 @@ from app.channels.store import ChannelStore
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from deerflow.config.app_config import AppConfig
|
||||
|
||||
# Channel name → import path for lazy loading
|
||||
_CHANNEL_REGISTRY: dict[str, str] = {
|
||||
"dingtalk": "app.channels.dingtalk:DingTalkChannel",
|
||||
"discord": "app.channels.discord:DiscordChannel",
|
||||
"feishu": "app.channels.feishu:FeishuChannel",
|
||||
"slack": "app.channels.slack:SlackChannel",
|
||||
"telegram": "app.channels.telegram:TelegramChannel",
|
||||
@@ -27,17 +22,6 @@ _CHANNEL_REGISTRY: dict[str, str] = {
|
||||
"wecom": "app.channels.wecom:WeComChannel",
|
||||
}
|
||||
|
||||
# Keys that indicate a user has configured credentials for a channel.
|
||||
_CHANNEL_CREDENTIAL_KEYS: dict[str, list[str]] = {
|
||||
"dingtalk": ["client_id", "client_secret"],
|
||||
"discord": ["bot_token"],
|
||||
"feishu": ["app_id", "app_secret"],
|
||||
"slack": ["bot_token", "app_token"],
|
||||
"telegram": ["bot_token"],
|
||||
"wecom": ["bot_id", "bot_secret"],
|
||||
"wechat": ["bot_token"],
|
||||
}
|
||||
|
||||
_CHANNELS_LANGGRAPH_URL_ENV = "DEER_FLOW_CHANNELS_LANGGRAPH_URL"
|
||||
_CHANNELS_GATEWAY_URL_ENV = "DEER_FLOW_CHANNELS_GATEWAY_URL"
|
||||
|
||||
@@ -80,15 +64,14 @@ class ChannelService:
|
||||
self._running = False
|
||||
|
||||
@classmethod
|
||||
def from_app_config(cls, app_config: AppConfig | None = None) -> ChannelService:
|
||||
def from_app_config(cls) -> ChannelService:
|
||||
"""Create a ChannelService from the application config."""
|
||||
if app_config is None:
|
||||
from deerflow.config.app_config import get_app_config
|
||||
|
||||
app_config = get_app_config()
|
||||
config = get_app_config()
|
||||
channels_config = {}
|
||||
# extra fields are allowed by AppConfig (extra="allow")
|
||||
extra = app_config.model_extra or {}
|
||||
extra = config.model_extra or {}
|
||||
if "channels" in extra:
|
||||
channels_config = extra["channels"]
|
||||
return cls(channels_config=channels_config)
|
||||
@@ -104,15 +87,6 @@ class ChannelService:
|
||||
if not isinstance(channel_config, dict):
|
||||
continue
|
||||
if not channel_config.get("enabled", False):
|
||||
cred_keys = _CHANNEL_CREDENTIAL_KEYS.get(name, [])
|
||||
has_creds = any(not isinstance(channel_config.get(k), bool) and channel_config.get(k) is not None and str(channel_config[k]).strip() for k in cred_keys)
|
||||
if has_creds:
|
||||
logger.warning(
|
||||
"Channel '%s' has credentials configured but is disabled. Set enabled: true under channels.%s in config.yaml to activate it.",
|
||||
name,
|
||||
name,
|
||||
)
|
||||
else:
|
||||
logger.info("Channel %s is disabled, skipping", name)
|
||||
continue
|
||||
|
||||
@@ -167,19 +141,12 @@ class ChannelService:
|
||||
return False
|
||||
|
||||
try:
|
||||
config = dict(config)
|
||||
config["channel_store"] = self.store
|
||||
channel = channel_cls(bus=self.bus, config=config)
|
||||
self._channels[name] = channel
|
||||
await channel.start()
|
||||
if not channel.is_running:
|
||||
self._channels.pop(name, None)
|
||||
logger.error("Channel %s did not enter a running state after start()", name)
|
||||
return False
|
||||
self._channels[name] = channel
|
||||
logger.info("Channel %s started", name)
|
||||
return True
|
||||
except Exception:
|
||||
self._channels.pop(name, None)
|
||||
logger.exception("Failed to start channel %s", name)
|
||||
return False
|
||||
|
||||
@@ -214,12 +181,12 @@ def get_channel_service() -> ChannelService | None:
|
||||
return _channel_service
|
||||
|
||||
|
||||
async def start_channel_service(app_config: AppConfig | None = None) -> ChannelService:
|
||||
async def start_channel_service() -> ChannelService:
|
||||
"""Create and start the global ChannelService from app config."""
|
||||
global _channel_service
|
||||
if _channel_service is not None:
|
||||
return _channel_service
|
||||
_channel_service = ChannelService.from_app_config(app_config)
|
||||
_channel_service = ChannelService.from_app_config()
|
||||
await _channel_service.start()
|
||||
return _channel_service
|
||||
|
||||
|
||||
@@ -9,7 +9,6 @@ from typing import Any
|
||||
from markdown_to_mrkdwn import SlackMarkdownConverter
|
||||
|
||||
from app.channels.base import Channel
|
||||
from app.channels.commands import is_known_channel_command
|
||||
from app.channels.message_bus import InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -17,45 +16,13 @@ logger = logging.getLogger(__name__)
|
||||
_slack_md_converter = SlackMarkdownConverter()
|
||||
|
||||
|
||||
def _normalize_allowed_users(allowed_users: Any) -> set[str]:
|
||||
if allowed_users is None:
|
||||
return set()
|
||||
if isinstance(allowed_users, str):
|
||||
values = [allowed_users]
|
||||
elif isinstance(allowed_users, list | tuple | set):
|
||||
values = allowed_users
|
||||
else:
|
||||
logger.warning(
|
||||
"Slack allowed_users should be a list of Slack user IDs or a single Slack user ID string; treating %s as one string value",
|
||||
type(allowed_users).__name__,
|
||||
)
|
||||
values = [allowed_users]
|
||||
return {str(user_id) for user_id in values if str(user_id)}
|
||||
|
||||
|
||||
def _strip_leading_slack_bot_mention(text: str, bot_user_id: str | None) -> str:
|
||||
if not bot_user_id:
|
||||
return text
|
||||
if not text.startswith("<@"):
|
||||
return text
|
||||
end = text.find(">")
|
||||
if end <= 2:
|
||||
return text
|
||||
mentioned_user_id = text[2:end].split("|", 1)[0].lstrip("!")
|
||||
if mentioned_user_id != bot_user_id:
|
||||
return text
|
||||
return text[end + 1 :].lstrip()
|
||||
|
||||
|
||||
class SlackChannel(Channel):
|
||||
"""Slack IM channel using Socket Mode (WebSocket, no public IP).
|
||||
|
||||
Configuration keys (in ``config.yaml`` under ``channels.slack``):
|
||||
- ``bot_token``: Slack Bot User OAuth Token (xoxb-...).
|
||||
- ``app_token``: Slack App-Level Token (xapp-...) for Socket Mode.
|
||||
- ``allowed_users``: (optional) List of allowed Slack user IDs, or a
|
||||
single Slack user ID string as shorthand. Empty = allow all. Other
|
||||
scalar values are treated as a single string with a warning.
|
||||
- ``allowed_users``: (optional) List of allowed Slack user IDs. Empty = allow all.
|
||||
"""
|
||||
|
||||
def __init__(self, bus: MessageBus, config: dict[str, Any]) -> None:
|
||||
@@ -63,9 +30,7 @@ class SlackChannel(Channel):
|
||||
self._socket_client = None
|
||||
self._web_client = None
|
||||
self._loop: asyncio.AbstractEventLoop | None = None
|
||||
self._allowed_users = _normalize_allowed_users(config.get("allowed_users", []))
|
||||
configured_bot_user_id = config.get("bot_user_id")
|
||||
self._bot_user_id = str(configured_bot_user_id).lstrip("@") if configured_bot_user_id else None
|
||||
self._allowed_users: set[str] = {str(user_id) for user_id in config.get("allowed_users", [])}
|
||||
|
||||
async def start(self) -> None:
|
||||
if self._running:
|
||||
@@ -89,17 +54,6 @@ class SlackChannel(Channel):
|
||||
return
|
||||
|
||||
self._web_client = WebClient(token=bot_token)
|
||||
if self._bot_user_id is None:
|
||||
try:
|
||||
auth_info = await asyncio.to_thread(self._web_client.auth_test)
|
||||
user_id = auth_info.get("user_id") if isinstance(auth_info, dict) else None
|
||||
if user_id is None:
|
||||
auth_get = getattr(auth_info, "get", None)
|
||||
user_id = auth_get("user_id") if callable(auth_get) else None
|
||||
if isinstance(user_id, str) and user_id:
|
||||
self._bot_user_id = user_id
|
||||
except Exception:
|
||||
logger.warning("[Slack] failed to resolve bot user id; app mention text may include the bot mention", exc_info=True)
|
||||
self._socket_client = SocketModeClient(
|
||||
app_token=app_token,
|
||||
web_client=self._web_client,
|
||||
@@ -238,12 +192,6 @@ class SlackChannel(Channel):
|
||||
if event_type != "events_api":
|
||||
return
|
||||
|
||||
if self._bot_user_id is None:
|
||||
authorization = next((item for item in req.payload.get("authorizations", []) if isinstance(item, dict)), None)
|
||||
user_id = authorization.get("user_id") if authorization else None
|
||||
if isinstance(user_id, str) and user_id:
|
||||
self._bot_user_id = user_id
|
||||
|
||||
event = req.payload.get("event", {})
|
||||
etype = event.get("type", "")
|
||||
|
||||
@@ -267,15 +215,13 @@ class SlackChannel(Channel):
|
||||
return
|
||||
|
||||
text = event.get("text", "").strip()
|
||||
if event.get("type") == "app_mention":
|
||||
text = _strip_leading_slack_bot_mention(text, self._bot_user_id)
|
||||
if not text:
|
||||
return
|
||||
|
||||
channel_id = event.get("channel", "")
|
||||
thread_ts = event.get("thread_ts") or event.get("ts", "")
|
||||
|
||||
if is_known_channel_command(text):
|
||||
if text.startswith("/"):
|
||||
msg_type = InboundMessageType.COMMAND
|
||||
else:
|
||||
msg_type = InboundMessageType.CHAT
|
||||
|
||||
@@ -60,17 +60,12 @@ class TelegramChannel(Channel):
|
||||
|
||||
# Command handlers
|
||||
app.add_handler(CommandHandler("start", self._cmd_start))
|
||||
app.add_handler(CommandHandler("bootstrap", self._cmd_generic))
|
||||
app.add_handler(CommandHandler("new", self._cmd_generic))
|
||||
app.add_handler(CommandHandler("status", self._cmd_generic))
|
||||
app.add_handler(CommandHandler("models", self._cmd_generic))
|
||||
app.add_handler(CommandHandler("memory", self._cmd_generic))
|
||||
app.add_handler(CommandHandler("help", self._cmd_generic))
|
||||
|
||||
# Slash skill commands are dynamic and cannot all be pre-registered
|
||||
# with Telegram, so route unknown slash commands through chat handling.
|
||||
app.add_handler(MessageHandler(filters.TEXT & filters.COMMAND, self._on_text))
|
||||
|
||||
# General message handler
|
||||
app.add_handler(MessageHandler(filters.TEXT & ~filters.COMMAND, self._on_text))
|
||||
|
||||
@@ -233,33 +228,6 @@ class TelegramChannel(Channel):
|
||||
return True
|
||||
return user_id in self._allowed_users
|
||||
|
||||
def _get_bot_username(self, context) -> str | None:
|
||||
bot = getattr(context, "bot", None)
|
||||
username = getattr(bot, "username", None)
|
||||
if not username and self._application is not None:
|
||||
username = getattr(getattr(self._application, "bot", None), "username", None)
|
||||
return str(username) if username else None
|
||||
|
||||
@staticmethod
|
||||
def _strip_bot_username_from_leading_command(text: str, bot_username: str | None) -> str:
|
||||
username = (bot_username or "").lstrip("@").lower()
|
||||
if not username or not text.startswith("/"):
|
||||
return text
|
||||
|
||||
parts = text.split(maxsplit=1)
|
||||
command_token = parts[0]
|
||||
if "@" not in command_token:
|
||||
return text
|
||||
|
||||
command_name, addressed_username = command_token[1:].rsplit("@", 1)
|
||||
if not command_name or addressed_username.lower() != username:
|
||||
return text
|
||||
|
||||
normalized = f"/{command_name}"
|
||||
if len(parts) > 1:
|
||||
normalized = f"{normalized} {parts[1]}"
|
||||
return normalized
|
||||
|
||||
async def _cmd_start(self, update, context) -> None:
|
||||
"""Handle /start command."""
|
||||
if not self._check_user(update.effective_user.id):
|
||||
@@ -275,7 +243,7 @@ class TelegramChannel(Channel):
|
||||
if not self._check_user(update.effective_user.id):
|
||||
return
|
||||
|
||||
text = self._strip_bot_username_from_leading_command(update.message.text.strip(), self._get_bot_username(context))
|
||||
text = update.message.text
|
||||
chat_id = str(update.effective_chat.id)
|
||||
user_id = str(update.effective_user.id)
|
||||
msg_id = str(update.message.message_id)
|
||||
@@ -311,7 +279,7 @@ class TelegramChannel(Channel):
|
||||
if not self._check_user(update.effective_user.id):
|
||||
return
|
||||
|
||||
text = self._strip_bot_username_from_leading_command(update.message.text.strip(), self._get_bot_username(context))
|
||||
text = update.message.text.strip()
|
||||
if not text:
|
||||
return
|
||||
|
||||
|
||||
@@ -22,7 +22,6 @@ from cryptography.hazmat.primitives import padding
|
||||
from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes
|
||||
|
||||
from app.channels.base import Channel
|
||||
from app.channels.commands import is_known_channel_command
|
||||
from app.channels.message_bus import InboundMessageType, MessageBus, OutboundMessage, ResolvedAttachment
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -621,7 +620,7 @@ class WechatChannel(Channel):
|
||||
chat_id=chat_id,
|
||||
user_id=chat_id,
|
||||
text=text,
|
||||
msg_type=InboundMessageType.COMMAND if is_known_channel_command(text) else InboundMessageType.CHAT,
|
||||
msg_type=InboundMessageType.COMMAND if text.startswith("/") else InboundMessageType.CHAT,
|
||||
thread_ts=thread_ts,
|
||||
files=files,
|
||||
metadata={
|
||||
|
||||
@@ -8,7 +8,6 @@ from collections.abc import Awaitable, Callable
|
||||
from typing import Any, cast
|
||||
|
||||
from app.channels.base import Channel
|
||||
from app.channels.commands import is_known_channel_command
|
||||
from app.channels.message_bus import (
|
||||
InboundMessageType,
|
||||
MessageBus,
|
||||
@@ -30,10 +29,6 @@ class WeComChannel(Channel):
|
||||
self._ws_stream_ids: dict[str, str] = {}
|
||||
self._working_message = "Working on it..."
|
||||
|
||||
@property
|
||||
def supports_streaming(self) -> bool:
|
||||
return True
|
||||
|
||||
def _clear_ws_context(self, thread_ts: str | None) -> None:
|
||||
if not thread_ts:
|
||||
return
|
||||
@@ -271,7 +266,7 @@ class WeComChannel(Channel):
|
||||
|
||||
user_id = (body.get("from") or {}).get("userid")
|
||||
|
||||
inbound_type = InboundMessageType.COMMAND if is_known_channel_command(text) else InboundMessageType.CHAT
|
||||
inbound_type = InboundMessageType.COMMAND if text.startswith("/") else InboundMessageType.CHAT
|
||||
inbound = self._make_inbound(
|
||||
chat_id=user_id, # keep user's conversation in memory
|
||||
user_id=user_id,
|
||||
|
||||
+15
-204
@@ -1,22 +1,16 @@
|
||||
import asyncio
|
||||
import logging
|
||||
from collections.abc import AsyncGenerator
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
|
||||
from app.gateway.auth_middleware import AuthMiddleware
|
||||
from app.gateway.config import get_gateway_config
|
||||
from app.gateway.csrf_middleware import CSRFMiddleware, get_configured_cors_origins
|
||||
from app.gateway.deps import langgraph_runtime
|
||||
from app.gateway.routers import (
|
||||
agents,
|
||||
artifacts,
|
||||
assistants_compat,
|
||||
auth,
|
||||
channels,
|
||||
feedback,
|
||||
mcp,
|
||||
memory,
|
||||
models,
|
||||
@@ -27,13 +21,9 @@ from app.gateway.routers import (
|
||||
threads,
|
||||
uploads,
|
||||
)
|
||||
from deerflow.config import app_config as deerflow_app_config
|
||||
from deerflow.config.app_config import apply_logging_level
|
||||
from deerflow.config.app_config import get_app_config
|
||||
|
||||
AppConfig = deerflow_app_config.AppConfig
|
||||
get_app_config = deerflow_app_config.get_app_config
|
||||
|
||||
# Default logging; lifespan overrides from config.yaml log_level.
|
||||
# Configure logging
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
|
||||
@@ -42,135 +32,14 @@ logging.basicConfig(
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Upper bound (seconds) each lifespan shutdown hook is allowed to run.
|
||||
# Bounds worker exit time so uvicorn's reload supervisor does not keep
|
||||
# firing signals into a worker that is stuck waiting for shutdown cleanup.
|
||||
_SHUTDOWN_HOOK_TIMEOUT_SECONDS = 5.0
|
||||
|
||||
|
||||
async def _ensure_admin_user(app: FastAPI) -> None:
|
||||
"""Startup hook: handle first boot and migrate orphan threads otherwise.
|
||||
|
||||
After admin creation, migrate orphan threads from the LangGraph
|
||||
store (metadata.user_id unset) to the admin account. This is the
|
||||
"no-auth → with-auth" upgrade path: users who ran DeerFlow without
|
||||
authentication have existing LangGraph thread data that needs an
|
||||
owner assigned.
|
||||
First boot (no admin exists):
|
||||
- Does NOT create any user accounts automatically.
|
||||
- The operator must visit ``/setup`` to create the first admin.
|
||||
|
||||
Subsequent boots (admin already exists):
|
||||
- Runs the one-time "no-auth → with-auth" orphan thread migration for
|
||||
existing LangGraph thread metadata that has no user_id.
|
||||
|
||||
No SQL persistence migration is needed: the four user_id columns
|
||||
(threads_meta, runs, run_events, feedback) only come into existence
|
||||
alongside the auth module via create_all, so freshly created tables
|
||||
never contain NULL-owner rows.
|
||||
"""
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.gateway.deps import get_local_provider
|
||||
from deerflow.persistence.engine import get_session_factory
|
||||
from deerflow.persistence.user.model import UserRow
|
||||
|
||||
try:
|
||||
provider = get_local_provider()
|
||||
except RuntimeError:
|
||||
# Auth persistence may not be initialized in some test/boot paths.
|
||||
# Skip admin migration work rather than failing gateway startup.
|
||||
logger.warning("Auth persistence not ready; skipping admin bootstrap check")
|
||||
return
|
||||
|
||||
sf = get_session_factory()
|
||||
if sf is None:
|
||||
return
|
||||
|
||||
admin_count = await provider.count_admin_users()
|
||||
|
||||
if admin_count == 0:
|
||||
logger.info("=" * 60)
|
||||
logger.info(" First boot detected — no admin account exists.")
|
||||
logger.info(" Visit /setup to complete admin account creation.")
|
||||
logger.info("=" * 60)
|
||||
return
|
||||
|
||||
# Admin already exists — run orphan thread migration for any
|
||||
# LangGraph thread metadata that pre-dates the auth module.
|
||||
async with sf() as session:
|
||||
stmt = select(UserRow).where(UserRow.system_role == "admin").limit(1)
|
||||
row = (await session.execute(stmt)).scalar_one_or_none()
|
||||
|
||||
if row is None:
|
||||
return # Should not happen (admin_count > 0 above), but be safe.
|
||||
|
||||
admin_id = str(row.id)
|
||||
|
||||
# LangGraph store orphan migration — non-fatal.
|
||||
# This covers the "no-auth → with-auth" upgrade path for users
|
||||
# whose existing LangGraph thread metadata has no user_id set.
|
||||
store = getattr(app.state, "store", None)
|
||||
if store is not None:
|
||||
try:
|
||||
migrated = await _migrate_orphaned_threads(store, admin_id)
|
||||
if migrated:
|
||||
logger.info("Migrated %d orphan LangGraph thread(s) to admin", migrated)
|
||||
except Exception:
|
||||
logger.exception("LangGraph thread migration failed (non-fatal)")
|
||||
|
||||
|
||||
async def _iter_store_items(store, namespace, *, page_size: int = 500):
|
||||
"""Paginated async iterator over a LangGraph store namespace.
|
||||
|
||||
Replaces the old hardcoded ``limit=1000`` call with a cursor-style
|
||||
loop so that environments with more than one page of orphans do
|
||||
not silently lose data. Terminates when a page is empty OR when a
|
||||
short page arrives (indicating the last page).
|
||||
"""
|
||||
offset = 0
|
||||
while True:
|
||||
batch = await store.asearch(namespace, limit=page_size, offset=offset)
|
||||
if not batch:
|
||||
return
|
||||
for item in batch:
|
||||
yield item
|
||||
if len(batch) < page_size:
|
||||
return
|
||||
offset += page_size
|
||||
|
||||
|
||||
async def _migrate_orphaned_threads(store, admin_user_id: str) -> int:
|
||||
"""Migrate LangGraph store threads with no user_id to the given admin.
|
||||
|
||||
Uses cursor pagination so all orphans are migrated regardless of
|
||||
count. Returns the number of rows migrated.
|
||||
"""
|
||||
migrated = 0
|
||||
async for item in _iter_store_items(store, ("threads",)):
|
||||
metadata = item.value.get("metadata", {})
|
||||
if not metadata.get("user_id"):
|
||||
metadata["user_id"] = admin_user_id
|
||||
item.value["metadata"] = metadata
|
||||
await store.aput(("threads",), item.key, item.value)
|
||||
migrated += 1
|
||||
return migrated
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
|
||||
"""Application lifespan handler."""
|
||||
|
||||
# Load config and check necessary environment variables at startup.
|
||||
# `startup_config` is a local snapshot used only for one-shot bootstrap
|
||||
# work (logging level, langgraph_runtime engines, channels). Request-time
|
||||
# config resolution always routes through `get_app_config()` in
|
||||
# `app/gateway/deps.py::get_config()` so `config.yaml` edits become
|
||||
# visible without a process restart. We deliberately do NOT cache this
|
||||
# snapshot on `app.state` to keep that contract enforceable.
|
||||
# Load config and check necessary environment variables at startup
|
||||
try:
|
||||
startup_config = get_app_config()
|
||||
apply_logging_level(startup_config.log_level)
|
||||
get_app_config()
|
||||
logger.info("Configuration loaded successfully")
|
||||
except Exception as e:
|
||||
error_msg = f"Failed to load configuration during gateway startup: {e}"
|
||||
@@ -179,57 +48,26 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
|
||||
config = get_gateway_config()
|
||||
logger.info(f"Starting API Gateway on {config.host}:{config.port}")
|
||||
|
||||
# Pre-warm tiktoken encoding cache so the first memory-injection request
|
||||
# never blocks on the BPE data download (which hits an OpenAI/Azure URL
|
||||
# that may be unreachable in restricted networks — see issue #3402).
|
||||
try:
|
||||
from deerflow.agents.memory.prompt import warm_tiktoken_cache
|
||||
|
||||
warmed = await asyncio.wait_for(
|
||||
asyncio.to_thread(warm_tiktoken_cache),
|
||||
timeout=5,
|
||||
)
|
||||
if warmed:
|
||||
logger.info("tiktoken encoding cache warmed successfully")
|
||||
else:
|
||||
logger.warning("tiktoken encoding cache warm-up failed; token counting will use character-based fallback")
|
||||
except TimeoutError:
|
||||
logger.warning("tiktoken encoding cache warm-up timed out; token counting will use character-based fallback")
|
||||
except Exception:
|
||||
logger.warning("tiktoken warm-up skipped", exc_info=True)
|
||||
|
||||
# Initialize LangGraph runtime components (StreamBridge, RunManager, checkpointer, store)
|
||||
async with langgraph_runtime(app, startup_config):
|
||||
async with langgraph_runtime(app):
|
||||
logger.info("LangGraph runtime initialised")
|
||||
|
||||
# Check admin bootstrap state and migrate orphan threads after admin exists.
|
||||
# Must run AFTER langgraph_runtime so app.state.store is available for thread migration
|
||||
await _ensure_admin_user(app)
|
||||
|
||||
# Start IM channel service if any channels are configured
|
||||
try:
|
||||
from app.channels.service import start_channel_service
|
||||
|
||||
channel_service = await start_channel_service(startup_config)
|
||||
channel_service = await start_channel_service()
|
||||
logger.info("Channel service started: %s", channel_service.get_status())
|
||||
except Exception:
|
||||
logger.exception("No IM channels configured or channel service failed to start")
|
||||
|
||||
yield
|
||||
|
||||
# Stop channel service on shutdown (bounded to prevent worker hang)
|
||||
# Stop channel service on shutdown
|
||||
try:
|
||||
from app.channels.service import stop_channel_service
|
||||
|
||||
await asyncio.wait_for(
|
||||
stop_channel_service(),
|
||||
timeout=_SHUTDOWN_HOOK_TIMEOUT_SECONDS,
|
||||
)
|
||||
except TimeoutError:
|
||||
logger.warning(
|
||||
"Channel service shutdown exceeded %.1fs; proceeding with worker exit.",
|
||||
_SHUTDOWN_HOOK_TIMEOUT_SECONDS,
|
||||
)
|
||||
await stop_channel_service()
|
||||
except Exception:
|
||||
logger.exception("Failed to stop channel service")
|
||||
|
||||
@@ -242,10 +80,6 @@ def create_app() -> FastAPI:
|
||||
Returns:
|
||||
Configured FastAPI application instance.
|
||||
"""
|
||||
config = get_gateway_config()
|
||||
docs_url = "/docs" if config.enable_docs else None
|
||||
redoc_url = "/redoc" if config.enable_docs else None
|
||||
openapi_url = "/openapi.json" if config.enable_docs else None
|
||||
|
||||
app = FastAPI(
|
||||
title="DeerFlow API Gateway",
|
||||
@@ -265,14 +99,14 @@ API Gateway for DeerFlow - A LangGraph-based AI agent backend with sandbox execu
|
||||
|
||||
### Architecture
|
||||
|
||||
LangGraph-compatible requests are routed through nginx to this gateway.
|
||||
This gateway provides runtime endpoints for agent runs plus custom endpoints for models, MCP configuration, skills, and artifacts.
|
||||
LangGraph requests are handled by nginx reverse proxy.
|
||||
This gateway provides custom endpoints for models, MCP configuration, skills, and artifacts.
|
||||
""",
|
||||
version="0.1.0",
|
||||
lifespan=lifespan,
|
||||
docs_url=docs_url,
|
||||
redoc_url=redoc_url,
|
||||
openapi_url=openapi_url,
|
||||
docs_url="/docs",
|
||||
redoc_url="/redoc",
|
||||
openapi_url="/openapi.json",
|
||||
openapi_tags=[
|
||||
{
|
||||
"name": "models",
|
||||
@@ -329,24 +163,7 @@ This gateway provides runtime endpoints for agent runs plus custom endpoints for
|
||||
],
|
||||
)
|
||||
|
||||
# Auth: reject unauthenticated requests to non-public paths (fail-closed safety net)
|
||||
app.add_middleware(AuthMiddleware)
|
||||
|
||||
# CSRF: Double Submit Cookie pattern for state-changing requests
|
||||
app.add_middleware(CSRFMiddleware)
|
||||
|
||||
# CORS: the unified nginx endpoint is same-origin by default. Split-origin
|
||||
# browser clients must opt in with this explicit Gateway allowlist so CORS
|
||||
# and CSRF origin checks share the same source of truth.
|
||||
cors_origins = sorted(get_configured_cors_origins())
|
||||
if cors_origins:
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=cors_origins,
|
||||
allow_credentials=True,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
# CORS is handled by nginx - no need for FastAPI middleware
|
||||
|
||||
# Include routers
|
||||
# Models API is mounted at /api/models
|
||||
@@ -382,12 +199,6 @@ This gateway provides runtime endpoints for agent runs plus custom endpoints for
|
||||
# Assistants compatibility API (LangGraph Platform stub)
|
||||
app.include_router(assistants_compat.router)
|
||||
|
||||
# Auth API is mounted at /api/v1/auth
|
||||
app.include_router(auth.router)
|
||||
|
||||
# Feedback API is mounted at /api/threads/{thread_id}/runs/{run_id}/feedback
|
||||
app.include_router(feedback.router)
|
||||
|
||||
# Thread Runs API (LangGraph Platform-compatible runs lifecycle)
|
||||
app.include_router(thread_runs.router)
|
||||
|
||||
@@ -395,7 +206,7 @@ This gateway provides runtime endpoints for agent runs plus custom endpoints for
|
||||
app.include_router(runs.router)
|
||||
|
||||
@app.get("/health", tags=["health"])
|
||||
async def health_check() -> dict[str, str]:
|
||||
async def health_check() -> dict:
|
||||
"""Health check endpoint.
|
||||
|
||||
Returns:
|
||||
|
||||
@@ -1,42 +0,0 @@
|
||||
"""Authentication module for DeerFlow.
|
||||
|
||||
This module provides:
|
||||
- JWT-based authentication
|
||||
- Provider Factory pattern for extensible auth methods
|
||||
- UserRepository interface for storage backends (SQLite)
|
||||
"""
|
||||
|
||||
from app.gateway.auth.config import AuthConfig, get_auth_config, set_auth_config
|
||||
from app.gateway.auth.errors import AuthErrorCode, AuthErrorResponse, TokenError
|
||||
from app.gateway.auth.jwt import TokenPayload, create_access_token, decode_token
|
||||
from app.gateway.auth.local_provider import LocalAuthProvider
|
||||
from app.gateway.auth.models import User, UserResponse
|
||||
from app.gateway.auth.password import hash_password, verify_password
|
||||
from app.gateway.auth.providers import AuthProvider
|
||||
from app.gateway.auth.repositories.base import UserRepository
|
||||
|
||||
__all__ = [
|
||||
# Config
|
||||
"AuthConfig",
|
||||
"get_auth_config",
|
||||
"set_auth_config",
|
||||
# Errors
|
||||
"AuthErrorCode",
|
||||
"AuthErrorResponse",
|
||||
"TokenError",
|
||||
# JWT
|
||||
"TokenPayload",
|
||||
"create_access_token",
|
||||
"decode_token",
|
||||
# Password
|
||||
"hash_password",
|
||||
"verify_password",
|
||||
# Models
|
||||
"User",
|
||||
"UserResponse",
|
||||
# Providers
|
||||
"AuthProvider",
|
||||
"LocalAuthProvider",
|
||||
# Repository
|
||||
"UserRepository",
|
||||
]
|
||||
@@ -1,85 +0,0 @@
|
||||
"""Authentication configuration for DeerFlow."""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import secrets
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_SECRET_FILE = ".jwt_secret"
|
||||
|
||||
|
||||
class AuthConfig(BaseModel):
|
||||
"""JWT and auth-related configuration. Parsed once at startup.
|
||||
|
||||
Note: the ``users`` table now lives in the shared persistence
|
||||
database managed by ``deerflow.persistence.engine``. The old
|
||||
``users_db_path`` config key has been removed — user storage is
|
||||
configured through ``config.database`` like every other table.
|
||||
"""
|
||||
|
||||
jwt_secret: str = Field(
|
||||
...,
|
||||
description="Secret key for JWT signing. MUST be set via AUTH_JWT_SECRET.",
|
||||
)
|
||||
token_expiry_days: int = Field(default=7, ge=1, le=30)
|
||||
oauth_github_client_id: str | None = Field(default=None)
|
||||
oauth_github_client_secret: str | None = Field(default=None)
|
||||
|
||||
|
||||
_auth_config: AuthConfig | None = None
|
||||
|
||||
|
||||
def _load_or_create_secret() -> str:
|
||||
"""Load persisted JWT secret from ``{base_dir}/.jwt_secret``, or generate and persist a new one."""
|
||||
from deerflow.config.paths import get_paths
|
||||
|
||||
paths = get_paths()
|
||||
secret_file = paths.base_dir / _SECRET_FILE
|
||||
|
||||
try:
|
||||
if secret_file.exists():
|
||||
secret = secret_file.read_text(encoding="utf-8").strip()
|
||||
if secret:
|
||||
return secret
|
||||
except OSError as exc:
|
||||
raise RuntimeError(f"Failed to read JWT secret from {secret_file}. Set AUTH_JWT_SECRET explicitly or fix DEER_FLOW_HOME/base directory permissions so DeerFlow can read its persisted auth secret.") from exc
|
||||
|
||||
secret = secrets.token_urlsafe(32)
|
||||
try:
|
||||
secret_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
fd = os.open(secret_file, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as fh:
|
||||
fh.write(secret)
|
||||
except OSError as exc:
|
||||
raise RuntimeError(f"Failed to persist JWT secret to {secret_file}. Set AUTH_JWT_SECRET explicitly or fix DEER_FLOW_HOME/base directory permissions so DeerFlow can store a stable auth secret.") from exc
|
||||
return secret
|
||||
|
||||
|
||||
def get_auth_config() -> AuthConfig:
|
||||
"""Get the global AuthConfig instance. Parses from env on first call."""
|
||||
global _auth_config
|
||||
if _auth_config is None:
|
||||
from dotenv import load_dotenv
|
||||
|
||||
load_dotenv()
|
||||
jwt_secret = os.environ.get("AUTH_JWT_SECRET")
|
||||
if not jwt_secret:
|
||||
jwt_secret = _load_or_create_secret()
|
||||
os.environ["AUTH_JWT_SECRET"] = jwt_secret
|
||||
logger.warning(
|
||||
"⚠ AUTH_JWT_SECRET is not set — using an auto-generated secret "
|
||||
"persisted to .jwt_secret. Sessions will survive restarts. "
|
||||
"For production, add AUTH_JWT_SECRET to your .env file: "
|
||||
'python -c "import secrets; print(secrets.token_urlsafe(32))"'
|
||||
)
|
||||
_auth_config = AuthConfig(jwt_secret=jwt_secret)
|
||||
return _auth_config
|
||||
|
||||
|
||||
def set_auth_config(config: AuthConfig) -> None:
|
||||
"""Set the global AuthConfig instance (for testing)."""
|
||||
global _auth_config
|
||||
_auth_config = config
|
||||
@@ -1,48 +0,0 @@
|
||||
"""Write initial admin credentials to a restricted file instead of logs.
|
||||
|
||||
Logging secrets to stdout/stderr is a well-known CodeQL finding
|
||||
(py/clear-text-logging-sensitive-data) — in production those logs
|
||||
get collected into ELK/Splunk/etc and become a secret sprawl
|
||||
source. This helper writes the credential to a 0600 file that only
|
||||
the process user can read, and returns the path so the caller can
|
||||
log **the path** (not the password) for the operator to pick up.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from deerflow.config.paths import get_paths
|
||||
|
||||
_CREDENTIAL_FILENAME = "admin_initial_credentials.txt"
|
||||
|
||||
|
||||
def write_initial_credentials(email: str, password: str, *, label: str = "initial") -> Path:
|
||||
"""Write the admin email + password to ``{base_dir}/admin_initial_credentials.txt``.
|
||||
|
||||
The file is created **atomically** with mode 0600 via ``os.open``
|
||||
so the password is never world-readable, even for the single syscall
|
||||
window between ``write_text`` and ``chmod``.
|
||||
|
||||
``label`` distinguishes "initial" (fresh creation) from "reset"
|
||||
(password reset) in the file header so an operator picking up the
|
||||
file after a restart can tell which event produced it.
|
||||
|
||||
Returns the absolute :class:`Path` to the file.
|
||||
"""
|
||||
target = get_paths().base_dir / _CREDENTIAL_FILENAME
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
content = (
|
||||
f"# DeerFlow admin {label} credentials\n# This file is generated on first boot or password reset.\n# Change the password after login via Settings -> Account,\n# then delete this file.\n#\nemail: {email}\npassword: {password}\n"
|
||||
)
|
||||
|
||||
# Atomic 0600 create-or-truncate. O_TRUNC (not O_EXCL) so the
|
||||
# reset-password path can rewrite an existing file without a
|
||||
# separate unlink-then-create dance.
|
||||
fd = os.open(target, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as fh:
|
||||
fh.write(content)
|
||||
|
||||
return target.resolve()
|
||||
@@ -1,45 +0,0 @@
|
||||
"""Typed error definitions for auth module.
|
||||
|
||||
AuthErrorCode: exhaustive enum of all auth failure conditions.
|
||||
TokenError: exhaustive enum of JWT decode failures.
|
||||
AuthErrorResponse: structured error payload for HTTP responses.
|
||||
"""
|
||||
|
||||
from enum import StrEnum
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
|
||||
class AuthErrorCode(StrEnum):
|
||||
"""Exhaustive list of auth error conditions."""
|
||||
|
||||
INVALID_CREDENTIALS = "invalid_credentials"
|
||||
TOKEN_EXPIRED = "token_expired"
|
||||
TOKEN_INVALID = "token_invalid"
|
||||
USER_NOT_FOUND = "user_not_found"
|
||||
EMAIL_ALREADY_EXISTS = "email_already_exists"
|
||||
PROVIDER_NOT_FOUND = "provider_not_found"
|
||||
NOT_AUTHENTICATED = "not_authenticated"
|
||||
SYSTEM_ALREADY_INITIALIZED = "system_already_initialized"
|
||||
|
||||
|
||||
class TokenError(StrEnum):
|
||||
"""Exhaustive list of JWT decode failure reasons."""
|
||||
|
||||
EXPIRED = "expired"
|
||||
INVALID_SIGNATURE = "invalid_signature"
|
||||
MALFORMED = "malformed"
|
||||
|
||||
|
||||
class AuthErrorResponse(BaseModel):
|
||||
"""Structured error response — replaces bare `detail` strings."""
|
||||
|
||||
code: AuthErrorCode
|
||||
message: str
|
||||
|
||||
|
||||
def token_error_to_code(err: TokenError) -> AuthErrorCode:
|
||||
"""Map TokenError to AuthErrorCode — single source of truth."""
|
||||
if err == TokenError.EXPIRED:
|
||||
return AuthErrorCode.TOKEN_EXPIRED
|
||||
return AuthErrorCode.TOKEN_INVALID
|
||||
@@ -1,55 +0,0 @@
|
||||
"""JWT token creation and verification."""
|
||||
|
||||
from datetime import UTC, datetime, timedelta
|
||||
|
||||
import jwt
|
||||
from pydantic import BaseModel
|
||||
|
||||
from app.gateway.auth.config import get_auth_config
|
||||
from app.gateway.auth.errors import TokenError
|
||||
|
||||
|
||||
class TokenPayload(BaseModel):
|
||||
"""JWT token payload."""
|
||||
|
||||
sub: str # user_id
|
||||
exp: datetime
|
||||
iat: datetime | None = None
|
||||
ver: int = 0 # token_version — must match User.token_version
|
||||
|
||||
|
||||
def create_access_token(user_id: str, expires_delta: timedelta | None = None, token_version: int = 0) -> str:
|
||||
"""Create a JWT access token.
|
||||
|
||||
Args:
|
||||
user_id: The user's UUID as string
|
||||
expires_delta: Optional custom expiry, defaults to 7 days
|
||||
token_version: User's current token_version for invalidation
|
||||
|
||||
Returns:
|
||||
Encoded JWT string
|
||||
"""
|
||||
config = get_auth_config()
|
||||
expiry = expires_delta or timedelta(days=config.token_expiry_days)
|
||||
|
||||
now = datetime.now(UTC)
|
||||
payload = {"sub": user_id, "exp": now + expiry, "iat": now, "ver": token_version}
|
||||
return jwt.encode(payload, config.jwt_secret, algorithm="HS256")
|
||||
|
||||
|
||||
def decode_token(token: str) -> TokenPayload | TokenError:
|
||||
"""Decode and validate a JWT token.
|
||||
|
||||
Returns:
|
||||
TokenPayload if valid, or a specific TokenError variant.
|
||||
"""
|
||||
config = get_auth_config()
|
||||
try:
|
||||
payload = jwt.decode(token, config.jwt_secret, algorithms=["HS256"])
|
||||
return TokenPayload(**payload)
|
||||
except jwt.ExpiredSignatureError:
|
||||
return TokenError.EXPIRED
|
||||
except jwt.InvalidSignatureError:
|
||||
return TokenError.INVALID_SIGNATURE
|
||||
except jwt.PyJWTError:
|
||||
return TokenError.MALFORMED
|
||||
@@ -1,104 +0,0 @@
|
||||
"""Local email/password authentication provider."""
|
||||
|
||||
import logging
|
||||
|
||||
from app.gateway.auth.models import User
|
||||
from app.gateway.auth.password import hash_password_async, needs_rehash, verify_password_async
|
||||
from app.gateway.auth.providers import AuthProvider
|
||||
from app.gateway.auth.repositories.base import UserRepository
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class LocalAuthProvider(AuthProvider):
|
||||
"""Email/password authentication provider using local database."""
|
||||
|
||||
def __init__(self, repository: UserRepository):
|
||||
"""Initialize with a UserRepository.
|
||||
|
||||
Args:
|
||||
repository: UserRepository implementation (SQLite)
|
||||
"""
|
||||
self._repo = repository
|
||||
|
||||
async def authenticate(self, credentials: dict) -> User | None:
|
||||
"""Authenticate with email and password.
|
||||
|
||||
Args:
|
||||
credentials: dict with 'email' and 'password' keys
|
||||
|
||||
Returns:
|
||||
User if authentication succeeds, None otherwise
|
||||
"""
|
||||
email = credentials.get("email")
|
||||
password = credentials.get("password")
|
||||
|
||||
if not email or not password:
|
||||
return None
|
||||
|
||||
user = await self._repo.get_user_by_email(email)
|
||||
if user is None:
|
||||
return None
|
||||
|
||||
if user.password_hash is None:
|
||||
# OAuth user without local password
|
||||
return None
|
||||
|
||||
if not await verify_password_async(password, user.password_hash):
|
||||
return None
|
||||
|
||||
if needs_rehash(user.password_hash):
|
||||
try:
|
||||
user.password_hash = await hash_password_async(password)
|
||||
await self._repo.update_user(user)
|
||||
except Exception:
|
||||
# Rehash is an opportunistic upgrade; a transient DB error must not
|
||||
# prevent an otherwise-valid login from succeeding.
|
||||
logger.warning("Failed to rehash password for user %s; login will still succeed", user.email, exc_info=True)
|
||||
|
||||
return user
|
||||
|
||||
async def get_user(self, user_id: str) -> User | None:
|
||||
"""Get user by ID."""
|
||||
return await self._repo.get_user_by_id(user_id)
|
||||
|
||||
async def create_user(self, email: str, password: str | None = None, system_role: str = "user", needs_setup: bool = False) -> User:
|
||||
"""Create a new local user.
|
||||
|
||||
Args:
|
||||
email: User email address
|
||||
password: Plain text password (will be hashed)
|
||||
system_role: Role to assign ("admin" or "user")
|
||||
needs_setup: If True, user must complete setup on first login
|
||||
|
||||
Returns:
|
||||
Created User instance
|
||||
"""
|
||||
password_hash = await hash_password_async(password) if password else None
|
||||
user = User(
|
||||
email=email,
|
||||
password_hash=password_hash,
|
||||
system_role=system_role,
|
||||
needs_setup=needs_setup,
|
||||
)
|
||||
return await self._repo.create_user(user)
|
||||
|
||||
async def get_user_by_oauth(self, provider: str, oauth_id: str) -> User | None:
|
||||
"""Get user by OAuth provider and ID."""
|
||||
return await self._repo.get_user_by_oauth(provider, oauth_id)
|
||||
|
||||
async def count_users(self) -> int:
|
||||
"""Return total number of registered users."""
|
||||
return await self._repo.count_users()
|
||||
|
||||
async def count_admin_users(self) -> int:
|
||||
"""Return number of admin users."""
|
||||
return await self._repo.count_admin_users()
|
||||
|
||||
async def update_user(self, user: User) -> User:
|
||||
"""Update an existing user."""
|
||||
return await self._repo.update_user(user)
|
||||
|
||||
async def get_user_by_email(self, email: str) -> User | None:
|
||||
"""Get user by email."""
|
||||
return await self._repo.get_user_by_email(email)
|
||||
@@ -1,41 +0,0 @@
|
||||
"""User Pydantic models for authentication."""
|
||||
|
||||
from datetime import UTC, datetime
|
||||
from typing import Literal
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, EmailStr, Field
|
||||
|
||||
|
||||
def _utc_now() -> datetime:
|
||||
"""Return current UTC time (timezone-aware)."""
|
||||
return datetime.now(UTC)
|
||||
|
||||
|
||||
class User(BaseModel):
|
||||
"""Internal user representation."""
|
||||
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
id: UUID = Field(default_factory=uuid4, description="Primary key")
|
||||
email: EmailStr = Field(..., description="Unique email address")
|
||||
password_hash: str | None = Field(None, description="bcrypt hash, nullable for OAuth users")
|
||||
system_role: Literal["admin", "user"] = Field(default="user")
|
||||
created_at: datetime = Field(default_factory=_utc_now)
|
||||
|
||||
# OAuth linkage (optional)
|
||||
oauth_provider: str | None = Field(None, description="e.g. 'github', 'google'")
|
||||
oauth_id: str | None = Field(None, description="User ID from OAuth provider")
|
||||
|
||||
# Auth lifecycle
|
||||
needs_setup: bool = Field(default=False, description="True when a reset account must complete setup")
|
||||
token_version: int = Field(default=0, description="Incremented on password change to invalidate old JWTs")
|
||||
|
||||
|
||||
class UserResponse(BaseModel):
|
||||
"""Response model for user info endpoint."""
|
||||
|
||||
id: str
|
||||
email: str
|
||||
system_role: Literal["admin", "user"]
|
||||
needs_setup: bool = False
|
||||
@@ -1,81 +0,0 @@
|
||||
"""Password hashing utilities with versioned hash format.
|
||||
|
||||
Hash format: ``$dfv<N>$<bcrypt_hash>`` where ``<N>`` is the version.
|
||||
|
||||
- **v1** (legacy): ``bcrypt(password)`` — plain bcrypt, susceptible to
|
||||
72-byte silent truncation.
|
||||
- **v2** (current): ``bcrypt(b64(sha256(password)))`` — SHA-256 pre-hash
|
||||
avoids the 72-byte truncation limit so the full password contributes
|
||||
to the hash.
|
||||
|
||||
Verification auto-detects the version and falls back to v1 for hashes
|
||||
without a prefix, so existing deployments upgrade transparently on next
|
||||
login.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import hashlib
|
||||
|
||||
import bcrypt
|
||||
|
||||
_CURRENT_VERSION = 2
|
||||
_PREFIX_V2 = "$dfv2$"
|
||||
_PREFIX_V1 = "$dfv1$"
|
||||
|
||||
|
||||
def _pre_hash_v2(password: str) -> bytes:
|
||||
"""SHA-256 pre-hash to bypass bcrypt's 72-byte limit."""
|
||||
return base64.b64encode(hashlib.sha256(password.encode("utf-8")).digest())
|
||||
|
||||
|
||||
def hash_password(password: str) -> str:
|
||||
"""Hash a password (current version: v2 — SHA-256 + bcrypt)."""
|
||||
raw = bcrypt.hashpw(_pre_hash_v2(password), bcrypt.gensalt()).decode("utf-8")
|
||||
return f"{_PREFIX_V2}{raw}"
|
||||
|
||||
|
||||
def verify_password(plain_password: str, hashed_password: str) -> bool:
|
||||
"""Verify a password, auto-detecting the hash version.
|
||||
|
||||
Accepts v2 (``$dfv2$…``), v1 (``$dfv1$…``), and bare bcrypt hashes
|
||||
(treated as v1 for backward compatibility with pre-versioning data).
|
||||
"""
|
||||
try:
|
||||
if hashed_password.startswith(_PREFIX_V2):
|
||||
bcrypt_hash = hashed_password[len(_PREFIX_V2) :]
|
||||
return bcrypt.checkpw(_pre_hash_v2(plain_password), bcrypt_hash.encode("utf-8"))
|
||||
|
||||
if hashed_password.startswith(_PREFIX_V1):
|
||||
bcrypt_hash = hashed_password[len(_PREFIX_V1) :]
|
||||
else:
|
||||
bcrypt_hash = hashed_password
|
||||
|
||||
return bcrypt.checkpw(plain_password.encode("utf-8"), bcrypt_hash.encode("utf-8"))
|
||||
except ValueError:
|
||||
# bcrypt raises ValueError for malformed or corrupt hashes (e.g., invalid salt).
|
||||
# Fail closed rather than crashing the request.
|
||||
return False
|
||||
|
||||
|
||||
def needs_rehash(hashed_password: str) -> bool:
|
||||
"""Return True if the hash uses an older version and should be rehashed."""
|
||||
return not hashed_password.startswith(_PREFIX_V2)
|
||||
|
||||
|
||||
async def hash_password_async(password: str) -> str:
|
||||
"""Hash a password using bcrypt (non-blocking).
|
||||
|
||||
Wraps the blocking bcrypt operation in a thread pool to avoid
|
||||
blocking the event loop during password hashing.
|
||||
"""
|
||||
return await asyncio.to_thread(hash_password, password)
|
||||
|
||||
|
||||
async def verify_password_async(plain_password: str, hashed_password: str) -> bool:
|
||||
"""Verify a password against its hash (non-blocking).
|
||||
|
||||
Wraps the blocking bcrypt operation in a thread pool to avoid
|
||||
blocking the event loop during password verification.
|
||||
"""
|
||||
return await asyncio.to_thread(verify_password, plain_password, hashed_password)
|
||||
@@ -1,24 +0,0 @@
|
||||
"""Auth provider abstraction."""
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
|
||||
|
||||
class AuthProvider(ABC):
|
||||
"""Abstract base class for authentication providers."""
|
||||
|
||||
@abstractmethod
|
||||
async def authenticate(self, credentials: dict) -> "User | None":
|
||||
"""Authenticate user with given credentials.
|
||||
|
||||
Returns User if authentication succeeds, None otherwise.
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def get_user(self, user_id: str) -> "User | None":
|
||||
"""Retrieve user by ID."""
|
||||
raise NotImplementedError
|
||||
|
||||
|
||||
# Import User at runtime to avoid circular imports
|
||||
from app.gateway.auth.models import User # noqa: E402
|
||||
@@ -1,102 +0,0 @@
|
||||
"""User repository interface for abstracting database operations."""
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
|
||||
from app.gateway.auth.models import User
|
||||
|
||||
|
||||
class UserNotFoundError(LookupError):
|
||||
"""Raised when a user repository operation targets a non-existent row.
|
||||
|
||||
Subclass of :class:`LookupError` so callers that already catch
|
||||
``LookupError`` for "missing entity" can keep working unchanged,
|
||||
while specific call sites can pin to this class to distinguish
|
||||
"concurrent delete during update" from other lookups.
|
||||
"""
|
||||
|
||||
|
||||
class UserRepository(ABC):
|
||||
"""Abstract interface for user data storage.
|
||||
|
||||
Implement this interface to support different storage backends
|
||||
(SQLite)
|
||||
"""
|
||||
|
||||
@abstractmethod
|
||||
async def create_user(self, user: User) -> User:
|
||||
"""Create a new user.
|
||||
|
||||
Args:
|
||||
user: User object to create
|
||||
|
||||
Returns:
|
||||
Created User with ID assigned
|
||||
|
||||
Raises:
|
||||
ValueError: If email already exists
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def get_user_by_id(self, user_id: str) -> User | None:
|
||||
"""Get user by ID.
|
||||
|
||||
Args:
|
||||
user_id: User UUID as string
|
||||
|
||||
Returns:
|
||||
User if found, None otherwise
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def get_user_by_email(self, email: str) -> User | None:
|
||||
"""Get user by email.
|
||||
|
||||
Args:
|
||||
email: User email address
|
||||
|
||||
Returns:
|
||||
User if found, None otherwise
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def update_user(self, user: User) -> User:
|
||||
"""Update an existing user.
|
||||
|
||||
Args:
|
||||
user: User object with updated fields
|
||||
|
||||
Returns:
|
||||
Updated User
|
||||
|
||||
Raises:
|
||||
UserNotFoundError: If no row exists for ``user.id``. This is
|
||||
a hard failure (not a no-op) so callers cannot mistake a
|
||||
concurrent-delete race for a successful update.
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def count_users(self) -> int:
|
||||
"""Return total number of registered users."""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def count_admin_users(self) -> int:
|
||||
"""Return number of users with system_role == 'admin'."""
|
||||
raise NotImplementedError
|
||||
|
||||
@abstractmethod
|
||||
async def get_user_by_oauth(self, provider: str, oauth_id: str) -> User | None:
|
||||
"""Get user by OAuth provider and ID.
|
||||
|
||||
Args:
|
||||
provider: OAuth provider name (e.g. 'github', 'google')
|
||||
oauth_id: User ID from the OAuth provider
|
||||
|
||||
Returns:
|
||||
User if found, None otherwise
|
||||
"""
|
||||
raise NotImplementedError
|
||||
@@ -1,127 +0,0 @@
|
||||
"""SQLAlchemy-backed UserRepository implementation.
|
||||
|
||||
Uses the shared async session factory from
|
||||
``deerflow.persistence.engine`` — the ``users`` table lives in the
|
||||
same database as ``threads_meta``, ``runs``, ``run_events``, and
|
||||
``feedback``.
|
||||
|
||||
Constructor takes the session factory directly (same pattern as the
|
||||
other four repositories in ``deerflow.persistence.*``). Callers
|
||||
construct this after ``init_engine_from_config()`` has run.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC
|
||||
from uuid import UUID
|
||||
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
|
||||
from app.gateway.auth.models import User
|
||||
from app.gateway.auth.repositories.base import UserNotFoundError, UserRepository
|
||||
from deerflow.persistence.user.model import UserRow
|
||||
|
||||
|
||||
class SQLiteUserRepository(UserRepository):
|
||||
"""Async user repository backed by the shared SQLAlchemy engine."""
|
||||
|
||||
def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:
|
||||
self._sf = session_factory
|
||||
|
||||
# ── Converters ────────────────────────────────────────────────────
|
||||
|
||||
@staticmethod
|
||||
def _row_to_user(row: UserRow) -> User:
|
||||
return User(
|
||||
id=UUID(row.id),
|
||||
email=row.email,
|
||||
password_hash=row.password_hash,
|
||||
system_role=row.system_role, # type: ignore[arg-type]
|
||||
# SQLite loses tzinfo on read; reattach UTC so downstream
|
||||
# code can compare timestamps reliably.
|
||||
created_at=row.created_at if row.created_at.tzinfo else row.created_at.replace(tzinfo=UTC),
|
||||
oauth_provider=row.oauth_provider,
|
||||
oauth_id=row.oauth_id,
|
||||
needs_setup=row.needs_setup,
|
||||
token_version=row.token_version,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _user_to_row(user: User) -> UserRow:
|
||||
return UserRow(
|
||||
id=str(user.id),
|
||||
email=user.email,
|
||||
password_hash=user.password_hash,
|
||||
system_role=user.system_role,
|
||||
created_at=user.created_at,
|
||||
oauth_provider=user.oauth_provider,
|
||||
oauth_id=user.oauth_id,
|
||||
needs_setup=user.needs_setup,
|
||||
token_version=user.token_version,
|
||||
)
|
||||
|
||||
# ── CRUD ──────────────────────────────────────────────────────────
|
||||
|
||||
async def create_user(self, user: User) -> User:
|
||||
"""Insert a new user. Raises ``ValueError`` on duplicate email."""
|
||||
row = self._user_to_row(user)
|
||||
async with self._sf() as session:
|
||||
session.add(row)
|
||||
try:
|
||||
await session.commit()
|
||||
except IntegrityError as exc:
|
||||
await session.rollback()
|
||||
raise ValueError(f"Email already registered: {user.email}") from exc
|
||||
return user
|
||||
|
||||
async def get_user_by_id(self, user_id: str) -> User | None:
|
||||
async with self._sf() as session:
|
||||
row = await session.get(UserRow, user_id)
|
||||
return self._row_to_user(row) if row is not None else None
|
||||
|
||||
async def get_user_by_email(self, email: str) -> User | None:
|
||||
stmt = select(UserRow).where(UserRow.email == email)
|
||||
async with self._sf() as session:
|
||||
result = await session.execute(stmt)
|
||||
row = result.scalar_one_or_none()
|
||||
return self._row_to_user(row) if row is not None else None
|
||||
|
||||
async def update_user(self, user: User) -> User:
|
||||
async with self._sf() as session:
|
||||
row = await session.get(UserRow, str(user.id))
|
||||
if row is None:
|
||||
# Hard fail on concurrent delete: callers (reset_admin,
|
||||
# password change handlers, _ensure_admin_user) all
|
||||
# fetched the user just before this call, so a missing
|
||||
# row here means the row vanished underneath us. Silent
|
||||
# success would let the caller log "password reset" for
|
||||
# a row that no longer exists.
|
||||
raise UserNotFoundError(f"User {user.id} no longer exists")
|
||||
row.email = user.email
|
||||
row.password_hash = user.password_hash
|
||||
row.system_role = user.system_role
|
||||
row.oauth_provider = user.oauth_provider
|
||||
row.oauth_id = user.oauth_id
|
||||
row.needs_setup = user.needs_setup
|
||||
row.token_version = user.token_version
|
||||
await session.commit()
|
||||
return user
|
||||
|
||||
async def count_users(self) -> int:
|
||||
stmt = select(func.count()).select_from(UserRow)
|
||||
async with self._sf() as session:
|
||||
return await session.scalar(stmt) or 0
|
||||
|
||||
async def count_admin_users(self) -> int:
|
||||
stmt = select(func.count()).select_from(UserRow).where(UserRow.system_role == "admin")
|
||||
async with self._sf() as session:
|
||||
return await session.scalar(stmt) or 0
|
||||
|
||||
async def get_user_by_oauth(self, provider: str, oauth_id: str) -> User | None:
|
||||
stmt = select(UserRow).where(UserRow.oauth_provider == provider, UserRow.oauth_id == oauth_id)
|
||||
async with self._sf() as session:
|
||||
result = await session.execute(stmt)
|
||||
row = result.scalar_one_or_none()
|
||||
return self._row_to_user(row) if row is not None else None
|
||||
@@ -1,91 +0,0 @@
|
||||
"""CLI tool to reset an admin password.
|
||||
|
||||
Usage:
|
||||
python -m app.gateway.auth.reset_admin
|
||||
python -m app.gateway.auth.reset_admin --email admin@example.com
|
||||
|
||||
Writes the new password to ``.deer-flow/admin_initial_credentials.txt``
|
||||
(mode 0600) instead of printing it, so CI / log aggregators never see
|
||||
the cleartext secret.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import secrets
|
||||
import sys
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.gateway.auth.credential_file import write_initial_credentials
|
||||
from app.gateway.auth.password import hash_password
|
||||
from app.gateway.auth.repositories.sqlite import SQLiteUserRepository
|
||||
from deerflow.persistence.user.model import UserRow
|
||||
|
||||
|
||||
async def _run(email: str | None) -> int:
|
||||
from deerflow.config import get_app_config
|
||||
from deerflow.persistence.engine import (
|
||||
close_engine,
|
||||
get_session_factory,
|
||||
init_engine_from_config,
|
||||
)
|
||||
|
||||
config = get_app_config()
|
||||
await init_engine_from_config(config.database)
|
||||
try:
|
||||
sf = get_session_factory()
|
||||
if sf is None:
|
||||
print("Error: persistence engine not available (check config.database).", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
repo = SQLiteUserRepository(sf)
|
||||
|
||||
if email:
|
||||
user = await repo.get_user_by_email(email)
|
||||
else:
|
||||
# Find first admin via direct SELECT — repository does not
|
||||
# expose a "first admin" helper and we do not want to add
|
||||
# one just for this CLI.
|
||||
async with sf() as session:
|
||||
stmt = select(UserRow).where(UserRow.system_role == "admin").limit(1)
|
||||
row = (await session.execute(stmt)).scalar_one_or_none()
|
||||
if row is None:
|
||||
user = None
|
||||
else:
|
||||
user = await repo.get_user_by_id(row.id)
|
||||
|
||||
if user is None:
|
||||
if email:
|
||||
print(f"Error: user '{email}' not found.", file=sys.stderr)
|
||||
else:
|
||||
print("Error: no admin user found.", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
new_password = secrets.token_urlsafe(16)
|
||||
user.password_hash = hash_password(new_password)
|
||||
user.token_version += 1
|
||||
user.needs_setup = True
|
||||
await repo.update_user(user)
|
||||
|
||||
cred_path = write_initial_credentials(user.email, new_password, label="reset")
|
||||
print(f"Password reset for: {user.email}")
|
||||
print(f"Credentials written to: {cred_path} (mode 0600)")
|
||||
print("Next login will require setup (new email + password).")
|
||||
return 0
|
||||
finally:
|
||||
await close_engine()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description="Reset admin password")
|
||||
parser.add_argument("--email", help="Admin email (default: first admin found)")
|
||||
args = parser.parse_args()
|
||||
|
||||
exit_code = asyncio.run(_run(args.email))
|
||||
sys.exit(exit_code)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,126 +0,0 @@
|
||||
"""Global authentication middleware — fail-closed safety net.
|
||||
|
||||
Rejects unauthenticated requests to non-public paths with 401. When a
|
||||
request passes the cookie check, resolves the JWT payload to a real
|
||||
``User`` object and stamps it into both ``request.state.user`` and the
|
||||
``deerflow.runtime.user_context`` contextvar so that repository-layer
|
||||
owner filtering works automatically via the sentinel pattern.
|
||||
|
||||
Fine-grained permission checks remain in authz.py decorators.
|
||||
"""
|
||||
|
||||
from collections.abc import Callable
|
||||
|
||||
from fastapi import HTTPException, Request, Response
|
||||
from starlette.middleware.base import BaseHTTPMiddleware
|
||||
from starlette.responses import JSONResponse
|
||||
from starlette.types import ASGIApp
|
||||
|
||||
from app.gateway.auth.errors import AuthErrorCode, AuthErrorResponse
|
||||
from app.gateway.authz import _ALL_PERMISSIONS, AuthContext
|
||||
from app.gateway.internal_auth import INTERNAL_AUTH_HEADER_NAME, get_internal_user, is_valid_internal_auth_token
|
||||
from deerflow.runtime.user_context import reset_current_user, set_current_user
|
||||
|
||||
# Paths that never require authentication.
|
||||
_PUBLIC_PATH_PREFIXES: tuple[str, ...] = (
|
||||
"/health",
|
||||
"/docs",
|
||||
"/redoc",
|
||||
"/openapi.json",
|
||||
)
|
||||
|
||||
# Exact auth paths that are public (login/register/status check).
|
||||
# /api/v1/auth/me, /api/v1/auth/change-password etc. are NOT public.
|
||||
_PUBLIC_EXACT_PATHS: frozenset[str] = frozenset(
|
||||
{
|
||||
"/api/v1/auth/login/local",
|
||||
"/api/v1/auth/register",
|
||||
"/api/v1/auth/logout",
|
||||
"/api/v1/auth/setup-status",
|
||||
"/api/v1/auth/initialize",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _is_public(path: str) -> bool:
|
||||
stripped = path.rstrip("/")
|
||||
if stripped in _PUBLIC_EXACT_PATHS:
|
||||
return True
|
||||
return any(path.startswith(prefix) for prefix in _PUBLIC_PATH_PREFIXES)
|
||||
|
||||
|
||||
class AuthMiddleware(BaseHTTPMiddleware):
|
||||
"""Strict auth gate: reject requests without a valid session.
|
||||
|
||||
Two-stage check for non-public paths:
|
||||
|
||||
1. Cookie presence — return 401 NOT_AUTHENTICATED if missing
|
||||
2. JWT validation via ``get_optional_user_from_request`` — return 401
|
||||
TOKEN_INVALID if the token is absent, malformed, expired, or the
|
||||
signed user does not exist / is stale
|
||||
|
||||
On success, stamps ``request.state.user`` and the
|
||||
``deerflow.runtime.user_context`` contextvar so that repository-layer
|
||||
owner filters work downstream without every route needing a
|
||||
``@require_auth`` decorator. Routes that need per-resource
|
||||
authorization (e.g. "user A cannot read user B's thread by guessing
|
||||
the URL") should additionally use ``@require_permission(...,
|
||||
owner_check=True)`` for explicit enforcement — but authentication
|
||||
itself is fully handled here.
|
||||
"""
|
||||
|
||||
def __init__(self, app: ASGIApp) -> None:
|
||||
super().__init__(app)
|
||||
|
||||
async def dispatch(self, request: Request, call_next: Callable) -> Response:
|
||||
if _is_public(request.url.path):
|
||||
return await call_next(request)
|
||||
|
||||
internal_user = None
|
||||
if is_valid_internal_auth_token(request.headers.get(INTERNAL_AUTH_HEADER_NAME)):
|
||||
internal_user = get_internal_user()
|
||||
|
||||
# Non-public path: require session cookie
|
||||
if internal_user is None and not request.cookies.get("access_token"):
|
||||
return JSONResponse(
|
||||
status_code=401,
|
||||
content={
|
||||
"detail": AuthErrorResponse(
|
||||
code=AuthErrorCode.NOT_AUTHENTICATED,
|
||||
message="Authentication required",
|
||||
).model_dump()
|
||||
},
|
||||
)
|
||||
|
||||
# Strict JWT validation: reject junk/expired tokens with 401
|
||||
# right here instead of silently passing through. This closes
|
||||
# the "junk cookie bypass" gap (AUTH_TEST_PLAN test 7.5.8):
|
||||
# without this, non-isolation routes like /api/models would
|
||||
# accept any cookie-shaped string as authentication.
|
||||
#
|
||||
# We call the *strict* resolver so that fine-grained error
|
||||
# codes (token_expired, token_invalid, user_not_found, …)
|
||||
# propagate from AuthErrorCode, not get flattened into one
|
||||
# generic code. BaseHTTPMiddleware doesn't let HTTPException
|
||||
# bubble up, so we catch and render it as JSONResponse here.
|
||||
from app.gateway.deps import get_current_user_from_request
|
||||
|
||||
if internal_user is not None:
|
||||
user = internal_user
|
||||
else:
|
||||
try:
|
||||
user = await get_current_user_from_request(request)
|
||||
except HTTPException as exc:
|
||||
return JSONResponse(status_code=exc.status_code, content={"detail": exc.detail})
|
||||
|
||||
# Stamp both request.state.user (for the contextvar pattern)
|
||||
# and request.state.auth (so @require_permission's "auth is
|
||||
# None" branch short-circuits instead of running the entire
|
||||
# JWT-decode + DB-lookup pipeline a second time per request).
|
||||
request.state.user = user
|
||||
request.state.auth = AuthContext(user=user, permissions=_ALL_PERMISSIONS)
|
||||
token = set_current_user(user)
|
||||
try:
|
||||
return await call_next(request)
|
||||
finally:
|
||||
reset_current_user(token)
|
||||
@@ -1,301 +0,0 @@
|
||||
"""Authorization decorators and context for DeerFlow.
|
||||
|
||||
Inspired by LangGraph Auth system: https://github.com/langchain-ai/langgraph/blob/main/libs/sdk-py/langgraph_sdk/auth/__init__.py
|
||||
|
||||
**Usage:**
|
||||
|
||||
1. Use ``@require_auth`` on routes that need authentication
|
||||
2. Use ``@require_permission("resource", "action", filter_key=...)`` for permission checks
|
||||
3. The decorator chain processes from bottom to top
|
||||
|
||||
**Example:**
|
||||
|
||||
@router.get("/{thread_id}")
|
||||
@require_auth
|
||||
@require_permission("threads", "read", owner_check=True)
|
||||
async def get_thread(thread_id: str, request: Request):
|
||||
# User is authenticated and has threads:read permission
|
||||
...
|
||||
|
||||
**Permission Model:**
|
||||
|
||||
- threads:read - View thread
|
||||
- threads:write - Create/update thread
|
||||
- threads:delete - Delete thread
|
||||
- runs:create - Run agent
|
||||
- runs:read - View run
|
||||
- runs:cancel - Cancel run
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import functools
|
||||
import inspect
|
||||
from collections.abc import Callable
|
||||
from types import SimpleNamespace
|
||||
from typing import TYPE_CHECKING, Any, ParamSpec, TypeVar
|
||||
|
||||
from fastapi import HTTPException, Request
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from app.gateway.auth.models import User
|
||||
|
||||
P = ParamSpec("P")
|
||||
T = TypeVar("T")
|
||||
|
||||
|
||||
# Permission constants
|
||||
class Permissions:
|
||||
"""Permission constants for resource:action format."""
|
||||
|
||||
# Threads
|
||||
THREADS_READ = "threads:read"
|
||||
THREADS_WRITE = "threads:write"
|
||||
THREADS_DELETE = "threads:delete"
|
||||
|
||||
# Runs
|
||||
RUNS_CREATE = "runs:create"
|
||||
RUNS_READ = "runs:read"
|
||||
RUNS_CANCEL = "runs:cancel"
|
||||
|
||||
|
||||
class AuthContext:
|
||||
"""Authentication context for the current request.
|
||||
|
||||
Stored in request.state.auth after require_auth decoration.
|
||||
|
||||
Attributes:
|
||||
user: The authenticated user, or None if anonymous
|
||||
permissions: List of permission strings (e.g., "threads:read")
|
||||
"""
|
||||
|
||||
__slots__ = ("user", "permissions")
|
||||
|
||||
def __init__(self, user: User | None = None, permissions: list[str] | None = None):
|
||||
self.user = user
|
||||
self.permissions = permissions or []
|
||||
|
||||
@property
|
||||
def is_authenticated(self) -> bool:
|
||||
"""Check if user is authenticated."""
|
||||
return self.user is not None
|
||||
|
||||
def has_permission(self, resource: str, action: str) -> bool:
|
||||
"""Check if context has permission for resource:action.
|
||||
|
||||
Args:
|
||||
resource: Resource name (e.g., "threads")
|
||||
action: Action name (e.g., "read")
|
||||
|
||||
Returns:
|
||||
True if user has permission
|
||||
"""
|
||||
permission = f"{resource}:{action}"
|
||||
return permission in self.permissions
|
||||
|
||||
def require_user(self) -> User:
|
||||
"""Get user or raise 401.
|
||||
|
||||
Raises:
|
||||
HTTPException 401 if not authenticated
|
||||
"""
|
||||
if not self.user:
|
||||
raise HTTPException(status_code=401, detail="Authentication required")
|
||||
return self.user
|
||||
|
||||
|
||||
def get_auth_context(request: Request) -> AuthContext | None:
|
||||
"""Get AuthContext from request state."""
|
||||
return getattr(request.state, "auth", None)
|
||||
|
||||
|
||||
_ALL_PERMISSIONS: list[str] = [
|
||||
Permissions.THREADS_READ,
|
||||
Permissions.THREADS_WRITE,
|
||||
Permissions.THREADS_DELETE,
|
||||
Permissions.RUNS_CREATE,
|
||||
Permissions.RUNS_READ,
|
||||
Permissions.RUNS_CANCEL,
|
||||
]
|
||||
|
||||
|
||||
def _make_test_request_stub() -> Any:
|
||||
"""Create a minimal request-like object for direct unit calls.
|
||||
|
||||
Used when decorated route handlers are invoked without FastAPI's
|
||||
request injection. Includes fields accessed by auth helpers.
|
||||
"""
|
||||
return SimpleNamespace(state=SimpleNamespace(), cookies={}, _deerflow_test_bypass_auth=True)
|
||||
|
||||
|
||||
async def _authenticate(request: Request) -> AuthContext:
|
||||
"""Authenticate request and return AuthContext.
|
||||
|
||||
Delegates to deps.get_optional_user_from_request() for the JWT→User pipeline.
|
||||
Returns AuthContext with user=None for anonymous requests.
|
||||
"""
|
||||
from app.gateway.deps import get_optional_user_from_request
|
||||
|
||||
user = await get_optional_user_from_request(request)
|
||||
if user is None:
|
||||
return AuthContext(user=None, permissions=[])
|
||||
|
||||
# In future, permissions could be stored in user record
|
||||
return AuthContext(user=user, permissions=_ALL_PERMISSIONS)
|
||||
|
||||
|
||||
def require_auth[**P, T](func: Callable[P, T]) -> Callable[P, T]:
|
||||
"""Decorator that authenticates the request and enforces authentication.
|
||||
|
||||
Independently raises HTTP 401 for unauthenticated requests, regardless of
|
||||
whether ``AuthMiddleware`` is present in the ASGI stack. Sets the resolved
|
||||
``AuthContext`` on ``request.state.auth`` for downstream handlers.
|
||||
|
||||
Must be placed ABOVE other decorators (executes after them).
|
||||
|
||||
Usage:
|
||||
@router.get("/{thread_id}")
|
||||
@require_auth # Bottom decorator (executes first after permission check)
|
||||
@require_permission("threads", "read")
|
||||
async def get_thread(thread_id: str, request: Request):
|
||||
auth: AuthContext = request.state.auth
|
||||
...
|
||||
|
||||
Raises:
|
||||
HTTPException: 401 if the request is unauthenticated.
|
||||
ValueError: If 'request' parameter is missing.
|
||||
"""
|
||||
|
||||
@functools.wraps(func)
|
||||
async def wrapper(*args: Any, **kwargs: Any) -> Any:
|
||||
request = kwargs.get("request")
|
||||
if request is None:
|
||||
# Unit tests may call decorated handlers directly without a
|
||||
# FastAPI Request object. Inject a minimal request stub when
|
||||
# the wrapped function declares `request`.
|
||||
if "request" in inspect.signature(func).parameters:
|
||||
kwargs["request"] = _make_test_request_stub()
|
||||
else:
|
||||
raise ValueError("require_auth decorator requires 'request' parameter")
|
||||
request = kwargs["request"]
|
||||
|
||||
if getattr(request, "_deerflow_test_bypass_auth", False):
|
||||
return await func(*args, **kwargs)
|
||||
|
||||
# Authenticate and set context
|
||||
auth_context = await _authenticate(request)
|
||||
request.state.auth = auth_context
|
||||
|
||||
if not auth_context.is_authenticated:
|
||||
raise HTTPException(status_code=401, detail="Authentication required")
|
||||
|
||||
return await func(*args, **kwargs)
|
||||
|
||||
return wrapper
|
||||
|
||||
|
||||
def require_permission(
|
||||
resource: str,
|
||||
action: str,
|
||||
owner_check: bool = False,
|
||||
require_existing: bool = False,
|
||||
) -> Callable[[Callable[P, T]], Callable[P, T]]:
|
||||
"""Decorator that checks permission for resource:action.
|
||||
|
||||
Must be used AFTER @require_auth.
|
||||
|
||||
Args:
|
||||
resource: Resource name (e.g., "threads", "runs")
|
||||
action: Action name (e.g., "read", "write", "delete")
|
||||
owner_check: If True, validates that the current user owns the resource.
|
||||
Requires 'thread_id' path parameter and performs ownership check.
|
||||
require_existing: Only meaningful with ``owner_check=True``. If True, a
|
||||
missing ``threads_meta`` row counts as a denial (404)
|
||||
instead of "untracked legacy thread, allow". Use on
|
||||
**destructive / mutating** routes (DELETE, PATCH,
|
||||
state-update) so a deleted thread can't be re-targeted
|
||||
by another user via the missing-row code path.
|
||||
|
||||
Usage:
|
||||
# Read-style: legacy untracked threads are allowed
|
||||
@require_permission("threads", "read", owner_check=True)
|
||||
async def get_thread(thread_id: str, request: Request):
|
||||
...
|
||||
|
||||
# Destructive: thread row MUST exist and be owned by caller
|
||||
@require_permission("threads", "delete", owner_check=True, require_existing=True)
|
||||
async def delete_thread(thread_id: str, request: Request):
|
||||
...
|
||||
|
||||
Raises:
|
||||
HTTPException 401: If authentication required but user is anonymous
|
||||
HTTPException 403: If user lacks permission
|
||||
HTTPException 404: If owner_check=True but user doesn't own the thread
|
||||
ValueError: If owner_check=True but 'thread_id' parameter is missing
|
||||
"""
|
||||
|
||||
def decorator(func: Callable[P, T]) -> Callable[P, T]:
|
||||
@functools.wraps(func)
|
||||
async def wrapper(*args: Any, **kwargs: Any) -> Any:
|
||||
request = kwargs.get("request")
|
||||
if request is None:
|
||||
# Unit tests may call decorated route handlers directly without
|
||||
# constructing a FastAPI Request object. Inject a minimal stub
|
||||
# when the wrapped function declares `request`.
|
||||
if "request" in inspect.signature(func).parameters:
|
||||
kwargs["request"] = _make_test_request_stub()
|
||||
else:
|
||||
return await func(*args, **kwargs)
|
||||
request = kwargs["request"]
|
||||
|
||||
if getattr(request, "_deerflow_test_bypass_auth", False):
|
||||
return await func(*args, **kwargs)
|
||||
|
||||
auth: AuthContext = getattr(request.state, "auth", None)
|
||||
if auth is None:
|
||||
auth = await _authenticate(request)
|
||||
request.state.auth = auth
|
||||
|
||||
if not auth.is_authenticated:
|
||||
raise HTTPException(status_code=401, detail="Authentication required")
|
||||
|
||||
# Check permission
|
||||
if not auth.has_permission(resource, action):
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail=f"Permission denied: {resource}:{action}",
|
||||
)
|
||||
|
||||
# Owner check for thread-specific resources.
|
||||
#
|
||||
# 2.0-rc moved thread metadata into the SQL persistence layer
|
||||
# (``threads_meta`` table). We verify ownership via
|
||||
# ``ThreadMetaStore.check_access``: it returns True for
|
||||
# missing rows (untracked legacy thread) and for rows whose
|
||||
# ``user_id`` is NULL (shared / pre-auth data), so this is
|
||||
# strict-deny rather than strict-allow — only an *existing*
|
||||
# row with a *different* user_id triggers 404.
|
||||
if owner_check:
|
||||
thread_id = kwargs.get("thread_id")
|
||||
if thread_id is None:
|
||||
raise ValueError("require_permission with owner_check=True requires 'thread_id' parameter")
|
||||
|
||||
from app.gateway.deps import get_thread_store
|
||||
|
||||
thread_store = get_thread_store(request)
|
||||
allowed = await thread_store.check_access(
|
||||
thread_id,
|
||||
str(auth.user.id),
|
||||
require_existing=require_existing,
|
||||
)
|
||||
if not allowed:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"Thread {thread_id} not found",
|
||||
)
|
||||
|
||||
return await func(*args, **kwargs)
|
||||
|
||||
return wrapper
|
||||
|
||||
return decorator
|
||||
@@ -8,7 +8,7 @@ class GatewayConfig(BaseModel):
|
||||
|
||||
host: str = Field(default="0.0.0.0", description="Host to bind the gateway server")
|
||||
port: int = Field(default=8001, description="Port to bind the gateway server")
|
||||
enable_docs: bool = Field(default=True, description="Enable Swagger/ReDoc/OpenAPI endpoints")
|
||||
cors_origins: list[str] = Field(default_factory=lambda: ["http://localhost:3000"], description="Allowed CORS origins")
|
||||
|
||||
|
||||
_gateway_config: GatewayConfig | None = None
|
||||
@@ -18,9 +18,10 @@ def get_gateway_config() -> GatewayConfig:
|
||||
"""Get gateway config, loading from environment if available."""
|
||||
global _gateway_config
|
||||
if _gateway_config is None:
|
||||
cors_origins_str = os.getenv("CORS_ORIGINS", "http://localhost:3000")
|
||||
_gateway_config = GatewayConfig(
|
||||
host=os.getenv("GATEWAY_HOST", "0.0.0.0"),
|
||||
port=int(os.getenv("GATEWAY_PORT", "8001")),
|
||||
enable_docs=os.getenv("GATEWAY_ENABLE_DOCS", "true").lower() == "true",
|
||||
cors_origins=cors_origins_str.split(","),
|
||||
)
|
||||
return _gateway_config
|
||||
|
||||
@@ -1,229 +0,0 @@
|
||||
"""CSRF protection middleware for FastAPI.
|
||||
|
||||
Per RFC-001:
|
||||
State-changing operations require CSRF protection.
|
||||
"""
|
||||
|
||||
import os
|
||||
import secrets
|
||||
from collections.abc import Awaitable, Callable
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from fastapi import Request, Response
|
||||
from starlette.middleware.base import BaseHTTPMiddleware
|
||||
from starlette.responses import JSONResponse
|
||||
from starlette.types import ASGIApp
|
||||
|
||||
CSRF_COOKIE_NAME = "csrf_token"
|
||||
CSRF_HEADER_NAME = "X-CSRF-Token"
|
||||
CSRF_TOKEN_LENGTH = 64 # bytes
|
||||
|
||||
|
||||
def is_secure_request(request: Request) -> bool:
|
||||
"""Detect whether the original client request was made over HTTPS."""
|
||||
return _request_scheme(request) == "https"
|
||||
|
||||
|
||||
def generate_csrf_token() -> str:
|
||||
"""Generate a secure random CSRF token."""
|
||||
return secrets.token_urlsafe(CSRF_TOKEN_LENGTH)
|
||||
|
||||
|
||||
def should_check_csrf(request: Request) -> bool:
|
||||
"""Determine if a request needs CSRF validation.
|
||||
|
||||
CSRF is checked for state-changing methods (POST, PUT, DELETE, PATCH).
|
||||
GET, HEAD, OPTIONS, and TRACE are exempt per RFC 7231.
|
||||
"""
|
||||
if request.method not in ("POST", "PUT", "DELETE", "PATCH"):
|
||||
return False
|
||||
|
||||
path = request.url.path.rstrip("/")
|
||||
# Exempt /api/v1/auth/me endpoint
|
||||
if path == "/api/v1/auth/me":
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
_AUTH_EXEMPT_PATHS: frozenset[str] = frozenset(
|
||||
{
|
||||
"/api/v1/auth/login/local",
|
||||
"/api/v1/auth/logout",
|
||||
"/api/v1/auth/register",
|
||||
"/api/v1/auth/initialize",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def is_auth_endpoint(request: Request) -> bool:
|
||||
"""Check if the request is to an auth endpoint.
|
||||
|
||||
Auth endpoints don't need CSRF validation on first call (no token).
|
||||
"""
|
||||
return request.url.path.rstrip("/") in _AUTH_EXEMPT_PATHS
|
||||
|
||||
|
||||
def _host_with_optional_port(hostname: str, port: int | None, scheme: str) -> str:
|
||||
"""Return normalized host[:port], omitting default ports."""
|
||||
host = hostname.lower()
|
||||
if ":" in host and not host.startswith("["):
|
||||
host = f"[{host}]"
|
||||
|
||||
if port is None or (scheme == "http" and port == 80) or (scheme == "https" and port == 443):
|
||||
return host
|
||||
return f"{host}:{port}"
|
||||
|
||||
|
||||
def _normalize_origin(origin: str) -> str | None:
|
||||
"""Return a normalized scheme://host[:port] origin, or None for invalid input."""
|
||||
try:
|
||||
parsed = urlsplit(origin.strip())
|
||||
port = parsed.port
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
scheme = parsed.scheme.lower()
|
||||
if scheme not in {"http", "https"} or not parsed.hostname:
|
||||
return None
|
||||
|
||||
# Browser Origin is only scheme/host/port. Reject URL-shaped or credentialed values.
|
||||
if parsed.username or parsed.password or parsed.path or parsed.query or parsed.fragment:
|
||||
return None
|
||||
|
||||
return f"{scheme}://{_host_with_optional_port(parsed.hostname, port, scheme)}"
|
||||
|
||||
|
||||
def _configured_cors_origins() -> set[str]:
|
||||
"""Return explicit configured browser origins that may call auth routes."""
|
||||
origins = set()
|
||||
for raw_origin in os.environ.get("GATEWAY_CORS_ORIGINS", "").split(","):
|
||||
origin = raw_origin.strip()
|
||||
if not origin or origin == "*":
|
||||
continue
|
||||
normalized = _normalize_origin(origin)
|
||||
if normalized:
|
||||
origins.add(normalized)
|
||||
return origins
|
||||
|
||||
|
||||
def get_configured_cors_origins() -> set[str]:
|
||||
"""Return normalized explicit browser origins from GATEWAY_CORS_ORIGINS."""
|
||||
return _configured_cors_origins()
|
||||
|
||||
|
||||
def _first_header_value(value: str | None) -> str | None:
|
||||
"""Return the first value from a comma-separated proxy header."""
|
||||
if not value:
|
||||
return None
|
||||
first = value.split(",", 1)[0].strip()
|
||||
return first or None
|
||||
|
||||
|
||||
def _forwarded_param(request: Request, name: str) -> str | None:
|
||||
"""Extract a parameter from the first RFC 7239 Forwarded header entry."""
|
||||
forwarded = _first_header_value(request.headers.get("forwarded"))
|
||||
if not forwarded:
|
||||
return None
|
||||
|
||||
for part in forwarded.split(";"):
|
||||
key, sep, value = part.strip().partition("=")
|
||||
if sep and key.lower() == name:
|
||||
return value.strip().strip('"') or None
|
||||
return None
|
||||
|
||||
|
||||
def _request_scheme(request: Request) -> str:
|
||||
"""Resolve the original request scheme from trusted proxy headers."""
|
||||
scheme = _forwarded_param(request, "proto") or _first_header_value(request.headers.get("x-forwarded-proto")) or request.url.scheme
|
||||
return scheme.lower()
|
||||
|
||||
|
||||
def _request_origin(request: Request) -> str | None:
|
||||
"""Build the origin for the URL the browser is targeting."""
|
||||
scheme = _request_scheme(request)
|
||||
host = _forwarded_param(request, "host") or _first_header_value(request.headers.get("x-forwarded-host")) or request.headers.get("host") or request.url.netloc
|
||||
|
||||
forwarded_port = _first_header_value(request.headers.get("x-forwarded-port"))
|
||||
if forwarded_port and ":" not in host.rsplit("]", 1)[-1]:
|
||||
host = f"{host}:{forwarded_port}"
|
||||
|
||||
return _normalize_origin(f"{scheme}://{host}")
|
||||
|
||||
|
||||
def is_allowed_auth_origin(request: Request) -> bool:
|
||||
"""Allow auth POSTs only from the same origin or explicit configured origins.
|
||||
|
||||
Login/register/initialize are exempt from the double-submit token because
|
||||
first-time browser clients do not have a CSRF token yet. They still create
|
||||
a session cookie, so browser requests with a hostile Origin header must be
|
||||
rejected to prevent login CSRF / session fixation. Requests without Origin
|
||||
are allowed for non-browser clients such as curl and mobile integrations.
|
||||
"""
|
||||
origin = request.headers.get("origin")
|
||||
if not origin:
|
||||
return True
|
||||
|
||||
normalized_origin = _normalize_origin(origin)
|
||||
if normalized_origin is None:
|
||||
return False
|
||||
|
||||
request_origin = _request_origin(request)
|
||||
return normalized_origin in _configured_cors_origins() or (request_origin is not None and normalized_origin == request_origin)
|
||||
|
||||
|
||||
class CSRFMiddleware(BaseHTTPMiddleware):
|
||||
"""Middleware that implements CSRF protection using Double Submit Cookie pattern."""
|
||||
|
||||
def __init__(self, app: ASGIApp) -> None:
|
||||
super().__init__(app)
|
||||
|
||||
async def dispatch(self, request: Request, call_next: Callable[[Request], Awaitable[Response]]) -> Response:
|
||||
_is_auth = is_auth_endpoint(request)
|
||||
|
||||
if should_check_csrf(request) and _is_auth and not is_allowed_auth_origin(request):
|
||||
return JSONResponse(
|
||||
status_code=403,
|
||||
content={"detail": "Cross-site auth request denied."},
|
||||
)
|
||||
|
||||
if should_check_csrf(request) and not _is_auth:
|
||||
cookie_token = request.cookies.get(CSRF_COOKIE_NAME)
|
||||
header_token = request.headers.get(CSRF_HEADER_NAME)
|
||||
|
||||
if not cookie_token or not header_token:
|
||||
return JSONResponse(
|
||||
status_code=403,
|
||||
content={"detail": "CSRF token missing. Include X-CSRF-Token header."},
|
||||
)
|
||||
|
||||
if not secrets.compare_digest(cookie_token, header_token):
|
||||
return JSONResponse(
|
||||
status_code=403,
|
||||
content={"detail": "CSRF token mismatch."},
|
||||
)
|
||||
|
||||
response = await call_next(request)
|
||||
|
||||
# For auth endpoints that set up session, also set CSRF cookie
|
||||
if _is_auth and request.method == "POST":
|
||||
# Generate a new CSRF token for the session
|
||||
csrf_token = generate_csrf_token()
|
||||
is_https = is_secure_request(request)
|
||||
response.set_cookie(
|
||||
key=CSRF_COOKIE_NAME,
|
||||
value=csrf_token,
|
||||
httponly=False, # Must be JS-readable for Double Submit Cookie pattern
|
||||
secure=is_https,
|
||||
samesite="strict",
|
||||
)
|
||||
|
||||
return response
|
||||
|
||||
|
||||
def get_csrf_token(request: Request) -> str | None:
|
||||
"""Get the CSRF token from the current request's cookies.
|
||||
|
||||
This is useful for server-side rendering where you need to embed
|
||||
token in forms or headers.
|
||||
"""
|
||||
return request.cookies.get(CSRF_COOKIE_NAME)
|
||||
+29
-347
@@ -3,237 +3,37 @@
|
||||
**Getters** (used by routers): raise 503 when a required dependency is
|
||||
missing, except ``get_store`` which returns ``None``.
|
||||
|
||||
``AppConfig`` is intentionally *not* cached on ``app.state``. Routers and the
|
||||
run path resolve it through :func:`deerflow.config.app_config.get_app_config`,
|
||||
which performs mtime-based hot reload, so edits to ``config.yaml`` take
|
||||
effect on the next request without a process restart. The engines created in
|
||||
:func:`langgraph_runtime` (stream bridge, persistence, checkpointer, store,
|
||||
run-event store) accept a ``startup_config`` snapshot — they are
|
||||
restart-required by design and stay bound to that snapshot to keep the live
|
||||
process consistent with itself.
|
||||
|
||||
Initialization is handled directly in ``app.py`` via :class:`AsyncExitStack`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from collections.abc import AsyncGenerator, Callable
|
||||
from collections.abc import AsyncGenerator
|
||||
from contextlib import AsyncExitStack, asynccontextmanager
|
||||
from typing import TYPE_CHECKING, TypeVar, cast
|
||||
|
||||
from fastapi import FastAPI, HTTPException, Request
|
||||
from langgraph.types import Checkpointer
|
||||
|
||||
from deerflow.config.app_config import AppConfig, get_app_config
|
||||
from deerflow.persistence.feedback import FeedbackRepository
|
||||
from deerflow.runtime import RunContext, RunManager, StreamBridge
|
||||
from deerflow.runtime.events.store.base import RunEventStore
|
||||
from deerflow.runtime.runs.store.base import RunStore
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Upper bound (seconds) for draining in-flight runs during shutdown, before the
|
||||
# AsyncExitStack tears down the checkpointer (and its connection pool). Kept
|
||||
# local to avoid an app -> deps -> app import cycle. This is a *separate* budget
|
||||
# from ``app.gateway.app._SHUTDOWN_HOOK_TIMEOUT_SECONDS`` (currently also 5.0s,
|
||||
# which bounds channel-service stop): the two govern independent teardown steps
|
||||
# and may diverge, but both count toward the lifespan shutdown window — revisit
|
||||
# them together if their sum must stay within the server's graceful-shutdown
|
||||
# timeout.
|
||||
_RUN_DRAIN_TIMEOUT_SECONDS = 5.0
|
||||
|
||||
|
||||
async def _drain_inflight_runs(run_manager: RunManager) -> None:
|
||||
"""Drain in-flight runs before the checkpointer is torn down (issue #3373).
|
||||
|
||||
Shields the (internally-bounded) drain so that even if the lifespan
|
||||
coroutine is itself cancelled mid-shutdown — a second SIGINT or the server's
|
||||
graceful-shutdown timeout, i.e. the same signal storm behind #3373 — the
|
||||
checkpointer pool is not closed while run tasks are still writing
|
||||
checkpoints. On such a cancellation we let the already-running drain finish
|
||||
(it is bounded by ``RunManager.shutdown``'s own timeout) and then propagate
|
||||
the cancellation.
|
||||
"""
|
||||
drain = asyncio.create_task(run_manager.shutdown(timeout=_RUN_DRAIN_TIMEOUT_SECONDS))
|
||||
try:
|
||||
await asyncio.shield(drain)
|
||||
except asyncio.CancelledError:
|
||||
# Re-shield so this second wait does not abandon the in-flight drain;
|
||||
# it is bounded, so this cannot hang. Then re-raise to honour shutdown.
|
||||
try:
|
||||
await asyncio.shield(drain)
|
||||
except Exception:
|
||||
logger.exception("In-flight run drain failed after shutdown cancellation")
|
||||
raise
|
||||
except Exception:
|
||||
logger.exception("Failed to drain in-flight runs during shutdown")
|
||||
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from app.gateway.auth.local_provider import LocalAuthProvider
|
||||
from app.gateway.auth.repositories.sqlite import SQLiteUserRepository
|
||||
from deerflow.persistence.thread_meta.base import ThreadMetaStore
|
||||
from deerflow.runtime import RunRecord
|
||||
|
||||
|
||||
T = TypeVar("T")
|
||||
|
||||
|
||||
async def _mark_latest_recovered_threads_error(
|
||||
run_manager: RunManager,
|
||||
thread_store: ThreadMetaStore,
|
||||
recovered_runs: list[RunRecord],
|
||||
) -> None:
|
||||
"""Mark thread status as error only when its newest run was recovered."""
|
||||
recovered_by_thread: dict[str, set[str]] = {}
|
||||
for record in recovered_runs:
|
||||
recovered_by_thread.setdefault(record.thread_id, set()).add(record.run_id)
|
||||
|
||||
for thread_id, recovered_run_ids in recovered_by_thread.items():
|
||||
try:
|
||||
latest_runs = await run_manager.list_by_thread(thread_id, user_id=None, limit=1)
|
||||
except Exception:
|
||||
logger.warning("Failed to find latest run for thread %s during run reconciliation", thread_id, exc_info=True)
|
||||
continue
|
||||
if not latest_runs or latest_runs[0].run_id not in recovered_run_ids:
|
||||
continue
|
||||
try:
|
||||
await thread_store.update_status(thread_id, "error", user_id=None)
|
||||
except Exception:
|
||||
logger.warning("Failed to mark thread %s as error during run reconciliation", thread_id, exc_info=True)
|
||||
|
||||
|
||||
def get_config() -> AppConfig:
|
||||
"""Return the freshest ``AppConfig`` for the current request.
|
||||
|
||||
Routes through :func:`deerflow.config.app_config.get_app_config`, which
|
||||
honours runtime ``ContextVar`` overrides and reloads ``config.yaml`` from
|
||||
disk when its mtime changes. ``AppConfig`` is not cached on ``app.state``
|
||||
at all — the only startup-time snapshot lives as a local
|
||||
``startup_config`` variable inside ``lifespan()`` and is passed
|
||||
explicitly into :func:`langgraph_runtime` for the engines that are
|
||||
restart-required by design. Routing every request through
|
||||
:func:`get_app_config` closes the bytedance/deer-flow issue #3107 BUG-001
|
||||
split-brain where the worker / lead-agent thread saw a stale startup
|
||||
snapshot.
|
||||
|
||||
Hot-reload boundary: fields backed by startup-time singletons
|
||||
(engines, sandbox provider, IM channels, logging handler) require a
|
||||
process restart to change at runtime. The authoritative list lives in
|
||||
:mod:`deerflow.config.reload_boundary` and is mirrored by the
|
||||
standardised ``"startup-only:"`` prefix on the matching
|
||||
``Field(description=...)`` in :class:`AppConfig` — IDE hover on those
|
||||
fields will surface the boundary inline. See
|
||||
``backend/CLAUDE.md`` "Config Hot-Reload Boundary" for the operator
|
||||
summary.
|
||||
|
||||
Any failure to materialise the config (missing file, permission denied,
|
||||
YAML parse error, validation error) is reported as 503 — semantically
|
||||
"the gateway cannot serve requests without a usable configuration" — and
|
||||
logged with the original exception so operators have something to debug.
|
||||
"""
|
||||
try:
|
||||
return get_app_config()
|
||||
except Exception as exc: # noqa: BLE001 - request boundary: log and degrade gracefully
|
||||
logger.exception("Failed to load AppConfig at request time")
|
||||
raise HTTPException(status_code=503, detail="Configuration not available") from exc
|
||||
from deerflow.runtime import RunManager, StreamBridge
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def langgraph_runtime(app: FastAPI, startup_config: AppConfig) -> AsyncGenerator[None, None]:
|
||||
async def langgraph_runtime(app: FastAPI) -> AsyncGenerator[None, None]:
|
||||
"""Bootstrap and tear down all LangGraph runtime singletons.
|
||||
|
||||
``startup_config`` is the ``AppConfig`` snapshot taken once during
|
||||
``lifespan()`` for one-shot infrastructure bootstrap. The engines and
|
||||
stores constructed here (stream bridge, persistence engine, checkpointer,
|
||||
store, run-event store) are restart-required by design — they hold live
|
||||
connections, file handles, or singleton providers — so they bind to this
|
||||
snapshot and survive across `config.yaml` edits. Request-time consumers
|
||||
must still go through :func:`get_config` for any field that should be
|
||||
hot-reloadable. See ``backend/CLAUDE.md`` "Config Hot-Reload Boundary".
|
||||
|
||||
The matching ``run_events_config`` is frozen onto ``app.state`` so
|
||||
:func:`get_run_context` pairs a freshly-loaded ``AppConfig`` with the
|
||||
*startup-time* run-events configuration the underlying ``event_store``
|
||||
was built from — otherwise the runtime could end up combining a live
|
||||
new ``run_events_config`` with an event store still bound to the
|
||||
previous backend.
|
||||
|
||||
Usage in ``app.py``::
|
||||
|
||||
async with langgraph_runtime(app, startup_config):
|
||||
async with langgraph_runtime(app):
|
||||
yield
|
||||
"""
|
||||
from deerflow.persistence.engine import close_engine, get_session_factory, init_engine_from_config
|
||||
from deerflow.agents.checkpointer.async_provider import make_checkpointer
|
||||
from deerflow.runtime import make_store, make_stream_bridge
|
||||
from deerflow.runtime.checkpointer.async_provider import make_checkpointer
|
||||
from deerflow.runtime.events.store import make_run_event_store
|
||||
|
||||
async with AsyncExitStack() as stack:
|
||||
config = startup_config
|
||||
|
||||
app.state.stream_bridge = await stack.enter_async_context(make_stream_bridge(config))
|
||||
|
||||
# Initialize persistence engine BEFORE checkpointer so that
|
||||
# auto-create-database logic runs first (postgres backend).
|
||||
await init_engine_from_config(config.database)
|
||||
|
||||
app.state.checkpointer = await stack.enter_async_context(make_checkpointer(config))
|
||||
app.state.store = await stack.enter_async_context(make_store(config))
|
||||
|
||||
# Initialize repositories — one get_session_factory() call for all.
|
||||
sf = get_session_factory()
|
||||
if sf is not None:
|
||||
from deerflow.persistence.feedback import FeedbackRepository
|
||||
from deerflow.persistence.run import RunRepository
|
||||
|
||||
app.state.run_store = RunRepository(sf)
|
||||
app.state.feedback_repo = FeedbackRepository(sf)
|
||||
else:
|
||||
from deerflow.runtime.runs.store.memory import MemoryRunStore
|
||||
|
||||
app.state.run_store = MemoryRunStore()
|
||||
app.state.feedback_repo = None
|
||||
|
||||
from deerflow.persistence.thread_meta import make_thread_store
|
||||
|
||||
app.state.thread_store = make_thread_store(sf, app.state.store)
|
||||
|
||||
# Run event store. The store and the matching ``run_events_config`` are
|
||||
# both frozen at startup so ``get_run_context`` does not combine a
|
||||
# freshly-reloaded ``AppConfig.run_events`` with a store still bound to
|
||||
# the previous backend.
|
||||
run_events_config = getattr(config, "run_events", None)
|
||||
app.state.run_events_config = run_events_config
|
||||
app.state.run_event_store = make_run_event_store(run_events_config)
|
||||
|
||||
# RunManager with store backing for persistence
|
||||
app.state.run_manager = RunManager(store=app.state.run_store)
|
||||
if getattr(config.database, "backend", None) == "sqlite":
|
||||
from deerflow.utils.time import now_iso
|
||||
|
||||
# Startup-only recovery: clean shutdowns return no active rows and
|
||||
# the thread-status update below becomes a no-op.
|
||||
recovered_runs = await app.state.run_manager.reconcile_orphaned_inflight_runs(
|
||||
error="Gateway restarted before this run reached a durable final state.",
|
||||
before=now_iso(),
|
||||
)
|
||||
await _mark_latest_recovered_threads_error(app.state.run_manager, app.state.thread_store, recovered_runs)
|
||||
|
||||
try:
|
||||
app.state.stream_bridge = await stack.enter_async_context(make_stream_bridge())
|
||||
app.state.checkpointer = await stack.enter_async_context(make_checkpointer())
|
||||
app.state.store = await stack.enter_async_context(make_store())
|
||||
app.state.run_manager = RunManager()
|
||||
yield
|
||||
finally:
|
||||
# Drain in-flight run tasks BEFORE the AsyncExitStack tears down the
|
||||
# checkpointer (and its connection pool). A run still mid-graph would
|
||||
# otherwise leak into asyncio.run() shutdown, where langgraph's
|
||||
# _checkpointer_put_after_previous aput races the closed pool and
|
||||
# raises PoolClosed (issue #3373).
|
||||
run_manager = getattr(app.state, "run_manager", None)
|
||||
if run_manager is not None:
|
||||
await _drain_inflight_runs(run_manager)
|
||||
await close_engine()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -241,148 +41,30 @@ async def langgraph_runtime(app: FastAPI, startup_config: AppConfig) -> AsyncGen
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _require(attr: str, label: str) -> Callable[[Request], T]:
|
||||
"""Create a FastAPI dependency that returns ``app.state.<attr>`` or 503."""
|
||||
|
||||
def dep(request: Request) -> T:
|
||||
val = getattr(request.app.state, attr, None)
|
||||
if val is None:
|
||||
raise HTTPException(status_code=503, detail=f"{label} not available")
|
||||
return cast(T, val)
|
||||
|
||||
dep.__name__ = dep.__qualname__ = f"get_{attr}"
|
||||
return dep
|
||||
def get_stream_bridge(request: Request) -> StreamBridge:
|
||||
"""Return the global :class:`StreamBridge`, or 503."""
|
||||
bridge = getattr(request.app.state, "stream_bridge", None)
|
||||
if bridge is None:
|
||||
raise HTTPException(status_code=503, detail="Stream bridge not available")
|
||||
return bridge
|
||||
|
||||
|
||||
get_stream_bridge: Callable[[Request], StreamBridge] = _require("stream_bridge", "Stream bridge")
|
||||
get_run_manager: Callable[[Request], RunManager] = _require("run_manager", "Run manager")
|
||||
get_checkpointer: Callable[[Request], Checkpointer] = _require("checkpointer", "Checkpointer")
|
||||
get_run_event_store: Callable[[Request], RunEventStore] = _require("run_event_store", "Run event store")
|
||||
get_feedback_repo: Callable[[Request], FeedbackRepository] = _require("feedback_repo", "Feedback")
|
||||
get_run_store: Callable[[Request], RunStore] = _require("run_store", "Run store")
|
||||
def get_run_manager(request: Request) -> RunManager:
|
||||
"""Return the global :class:`RunManager`, or 503."""
|
||||
mgr = getattr(request.app.state, "run_manager", None)
|
||||
if mgr is None:
|
||||
raise HTTPException(status_code=503, detail="Run manager not available")
|
||||
return mgr
|
||||
|
||||
|
||||
def get_checkpointer(request: Request):
|
||||
"""Return the global checkpointer, or 503."""
|
||||
cp = getattr(request.app.state, "checkpointer", None)
|
||||
if cp is None:
|
||||
raise HTTPException(status_code=503, detail="Checkpointer not available")
|
||||
return cp
|
||||
|
||||
|
||||
def get_store(request: Request):
|
||||
"""Return the global store (may be ``None`` if not configured)."""
|
||||
return getattr(request.app.state, "store", None)
|
||||
|
||||
|
||||
def get_thread_store(request: Request) -> ThreadMetaStore:
|
||||
"""Return the thread metadata store (SQL or memory-backed)."""
|
||||
val = getattr(request.app.state, "thread_store", None)
|
||||
if val is None:
|
||||
raise HTTPException(status_code=503, detail="Thread metadata store not available")
|
||||
return val
|
||||
|
||||
|
||||
def get_run_context(request: Request) -> RunContext:
|
||||
"""Build a :class:`RunContext` from ``app.state`` singletons.
|
||||
|
||||
Returns a *base* context with infrastructure dependencies. The
|
||||
``app_config`` field is resolved live so per-run fields (e.g.
|
||||
``models[*].max_tokens``) follow ``config.yaml`` edits; the
|
||||
``event_store`` / ``run_events_config`` pair stays frozen to the snapshot
|
||||
captured in :func:`langgraph_runtime` so callers never see a store bound
|
||||
to one backend paired with a config pointing at another.
|
||||
"""
|
||||
return RunContext(
|
||||
checkpointer=get_checkpointer(request),
|
||||
store=get_store(request),
|
||||
event_store=get_run_event_store(request),
|
||||
run_events_config=getattr(request.app.state, "run_events_config", None),
|
||||
thread_store=get_thread_store(request),
|
||||
app_config=get_config(),
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Auth helpers (used by authz.py and auth middleware)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Cached singletons to avoid repeated instantiation per request
|
||||
_cached_local_provider: LocalAuthProvider | None = None
|
||||
_cached_repo: SQLiteUserRepository | None = None
|
||||
|
||||
|
||||
def get_local_provider() -> LocalAuthProvider:
|
||||
"""Get or create the cached LocalAuthProvider singleton.
|
||||
|
||||
Must be called after ``init_engine_from_config()`` — the shared
|
||||
session factory is required to construct the user repository.
|
||||
"""
|
||||
global _cached_local_provider, _cached_repo
|
||||
if _cached_repo is None:
|
||||
from app.gateway.auth.repositories.sqlite import SQLiteUserRepository
|
||||
from deerflow.persistence.engine import get_session_factory
|
||||
|
||||
sf = get_session_factory()
|
||||
if sf is None:
|
||||
raise RuntimeError("get_local_provider() called before init_engine_from_config(); cannot access users table")
|
||||
_cached_repo = SQLiteUserRepository(sf)
|
||||
if _cached_local_provider is None:
|
||||
from app.gateway.auth.local_provider import LocalAuthProvider
|
||||
|
||||
_cached_local_provider = LocalAuthProvider(repository=_cached_repo)
|
||||
return _cached_local_provider
|
||||
|
||||
|
||||
async def get_current_user_from_request(request: Request):
|
||||
"""Get the current authenticated user from the request cookie.
|
||||
|
||||
Raises HTTPException 401 if not authenticated.
|
||||
"""
|
||||
from app.gateway.auth import decode_token
|
||||
from app.gateway.auth.errors import AuthErrorCode, AuthErrorResponse, TokenError, token_error_to_code
|
||||
|
||||
access_token = request.cookies.get("access_token")
|
||||
if not access_token:
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail=AuthErrorResponse(code=AuthErrorCode.NOT_AUTHENTICATED, message="Not authenticated").model_dump(),
|
||||
)
|
||||
|
||||
payload = decode_token(access_token)
|
||||
if isinstance(payload, TokenError):
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail=AuthErrorResponse(code=token_error_to_code(payload), message=f"Token error: {payload.value}").model_dump(),
|
||||
)
|
||||
|
||||
provider = get_local_provider()
|
||||
user = await provider.get_user(payload.sub)
|
||||
if user is None:
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail=AuthErrorResponse(code=AuthErrorCode.USER_NOT_FOUND, message="User not found").model_dump(),
|
||||
)
|
||||
|
||||
# Token version mismatch → password was changed, token is stale
|
||||
if user.token_version != payload.ver:
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail=AuthErrorResponse(code=AuthErrorCode.TOKEN_INVALID, message="Token revoked (password changed)").model_dump(),
|
||||
)
|
||||
|
||||
return user
|
||||
|
||||
|
||||
async def get_optional_user_from_request(request: Request):
|
||||
"""Get optional authenticated user from request.
|
||||
|
||||
Returns None if not authenticated.
|
||||
"""
|
||||
try:
|
||||
return await get_current_user_from_request(request)
|
||||
except HTTPException:
|
||||
return None
|
||||
|
||||
|
||||
async def get_current_user(request: Request) -> str | None:
|
||||
"""Extract user_id from request cookie, or None if not authenticated.
|
||||
|
||||
Thin adapter that returns the string id for callers that only need
|
||||
identification (e.g., ``feedback.py``). Full-user callers should use
|
||||
``get_current_user_from_request`` or ``get_optional_user_from_request``.
|
||||
"""
|
||||
user = await get_optional_user_from_request(request)
|
||||
return str(user.id) if user else None
|
||||
|
||||
@@ -1,38 +0,0 @@
|
||||
"""Authentication for trusted Gateway internal callers."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import secrets
|
||||
from types import SimpleNamespace
|
||||
|
||||
from deerflow.runtime.user_context import DEFAULT_USER_ID
|
||||
|
||||
INTERNAL_AUTH_HEADER_NAME = "X-DeerFlow-Internal-Token"
|
||||
INTERNAL_AUTH_ENV_VAR = "DEER_FLOW_INTERNAL_AUTH_TOKEN"
|
||||
INTERNAL_SYSTEM_ROLE = "internal"
|
||||
|
||||
|
||||
def _load_internal_auth_token() -> str:
|
||||
token = os.environ.get(INTERNAL_AUTH_ENV_VAR)
|
||||
if token:
|
||||
return token
|
||||
return secrets.token_urlsafe(32)
|
||||
|
||||
|
||||
_INTERNAL_AUTH_TOKEN = _load_internal_auth_token()
|
||||
|
||||
|
||||
def create_internal_auth_headers() -> dict[str, str]:
|
||||
"""Return headers that authenticate trusted Gateway internal calls."""
|
||||
return {INTERNAL_AUTH_HEADER_NAME: _INTERNAL_AUTH_TOKEN}
|
||||
|
||||
|
||||
def is_valid_internal_auth_token(token: str | None) -> bool:
|
||||
"""Return True when *token* matches this Gateway worker's internal token."""
|
||||
return bool(token) and secrets.compare_digest(token, _INTERNAL_AUTH_TOKEN)
|
||||
|
||||
|
||||
def get_internal_user():
|
||||
"""Return the synthetic user used for trusted internal channel calls."""
|
||||
return SimpleNamespace(id=DEFAULT_USER_ID, system_role=INTERNAL_SYSTEM_ROLE)
|
||||
@@ -1,110 +0,0 @@
|
||||
"""LangGraph compatibility auth handler — shares JWT logic with Gateway.
|
||||
|
||||
The default DeerFlow runtime is embedded in the FastAPI Gateway; scripts and
|
||||
Docker deployments do not load this module. It is retained for LangGraph
|
||||
tooling, Studio, or direct LangGraph Server compatibility through
|
||||
``langgraph.json``'s ``auth.path``.
|
||||
|
||||
When that compatibility path is used, this module reuses the same JWT and CSRF
|
||||
rules as Gateway so both modes validate sessions consistently.
|
||||
|
||||
Two layers:
|
||||
1. @auth.authenticate — validates JWT cookie, extracts user_id,
|
||||
and enforces CSRF on state-changing methods (POST/PUT/DELETE/PATCH)
|
||||
2. @auth.on — returns metadata filter so each user only sees own threads
|
||||
"""
|
||||
|
||||
import secrets
|
||||
|
||||
from langgraph_sdk import Auth
|
||||
|
||||
from app.gateway.auth.errors import TokenError
|
||||
from app.gateway.auth.jwt import decode_token
|
||||
from app.gateway.deps import get_local_provider
|
||||
|
||||
auth = Auth()
|
||||
|
||||
# Methods that require CSRF validation (state-changing per RFC 7231).
|
||||
_CSRF_METHODS = frozenset({"POST", "PUT", "DELETE", "PATCH"})
|
||||
|
||||
|
||||
def _check_csrf(request) -> None:
|
||||
"""Enforce Double Submit Cookie CSRF check for state-changing requests.
|
||||
|
||||
Mirrors Gateway's CSRFMiddleware logic so that LangGraph routes
|
||||
proxied directly by nginx have the same CSRF protection.
|
||||
"""
|
||||
method = getattr(request, "method", "") or ""
|
||||
if method.upper() not in _CSRF_METHODS:
|
||||
return
|
||||
|
||||
cookie_token = request.cookies.get("csrf_token")
|
||||
header_token = request.headers.get("x-csrf-token")
|
||||
|
||||
if not cookie_token or not header_token:
|
||||
raise Auth.exceptions.HTTPException(
|
||||
status_code=403,
|
||||
detail="CSRF token missing. Include X-CSRF-Token header.",
|
||||
)
|
||||
|
||||
if not secrets.compare_digest(cookie_token, header_token):
|
||||
raise Auth.exceptions.HTTPException(
|
||||
status_code=403,
|
||||
detail="CSRF token mismatch.",
|
||||
)
|
||||
|
||||
|
||||
@auth.authenticate
|
||||
async def authenticate(request):
|
||||
"""Validate the session cookie, decode JWT, and check token_version.
|
||||
|
||||
Same validation chain as Gateway's get_current_user_from_request:
|
||||
cookie → decode JWT → DB lookup → token_version match
|
||||
Also enforces CSRF on state-changing methods.
|
||||
"""
|
||||
# CSRF check before authentication so forged cross-site requests
|
||||
# are rejected early, even if the cookie carries a valid JWT.
|
||||
_check_csrf(request)
|
||||
|
||||
token = request.cookies.get("access_token")
|
||||
if not token:
|
||||
raise Auth.exceptions.HTTPException(
|
||||
status_code=401,
|
||||
detail="Not authenticated",
|
||||
)
|
||||
|
||||
payload = decode_token(token)
|
||||
if isinstance(payload, TokenError):
|
||||
raise Auth.exceptions.HTTPException(
|
||||
status_code=401,
|
||||
detail="Invalid token",
|
||||
)
|
||||
|
||||
user = await get_local_provider().get_user(payload.sub)
|
||||
if user is None:
|
||||
raise Auth.exceptions.HTTPException(
|
||||
status_code=401,
|
||||
detail="User not found",
|
||||
)
|
||||
if user.token_version != payload.ver:
|
||||
raise Auth.exceptions.HTTPException(
|
||||
status_code=401,
|
||||
detail="Token revoked (password changed)",
|
||||
)
|
||||
|
||||
return payload.sub
|
||||
|
||||
|
||||
@auth.on
|
||||
async def add_owner_filter(ctx: Auth.types.AuthContext, value: dict):
|
||||
"""Inject user_id metadata on writes; filter by user_id on reads.
|
||||
|
||||
Gateway stores thread ownership as ``metadata.user_id``.
|
||||
This handler ensures LangGraph Server enforces the same isolation.
|
||||
"""
|
||||
# On create/update: stamp user_id into metadata
|
||||
metadata = value.setdefault("metadata", {})
|
||||
metadata["user_id"] = ctx.user.identity
|
||||
|
||||
# Return filter dict — LangGraph applies it to search/read/delete
|
||||
return {"user_id": ctx.user.identity}
|
||||
@@ -1,15 +0,0 @@
|
||||
"""Shared pagination helpers for gateway routers."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
def trim_run_message_page(rows: list[dict], *, limit: int, after_seq: int | None) -> tuple[list[dict], bool]:
|
||||
"""Trim a ``limit + 1`` run-message page while preserving page boundaries."""
|
||||
has_more = len(rows) > limit
|
||||
if not has_more:
|
||||
return rows, False
|
||||
|
||||
if after_seq is not None:
|
||||
return rows[:limit], True
|
||||
|
||||
return rows[-limit:], True
|
||||
@@ -5,7 +5,6 @@ from pathlib import Path
|
||||
from fastapi import HTTPException
|
||||
|
||||
from deerflow.config.paths import get_paths
|
||||
from deerflow.runtime.user_context import get_effective_user_id
|
||||
|
||||
|
||||
def resolve_thread_virtual_path(thread_id: str, virtual_path: str) -> Path:
|
||||
@@ -23,7 +22,7 @@ def resolve_thread_virtual_path(thread_id: str, virtual_path: str) -> Path:
|
||||
HTTPException: If the path is invalid or outside allowed directories.
|
||||
"""
|
||||
try:
|
||||
return get_paths().resolve_virtual_path(thread_id, virtual_path, user_id=get_effective_user_id())
|
||||
return get_paths().resolve_virtual_path(thread_id, virtual_path)
|
||||
except ValueError as e:
|
||||
status = 403 if "traversal" in str(e) else 400
|
||||
raise HTTPException(status_code=status, detail=str(e))
|
||||
|
||||
@@ -1,6 +1,5 @@
|
||||
"""CRUD API for custom agents."""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import re
|
||||
import shutil
|
||||
@@ -9,10 +8,8 @@ import yaml
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from deerflow.config.agents_api_config import get_agents_api_config
|
||||
from deerflow.config.agents_config import AgentConfig, list_custom_agents, load_agent_config, load_agent_soul
|
||||
from deerflow.config.paths import get_paths
|
||||
from deerflow.runtime.user_context import get_effective_user_id
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/api", tags=["agents"])
|
||||
@@ -27,7 +24,6 @@ class AgentResponse(BaseModel):
|
||||
description: str = Field(default="", description="Agent description")
|
||||
model: str | None = Field(default=None, description="Optional model override")
|
||||
tool_groups: list[str] | None = Field(default=None, description="Optional tool group whitelist")
|
||||
skills: list[str] | None = Field(default=None, description="Optional skill whitelist (None=all, []=none)")
|
||||
soul: str | None = Field(default=None, description="SOUL.md content")
|
||||
|
||||
|
||||
@@ -44,7 +40,6 @@ class AgentCreateRequest(BaseModel):
|
||||
description: str = Field(default="", description="Agent description")
|
||||
model: str | None = Field(default=None, description="Optional model override")
|
||||
tool_groups: list[str] | None = Field(default=None, description="Optional tool group whitelist")
|
||||
skills: list[str] | None = Field(default=None, description="Optional skill whitelist (None=all enabled, []=none)")
|
||||
soul: str = Field(default="", description="SOUL.md content — agent personality and behavioral guardrails")
|
||||
|
||||
|
||||
@@ -54,7 +49,6 @@ class AgentUpdateRequest(BaseModel):
|
||||
description: str | None = Field(default=None, description="Updated description")
|
||||
model: str | None = Field(default=None, description="Updated model override")
|
||||
tool_groups: list[str] | None = Field(default=None, description="Updated tool group whitelist")
|
||||
skills: list[str] | None = Field(default=None, description="Updated skill whitelist (None=all, []=none)")
|
||||
soul: str | None = Field(default=None, description="Updated SOUL.md content")
|
||||
|
||||
|
||||
@@ -79,27 +73,17 @@ def _normalize_agent_name(name: str) -> str:
|
||||
return name.lower()
|
||||
|
||||
|
||||
def _require_agents_api_enabled() -> None:
|
||||
"""Reject access unless the custom-agent management API is explicitly enabled."""
|
||||
if not get_agents_api_config().enabled:
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail=("Custom-agent management API is disabled. Set agents_api.enabled=true to expose agent and user-profile routes over HTTP."),
|
||||
)
|
||||
|
||||
|
||||
def _agent_config_to_response(agent_cfg: AgentConfig, include_soul: bool = False, *, user_id: str | None = None) -> AgentResponse:
|
||||
def _agent_config_to_response(agent_cfg: AgentConfig, include_soul: bool = False) -> AgentResponse:
|
||||
"""Convert AgentConfig to AgentResponse."""
|
||||
soul: str | None = None
|
||||
if include_soul:
|
||||
soul = load_agent_soul(agent_cfg.name, user_id=user_id) or ""
|
||||
soul = load_agent_soul(agent_cfg.name) or ""
|
||||
|
||||
return AgentResponse(
|
||||
name=agent_cfg.name,
|
||||
description=agent_cfg.description,
|
||||
model=agent_cfg.model,
|
||||
tool_groups=agent_cfg.tool_groups,
|
||||
skills=agent_cfg.skills,
|
||||
soul=soul,
|
||||
)
|
||||
|
||||
@@ -116,12 +100,9 @@ async def list_agents() -> AgentsListResponse:
|
||||
Returns:
|
||||
List of all custom agents with their metadata and soul content.
|
||||
"""
|
||||
_require_agents_api_enabled()
|
||||
|
||||
user_id = get_effective_user_id()
|
||||
try:
|
||||
agents = list_custom_agents(user_id=user_id)
|
||||
return AgentsListResponse(agents=[_agent_config_to_response(a, include_soul=True, user_id=user_id) for a in agents])
|
||||
agents = list_custom_agents()
|
||||
return AgentsListResponse(agents=[_agent_config_to_response(a, include_soul=True) for a in agents])
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to list agents: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to list agents: {str(e)}")
|
||||
@@ -144,15 +125,9 @@ async def check_agent_name(name: str) -> dict:
|
||||
Raises:
|
||||
HTTPException: 422 if the name is invalid.
|
||||
"""
|
||||
_require_agents_api_enabled()
|
||||
_validate_agent_name(name)
|
||||
normalized = _normalize_agent_name(name)
|
||||
user_id = get_effective_user_id()
|
||||
paths = get_paths()
|
||||
# Treat the name as taken if either the per-user path or the legacy shared
|
||||
# path holds an agent — picking a name that collides with an unmigrated
|
||||
# legacy agent would shadow the legacy entry once migration runs.
|
||||
available = not paths.user_agent_dir(user_id, normalized).exists() and not paths.agent_dir(normalized).exists()
|
||||
available = not get_paths().agent_dir(normalized).exists()
|
||||
return {"available": available, "name": normalized}
|
||||
|
||||
|
||||
@@ -174,14 +149,12 @@ async def get_agent(name: str) -> AgentResponse:
|
||||
Raises:
|
||||
HTTPException: 404 if agent not found.
|
||||
"""
|
||||
_require_agents_api_enabled()
|
||||
_validate_agent_name(name)
|
||||
name = _normalize_agent_name(name)
|
||||
user_id = get_effective_user_id()
|
||||
|
||||
try:
|
||||
agent_cfg = load_agent_config(name, user_id=user_id)
|
||||
return _agent_config_to_response(agent_cfg, include_soul=True, user_id=user_id)
|
||||
agent_cfg = load_agent_config(name)
|
||||
return _agent_config_to_response(agent_cfg, include_soul=True)
|
||||
except FileNotFoundError:
|
||||
raise HTTPException(status_code=404, detail=f"Agent '{name}' not found")
|
||||
except Exception as e:
|
||||
@@ -208,27 +181,17 @@ async def create_agent_endpoint(request: AgentCreateRequest) -> AgentResponse:
|
||||
Raises:
|
||||
HTTPException: 409 if agent already exists, 422 if name is invalid.
|
||||
"""
|
||||
_require_agents_api_enabled()
|
||||
_validate_agent_name(request.name)
|
||||
normalized_name = _normalize_agent_name(request.name)
|
||||
user_id = get_effective_user_id()
|
||||
paths = get_paths()
|
||||
|
||||
def _create_agent() -> AgentResponse | None:
|
||||
# Worker thread: base-dir resolution, existence checks, directory/file
|
||||
# creation, read-back, and failure cleanup are all blocking filesystem
|
||||
# IO that must stay off the event loop.
|
||||
agent_dir = paths.user_agent_dir(user_id, normalized_name)
|
||||
legacy_dir = paths.agent_dir(normalized_name)
|
||||
agent_dir = get_paths().agent_dir(normalized_name)
|
||||
|
||||
if legacy_dir.exists():
|
||||
return None # signals 409 to the caller
|
||||
if agent_dir.exists():
|
||||
raise HTTPException(status_code=409, detail=f"Agent '{normalized_name}' already exists")
|
||||
|
||||
try:
|
||||
try:
|
||||
agent_dir.mkdir(parents=True, exist_ok=False)
|
||||
except FileExistsError:
|
||||
return None # signals 409 to the caller
|
||||
agent_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Write config.yaml
|
||||
config_data: dict = {"name": normalized_name}
|
||||
if request.description:
|
||||
@@ -237,8 +200,6 @@ async def create_agent_endpoint(request: AgentCreateRequest) -> AgentResponse:
|
||||
config_data["model"] = request.model
|
||||
if request.tool_groups is not None:
|
||||
config_data["tool_groups"] = request.tool_groups
|
||||
if request.skills is not None:
|
||||
config_data["skills"] = request.skills
|
||||
|
||||
config_file = agent_dir / "config.yaml"
|
||||
with open(config_file, "w", encoding="utf-8") as f:
|
||||
@@ -250,25 +211,18 @@ async def create_agent_endpoint(request: AgentCreateRequest) -> AgentResponse:
|
||||
|
||||
logger.info(f"Created agent '{normalized_name}' at {agent_dir}")
|
||||
|
||||
agent_cfg = load_agent_config(normalized_name, user_id=user_id)
|
||||
return _agent_config_to_response(agent_cfg, include_soul=True, user_id=user_id)
|
||||
except Exception:
|
||||
# Clean up partial state on failure before surfacing the error.
|
||||
agent_cfg = load_agent_config(normalized_name)
|
||||
return _agent_config_to_response(agent_cfg, include_soul=True)
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
# Clean up on failure
|
||||
if agent_dir.exists():
|
||||
shutil.rmtree(agent_dir)
|
||||
raise
|
||||
|
||||
try:
|
||||
response = await asyncio.to_thread(_create_agent)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to create agent '{request.name}': {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to create agent: {str(e)}")
|
||||
|
||||
if response is None:
|
||||
raise HTTPException(status_code=409, detail=f"Agent '{normalized_name}' already exists")
|
||||
|
||||
return response
|
||||
|
||||
|
||||
@router.put(
|
||||
"/agents/{name}",
|
||||
@@ -289,52 +243,33 @@ async def update_agent(name: str, request: AgentUpdateRequest) -> AgentResponse:
|
||||
Raises:
|
||||
HTTPException: 404 if agent not found.
|
||||
"""
|
||||
_require_agents_api_enabled()
|
||||
_validate_agent_name(name)
|
||||
name = _normalize_agent_name(name)
|
||||
user_id = get_effective_user_id()
|
||||
|
||||
try:
|
||||
agent_cfg = load_agent_config(name, user_id=user_id)
|
||||
agent_cfg = load_agent_config(name)
|
||||
except FileNotFoundError:
|
||||
raise HTTPException(status_code=404, detail=f"Agent '{name}' not found")
|
||||
|
||||
paths = get_paths()
|
||||
agent_dir = paths.user_agent_dir(user_id, name)
|
||||
if not agent_dir.exists() and paths.agent_dir(name).exists():
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail=(f"Agent '{name}' only exists in the legacy shared layout and is not scoped to a user. Run scripts/migrate_user_isolation.py to move legacy agents into the per-user layout before updating."),
|
||||
)
|
||||
agent_dir = get_paths().agent_dir(name)
|
||||
|
||||
try:
|
||||
# Update config if any config fields changed
|
||||
# Use model_fields_set to distinguish "field omitted" from "explicitly set to null".
|
||||
# This is critical for skills where None means "inherit all" (not "don't change").
|
||||
fields_set = request.model_fields_set
|
||||
config_changed = bool(fields_set & {"description", "model", "tool_groups", "skills"})
|
||||
config_changed = any(v is not None for v in [request.description, request.model, request.tool_groups])
|
||||
|
||||
if config_changed:
|
||||
updated: dict = {
|
||||
"name": agent_cfg.name,
|
||||
"description": request.description if "description" in fields_set else agent_cfg.description,
|
||||
"description": request.description if request.description is not None else agent_cfg.description,
|
||||
}
|
||||
new_model = request.model if "model" in fields_set else agent_cfg.model
|
||||
new_model = request.model if request.model is not None else agent_cfg.model
|
||||
if new_model is not None:
|
||||
updated["model"] = new_model
|
||||
|
||||
new_tool_groups = request.tool_groups if "tool_groups" in fields_set else agent_cfg.tool_groups
|
||||
new_tool_groups = request.tool_groups if request.tool_groups is not None else agent_cfg.tool_groups
|
||||
if new_tool_groups is not None:
|
||||
updated["tool_groups"] = new_tool_groups
|
||||
|
||||
# skills: None = inherit all, [] = no skills, ["a","b"] = whitelist
|
||||
if "skills" in fields_set:
|
||||
new_skills = request.skills
|
||||
else:
|
||||
new_skills = agent_cfg.skills
|
||||
if new_skills is not None:
|
||||
updated["skills"] = new_skills
|
||||
|
||||
config_file = agent_dir / "config.yaml"
|
||||
with open(config_file, "w", encoding="utf-8") as f:
|
||||
yaml.dump(updated, f, default_flow_style=False, allow_unicode=True)
|
||||
@@ -346,8 +281,8 @@ async def update_agent(name: str, request: AgentUpdateRequest) -> AgentResponse:
|
||||
|
||||
logger.info(f"Updated agent '{name}'")
|
||||
|
||||
refreshed_cfg = load_agent_config(name, user_id=user_id)
|
||||
return _agent_config_to_response(refreshed_cfg, include_soul=True, user_id=user_id)
|
||||
refreshed_cfg = load_agent_config(name)
|
||||
return _agent_config_to_response(refreshed_cfg, include_soul=True)
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
@@ -380,8 +315,6 @@ async def get_user_profile() -> UserProfileResponse:
|
||||
Returns:
|
||||
UserProfileResponse with content=None if USER.md does not exist yet.
|
||||
"""
|
||||
_require_agents_api_enabled()
|
||||
|
||||
try:
|
||||
user_md_path = get_paths().user_md_file
|
||||
if not user_md_path.exists():
|
||||
@@ -408,8 +341,6 @@ async def update_user_profile(request: UserProfileUpdateRequest) -> UserProfileR
|
||||
Returns:
|
||||
UserProfileResponse with the saved content.
|
||||
"""
|
||||
_require_agents_api_enabled()
|
||||
|
||||
try:
|
||||
paths = get_paths()
|
||||
paths.base_dir.mkdir(parents=True, exist_ok=True)
|
||||
@@ -434,38 +365,19 @@ async def delete_agent(name: str) -> None:
|
||||
name: The agent name.
|
||||
|
||||
Raises:
|
||||
HTTPException: 404 if no per-user copy exists; 409 if only a legacy
|
||||
shared copy exists (suggesting the migration script).
|
||||
HTTPException: 404 if agent not found.
|
||||
"""
|
||||
_require_agents_api_enabled()
|
||||
_validate_agent_name(name)
|
||||
name = _normalize_agent_name(name)
|
||||
user_id = get_effective_user_id()
|
||||
paths = get_paths()
|
||||
|
||||
def _remove_agent_dir() -> tuple[str, str]:
|
||||
# Runs in a worker thread: resolving the base dir, probing the directory
|
||||
# (`exists`), and removing it (`rmtree`) are all blocking filesystem IO
|
||||
# that must stay off the event loop.
|
||||
agent_dir = paths.user_agent_dir(user_id, name)
|
||||
agent_dir = get_paths().agent_dir(name)
|
||||
|
||||
if not agent_dir.exists():
|
||||
outcome = "legacy" if paths.agent_dir(name).exists() else "missing"
|
||||
return outcome, str(agent_dir)
|
||||
shutil.rmtree(agent_dir)
|
||||
return "deleted", str(agent_dir)
|
||||
raise HTTPException(status_code=404, detail=f"Agent '{name}' not found")
|
||||
|
||||
try:
|
||||
outcome, agent_dir = await asyncio.to_thread(_remove_agent_dir)
|
||||
shutil.rmtree(agent_dir)
|
||||
logger.info(f"Deleted agent '{name}' from {agent_dir}")
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to delete agent '{name}': {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to delete agent: {str(e)}")
|
||||
|
||||
if outcome == "legacy":
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail=(f"Agent '{name}' only exists in the legacy shared layout and is not scoped to a user. Run scripts/migrate_user_isolation.py to move legacy agents into the per-user layout before deleting."),
|
||||
)
|
||||
if outcome == "missing":
|
||||
raise HTTPException(status_code=404, detail=f"Agent '{name}' not found")
|
||||
|
||||
logger.info(f"Deleted agent '{name}' from {agent_dir}")
|
||||
|
||||
@@ -7,7 +7,6 @@ from urllib.parse import quote
|
||||
from fastapi import APIRouter, HTTPException, Request
|
||||
from fastapi.responses import FileResponse, PlainTextResponse, Response
|
||||
|
||||
from app.gateway.authz import require_permission
|
||||
from app.gateway.path_utils import resolve_thread_virtual_path
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -20,9 +19,6 @@ ACTIVE_CONTENT_MIME_TYPES = {
|
||||
"image/svg+xml",
|
||||
}
|
||||
|
||||
MAX_SKILL_ARCHIVE_MEMBER_BYTES = 16 * 1024 * 1024
|
||||
_SKILL_ARCHIVE_READ_CHUNK_SIZE = 64 * 1024
|
||||
|
||||
|
||||
def _build_content_disposition(disposition_type: str, filename: str) -> str:
|
||||
"""Build an RFC 5987 encoded Content-Disposition header value."""
|
||||
@@ -47,22 +43,6 @@ def is_text_file_by_content(path: Path, sample_size: int = 8192) -> bool:
|
||||
return False
|
||||
|
||||
|
||||
def _read_skill_archive_member(zip_ref: zipfile.ZipFile, info: zipfile.ZipInfo) -> bytes:
|
||||
"""Read a .skill archive member while enforcing an uncompressed size cap."""
|
||||
if info.file_size > MAX_SKILL_ARCHIVE_MEMBER_BYTES:
|
||||
raise HTTPException(status_code=413, detail="Skill archive member is too large to preview")
|
||||
|
||||
chunks: list[bytes] = []
|
||||
total_read = 0
|
||||
with zip_ref.open(info, "r") as src:
|
||||
while chunk := src.read(_SKILL_ARCHIVE_READ_CHUNK_SIZE):
|
||||
total_read += len(chunk)
|
||||
if total_read > MAX_SKILL_ARCHIVE_MEMBER_BYTES:
|
||||
raise HTTPException(status_code=413, detail="Skill archive member is too large to preview")
|
||||
chunks.append(chunk)
|
||||
return b"".join(chunks)
|
||||
|
||||
|
||||
def _extract_file_from_skill_archive(zip_path: Path, internal_path: str) -> bytes | None:
|
||||
"""Extract a file from a .skill ZIP archive.
|
||||
|
||||
@@ -79,16 +59,16 @@ def _extract_file_from_skill_archive(zip_path: Path, internal_path: str) -> byte
|
||||
try:
|
||||
with zipfile.ZipFile(zip_path, "r") as zip_ref:
|
||||
# List all files in the archive
|
||||
infos_by_name = {info.filename: info for info in zip_ref.infolist()}
|
||||
namelist = zip_ref.namelist()
|
||||
|
||||
# Try direct path first
|
||||
if internal_path in infos_by_name:
|
||||
return _read_skill_archive_member(zip_ref, infos_by_name[internal_path])
|
||||
if internal_path in namelist:
|
||||
return zip_ref.read(internal_path)
|
||||
|
||||
# Try with any top-level directory prefix (e.g., "skill-name/SKILL.md")
|
||||
for name, info in infos_by_name.items():
|
||||
for name in namelist:
|
||||
if name.endswith("/" + internal_path) or name == internal_path:
|
||||
return _read_skill_archive_member(zip_ref, info)
|
||||
return zip_ref.read(name)
|
||||
|
||||
# Not found
|
||||
return None
|
||||
@@ -101,7 +81,6 @@ def _extract_file_from_skill_archive(zip_path: Path, internal_path: str) -> byte
|
||||
summary="Get Artifact File",
|
||||
description="Retrieve an artifact file generated by the AI agent. Text and binary files can be viewed inline, while active web content is always downloaded.",
|
||||
)
|
||||
@require_permission("threads", "read", owner_check=True)
|
||||
async def get_artifact(thread_id: str, path: str, request: Request, download: bool = False) -> Response:
|
||||
"""Get an artifact file by its path.
|
||||
|
||||
|
||||
@@ -1,527 +0,0 @@
|
||||
"""Authentication endpoints."""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
from ipaddress import ip_address, ip_network
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request, Response, status
|
||||
from fastapi.security import OAuth2PasswordRequestForm
|
||||
from pydantic import BaseModel, EmailStr, Field, field_validator
|
||||
|
||||
from app.gateway.auth import (
|
||||
UserResponse,
|
||||
create_access_token,
|
||||
)
|
||||
from app.gateway.auth.config import get_auth_config
|
||||
from app.gateway.auth.errors import AuthErrorCode, AuthErrorResponse
|
||||
from app.gateway.csrf_middleware import is_secure_request
|
||||
from app.gateway.deps import get_current_user_from_request, get_local_provider
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/v1/auth", tags=["auth"])
|
||||
|
||||
|
||||
# ── Request/Response Models ──────────────────────────────────────────────
|
||||
|
||||
|
||||
class LoginResponse(BaseModel):
|
||||
"""Response model for login — token only lives in HttpOnly cookie."""
|
||||
|
||||
expires_in: int # seconds
|
||||
needs_setup: bool = False
|
||||
|
||||
|
||||
# Top common-password blocklist. Drawn from the public SecLists "10k worst
|
||||
# passwords" set, lowercased + length>=8 only (shorter ones already fail
|
||||
# the min_length check). Kept tight on purpose: this is the **lower bound**
|
||||
# defense, not a full HIBP / passlib check, and runs in-process per request.
|
||||
_COMMON_PASSWORDS: frozenset[str] = frozenset(
|
||||
{
|
||||
"password",
|
||||
"password1",
|
||||
"password12",
|
||||
"password123",
|
||||
"password1234",
|
||||
"12345678",
|
||||
"123456789",
|
||||
"1234567890",
|
||||
"qwerty12",
|
||||
"qwertyui",
|
||||
"qwerty123",
|
||||
"abc12345",
|
||||
"abcd1234",
|
||||
"iloveyou",
|
||||
"letmein1",
|
||||
"welcome1",
|
||||
"welcome123",
|
||||
"admin123",
|
||||
"administrator",
|
||||
"passw0rd",
|
||||
"p@ssw0rd",
|
||||
"monkey12",
|
||||
"trustno1",
|
||||
"sunshine",
|
||||
"princess",
|
||||
"football",
|
||||
"baseball",
|
||||
"superman",
|
||||
"batman123",
|
||||
"starwars",
|
||||
"dragon123",
|
||||
"master123",
|
||||
"shadow12",
|
||||
"michael1",
|
||||
"jennifer",
|
||||
"computer",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _password_is_common(password: str) -> bool:
|
||||
"""Case-insensitive blocklist check.
|
||||
|
||||
Lowercases the input so trivial mutations like ``Password`` /
|
||||
``PASSWORD`` are also rejected. Does not normalize digit substitutions
|
||||
(``p@ssw0rd`` is included as a literal entry instead) — keeping the
|
||||
rule cheap and predictable.
|
||||
"""
|
||||
return password.lower() in _COMMON_PASSWORDS
|
||||
|
||||
|
||||
def _validate_strong_password(value: str) -> str:
|
||||
"""Pydantic field-validator body shared by Register + ChangePassword.
|
||||
|
||||
Constraint = function, not type-level mixin. The two request models
|
||||
have no "is-a" relationship; they only share the password-strength
|
||||
rule. Lifting it into a free function lets each model bind it via
|
||||
``@field_validator(field_name)`` without inheritance gymnastics.
|
||||
"""
|
||||
if _password_is_common(value):
|
||||
raise ValueError("Password is too common; choose a stronger password.")
|
||||
return value
|
||||
|
||||
|
||||
class RegisterRequest(BaseModel):
|
||||
"""Request model for user registration."""
|
||||
|
||||
email: EmailStr
|
||||
password: str = Field(..., min_length=8)
|
||||
|
||||
_strong_password = field_validator("password")(classmethod(lambda cls, v: _validate_strong_password(v)))
|
||||
|
||||
|
||||
class ChangePasswordRequest(BaseModel):
|
||||
"""Request model for password change (also handles setup flow)."""
|
||||
|
||||
current_password: str
|
||||
new_password: str = Field(..., min_length=8)
|
||||
new_email: EmailStr | None = None
|
||||
|
||||
_strong_password = field_validator("new_password")(classmethod(lambda cls, v: _validate_strong_password(v)))
|
||||
|
||||
|
||||
class MessageResponse(BaseModel):
|
||||
"""Generic message response."""
|
||||
|
||||
message: str
|
||||
|
||||
|
||||
# ── Helpers ───────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _set_session_cookie(response: Response, token: str, request: Request) -> None:
|
||||
"""Set the access_token HttpOnly cookie on the response."""
|
||||
config = get_auth_config()
|
||||
is_https = is_secure_request(request)
|
||||
response.set_cookie(
|
||||
key="access_token",
|
||||
value=token,
|
||||
httponly=True,
|
||||
secure=is_https,
|
||||
samesite="lax",
|
||||
max_age=config.token_expiry_days * 24 * 3600 if is_https else None,
|
||||
)
|
||||
|
||||
|
||||
# ── Rate Limiting ────────────────────────────────────────────────────────
|
||||
# In-process dict — not shared across workers.
|
||||
#
|
||||
# **Limitation**: with multi-worker deployments (e.g., gunicorn -w N), each
|
||||
# worker maintains its own lockout table, so an attacker effectively gets
|
||||
# N × _MAX_LOGIN_ATTEMPTS guesses before being locked out everywhere. For
|
||||
# production multi-worker setups, replace this with a shared store (Redis,
|
||||
# database-backed counter) to enforce a true per-IP limit.
|
||||
|
||||
_MAX_LOGIN_ATTEMPTS = 5
|
||||
_LOCKOUT_SECONDS = 300 # 5 minutes
|
||||
|
||||
# ip → (fail_count, lock_until_timestamp)
|
||||
_login_attempts: dict[str, tuple[int, float]] = {}
|
||||
|
||||
|
||||
def _trusted_proxies() -> list:
|
||||
"""Parse ``AUTH_TRUSTED_PROXIES`` env var into a list of ip_network objects.
|
||||
|
||||
Comma-separated CIDR or single-IP entries. Empty / unset = no proxy is
|
||||
trusted (direct mode). Invalid entries are skipped with a logger warning.
|
||||
Read live so env-var overrides take effect immediately and tests can
|
||||
``monkeypatch.setenv`` without poking a module-level cache.
|
||||
"""
|
||||
raw = os.getenv("AUTH_TRUSTED_PROXIES", "").strip()
|
||||
if not raw:
|
||||
return []
|
||||
nets = []
|
||||
for entry in raw.split(","):
|
||||
entry = entry.strip()
|
||||
if not entry:
|
||||
continue
|
||||
try:
|
||||
nets.append(ip_network(entry, strict=False))
|
||||
except ValueError:
|
||||
logger.warning("AUTH_TRUSTED_PROXIES: ignoring invalid entry %r", entry)
|
||||
return nets
|
||||
|
||||
|
||||
def _get_client_ip(request: Request) -> str:
|
||||
"""Extract the real client IP for rate limiting.
|
||||
|
||||
Trust model:
|
||||
|
||||
- The TCP peer (``request.client.host``) is always the baseline. It is
|
||||
whatever the kernel reports as the connecting socket — unforgeable
|
||||
by the client itself.
|
||||
- ``X-Real-IP`` is **only** honored if the TCP peer is in the
|
||||
``AUTH_TRUSTED_PROXIES`` allowlist (set via env var, comma-separated
|
||||
CIDR or single IPs). When set, the gateway is assumed to be behind a
|
||||
reverse proxy (nginx, Cloudflare, ALB, …) that overwrites
|
||||
``X-Real-IP`` with the original client address.
|
||||
- With no ``AUTH_TRUSTED_PROXIES`` set, ``X-Real-IP`` is silently
|
||||
ignored — closing the bypass where any client could rotate the
|
||||
header to dodge per-IP rate limits in dev / direct-gateway mode.
|
||||
|
||||
``X-Forwarded-For`` is intentionally NOT used because it is naturally
|
||||
client-controlled at the *first* hop and the trust chain is harder to
|
||||
audit per-request.
|
||||
"""
|
||||
peer_host = request.client.host if request.client else None
|
||||
|
||||
trusted = _trusted_proxies()
|
||||
if trusted and peer_host:
|
||||
try:
|
||||
peer_ip = ip_address(peer_host)
|
||||
if any(peer_ip in net for net in trusted):
|
||||
real_ip = request.headers.get("x-real-ip", "").strip()
|
||||
if real_ip:
|
||||
return real_ip
|
||||
except ValueError:
|
||||
# peer_host wasn't a parseable IP (e.g. "unknown") — fall through
|
||||
pass
|
||||
|
||||
return peer_host or "unknown"
|
||||
|
||||
|
||||
def _check_rate_limit(ip: str) -> None:
|
||||
"""Raise 429 if the IP is currently locked out."""
|
||||
record = _login_attempts.get(ip)
|
||||
if record is None:
|
||||
return
|
||||
fail_count, lock_until = record
|
||||
if fail_count >= _MAX_LOGIN_ATTEMPTS:
|
||||
if time.time() < lock_until:
|
||||
raise HTTPException(
|
||||
status_code=429,
|
||||
detail="Too many login attempts. Try again later.",
|
||||
)
|
||||
del _login_attempts[ip]
|
||||
|
||||
|
||||
_MAX_TRACKED_IPS = 10000
|
||||
|
||||
|
||||
def _record_login_failure(ip: str) -> None:
|
||||
"""Record a failed login attempt for the given IP."""
|
||||
# Evict expired lockouts when dict grows too large
|
||||
if len(_login_attempts) >= _MAX_TRACKED_IPS:
|
||||
now = time.time()
|
||||
expired = [k for k, (c, t) in _login_attempts.items() if c >= _MAX_LOGIN_ATTEMPTS and now >= t]
|
||||
for k in expired:
|
||||
del _login_attempts[k]
|
||||
# If still too large, evict cheapest-to-lose half: below-threshold
|
||||
# IPs (lock_until=0.0) sort first, then earliest-expiring lockouts.
|
||||
if len(_login_attempts) >= _MAX_TRACKED_IPS:
|
||||
by_time = sorted(_login_attempts.items(), key=lambda kv: kv[1][1])
|
||||
for k, _ in by_time[: len(by_time) // 2]:
|
||||
del _login_attempts[k]
|
||||
|
||||
record = _login_attempts.get(ip)
|
||||
if record is None:
|
||||
_login_attempts[ip] = (1, 0.0)
|
||||
else:
|
||||
new_count = record[0] + 1
|
||||
lock_until = time.time() + _LOCKOUT_SECONDS if new_count >= _MAX_LOGIN_ATTEMPTS else 0.0
|
||||
_login_attempts[ip] = (new_count, lock_until)
|
||||
|
||||
|
||||
def _record_login_success(ip: str) -> None:
|
||||
"""Clear failure counter for the given IP on successful login."""
|
||||
_login_attempts.pop(ip, None)
|
||||
|
||||
|
||||
# ── Endpoints ─────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.post("/login/local", response_model=LoginResponse)
|
||||
async def login_local(
|
||||
request: Request,
|
||||
response: Response,
|
||||
form_data: OAuth2PasswordRequestForm = Depends(),
|
||||
):
|
||||
"""Local email/password login."""
|
||||
client_ip = _get_client_ip(request)
|
||||
_check_rate_limit(client_ip)
|
||||
|
||||
user = await get_local_provider().authenticate({"email": form_data.username, "password": form_data.password})
|
||||
|
||||
if user is None:
|
||||
_record_login_failure(client_ip)
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail=AuthErrorResponse(code=AuthErrorCode.INVALID_CREDENTIALS, message="Incorrect email or password").model_dump(),
|
||||
)
|
||||
|
||||
_record_login_success(client_ip)
|
||||
token = create_access_token(str(user.id), token_version=user.token_version)
|
||||
_set_session_cookie(response, token, request)
|
||||
|
||||
return LoginResponse(
|
||||
expires_in=get_auth_config().token_expiry_days * 24 * 3600,
|
||||
needs_setup=user.needs_setup,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/register", response_model=UserResponse, status_code=status.HTTP_201_CREATED)
|
||||
async def register(request: Request, response: Response, body: RegisterRequest):
|
||||
"""Register a new user account (always 'user' role).
|
||||
|
||||
The first admin is created explicitly through /initialize. This endpoint creates regular users.
|
||||
Auto-login by setting the session cookie.
|
||||
"""
|
||||
try:
|
||||
user = await get_local_provider().create_user(email=body.email, password=body.password, system_role="user")
|
||||
except ValueError:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail=AuthErrorResponse(code=AuthErrorCode.EMAIL_ALREADY_EXISTS, message="Email already registered").model_dump(),
|
||||
)
|
||||
|
||||
token = create_access_token(str(user.id), token_version=user.token_version)
|
||||
_set_session_cookie(response, token, request)
|
||||
|
||||
return UserResponse(id=str(user.id), email=user.email, system_role=user.system_role)
|
||||
|
||||
|
||||
@router.post("/logout", response_model=MessageResponse)
|
||||
async def logout(request: Request, response: Response):
|
||||
"""Logout current user by clearing the cookie."""
|
||||
response.delete_cookie(key="access_token", secure=is_secure_request(request), samesite="lax")
|
||||
return MessageResponse(message="Successfully logged out")
|
||||
|
||||
|
||||
@router.post("/change-password", response_model=MessageResponse)
|
||||
async def change_password(request: Request, response: Response, body: ChangePasswordRequest):
|
||||
"""Change password for the currently authenticated user.
|
||||
|
||||
Also handles the first-boot setup flow:
|
||||
- If new_email is provided, updates email (checks uniqueness)
|
||||
- If user.needs_setup is True and new_email is given, clears needs_setup
|
||||
- Always increments token_version to invalidate old sessions
|
||||
- Re-issues session cookie with new token_version
|
||||
"""
|
||||
from app.gateway.auth.password import hash_password_async, verify_password_async
|
||||
|
||||
user = await get_current_user_from_request(request)
|
||||
|
||||
if user.password_hash is None:
|
||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=AuthErrorResponse(code=AuthErrorCode.INVALID_CREDENTIALS, message="OAuth users cannot change password").model_dump())
|
||||
|
||||
if not await verify_password_async(body.current_password, user.password_hash):
|
||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=AuthErrorResponse(code=AuthErrorCode.INVALID_CREDENTIALS, message="Current password is incorrect").model_dump())
|
||||
|
||||
provider = get_local_provider()
|
||||
|
||||
# Update email if provided
|
||||
if body.new_email is not None:
|
||||
existing = await provider.get_user_by_email(body.new_email)
|
||||
if existing and str(existing.id) != str(user.id):
|
||||
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=AuthErrorResponse(code=AuthErrorCode.EMAIL_ALREADY_EXISTS, message="Email already in use").model_dump())
|
||||
user.email = body.new_email
|
||||
|
||||
# Update password + bump version
|
||||
user.password_hash = await hash_password_async(body.new_password)
|
||||
user.token_version += 1
|
||||
|
||||
# Clear setup flag if this is the setup flow
|
||||
if user.needs_setup and body.new_email is not None:
|
||||
user.needs_setup = False
|
||||
|
||||
await provider.update_user(user)
|
||||
|
||||
# Re-issue cookie with new token_version
|
||||
token = create_access_token(str(user.id), token_version=user.token_version)
|
||||
_set_session_cookie(response, token, request)
|
||||
|
||||
return MessageResponse(message="Password changed successfully")
|
||||
|
||||
|
||||
@router.get("/me", response_model=UserResponse)
|
||||
async def get_me(request: Request):
|
||||
"""Get current authenticated user info."""
|
||||
user = await get_current_user_from_request(request)
|
||||
return UserResponse(id=str(user.id), email=user.email, system_role=user.system_role, needs_setup=user.needs_setup)
|
||||
|
||||
|
||||
# Per-IP cache: ip → (timestamp, result_dict).
|
||||
# Returns the cached result within the TTL instead of 429, because
|
||||
# the answer (whether an admin exists) rarely changes and returning
|
||||
# 429 breaks multi-tab / post-restart reconnection storms.
|
||||
_SETUP_STATUS_CACHE: dict[str, tuple[float, dict]] = {}
|
||||
_SETUP_STATUS_CACHE_TTL_SECONDS = 60
|
||||
_MAX_TRACKED_SETUP_STATUS_IPS = 10000
|
||||
_SETUP_STATUS_INFLIGHT: dict[str, asyncio.Task[dict]] = {}
|
||||
_SETUP_STATUS_INFLIGHT_GUARD = asyncio.Lock()
|
||||
|
||||
|
||||
@router.get("/setup-status")
|
||||
async def setup_status(request: Request):
|
||||
"""Check if an admin account exists. Returns needs_setup=True when no admin exists."""
|
||||
client_ip = _get_client_ip(request)
|
||||
now = time.time()
|
||||
|
||||
# Return cached result when within TTL — avoids 429 on multi-tab reconnection.
|
||||
cached = _SETUP_STATUS_CACHE.get(client_ip)
|
||||
if cached is not None:
|
||||
cached_time, cached_result = cached
|
||||
if now - cached_time < _SETUP_STATUS_CACHE_TTL_SECONDS:
|
||||
return cached_result
|
||||
|
||||
async with _SETUP_STATUS_INFLIGHT_GUARD:
|
||||
# Recheck cache after waiting for the inflight guard.
|
||||
now = time.time()
|
||||
cached = _SETUP_STATUS_CACHE.get(client_ip)
|
||||
if cached is not None:
|
||||
cached_time, cached_result = cached
|
||||
if now - cached_time < _SETUP_STATUS_CACHE_TTL_SECONDS:
|
||||
return cached_result
|
||||
|
||||
task = _SETUP_STATUS_INFLIGHT.get(client_ip)
|
||||
if task is None:
|
||||
# Evict stale entries when dict grows too large to bound memory usage.
|
||||
if len(_SETUP_STATUS_CACHE) >= _MAX_TRACKED_SETUP_STATUS_IPS:
|
||||
cutoff = now - _SETUP_STATUS_CACHE_TTL_SECONDS
|
||||
stale = [k for k, (t, _) in _SETUP_STATUS_CACHE.items() if t < cutoff]
|
||||
for k in stale:
|
||||
del _SETUP_STATUS_CACHE[k]
|
||||
if len(_SETUP_STATUS_CACHE) >= _MAX_TRACKED_SETUP_STATUS_IPS:
|
||||
by_time = sorted(_SETUP_STATUS_CACHE.items(), key=lambda entry: entry[1][0])
|
||||
for k, _ in by_time[: len(by_time) // 2]:
|
||||
del _SETUP_STATUS_CACHE[k]
|
||||
|
||||
async def _compute_setup_status() -> dict:
|
||||
admin_count = await get_local_provider().count_admin_users()
|
||||
return {"needs_setup": admin_count == 0}
|
||||
|
||||
task = asyncio.create_task(_compute_setup_status())
|
||||
_SETUP_STATUS_INFLIGHT[client_ip] = task
|
||||
|
||||
try:
|
||||
result = await task
|
||||
finally:
|
||||
async with _SETUP_STATUS_INFLIGHT_GUARD:
|
||||
if _SETUP_STATUS_INFLIGHT.get(client_ip) is task:
|
||||
del _SETUP_STATUS_INFLIGHT[client_ip]
|
||||
|
||||
# Cache only the stable "initialized" result to avoid stale setup redirects.
|
||||
if result["needs_setup"] is False:
|
||||
_SETUP_STATUS_CACHE[client_ip] = (time.time(), result)
|
||||
else:
|
||||
_SETUP_STATUS_CACHE.pop(client_ip, None)
|
||||
return result
|
||||
|
||||
|
||||
class InitializeAdminRequest(BaseModel):
|
||||
"""Request model for first-boot admin account creation."""
|
||||
|
||||
email: EmailStr
|
||||
password: str = Field(..., min_length=8)
|
||||
|
||||
_strong_password = field_validator("password")(classmethod(lambda cls, v: _validate_strong_password(v)))
|
||||
|
||||
|
||||
@router.post("/initialize", response_model=UserResponse, status_code=status.HTTP_201_CREATED)
|
||||
async def initialize_admin(request: Request, response: Response, body: InitializeAdminRequest):
|
||||
"""Create the first admin account on initial system setup.
|
||||
|
||||
Only callable when no admin exists. Returns 409 Conflict if an admin
|
||||
already exists.
|
||||
|
||||
On success, the admin account is created with ``needs_setup=False`` and
|
||||
the session cookie is set.
|
||||
"""
|
||||
admin_count = await get_local_provider().count_admin_users()
|
||||
if admin_count > 0:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_409_CONFLICT,
|
||||
detail=AuthErrorResponse(code=AuthErrorCode.SYSTEM_ALREADY_INITIALIZED, message="System already initialized").model_dump(),
|
||||
)
|
||||
|
||||
try:
|
||||
user = await get_local_provider().create_user(email=body.email, password=body.password, system_role="admin", needs_setup=False)
|
||||
except ValueError:
|
||||
# DB unique-constraint race: another concurrent request beat us.
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_409_CONFLICT,
|
||||
detail=AuthErrorResponse(code=AuthErrorCode.SYSTEM_ALREADY_INITIALIZED, message="System already initialized").model_dump(),
|
||||
)
|
||||
|
||||
token = create_access_token(str(user.id), token_version=user.token_version)
|
||||
_set_session_cookie(response, token, request)
|
||||
|
||||
return UserResponse(id=str(user.id), email=user.email, system_role=user.system_role)
|
||||
|
||||
|
||||
# ── OAuth Endpoints (Future/Placeholder) ─────────────────────────────────
|
||||
|
||||
|
||||
@router.get("/oauth/{provider}")
|
||||
async def oauth_login(provider: str):
|
||||
"""Initiate OAuth login flow.
|
||||
|
||||
Redirects to the OAuth provider's authorization URL.
|
||||
Currently a placeholder - requires OAuth provider implementation.
|
||||
"""
|
||||
if provider not in ["github", "google"]:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail=f"Unsupported OAuth provider: {provider}",
|
||||
)
|
||||
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_501_NOT_IMPLEMENTED,
|
||||
detail="OAuth login not yet implemented",
|
||||
)
|
||||
|
||||
|
||||
@router.get("/callback/{provider}")
|
||||
async def oauth_callback(provider: str, code: str, state: str):
|
||||
"""OAuth callback endpoint.
|
||||
|
||||
Handles the OAuth provider's callback after user authorization.
|
||||
Currently a placeholder.
|
||||
"""
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_501_NOT_IMPLEMENTED,
|
||||
detail="OAuth callback not yet implemented",
|
||||
)
|
||||
@@ -1,188 +0,0 @@
|
||||
"""Feedback endpoints — create, list, stats, delete.
|
||||
|
||||
Allows users to submit thumbs-up/down feedback on runs,
|
||||
optionally scoped to a specific message.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Request
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.gateway.authz import require_permission
|
||||
from app.gateway.deps import get_current_user, get_feedback_repo, get_run_store
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/api/threads", tags=["feedback"])
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Request / response models
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class FeedbackCreateRequest(BaseModel):
|
||||
rating: int = Field(..., description="Feedback rating: +1 (positive) or -1 (negative)")
|
||||
comment: str | None = Field(default=None, description="Optional text feedback")
|
||||
message_id: str | None = Field(default=None, description="Optional: scope feedback to a specific message")
|
||||
|
||||
|
||||
class FeedbackUpsertRequest(BaseModel):
|
||||
rating: int = Field(..., description="Feedback rating: +1 (positive) or -1 (negative)")
|
||||
comment: str | None = Field(default=None, description="Optional text feedback")
|
||||
|
||||
|
||||
class FeedbackResponse(BaseModel):
|
||||
feedback_id: str
|
||||
run_id: str
|
||||
thread_id: str
|
||||
user_id: str | None = None
|
||||
message_id: str | None = None
|
||||
rating: int
|
||||
comment: str | None = None
|
||||
created_at: str = ""
|
||||
|
||||
|
||||
class FeedbackStatsResponse(BaseModel):
|
||||
run_id: str
|
||||
total: int = 0
|
||||
positive: int = 0
|
||||
negative: int = 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Endpoints
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@router.put("/{thread_id}/runs/{run_id}/feedback", response_model=FeedbackResponse)
|
||||
@require_permission("threads", "write", owner_check=True, require_existing=True)
|
||||
async def upsert_feedback(
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
body: FeedbackUpsertRequest,
|
||||
request: Request,
|
||||
) -> dict[str, Any]:
|
||||
"""Create or update feedback for a run (idempotent)."""
|
||||
if body.rating not in (1, -1):
|
||||
raise HTTPException(status_code=400, detail="rating must be +1 or -1")
|
||||
|
||||
user_id = await get_current_user(request)
|
||||
|
||||
run_store = get_run_store(request)
|
||||
run = await run_store.get(run_id)
|
||||
if run is None:
|
||||
raise HTTPException(status_code=404, detail=f"Run {run_id} not found")
|
||||
if run.get("thread_id") != thread_id:
|
||||
raise HTTPException(status_code=404, detail=f"Run {run_id} not found in thread {thread_id}")
|
||||
|
||||
feedback_repo = get_feedback_repo(request)
|
||||
return await feedback_repo.upsert(
|
||||
run_id=run_id,
|
||||
thread_id=thread_id,
|
||||
rating=body.rating,
|
||||
user_id=user_id,
|
||||
comment=body.comment,
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/{thread_id}/runs/{run_id}/feedback")
|
||||
@require_permission("threads", "delete", owner_check=True, require_existing=True)
|
||||
async def delete_run_feedback(
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
request: Request,
|
||||
) -> dict[str, bool]:
|
||||
"""Delete the current user's feedback for a run."""
|
||||
user_id = await get_current_user(request)
|
||||
feedback_repo = get_feedback_repo(request)
|
||||
deleted = await feedback_repo.delete_by_run(
|
||||
thread_id=thread_id,
|
||||
run_id=run_id,
|
||||
user_id=user_id,
|
||||
)
|
||||
if not deleted:
|
||||
raise HTTPException(status_code=404, detail="No feedback found for this run")
|
||||
return {"success": True}
|
||||
|
||||
|
||||
@router.post("/{thread_id}/runs/{run_id}/feedback", response_model=FeedbackResponse)
|
||||
@require_permission("threads", "write", owner_check=True, require_existing=True)
|
||||
async def create_feedback(
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
body: FeedbackCreateRequest,
|
||||
request: Request,
|
||||
) -> dict[str, Any]:
|
||||
"""Submit feedback (thumbs-up/down) for a run."""
|
||||
if body.rating not in (1, -1):
|
||||
raise HTTPException(status_code=400, detail="rating must be +1 or -1")
|
||||
|
||||
user_id = await get_current_user(request)
|
||||
|
||||
# Validate run exists and belongs to thread
|
||||
run_store = get_run_store(request)
|
||||
run = await run_store.get(run_id)
|
||||
if run is None:
|
||||
raise HTTPException(status_code=404, detail=f"Run {run_id} not found")
|
||||
if run.get("thread_id") != thread_id:
|
||||
raise HTTPException(status_code=404, detail=f"Run {run_id} not found in thread {thread_id}")
|
||||
|
||||
feedback_repo = get_feedback_repo(request)
|
||||
return await feedback_repo.create(
|
||||
run_id=run_id,
|
||||
thread_id=thread_id,
|
||||
rating=body.rating,
|
||||
user_id=user_id,
|
||||
message_id=body.message_id,
|
||||
comment=body.comment,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{thread_id}/runs/{run_id}/feedback", response_model=list[FeedbackResponse])
|
||||
@require_permission("threads", "read", owner_check=True)
|
||||
async def list_feedback(
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
request: Request,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""List all feedback for a run."""
|
||||
feedback_repo = get_feedback_repo(request)
|
||||
return await feedback_repo.list_by_run(thread_id, run_id)
|
||||
|
||||
|
||||
@router.get("/{thread_id}/runs/{run_id}/feedback/stats", response_model=FeedbackStatsResponse)
|
||||
@require_permission("threads", "read", owner_check=True)
|
||||
async def feedback_stats(
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
request: Request,
|
||||
) -> dict[str, Any]:
|
||||
"""Get aggregated feedback stats (positive/negative counts) for a run."""
|
||||
feedback_repo = get_feedback_repo(request)
|
||||
return await feedback_repo.aggregate_by_run(thread_id, run_id)
|
||||
|
||||
|
||||
@router.delete("/{thread_id}/runs/{run_id}/feedback/{feedback_id}")
|
||||
@require_permission("threads", "delete", owner_check=True, require_existing=True)
|
||||
async def delete_feedback(
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
feedback_id: str,
|
||||
request: Request,
|
||||
) -> dict[str, bool]:
|
||||
"""Delete a feedback record."""
|
||||
feedback_repo = get_feedback_repo(request)
|
||||
# Verify feedback belongs to the specified thread/run before deleting
|
||||
existing = await feedback_repo.get(feedback_id)
|
||||
if existing is None:
|
||||
raise HTTPException(status_code=404, detail=f"Feedback {feedback_id} not found")
|
||||
if existing.get("thread_id") != thread_id or existing.get("run_id") != run_id:
|
||||
raise HTTPException(status_code=404, detail=f"Feedback {feedback_id} not found in run {run_id}")
|
||||
deleted = await feedback_repo.delete(feedback_id)
|
||||
if not deleted:
|
||||
raise HTTPException(status_code=404, detail=f"Feedback {feedback_id} not found")
|
||||
return {"success": True}
|
||||
@@ -1,10 +1,9 @@
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Literal
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Request, status
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from deerflow.config.extensions_config import ExtensionsConfig, get_extensions_config, reload_extensions_config
|
||||
@@ -13,11 +12,6 @@ logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/api", tags=["mcp"])
|
||||
|
||||
|
||||
_MCP_STDIO_COMMAND_ALLOWLIST_ENV = "DEER_FLOW_MCP_STDIO_COMMAND_ALLOWLIST"
|
||||
_DEFAULT_MCP_STDIO_COMMAND_ALLOWLIST = frozenset({"npx", "uvx"})
|
||||
_SHELL_METACHARS = frozenset(";|&`$<>\n\r")
|
||||
|
||||
|
||||
class McpOAuthConfigResponse(BaseModel):
|
||||
"""OAuth configuration for an MCP server."""
|
||||
|
||||
@@ -69,178 +63,13 @@ class McpConfigUpdateRequest(BaseModel):
|
||||
)
|
||||
|
||||
|
||||
_MASKED_VALUE = "***"
|
||||
|
||||
|
||||
async def _require_admin_user(request: Request) -> None:
|
||||
"""Require the authenticated caller to be an admin user.
|
||||
|
||||
``AuthMiddleware`` normally stamps ``request.state.user`` before the
|
||||
request reaches this router. Falling back to the strict dependency keeps
|
||||
this route safe even in tests or alternative ASGI compositions that mount
|
||||
the router without the global middleware.
|
||||
"""
|
||||
user = getattr(request.state, "user", None)
|
||||
if user is None:
|
||||
from app.gateway.deps import get_current_user_from_request
|
||||
|
||||
user = await get_current_user_from_request(request)
|
||||
|
||||
if getattr(user, "system_role", None) != "admin":
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail="Admin privileges required to manage MCP configuration.",
|
||||
)
|
||||
|
||||
|
||||
def _allowed_stdio_commands() -> set[str]:
|
||||
"""Return executable names allowed for API-managed stdio MCP servers."""
|
||||
raw = os.environ.get(_MCP_STDIO_COMMAND_ALLOWLIST_ENV)
|
||||
base = set(_DEFAULT_MCP_STDIO_COMMAND_ALLOWLIST)
|
||||
if raw is None:
|
||||
return base
|
||||
extra = {item.strip() for item in raw.split(",") if item.strip()}
|
||||
return base | extra
|
||||
|
||||
|
||||
def _stdio_command_name(command: str | None, *, server_name: str) -> str:
|
||||
"""Normalize and validate a stdio command field from the API boundary."""
|
||||
if command is None or not command.strip():
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail=f"MCP server '{server_name}' with stdio transport requires a command.",
|
||||
)
|
||||
|
||||
stripped = command.strip()
|
||||
has_path_separator = "/" in stripped or "\\" in stripped
|
||||
if stripped != command or has_path_separator or any(ch.isspace() for ch in stripped) or any(ch in stripped for ch in _SHELL_METACHARS):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail=(f"MCP server '{server_name}' command must be a single executable name; put parameters in args instead."),
|
||||
)
|
||||
|
||||
return stripped
|
||||
|
||||
|
||||
def _validate_mcp_update_request(request: McpConfigUpdateRequest) -> None:
|
||||
"""Validate API-submitted MCP config before it is persisted.
|
||||
|
||||
Local config files can still express arbitrary advanced setups, but the
|
||||
HTTP API is an untrusted boundary. Restricting stdio commands here reduces
|
||||
the blast radius of a compromised authenticated browser session.
|
||||
"""
|
||||
allowed_commands = _allowed_stdio_commands()
|
||||
for name, server in request.mcp_servers.items():
|
||||
transport_type = (server.type or "stdio").lower()
|
||||
if transport_type != "stdio":
|
||||
continue
|
||||
|
||||
command_name = _stdio_command_name(server.command, server_name=name)
|
||||
if command_name not in allowed_commands:
|
||||
allowed = ", ".join(sorted(allowed_commands)) or "<none>"
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail=(f"MCP server '{name}' uses disallowed stdio command '{command_name}'. Allowed commands: {allowed}. Configure {_MCP_STDIO_COMMAND_ALLOWLIST_ENV} to extend this list."),
|
||||
)
|
||||
|
||||
|
||||
def _mask_server_config(server: McpServerConfigResponse) -> McpServerConfigResponse:
|
||||
"""Return a copy of server config with sensitive fields masked.
|
||||
|
||||
Masks env values, header values, and removes OAuth secrets so they
|
||||
are not exposed through the GET API endpoint.
|
||||
"""
|
||||
masked_env = {k: _MASKED_VALUE for k in server.env}
|
||||
masked_headers = {k: _MASKED_VALUE for k in server.headers}
|
||||
masked_oauth = None
|
||||
if server.oauth is not None:
|
||||
masked_oauth = server.oauth.model_copy(
|
||||
update={
|
||||
"client_secret": None,
|
||||
"refresh_token": None,
|
||||
}
|
||||
)
|
||||
return server.model_copy(
|
||||
update={
|
||||
"env": masked_env,
|
||||
"headers": masked_headers,
|
||||
"oauth": masked_oauth,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _merge_preserving_secrets(
|
||||
incoming: McpServerConfigResponse,
|
||||
existing: McpServerConfigResponse,
|
||||
) -> McpServerConfigResponse:
|
||||
"""Merge incoming config with existing, preserving secrets masked by GET.
|
||||
|
||||
When the frontend toggles ``enabled`` it round-trips the full config:
|
||||
GET (masked) → modify enabled → PUT (masked values sent back).
|
||||
This function ensures masked values (``***``) are replaced with the
|
||||
real secrets from the current on-disk config.
|
||||
|
||||
``***`` is only accepted for keys that already exist in *existing*.
|
||||
New keys must provide a real value.
|
||||
|
||||
For OAuth secrets, ``None`` means "preserve the existing stored value"
|
||||
so masked GET responses can be safely round-tripped. To explicitly clear
|
||||
a stored secret, clients may send an empty string, which is converted
|
||||
to ``None`` before persisting.
|
||||
"""
|
||||
merged_env = {}
|
||||
for k, v in incoming.env.items():
|
||||
if v == _MASKED_VALUE:
|
||||
if k in existing.env:
|
||||
merged_env[k] = existing.env[k]
|
||||
else:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Cannot set env key '{k}' to masked value '***'; provide a real value.",
|
||||
)
|
||||
else:
|
||||
merged_env[k] = v
|
||||
|
||||
merged_headers = {}
|
||||
for k, v in incoming.headers.items():
|
||||
if v == _MASKED_VALUE:
|
||||
if k in existing.headers:
|
||||
merged_headers[k] = existing.headers[k]
|
||||
else:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Cannot set header '{k}' to masked value '***'; provide a real value.",
|
||||
)
|
||||
else:
|
||||
merged_headers[k] = v
|
||||
|
||||
merged_oauth = incoming.oauth
|
||||
if incoming.oauth is not None and existing.oauth is not None:
|
||||
# None = preserve (masked round-trip), "" = explicitly clear, else = new value
|
||||
merged_client_secret = existing.oauth.client_secret if incoming.oauth.client_secret is None else (None if incoming.oauth.client_secret == "" else incoming.oauth.client_secret)
|
||||
merged_refresh_token = existing.oauth.refresh_token if incoming.oauth.refresh_token is None else (None if incoming.oauth.refresh_token == "" else incoming.oauth.refresh_token)
|
||||
merged_oauth = incoming.oauth.model_copy(
|
||||
update={
|
||||
"client_secret": merged_client_secret,
|
||||
"refresh_token": merged_refresh_token,
|
||||
}
|
||||
)
|
||||
return incoming.model_copy(
|
||||
update={
|
||||
"env": merged_env,
|
||||
"headers": merged_headers,
|
||||
"oauth": merged_oauth,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/mcp/config",
|
||||
response_model=McpConfigResponse,
|
||||
summary="Get MCP Configuration",
|
||||
description="Retrieve the current Model Context Protocol (MCP) server configurations.",
|
||||
)
|
||||
async def get_mcp_configuration(request: Request) -> McpConfigResponse:
|
||||
async def get_mcp_configuration() -> McpConfigResponse:
|
||||
"""Get the current MCP configuration.
|
||||
|
||||
Returns:
|
||||
@@ -254,19 +83,16 @@ async def get_mcp_configuration(request: Request) -> McpConfigResponse:
|
||||
"enabled": true,
|
||||
"command": "npx",
|
||||
"args": ["-y", "@modelcontextprotocol/server-github"],
|
||||
"env": {"GITHUB_TOKEN": "***"},
|
||||
"env": {"GITHUB_TOKEN": "ghp_xxx"},
|
||||
"description": "GitHub MCP server for repository operations"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
"""
|
||||
await _require_admin_user(request)
|
||||
|
||||
config = get_extensions_config()
|
||||
|
||||
servers = {name: _mask_server_config(McpServerConfigResponse(**server.model_dump())) for name, server in config.mcp_servers.items()}
|
||||
return McpConfigResponse(mcp_servers=servers)
|
||||
return McpConfigResponse(mcp_servers={name: McpServerConfigResponse(**server.model_dump()) for name, server in config.mcp_servers.items()})
|
||||
|
||||
|
||||
@router.put(
|
||||
@@ -275,7 +101,7 @@ async def get_mcp_configuration(request: Request) -> McpConfigResponse:
|
||||
summary="Update MCP Configuration",
|
||||
description="Update Model Context Protocol (MCP) server configurations and save to file.",
|
||||
)
|
||||
async def update_mcp_configuration(request: Request, body: McpConfigUpdateRequest) -> McpConfigResponse:
|
||||
async def update_mcp_configuration(request: McpConfigUpdateRequest) -> McpConfigResponse:
|
||||
"""Update the MCP configuration.
|
||||
|
||||
This will:
|
||||
@@ -308,9 +134,6 @@ async def update_mcp_configuration(request: Request, body: McpConfigUpdateReques
|
||||
```
|
||||
"""
|
||||
try:
|
||||
await _require_admin_user(request)
|
||||
_validate_mcp_update_request(body)
|
||||
|
||||
# Get the current config path (or determine where to save it)
|
||||
config_path = ExtensionsConfig.resolve_config_path()
|
||||
|
||||
@@ -319,39 +142,14 @@ async def update_mcp_configuration(request: Request, body: McpConfigUpdateReques
|
||||
config_path = Path.cwd().parent / "extensions_config.json"
|
||||
logger.info(f"No existing extensions config found. Creating new config at: {config_path}")
|
||||
|
||||
# Load current config to preserve skills
|
||||
# Load current config to preserve skills configuration
|
||||
current_config = get_extensions_config()
|
||||
|
||||
# Load raw (un-resolved) JSON from disk to use as the merge source.
|
||||
# This preserves $VAR placeholders in env values and top-level keys
|
||||
# like mcpInterceptors that would otherwise be lost.
|
||||
raw_servers: dict[str, dict] = {}
|
||||
raw_other_keys: dict = {}
|
||||
if config_path is not None and config_path.exists():
|
||||
with open(config_path, encoding="utf-8") as f:
|
||||
raw_data = json.load(f)
|
||||
raw_servers = raw_data.get("mcpServers", {})
|
||||
# Preserve any top-level keys beyond mcpServers/skills
|
||||
for key, value in raw_data.items():
|
||||
if key not in ("mcpServers", "skills"):
|
||||
raw_other_keys[key] = value
|
||||
|
||||
# Merge incoming server configs with raw on-disk secrets
|
||||
merged_servers: dict[str, McpServerConfigResponse] = {}
|
||||
for name, incoming in body.mcp_servers.items():
|
||||
raw_server = raw_servers.get(name)
|
||||
if raw_server is not None:
|
||||
merged_servers[name] = _merge_preserving_secrets(
|
||||
incoming,
|
||||
McpServerConfigResponse(**raw_server),
|
||||
)
|
||||
else:
|
||||
merged_servers[name] = incoming
|
||||
|
||||
# Build config data preserving all top-level keys from the original file
|
||||
config_data = dict(raw_other_keys)
|
||||
config_data["mcpServers"] = {name: server.model_dump() for name, server in merged_servers.items()}
|
||||
config_data["skills"] = {name: {"enabled": skill.enabled} for name, skill in current_config.skills.items()}
|
||||
# Convert request to dict format for JSON serialization
|
||||
config_data = {
|
||||
"mcpServers": {name: server.model_dump() for name, server in request.mcp_servers.items()},
|
||||
"skills": {name: {"enabled": skill.enabled} for name, skill in current_config.skills.items()},
|
||||
}
|
||||
|
||||
# Write the configuration to file
|
||||
with open(config_path, "w", encoding="utf-8") as f:
|
||||
@@ -359,15 +157,13 @@ async def update_mcp_configuration(request: Request, body: McpConfigUpdateReques
|
||||
|
||||
logger.info(f"MCP configuration updated and saved to: {config_path}")
|
||||
|
||||
# Reload the Gateway configuration and update the global cache. The
|
||||
# agent runtime lives in Gateway, so this keeps API reads and tool
|
||||
# execution aligned after extensions_config.json changes.
|
||||
reloaded_config = reload_extensions_config()
|
||||
servers = {name: _mask_server_config(McpServerConfigResponse(**server.model_dump())) for name, server in reloaded_config.mcp_servers.items()}
|
||||
return McpConfigResponse(mcp_servers=servers)
|
||||
# NOTE: No need to reload/reset cache here - LangGraph Server (separate process)
|
||||
# will detect config file changes via mtime and reinitialize MCP tools automatically
|
||||
|
||||
# Reload the configuration and update the global cache
|
||||
reloaded_config = reload_extensions_config()
|
||||
return McpConfigResponse(mcp_servers={name: McpServerConfigResponse(**server.model_dump()) for name, server in reloaded_config.mcp_servers.items()})
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to update MCP configuration: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to update MCP configuration: {str(e)}")
|
||||
|
||||
@@ -13,7 +13,6 @@ from deerflow.agents.memory.updater import (
|
||||
update_memory_fact,
|
||||
)
|
||||
from deerflow.config.memory_config import get_memory_config
|
||||
from deerflow.runtime.user_context import get_effective_user_id
|
||||
|
||||
router = APIRouter(prefix="/api", tags=["memory"])
|
||||
|
||||
@@ -148,7 +147,7 @@ async def get_memory() -> MemoryResponse:
|
||||
}
|
||||
```
|
||||
"""
|
||||
memory_data = get_memory_data(user_id=get_effective_user_id())
|
||||
memory_data = get_memory_data()
|
||||
return MemoryResponse(**memory_data)
|
||||
|
||||
|
||||
@@ -168,7 +167,7 @@ async def reload_memory() -> MemoryResponse:
|
||||
Returns:
|
||||
The reloaded memory data.
|
||||
"""
|
||||
memory_data = reload_memory_data(user_id=get_effective_user_id())
|
||||
memory_data = reload_memory_data()
|
||||
return MemoryResponse(**memory_data)
|
||||
|
||||
|
||||
@@ -182,7 +181,7 @@ async def reload_memory() -> MemoryResponse:
|
||||
async def clear_memory() -> MemoryResponse:
|
||||
"""Clear all persisted memory data."""
|
||||
try:
|
||||
memory_data = clear_memory_data(user_id=get_effective_user_id())
|
||||
memory_data = clear_memory_data()
|
||||
except OSError as exc:
|
||||
raise HTTPException(status_code=500, detail="Failed to clear memory data.") from exc
|
||||
|
||||
@@ -203,7 +202,6 @@ async def create_memory_fact_endpoint(request: FactCreateRequest) -> MemoryRespo
|
||||
content=request.content,
|
||||
category=request.category,
|
||||
confidence=request.confidence,
|
||||
user_id=get_effective_user_id(),
|
||||
)
|
||||
except ValueError as exc:
|
||||
raise _map_memory_fact_value_error(exc) from exc
|
||||
@@ -223,7 +221,7 @@ async def create_memory_fact_endpoint(request: FactCreateRequest) -> MemoryRespo
|
||||
async def delete_memory_fact_endpoint(fact_id: str) -> MemoryResponse:
|
||||
"""Delete a single fact from memory by fact id."""
|
||||
try:
|
||||
memory_data = delete_memory_fact(fact_id, user_id=get_effective_user_id())
|
||||
memory_data = delete_memory_fact(fact_id)
|
||||
except KeyError as exc:
|
||||
raise HTTPException(status_code=404, detail=f"Memory fact '{fact_id}' not found.") from exc
|
||||
except OSError as exc:
|
||||
@@ -247,7 +245,6 @@ async def update_memory_fact_endpoint(fact_id: str, request: FactPatchRequest) -
|
||||
content=request.content,
|
||||
category=request.category,
|
||||
confidence=request.confidence,
|
||||
user_id=get_effective_user_id(),
|
||||
)
|
||||
except ValueError as exc:
|
||||
raise _map_memory_fact_value_error(exc) from exc
|
||||
@@ -268,7 +265,7 @@ async def update_memory_fact_endpoint(fact_id: str, request: FactPatchRequest) -
|
||||
)
|
||||
async def export_memory() -> MemoryResponse:
|
||||
"""Export the current memory data."""
|
||||
memory_data = get_memory_data(user_id=get_effective_user_id())
|
||||
memory_data = get_memory_data()
|
||||
return MemoryResponse(**memory_data)
|
||||
|
||||
|
||||
@@ -282,7 +279,7 @@ async def export_memory() -> MemoryResponse:
|
||||
async def import_memory(request: MemoryResponse) -> MemoryResponse:
|
||||
"""Import and persist memory data."""
|
||||
try:
|
||||
memory_data = import_memory_data(request.model_dump(), user_id=get_effective_user_id())
|
||||
memory_data = import_memory_data(request.model_dump())
|
||||
except OSError as exc:
|
||||
raise HTTPException(status_code=500, detail="Failed to import memory data.") from exc
|
||||
|
||||
@@ -340,7 +337,7 @@ async def get_memory_status() -> MemoryStatusResponse:
|
||||
Combined memory configuration and current data.
|
||||
"""
|
||||
config = get_memory_config()
|
||||
memory_data = get_memory_data(user_id=get_effective_user_id())
|
||||
memory_data = get_memory_data()
|
||||
|
||||
return MemoryStatusResponse(
|
||||
config=MemoryConfigResponse(
|
||||
|
||||
@@ -1,8 +1,7 @@
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.gateway.deps import get_config
|
||||
from deerflow.config.app_config import AppConfig
|
||||
from deerflow.config import get_app_config
|
||||
|
||||
router = APIRouter(prefix="/api", tags=["models"])
|
||||
|
||||
@@ -18,17 +17,10 @@ class ModelResponse(BaseModel):
|
||||
supports_reasoning_effort: bool = Field(default=False, description="Whether model supports reasoning effort")
|
||||
|
||||
|
||||
class TokenUsageResponse(BaseModel):
|
||||
"""Token usage display configuration."""
|
||||
|
||||
enabled: bool = Field(default=False, description="Whether token usage display is enabled")
|
||||
|
||||
|
||||
class ModelsListResponse(BaseModel):
|
||||
"""Response model for listing all models."""
|
||||
|
||||
models: list[ModelResponse]
|
||||
token_usage: TokenUsageResponse
|
||||
|
||||
|
||||
@router.get(
|
||||
@@ -37,14 +29,14 @@ class ModelsListResponse(BaseModel):
|
||||
summary="List All Models",
|
||||
description="Retrieve a list of all available AI models configured in the system.",
|
||||
)
|
||||
async def list_models(config: AppConfig = Depends(get_config)) -> ModelsListResponse:
|
||||
async def list_models() -> ModelsListResponse:
|
||||
"""List all available models from configuration.
|
||||
|
||||
Returns model information suitable for frontend display,
|
||||
excluding sensitive fields like API keys and internal configuration.
|
||||
|
||||
Returns:
|
||||
A list of all configured models with their metadata and token usage display settings.
|
||||
A list of all configured models with their metadata.
|
||||
|
||||
Example Response:
|
||||
```json
|
||||
@@ -52,27 +44,21 @@ async def list_models(config: AppConfig = Depends(get_config)) -> ModelsListResp
|
||||
"models": [
|
||||
{
|
||||
"name": "gpt-4",
|
||||
"model": "gpt-4",
|
||||
"display_name": "GPT-4",
|
||||
"description": "OpenAI GPT-4 model",
|
||||
"supports_thinking": false,
|
||||
"supports_reasoning_effort": false
|
||||
"supports_thinking": false
|
||||
},
|
||||
{
|
||||
"name": "claude-3-opus",
|
||||
"model": "claude-3-opus",
|
||||
"display_name": "Claude 3 Opus",
|
||||
"description": "Anthropic Claude 3 Opus model",
|
||||
"supports_thinking": true,
|
||||
"supports_reasoning_effort": false
|
||||
}
|
||||
],
|
||||
"token_usage": {
|
||||
"enabled": true
|
||||
"supports_thinking": true
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
"""
|
||||
config = get_app_config()
|
||||
models = [
|
||||
ModelResponse(
|
||||
name=model.name,
|
||||
@@ -84,10 +70,7 @@ async def list_models(config: AppConfig = Depends(get_config)) -> ModelsListResp
|
||||
)
|
||||
for model in config.models
|
||||
]
|
||||
return ModelsListResponse(
|
||||
models=models,
|
||||
token_usage=TokenUsageResponse(enabled=config.token_usage.enabled),
|
||||
)
|
||||
return ModelsListResponse(models=models)
|
||||
|
||||
|
||||
@router.get(
|
||||
@@ -96,7 +79,7 @@ async def list_models(config: AppConfig = Depends(get_config)) -> ModelsListResp
|
||||
summary="Get Model Details",
|
||||
description="Retrieve detailed information about a specific AI model by its name.",
|
||||
)
|
||||
async def get_model(model_name: str, config: AppConfig = Depends(get_config)) -> ModelResponse:
|
||||
async def get_model(model_name: str) -> ModelResponse:
|
||||
"""Get a specific model by name.
|
||||
|
||||
Args:
|
||||
@@ -118,6 +101,7 @@ async def get_model(model_name: str, config: AppConfig = Depends(get_config)) ->
|
||||
}
|
||||
```
|
||||
"""
|
||||
config = get_app_config()
|
||||
model = config.get_model_config(model_name)
|
||||
if model is None:
|
||||
raise HTTPException(status_code=404, detail=f"Model '{model_name}' not found")
|
||||
|
||||
@@ -7,17 +7,16 @@ is reused so that conversation history is preserved across calls.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import uuid
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Query, Request
|
||||
from fastapi import APIRouter, Request
|
||||
from fastapi.responses import StreamingResponse
|
||||
|
||||
from app.gateway.authz import require_permission
|
||||
from app.gateway.deps import get_checkpointer, get_feedback_repo, get_run_event_store, get_run_manager, get_run_store, get_stream_bridge
|
||||
from app.gateway.pagination import trim_run_message_page
|
||||
from app.gateway.deps import get_checkpointer, get_run_manager, get_stream_bridge
|
||||
from app.gateway.routers.thread_runs import RunCreateRequest
|
||||
from app.gateway.services import sse_consumer, start_run, wait_for_run_completion
|
||||
from app.gateway.services import sse_consumer, start_run
|
||||
from deerflow.runtime import serialize_channel_values
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -66,15 +65,14 @@ async def stateless_wait(body: RunCreateRequest, request: Request) -> dict:
|
||||
Otherwise a new temporary thread is created.
|
||||
"""
|
||||
thread_id = _resolve_thread_id(body)
|
||||
bridge = get_stream_bridge(request)
|
||||
run_mgr = get_run_manager(request)
|
||||
record = await start_run(body, thread_id, request)
|
||||
|
||||
completed = True
|
||||
if record.task is not None:
|
||||
completed = await wait_for_run_completion(bridge, record, request, run_mgr)
|
||||
try:
|
||||
await record.task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
|
||||
if completed:
|
||||
checkpointer = get_checkpointer(request)
|
||||
config = {"configurable": {"thread_id": thread_id}}
|
||||
try:
|
||||
@@ -87,57 +85,3 @@ async def stateless_wait(body: RunCreateRequest, request: Request) -> dict:
|
||||
logger.exception("Failed to fetch final state for run %s", record.run_id)
|
||||
|
||||
return {"status": record.status.value, "error": record.error}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Run-scoped read endpoints
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def _resolve_run(run_id: str, request: Request) -> dict:
|
||||
"""Fetch run by run_id with user ownership check. Raises 404 if not found."""
|
||||
run_store = get_run_store(request)
|
||||
record = await run_store.get(run_id) # user_id=AUTO filters by contextvar
|
||||
if record is None:
|
||||
raise HTTPException(status_code=404, detail=f"Run {run_id} not found")
|
||||
return record
|
||||
|
||||
|
||||
@router.get("/{run_id}/messages")
|
||||
@require_permission("runs", "read")
|
||||
async def run_messages(
|
||||
run_id: str,
|
||||
request: Request,
|
||||
limit: int = Query(default=50, le=200, ge=1),
|
||||
before_seq: int | None = Query(default=None),
|
||||
after_seq: int | None = Query(default=None),
|
||||
) -> dict:
|
||||
"""Return paginated messages for a run (cursor-based).
|
||||
|
||||
Pagination:
|
||||
- after_seq: messages with seq > after_seq (forward)
|
||||
- before_seq: messages with seq < before_seq (backward)
|
||||
- neither: latest messages
|
||||
|
||||
Response: { data: [...], has_more: bool }
|
||||
"""
|
||||
run = await _resolve_run(run_id, request)
|
||||
event_store = get_run_event_store(request)
|
||||
rows = await event_store.list_messages_by_run(
|
||||
run["thread_id"],
|
||||
run_id,
|
||||
limit=limit + 1,
|
||||
before_seq=before_seq,
|
||||
after_seq=after_seq,
|
||||
)
|
||||
data, has_more = trim_run_message_page(rows, limit=limit, after_seq=after_seq)
|
||||
return {"data": data, "has_more": has_more}
|
||||
|
||||
|
||||
@router.get("/{run_id}/feedback")
|
||||
@require_permission("runs", "read")
|
||||
async def run_feedback(run_id: str, request: Request) -> list[dict]:
|
||||
"""Return all feedback for a run."""
|
||||
run = await _resolve_run(run_id, request)
|
||||
feedback_repo = get_feedback_repo(request)
|
||||
return await feedback_repo.list_by_run(run["thread_id"], run_id)
|
||||
|
||||
@@ -1,20 +1,29 @@
|
||||
import json
|
||||
import logging
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.gateway.deps import get_config
|
||||
from app.gateway.path_utils import resolve_thread_virtual_path
|
||||
from deerflow.agents.lead_agent.prompt import refresh_skills_system_prompt_cache_async
|
||||
from deerflow.config.app_config import AppConfig
|
||||
from deerflow.config.extensions_config import ExtensionsConfig, SkillStateConfig, get_extensions_config, reload_extensions_config
|
||||
from deerflow.skills import Skill
|
||||
from deerflow.skills.installer import SkillAlreadyExistsError
|
||||
from deerflow.skills import Skill, load_skills
|
||||
from deerflow.skills.installer import SkillAlreadyExistsError, install_skill_from_archive
|
||||
from deerflow.skills.manager import (
|
||||
append_history,
|
||||
atomic_write,
|
||||
custom_skill_exists,
|
||||
ensure_custom_skill_is_editable,
|
||||
get_custom_skill_dir,
|
||||
get_custom_skill_file,
|
||||
get_skill_history_file,
|
||||
read_custom_skill_content,
|
||||
read_history,
|
||||
validate_skill_markdown_content,
|
||||
)
|
||||
from deerflow.skills.security_scanner import scan_skill_content
|
||||
from deerflow.skills.storage import get_or_new_skill_storage
|
||||
from deerflow.skills.types import SKILL_MD_FILE, SkillCategory
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -27,7 +36,7 @@ class SkillResponse(BaseModel):
|
||||
name: str = Field(..., description="Name of the skill")
|
||||
description: str = Field(..., description="Description of what the skill does")
|
||||
license: str | None = Field(None, description="License information")
|
||||
category: SkillCategory = Field(..., description="Category of the skill (public or custom)")
|
||||
category: str = Field(..., description="Category of the skill (public or custom)")
|
||||
enabled: bool = Field(default=True, description="Whether this skill is enabled")
|
||||
|
||||
|
||||
@@ -91,9 +100,9 @@ def _skill_to_response(skill: Skill) -> SkillResponse:
|
||||
summary="List All Skills",
|
||||
description="Retrieve a list of all available skills from both public and custom directories.",
|
||||
)
|
||||
async def list_skills(config: AppConfig = Depends(get_config)) -> SkillsListResponse:
|
||||
async def list_skills() -> SkillsListResponse:
|
||||
try:
|
||||
skills = get_or_new_skill_storage(app_config=config).load_skills(enabled_only=False)
|
||||
skills = load_skills(enabled_only=False)
|
||||
return SkillsListResponse(skills=[_skill_to_response(skill) for skill in skills])
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to load skills: {e}", exc_info=True)
|
||||
@@ -106,10 +115,10 @@ async def list_skills(config: AppConfig = Depends(get_config)) -> SkillsListResp
|
||||
summary="Install Skill",
|
||||
description="Install a skill from a .skill file (ZIP archive) located in the thread's user-data directory.",
|
||||
)
|
||||
async def install_skill(request: SkillInstallRequest, config: AppConfig = Depends(get_config)) -> SkillInstallResponse:
|
||||
async def install_skill(request: SkillInstallRequest) -> SkillInstallResponse:
|
||||
try:
|
||||
skill_file_path = resolve_thread_virtual_path(request.thread_id, request.path)
|
||||
result = await get_or_new_skill_storage(app_config=config).ainstall_skill_from_archive(skill_file_path)
|
||||
result = install_skill_from_archive(skill_file_path)
|
||||
await refresh_skills_system_prompt_cache_async()
|
||||
return SkillInstallResponse(**result)
|
||||
except FileNotFoundError as e:
|
||||
@@ -126,9 +135,9 @@ async def install_skill(request: SkillInstallRequest, config: AppConfig = Depend
|
||||
|
||||
|
||||
@router.get("/skills/custom", response_model=SkillsListResponse, summary="List Custom Skills")
|
||||
async def list_custom_skills(config: AppConfig = Depends(get_config)) -> SkillsListResponse:
|
||||
async def list_custom_skills() -> SkillsListResponse:
|
||||
try:
|
||||
skills = [skill for skill in get_or_new_skill_storage(app_config=config).load_skills(enabled_only=False) if skill.category == SkillCategory.CUSTOM]
|
||||
skills = [skill for skill in load_skills(enabled_only=False) if skill.category == "custom"]
|
||||
return SkillsListResponse(skills=[_skill_to_response(skill) for skill in skills])
|
||||
except Exception as e:
|
||||
logger.error("Failed to list custom skills: %s", e, exc_info=True)
|
||||
@@ -136,14 +145,13 @@ async def list_custom_skills(config: AppConfig = Depends(get_config)) -> SkillsL
|
||||
|
||||
|
||||
@router.get("/skills/custom/{skill_name}", response_model=CustomSkillContentResponse, summary="Get Custom Skill Content")
|
||||
async def get_custom_skill(skill_name: str, config: AppConfig = Depends(get_config)) -> CustomSkillContentResponse:
|
||||
async def get_custom_skill(skill_name: str) -> CustomSkillContentResponse:
|
||||
try:
|
||||
skill_name = skill_name.replace("\r\n", "").replace("\n", "")
|
||||
skills = get_or_new_skill_storage(app_config=config).load_skills(enabled_only=False)
|
||||
skill = next((s for s in skills if s.name == skill_name and s.category == SkillCategory.CUSTOM), None)
|
||||
skills = load_skills(enabled_only=False)
|
||||
skill = next((s for s in skills if s.name == skill_name and s.category == "custom"), None)
|
||||
if skill is None:
|
||||
raise HTTPException(status_code=404, detail=f"Custom skill '{skill_name}' not found")
|
||||
return CustomSkillContentResponse(**_skill_to_response(skill).model_dump(), content=get_or_new_skill_storage(app_config=config).read_custom_skill(skill_name))
|
||||
return CustomSkillContentResponse(**_skill_to_response(skill).model_dump(), content=read_custom_skill_content(skill_name))
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
@@ -152,31 +160,30 @@ async def get_custom_skill(skill_name: str, config: AppConfig = Depends(get_conf
|
||||
|
||||
|
||||
@router.put("/skills/custom/{skill_name}", response_model=CustomSkillContentResponse, summary="Edit Custom Skill")
|
||||
async def update_custom_skill(skill_name: str, request: CustomSkillUpdateRequest, config: AppConfig = Depends(get_config)) -> CustomSkillContentResponse:
|
||||
async def update_custom_skill(skill_name: str, request: CustomSkillUpdateRequest) -> CustomSkillContentResponse:
|
||||
try:
|
||||
skill_name = skill_name.replace("\r\n", "").replace("\n", "")
|
||||
storage = get_or_new_skill_storage(app_config=config)
|
||||
storage.ensure_custom_skill_is_editable(skill_name)
|
||||
storage.validate_skill_markdown_content(skill_name, request.content)
|
||||
scan = await scan_skill_content(request.content, executable=False, location=f"{skill_name}/{SKILL_MD_FILE}", app_config=config)
|
||||
ensure_custom_skill_is_editable(skill_name)
|
||||
validate_skill_markdown_content(skill_name, request.content)
|
||||
scan = await scan_skill_content(request.content, executable=False, location=f"{skill_name}/SKILL.md")
|
||||
if scan.decision == "block":
|
||||
raise HTTPException(status_code=400, detail=f"Security scan blocked the edit: {scan.reason}")
|
||||
prev_content = storage.read_custom_skill(skill_name)
|
||||
storage.write_custom_skill(skill_name, SKILL_MD_FILE, request.content)
|
||||
storage.append_history(
|
||||
skill_file = get_custom_skill_dir(skill_name) / "SKILL.md"
|
||||
prev_content = skill_file.read_text(encoding="utf-8")
|
||||
atomic_write(skill_file, request.content)
|
||||
append_history(
|
||||
skill_name,
|
||||
{
|
||||
"action": "human_edit",
|
||||
"author": "human",
|
||||
"thread_id": None,
|
||||
"file_path": SKILL_MD_FILE,
|
||||
"file_path": "SKILL.md",
|
||||
"prev_content": prev_content,
|
||||
"new_content": request.content,
|
||||
"scanner": {"decision": scan.decision, "reason": scan.reason},
|
||||
},
|
||||
)
|
||||
await refresh_skills_system_prompt_cache_async()
|
||||
return await get_custom_skill(skill_name, config)
|
||||
return await get_custom_skill(skill_name)
|
||||
except HTTPException:
|
||||
raise
|
||||
except FileNotFoundError as e:
|
||||
@@ -189,22 +196,24 @@ async def update_custom_skill(skill_name: str, request: CustomSkillUpdateRequest
|
||||
|
||||
|
||||
@router.delete("/skills/custom/{skill_name}", summary="Delete Custom Skill")
|
||||
async def delete_custom_skill(skill_name: str, config: AppConfig = Depends(get_config)) -> dict[str, bool]:
|
||||
async def delete_custom_skill(skill_name: str) -> dict[str, bool]:
|
||||
try:
|
||||
skill_name = skill_name.replace("\r\n", "").replace("\n", "")
|
||||
storage = get_or_new_skill_storage(app_config=config)
|
||||
storage.delete_custom_skill(
|
||||
ensure_custom_skill_is_editable(skill_name)
|
||||
skill_dir = get_custom_skill_dir(skill_name)
|
||||
prev_content = read_custom_skill_content(skill_name)
|
||||
append_history(
|
||||
skill_name,
|
||||
history_meta={
|
||||
{
|
||||
"action": "human_delete",
|
||||
"author": "human",
|
||||
"thread_id": None,
|
||||
"file_path": SKILL_MD_FILE,
|
||||
"prev_content": None,
|
||||
"file_path": "SKILL.md",
|
||||
"prev_content": prev_content,
|
||||
"new_content": None,
|
||||
"scanner": {"decision": "allow", "reason": "Deletion requested."},
|
||||
},
|
||||
)
|
||||
shutil.rmtree(skill_dir)
|
||||
await refresh_skills_system_prompt_cache_async()
|
||||
return {"success": True}
|
||||
except FileNotFoundError as e:
|
||||
@@ -217,13 +226,11 @@ async def delete_custom_skill(skill_name: str, config: AppConfig = Depends(get_c
|
||||
|
||||
|
||||
@router.get("/skills/custom/{skill_name}/history", response_model=CustomSkillHistoryResponse, summary="Get Custom Skill History")
|
||||
async def get_custom_skill_history(skill_name: str, config: AppConfig = Depends(get_config)) -> CustomSkillHistoryResponse:
|
||||
async def get_custom_skill_history(skill_name: str) -> CustomSkillHistoryResponse:
|
||||
try:
|
||||
skill_name = skill_name.replace("\r\n", "").replace("\n", "")
|
||||
storage = get_or_new_skill_storage(app_config=config)
|
||||
if not storage.custom_skill_exists(skill_name) and not storage.get_skill_history_file(skill_name).exists():
|
||||
if not custom_skill_exists(skill_name) and not get_skill_history_file(skill_name).exists():
|
||||
raise HTTPException(status_code=404, detail=f"Custom skill '{skill_name}' not found")
|
||||
return CustomSkillHistoryResponse(history=storage.read_history(skill_name))
|
||||
return CustomSkillHistoryResponse(history=read_history(skill_name))
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
@@ -232,39 +239,38 @@ async def get_custom_skill_history(skill_name: str, config: AppConfig = Depends(
|
||||
|
||||
|
||||
@router.post("/skills/custom/{skill_name}/rollback", response_model=CustomSkillContentResponse, summary="Rollback Custom Skill")
|
||||
async def rollback_custom_skill(skill_name: str, request: SkillRollbackRequest, config: AppConfig = Depends(get_config)) -> CustomSkillContentResponse:
|
||||
async def rollback_custom_skill(skill_name: str, request: SkillRollbackRequest) -> CustomSkillContentResponse:
|
||||
try:
|
||||
storage = get_or_new_skill_storage(app_config=config)
|
||||
if not storage.custom_skill_exists(skill_name) and not storage.get_skill_history_file(skill_name).exists():
|
||||
if not custom_skill_exists(skill_name) and not get_skill_history_file(skill_name).exists():
|
||||
raise HTTPException(status_code=404, detail=f"Custom skill '{skill_name}' not found")
|
||||
history = storage.read_history(skill_name)
|
||||
history = read_history(skill_name)
|
||||
if not history:
|
||||
raise HTTPException(status_code=400, detail=f"Custom skill '{skill_name}' has no history")
|
||||
record = history[request.history_index]
|
||||
target_content = record.get("prev_content")
|
||||
if target_content is None:
|
||||
raise HTTPException(status_code=400, detail="Selected history entry has no previous content to roll back to")
|
||||
storage.validate_skill_markdown_content(skill_name, target_content)
|
||||
scan = await scan_skill_content(target_content, executable=False, location=f"{skill_name}/{SKILL_MD_FILE}", app_config=config)
|
||||
skill_file = storage.get_custom_skill_file(skill_name)
|
||||
validate_skill_markdown_content(skill_name, target_content)
|
||||
scan = await scan_skill_content(target_content, executable=False, location=f"{skill_name}/SKILL.md")
|
||||
skill_file = get_custom_skill_file(skill_name)
|
||||
current_content = skill_file.read_text(encoding="utf-8") if skill_file.exists() else None
|
||||
history_entry = {
|
||||
"action": "rollback",
|
||||
"author": "human",
|
||||
"thread_id": None,
|
||||
"file_path": SKILL_MD_FILE,
|
||||
"file_path": "SKILL.md",
|
||||
"prev_content": current_content,
|
||||
"new_content": target_content,
|
||||
"rollback_from_ts": record.get("ts"),
|
||||
"scanner": {"decision": scan.decision, "reason": scan.reason},
|
||||
}
|
||||
if scan.decision == "block":
|
||||
storage.append_history(skill_name, history_entry)
|
||||
append_history(skill_name, history_entry)
|
||||
raise HTTPException(status_code=400, detail=f"Rollback blocked by security scanner: {scan.reason}")
|
||||
storage.write_custom_skill(skill_name, SKILL_MD_FILE, target_content)
|
||||
storage.append_history(skill_name, history_entry)
|
||||
atomic_write(skill_file, target_content)
|
||||
append_history(skill_name, history_entry)
|
||||
await refresh_skills_system_prompt_cache_async()
|
||||
return await get_custom_skill(skill_name, config)
|
||||
return await get_custom_skill(skill_name)
|
||||
except HTTPException:
|
||||
raise
|
||||
except IndexError:
|
||||
@@ -284,10 +290,9 @@ async def rollback_custom_skill(skill_name: str, request: SkillRollbackRequest,
|
||||
summary="Get Skill Details",
|
||||
description="Retrieve detailed information about a specific skill by its name.",
|
||||
)
|
||||
async def get_skill(skill_name: str, config: AppConfig = Depends(get_config)) -> SkillResponse:
|
||||
async def get_skill(skill_name: str) -> SkillResponse:
|
||||
try:
|
||||
skill_name = skill_name.replace("\r\n", "").replace("\n", "")
|
||||
skills = get_or_new_skill_storage(app_config=config).load_skills(enabled_only=False)
|
||||
skills = load_skills(enabled_only=False)
|
||||
skill = next((s for s in skills if s.name == skill_name), None)
|
||||
|
||||
if skill is None:
|
||||
@@ -307,10 +312,9 @@ async def get_skill(skill_name: str, config: AppConfig = Depends(get_config)) ->
|
||||
summary="Update Skill",
|
||||
description="Update a skill's enabled status by modifying the extensions_config.json file.",
|
||||
)
|
||||
async def update_skill(skill_name: str, request: SkillUpdateRequest, config: AppConfig = Depends(get_config)) -> SkillResponse:
|
||||
async def update_skill(skill_name: str, request: SkillUpdateRequest) -> SkillResponse:
|
||||
try:
|
||||
skill_name = skill_name.replace("\r\n", "").replace("\n", "")
|
||||
skills = get_or_new_skill_storage(app_config=config).load_skills(enabled_only=False)
|
||||
skills = load_skills(enabled_only=False)
|
||||
skill = next((s for s in skills if s.name == skill_name), None)
|
||||
|
||||
if skill is None:
|
||||
@@ -336,7 +340,7 @@ async def update_skill(skill_name: str, request: SkillUpdateRequest, config: App
|
||||
reload_extensions_config()
|
||||
await refresh_skills_system_prompt_cache_async()
|
||||
|
||||
skills = get_or_new_skill_storage(app_config=config).load_skills(enabled_only=False)
|
||||
skills = load_skills(enabled_only=False)
|
||||
updated_skill = next((s for s in skills if s.name == skill_name), None)
|
||||
|
||||
if updated_skill is None:
|
||||
|
||||
@@ -1,14 +1,10 @@
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
|
||||
from fastapi import APIRouter, Depends, Request
|
||||
from fastapi import APIRouter
|
||||
from langchain_core.messages import HumanMessage, SystemMessage
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.gateway.authz import require_permission
|
||||
from app.gateway.deps import get_config
|
||||
from deerflow.config.app_config import AppConfig
|
||||
from deerflow.models import create_chat_model
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -31,31 +27,6 @@ class SuggestionsResponse(BaseModel):
|
||||
suggestions: list[str] = Field(default_factory=list, description="Suggested follow-up questions")
|
||||
|
||||
|
||||
# Matches a complete <think>...</think> block (case-insensitive, spans newlines).
|
||||
_THINK_BLOCK_RE = re.compile(r"<think\b[^>]*>.*?</think\s*>", re.IGNORECASE | re.DOTALL)
|
||||
# Matches a dangling, unclosed <think> (model truncated at max_tokens mid-thought).
|
||||
_OPEN_THINK_RE = re.compile(r"<think\b[^>]*>", re.IGNORECASE)
|
||||
|
||||
|
||||
def _strip_think_blocks(text: str) -> str:
|
||||
"""Remove reasoning-model ``<think>...</think>`` blocks from the response.
|
||||
|
||||
Reasoning models such as MiniMax-M3 inline their chain-of-thought into the
|
||||
message ``content`` wrapped in ``<think>...</think>`` (``reasoning_split``
|
||||
defaults to false), rather than exposing a separate ``reasoning_content``
|
||||
field. The thinking text frequently contains ``[`` / ``]`` characters, which
|
||||
corrupted the downstream ``find('[')`` / ``rfind(']')`` JSON extraction and
|
||||
produced empty suggestions. We strip the reasoning before parsing so only
|
||||
the actual answer remains.
|
||||
"""
|
||||
text = _THINK_BLOCK_RE.sub("", text)
|
||||
# Drop any unclosed <think> (and everything after it) left by truncation.
|
||||
open_match = _OPEN_THINK_RE.search(text)
|
||||
if open_match:
|
||||
text = text[: open_match.start()]
|
||||
return text.strip()
|
||||
|
||||
|
||||
def _strip_markdown_code_fence(text: str) -> str:
|
||||
stripped = text.strip()
|
||||
if not stripped.startswith("```"):
|
||||
@@ -67,8 +38,7 @@ def _strip_markdown_code_fence(text: str) -> str:
|
||||
|
||||
|
||||
def _parse_json_string_list(text: str) -> list[str] | None:
|
||||
candidate = _strip_think_blocks(text)
|
||||
candidate = _strip_markdown_code_fence(candidate)
|
||||
candidate = _strip_markdown_code_fence(text)
|
||||
start = candidate.find("[")
|
||||
end = candidate.rfind("]")
|
||||
if start == -1 or end == -1 or end <= start:
|
||||
@@ -128,18 +98,12 @@ def _format_conversation(messages: list[SuggestionMessage]) -> str:
|
||||
summary="Generate Follow-up Questions",
|
||||
description="Generate short follow-up questions a user might ask next, based on recent conversation context.",
|
||||
)
|
||||
@require_permission("threads", "read", owner_check=True)
|
||||
async def generate_suggestions(
|
||||
thread_id: str,
|
||||
body: SuggestionsRequest,
|
||||
request: Request,
|
||||
config: AppConfig = Depends(get_config),
|
||||
) -> SuggestionsResponse:
|
||||
if not body.messages:
|
||||
async def generate_suggestions(thread_id: str, request: SuggestionsRequest) -> SuggestionsResponse:
|
||||
if not request.messages:
|
||||
return SuggestionsResponse(suggestions=[])
|
||||
|
||||
n = body.n
|
||||
conversation = _format_conversation(body.messages)
|
||||
n = request.n
|
||||
conversation = _format_conversation(request.messages)
|
||||
if not conversation:
|
||||
return SuggestionsResponse(suggestions=[])
|
||||
|
||||
@@ -156,8 +120,8 @@ async def generate_suggestions(
|
||||
user_content = f"Conversation Context:\n{conversation}\n\nGenerate {n} follow-up questions"
|
||||
|
||||
try:
|
||||
model = create_chat_model(name=body.model_name, thinking_enabled=False, app_config=config)
|
||||
response = await model.ainvoke([SystemMessage(content=system_instruction), HumanMessage(content=user_content)], config={"run_name": "suggest_agent"})
|
||||
model = create_chat_model(name=request.model_name, thinking_enabled=False)
|
||||
response = await model.ainvoke([SystemMessage(content=system_instruction), HumanMessage(content=user_content)])
|
||||
raw = _extract_response_text(response.content)
|
||||
suggestions = _parse_json_string_list(raw) or []
|
||||
cleaned = [s.replace("\n", " ").strip() for s in suggestions if s.strip()]
|
||||
|
||||
@@ -19,11 +19,9 @@ from fastapi import APIRouter, HTTPException, Query, Request
|
||||
from fastapi.responses import Response, StreamingResponse
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.gateway.authz import require_permission
|
||||
from app.gateway.deps import get_checkpointer, get_current_user, get_feedback_repo, get_run_event_store, get_run_manager, get_run_store, get_stream_bridge
|
||||
from app.gateway.pagination import trim_run_message_page
|
||||
from app.gateway.services import sse_consumer, start_run, wait_for_run_completion
|
||||
from deerflow.runtime import RunRecord, RunStatus, serialize_channel_values
|
||||
from app.gateway.deps import get_checkpointer, get_run_manager, get_stream_bridge
|
||||
from app.gateway.services import sse_consumer, start_run
|
||||
from deerflow.runtime import RunRecord, serialize_channel_values
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/api/threads", tags=["runs"])
|
||||
@@ -67,35 +65,6 @@ class RunResponse(BaseModel):
|
||||
multitask_strategy: str = "reject"
|
||||
created_at: str = ""
|
||||
updated_at: str = ""
|
||||
total_input_tokens: int = 0
|
||||
total_output_tokens: int = 0
|
||||
total_tokens: int = 0
|
||||
llm_call_count: int = 0
|
||||
lead_agent_tokens: int = 0
|
||||
subagent_tokens: int = 0
|
||||
middleware_tokens: int = 0
|
||||
message_count: int = 0
|
||||
|
||||
|
||||
class ThreadTokenUsageModelBreakdown(BaseModel):
|
||||
tokens: int = 0
|
||||
runs: int = 0
|
||||
|
||||
|
||||
class ThreadTokenUsageCallerBreakdown(BaseModel):
|
||||
lead_agent: int = 0
|
||||
subagent: int = 0
|
||||
middleware: int = 0
|
||||
|
||||
|
||||
class ThreadTokenUsageResponse(BaseModel):
|
||||
thread_id: str
|
||||
total_tokens: int = 0
|
||||
total_input_tokens: int = 0
|
||||
total_output_tokens: int = 0
|
||||
total_runs: int = 0
|
||||
by_model: dict[str, ThreadTokenUsageModelBreakdown] = Field(default_factory=dict)
|
||||
by_caller: ThreadTokenUsageCallerBreakdown = Field(default_factory=ThreadTokenUsageCallerBreakdown)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -103,12 +72,6 @@ class ThreadTokenUsageResponse(BaseModel):
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _cancel_conflict_detail(run_id: str, record: RunRecord) -> str:
|
||||
if record.status in (RunStatus.pending, RunStatus.running):
|
||||
return f"Run {run_id} is not active on this worker and cannot be cancelled"
|
||||
return f"Run {run_id} is not cancellable (status: {record.status.value})"
|
||||
|
||||
|
||||
def _record_to_response(record: RunRecord) -> RunResponse:
|
||||
return RunResponse(
|
||||
run_id=record.run_id,
|
||||
@@ -120,14 +83,6 @@ def _record_to_response(record: RunRecord) -> RunResponse:
|
||||
multitask_strategy=record.multitask_strategy,
|
||||
created_at=record.created_at,
|
||||
updated_at=record.updated_at,
|
||||
total_input_tokens=record.total_input_tokens,
|
||||
total_output_tokens=record.total_output_tokens,
|
||||
total_tokens=record.total_tokens,
|
||||
llm_call_count=record.llm_call_count,
|
||||
lead_agent_tokens=record.lead_agent_tokens,
|
||||
subagent_tokens=record.subagent_tokens,
|
||||
middleware_tokens=record.middleware_tokens,
|
||||
message_count=record.message_count,
|
||||
)
|
||||
|
||||
|
||||
@@ -137,7 +92,6 @@ def _record_to_response(record: RunRecord) -> RunResponse:
|
||||
|
||||
|
||||
@router.post("/{thread_id}/runs", response_model=RunResponse)
|
||||
@require_permission("runs", "create", owner_check=True, require_existing=True)
|
||||
async def create_run(thread_id: str, body: RunCreateRequest, request: Request) -> RunResponse:
|
||||
"""Create a background run (returns immediately)."""
|
||||
record = await start_run(body, thread_id, request)
|
||||
@@ -145,7 +99,6 @@ async def create_run(thread_id: str, body: RunCreateRequest, request: Request) -
|
||||
|
||||
|
||||
@router.post("/{thread_id}/runs/stream")
|
||||
@require_permission("runs", "create", owner_check=True, require_existing=True)
|
||||
async def stream_run(thread_id: str, body: RunCreateRequest, request: Request) -> StreamingResponse:
|
||||
"""Create a run and stream events via SSE.
|
||||
|
||||
@@ -173,18 +126,16 @@ async def stream_run(thread_id: str, body: RunCreateRequest, request: Request) -
|
||||
|
||||
|
||||
@router.post("/{thread_id}/runs/wait", response_model=dict)
|
||||
@require_permission("runs", "create", owner_check=True, require_existing=True)
|
||||
async def wait_run(thread_id: str, body: RunCreateRequest, request: Request) -> dict:
|
||||
"""Create a run and block until it completes, returning the final state."""
|
||||
bridge = get_stream_bridge(request)
|
||||
run_mgr = get_run_manager(request)
|
||||
record = await start_run(body, thread_id, request)
|
||||
|
||||
completed = True
|
||||
if record.task is not None:
|
||||
completed = await wait_for_run_completion(bridge, record, request, run_mgr)
|
||||
try:
|
||||
await record.task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
|
||||
if completed:
|
||||
checkpointer = get_checkpointer(request)
|
||||
config = {"configurable": {"thread_id": thread_id}}
|
||||
try:
|
||||
@@ -200,29 +151,24 @@ async def wait_run(thread_id: str, body: RunCreateRequest, request: Request) ->
|
||||
|
||||
|
||||
@router.get("/{thread_id}/runs", response_model=list[RunResponse])
|
||||
@require_permission("runs", "read", owner_check=True)
|
||||
async def list_runs(thread_id: str, request: Request) -> list[RunResponse]:
|
||||
"""List all runs for a thread."""
|
||||
run_mgr = get_run_manager(request)
|
||||
user_id = await get_current_user(request)
|
||||
records = await run_mgr.list_by_thread(thread_id, user_id=user_id)
|
||||
records = await run_mgr.list_by_thread(thread_id)
|
||||
return [_record_to_response(r) for r in records]
|
||||
|
||||
|
||||
@router.get("/{thread_id}/runs/{run_id}", response_model=RunResponse)
|
||||
@require_permission("runs", "read", owner_check=True)
|
||||
async def get_run(thread_id: str, run_id: str, request: Request) -> RunResponse:
|
||||
"""Get details of a specific run."""
|
||||
run_mgr = get_run_manager(request)
|
||||
user_id = await get_current_user(request)
|
||||
record = await run_mgr.get(run_id, user_id=user_id)
|
||||
record = run_mgr.get(run_id)
|
||||
if record is None or record.thread_id != thread_id:
|
||||
raise HTTPException(status_code=404, detail=f"Run {run_id} not found")
|
||||
return _record_to_response(record)
|
||||
|
||||
|
||||
@router.post("/{thread_id}/runs/{run_id}/cancel")
|
||||
@require_permission("runs", "cancel", owner_check=True, require_existing=True)
|
||||
async def cancel_run(
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
@@ -238,13 +184,16 @@ async def cancel_run(
|
||||
- wait=false: Return immediately with 202
|
||||
"""
|
||||
run_mgr = get_run_manager(request)
|
||||
record = await run_mgr.get(run_id)
|
||||
record = run_mgr.get(run_id)
|
||||
if record is None or record.thread_id != thread_id:
|
||||
raise HTTPException(status_code=404, detail=f"Run {run_id} not found")
|
||||
|
||||
cancelled = await run_mgr.cancel(run_id, action=action)
|
||||
if not cancelled:
|
||||
raise HTTPException(status_code=409, detail=_cancel_conflict_detail(run_id, record))
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail=f"Run {run_id} is not cancellable (status: {record.status.value})",
|
||||
)
|
||||
|
||||
if wait and record.task is not None:
|
||||
try:
|
||||
@@ -257,17 +206,14 @@ async def cancel_run(
|
||||
|
||||
|
||||
@router.get("/{thread_id}/runs/{run_id}/join")
|
||||
@require_permission("runs", "read", owner_check=True)
|
||||
async def join_run(thread_id: str, run_id: str, request: Request) -> StreamingResponse:
|
||||
"""Join an existing run's SSE stream."""
|
||||
bridge = get_stream_bridge(request)
|
||||
run_mgr = get_run_manager(request)
|
||||
record = await run_mgr.get(run_id)
|
||||
record = run_mgr.get(run_id)
|
||||
if record is None or record.thread_id != thread_id:
|
||||
raise HTTPException(status_code=404, detail=f"Run {run_id} not found")
|
||||
if record.store_only:
|
||||
raise HTTPException(status_code=409, detail=f"Run {run_id} is not active on this worker and cannot be streamed")
|
||||
|
||||
bridge = get_stream_bridge(request)
|
||||
return StreamingResponse(
|
||||
sse_consumer(bridge, record, request, run_mgr),
|
||||
media_type="text/event-stream",
|
||||
@@ -279,13 +225,7 @@ async def join_run(thread_id: str, run_id: str, request: Request) -> StreamingRe
|
||||
)
|
||||
|
||||
|
||||
# Register GET and POST as separate routes so each method gets a unique OpenAPI
|
||||
# operationId. ``api_route(methods=["GET", "POST"])`` shares one route registration
|
||||
# across both methods, which makes FastAPI emit the same ``operationId`` twice and
|
||||
# warn about a duplicate operation id during OpenAPI generation.
|
||||
@router.get("/{thread_id}/runs/{run_id}/stream", response_model=None)
|
||||
@router.post("/{thread_id}/runs/{run_id}/stream", response_model=None)
|
||||
@require_permission("runs", "read", owner_check=True)
|
||||
@router.api_route("/{thread_id}/runs/{run_id}/stream", methods=["GET", "POST"], response_model=None)
|
||||
async def stream_existing_run(
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
@@ -301,18 +241,14 @@ async def stream_existing_run(
|
||||
remaining buffered events so the client observes a clean shutdown.
|
||||
"""
|
||||
run_mgr = get_run_manager(request)
|
||||
record = await run_mgr.get(run_id)
|
||||
record = run_mgr.get(run_id)
|
||||
if record is None or record.thread_id != thread_id:
|
||||
raise HTTPException(status_code=404, detail=f"Run {run_id} not found")
|
||||
if record.store_only and action is None:
|
||||
raise HTTPException(status_code=409, detail=f"Run {run_id} is not active on this worker and cannot be streamed")
|
||||
|
||||
# Cancel if an action was requested (stop-button / interrupt flow)
|
||||
if action is not None:
|
||||
cancelled = await run_mgr.cancel(run_id, action=action)
|
||||
if not cancelled:
|
||||
raise HTTPException(status_code=409, detail=_cancel_conflict_detail(run_id, record))
|
||||
if wait and record.task is not None:
|
||||
if cancelled and wait and record.task is not None:
|
||||
try:
|
||||
await record.task
|
||||
except (asyncio.CancelledError, Exception):
|
||||
@@ -329,110 +265,3 @@ async def stream_existing_run(
|
||||
"X-Accel-Buffering": "no",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Messages / Events / Token usage endpoints
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@router.get("/{thread_id}/messages")
|
||||
@require_permission("runs", "read", owner_check=True)
|
||||
async def list_thread_messages(
|
||||
thread_id: str,
|
||||
request: Request,
|
||||
limit: int = Query(default=50, le=200),
|
||||
before_seq: int | None = Query(default=None),
|
||||
after_seq: int | None = Query(default=None),
|
||||
) -> list[dict]:
|
||||
"""Return displayable messages for a thread (across all runs), with feedback attached."""
|
||||
event_store = get_run_event_store(request)
|
||||
messages = await event_store.list_messages(thread_id, limit=limit, before_seq=before_seq, after_seq=after_seq)
|
||||
|
||||
# Attach feedback to the last AI message of each run
|
||||
feedback_repo = get_feedback_repo(request)
|
||||
user_id = await get_current_user(request)
|
||||
feedback_map = await feedback_repo.list_by_thread_grouped(thread_id, user_id=user_id)
|
||||
|
||||
# Find the last ai_message per run_id
|
||||
last_ai_per_run: dict[str, int] = {} # run_id -> index in messages list
|
||||
for i, msg in enumerate(messages):
|
||||
if msg.get("event_type") == "ai_message":
|
||||
last_ai_per_run[msg["run_id"]] = i
|
||||
|
||||
# Attach feedback field
|
||||
last_ai_indices = set(last_ai_per_run.values())
|
||||
for i, msg in enumerate(messages):
|
||||
if i in last_ai_indices:
|
||||
run_id = msg["run_id"]
|
||||
fb = feedback_map.get(run_id)
|
||||
msg["feedback"] = (
|
||||
{
|
||||
"feedback_id": fb["feedback_id"],
|
||||
"rating": fb["rating"],
|
||||
"comment": fb.get("comment"),
|
||||
}
|
||||
if fb
|
||||
else None
|
||||
)
|
||||
else:
|
||||
msg["feedback"] = None
|
||||
|
||||
return messages
|
||||
|
||||
|
||||
@router.get("/{thread_id}/runs/{run_id}/messages")
|
||||
@require_permission("runs", "read", owner_check=True)
|
||||
async def list_run_messages(
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
request: Request,
|
||||
limit: int = Query(default=50, le=200, ge=1),
|
||||
before_seq: int | None = Query(default=None),
|
||||
after_seq: int | None = Query(default=None),
|
||||
) -> dict:
|
||||
"""Return paginated messages for a specific run.
|
||||
|
||||
Response: { data: [...], has_more: bool }
|
||||
"""
|
||||
event_store = get_run_event_store(request)
|
||||
rows = await event_store.list_messages_by_run(
|
||||
thread_id,
|
||||
run_id,
|
||||
limit=limit + 1,
|
||||
before_seq=before_seq,
|
||||
after_seq=after_seq,
|
||||
)
|
||||
data, has_more = trim_run_message_page(rows, limit=limit, after_seq=after_seq)
|
||||
return {"data": data, "has_more": has_more}
|
||||
|
||||
|
||||
@router.get("/{thread_id}/runs/{run_id}/events")
|
||||
@require_permission("runs", "read", owner_check=True)
|
||||
async def list_run_events(
|
||||
thread_id: str,
|
||||
run_id: str,
|
||||
request: Request,
|
||||
event_types: str | None = Query(default=None),
|
||||
limit: int = Query(default=500, le=2000),
|
||||
) -> list[dict]:
|
||||
"""Return the full event stream for a run (debug/audit)."""
|
||||
event_store = get_run_event_store(request)
|
||||
types = event_types.split(",") if event_types else None
|
||||
return await event_store.list_events(thread_id, run_id, event_types=types, limit=limit)
|
||||
|
||||
|
||||
@router.get("/{thread_id}/token-usage", response_model=ThreadTokenUsageResponse)
|
||||
@require_permission("threads", "read", owner_check=True)
|
||||
async def thread_token_usage(
|
||||
thread_id: str,
|
||||
request: Request,
|
||||
include_active: bool = Query(default=False, description="Include running run progress snapshots"),
|
||||
) -> ThreadTokenUsageResponse:
|
||||
"""Thread-level token usage aggregation."""
|
||||
run_store = get_run_store(request)
|
||||
if include_active:
|
||||
agg = await run_store.aggregate_tokens_by_thread(thread_id, include_active=True)
|
||||
else:
|
||||
agg = await run_store.aggregate_tokens_by_thread(thread_id)
|
||||
return ThreadTokenUsageResponse(thread_id=thread_id, **agg)
|
||||
|
||||
@@ -13,41 +13,28 @@ matching the LangGraph Platform wire format expected by the
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import time
|
||||
import uuid
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Request
|
||||
from langgraph.checkpoint.base import empty_checkpoint, uuid6
|
||||
from pydantic import BaseModel, Field, field_validator
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.gateway.authz import require_permission
|
||||
from app.gateway.deps import get_checkpointer
|
||||
from app.gateway.utils import sanitize_log_param
|
||||
from app.gateway.deps import get_checkpointer, get_store
|
||||
from deerflow.config.paths import Paths, get_paths
|
||||
from deerflow.runtime import serialize_channel_values
|
||||
from deerflow.runtime.user_context import get_effective_user_id
|
||||
from deerflow.utils.time import coerce_iso, now_iso
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Store namespace
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
THREADS_NS: tuple[str, ...] = ("threads",)
|
||||
"""Namespace used by the Store for thread metadata records."""
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/api/threads", tags=["threads"])
|
||||
|
||||
|
||||
# Metadata keys that the server controls; clients are not allowed to set
|
||||
# them. Pydantic ``@field_validator("metadata")`` strips them on every
|
||||
# inbound model below so a malicious client cannot reflect a forged
|
||||
# owner identity through the API surface. Defense-in-depth — the
|
||||
# row-level invariant is still ``threads_meta.user_id`` populated from
|
||||
# the auth contextvar; this list closes the metadata-blob echo gap.
|
||||
_SERVER_RESERVED_METADATA_KEYS: frozenset[str] = frozenset({"owner_id", "user_id"})
|
||||
|
||||
|
||||
def _strip_reserved_metadata(metadata: dict[str, Any] | None) -> dict[str, Any]:
|
||||
"""Return ``metadata`` with server-controlled keys removed."""
|
||||
if not metadata:
|
||||
return metadata or {}
|
||||
return {k: v for k, v in metadata.items() if k not in _SERVER_RESERVED_METADATA_KEYS}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Response / request models
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -76,11 +63,8 @@ class ThreadCreateRequest(BaseModel):
|
||||
"""Request body for creating a thread."""
|
||||
|
||||
thread_id: str | None = Field(default=None, description="Optional thread ID (auto-generated if omitted)")
|
||||
assistant_id: str | None = Field(default=None, description="Associate thread with an assistant")
|
||||
metadata: dict[str, Any] = Field(default_factory=dict, description="Initial metadata")
|
||||
|
||||
_strip_reserved = field_validator("metadata")(classmethod(lambda cls, v: _strip_reserved_metadata(v)))
|
||||
|
||||
|
||||
class ThreadSearchRequest(BaseModel):
|
||||
"""Request body for searching threads."""
|
||||
@@ -90,28 +74,6 @@ class ThreadSearchRequest(BaseModel):
|
||||
offset: int = Field(default=0, ge=0, description="Pagination offset")
|
||||
status: str | None = Field(default=None, description="Filter by thread status")
|
||||
|
||||
@field_validator("metadata")
|
||||
@classmethod
|
||||
def _validate_metadata_filters(cls, v: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Reject filter entries the SQL backend cannot compile.
|
||||
|
||||
Enforces consistent behaviour across SQL and memory backends.
|
||||
See ``deerflow.persistence.json_compat`` for the shared validators.
|
||||
"""
|
||||
if not v:
|
||||
return v
|
||||
from deerflow.persistence.json_compat import validate_metadata_filter_key, validate_metadata_filter_value
|
||||
|
||||
bad_entries: list[str] = []
|
||||
for key, value in v.items():
|
||||
if not validate_metadata_filter_key(key):
|
||||
bad_entries.append(f"{key!r} (unsafe key)")
|
||||
elif not validate_metadata_filter_value(value):
|
||||
bad_entries.append(f"{key!r} (unsupported value type {type(value).__name__})")
|
||||
if bad_entries:
|
||||
raise ValueError(f"Invalid metadata filter entries: {', '.join(bad_entries)}")
|
||||
return v
|
||||
|
||||
|
||||
class ThreadStateResponse(BaseModel):
|
||||
"""Response model for thread state."""
|
||||
@@ -131,8 +93,6 @@ class ThreadPatchRequest(BaseModel):
|
||||
|
||||
metadata: dict[str, Any] = Field(default_factory=dict, description="Metadata to merge")
|
||||
|
||||
_strip_reserved = field_validator("metadata")(classmethod(lambda cls, v: _strip_reserved_metadata(v)))
|
||||
|
||||
|
||||
class ThreadStateUpdateRequest(BaseModel):
|
||||
"""Request body for updating thread state (human-in-the-loop resume)."""
|
||||
@@ -166,25 +126,70 @@ class ThreadHistoryRequest(BaseModel):
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _delete_thread_data(thread_id: str, paths: Paths | None = None, *, user_id: str | None = None) -> ThreadDeleteResponse:
|
||||
def _delete_thread_data(thread_id: str, paths: Paths | None = None) -> ThreadDeleteResponse:
|
||||
"""Delete local persisted filesystem data for a thread."""
|
||||
path_manager = paths or get_paths()
|
||||
try:
|
||||
path_manager.delete_thread_dir(thread_id, user_id=user_id)
|
||||
path_manager.delete_thread_dir(thread_id)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=422, detail=str(exc)) from exc
|
||||
except FileNotFoundError:
|
||||
# Not critical — thread data may not exist on disk
|
||||
logger.debug("No local thread data to delete for %s", sanitize_log_param(thread_id))
|
||||
logger.debug("No local thread data to delete for %s", thread_id)
|
||||
return ThreadDeleteResponse(success=True, message=f"No local data for {thread_id}")
|
||||
except Exception as exc:
|
||||
logger.exception("Failed to delete thread data for %s", sanitize_log_param(thread_id))
|
||||
logger.exception("Failed to delete thread data for %s", thread_id)
|
||||
raise HTTPException(status_code=500, detail="Failed to delete local thread data.") from exc
|
||||
|
||||
logger.info("Deleted local thread data for %s", sanitize_log_param(thread_id))
|
||||
logger.info("Deleted local thread data for %s", thread_id)
|
||||
return ThreadDeleteResponse(success=True, message=f"Deleted local thread data for {thread_id}")
|
||||
|
||||
|
||||
async def _store_get(store, thread_id: str) -> dict | None:
|
||||
"""Fetch a thread record from the Store; returns ``None`` if absent."""
|
||||
item = await store.aget(THREADS_NS, thread_id)
|
||||
return item.value if item is not None else None
|
||||
|
||||
|
||||
async def _store_put(store, record: dict) -> None:
|
||||
"""Write a thread record to the Store."""
|
||||
await store.aput(THREADS_NS, record["thread_id"], record)
|
||||
|
||||
|
||||
async def _store_upsert(store, thread_id: str, *, metadata: dict | None = None, values: dict | None = None) -> None:
|
||||
"""Create or refresh a thread record in the Store.
|
||||
|
||||
On creation the record is written with ``status="idle"``. On update only
|
||||
``updated_at`` (and optionally ``metadata`` / ``values``) are changed so
|
||||
that existing fields are preserved.
|
||||
|
||||
``values`` carries the agent-state snapshot exposed to the frontend
|
||||
(currently just ``{"title": "..."}``).
|
||||
"""
|
||||
now = time.time()
|
||||
existing = await _store_get(store, thread_id)
|
||||
if existing is None:
|
||||
await _store_put(
|
||||
store,
|
||||
{
|
||||
"thread_id": thread_id,
|
||||
"status": "idle",
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
"metadata": metadata or {},
|
||||
"values": values or {},
|
||||
},
|
||||
)
|
||||
else:
|
||||
val = dict(existing)
|
||||
val["updated_at"] = now
|
||||
if metadata:
|
||||
val.setdefault("metadata", {}).update(metadata)
|
||||
if values:
|
||||
val.setdefault("values", {}).update(values)
|
||||
await _store_put(store, val)
|
||||
|
||||
|
||||
def _derive_thread_status(checkpoint_tuple) -> str:
|
||||
"""Derive thread status from checkpoint metadata."""
|
||||
if checkpoint_tuple is None:
|
||||
@@ -210,18 +215,22 @@ def _derive_thread_status(checkpoint_tuple) -> str:
|
||||
|
||||
|
||||
@router.delete("/{thread_id}", response_model=ThreadDeleteResponse)
|
||||
@require_permission("threads", "delete", owner_check=True, require_existing=True)
|
||||
async def delete_thread_data(thread_id: str, request: Request) -> ThreadDeleteResponse:
|
||||
"""Delete local persisted filesystem data for a thread.
|
||||
|
||||
Cleans DeerFlow-managed thread directories, removes checkpoint data,
|
||||
and removes the thread_meta row from the configured ThreadMetaStore
|
||||
(sqlite or memory).
|
||||
and removes the thread record from the Store.
|
||||
"""
|
||||
from app.gateway.deps import get_thread_store
|
||||
|
||||
# Clean local filesystem
|
||||
response = _delete_thread_data(thread_id, user_id=get_effective_user_id())
|
||||
response = _delete_thread_data(thread_id)
|
||||
|
||||
# Remove from Store (best-effort)
|
||||
store = get_store(request)
|
||||
if store is not None:
|
||||
try:
|
||||
await store.adelete(THREADS_NS, thread_id)
|
||||
except Exception:
|
||||
logger.debug("Could not delete store record for thread %s (not critical)", thread_id)
|
||||
|
||||
# Remove checkpoints (best-effort)
|
||||
checkpointer = getattr(request.app.state, "checkpointer", None)
|
||||
@@ -230,15 +239,7 @@ async def delete_thread_data(thread_id: str, request: Request) -> ThreadDeleteRe
|
||||
if hasattr(checkpointer, "adelete_thread"):
|
||||
await checkpointer.adelete_thread(thread_id)
|
||||
except Exception:
|
||||
logger.debug("Could not delete checkpoints for thread %s (not critical)", sanitize_log_param(thread_id))
|
||||
|
||||
# Remove thread_meta row (best-effort) — required for sqlite backend
|
||||
# so the deleted thread no longer appears in /threads/search.
|
||||
try:
|
||||
thread_store = get_thread_store(request)
|
||||
await thread_store.delete(thread_id)
|
||||
except Exception:
|
||||
logger.debug("Could not delete thread_meta for %s (not critical)", sanitize_log_param(thread_id))
|
||||
logger.debug("Could not delete checkpoints for thread %s (not critical)", thread_id)
|
||||
|
||||
return response
|
||||
|
||||
@@ -247,44 +248,49 @@ async def delete_thread_data(thread_id: str, request: Request) -> ThreadDeleteRe
|
||||
async def create_thread(body: ThreadCreateRequest, request: Request) -> ThreadResponse:
|
||||
"""Create a new thread.
|
||||
|
||||
Writes a thread_meta record (so the thread appears in /threads/search)
|
||||
and an empty checkpoint (so state endpoints work immediately).
|
||||
The thread record is written to the Store (for fast listing) and an
|
||||
empty checkpoint is written to the checkpointer (for state reads).
|
||||
Idempotent: returns the existing record when ``thread_id`` already exists.
|
||||
"""
|
||||
from app.gateway.deps import get_thread_store
|
||||
|
||||
store = get_store(request)
|
||||
checkpointer = get_checkpointer(request)
|
||||
thread_store = get_thread_store(request)
|
||||
thread_id = body.thread_id or str(uuid.uuid4())
|
||||
now = now_iso()
|
||||
# ``body.metadata`` is already stripped of server-reserved keys by
|
||||
# ``ThreadCreateRequest._strip_reserved`` — see the model definition.
|
||||
now = time.time()
|
||||
|
||||
# Idempotency: return existing record when already present
|
||||
existing_record = await thread_store.get(thread_id)
|
||||
# Idempotency: return existing record from Store when already present
|
||||
if store is not None:
|
||||
existing_record = await _store_get(store, thread_id)
|
||||
if existing_record is not None:
|
||||
return ThreadResponse(
|
||||
thread_id=thread_id,
|
||||
status=existing_record.get("status", "idle"),
|
||||
created_at=coerce_iso(existing_record.get("created_at", "")),
|
||||
updated_at=coerce_iso(existing_record.get("updated_at", "")),
|
||||
created_at=str(existing_record.get("created_at", "")),
|
||||
updated_at=str(existing_record.get("updated_at", "")),
|
||||
metadata=existing_record.get("metadata", {}),
|
||||
)
|
||||
|
||||
# Write thread_meta so the thread appears in /threads/search immediately
|
||||
# Write thread record to Store
|
||||
if store is not None:
|
||||
try:
|
||||
await thread_store.create(
|
||||
thread_id,
|
||||
assistant_id=getattr(body, "assistant_id", None),
|
||||
metadata=body.metadata,
|
||||
await _store_put(
|
||||
store,
|
||||
{
|
||||
"thread_id": thread_id,
|
||||
"status": "idle",
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
"metadata": body.metadata,
|
||||
},
|
||||
)
|
||||
except Exception:
|
||||
logger.exception("Failed to write thread_meta for %s", sanitize_log_param(thread_id))
|
||||
logger.exception("Failed to write thread %s to store", thread_id)
|
||||
raise HTTPException(status_code=500, detail="Failed to create thread")
|
||||
|
||||
# Write an empty checkpoint so state endpoints work immediately
|
||||
config = {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}}
|
||||
try:
|
||||
from langgraph.checkpoint.base import empty_checkpoint
|
||||
|
||||
ckpt_metadata = {
|
||||
"step": -1,
|
||||
"source": "input",
|
||||
@@ -295,15 +301,15 @@ async def create_thread(body: ThreadCreateRequest, request: Request) -> ThreadRe
|
||||
}
|
||||
await checkpointer.aput(config, empty_checkpoint(), ckpt_metadata, {})
|
||||
except Exception:
|
||||
logger.exception("Failed to create checkpoint for thread %s", sanitize_log_param(thread_id))
|
||||
logger.exception("Failed to create checkpoint for thread %s", thread_id)
|
||||
raise HTTPException(status_code=500, detail="Failed to create thread")
|
||||
|
||||
logger.info("Thread created: %s", sanitize_log_param(thread_id))
|
||||
logger.info("Thread created: %s", thread_id)
|
||||
return ThreadResponse(
|
||||
thread_id=thread_id,
|
||||
status="idle",
|
||||
created_at=now,
|
||||
updated_at=now,
|
||||
created_at=str(now),
|
||||
updated_at=str(now),
|
||||
metadata=body.metadata,
|
||||
)
|
||||
|
||||
@@ -312,105 +318,173 @@ async def create_thread(body: ThreadCreateRequest, request: Request) -> ThreadRe
|
||||
async def search_threads(body: ThreadSearchRequest, request: Request) -> list[ThreadResponse]:
|
||||
"""Search and list threads.
|
||||
|
||||
Delegates to the configured ThreadMetaStore implementation
|
||||
(SQL-backed for sqlite/postgres, Store-backed for memory mode).
|
||||
"""
|
||||
from app.gateway.deps import get_thread_store
|
||||
from deerflow.persistence.thread_meta import InvalidMetadataFilterError
|
||||
Two-phase approach:
|
||||
|
||||
repo = get_thread_store(request)
|
||||
**Phase 1 — Store (fast path, O(threads))**: returns threads that were
|
||||
created or run through this Gateway. Store records are tiny metadata
|
||||
dicts so fetching all of them at once is cheap.
|
||||
|
||||
**Phase 2 — Checkpointer supplement (lazy migration)**: threads that
|
||||
were created directly by LangGraph Server (and therefore absent from the
|
||||
Store) are discovered here by iterating the shared checkpointer. Any
|
||||
newly found thread is immediately written to the Store so that the next
|
||||
search skips Phase 2 for that thread — the Store converges to a full
|
||||
index over time without a one-shot migration job.
|
||||
"""
|
||||
store = get_store(request)
|
||||
checkpointer = get_checkpointer(request)
|
||||
|
||||
# -----------------------------------------------------------------------
|
||||
# Phase 1: Store
|
||||
# -----------------------------------------------------------------------
|
||||
merged: dict[str, ThreadResponse] = {}
|
||||
|
||||
if store is not None:
|
||||
try:
|
||||
rows = await repo.search(
|
||||
metadata=body.metadata or None,
|
||||
status=body.status,
|
||||
limit=body.limit,
|
||||
offset=body.offset,
|
||||
items = await store.asearch(THREADS_NS, limit=10_000)
|
||||
except Exception:
|
||||
logger.warning("Store search failed — falling back to checkpointer only", exc_info=True)
|
||||
items = []
|
||||
|
||||
for item in items:
|
||||
val = item.value
|
||||
merged[val["thread_id"]] = ThreadResponse(
|
||||
thread_id=val["thread_id"],
|
||||
status=val.get("status", "idle"),
|
||||
created_at=str(val.get("created_at", "")),
|
||||
updated_at=str(val.get("updated_at", "")),
|
||||
metadata=val.get("metadata", {}),
|
||||
values=val.get("values", {}),
|
||||
)
|
||||
except InvalidMetadataFilterError as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc)) from exc
|
||||
return [
|
||||
ThreadResponse(
|
||||
thread_id=r["thread_id"],
|
||||
status=r.get("status", "idle"),
|
||||
# ``coerce_iso`` heals legacy unix-second values that
|
||||
# ``MemoryThreadMetaStore`` historically wrote with ``time.time()``;
|
||||
# SQL-backed rows already arrive as ISO strings and pass through.
|
||||
created_at=coerce_iso(r.get("created_at", "")),
|
||||
updated_at=coerce_iso(r.get("updated_at", "")),
|
||||
metadata=r.get("metadata", {}),
|
||||
values={"title": r["display_name"]} if r.get("display_name") else {},
|
||||
interrupts={},
|
||||
|
||||
# -----------------------------------------------------------------------
|
||||
# Phase 2: Checkpointer supplement
|
||||
# Discovers threads not yet in the Store (e.g. created by LangGraph
|
||||
# Server) and lazily migrates them so future searches skip this phase.
|
||||
# -----------------------------------------------------------------------
|
||||
try:
|
||||
async for checkpoint_tuple in checkpointer.alist(None):
|
||||
cfg = getattr(checkpoint_tuple, "config", {})
|
||||
thread_id = cfg.get("configurable", {}).get("thread_id")
|
||||
if not thread_id or thread_id in merged:
|
||||
continue
|
||||
|
||||
# Skip sub-graph checkpoints (checkpoint_ns is non-empty for those)
|
||||
if cfg.get("configurable", {}).get("checkpoint_ns", ""):
|
||||
continue
|
||||
|
||||
ckpt_meta = getattr(checkpoint_tuple, "metadata", {}) or {}
|
||||
# Strip LangGraph internal keys from the user-visible metadata dict
|
||||
user_meta = {k: v for k, v in ckpt_meta.items() if k not in ("created_at", "updated_at", "step", "source", "writes", "parents")}
|
||||
|
||||
# Extract state values (title) from the checkpoint's channel_values
|
||||
checkpoint_data = getattr(checkpoint_tuple, "checkpoint", {}) or {}
|
||||
channel_values = checkpoint_data.get("channel_values", {})
|
||||
ckpt_values = {}
|
||||
if title := channel_values.get("title"):
|
||||
ckpt_values["title"] = title
|
||||
|
||||
thread_resp = ThreadResponse(
|
||||
thread_id=thread_id,
|
||||
status=_derive_thread_status(checkpoint_tuple),
|
||||
created_at=str(ckpt_meta.get("created_at", "")),
|
||||
updated_at=str(ckpt_meta.get("updated_at", ckpt_meta.get("created_at", ""))),
|
||||
metadata=user_meta,
|
||||
values=ckpt_values,
|
||||
)
|
||||
for r in rows
|
||||
]
|
||||
merged[thread_id] = thread_resp
|
||||
|
||||
# Lazy migration — write to Store so the next search finds it there
|
||||
if store is not None:
|
||||
try:
|
||||
await _store_upsert(store, thread_id, metadata=user_meta, values=ckpt_values or None)
|
||||
except Exception:
|
||||
logger.debug("Failed to migrate thread %s to store (non-fatal)", thread_id)
|
||||
except Exception:
|
||||
logger.exception("Checkpointer scan failed during thread search")
|
||||
# Don't raise — return whatever was collected from Store + partial scan
|
||||
|
||||
# -----------------------------------------------------------------------
|
||||
# Phase 3: Filter → sort → paginate
|
||||
# -----------------------------------------------------------------------
|
||||
results = list(merged.values())
|
||||
|
||||
if body.metadata:
|
||||
results = [r for r in results if all(r.metadata.get(k) == v for k, v in body.metadata.items())]
|
||||
|
||||
if body.status:
|
||||
results = [r for r in results if r.status == body.status]
|
||||
|
||||
results.sort(key=lambda r: r.updated_at, reverse=True)
|
||||
return results[body.offset : body.offset + body.limit]
|
||||
|
||||
|
||||
@router.patch("/{thread_id}", response_model=ThreadResponse)
|
||||
@require_permission("threads", "write", owner_check=True, require_existing=True)
|
||||
async def patch_thread(thread_id: str, body: ThreadPatchRequest, request: Request) -> ThreadResponse:
|
||||
"""Merge metadata into a thread record."""
|
||||
from app.gateway.deps import get_thread_store
|
||||
store = get_store(request)
|
||||
if store is None:
|
||||
raise HTTPException(status_code=503, detail="Store not available")
|
||||
|
||||
thread_store = get_thread_store(request)
|
||||
record = await thread_store.get(thread_id)
|
||||
record = await _store_get(store, thread_id)
|
||||
if record is None:
|
||||
raise HTTPException(status_code=404, detail=f"Thread {thread_id} not found")
|
||||
|
||||
# ``body.metadata`` already stripped by ``ThreadPatchRequest._strip_reserved``.
|
||||
now = time.time()
|
||||
updated = dict(record)
|
||||
updated.setdefault("metadata", {}).update(body.metadata)
|
||||
updated["updated_at"] = now
|
||||
|
||||
try:
|
||||
await thread_store.update_metadata(thread_id, body.metadata)
|
||||
await _store_put(store, updated)
|
||||
except Exception:
|
||||
logger.exception("Failed to patch thread %s", sanitize_log_param(thread_id))
|
||||
logger.exception("Failed to patch thread %s", thread_id)
|
||||
raise HTTPException(status_code=500, detail="Failed to update thread")
|
||||
|
||||
# Re-read to get the merged metadata + refreshed updated_at
|
||||
record = await thread_store.get(thread_id) or record
|
||||
return ThreadResponse(
|
||||
thread_id=thread_id,
|
||||
status=record.get("status", "idle"),
|
||||
created_at=coerce_iso(record.get("created_at", "")),
|
||||
updated_at=coerce_iso(record.get("updated_at", "")),
|
||||
metadata=record.get("metadata", {}),
|
||||
status=updated.get("status", "idle"),
|
||||
created_at=str(updated.get("created_at", "")),
|
||||
updated_at=str(now),
|
||||
metadata=updated.get("metadata", {}),
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{thread_id}", response_model=ThreadResponse)
|
||||
@require_permission("threads", "read", owner_check=True)
|
||||
async def get_thread(thread_id: str, request: Request) -> ThreadResponse:
|
||||
"""Get thread info.
|
||||
|
||||
Reads metadata from the ThreadMetaStore and derives the accurate
|
||||
execution status from the checkpointer. Falls back to the checkpointer
|
||||
alone for threads that pre-date ThreadMetaStore adoption (backward compat).
|
||||
Reads metadata from the Store and derives the accurate execution
|
||||
status from the checkpointer. Falls back to the checkpointer alone
|
||||
for threads that pre-date Store adoption (backward compat).
|
||||
"""
|
||||
from app.gateway.deps import get_thread_store
|
||||
|
||||
thread_store = get_thread_store(request)
|
||||
store = get_store(request)
|
||||
checkpointer = get_checkpointer(request)
|
||||
|
||||
record: dict | None = await thread_store.get(thread_id)
|
||||
record: dict | None = None
|
||||
if store is not None:
|
||||
record = await _store_get(store, thread_id)
|
||||
|
||||
# Derive accurate status from the checkpointer
|
||||
config = {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}}
|
||||
try:
|
||||
checkpoint_tuple = await checkpointer.aget_tuple(config)
|
||||
except Exception:
|
||||
logger.exception("Failed to get checkpoint for thread %s", sanitize_log_param(thread_id))
|
||||
logger.exception("Failed to get checkpoint for thread %s", thread_id)
|
||||
raise HTTPException(status_code=500, detail="Failed to get thread")
|
||||
|
||||
if record is None and checkpoint_tuple is None:
|
||||
raise HTTPException(status_code=404, detail=f"Thread {thread_id} not found")
|
||||
|
||||
# If the thread exists in the checkpointer but not in thread_meta (e.g.
|
||||
# legacy data created before thread_meta adoption), synthesize a minimal
|
||||
# record from the checkpoint metadata.
|
||||
# If the thread exists in the checkpointer but not the store (e.g. legacy
|
||||
# data), synthesize a minimal store record from the checkpoint metadata.
|
||||
if record is None and checkpoint_tuple is not None:
|
||||
ckpt_meta = getattr(checkpoint_tuple, "metadata", {}) or {}
|
||||
record = {
|
||||
"thread_id": thread_id,
|
||||
"status": "idle",
|
||||
"created_at": coerce_iso(ckpt_meta.get("created_at", "")),
|
||||
"updated_at": coerce_iso(ckpt_meta.get("updated_at", ckpt_meta.get("created_at", ""))),
|
||||
"created_at": ckpt_meta.get("created_at", ""),
|
||||
"updated_at": ckpt_meta.get("updated_at", ckpt_meta.get("created_at", "")),
|
||||
"metadata": {k: v for k, v in ckpt_meta.items() if k not in ("created_at", "updated_at", "step", "source", "writes", "parents")},
|
||||
}
|
||||
|
||||
@@ -424,16 +498,14 @@ async def get_thread(thread_id: str, request: Request) -> ThreadResponse:
|
||||
return ThreadResponse(
|
||||
thread_id=thread_id,
|
||||
status=status,
|
||||
created_at=coerce_iso(record.get("created_at", "")),
|
||||
updated_at=coerce_iso(record.get("updated_at", "")),
|
||||
created_at=str(record.get("created_at", "")),
|
||||
updated_at=str(record.get("updated_at", "")),
|
||||
metadata=record.get("metadata", {}),
|
||||
values=serialize_channel_values(channel_values),
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@router.get("/{thread_id}/state", response_model=ThreadStateResponse)
|
||||
@require_permission("threads", "read", owner_check=True)
|
||||
async def get_thread_state(thread_id: str, request: Request) -> ThreadStateResponse:
|
||||
"""Get the latest state snapshot for a thread.
|
||||
|
||||
@@ -446,7 +518,7 @@ async def get_thread_state(thread_id: str, request: Request) -> ThreadStateRespo
|
||||
try:
|
||||
checkpoint_tuple = await checkpointer.aget_tuple(config)
|
||||
except Exception:
|
||||
logger.exception("Failed to get state for thread %s", sanitize_log_param(thread_id))
|
||||
logger.exception("Failed to get state for thread %s", thread_id)
|
||||
raise HTTPException(status_code=500, detail="Failed to get thread state")
|
||||
|
||||
if checkpoint_tuple is None:
|
||||
@@ -470,34 +542,28 @@ async def get_thread_state(thread_id: str, request: Request) -> ThreadStateRespo
|
||||
next_tasks = [t.name for t in tasks_raw if hasattr(t, "name")]
|
||||
tasks = [{"id": getattr(t, "id", ""), "name": getattr(t, "name", "")} for t in tasks_raw]
|
||||
|
||||
values = serialize_channel_values(channel_values)
|
||||
|
||||
return ThreadStateResponse(
|
||||
values=values,
|
||||
values=serialize_channel_values(channel_values),
|
||||
next=next_tasks,
|
||||
metadata=metadata,
|
||||
checkpoint={"id": checkpoint_id, "ts": coerce_iso(metadata.get("created_at", ""))},
|
||||
checkpoint={"id": checkpoint_id, "ts": str(metadata.get("created_at", ""))},
|
||||
checkpoint_id=checkpoint_id,
|
||||
parent_checkpoint_id=parent_checkpoint_id,
|
||||
created_at=coerce_iso(metadata.get("created_at", "")),
|
||||
created_at=str(metadata.get("created_at", "")),
|
||||
tasks=tasks,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/{thread_id}/state", response_model=ThreadStateResponse)
|
||||
@require_permission("threads", "write", owner_check=True, require_existing=True)
|
||||
async def update_thread_state(thread_id: str, body: ThreadStateUpdateRequest, request: Request) -> ThreadStateResponse:
|
||||
"""Update thread state (e.g. for human-in-the-loop resume or title rename).
|
||||
|
||||
Writes a new checkpoint that merges *body.values* into the latest
|
||||
channel values, then syncs any updated ``title`` field through the
|
||||
ThreadMetaStore abstraction so that ``/threads/search`` reflects the
|
||||
change immediately in both sqlite and memory backends.
|
||||
channel values, then syncs any updated ``title`` field back to the Store
|
||||
so that ``/threads/search`` reflects the change immediately.
|
||||
"""
|
||||
from app.gateway.deps import get_thread_store
|
||||
|
||||
checkpointer = get_checkpointer(request)
|
||||
thread_store = get_thread_store(request)
|
||||
store = get_store(request)
|
||||
|
||||
# checkpoint_ns must be present in the config for aput — default to ""
|
||||
# (the root graph namespace). checkpoint_id is optional; omitting it
|
||||
@@ -514,7 +580,7 @@ async def update_thread_state(thread_id: str, body: ThreadStateUpdateRequest, re
|
||||
try:
|
||||
checkpoint_tuple = await checkpointer.aget_tuple(read_config)
|
||||
except Exception:
|
||||
logger.exception("Failed to get state for thread %s", sanitize_log_param(thread_id))
|
||||
logger.exception("Failed to get state for thread %s", thread_id)
|
||||
raise HTTPException(status_code=500, detail="Failed to get thread state")
|
||||
|
||||
if checkpoint_tuple is None:
|
||||
@@ -529,28 +595,16 @@ async def update_thread_state(thread_id: str, body: ThreadStateUpdateRequest, re
|
||||
channel_values.update(body.values)
|
||||
|
||||
checkpoint["channel_values"] = channel_values
|
||||
metadata["updated_at"] = now_iso()
|
||||
metadata["updated_at"] = time.time()
|
||||
|
||||
if body.as_node:
|
||||
metadata["source"] = "update"
|
||||
metadata["step"] = metadata.get("step", 0) + 1
|
||||
metadata["writes"] = {body.as_node: body.values}
|
||||
|
||||
# Assign a new checkpoint ID so aput performs an INSERT rather than an
|
||||
# in-place REPLACE of the existing row. Use uuid6 (time-ordered) rather
|
||||
# than uuid4 (random) so the new ID is always lexicographically greater
|
||||
# than the previous one — LangGraph's checkpointers determine the "latest"
|
||||
# checkpoint by max(checkpoint_ids) string order, matching the uuid6 epoch.
|
||||
checkpoint["id"] = str(uuid6())
|
||||
|
||||
# aput requires checkpoint_ns in the config — use the same config used for the
|
||||
# read (which always includes checkpoint_ns=""). The fresh checkpoint ID is
|
||||
# assigned above via checkpoint["id"]; keep checkpoint_id out of the config so
|
||||
# the write is keyed by the new checkpoint payload rather than the prior read.
|
||||
# All supported savers (InMemorySaver, AsyncSqliteSaver, AsyncPostgresSaver)
|
||||
# persist and echo back checkpoint["id"] verbatim — none mint their own — so
|
||||
# the new_config below carries the uuid6 we assigned here. (Regression-locked
|
||||
# by test_update_thread_state_inserts_new_checkpoint_each_call.)
|
||||
# read (which always includes checkpoint_ns=""). Do NOT include checkpoint_id
|
||||
# so that aput generates a fresh checkpoint ID for the new snapshot.
|
||||
write_config: dict[str, Any] = {
|
||||
"configurable": {
|
||||
"thread_id": thread_id,
|
||||
@@ -560,43 +614,32 @@ async def update_thread_state(thread_id: str, body: ThreadStateUpdateRequest, re
|
||||
try:
|
||||
new_config = await checkpointer.aput(write_config, checkpoint, metadata, {})
|
||||
except Exception:
|
||||
logger.exception("Failed to update state for thread %s", sanitize_log_param(thread_id))
|
||||
logger.exception("Failed to update state for thread %s", thread_id)
|
||||
raise HTTPException(status_code=500, detail="Failed to update thread state")
|
||||
|
||||
new_checkpoint_id: str | None = None
|
||||
if isinstance(new_config, dict):
|
||||
new_checkpoint_id = new_config.get("configurable", {}).get("checkpoint_id")
|
||||
|
||||
# Sync title changes through the ThreadMetaStore abstraction so /threads/search
|
||||
# reflects them immediately in both sqlite and memory backends.
|
||||
if thread_store and body.values and "title" in body.values:
|
||||
new_title = body.values["title"]
|
||||
if new_title: # Skip empty strings and None
|
||||
# Sync title changes to the Store so /threads/search reflects them immediately.
|
||||
if store is not None and body.values and "title" in body.values:
|
||||
try:
|
||||
await thread_store.update_display_name(thread_id, new_title)
|
||||
await _store_upsert(store, thread_id, values={"title": body.values["title"]})
|
||||
except Exception:
|
||||
logger.debug("Failed to sync title to thread_meta for %s (non-fatal)", sanitize_log_param(thread_id))
|
||||
logger.debug("Failed to sync title to store for thread %s (non-fatal)", thread_id)
|
||||
|
||||
return ThreadStateResponse(
|
||||
values=serialize_channel_values(channel_values),
|
||||
next=[],
|
||||
metadata=metadata,
|
||||
checkpoint_id=new_checkpoint_id,
|
||||
created_at=coerce_iso(metadata.get("created_at", "")),
|
||||
created_at=str(metadata.get("created_at", "")),
|
||||
)
|
||||
|
||||
|
||||
@router.post("/{thread_id}/history", response_model=list[HistoryEntry])
|
||||
@require_permission("threads", "read", owner_check=True)
|
||||
async def get_thread_history(thread_id: str, body: ThreadHistoryRequest, request: Request) -> list[HistoryEntry]:
|
||||
"""Get checkpoint history for a thread.
|
||||
|
||||
Messages are read from the checkpointer's channel values (the
|
||||
authoritative source) and serialized via
|
||||
:func:`~deerflow.runtime.serialization.serialize_channel_values`.
|
||||
Only the latest (first) checkpoint carries the ``messages`` key to
|
||||
avoid duplicating them across every entry.
|
||||
"""
|
||||
"""Get checkpoint history for a thread."""
|
||||
checkpointer = get_checkpointer(request)
|
||||
|
||||
config: dict[str, Any] = {"configurable": {"thread_id": thread_id}}
|
||||
@@ -604,7 +647,6 @@ async def get_thread_history(thread_id: str, body: ThreadHistoryRequest, request
|
||||
config["configurable"]["checkpoint_id"] = body.before
|
||||
|
||||
entries: list[HistoryEntry] = []
|
||||
is_latest_checkpoint = True
|
||||
try:
|
||||
async for checkpoint_tuple in checkpointer.alist(config, limit=body.limit):
|
||||
ckpt_config = getattr(checkpoint_tuple, "config", {})
|
||||
@@ -619,42 +661,22 @@ async def get_thread_history(thread_id: str, body: ThreadHistoryRequest, request
|
||||
|
||||
channel_values = checkpoint.get("channel_values", {})
|
||||
|
||||
# Build values from checkpoint channel_values
|
||||
values: dict[str, Any] = {}
|
||||
if title := channel_values.get("title"):
|
||||
values["title"] = title
|
||||
if thread_data := channel_values.get("thread_data"):
|
||||
values["thread_data"] = thread_data
|
||||
|
||||
# Attach messages only to the latest checkpoint entry.
|
||||
if is_latest_checkpoint:
|
||||
messages = channel_values.get("messages")
|
||||
if messages:
|
||||
values["messages"] = serialize_channel_values({"messages": messages}).get("messages", [])
|
||||
is_latest_checkpoint = False
|
||||
|
||||
# Derive next tasks
|
||||
tasks_raw = getattr(checkpoint_tuple, "tasks", []) or []
|
||||
next_tasks = [t.name for t in tasks_raw if hasattr(t, "name")]
|
||||
|
||||
# Strip LangGraph internal keys from metadata
|
||||
user_meta = {k: v for k, v in metadata.items() if k not in ("created_at", "updated_at", "step", "source", "writes", "parents")}
|
||||
# Keep step for ordering context
|
||||
if "step" in metadata:
|
||||
user_meta["step"] = metadata["step"]
|
||||
|
||||
entries.append(
|
||||
HistoryEntry(
|
||||
checkpoint_id=checkpoint_id,
|
||||
parent_checkpoint_id=parent_id,
|
||||
metadata=user_meta,
|
||||
values=values,
|
||||
created_at=coerce_iso(metadata.get("created_at", "")),
|
||||
metadata=metadata,
|
||||
values=serialize_channel_values(channel_values),
|
||||
created_at=str(metadata.get("created_at", "")),
|
||||
next=next_tasks,
|
||||
)
|
||||
)
|
||||
except Exception:
|
||||
logger.exception("Failed to get history for thread %s", sanitize_log_param(thread_id))
|
||||
logger.exception("Failed to get history for thread %s", thread_id)
|
||||
raise HTTPException(status_code=500, detail="Failed to get thread history")
|
||||
|
||||
return entries
|
||||
|
||||
@@ -4,26 +4,19 @@ import logging
|
||||
import os
|
||||
import stat
|
||||
|
||||
from fastapi import APIRouter, Depends, File, HTTPException, Request, UploadFile
|
||||
from pydantic import BaseModel, Field
|
||||
from fastapi import APIRouter, File, HTTPException, UploadFile
|
||||
from pydantic import BaseModel
|
||||
|
||||
from app.gateway.authz import require_permission
|
||||
from app.gateway.deps import get_config
|
||||
from deerflow.config.app_config import AppConfig
|
||||
from deerflow.config.paths import get_paths
|
||||
from deerflow.runtime.user_context import get_effective_user_id
|
||||
from deerflow.sandbox.sandbox_provider import SandboxProvider, get_sandbox_provider
|
||||
from deerflow.sandbox.sandbox_provider import get_sandbox_provider
|
||||
from deerflow.uploads.manager import (
|
||||
PathTraversalError,
|
||||
UnsafeUploadPathError,
|
||||
claim_unique_filename,
|
||||
delete_file_safe,
|
||||
enrich_file_listing,
|
||||
ensure_uploads_dir,
|
||||
get_uploads_dir,
|
||||
list_files_in_dir,
|
||||
normalize_filename,
|
||||
open_upload_file_no_symlink,
|
||||
upload_artifact_url,
|
||||
upload_virtual_path,
|
||||
)
|
||||
@@ -33,51 +26,13 @@ logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/threads/{thread_id}/uploads", tags=["uploads"])
|
||||
|
||||
UPLOAD_CHUNK_SIZE = 8192
|
||||
DEFAULT_MAX_FILES = 10
|
||||
DEFAULT_MAX_FILE_SIZE = 50 * 1024 * 1024
|
||||
DEFAULT_MAX_TOTAL_SIZE = 100 * 1024 * 1024
|
||||
|
||||
|
||||
class UploadedFileInfo(BaseModel):
|
||||
"""Uploaded file metadata exposed by upload and list APIs."""
|
||||
|
||||
filename: str
|
||||
size: int
|
||||
path: str
|
||||
virtual_path: str
|
||||
artifact_url: str
|
||||
extension: str | None = None
|
||||
modified: float | None = None
|
||||
original_filename: str | None = None
|
||||
markdown_file: str | None = None
|
||||
markdown_path: str | None = None
|
||||
markdown_virtual_path: str | None = None
|
||||
markdown_artifact_url: str | None = None
|
||||
|
||||
|
||||
class UploadResponse(BaseModel):
|
||||
"""Response model for file upload."""
|
||||
|
||||
success: bool
|
||||
files: list[UploadedFileInfo]
|
||||
files: list[dict[str, str]]
|
||||
message: str
|
||||
skipped_files: list[str] = Field(default_factory=list)
|
||||
|
||||
|
||||
class UploadListResponse(BaseModel):
|
||||
"""Response model for uploaded file listing."""
|
||||
|
||||
files: list[UploadedFileInfo]
|
||||
count: int
|
||||
|
||||
|
||||
class UploadLimits(BaseModel):
|
||||
"""Application-level upload limits exposed to clients."""
|
||||
|
||||
max_files: int
|
||||
max_file_size: int
|
||||
max_total_size: int
|
||||
|
||||
|
||||
def _make_file_sandbox_writable(file_path: os.PathLike[str] | str) -> None:
|
||||
@@ -93,212 +48,71 @@ def _make_file_sandbox_writable(file_path: os.PathLike[str] | str) -> None:
|
||||
logger.warning("Skipping sandbox chmod for symlinked upload path: %s", file_path)
|
||||
return
|
||||
|
||||
writable_mode = stat.S_IMODE(file_stat.st_mode) | stat.S_IWUSR | stat.S_IWGRP | stat.S_IWOTH | stat.S_IRGRP | stat.S_IROTH
|
||||
writable_mode = stat.S_IMODE(file_stat.st_mode) | stat.S_IWUSR | stat.S_IWGRP | stat.S_IWOTH
|
||||
chmod_kwargs = {"follow_symlinks": False} if os.chmod in os.supports_follow_symlinks else {}
|
||||
os.chmod(file_path, writable_mode, **chmod_kwargs)
|
||||
|
||||
|
||||
def _make_file_sandbox_readable(file_path: os.PathLike[str] | str) -> None:
|
||||
"""Ensure uploaded files are readable by the sandbox process.
|
||||
|
||||
For Docker sandboxes (AIO), the gateway writes files as root with 0o600
|
||||
permissions, then bind-mounts the host directory into the container. The
|
||||
sandbox process inside the container runs as a non-root user and cannot
|
||||
read those files without group/other read bits. This function adds
|
||||
``S_IRGRP | S_IROTH`` so the sandbox can read the uploaded content.
|
||||
"""
|
||||
file_stat = os.lstat(file_path)
|
||||
if stat.S_ISLNK(file_stat.st_mode):
|
||||
logger.warning("Skipping sandbox chmod for symlinked upload path: %s", file_path)
|
||||
return
|
||||
|
||||
readable_mode = stat.S_IMODE(file_stat.st_mode) | stat.S_IRGRP | stat.S_IROTH
|
||||
chmod_kwargs = {"follow_symlinks": False} if os.chmod in os.supports_follow_symlinks else {}
|
||||
os.chmod(file_path, readable_mode, **chmod_kwargs)
|
||||
|
||||
|
||||
def _uses_thread_data_mounts(sandbox_provider: SandboxProvider) -> bool:
|
||||
return bool(getattr(sandbox_provider, "uses_thread_data_mounts", False))
|
||||
|
||||
|
||||
def _get_uploads_config_value(app_config: AppConfig, key: str, default: object) -> object:
|
||||
"""Read a value from the uploads config, supporting dict and attribute access."""
|
||||
uploads_cfg = getattr(app_config, "uploads", None)
|
||||
if isinstance(uploads_cfg, dict):
|
||||
return uploads_cfg.get(key, default)
|
||||
return getattr(uploads_cfg, key, default)
|
||||
|
||||
|
||||
def _get_upload_limit(app_config: AppConfig, key: str, default: int, *, legacy_key: str | None = None) -> int:
|
||||
try:
|
||||
value = _get_uploads_config_value(app_config, key, None)
|
||||
if value is None and legacy_key is not None:
|
||||
value = _get_uploads_config_value(app_config, legacy_key, None)
|
||||
if value is None:
|
||||
value = default
|
||||
limit = int(value)
|
||||
if limit <= 0:
|
||||
raise ValueError
|
||||
return limit
|
||||
except Exception:
|
||||
logger.warning("Invalid uploads.%s value; falling back to %d", key, default)
|
||||
return default
|
||||
|
||||
|
||||
def _get_upload_limits(app_config: AppConfig) -> UploadLimits:
|
||||
return UploadLimits(
|
||||
max_files=_get_upload_limit(app_config, "max_files", DEFAULT_MAX_FILES, legacy_key="max_file_count"),
|
||||
max_file_size=_get_upload_limit(app_config, "max_file_size", DEFAULT_MAX_FILE_SIZE, legacy_key="max_single_file_size"),
|
||||
max_total_size=_get_upload_limit(app_config, "max_total_size", DEFAULT_MAX_TOTAL_SIZE),
|
||||
)
|
||||
|
||||
|
||||
def _cleanup_uploaded_paths(paths: list[os.PathLike[str] | str]) -> None:
|
||||
for path in reversed(paths):
|
||||
try:
|
||||
os.unlink(path)
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
except Exception:
|
||||
logger.warning("Failed to clean up upload path after rejected request: %s", path, exc_info=True)
|
||||
|
||||
|
||||
async def _write_upload_file_with_limits(
|
||||
file: UploadFile,
|
||||
*,
|
||||
uploads_dir: os.PathLike[str] | str,
|
||||
display_filename: str,
|
||||
max_single_file_size: int,
|
||||
max_total_size: int,
|
||||
total_size: int,
|
||||
) -> tuple[os.PathLike[str] | str, int, int]:
|
||||
file_size = 0
|
||||
file_path, fh = open_upload_file_no_symlink(uploads_dir, display_filename)
|
||||
try:
|
||||
while chunk := await file.read(UPLOAD_CHUNK_SIZE):
|
||||
file_size += len(chunk)
|
||||
total_size += len(chunk)
|
||||
if file_size > max_single_file_size:
|
||||
raise HTTPException(status_code=413, detail=f"File too large: {display_filename}")
|
||||
if total_size > max_total_size:
|
||||
raise HTTPException(status_code=413, detail="Total upload size too large")
|
||||
fh.write(chunk)
|
||||
except Exception:
|
||||
fh.close()
|
||||
try:
|
||||
os.unlink(file_path)
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
raise
|
||||
else:
|
||||
fh.close()
|
||||
return file_path, file_size, total_size
|
||||
|
||||
|
||||
def _auto_convert_documents_enabled(app_config: AppConfig) -> bool:
|
||||
"""Return whether automatic host-side document conversion is enabled.
|
||||
|
||||
The secure default is disabled unless an operator explicitly opts in via
|
||||
uploads.auto_convert_documents in config.yaml.
|
||||
"""
|
||||
try:
|
||||
raw = _get_uploads_config_value(app_config, "auto_convert_documents", False)
|
||||
if isinstance(raw, str):
|
||||
return raw.strip().lower() in {"1", "true", "yes", "on"}
|
||||
return bool(raw)
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
@router.post("", response_model=UploadResponse)
|
||||
@require_permission("threads", "write", owner_check=True, require_existing=False)
|
||||
async def upload_files(
|
||||
thread_id: str,
|
||||
request: Request,
|
||||
files: list[UploadFile] = File(...),
|
||||
config: AppConfig = Depends(get_config),
|
||||
) -> UploadResponse:
|
||||
"""Upload multiple files to a thread's uploads directory."""
|
||||
if not files:
|
||||
raise HTTPException(status_code=400, detail="No files provided")
|
||||
|
||||
limits = _get_upload_limits(config)
|
||||
if len(files) > limits.max_files:
|
||||
raise HTTPException(status_code=413, detail=f"Too many files: maximum is {limits.max_files}")
|
||||
|
||||
try:
|
||||
uploads_dir = ensure_uploads_dir(thread_id)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
sandbox_uploads = get_paths().sandbox_uploads_dir(thread_id, user_id=get_effective_user_id())
|
||||
sandbox_uploads = get_paths().sandbox_uploads_dir(thread_id)
|
||||
uploaded_files = []
|
||||
written_paths = []
|
||||
sandbox_sync_targets = []
|
||||
skipped_files = []
|
||||
total_size = 0
|
||||
# Track filenames within this request so duplicate form parts do not
|
||||
# silently truncate each other. Existing uploads keep the historical
|
||||
# overwrite behavior for a single replacement upload.
|
||||
seen_filenames: set[str] = set()
|
||||
|
||||
sandbox_provider = get_sandbox_provider()
|
||||
sync_to_sandbox = not _uses_thread_data_mounts(sandbox_provider)
|
||||
sandbox = None
|
||||
if sync_to_sandbox:
|
||||
sandbox_id = sandbox_provider.acquire(thread_id)
|
||||
sandbox = sandbox_provider.get(sandbox_id)
|
||||
if sandbox is None:
|
||||
raise HTTPException(status_code=500, detail="Failed to acquire sandbox")
|
||||
auto_convert_documents = _auto_convert_documents_enabled(config)
|
||||
|
||||
for file in files:
|
||||
if not file.filename:
|
||||
continue
|
||||
|
||||
try:
|
||||
original_filename = normalize_filename(file.filename)
|
||||
safe_filename = claim_unique_filename(original_filename, seen_filenames)
|
||||
safe_filename = normalize_filename(file.filename)
|
||||
except ValueError:
|
||||
logger.warning(f"Skipping file with unsafe filename: {file.filename!r}")
|
||||
continue
|
||||
|
||||
try:
|
||||
file_path, file_size, total_size = await _write_upload_file_with_limits(
|
||||
file,
|
||||
uploads_dir=uploads_dir,
|
||||
display_filename=safe_filename,
|
||||
max_single_file_size=limits.max_file_size,
|
||||
max_total_size=limits.max_total_size,
|
||||
total_size=total_size,
|
||||
)
|
||||
written_paths.append(file_path)
|
||||
content = await file.read()
|
||||
file_path = uploads_dir / safe_filename
|
||||
file_path.write_bytes(content)
|
||||
|
||||
virtual_path = upload_virtual_path(safe_filename)
|
||||
|
||||
if sync_to_sandbox:
|
||||
sandbox_sync_targets.append((file_path, virtual_path))
|
||||
if sandbox_id != "local":
|
||||
_make_file_sandbox_writable(file_path)
|
||||
sandbox.update_file(virtual_path, content)
|
||||
|
||||
file_info = {
|
||||
"filename": safe_filename,
|
||||
"size": file_size,
|
||||
"size": str(len(content)),
|
||||
"path": str(sandbox_uploads / safe_filename),
|
||||
"virtual_path": virtual_path,
|
||||
"artifact_url": upload_artifact_url(thread_id, safe_filename),
|
||||
}
|
||||
if safe_filename != original_filename:
|
||||
file_info["original_filename"] = original_filename
|
||||
|
||||
logger.info(f"Saved file: {safe_filename} ({file_size} bytes) to {file_info['path']}")
|
||||
logger.info(f"Saved file: {safe_filename} ({len(content)} bytes) to {file_info['path']}")
|
||||
|
||||
file_ext = file_path.suffix.lower()
|
||||
if auto_convert_documents and file_ext in CONVERTIBLE_EXTENSIONS:
|
||||
if file_ext in CONVERTIBLE_EXTENSIONS:
|
||||
md_path = await convert_file_to_markdown(file_path)
|
||||
if md_path:
|
||||
written_paths.append(md_path)
|
||||
md_virtual_path = upload_virtual_path(md_path.name)
|
||||
|
||||
if sync_to_sandbox:
|
||||
sandbox_sync_targets.append((md_path, md_virtual_path))
|
||||
if sandbox_id != "local":
|
||||
_make_file_sandbox_writable(md_path)
|
||||
sandbox.update_file(md_virtual_path, md_path.read_bytes())
|
||||
|
||||
file_info["markdown_file"] = md_path.name
|
||||
file_info["markdown_path"] = str(sandbox_uploads / md_path.name)
|
||||
@@ -307,59 +121,19 @@ async def upload_files(
|
||||
|
||||
uploaded_files.append(file_info)
|
||||
|
||||
except HTTPException as e:
|
||||
_cleanup_uploaded_paths(written_paths)
|
||||
raise e
|
||||
except UnsafeUploadPathError as e:
|
||||
logger.warning("Skipping upload with unsafe destination %s: %s", file.filename, e)
|
||||
skipped_files.append(safe_filename)
|
||||
continue
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to upload {file.filename}: {e}")
|
||||
_cleanup_uploaded_paths(written_paths)
|
||||
raise HTTPException(status_code=500, detail=f"Failed to upload {file.filename}: {str(e)}")
|
||||
|
||||
# Uploaded files are created with 0o600 permissions (owner read/write only).
|
||||
# In Docker sandbox deployments the gateway writes as root but the sandbox
|
||||
# process runs as a non-root user (typically UID 1000). Without group/other
|
||||
# read bits the sandbox cannot access the files — whether the uploads
|
||||
# directory is bind-mounted into the container or synced via
|
||||
# sandbox.update_file. Always add group/other read bits so every sandbox
|
||||
# configuration can read the uploaded content.
|
||||
for file_path in written_paths:
|
||||
_make_file_sandbox_readable(file_path)
|
||||
|
||||
if sync_to_sandbox:
|
||||
for file_path, virtual_path in sandbox_sync_targets:
|
||||
_make_file_sandbox_writable(file_path)
|
||||
sandbox.update_file(virtual_path, file_path.read_bytes())
|
||||
|
||||
message = f"Successfully uploaded {len(uploaded_files)} file(s)"
|
||||
if skipped_files:
|
||||
message += f"; skipped {len(skipped_files)} unsafe file(s)"
|
||||
|
||||
return UploadResponse(
|
||||
success=not skipped_files,
|
||||
success=True,
|
||||
files=uploaded_files,
|
||||
message=message,
|
||||
skipped_files=skipped_files,
|
||||
message=f"Successfully uploaded {len(uploaded_files)} file(s)",
|
||||
)
|
||||
|
||||
|
||||
@router.get("/limits", response_model=UploadLimits)
|
||||
@require_permission("threads", "read", owner_check=True)
|
||||
async def get_upload_limits(
|
||||
thread_id: str,
|
||||
request: Request,
|
||||
config: AppConfig = Depends(get_config),
|
||||
) -> UploadLimits:
|
||||
"""Return upload limits used by the gateway for this thread."""
|
||||
return _get_upload_limits(config)
|
||||
|
||||
|
||||
@router.get("/list", response_model=UploadListResponse)
|
||||
@require_permission("threads", "read", owner_check=True)
|
||||
async def list_uploaded_files(thread_id: str, request: Request) -> UploadListResponse:
|
||||
@router.get("/list", response_model=dict)
|
||||
async def list_uploaded_files(thread_id: str) -> dict:
|
||||
"""List all files in a thread's uploads directory."""
|
||||
try:
|
||||
uploads_dir = get_uploads_dir(thread_id)
|
||||
@@ -369,16 +143,15 @@ async def list_uploaded_files(thread_id: str, request: Request) -> UploadListRes
|
||||
enrich_file_listing(result, thread_id)
|
||||
|
||||
# Gateway additionally includes the sandbox-relative path.
|
||||
sandbox_uploads = get_paths().sandbox_uploads_dir(thread_id, user_id=get_effective_user_id())
|
||||
sandbox_uploads = get_paths().sandbox_uploads_dir(thread_id)
|
||||
for f in result["files"]:
|
||||
f["path"] = str(sandbox_uploads / f["filename"])
|
||||
|
||||
return UploadListResponse(**result)
|
||||
return result
|
||||
|
||||
|
||||
@router.delete("/{filename}")
|
||||
@require_permission("threads", "delete", owner_check=True, require_existing=True)
|
||||
async def delete_uploaded_file(thread_id: str, filename: str, request: Request) -> dict:
|
||||
async def delete_uploaded_file(thread_id: str, filename: str) -> dict:
|
||||
"""Delete a file from a thread's uploads directory."""
|
||||
try:
|
||||
uploads_dir = get_uploads_dir(thread_id)
|
||||
|
||||
+121
-219
@@ -11,17 +11,13 @@ import asyncio
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
from collections.abc import Mapping
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
from fastapi import HTTPException, Request
|
||||
from langchain_core.messages import BaseMessage
|
||||
from langchain_core.messages.utils import convert_to_messages
|
||||
from langchain_core.messages import HumanMessage
|
||||
|
||||
from app.gateway.deps import get_run_context, get_run_manager, get_stream_bridge
|
||||
from app.gateway.internal_auth import INTERNAL_SYSTEM_ROLE
|
||||
from app.gateway.utils import sanitize_log_param
|
||||
from deerflow.config.app_config import get_app_config
|
||||
from app.gateway.deps import get_checkpointer, get_run_manager, get_store, get_stream_bridge
|
||||
from deerflow.runtime import (
|
||||
END_SENTINEL,
|
||||
HEARTBEAT_SENTINEL,
|
||||
@@ -34,7 +30,6 @@ from deerflow.runtime import (
|
||||
UnsupportedStrategyError,
|
||||
run_agent,
|
||||
)
|
||||
from deerflow.runtime.runs.naming import resolve_root_run_name
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -78,35 +73,21 @@ def normalize_stream_modes(raw: list[str] | str | None) -> list[str]:
|
||||
|
||||
|
||||
def normalize_input(raw_input: dict[str, Any] | None) -> dict[str, Any]:
|
||||
"""Convert LangGraph Platform input format to LangChain state dict.
|
||||
|
||||
Delegates dict→message coercion to ``langchain_core.messages.utils.convert_to_messages``
|
||||
so that ``additional_kwargs`` (e.g. uploaded-file metadata — gh #3132), ``id``,
|
||||
``name``, and non-human roles (ai/system/tool) survive unchanged. An earlier
|
||||
hand-rolled version only forwarded ``content`` and collapsed every role to
|
||||
``HumanMessage``, which silently stripped frontend-supplied attachments.
|
||||
|
||||
Malformed message dicts (missing ``role``/``type``/``content``, unsupported
|
||||
role, etc.) raise ``HTTPException(400)`` with the offending index, instead
|
||||
of bubbling up as a 500. The gateway is a system boundary, so per-entry
|
||||
validation errors are the right shape for clients to retry against.
|
||||
"""
|
||||
"""Convert LangGraph Platform input format to LangChain state dict."""
|
||||
if raw_input is None:
|
||||
return {}
|
||||
messages = raw_input.get("messages")
|
||||
if messages and isinstance(messages, list):
|
||||
converted: list[Any] = []
|
||||
for index, msg in enumerate(messages):
|
||||
if isinstance(msg, BaseMessage):
|
||||
converted.append(msg)
|
||||
elif isinstance(msg, dict):
|
||||
try:
|
||||
converted.extend(convert_to_messages([msg]))
|
||||
except (ValueError, TypeError, NotImplementedError) as exc:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Invalid message at input.messages[{index}]: {exc}",
|
||||
) from exc
|
||||
converted = []
|
||||
for msg in messages:
|
||||
if isinstance(msg, dict):
|
||||
role = msg.get("role", msg.get("type", "user"))
|
||||
content = msg.get("content", "")
|
||||
if role in ("user", "human"):
|
||||
converted.append(HumanMessage(content=content))
|
||||
else:
|
||||
# TODO: handle other message types (system, ai, tool)
|
||||
converted.append(HumanMessage(content=content))
|
||||
else:
|
||||
converted.append(msg)
|
||||
return {**raw_input, "messages": converted}
|
||||
@@ -116,82 +97,13 @@ def normalize_input(raw_input: dict[str, Any] | None) -> dict[str, Any]:
|
||||
_DEFAULT_ASSISTANT_ID = "lead_agent"
|
||||
|
||||
|
||||
# Whitelist of run-context keys that the langgraph-compat layer forwards from
|
||||
# ``body.context`` into the run config. ``config["context"]`` exists in
|
||||
# LangGraph >=0.6, but these values must be written to both ``configurable``
|
||||
# (for legacy ``_get_runtime_config`` consumers) and ``context`` because
|
||||
# LangGraph >=1.1.9 no longer makes ``ToolRuntime.context`` fall back to
|
||||
# ``configurable`` for consumers like ``setup_agent``.
|
||||
_CONTEXT_CONFIGURABLE_KEYS: frozenset[str] = frozenset(
|
||||
{
|
||||
"model_name",
|
||||
"mode",
|
||||
"thinking_enabled",
|
||||
"reasoning_effort",
|
||||
"is_plan_mode",
|
||||
"subagent_enabled",
|
||||
"max_concurrent_subagents",
|
||||
"agent_name",
|
||||
"is_bootstrap",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def merge_run_context_overrides(config: dict[str, Any], context: Mapping[str, Any] | None) -> None:
|
||||
"""Merge whitelisted keys from ``body.context`` into both ``config['configurable']``
|
||||
and ``config['context']`` so they are visible to legacy configurable readers and
|
||||
to LangGraph ``ToolRuntime.context`` consumers (e.g. the ``setup_agent`` tool —
|
||||
see issue #2677).
|
||||
|
||||
``user_id`` is intentionally propagated into ``config['context']`` in addition to
|
||||
the whitelisted keys, so non-web callers (e.g. IM channels) that supply identity in
|
||||
``body.context`` keep it on ``ToolRuntime.context``. It is merged with
|
||||
``setdefault`` so a server-authenticated id stamped by
|
||||
:func:`inject_authenticated_user_context` always wins over the client-supplied one.
|
||||
"""
|
||||
if not context:
|
||||
return
|
||||
configurable = config.setdefault("configurable", {})
|
||||
runtime_context = config.setdefault("context", {})
|
||||
for key in _CONTEXT_CONFIGURABLE_KEYS:
|
||||
if key in context:
|
||||
if isinstance(configurable, dict):
|
||||
configurable.setdefault(key, context[key])
|
||||
if isinstance(runtime_context, dict):
|
||||
runtime_context.setdefault(key, context[key])
|
||||
if "user_id" in context and isinstance(runtime_context, dict):
|
||||
runtime_context.setdefault("user_id", context["user_id"])
|
||||
|
||||
|
||||
def inject_authenticated_user_context(config: dict[str, Any], request: Request) -> None:
|
||||
"""Stamp the authenticated user into the run context for background tools.
|
||||
|
||||
Tool execution may happen after the request handler has returned, so tools
|
||||
that persist user-scoped files should not rely only on ambient ContextVars.
|
||||
The value comes from server-side auth state, never from client context.
|
||||
"""
|
||||
|
||||
user = getattr(request.state, "user", None)
|
||||
user_id = getattr(user, "id", None)
|
||||
if user_id is None:
|
||||
return
|
||||
|
||||
if getattr(user, "system_role", None) == INTERNAL_SYSTEM_ROLE:
|
||||
return
|
||||
|
||||
runtime_context = config.setdefault("context", {})
|
||||
if isinstance(runtime_context, dict):
|
||||
runtime_context["user_id"] = str(user_id)
|
||||
|
||||
|
||||
def resolve_agent_factory(assistant_id: str | None):
|
||||
"""Resolve the agent factory callable from config.
|
||||
|
||||
Custom agents are implemented as ``lead_agent`` + an ``agent_name``
|
||||
injected into ``configurable`` or ``context`` — see
|
||||
:func:`build_run_config`. All ``assistant_id`` values therefore map to the
|
||||
same factory; the routing happens inside ``make_lead_agent`` when it reads
|
||||
``cfg["agent_name"]``.
|
||||
injected into ``configurable`` — see :func:`build_run_config`. All
|
||||
``assistant_id`` values therefore map to the same factory; the routing
|
||||
happens inside ``make_lead_agent`` when it reads ``cfg["agent_name"]``.
|
||||
"""
|
||||
from deerflow.agents.lead_agent.agent import make_lead_agent
|
||||
|
||||
@@ -208,12 +120,10 @@ def build_run_config(
|
||||
"""Build a RunnableConfig dict for the agent.
|
||||
|
||||
When *assistant_id* refers to a custom agent (anything other than
|
||||
``"lead_agent"`` / ``None``), the name is forwarded as ``agent_name`` in
|
||||
whichever runtime options container is active: ``context`` for
|
||||
LangGraph >= 0.6.0 requests, otherwise ``configurable``.
|
||||
``make_lead_agent`` reads this key to load the matching
|
||||
``agents/<name>/SOUL.md`` and per-agent config — without it the agent
|
||||
silently runs as the default lead agent.
|
||||
``"lead_agent"`` / ``None``), the name is forwarded as
|
||||
``configurable["agent_name"]``. ``make_lead_agent`` reads this key to
|
||||
load the matching ``agents/<name>/SOUL.md`` and per-agent config —
|
||||
without it the agent silently runs as the default lead agent.
|
||||
|
||||
This mirrors the channel manager's ``_resolve_run_params`` logic so that
|
||||
the LangGraph Platform-compatible HTTP API and the IM channel path behave
|
||||
@@ -232,14 +142,7 @@ def build_run_config(
|
||||
thread_id,
|
||||
list(request_config.get("configurable", {}).keys()),
|
||||
)
|
||||
context_value = request_config["context"]
|
||||
if context_value is None:
|
||||
context = {}
|
||||
elif isinstance(context_value, Mapping):
|
||||
context = dict(context_value)
|
||||
else:
|
||||
raise ValueError("request config 'context' must be a mapping or null.")
|
||||
config["context"] = context
|
||||
config["context"] = request_config["context"]
|
||||
else:
|
||||
configurable = {"thread_id": thread_id}
|
||||
configurable.update(request_config.get("configurable", {}))
|
||||
@@ -251,20 +154,13 @@ def build_run_config(
|
||||
config["configurable"] = {"thread_id": thread_id}
|
||||
|
||||
# Inject custom agent name when the caller specified a non-default assistant.
|
||||
# Honour an explicit agent_name in the active runtime options container.
|
||||
if assistant_id and assistant_id != _DEFAULT_ASSISTANT_ID:
|
||||
# Honour an explicit configurable["agent_name"] in the request if already set.
|
||||
if assistant_id and assistant_id != _DEFAULT_ASSISTANT_ID and "configurable" in config:
|
||||
if "agent_name" not in config["configurable"]:
|
||||
normalized = assistant_id.strip().lower().replace("_", "-")
|
||||
if not normalized or not re.fullmatch(r"[a-z0-9-]+", normalized):
|
||||
raise ValueError(f"Invalid assistant_id {assistant_id!r}: must contain only letters, digits, and hyphens after normalization.")
|
||||
if "configurable" in config:
|
||||
target = config["configurable"]
|
||||
elif "context" in config:
|
||||
target = config["context"]
|
||||
else:
|
||||
target = config.setdefault("configurable", {})
|
||||
if target is not None and "agent_name" not in target:
|
||||
target["agent_name"] = normalized
|
||||
config.setdefault("run_name", resolve_root_run_name(config, normalized))
|
||||
config["configurable"]["agent_name"] = normalized
|
||||
if metadata:
|
||||
config.setdefault("metadata", {}).update(metadata)
|
||||
return config
|
||||
@@ -275,6 +171,71 @@ def build_run_config(
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def _upsert_thread_in_store(store, thread_id: str, metadata: dict | None) -> None:
|
||||
"""Create or refresh the thread record in the Store.
|
||||
|
||||
Called from :func:`start_run` so that threads created via the stateless
|
||||
``/runs/stream`` endpoint (which never calls ``POST /threads``) still
|
||||
appear in ``/threads/search`` results.
|
||||
"""
|
||||
# Deferred import to avoid circular import with the threads router module.
|
||||
from app.gateway.routers.threads import _store_upsert
|
||||
|
||||
try:
|
||||
await _store_upsert(store, thread_id, metadata=metadata)
|
||||
except Exception:
|
||||
logger.warning("Failed to upsert thread %s in store (non-fatal)", thread_id)
|
||||
|
||||
|
||||
async def _sync_thread_title_after_run(
|
||||
run_task: asyncio.Task,
|
||||
thread_id: str,
|
||||
checkpointer: Any,
|
||||
store: Any,
|
||||
) -> None:
|
||||
"""Wait for *run_task* to finish, then persist the generated title to the Store.
|
||||
|
||||
TitleMiddleware writes the generated title to the LangGraph agent state
|
||||
(checkpointer) but the Gateway's Store record is not updated automatically.
|
||||
This coroutine closes that gap by reading the final checkpoint after the
|
||||
run completes and syncing ``values.title`` into the Store record so that
|
||||
subsequent ``/threads/search`` responses include the correct title.
|
||||
|
||||
Runs as a fire-and-forget :func:`asyncio.create_task`; failures are
|
||||
logged at DEBUG level and never propagate.
|
||||
"""
|
||||
# Wait for the background run task to complete (any outcome).
|
||||
# asyncio.wait does not propagate task exceptions — it just returns
|
||||
# when the task is done, cancelled, or failed.
|
||||
await asyncio.wait({run_task})
|
||||
|
||||
# Deferred import to avoid circular import with the threads router module.
|
||||
from app.gateway.routers.threads import _store_get, _store_put
|
||||
|
||||
try:
|
||||
ckpt_config = {"configurable": {"thread_id": thread_id, "checkpoint_ns": ""}}
|
||||
ckpt_tuple = await checkpointer.aget_tuple(ckpt_config)
|
||||
if ckpt_tuple is None:
|
||||
return
|
||||
|
||||
channel_values = ckpt_tuple.checkpoint.get("channel_values", {})
|
||||
title = channel_values.get("title")
|
||||
if not title:
|
||||
return
|
||||
|
||||
existing = await _store_get(store, thread_id)
|
||||
if existing is None:
|
||||
return
|
||||
|
||||
updated = dict(existing)
|
||||
updated.setdefault("values", {})["title"] = title
|
||||
updated["updated_at"] = time.time()
|
||||
await _store_put(store, updated)
|
||||
logger.debug("Synced title %r for thread %s", title, thread_id)
|
||||
except Exception:
|
||||
logger.debug("Failed to sync title for thread %s (non-fatal)", thread_id, exc_info=True)
|
||||
|
||||
|
||||
async def start_run(
|
||||
body: Any,
|
||||
thread_id: str,
|
||||
@@ -294,27 +255,11 @@ async def start_run(
|
||||
"""
|
||||
bridge = get_stream_bridge(request)
|
||||
run_mgr = get_run_manager(request)
|
||||
run_ctx = get_run_context(request)
|
||||
checkpointer = get_checkpointer(request)
|
||||
store = get_store(request)
|
||||
|
||||
disconnect = DisconnectMode.cancel if body.on_disconnect == "cancel" else DisconnectMode.continue_
|
||||
|
||||
body_context = getattr(body, "context", None) or {}
|
||||
model_name = body_context.get("model_name")
|
||||
|
||||
# Coerce non-string model_name values to str before truncation.
|
||||
if model_name is not None and not isinstance(model_name, str):
|
||||
model_name = str(model_name)
|
||||
|
||||
# Validate model against the allowlist when a model_name is provided.
|
||||
if model_name:
|
||||
app_config = get_app_config()
|
||||
resolved = app_config.get_model_config(model_name)
|
||||
if resolved is None:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Model {model_name!r} is not in the configured model allowlist",
|
||||
)
|
||||
|
||||
try:
|
||||
record = await run_mgr.create_or_reject(
|
||||
thread_id,
|
||||
@@ -323,39 +268,41 @@ async def start_run(
|
||||
metadata=body.metadata or {},
|
||||
kwargs={"input": body.input, "config": body.config},
|
||||
multitask_strategy=body.multitask_strategy,
|
||||
model_name=model_name,
|
||||
)
|
||||
except ConflictError as exc:
|
||||
raise HTTPException(status_code=409, detail=str(exc)) from exc
|
||||
except UnsupportedStrategyError as exc:
|
||||
raise HTTPException(status_code=501, detail=str(exc)) from exc
|
||||
|
||||
# Upsert thread metadata so the thread appears in /threads/search,
|
||||
# even for threads that were never explicitly created via POST /threads
|
||||
# (e.g. stateless runs).
|
||||
try:
|
||||
existing = await run_ctx.thread_store.get(thread_id)
|
||||
if existing is None:
|
||||
await run_ctx.thread_store.create(
|
||||
thread_id,
|
||||
assistant_id=body.assistant_id,
|
||||
metadata=body.metadata,
|
||||
)
|
||||
else:
|
||||
await run_ctx.thread_store.update_status(thread_id, "running")
|
||||
except Exception:
|
||||
logger.warning("Failed to upsert thread_meta for %s (non-fatal)", sanitize_log_param(thread_id))
|
||||
# Ensure the thread is visible in /threads/search, even for threads that
|
||||
# were never explicitly created via POST /threads (e.g. stateless runs).
|
||||
store = get_store(request)
|
||||
if store is not None:
|
||||
await _upsert_thread_in_store(store, thread_id, body.metadata)
|
||||
|
||||
agent_factory = resolve_agent_factory(body.assistant_id)
|
||||
graph_input = normalize_input(body.input)
|
||||
config = build_run_config(thread_id, body.config, body.metadata, assistant_id=body.assistant_id)
|
||||
|
||||
# Merge DeerFlow-specific context overrides into both ``configurable`` and ``context``.
|
||||
# Merge DeerFlow-specific context overrides into configurable.
|
||||
# The ``context`` field is a custom extension for the langgraph-compat layer
|
||||
# that carries agent configuration (model_name, thinking_enabled, etc.).
|
||||
# Only agent-relevant keys are forwarded; unknown keys (e.g. thread_id) are ignored.
|
||||
merge_run_context_overrides(config, getattr(body, "context", None))
|
||||
inject_authenticated_user_context(config, request)
|
||||
context = getattr(body, "context", None)
|
||||
if context:
|
||||
_CONTEXT_CONFIGURABLE_KEYS = {
|
||||
"model_name",
|
||||
"mode",
|
||||
"thinking_enabled",
|
||||
"reasoning_effort",
|
||||
"is_plan_mode",
|
||||
"subagent_enabled",
|
||||
"max_concurrent_subagents",
|
||||
}
|
||||
configurable = config.setdefault("configurable", {})
|
||||
for key in _CONTEXT_CONFIGURABLE_KEYS:
|
||||
if key in context:
|
||||
configurable.setdefault(key, context[key])
|
||||
|
||||
stream_modes = normalize_stream_modes(body.stream_mode)
|
||||
|
||||
@@ -364,7 +311,8 @@ async def start_run(
|
||||
bridge,
|
||||
run_mgr,
|
||||
record,
|
||||
ctx=run_ctx,
|
||||
checkpointer=checkpointer,
|
||||
store=store,
|
||||
agent_factory=agent_factory,
|
||||
graph_input=graph_input,
|
||||
config=config,
|
||||
@@ -376,9 +324,11 @@ async def start_run(
|
||||
)
|
||||
record.task = task
|
||||
|
||||
# Title sync is handled by worker.py's finally block which reads the
|
||||
# title from the checkpoint and calls thread_store.update_display_name
|
||||
# after the run completes.
|
||||
# After the run completes, sync the title generated by TitleMiddleware from
|
||||
# the checkpointer into the Store record so that /threads/search returns the
|
||||
# correct title instead of an empty values dict.
|
||||
if store is not None:
|
||||
asyncio.create_task(_sync_thread_title_after_run(task, thread_id, checkpointer, store))
|
||||
|
||||
return record
|
||||
|
||||
@@ -415,51 +365,3 @@ async def sse_consumer(
|
||||
if record.status in (RunStatus.pending, RunStatus.running):
|
||||
if record.on_disconnect == DisconnectMode.cancel:
|
||||
await run_mgr.cancel(record.run_id)
|
||||
|
||||
|
||||
async def wait_for_run_completion(
|
||||
bridge: StreamBridge,
|
||||
record: RunRecord,
|
||||
request: Request,
|
||||
run_mgr: RunManager,
|
||||
) -> bool:
|
||||
"""Block until the run publishes ``END_SENTINEL``, honouring on_disconnect.
|
||||
|
||||
The non-streaming ``/wait`` endpoints used to ``await record.task``
|
||||
directly with no disconnect handling. When the client (or an
|
||||
intermediate HTTP proxy) timed out during a long tool call such as
|
||||
``pip install``, the handler would swallow ``CancelledError`` and
|
||||
serialize whatever checkpoint happened to exist — masking a half-finished
|
||||
run as a normal completion (issue #3265).
|
||||
|
||||
This helper consumes the same bridge that ``sse_consumer`` does so the
|
||||
wait path shares its disconnect semantics: each wake-up polls
|
||||
``request.is_disconnected()``; on a real disconnect it cancels the
|
||||
background run when ``record.on_disconnect`` is ``cancel``. The bridge's
|
||||
heartbeat sentinels guarantee at least one wake-up per
|
||||
``heartbeat_interval`` even when the agent emits no events for a while.
|
||||
|
||||
Returns:
|
||||
``True`` when ``END_SENTINEL`` was observed (run reached a terminal
|
||||
state), ``False`` when the loop exited because the client
|
||||
disconnected. Callers must skip checkpoint serialization on
|
||||
``False`` so a partial checkpoint is not returned as a normal
|
||||
response.
|
||||
"""
|
||||
completed = False
|
||||
try:
|
||||
async for entry in bridge.subscribe(record.run_id):
|
||||
# END_SENTINEL means the run reached a terminal state; honour it
|
||||
# even if the client just disconnected so the caller still serializes
|
||||
# the real final checkpoint.
|
||||
if entry is END_SENTINEL:
|
||||
completed = True
|
||||
return True
|
||||
if await request.is_disconnected():
|
||||
break
|
||||
# Heartbeats and regular events: keep waiting for END_SENTINEL.
|
||||
return completed
|
||||
finally:
|
||||
if not completed and record.status in (RunStatus.pending, RunStatus.running):
|
||||
if record.on_disconnect == DisconnectMode.cancel:
|
||||
await run_mgr.cancel(record.run_id)
|
||||
|
||||
@@ -1,6 +0,0 @@
|
||||
"""Shared utility helpers for the Gateway layer."""
|
||||
|
||||
|
||||
def sanitize_log_param(value: str) -> str:
|
||||
"""Strip control characters to prevent log injection."""
|
||||
return value.replace("\n", "").replace("\r", "").replace("\x00", "")
|
||||
+12
-89
@@ -19,72 +19,24 @@ import asyncio
|
||||
import logging
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from langchain_core.messages import HumanMessage
|
||||
|
||||
try:
|
||||
from prompt_toolkit import PromptSession
|
||||
from prompt_toolkit.history import InMemoryHistory
|
||||
|
||||
_HAS_PROMPT_TOOLKIT = True
|
||||
except ImportError:
|
||||
_HAS_PROMPT_TOOLKIT = False
|
||||
from deerflow.agents import make_lead_agent
|
||||
|
||||
load_dotenv()
|
||||
|
||||
_LOG_FMT = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
|
||||
_LOG_DATEFMT = "%Y-%m-%d %H:%M:%S"
|
||||
|
||||
|
||||
def _setup_logging(log_level: int = logging.INFO) -> None:
|
||||
"""Route logs to ``debug.log`` using *log_level* for the initial root/file setup.
|
||||
|
||||
This configures the root logger and the ``debug.log`` file handler so logs do
|
||||
not print on the interactive console. It is idempotent: any pre-existing
|
||||
handlers on the root logger (e.g. installed by ``logging.basicConfig`` in
|
||||
transitively imported modules) are removed so the debug session output only
|
||||
lands in ``debug.log``.
|
||||
|
||||
Note: later config-driven logging adjustments may change named logger
|
||||
verbosity without raising the root logger or file-handler thresholds set
|
||||
here, so the eventual contents of ``debug.log`` may not be filtered solely by
|
||||
this function's ``log_level`` argument.
|
||||
"""
|
||||
root = logging.root
|
||||
for h in list(root.handlers):
|
||||
root.removeHandler(h)
|
||||
h.close()
|
||||
root.setLevel(log_level)
|
||||
|
||||
file_handler = logging.FileHandler("debug.log", mode="a", encoding="utf-8")
|
||||
file_handler.setLevel(log_level)
|
||||
file_handler.setFormatter(logging.Formatter(_LOG_FMT, datefmt=_LOG_DATEFMT))
|
||||
root.addHandler(file_handler)
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
|
||||
datefmt="%Y-%m-%d %H:%M:%S",
|
||||
)
|
||||
|
||||
|
||||
async def main():
|
||||
# Install file logging first so warnings emitted while loading config do not
|
||||
# leak onto the interactive terminal via Python's lastResort handler.
|
||||
_setup_logging()
|
||||
|
||||
from deerflow.config import get_app_config
|
||||
from deerflow.config.app_config import apply_logging_level
|
||||
|
||||
app_config = get_app_config()
|
||||
apply_logging_level(app_config.log_level)
|
||||
|
||||
# Delay the rest of the deerflow imports until *after* logging is installed
|
||||
# so that any import-time side effects (e.g. deerflow.agents starts a
|
||||
# background skill-loader thread on import) emit logs to debug.log instead
|
||||
# of leaking onto the interactive terminal via Python's lastResort handler.
|
||||
from langchain_core.messages import HumanMessage
|
||||
from langgraph.runtime import Runtime
|
||||
|
||||
from deerflow.agents import make_lead_agent
|
||||
from deerflow.config.paths import get_paths
|
||||
from deerflow.mcp import initialize_mcp_tools
|
||||
from deerflow.runtime.user_context import get_effective_user_id
|
||||
|
||||
# Initialize MCP tools at startup
|
||||
try:
|
||||
from deerflow.mcp import initialize_mcp_tools
|
||||
|
||||
await initialize_mcp_tools()
|
||||
except Exception as e:
|
||||
print(f"Warning: Failed to initialize MCP tools: {e}")
|
||||
@@ -100,28 +52,15 @@ async def main():
|
||||
}
|
||||
}
|
||||
|
||||
runtime = Runtime(context={"thread_id": config["configurable"]["thread_id"]})
|
||||
config["configurable"]["__pregel_runtime"] = runtime
|
||||
|
||||
agent = make_lead_agent(config)
|
||||
|
||||
session = PromptSession(history=InMemoryHistory()) if _HAS_PROMPT_TOOLKIT else None
|
||||
|
||||
print("=" * 50)
|
||||
print("Lead Agent Debug Mode")
|
||||
print("Type 'quit' or 'exit' to stop")
|
||||
print(f"Logs: debug.log (log_level={app_config.log_level})")
|
||||
if not _HAS_PROMPT_TOOLKIT:
|
||||
print("Tip: `uv sync --group dev` to enable arrow-key & history support")
|
||||
print("=" * 50)
|
||||
|
||||
seen_artifacts: set[str] = set()
|
||||
|
||||
while True:
|
||||
try:
|
||||
if session:
|
||||
user_input = (await session.prompt_async("\nYou: ")).strip()
|
||||
else:
|
||||
user_input = input("\nYou: ").strip()
|
||||
if not user_input:
|
||||
continue
|
||||
@@ -131,31 +70,15 @@ async def main():
|
||||
|
||||
# Invoke the agent
|
||||
state = {"messages": [HumanMessage(content=user_input)]}
|
||||
result = await agent.ainvoke(state, config=config)
|
||||
result = await agent.ainvoke(state, config=config, context={"thread_id": "debug-thread-001"})
|
||||
|
||||
# Print the response
|
||||
if result.get("messages"):
|
||||
last_message = result["messages"][-1]
|
||||
print(f"\nAgent: {last_message.content}")
|
||||
|
||||
# Show files presented to the user this turn (new artifacts only)
|
||||
artifacts = result.get("artifacts") or []
|
||||
new_artifacts = [p for p in artifacts if p not in seen_artifacts]
|
||||
if new_artifacts:
|
||||
thread_id = config["configurable"]["thread_id"]
|
||||
user_id = get_effective_user_id()
|
||||
paths = get_paths()
|
||||
print("\n[Presented files]")
|
||||
for virtual in new_artifacts:
|
||||
try:
|
||||
physical = paths.resolve_virtual_path(thread_id, virtual, user_id=user_id)
|
||||
print(f" - {virtual}\n → {physical}")
|
||||
except ValueError as exc:
|
||||
print(f" - {virtual} (failed to resolve physical path: {exc})")
|
||||
seen_artifacts.update(new_artifacts)
|
||||
|
||||
except (KeyboardInterrupt, EOFError):
|
||||
print("\nGoodbye!")
|
||||
except KeyboardInterrupt:
|
||||
print("\nInterrupted. Goodbye!")
|
||||
break
|
||||
except Exception as e:
|
||||
print(f"\nError: {e}")
|
||||
|
||||
+46
-74
@@ -6,16 +6,16 @@ This document provides a complete reference for the DeerFlow backend APIs.
|
||||
|
||||
DeerFlow backend exposes two sets of APIs:
|
||||
|
||||
1. **LangGraph-compatible API** - Agent interactions, threads, and streaming (`/api/langgraph/*`)
|
||||
1. **LangGraph API** - Agent interactions, threads, and streaming (`/api/langgraph/*`)
|
||||
2. **Gateway API** - Models, MCP, skills, uploads, and artifacts (`/api/*`)
|
||||
|
||||
All APIs are accessed through the Nginx reverse proxy at port 2026.
|
||||
|
||||
## LangGraph-compatible API
|
||||
## LangGraph API
|
||||
|
||||
Base URL: `/api/langgraph`
|
||||
|
||||
The public LangGraph-compatible API follows LangGraph SDK conventions. In the unified nginx deployment, Gateway owns `/api/langgraph/*` and translates those paths to its native `/api/*` run, thread, and streaming routers.
|
||||
The LangGraph API is provided by the LangGraph server and follows the LangGraph SDK conventions.
|
||||
|
||||
### Threads
|
||||
|
||||
@@ -104,11 +104,17 @@ Content-Type: application/json
|
||||
**Recursion Limit:**
|
||||
|
||||
`config.recursion_limit` caps the number of graph steps LangGraph will execute
|
||||
in a single run. The unified Gateway path defaults to `100` in
|
||||
`build_run_config` (see `backend/app/gateway/services.py`), which is a safer
|
||||
starting point for plan-mode or subagent-heavy runs. Clients can still set
|
||||
`recursion_limit` explicitly in the request body; increase it if you run deeply
|
||||
nested subagent graphs.
|
||||
in a single run. The `/api/langgraph/*` endpoints go straight to the LangGraph
|
||||
server and therefore inherit LangGraph's native default of **25**, which is
|
||||
too low for plan-mode or subagent-heavy runs — the agent typically errors out
|
||||
with `GraphRecursionError` after the first round of subagent results comes
|
||||
back, before the lead agent can synthesize the final answer.
|
||||
|
||||
DeerFlow's own Gateway and IM-channel paths mitigate this by defaulting to
|
||||
`100` in `build_run_config` (see `backend/app/gateway/services.py`), but
|
||||
clients calling the LangGraph API directly must set `recursion_limit`
|
||||
explicitly in the request body. `100` matches the Gateway default and is a
|
||||
safe starting point; increase it if you run deeply nested subagent graphs.
|
||||
|
||||
**Configurable Options:**
|
||||
- `model_name` (string): Override the default model
|
||||
@@ -228,13 +234,10 @@ Get current MCP server configurations.
|
||||
GET /api/mcp/config
|
||||
```
|
||||
|
||||
Requires an authenticated admin session. Sensitive env/header/OAuth secret
|
||||
values are masked in the response.
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"mcp_servers": {
|
||||
"mcpServers": {
|
||||
"github": {
|
||||
"enabled": true,
|
||||
"type": "stdio",
|
||||
@@ -244,6 +247,13 @@ values are masked in the response.
|
||||
"GITHUB_TOKEN": "***"
|
||||
},
|
||||
"description": "GitHub operations"
|
||||
},
|
||||
"filesystem": {
|
||||
"enabled": false,
|
||||
"type": "stdio",
|
||||
"command": "npx",
|
||||
"args": ["-y", "@modelcontextprotocol/server-filesystem"],
|
||||
"description": "File system access"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -258,15 +268,10 @@ PUT /api/mcp/config
|
||||
Content-Type: application/json
|
||||
```
|
||||
|
||||
Requires an authenticated admin session. API-managed `stdio` MCP servers may
|
||||
only use allowed executable names for `command` (default: `npx`, `uvx`). Set
|
||||
`DEER_FLOW_MCP_STDIO_COMMAND_ALLOWLIST` to a comma-separated list when a
|
||||
deployment needs additional trusted launchers.
|
||||
|
||||
**Request Body:**
|
||||
```json
|
||||
{
|
||||
"mcp_servers": {
|
||||
"mcpServers": {
|
||||
"github": {
|
||||
"enabled": true,
|
||||
"type": "stdio",
|
||||
@@ -284,18 +289,8 @@ deployment needs additional trusted launchers.
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"mcp_servers": {
|
||||
"github": {
|
||||
"enabled": true,
|
||||
"type": "stdio",
|
||||
"command": "npx",
|
||||
"args": ["-y", "@modelcontextprotocol/server-github"],
|
||||
"env": {
|
||||
"GITHUB_TOKEN": "***"
|
||||
},
|
||||
"description": "GitHub operations"
|
||||
}
|
||||
}
|
||||
"success": true,
|
||||
"message": "MCP configuration updated"
|
||||
}
|
||||
```
|
||||
|
||||
@@ -546,28 +541,14 @@ All APIs return errors in a consistent format:
|
||||
|
||||
## Authentication
|
||||
|
||||
DeerFlow enforces authentication for all non-public HTTP routes. Public routes are limited to health/docs metadata and these public auth endpoints:
|
||||
Currently, DeerFlow does not implement authentication. All APIs are accessible without credentials.
|
||||
|
||||
- `POST /api/v1/auth/initialize` creates the first admin account when no admin exists.
|
||||
- `POST /api/v1/auth/login/local` logs in with email/password and sets an HttpOnly `access_token` cookie.
|
||||
- `POST /api/v1/auth/register` creates a regular `user` account and sets the session cookie.
|
||||
- `POST /api/v1/auth/logout` clears the session cookie.
|
||||
- `GET /api/v1/auth/setup-status` reports whether the first admin still needs to be created.
|
||||
Note: This is about DeerFlow API authentication. MCP outbound connections can still use OAuth for configured HTTP/SSE MCP servers.
|
||||
|
||||
The authenticated auth endpoints are:
|
||||
|
||||
- `GET /api/v1/auth/me` returns the current user.
|
||||
- `POST /api/v1/auth/change-password` changes password, optionally changes email during setup, increments `token_version`, and reissues the cookie.
|
||||
|
||||
Protected state-changing requests also require the CSRF double-submit token: send the `csrf_token` cookie value as the `X-CSRF-Token` header. Login/register/initialize/logout are bootstrap auth endpoints: they are exempt from the double-submit token but still reject hostile browser `Origin` headers.
|
||||
|
||||
User isolation is enforced from the authenticated user context:
|
||||
|
||||
- Thread metadata is scoped by `threads_meta.user_id`; search/read/write/delete APIs only expose the current user's threads.
|
||||
- Thread files live under `{base_dir}/users/{user_id}/threads/{thread_id}/user-data/` and are exposed inside the sandbox as `/mnt/user-data/`.
|
||||
- Memory and custom agents are stored under `{base_dir}/users/{user_id}/...`.
|
||||
|
||||
Note: MCP outbound connections can still use OAuth for configured HTTP/SSE MCP servers; that is separate from DeerFlow API authentication.
|
||||
For production deployments, it is recommended to:
|
||||
1. Use Nginx for basic auth or OAuth integration
|
||||
2. Deploy behind a VPN or private network
|
||||
3. Implement custom authentication middleware
|
||||
|
||||
---
|
||||
|
||||
@@ -586,13 +567,12 @@ location /api/ {
|
||||
|
||||
---
|
||||
|
||||
## Streaming Support
|
||||
## WebSocket Support
|
||||
|
||||
Gateway's LangGraph-compatible API streams run events with Server-Sent Events (SSE):
|
||||
The LangGraph server supports WebSocket connections for real-time streaming. Connect to:
|
||||
|
||||
```http
|
||||
POST /api/langgraph/threads/{thread_id}/runs/stream
|
||||
Accept: text/event-stream
|
||||
```
|
||||
ws://localhost:2026/api/langgraph/threads/{thread_id}/runs/stream
|
||||
```
|
||||
|
||||
---
|
||||
@@ -628,21 +608,13 @@ const response = await fetch('/api/models');
|
||||
const data = await response.json();
|
||||
console.log(data.models);
|
||||
|
||||
// Create a run and stream SSE events
|
||||
const streamResponse = await fetch(`/api/langgraph/threads/${threadId}/runs/stream`, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
Accept: "text/event-stream",
|
||||
},
|
||||
body: JSON.stringify({
|
||||
input: { messages: [{ role: "user", content: "Hello" }] },
|
||||
stream_mode: ["values", "messages-tuple", "custom"],
|
||||
}),
|
||||
});
|
||||
|
||||
const reader = streamResponse.body?.getReader();
|
||||
// Decode and parse SSE frames from reader in your client code.
|
||||
// Using EventSource for streaming
|
||||
const eventSource = new EventSource(
|
||||
`/api/langgraph/threads/${threadId}/runs/stream`
|
||||
);
|
||||
eventSource.onmessage = (event) => {
|
||||
console.log(JSON.parse(event.data));
|
||||
};
|
||||
```
|
||||
|
||||
### cURL Examples
|
||||
@@ -677,7 +649,7 @@ curl -X POST http://localhost:2026/api/langgraph/threads/abc123/runs \
|
||||
}'
|
||||
```
|
||||
|
||||
> The unified Gateway path defaults `config.recursion_limit` to 100 for
|
||||
> plan-mode and subagent-heavy runs. Clients may still set
|
||||
> `config.recursion_limit` explicitly — see the [Create Run](#create-run)
|
||||
> section for details.
|
||||
> The `/api/langgraph/*` endpoints bypass DeerFlow's Gateway and inherit
|
||||
> LangGraph's native `recursion_limit` default of 25, which is too low for
|
||||
> plan-mode or subagent runs. Set `config.recursion_limit` explicitly — see
|
||||
> the [Create Run](#create-run) section for details.
|
||||
|
||||
@@ -14,28 +14,30 @@ This document provides a comprehensive overview of the DeerFlow backend architec
|
||||
│ Nginx (Port 2026) │
|
||||
│ Unified Reverse Proxy Entry Point │
|
||||
│ ┌────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ /api/langgraph/* → Gateway LangGraph-compatible runtime (8001) │ │
|
||||
│ │ /api/* → Gateway REST APIs (8001) │ │
|
||||
│ │ /api/langgraph/* → LangGraph Server (2024) │ │
|
||||
│ │ /api/* → Gateway API (8001) │ │
|
||||
│ │ /* → Frontend (3000) │ │
|
||||
│ └────────────────────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────┬────────────────────────────────────────┘
|
||||
│
|
||||
┌───────────────────────┴───────────────────────┐
|
||||
┌───────────────────────┼───────────────────────┐
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌─────────────────────┐ ┌─────────────────────┐ ┌─────────────────────┐
|
||||
│ LangGraph Server │ │ Gateway API │ │ Frontend │
|
||||
│ (Port 2024) │ │ (Port 8001) │ │ (Port 3000) │
|
||||
│ │ │ │ │ │
|
||||
│ - Agent Runtime │ │ - Models API │ │ - Next.js App │
|
||||
│ - Thread Mgmt │ │ - MCP Config │ │ - React UI │
|
||||
│ - SSE Streaming │ │ - Skills Mgmt │ │ - Chat Interface │
|
||||
│ - Checkpointing │ │ - File Uploads │ │ │
|
||||
│ │ │ - Thread Cleanup │ │ │
|
||||
│ │ │ - Artifacts │ │ │
|
||||
└─────────────────────┘ └─────────────────────┘ └─────────────────────┘
|
||||
│ │
|
||||
│ ┌─────────────────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
┌─────────────────────────────────────────────┐ ┌─────────────────────┐
|
||||
│ Gateway API │ │ Frontend │
|
||||
│ (Port 8001) │ │ (Port 3000) │
|
||||
│ │ │ │
|
||||
│ - LangGraph-compatible runs/threads API │ │ - Next.js App │
|
||||
│ - Embedded Agent Runtime │ │ - React UI │
|
||||
│ - SSE Streaming │ │ - Chat Interface │
|
||||
│ - Checkpointing │ │ │
|
||||
│ - Models, MCP, Skills, Uploads, Artifacts │ │ │
|
||||
│ - Thread Cleanup │ │ │
|
||||
└─────────────────────────────────────────────┘ └─────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────────────────────┐
|
||||
│ Shared Configuration │
|
||||
│ ┌─────────────────────────┐ ┌────────────────────────────────────────┐ │
|
||||
@@ -50,9 +52,9 @@ This document provides a comprehensive overview of the DeerFlow backend architec
|
||||
|
||||
## Component Details
|
||||
|
||||
### Gateway Embedded Agent Runtime
|
||||
### LangGraph Server
|
||||
|
||||
The agent runtime is embedded in the FastAPI Gateway and built on LangGraph for robust multi-agent workflow orchestration. Nginx rewrites `/api/langgraph/*` to Gateway's native `/api/*` routes, so the public API remains compatible with LangGraph SDK clients without running a separate LangGraph server.
|
||||
The LangGraph server is the core agent runtime, built on LangGraph for robust multi-agent workflow orchestration.
|
||||
|
||||
**Entry Point**: `packages/harness/deerflow/agents/lead_agent/agent.py:make_lead_agent`
|
||||
|
||||
@@ -63,8 +65,7 @@ The agent runtime is embedded in the FastAPI Gateway and built on LangGraph for
|
||||
- Tool execution orchestration
|
||||
- SSE streaming for real-time responses
|
||||
|
||||
**Graph registry**: `langgraph.json` remains available for tooling, Studio, or direct LangGraph Server compatibility.
|
||||
It is not the default service entrypoint; scripts and Docker deployments run the Gateway embedded runtime.
|
||||
**Configuration**: `langgraph.json`
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -77,13 +78,12 @@ It is not the default service entrypoint; scripts and Docker deployments run the
|
||||
|
||||
### Gateway API
|
||||
|
||||
FastAPI application providing REST endpoints plus the public LangGraph-compatible `/api/langgraph/*` runtime routes.
|
||||
FastAPI application providing REST endpoints for non-agent operations.
|
||||
|
||||
**Entry Point**: `app/gateway/app.py`
|
||||
|
||||
**Routers**:
|
||||
- `models.py` - `/api/models` - Model listing and details
|
||||
- `thread_runs.py` / `runs.py` - `/api/threads/{id}/runs`, `/api/runs/*` - LangGraph-compatible runs and streaming
|
||||
- `mcp.py` - `/api/mcp` - MCP server configuration
|
||||
- `skills.py` - `/api/skills` - Skills management
|
||||
- `uploads.py` - `/api/threads/{id}/uploads` - File upload
|
||||
@@ -91,7 +91,7 @@ FastAPI application providing REST endpoints plus the public LangGraph-compatibl
|
||||
- `artifacts.py` - `/api/threads/{id}/artifacts` - Artifact serving
|
||||
- `suggestions.py` - `/api/threads/{id}/suggestions` - Follow-up suggestion generation
|
||||
|
||||
The web conversation delete flow first deletes Gateway-managed thread state through the LangGraph-compatible route, then the Gateway `threads.py` router removes DeerFlow-managed filesystem data via `Paths.delete_thread_dir()`.
|
||||
The web conversation delete flow is now split across both backend surfaces: LangGraph handles `DELETE /api/langgraph/threads/{thread_id}` for thread state, then the Gateway `threads.py` router removes DeerFlow-managed filesystem data via `Paths.delete_thread_dir()`.
|
||||
|
||||
### Agent Architecture
|
||||
|
||||
@@ -199,7 +199,7 @@ class ThreadState(AgentState):
|
||||
│ Built-in Tools │ │ Configured Tools │ │ MCP Tools │
|
||||
│ (packages/harness/deerflow/tools/) │ │ (config.yaml) │ │ (extensions.json) │
|
||||
├─────────────────────┤ ├─────────────────────┤ ├─────────────────────┤
|
||||
│ - present_files │ │ - web_search │ │ - github │
|
||||
│ - present_file │ │ - web_search │ │ - github │
|
||||
│ - ask_clarification │ │ - web_fetch │ │ - filesystem │
|
||||
│ - view_image │ │ - bash │ │ - postgres │
|
||||
│ │ │ - read_file │ │ - brave-search │
|
||||
@@ -353,10 +353,10 @@ SKILL.md Format:
|
||||
POST /api/langgraph/threads/{thread_id}/runs
|
||||
{"input": {"messages": [{"role": "user", "content": "Hello"}]}}
|
||||
|
||||
2. Nginx → Gateway API (8001)
|
||||
`/api/langgraph/*` is rewritten to Gateway's LangGraph-compatible `/api/*` routes
|
||||
2. Nginx → LangGraph Server (2024)
|
||||
Proxied to LangGraph server
|
||||
|
||||
3. Gateway embedded runtime
|
||||
3. LangGraph Server
|
||||
a. Load/create thread state
|
||||
b. Execute middleware chain:
|
||||
- ThreadDataMiddleware: Set up paths
|
||||
@@ -412,7 +412,7 @@ SKILL.md Format:
|
||||
### Thread Cleanup Flow
|
||||
|
||||
```
|
||||
1. Client deletes conversation via the LangGraph-compatible Gateway route
|
||||
1. Client deletes conversation via LangGraph
|
||||
DELETE /api/langgraph/threads/{thread_id}
|
||||
|
||||
2. Web UI follows up with Gateway cleanup
|
||||
|
||||
@@ -1,331 +0,0 @@
|
||||
# 用户认证与隔离设计
|
||||
|
||||
本文档描述 DeerFlow 当前内置认证模块的设计,而不是历史 RFC。它覆盖浏览器登录、API 认证、CSRF、用户隔离、首次初始化、密码重置、内部调用和升级迁移。
|
||||
|
||||
## 设计目标
|
||||
|
||||
认证模块的核心目标是把 DeerFlow 从“本地单用户工具”提升为“可多用户部署的 agent runtime”,并让用户身份贯穿 HTTP API、LangGraph-compatible runtime、文件系统、memory、自定义 agent 和反馈数据。
|
||||
|
||||
设计约束:
|
||||
|
||||
- 默认强制认证:除健康检查、文档和 auth bootstrap 端点外,HTTP 路由都必须有有效 session。
|
||||
- 服务端持有所有权:客户端 metadata 不能声明 `user_id` 或 `owner_id`。
|
||||
- 隔离默认开启:repository(仓储)、文件路径、memory、agent 配置默认按当前用户解析。
|
||||
- 旧数据可升级:无认证版本留下的 thread 可以在 admin 存在后迁移到 admin。
|
||||
- 密码不进日志:首次初始化由操作者设置密码;`reset_admin` 只写 0600 凭据文件。
|
||||
|
||||
非目标:
|
||||
|
||||
- 当前 OAuth 端点只是占位,尚未实现第三方登录。
|
||||
- 当前用户角色只有 `admin` 和 `user`,尚未实现细粒度 RBAC。
|
||||
- 当前登录限速是进程内字典,多 worker 下不是全局精确限速。
|
||||
|
||||
## 核心模型
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
classDef actor fill:#D8CFC4,stroke:#6E6259,color:#2F2A26;
|
||||
classDef api fill:#C9D7D2,stroke:#5D706A,color:#21302C;
|
||||
classDef state fill:#D7D3E8,stroke:#6B6680,color:#29263A;
|
||||
classDef data fill:#E5D2C4,stroke:#806A5B,color:#30251E;
|
||||
|
||||
Browser["Browser — access_token cookie and csrf_token cookie"]:::actor
|
||||
AuthMiddleware["AuthMiddleware — strict session gate"]:::api
|
||||
CSRFMiddleware["CSRFMiddleware — double-submit token and Origin check"]:::api
|
||||
AuthRoutes["Auth routes — initialize login register logout me change-password"]:::api
|
||||
UserContext["Current user ContextVar — request-scoped identity"]:::state
|
||||
Repositories["Repositories — AUTO resolves user_id from context"]:::state
|
||||
Files["Filesystem — users/{user_id}/threads/{thread_id}/user-data"]:::data
|
||||
Memory["Memory and agents — users/{user_id}/memory.json and agents"]:::data
|
||||
|
||||
Browser --> AuthMiddleware
|
||||
Browser --> CSRFMiddleware
|
||||
AuthMiddleware --> AuthRoutes
|
||||
AuthMiddleware --> UserContext
|
||||
UserContext --> Repositories
|
||||
UserContext --> Files
|
||||
UserContext --> Memory
|
||||
```
|
||||
|
||||
### 用户表
|
||||
|
||||
用户记录定义在 `app.gateway.auth.models.User`,持久化到 `users` 表。关键字段:
|
||||
|
||||
| 字段 | 语义 |
|
||||
|---|---|
|
||||
| `id` | 用户主键,JWT `sub` 使用该值 |
|
||||
| `email` | 唯一登录名 |
|
||||
| `password_hash` | bcrypt hash,OAuth 用户可为空 |
|
||||
| `system_role` | `admin` 或 `user` |
|
||||
| `needs_setup` | reset 后要求用户完成邮箱 / 密码设置 |
|
||||
| `token_version` | 改密码或 reset 时递增,用于废弃旧 JWT |
|
||||
|
||||
### 运行时身份
|
||||
|
||||
认证成功后,`AuthMiddleware` 把用户同时写入:
|
||||
|
||||
- `request.state.user`
|
||||
- `request.state.auth`
|
||||
- `deerflow.runtime.user_context` 的 `ContextVar`
|
||||
|
||||
`ContextVar` 是这里的核心边界。上层 Gateway 负责写入身份,下层 persistence / file path 只读取结构化的当前用户,不反向依赖 `app.gateway.auth` 具体类型。
|
||||
|
||||
可以把 repository 调用的用户参数理解成一个三态 ADT:
|
||||
|
||||
```scala
|
||||
enum UserScope:
|
||||
case AutoFromContext
|
||||
case Explicit(userId: String)
|
||||
case BypassForMigration
|
||||
```
|
||||
|
||||
对应 Python 实现是 `AUTO | str | None`:
|
||||
|
||||
- `AUTO`:从 `ContextVar` 解析当前用户;没有上下文则抛错。
|
||||
- `str`:显式指定用户,主要用于测试或管理脚本。
|
||||
- `None`:跳过用户过滤,只允许迁移脚本或 admin CLI 使用。
|
||||
|
||||
## 登录与初始化流程
|
||||
|
||||
### 首次初始化
|
||||
|
||||
首次启动时,如果没有 admin,服务不会自动创建账号,只记录日志提示访问 `/setup`。
|
||||
|
||||
流程:
|
||||
|
||||
1. 用户访问 `/setup`。
|
||||
2. 前端调用 `GET /api/v1/auth/setup-status`。
|
||||
3. 如果返回 `{"needs_setup": true}`,前端展示创建 admin 表单。
|
||||
4. 表单提交 `POST /api/v1/auth/initialize`。
|
||||
5. 服务端确认当前没有 admin,创建 `system_role="admin"`、`needs_setup=false` 的用户。
|
||||
6. 服务端设置 `access_token` HttpOnly cookie,用户进入 workspace。
|
||||
|
||||
`/api/v1/auth/initialize` 只在没有 admin 时可用。并发初始化由数据库唯一约束兜底,失败方返回 409。
|
||||
|
||||
### 普通登录
|
||||
|
||||
`POST /api/v1/auth/login/local` 使用 `OAuth2PasswordRequestForm`:
|
||||
|
||||
- `username` 是邮箱。
|
||||
- `password` 是密码。
|
||||
- 成功后签发 JWT,放入 `access_token` HttpOnly cookie。
|
||||
- 响应体只返回 `expires_in` 和 `needs_setup`,不返回 token。
|
||||
|
||||
登录失败会按客户端 IP 计数。IP 解析只在 TCP peer 属于 `AUTH_TRUSTED_PROXIES` 时信任 `X-Real-IP`,不使用 `X-Forwarded-For`。
|
||||
|
||||
### 注册
|
||||
|
||||
`POST /api/v1/auth/register` 创建普通 `user`,并自动登录。
|
||||
|
||||
当前实现允许在没有 admin 时注册普通用户,但 `setup-status` 仍会返回 `needs_setup=true`,因为 admin 仍不存在。这是当前产品策略边界:如果后续要求“必须先初始化 admin 才能注册普通用户”,需要在 `/register` 增加 admin-exists gate。
|
||||
|
||||
### 改密码与 reset setup
|
||||
|
||||
`POST /api/v1/auth/change-password` 需要当前密码和新密码:
|
||||
|
||||
- 校验当前密码。
|
||||
- 更新 bcrypt hash。
|
||||
- `token_version += 1`,使旧 JWT 立即失效。
|
||||
- 重新签发 cookie。
|
||||
- 如果 `needs_setup=true` 且传了 `new_email`,则更新邮箱并清除 `needs_setup`。
|
||||
|
||||
`python -m app.gateway.auth.reset_admin` 会:
|
||||
|
||||
- 找到 admin 或指定邮箱用户。
|
||||
- 生成随机密码。
|
||||
- 更新密码 hash。
|
||||
- `token_version += 1`。
|
||||
- 设置 `needs_setup=true`。
|
||||
- 写入 `.deer-flow/admin_initial_credentials.txt`,权限 `0600`。
|
||||
|
||||
命令行只输出凭据文件路径,不输出明文密码。
|
||||
|
||||
## HTTP 认证边界
|
||||
|
||||
`AuthMiddleware` 是 fail-closed(默认拒绝)的全局认证门。
|
||||
|
||||
公开路径:
|
||||
|
||||
- `/health`
|
||||
- `/docs`
|
||||
- `/redoc`
|
||||
- `/openapi.json`
|
||||
- `/api/v1/auth/login/local`
|
||||
- `/api/v1/auth/register`
|
||||
- `/api/v1/auth/logout`
|
||||
- `/api/v1/auth/setup-status`
|
||||
- `/api/v1/auth/initialize`
|
||||
|
||||
其余路径都要求有效 `access_token` cookie。存在 cookie 但 JWT 无效、过期、用户不存在或 `token_version` 不匹配时,直接返回 401,而不是让请求穿透到业务路由。
|
||||
|
||||
路由级别的 owner check 由 `require_permission(..., owner_check=True)` 完成:
|
||||
|
||||
- 读类请求允许旧的未追踪 legacy thread 兼容读取。
|
||||
- 写 / 删除类请求使用 `require_existing=True`,要求 thread row 存在且属于当前用户,避免删除后缺 row 导致其他用户误通过。
|
||||
|
||||
## CSRF 设计
|
||||
|
||||
DeerFlow 使用 Double Submit Cookie:
|
||||
|
||||
- 服务端设置 `csrf_token` cookie。
|
||||
- 前端 state-changing 请求发送同值 `X-CSRF-Token` header。
|
||||
- 服务端用 `secrets.compare_digest` 比较 cookie/header。
|
||||
|
||||
需要 CSRF 的方法:
|
||||
|
||||
- `POST`
|
||||
- `PUT`
|
||||
- `DELETE`
|
||||
- `PATCH`
|
||||
|
||||
auth bootstrap 端点(login/register/initialize/logout)不要求 double-submit token,因为首次调用时浏览器还没有 token;但这些端点会校验 browser `Origin`,拒绝 hostile Origin,避免 login CSRF / session fixation。
|
||||
|
||||
## 用户隔离
|
||||
|
||||
### Thread metadata
|
||||
|
||||
Thread metadata 存在 `threads_meta`,关键隔离字段是 `user_id`。
|
||||
|
||||
创建 thread 时:
|
||||
|
||||
- 客户端传入的 `metadata.user_id` 和 `metadata.owner_id` 会被剥离。
|
||||
- `ThreadMetaRepository.create(..., user_id=AUTO)` 从 `ContextVar` 解析真实用户。
|
||||
- `/api/threads/search` 默认只返回当前用户的 thread。
|
||||
|
||||
读取 / 修改 / 删除时:
|
||||
|
||||
- `get()` 默认按当前用户过滤。
|
||||
- `check_access()` 用于路由 owner check。
|
||||
- 对其他用户的 thread 返回 404,避免泄露资源存在性。
|
||||
|
||||
### 文件系统
|
||||
|
||||
当前线程文件布局:
|
||||
|
||||
```text
|
||||
{base_dir}/users/{user_id}/threads/{thread_id}/user-data/
|
||||
├── workspace/
|
||||
├── uploads/
|
||||
└── outputs/
|
||||
```
|
||||
|
||||
agent 在 sandbox 内看到统一虚拟路径:
|
||||
|
||||
```text
|
||||
/mnt/user-data/workspace
|
||||
/mnt/user-data/uploads
|
||||
/mnt/user-data/outputs
|
||||
```
|
||||
|
||||
`ThreadDataMiddleware` 使用 `get_effective_user_id()` 解析当前用户并生成线程路径。没有认证上下文时会落到 `default` 用户桶,主要用于内部调用、嵌入式 client 或无 HTTP 的本地执行路径。
|
||||
|
||||
### Memory
|
||||
|
||||
默认 memory 存储:
|
||||
|
||||
```text
|
||||
{base_dir}/users/{user_id}/memory.json
|
||||
{base_dir}/users/{user_id}/agents/{agent_name}/memory.json
|
||||
```
|
||||
|
||||
有用户上下文时,空或相对 `memory.storage_path` 都使用上述 per-user 默认路径;只有绝对 `memory.storage_path` 会视为显式 opt-out(退出) per-user isolation,所有用户共享该路径。无用户上下文的 legacy 路径仍会把相对 `storage_path` 解析到 `Paths.base_dir` 下。
|
||||
|
||||
### 自定义 agent
|
||||
|
||||
用户自定义 agent 写入:
|
||||
|
||||
```text
|
||||
{base_dir}/users/{user_id}/agents/{agent_name}/
|
||||
├── config.yaml
|
||||
├── SOUL.md
|
||||
└── memory.json
|
||||
```
|
||||
|
||||
旧布局 `{base_dir}/agents/{agent_name}/` 只作为只读兼容回退。更新或删除旧共享 agent 会要求先运行迁移脚本。
|
||||
|
||||
## 内部调用与 IM 渠道
|
||||
|
||||
IM channel worker 不是浏览器用户,不持有浏览器 cookie。它们通过 Gateway 内部认证:
|
||||
|
||||
- 请求带 `X-DeerFlow-Internal-Token`。
|
||||
- 同时带匹配的 CSRF cookie/header。
|
||||
- 服务端识别为内部用户,`id="default"`、`system_role="internal"`。
|
||||
|
||||
这意味着 channel 产生的数据默认进入 `default` 用户桶。这个选择适合“平台级 bot 身份”,但不是“每个 IM 用户单独隔离”。如果后续要做到外部 IM 用户隔离,需要把外部 platform user 映射到 DeerFlow user,并让 channel manager 设置对应的 scoped identity。
|
||||
|
||||
## LangGraph-compatible 认证
|
||||
|
||||
Gateway 内嵌 runtime 路径由 `AuthMiddleware` 和 `CSRFMiddleware` 保护。
|
||||
|
||||
仓库仍保留 `app.gateway.langgraph_auth`,用于 LangGraph Server 直连模式:
|
||||
|
||||
- `@auth.authenticate` 校验 JWT cookie、CSRF、用户存在性和 `token_version`。
|
||||
- `@auth.on` 在写入 metadata 时注入 `user_id`,并在读路径返回 `{"user_id": current_user}` 过滤条件。
|
||||
|
||||
这保证 Gateway 路由和 LangGraph-compatible 直连模式使用同一 JWT 语义。
|
||||
|
||||
## 升级与迁移
|
||||
|
||||
从无认证版本升级时,可能存在没有 `user_id` 的历史 thread。
|
||||
|
||||
当前策略:
|
||||
|
||||
1. 首次启动如果没有 admin,只提示访问 `/setup`,不迁移。
|
||||
2. 操作者创建 admin。
|
||||
3. 后续启动时,`_ensure_admin_user()` 找到 admin,并把 LangGraph store 中缺少 `metadata.user_id` 的 thread 迁移到 admin。
|
||||
|
||||
文件系统旧布局迁移由脚本处理:
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
PYTHONPATH=. python scripts/migrate_user_isolation.py --dry-run
|
||||
PYTHONPATH=. python scripts/migrate_user_isolation.py --user-id <target-user-id>
|
||||
```
|
||||
|
||||
迁移脚本覆盖 legacy `memory.json`、`threads/` 和 `agents/` 到 per-user layout。
|
||||
|
||||
## 安全不变量
|
||||
|
||||
必须长期保持的不变量:
|
||||
|
||||
- JWT 只在 HttpOnly cookie 中传输,不出现在响应 JSON。
|
||||
- 任何非 public HTTP 路由都不能只靠“cookie 存在”放行,必须严格验证 JWT。
|
||||
- `token_version` 不匹配必须拒绝,保证改密码 / reset 后旧 session 失效。
|
||||
- 客户端 metadata 中的 `user_id` / `owner_id` 必须剥离。
|
||||
- repository 默认 `AUTO` 必须从当前用户上下文解析,不能静默退化成全局查询。
|
||||
- 只有迁移脚本和 admin CLI 可以显式传 `user_id=None` 绕过隔离。
|
||||
- 本地文件路径必须通过 `Paths` 和 sandbox path validation 解析,不能拼接未校验的用户输入。
|
||||
- 捕获认证、迁移、后台任务异常必须记录日志;不能空 catch。
|
||||
|
||||
## 已知边界
|
||||
|
||||
| 边界 | 当前行为 | 后续方向 |
|
||||
|---|---|---|
|
||||
| 无 admin 时注册普通用户 | 允许注册普通 `user` | 如产品要求先初始化 admin,给 `/register` 加 gate |
|
||||
| 登录限速 | 进程内 dict,单 worker 精确,多 worker 近似 | Redis / DB-backed rate limiter |
|
||||
| OAuth | 端点占位,未实现 | 接入 provider 并统一 `token_version` / role 语义 |
|
||||
| IM 用户隔离 | channel 使用 `default` 内部用户 | 建立外部用户到 DeerFlow user 的映射 |
|
||||
| 绝对 memory path | 显式共享 memory | UI / docs 明确提示 opt-out 风险 |
|
||||
|
||||
## 相关文件
|
||||
|
||||
| 文件 | 职责 |
|
||||
|---|---|
|
||||
| `app/gateway/auth_middleware.py` | 全局认证门、JWT 严格验证、写入 user context |
|
||||
| `app/gateway/csrf_middleware.py` | CSRF double-submit 和 auth Origin 校验 |
|
||||
| `app/gateway/routers/auth.py` | initialize/login/register/logout/me/change-password |
|
||||
| `app/gateway/auth/jwt.py` | JWT 创建与解析 |
|
||||
| `app/gateway/auth/reset_admin.py` | 密码 reset CLI |
|
||||
| `app/gateway/auth/credential_file.py` | 0600 凭据文件写入 |
|
||||
| `app/gateway/authz.py` | 路由权限与 owner check |
|
||||
| `deerflow/runtime/user_context.py` | 当前用户 ContextVar 与 `AUTO` sentinel |
|
||||
| `deerflow/persistence/thread_meta/` | thread metadata owner filter |
|
||||
| `deerflow/config/paths.py` | per-user filesystem layout |
|
||||
| `deerflow/agents/middlewares/thread_data_middleware.py` | run 时解析用户线程目录 |
|
||||
| `deerflow/agents/memory/storage.py` | per-user memory storage |
|
||||
| `deerflow/config/agents_config.py` | per-user custom agents |
|
||||
| `app/channels/manager.py` | IM channel 内部认证调用 |
|
||||
| `scripts/migrate_user_isolation.py` | legacy 数据迁移到 per-user layout |
|
||||
| `.deer-flow/data/deerflow.db` | 统一 SQLite 数据库,包含 users / threads_meta / runs / feedback 等表 |
|
||||
| `.deer-flow/users/{user_id}/agents/{agent_name}/` | 用户自定义 agent 配置、SOUL 和 agent memory |
|
||||
| `.deer-flow/admin_initial_credentials.txt` | `reset_admin` 生成的新凭据文件(0600,读完应删除) |
|
||||
@@ -1,77 +0,0 @@
|
||||
# Docker Test Gap (Section 七 7.4)
|
||||
|
||||
This file documents the only **un-executed** test cases from
|
||||
`backend/docs/AUTH_TEST_PLAN.md` after the full release validation pass.
|
||||
|
||||
## Why this gap exists
|
||||
|
||||
The release validation environment (sg_dev: `10.251.229.92`) **does not have
|
||||
a Docker daemon installed**. The TC-DOCKER cases are container-runtime
|
||||
behavior tests that need an actual Docker engine to spin up
|
||||
`docker/docker-compose.yaml` services.
|
||||
|
||||
```bash
|
||||
$ ssh sg_dev "which docker; docker --version"
|
||||
# (empty)
|
||||
# bash: docker: command not found
|
||||
```
|
||||
|
||||
All other test plan sections were executed against either:
|
||||
- The local dev box (Mac, all services running locally), or
|
||||
- The deployed sg_dev instance (gateway + frontend + nginx via SSH tunnel)
|
||||
|
||||
## Cases not executed
|
||||
|
||||
| Case | Title | What it covers | Why not run |
|
||||
|---|---|---|---|
|
||||
| TC-DOCKER-01 | `deerflow.db` volume persistence | Verify the `DEER_FLOW_HOME` bind mount survives container restart | needs `docker compose up` |
|
||||
| TC-DOCKER-02 | Session persistence across container restart | `AUTH_JWT_SECRET` env var keeps cookies valid after `docker compose down && up` | needs `docker compose down/up` |
|
||||
| TC-DOCKER-03 | Per-worker rate limiter divergence | Confirms in-process `_login_attempts` dict doesn't share state across `gunicorn` workers (4 by default in the compose file); known limitation, documented | needs multi-worker container |
|
||||
| TC-DOCKER-04 | IM channels use internal Gateway auth | Verify Feishu/Slack/Telegram dispatchers attach the process-local internal auth header plus CSRF cookie/header when calling Gateway-compatible LangGraph APIs | needs `docker logs` |
|
||||
| TC-DOCKER-05 | Reset credentials surfacing | `reset_admin` writes a 0600 credential file in `DEER_FLOW_HOME` instead of logging plaintext. The file-based behavior is validated by non-Docker reset tests, so the only Docker-specific gap is verifying the volume mount carries the file out to the host | needs container + host volume |
|
||||
| TC-DOCKER-06 | Docker deploy uses Gateway embedded runtime | `./scripts/deploy.sh` produces a Gateway + frontend + nginx topology (no `langgraph` container); same auth flow as local `make dev` | needs `docker compose up` |
|
||||
|
||||
## Coverage already provided by non-Docker tests
|
||||
|
||||
The **auth-relevant** behavior in each Docker case is already exercised by
|
||||
the test cases that ran on sg_dev or local:
|
||||
|
||||
| Docker case | Auth behavior covered by |
|
||||
|---|---|
|
||||
| TC-DOCKER-01 (volume persistence) | TC-REENT-01 on sg_dev (admin row survives gateway restart) — same SQLite file, just no container layer between |
|
||||
| TC-DOCKER-02 (session persistence) | TC-API-02/03/06 (cookie roundtrip), plus TC-REENT-04 (multi-cookie) — JWT verification is process-state-free, container restart is equivalent to `pkill uvicorn && uv run uvicorn` |
|
||||
| TC-DOCKER-03 (per-worker rate limit) | TC-GW-04 + TC-REENT-09 (single-worker rate limit + 5min expiry). The cross-worker divergence is an architectural property of the in-memory dict; no auth code path differs |
|
||||
| TC-DOCKER-04 (IM channels use internal auth) | Code-level: `app/channels/manager.py` creates the `langgraph_sdk` client with `create_internal_auth_headers()` plus CSRF cookie/header, so channel workers do not rely on browser cookies |
|
||||
| TC-DOCKER-05 (credential surfacing) | `reset_admin` writes `.deer-flow/admin_initial_credentials.txt` with mode 0600 and logs only the path — the only Docker-unique step is whether the bind mount projects this path onto the host, which is a `docker compose` config check, not a runtime behavior change |
|
||||
| TC-DOCKER-06 (Gateway embedded runtime container) | Section 七 7.2 covered by TC-GW-01..05 + Section 二 (Gateway auth flow on sg_dev) — same Gateway code, container is just a packaging change |
|
||||
|
||||
## Reproduction steps when Docker becomes available
|
||||
|
||||
Anyone with `docker` + `docker compose` installed can reproduce the gap by
|
||||
running the test plan section verbatim. Pre-flight:
|
||||
|
||||
```bash
|
||||
# Required on the host
|
||||
docker --version # >=24.x
|
||||
docker compose version # plugin >=2.x
|
||||
|
||||
# Required env var (otherwise sessions reset on every container restart)
|
||||
echo "AUTH_JWT_SECRET=$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')" \
|
||||
>> .env
|
||||
|
||||
# Optional: pin DEER_FLOW_HOME to a stable host path
|
||||
echo "DEER_FLOW_HOME=$HOME/deer-flow-data" >> .env
|
||||
```
|
||||
|
||||
Then run TC-DOCKER-01..06 from the test plan as written.
|
||||
|
||||
## Decision log
|
||||
|
||||
- **Not blocking the release.** The auth-relevant behavior in every Docker
|
||||
case has an already-validated equivalent on bare metal. The gap is purely
|
||||
about *container packaging* details (bind mounts, multi-worker, log
|
||||
collection), not about whether the auth code paths work.
|
||||
- **TC-DOCKER-05 was updated in place** in `AUTH_TEST_PLAN.md` to reflect
|
||||
the current reset flow (`reset_admin` → 0600 credentials file, no log leak).
|
||||
The old "grep 'Password:' in docker logs" expectation would have failed
|
||||
silently and given a false sense of coverage.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,140 +0,0 @@
|
||||
# Authentication Upgrade Guide
|
||||
|
||||
DeerFlow 内置了认证模块。本文档面向从无认证版本升级的用户。
|
||||
|
||||
完整设计见 [AUTH_DESIGN.md](AUTH_DESIGN.md)。
|
||||
|
||||
## 核心概念
|
||||
|
||||
认证模块采用**始终强制**策略:
|
||||
|
||||
- 首次启动时不会自动创建账号;首次访问 `/setup` 时由操作者创建第一个 admin 账号
|
||||
- 认证从一开始就是强制的,无竞争窗口
|
||||
- 已有 admin 后,服务启动时会把历史对话(升级前创建且缺少 `user_id` 的 thread)迁移到 admin 名下
|
||||
- 新数据按用户隔离:thread、workspace/uploads/outputs、memory、自定义 agent 都归属当前用户
|
||||
|
||||
## 升级步骤
|
||||
|
||||
### 1. 更新代码
|
||||
|
||||
```bash
|
||||
git pull origin main
|
||||
cd backend && make install
|
||||
```
|
||||
|
||||
### 2. 首次启动
|
||||
|
||||
```bash
|
||||
make dev
|
||||
```
|
||||
|
||||
如果没有 admin 账号,控制台只会提示:
|
||||
|
||||
```
|
||||
============================================================
|
||||
First boot detected — no admin account exists.
|
||||
Visit /setup to complete admin account creation.
|
||||
============================================================
|
||||
```
|
||||
|
||||
首次启动不会在日志里打印随机密码,也不会写入默认 admin。这样避免启动日志泄露凭据,也避免在操作者创建账号前出现可被猜测的默认身份。
|
||||
|
||||
### 3. 创建 admin
|
||||
|
||||
访问 `http://localhost:2026/setup`,填写邮箱和密码创建第一个 admin 账号。创建成功后会自动登录并进入 workspace。
|
||||
|
||||
如果这是从无认证版本升级,创建 admin 后重启一次服务,让启动迁移把缺少 `user_id` 的历史 thread 归属到 admin。
|
||||
|
||||
### 4. 登录
|
||||
|
||||
后续访问 `http://localhost:2026/login`,使用已创建的邮箱和密码登录。
|
||||
|
||||
### 5. 添加用户(可选)
|
||||
|
||||
其他用户通过 `/login` 页面注册,自动获得 **user** 角色。每个用户只能看到自己的对话、上传文件、输出文件、memory 和自定义 agent。
|
||||
|
||||
## 安全机制
|
||||
|
||||
| 机制 | 说明 |
|
||||
|------|------|
|
||||
| JWT HttpOnly Cookie | Token 不暴露给 JavaScript,防止 XSS 窃取 |
|
||||
| CSRF Double Submit Cookie | 受保护的 POST/PUT/PATCH/DELETE 请求需携带 `X-CSRF-Token`;登录/注册/初始化/登出走 auth 端点 Origin 校验 |
|
||||
| bcrypt 密码哈希 | 密码不以明文存储 |
|
||||
| Thread owner filter | `threads_meta.user_id` 由服务端认证上下文写入,搜索、读取、更新、删除默认按当前用户过滤 |
|
||||
| 文件系统隔离 | 线程数据写入 `{base_dir}/users/{user_id}/threads/{thread_id}/user-data/`,sandbox 内统一映射为 `/mnt/user-data/` |
|
||||
| Memory / agent 隔离 | 用户 memory 和自定义 agent 写入 `{base_dir}/users/{user_id}/...`;旧共享 agent 只作为只读兼容回退 |
|
||||
| HTTPS 自适应 | 检测 `x-forwarded-proto`,自动设置 `Secure` cookie 标志 |
|
||||
|
||||
## 常见操作
|
||||
|
||||
### 忘记密码
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
|
||||
# 重置 admin 密码
|
||||
python -m app.gateway.auth.reset_admin
|
||||
|
||||
# 重置指定用户密码
|
||||
python -m app.gateway.auth.reset_admin --email user@example.com
|
||||
```
|
||||
|
||||
会把新的随机密码写入 `.deer-flow/admin_initial_credentials.txt`,文件权限为 `0600`。命令行只输出文件路径,不输出明文密码。
|
||||
|
||||
### 完全重置
|
||||
|
||||
删除统一 SQLite 数据库,重启后重新访问 `/setup` 创建新 admin:
|
||||
|
||||
```bash
|
||||
rm -f backend/.deer-flow/data/deerflow.db
|
||||
# 重启服务后访问 http://localhost:2026/setup
|
||||
```
|
||||
|
||||
## 数据存储
|
||||
|
||||
| 文件 | 内容 |
|
||||
|------|------|
|
||||
| `.deer-flow/data/deerflow.db` | 统一 SQLite 数据库(users、threads_meta、runs、feedback 等应用数据) |
|
||||
| `.deer-flow/users/{user_id}/threads/{thread_id}/user-data/` | 用户线程的 workspace、uploads、outputs |
|
||||
| `.deer-flow/users/{user_id}/memory.json` | 用户级 memory |
|
||||
| `.deer-flow/users/{user_id}/agents/{agent_name}/` | 用户自定义 agent 配置、SOUL 和 agent memory |
|
||||
| `.deer-flow/admin_initial_credentials.txt` | `reset_admin` 生成的新凭据文件(0600,读完应删除) |
|
||||
| `.env` 中的 `AUTH_JWT_SECRET` | JWT 签名密钥(未设置时自动生成并持久化到 `.deer-flow/.jwt_secret`,重启后 session 保持) |
|
||||
|
||||
### 生产环境建议
|
||||
|
||||
```bash
|
||||
# 生成持久化 JWT 密钥,避免重启后所有用户需重新登录
|
||||
python -c "import secrets; print(secrets.token_urlsafe(32))"
|
||||
# 将输出添加到 .env:
|
||||
# AUTH_JWT_SECRET=<生成的密钥>
|
||||
```
|
||||
|
||||
## API 端点
|
||||
|
||||
| 端点 | 方法 | 说明 |
|
||||
|------|------|------|
|
||||
| `/api/v1/auth/login/local` | POST | 邮箱密码登录(OAuth2 form) |
|
||||
| `/api/v1/auth/register` | POST | 注册新用户(user 角色) |
|
||||
| `/api/v1/auth/logout` | POST | 登出(清除 cookie) |
|
||||
| `/api/v1/auth/me` | GET | 获取当前用户信息 |
|
||||
| `/api/v1/auth/change-password` | POST | 修改密码 |
|
||||
| `/api/v1/auth/setup-status` | GET | 检查 admin 是否存在 |
|
||||
| `/api/v1/auth/initialize` | POST | 首次初始化第一个 admin(仅无 admin 时可调用) |
|
||||
|
||||
## 兼容性
|
||||
|
||||
- **本地开发**(`make dev`):Gateway embedded runtime 完全兼容;无 admin 时访问 `/setup` 初始化
|
||||
- **Gateway embedded runtime**:标准脚本、Docker dev 和生产部署均通过 Gateway 提供认证与 LangGraph-compatible API
|
||||
- **Docker 部署**:完全兼容,`.deer-flow/data/deerflow.db` 需持久化卷挂载
|
||||
- **IM 渠道**(Feishu/Slack/Telegram):通过 Gateway 内部认证通信,使用 `default` 用户桶
|
||||
- **DeerFlowClient**(嵌入式):不经过 HTTP,不受认证影响
|
||||
|
||||
## 故障排查
|
||||
|
||||
| 症状 | 原因 | 解决 |
|
||||
|------|------|------|
|
||||
| 启动后没看到密码 | 当前实现不在启动日志输出密码 | 首次安装访问 `/setup`;忘记密码用 `reset_admin` |
|
||||
| `/login` 自动跳到 `/setup` | 系统还没有 admin | 在 `/setup` 创建第一个 admin |
|
||||
| 登录后 POST 返回 403 | CSRF token 缺失 | 确认前端已更新 |
|
||||
| 重启后需要重新登录 | `.jwt_secret` 文件被删除且 `.env` 未设置 `AUTH_JWT_SECRET` | 在 `.env` 中设置固定密钥 |
|
||||
@@ -1,154 +0,0 @@
|
||||
# Blocking IO detection usage and maintenance
|
||||
|
||||
This document describes how to use and maintain DeerFlow backend blocking-IO
|
||||
detection for async event-loop safety.
|
||||
|
||||
The goal is narrow: find and prevent synchronous IO from blocking backend
|
||||
async event-loop paths. Static and runtime detection are complementary, but
|
||||
they have different jobs.
|
||||
|
||||
## Static detector
|
||||
|
||||
The static detector is the discovery tool. It scans backend source code and
|
||||
reports candidate blocking-IO call sites that may need human review.
|
||||
|
||||
Run it from the repository root:
|
||||
|
||||
```bash
|
||||
make detect-blocking-io
|
||||
```
|
||||
|
||||
Or from `backend/`:
|
||||
|
||||
```bash
|
||||
make detect-blocking-io
|
||||
```
|
||||
|
||||
The report is written to:
|
||||
|
||||
```text
|
||||
.deer-flow/blocking-io-findings.json
|
||||
```
|
||||
|
||||
Use this output for review and triage. A static finding is a candidate, not
|
||||
proof that production blocks the event loop at runtime. The current static
|
||||
rules are intentionally broad; prefer triaging existing output before adding
|
||||
new static rules.
|
||||
|
||||
Add a static rule only when review finds a recurring high-risk blocking
|
||||
pattern that is invisible to the current detector.
|
||||
|
||||
## Runtime detector
|
||||
|
||||
The runtime detector is the CI regression guard. It uses Blockbuster to fail a
|
||||
focused test when code under `app.*` or `deerflow.*` performs blocking IO on
|
||||
the asyncio event-loop thread.
|
||||
|
||||
Run it from `backend/`:
|
||||
|
||||
```bash
|
||||
make test-blocking-io
|
||||
```
|
||||
|
||||
The runtime gate starts from confirmed production bugs and protects those
|
||||
paths from regressing. It does not prove that the entire backend is free of
|
||||
blocking IO; it only covers the production paths exercised by
|
||||
`backend/tests/blocking_io/`.
|
||||
|
||||
## Maintenance workflow
|
||||
|
||||
Use the static detector to find candidates, then use review to decide which
|
||||
async production paths are worth protecting in CI.
|
||||
|
||||
The normal workflow is:
|
||||
|
||||
1. Run the static detector to find backend blocking-IO candidates.
|
||||
2. Use human review to pick high-risk production async paths.
|
||||
3. Add or update a focused runtime anchor in `backend/tests/blocking_io/`.
|
||||
4. Let CI prevent that path from regressing.
|
||||
|
||||
Runtime detection has two maintenance paths.
|
||||
|
||||
### Add a runtime rule
|
||||
|
||||
Add a runtime rule when Blockbuster's default rules do not cover a generic
|
||||
blocking primitive used by production code.
|
||||
|
||||
Rules belong in:
|
||||
|
||||
```text
|
||||
backend/tests/support/detectors/blocking_io_runtime.py
|
||||
```
|
||||
|
||||
Add them to `_PROJECT_BLOCKING_RULES`, not directly inside individual tests.
|
||||
Keeping rules centralized makes it clear which extra primitives DeerFlow
|
||||
expects Blockbuster to catch.
|
||||
|
||||
Example shape:
|
||||
|
||||
```python
|
||||
import subprocess
|
||||
|
||||
from blockbuster import BlockBusterFunction
|
||||
|
||||
_PROJECT_BLOCKING_RULES = (
|
||||
(
|
||||
"subprocess.Popen.__init__",
|
||||
BlockBusterFunction(
|
||||
subprocess.Popen,
|
||||
"__init__",
|
||||
scanned_modules=["app", "deerflow"],
|
||||
),
|
||||
),
|
||||
)
|
||||
```
|
||||
|
||||
Do not add a runtime rule just because a business path is not tested. A rule
|
||||
only expands what Blockbuster can intercept after code runs.
|
||||
|
||||
### Add a runtime anchor
|
||||
|
||||
Add a runtime anchor when a high-risk async production path should be protected
|
||||
by CI but no existing `backend/tests/blocking_io/` test executes it.
|
||||
|
||||
Anchors belong in:
|
||||
|
||||
```text
|
||||
backend/tests/blocking_io/
|
||||
```
|
||||
|
||||
A good anchor should:
|
||||
|
||||
- Call the real production async entry point.
|
||||
- Avoid bypassing the blocking surface with test-only `asyncio.to_thread`
|
||||
wrappers.
|
||||
- Use real local filesystem inputs when the bug shape is filesystem IO.
|
||||
- Mock only the external dependency boundary, such as a network service or
|
||||
third-party saver class.
|
||||
- Fail if a future change moves the blocking operation back onto the event
|
||||
loop.
|
||||
|
||||
Avoid testing only the low-level helper unless that helper is the production
|
||||
async entry point. The runtime gate is most useful when it protects the caller
|
||||
that production actually executes.
|
||||
|
||||
## Current runtime coverage
|
||||
|
||||
The runtime anchors protect confirmed blocking-IO bug shapes:
|
||||
|
||||
- SQLite checkpointer setup, including path resolution and parent-directory
|
||||
creation.
|
||||
- Subagent skill metadata loading through `SubagentExecutor._load_skills()`.
|
||||
- `JsonlRunEventStore` async API (`put` / `list_*` / `delete_*`): the JSONL
|
||||
run-event backend offloads its synchronous file IO via `asyncio.to_thread`
|
||||
(fix #3084); this anchor drives the real async API under the gate so any
|
||||
blocking IO reintroduced on the loop fails, not only removal of one
|
||||
`to_thread` call.
|
||||
- `UploadsMiddleware.before_agent` uploads-directory scan: a sync-only middleware
|
||||
hook runs on the event loop under async graph execution, so the scan is
|
||||
offloaded via `abefore_agent` + `run_in_executor`.
|
||||
- Gate health checks: Blockbuster catches unoffloaded calls, opt-out works, and
|
||||
patches are restored after exceptions.
|
||||
|
||||
As static detection and review identify more high-risk async paths, add new
|
||||
runtime anchors incrementally.
|
||||
@@ -36,7 +36,6 @@ models:
|
||||
- OpenAI (`langchain_openai:ChatOpenAI`)
|
||||
- Anthropic (`langchain_anthropic:ChatAnthropic`)
|
||||
- DeepSeek (`langchain_deepseek:ChatDeepSeek`)
|
||||
- Xiaomi MiMo (`deerflow.models.patched_mimo:PatchedChatMiMo`)
|
||||
- Claude Code OAuth (`deerflow.models.claude_provider:ClaudeChatModel`)
|
||||
- Codex CLI (`deerflow.models.openai_codex_provider:CodexChatModel`)
|
||||
- Any LangChain-compatible provider
|
||||
@@ -95,35 +94,25 @@ models:
|
||||
thinking:
|
||||
type: enabled
|
||||
|
||||
- name: minimax-m3
|
||||
display_name: MiniMax M3
|
||||
- name: minimax-m2.5
|
||||
display_name: MiniMax M2.5
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: MiniMax-M3
|
||||
model: MiniMax-M2.5
|
||||
api_key: $MINIMAX_API_KEY
|
||||
base_url: https://api.minimax.io/v1
|
||||
max_tokens: 4096
|
||||
temperature: 1.0 # MiniMax requires temperature in (0.0, 1.0]
|
||||
supports_vision: true
|
||||
|
||||
- name: minimax-m2.7
|
||||
display_name: MiniMax M2.7
|
||||
- name: minimax-m2.5-highspeed
|
||||
display_name: MiniMax M2.5 Highspeed
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: MiniMax-M2.7
|
||||
model: MiniMax-M2.5-highspeed
|
||||
api_key: $MINIMAX_API_KEY
|
||||
base_url: https://api.minimax.io/v1
|
||||
max_tokens: 4096
|
||||
temperature: 1.0 # MiniMax requires temperature in (0.0, 1.0]
|
||||
supports_vision: false # M2.7 is text-only; M3 supports vision
|
||||
|
||||
- name: minimax-m2.7-highspeed
|
||||
display_name: MiniMax M2.7 Highspeed
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: MiniMax-M2.7-highspeed
|
||||
api_key: $MINIMAX_API_KEY
|
||||
base_url: https://api.minimax.io/v1
|
||||
max_tokens: 4096
|
||||
temperature: 1.0 # MiniMax requires temperature in (0.0, 1.0]
|
||||
supports_vision: false # M2.7 is text-only; M3 supports vision
|
||||
supports_vision: true
|
||||
- name: openrouter-gemini-2.5-flash
|
||||
display_name: Gemini 2.5 Flash (OpenRouter)
|
||||
use: langchain_openai:ChatOpenAI
|
||||
@@ -177,37 +166,6 @@ models:
|
||||
|
||||
For Gemini accessed **without** thinking (e.g. via OpenRouter where thinking is not activated), the plain `langchain_openai:ChatOpenAI` with `supports_thinking: false` is sufficient and no patch is needed.
|
||||
|
||||
**MiMo with thinking via OpenAI-compatible API**:
|
||||
|
||||
MiMo returns `reasoning_content` on assistant messages in thinking mode. In multi-turn agent conversations with tool calls, subsequent requests must preserve that historical `reasoning_content` on assistant messages or the MiMo API can return HTTP 400. Standard `langchain_openai:ChatOpenAI` drops this provider-specific field, so use `deerflow.models.patched_mimo:PatchedChatMiMo`:
|
||||
|
||||
For pay-as-you-go API keys (`sk-...`), use `https://api.xiaomimimo.com/v1`. For Token Plan keys (`tp-...`), use the regional Token Plan Base URL shown in the MiMo console, such as `https://token-plan-cn.xiaomimimo.com/v1`. MiMo documents these key types as separate and non-interchangeable.
|
||||
|
||||
`PatchedChatMiMo` is model-id agnostic. Use it for every MiMo thinking model entry you configure, including model entries referenced by `subagents.*.model` overrides (for example `mimo-v2.5-pro`, `mimo-v2.5`, `mimo-v2-pro`, `mimo-v2-omni`, or `mimo-v2-flash`).
|
||||
|
||||
```yaml
|
||||
models:
|
||||
- name: mimo-v2.5-pro
|
||||
display_name: MiMo V2.5 Pro
|
||||
use: deerflow.models.patched_mimo:PatchedChatMiMo
|
||||
model: mimo-v2.5-pro
|
||||
api_key: $MIMO_API_KEY
|
||||
base_url: https://api.xiaomimimo.com/v1
|
||||
max_tokens: 8192
|
||||
supports_thinking: true
|
||||
supports_vision: false
|
||||
when_thinking_enabled:
|
||||
extra_body:
|
||||
thinking:
|
||||
type: enabled
|
||||
when_thinking_disabled:
|
||||
extra_body:
|
||||
thinking:
|
||||
type: disabled
|
||||
```
|
||||
|
||||
`PatchedChatMiMo` preserves MiMo's `choices[].message.reasoning_content`, streaming `delta.reasoning_content`, and request-history assistant `reasoning_content` fields. It does not reuse the DeepSeek provider.
|
||||
|
||||
### Tool Groups
|
||||
|
||||
Organize tools into logical groups:
|
||||
@@ -301,8 +259,6 @@ sandbox:
|
||||
|
||||
When you configure `sandbox.mounts`, DeerFlow exposes those `container_path` values in the agent prompt so the agent can discover and operate on mounted directories directly instead of assuming everything must live under `/mnt/user-data`.
|
||||
|
||||
For bare-metal Docker sandbox runs that use localhost, DeerFlow binds the sandbox HTTP port to `127.0.0.1` by default so it is not exposed on every host interface. Docker-outside-of-Docker deployments that connect through `host.docker.internal` keep the broad legacy bind for compatibility. Set `DEER_FLOW_SANDBOX_BIND_HOST` explicitly if your deployment needs a different bind address.
|
||||
|
||||
### Skills
|
||||
|
||||
Configure the skills directory for specialized workflows:
|
||||
@@ -361,19 +317,13 @@ models:
|
||||
- `OPENAI_API_KEY` - OpenAI API key
|
||||
- `ANTHROPIC_API_KEY` - Anthropic API key
|
||||
- `DEEPSEEK_API_KEY` - DeepSeek API key
|
||||
- `MIMO_API_KEY` - Xiaomi MiMo API key
|
||||
- `NOVITA_API_KEY` - Novita API key (OpenAI-compatible endpoint)
|
||||
- `TAVILY_API_KEY` - Tavily search API key
|
||||
- `DEER_FLOW_PROJECT_ROOT` - Project root for relative runtime paths
|
||||
- `DEER_FLOW_CONFIG_PATH` - Custom config file path
|
||||
- `DEER_FLOW_EXTENSIONS_CONFIG_PATH` - Custom extensions config file path
|
||||
- `DEER_FLOW_HOME` - Runtime state directory (defaults to `.deer-flow` under the project root)
|
||||
- `DEER_FLOW_SKILLS_PATH` - Skills directory when `skills.path` is omitted
|
||||
- `GATEWAY_ENABLE_DOCS` - Set to `false` to disable Swagger UI (`/docs`), ReDoc (`/redoc`), and OpenAPI schema (`/openapi.json`) endpoints (default: `true`)
|
||||
|
||||
## Configuration Location
|
||||
|
||||
The configuration file should be placed in the **project root directory** (`deer-flow/config.yaml`). Set `DEER_FLOW_PROJECT_ROOT` when the process may start from another working directory, or set `DEER_FLOW_CONFIG_PATH` to point at a specific file.
|
||||
The configuration file should be placed in the **project root directory** (`deer-flow/config.yaml`), not in the backend directory.
|
||||
|
||||
## Configuration Priority
|
||||
|
||||
@@ -381,12 +331,12 @@ DeerFlow searches for configuration in this order:
|
||||
|
||||
1. Path specified in code via `config_path` argument
|
||||
2. Path from `DEER_FLOW_CONFIG_PATH` environment variable
|
||||
3. `config.yaml` under `DEER_FLOW_PROJECT_ROOT`, or under the current working directory when `DEER_FLOW_PROJECT_ROOT` is unset
|
||||
4. Legacy backend/repository-root locations for monorepo compatibility
|
||||
3. `config.yaml` in current working directory (typically `backend/` when running)
|
||||
4. `config.yaml` in parent directory (project root: `deer-flow/`)
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Place `config.yaml` in project root** - Set `DEER_FLOW_PROJECT_ROOT` if the runtime starts elsewhere
|
||||
1. **Place `config.yaml` in project root** - Not in `backend/` directory
|
||||
2. **Never commit `config.yaml`** - It's already in `.gitignore`
|
||||
3. **Use environment variables for secrets** - Don't hardcode API keys
|
||||
4. **Keep `config.example.yaml` updated** - Document all new options
|
||||
@@ -397,7 +347,7 @@ DeerFlow searches for configuration in this order:
|
||||
|
||||
### "Config file not found"
|
||||
- Ensure `config.yaml` exists in the **project root** directory (`deer-flow/config.yaml`)
|
||||
- If the runtime starts outside the project root, set `DEER_FLOW_PROJECT_ROOT`
|
||||
- The backend searches parent directory by default, so root location is preferred
|
||||
- Alternatively, set `DEER_FLOW_CONFIG_PATH` environment variable to custom location
|
||||
|
||||
### "Invalid API key"
|
||||
@@ -407,7 +357,7 @@ DeerFlow searches for configuration in this order:
|
||||
### "Skills not loading"
|
||||
- Check that `deer-flow/skills/` directory exists
|
||||
- Verify skills have valid `SKILL.md` files
|
||||
- Check `skills.path` or `DEER_FLOW_SKILLS_PATH` if using a custom path
|
||||
- Check `skills.path` configuration if using custom path
|
||||
|
||||
### "Docker sandbox fails to start"
|
||||
- Ensure Docker is running
|
||||
|
||||
@@ -2,12 +2,12 @@
|
||||
|
||||
## 概述
|
||||
|
||||
DeerFlow 后端提供了完整的文件上传功能,支持多文件上传,并可选地将 Office 文档和 PDF 转换为 Markdown 格式。
|
||||
DeerFlow 后端提供了完整的文件上传功能,支持多文件上传,并自动将 Office 文档和 PDF 转换为 Markdown 格式。
|
||||
|
||||
## 功能特性
|
||||
|
||||
- ✅ 支持多文件同时上传
|
||||
- ✅ 可选地转换文档为 Markdown(PDF、PPT、Excel、Word)
|
||||
- ✅ 自动转换文档为 Markdown(PDF、PPT、Excel、Word)
|
||||
- ✅ 文件存储在线程隔离的目录中
|
||||
- ✅ Agent 自动感知已上传的文件
|
||||
- ✅ 支持文件列表查询和删除
|
||||
@@ -22,8 +22,6 @@ POST /api/threads/{thread_id}/uploads
|
||||
**请求体:** `multipart/form-data`
|
||||
- `files`: 一个或多个文件
|
||||
|
||||
网关会在应用层限制上传规模,默认最多 10 个文件、单文件 50 MiB、单次请求总计 100 MiB。可通过 `config.yaml` 的 `uploads.max_files`、`uploads.max_file_size`、`uploads.max_total_size` 调整;前端会读取同一组限制并在选择文件时提示,超过限制时后端返回 `413 Payload Too Large`。
|
||||
|
||||
**响应:**
|
||||
```json
|
||||
{
|
||||
@@ -50,23 +48,7 @@ POST /api/threads/{thread_id}/uploads
|
||||
- `virtual_path`: Agent 在沙箱中使用的虚拟路径
|
||||
- `artifact_url`: 前端通过 HTTP 访问文件的 URL
|
||||
|
||||
### 2. 查询上传限制
|
||||
```
|
||||
GET /api/threads/{thread_id}/uploads/limits
|
||||
```
|
||||
|
||||
返回网关当前生效的上传限制,供前端在用户选择文件前提示和拦截。
|
||||
|
||||
**响应:**
|
||||
```json
|
||||
{
|
||||
"max_files": 10,
|
||||
"max_file_size": 52428800,
|
||||
"max_total_size": 104857600
|
||||
}
|
||||
```
|
||||
|
||||
### 3. 列出已上传文件
|
||||
### 2. 列出已上传文件
|
||||
```
|
||||
GET /api/threads/{thread_id}/uploads/list
|
||||
```
|
||||
@@ -89,7 +71,7 @@ GET /api/threads/{thread_id}/uploads/list
|
||||
}
|
||||
```
|
||||
|
||||
### 4. 删除文件
|
||||
### 3. 删除文件
|
||||
```
|
||||
DELETE /api/threads/{thread_id}/uploads/{filename}
|
||||
```
|
||||
@@ -104,7 +86,7 @@ DELETE /api/threads/{thread_id}/uploads/{filename}
|
||||
|
||||
## 支持的文档格式
|
||||
|
||||
以下格式在显式启用 `uploads.auto_convert_documents: true` 时会自动转换为 Markdown:
|
||||
以下格式会自动转换为 Markdown:
|
||||
- PDF (`.pdf`)
|
||||
- PowerPoint (`.ppt`, `.pptx`)
|
||||
- Excel (`.xls`, `.xlsx`)
|
||||
@@ -112,8 +94,6 @@ DELETE /api/threads/{thread_id}/uploads/{filename}
|
||||
|
||||
转换后的 Markdown 文件会保存在同一目录下,文件名为原文件名 + `.md` 扩展名。
|
||||
|
||||
默认情况下,自动转换是关闭的,以避免在网关主机上对不受信任的 Office/PDF 上传执行解析。只有在受信任部署中明确接受此风险时,才应将 `uploads.auto_convert_documents` 设置为 `true`。
|
||||
|
||||
## Agent 集成
|
||||
|
||||
### 自动文件列举
|
||||
@@ -227,7 +207,6 @@ backend/.deer-flow/threads/
|
||||
- 最大文件大小:100MB(可在 nginx.conf 中配置 `client_max_body_size`)
|
||||
- 文件名安全性:系统会自动验证文件路径,防止目录遍历攻击
|
||||
- 线程隔离:每个线程的上传文件相互隔离,无法跨线程访问
|
||||
- 自动文档转换默认关闭;如需启用,需在 `config.yaml` 中显式设置 `uploads.auto_convert_documents: true`
|
||||
|
||||
## 技术实现
|
||||
|
||||
|
||||
@@ -296,7 +296,7 @@ These are the tool names your provider will see in `request.tool_name`:
|
||||
| `web_search` | Web search query |
|
||||
| `web_fetch` | Fetch URL content |
|
||||
| `image_search` | Image search |
|
||||
| `present_files` | Present file to user |
|
||||
| `present_file` | Present file to user |
|
||||
| `view_image` | Display image |
|
||||
| `ask_clarification` | Ask user a question |
|
||||
| `task` | Delegate to subagent |
|
||||
|
||||
@@ -0,0 +1,343 @@
|
||||
# DeerFlow 后端拆分设计文档:Harness + App
|
||||
|
||||
> 状态:Draft
|
||||
> 作者:DeerFlow Team
|
||||
> 日期:2026-03-13
|
||||
|
||||
## 1. 背景与动机
|
||||
|
||||
DeerFlow 后端当前是一个单一 Python 包(`src.*`),包含了从底层 agent 编排到上层用户产品的所有代码。随着项目发展,这种结构带来了几个问题:
|
||||
|
||||
- **复用困难**:其他产品(CLI 工具、Slack bot、第三方集成)想用 agent 能力,必须依赖整个后端,包括 FastAPI、IM SDK 等不需要的依赖
|
||||
- **职责模糊**:agent 编排逻辑和用户产品逻辑混在同一个 `src/` 下,边界不清晰
|
||||
- **依赖膨胀**:LangGraph Server 运行时不需要 FastAPI/uvicorn/Slack SDK,但当前必须安装全部依赖
|
||||
|
||||
本文档提出将后端拆分为两部分:**deerflow-harness**(可发布的 agent 框架包)和 **app**(不打包的用户产品代码)。
|
||||
|
||||
## 2. 核心概念
|
||||
|
||||
### 2.1 Harness(线束/框架层)
|
||||
|
||||
Harness 是 agent 的构建与编排框架,回答 **"如何构建和运行 agent"** 的问题:
|
||||
|
||||
- Agent 工厂与生命周期管理
|
||||
- Middleware pipeline
|
||||
- 工具系统(内置工具 + MCP + 社区工具)
|
||||
- 沙箱执行环境
|
||||
- 子 agent 委派
|
||||
- 记忆系统
|
||||
- 技能加载与注入
|
||||
- 模型工厂
|
||||
- 配置系统
|
||||
|
||||
**Harness 是一个可发布的 Python 包**(`deerflow-harness`),可以独立安装和使用。
|
||||
|
||||
**Harness 的设计原则**:对上层应用完全无感知。它不知道也不关心谁在调用它——可以是 Web App、CLI、Slack Bot、或者一个单元测试。
|
||||
|
||||
### 2.2 App(应用层)
|
||||
|
||||
App 是面向用户的产品代码,回答 **"如何将 agent 呈现给用户"** 的问题:
|
||||
|
||||
- Gateway API(FastAPI REST 接口)
|
||||
- IM Channels(飞书、Slack、Telegram 集成)
|
||||
- Custom Agent 的 CRUD 管理
|
||||
- 文件上传/下载的 HTTP 接口
|
||||
|
||||
**App 不打包、不发布**,它是 DeerFlow 项目内部的应用代码,直接运行。
|
||||
|
||||
**App 依赖 Harness,但 Harness 不依赖 App。**
|
||||
|
||||
### 2.3 边界划分
|
||||
|
||||
| 模块 | 归属 | 说明 |
|
||||
|------|------|------|
|
||||
| `config/` | Harness | 配置系统是基础设施 |
|
||||
| `reflection/` | Harness | 动态模块加载工具 |
|
||||
| `utils/` | Harness | 通用工具函数 |
|
||||
| `agents/` | Harness | Agent 工厂、middleware、state、memory |
|
||||
| `subagents/` | Harness | 子 agent 委派系统 |
|
||||
| `sandbox/` | Harness | 沙箱执行环境 |
|
||||
| `tools/` | Harness | 工具注册与发现 |
|
||||
| `mcp/` | Harness | MCP 协议集成 |
|
||||
| `skills/` | Harness | 技能加载、解析、定义 schema |
|
||||
| `models/` | Harness | LLM 模型工厂 |
|
||||
| `community/` | Harness | 社区工具(tavily、jina 等) |
|
||||
| `client.py` | Harness | 嵌入式 Python 客户端 |
|
||||
| `gateway/` | App | FastAPI REST API |
|
||||
| `channels/` | App | IM 平台集成 |
|
||||
|
||||
**关于 Custom Agents**:agent 定义格式(`config.yaml` + `SOUL.md` schema)由 Harness 层的 `config/agents_config.py` 定义,但文件的存储、CRUD、发现机制由 App 层的 `gateway/routers/agents.py` 负责。
|
||||
|
||||
## 3. 目标架构
|
||||
|
||||
### 3.1 目录结构
|
||||
|
||||
```
|
||||
backend/
|
||||
├── packages/
|
||||
│ └── harness/
|
||||
│ ├── pyproject.toml # deerflow-harness 包定义
|
||||
│ └── deerflow/ # Python 包根(import 前缀: deerflow.*)
|
||||
│ ├── __init__.py
|
||||
│ ├── config/
|
||||
│ ├── reflection/
|
||||
│ ├── utils/
|
||||
│ ├── agents/
|
||||
│ │ ├── lead_agent/
|
||||
│ │ ├── middlewares/
|
||||
│ │ ├── memory/
|
||||
│ │ ├── checkpointer/
|
||||
│ │ └── thread_state.py
|
||||
│ ├── subagents/
|
||||
│ ├── sandbox/
|
||||
│ ├── tools/
|
||||
│ ├── mcp/
|
||||
│ ├── skills/
|
||||
│ ├── models/
|
||||
│ ├── community/
|
||||
│ └── client.py
|
||||
├── app/ # 不打包(import 前缀: app.*)
|
||||
│ ├── __init__.py
|
||||
│ ├── gateway/
|
||||
│ │ ├── __init__.py
|
||||
│ │ ├── app.py
|
||||
│ │ ├── config.py
|
||||
│ │ ├── path_utils.py
|
||||
│ │ └── routers/
|
||||
│ └── channels/
|
||||
│ ├── __init__.py
|
||||
│ ├── base.py
|
||||
│ ├── manager.py
|
||||
│ ├── service.py
|
||||
│ ├── store.py
|
||||
│ ├── message_bus.py
|
||||
│ ├── feishu.py
|
||||
│ ├── slack.py
|
||||
│ └── telegram.py
|
||||
├── pyproject.toml # uv workspace root
|
||||
├── langgraph.json
|
||||
├── tests/
|
||||
├── docs/
|
||||
└── Makefile
|
||||
```
|
||||
|
||||
### 3.2 Import 规则
|
||||
|
||||
两个层使用不同的 import 前缀,职责边界一目了然:
|
||||
|
||||
```python
|
||||
# ---------------------------------------------------------------
|
||||
# Harness 内部互相引用(deerflow.* 前缀)
|
||||
# ---------------------------------------------------------------
|
||||
from deerflow.agents import make_lead_agent
|
||||
from deerflow.models import create_chat_model
|
||||
from deerflow.config import get_app_config
|
||||
from deerflow.tools import get_available_tools
|
||||
|
||||
# ---------------------------------------------------------------
|
||||
# App 内部互相引用(app.* 前缀)
|
||||
# ---------------------------------------------------------------
|
||||
from app.gateway.app import app
|
||||
from app.gateway.routers.uploads import upload_files
|
||||
from app.channels.service import start_channel_service
|
||||
|
||||
# ---------------------------------------------------------------
|
||||
# App 调用 Harness(单向依赖,Harness 永远不 import app)
|
||||
# ---------------------------------------------------------------
|
||||
from deerflow.agents import make_lead_agent
|
||||
from deerflow.models import create_chat_model
|
||||
from deerflow.skills import load_skills
|
||||
from deerflow.config.extensions_config import get_extensions_config
|
||||
```
|
||||
|
||||
**App 调用 Harness 示例 — Gateway 中启动 agent**:
|
||||
|
||||
```python
|
||||
# app/gateway/routers/chat.py
|
||||
from deerflow.agents.lead_agent.agent import make_lead_agent
|
||||
from deerflow.models import create_chat_model
|
||||
from deerflow.config import get_app_config
|
||||
|
||||
async def create_chat_session(thread_id: str, model_name: str):
|
||||
config = get_app_config()
|
||||
model = create_chat_model(name=model_name)
|
||||
agent = make_lead_agent(config=...)
|
||||
# ... 使用 agent 处理用户消息
|
||||
```
|
||||
|
||||
**App 调用 Harness 示例 — Channel 中查询 skills**:
|
||||
|
||||
```python
|
||||
# app/channels/manager.py
|
||||
from deerflow.skills import load_skills
|
||||
from deerflow.agents.memory.updater import get_memory_data
|
||||
|
||||
def handle_status_command():
|
||||
skills = load_skills(enabled_only=True)
|
||||
memory = get_memory_data()
|
||||
return f"Skills: {len(skills)}, Memory facts: {len(memory.get('facts', []))}"
|
||||
```
|
||||
|
||||
**禁止方向**:Harness 代码中绝不能出现 `from app.` 或 `import app.`。
|
||||
|
||||
### 3.3 为什么 App 不打包
|
||||
|
||||
| 方面 | 打包(放 packages/ 下) | 不打包(放 backend/app/) |
|
||||
|------|------------------------|--------------------------|
|
||||
| 命名空间 | 需要 pkgutil `extend_path` 合并,或独立前缀 | 天然独立,`app.*` vs `deerflow.*` |
|
||||
| 发布需求 | 没有——App 是项目内部代码 | 不需要 pyproject.toml |
|
||||
| 复杂度 | 需要管理两个包的构建、版本、依赖声明 | 直接运行,零额外配置 |
|
||||
| 运行方式 | `pip install deerflow-app` | `PYTHONPATH=. uvicorn app.gateway.app:app` |
|
||||
|
||||
App 的唯一消费者是 DeerFlow 项目自身,没有独立发布的需求。放在 `backend/app/` 下作为普通 Python 包,通过 `PYTHONPATH` 或 editable install 让 Python 找到即可。
|
||||
|
||||
### 3.4 依赖关系
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────┐
|
||||
│ app/ (不打包,直接运行) │
|
||||
│ ├── fastapi, uvicorn │
|
||||
│ ├── slack-sdk, lark-oapi, ... │
|
||||
│ └── import deerflow.* │
|
||||
└──────────────┬──────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────┐
|
||||
│ deerflow-harness (可发布的包) │
|
||||
│ ├── langgraph, langchain │
|
||||
│ ├── markitdown, pydantic, ... │
|
||||
│ └── 零 app 依赖 │
|
||||
└─────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**依赖分类**:
|
||||
|
||||
| 分类 | 依赖包 |
|
||||
|------|--------|
|
||||
| Harness only | agent-sandbox, langchain*, langgraph*, markdownify, markitdown, pydantic, pyyaml, readabilipy, tavily-python, firecrawl-py, tiktoken, ddgs, duckdb, httpx, kubernetes, dotenv |
|
||||
| App only | fastapi, uvicorn, sse-starlette, python-multipart, lark-oapi, slack-sdk, python-telegram-bot, markdown-to-mrkdwn |
|
||||
| Shared | langgraph-sdk(channels 用 HTTP client), pydantic, httpx |
|
||||
|
||||
### 3.5 Workspace 配置
|
||||
|
||||
`backend/pyproject.toml`(workspace root):
|
||||
|
||||
```toml
|
||||
[project]
|
||||
name = "deer-flow"
|
||||
version = "0.1.0"
|
||||
requires-python = ">=3.12"
|
||||
dependencies = ["deerflow-harness"]
|
||||
|
||||
[dependency-groups]
|
||||
dev = ["pytest>=8.0.0", "ruff>=0.14.11"]
|
||||
# App 的额外依赖(fastapi 等)也声明在 workspace root,因为 app 不打包
|
||||
app = ["fastapi", "uvicorn", "sse-starlette", "python-multipart"]
|
||||
channels = ["lark-oapi", "slack-sdk", "python-telegram-bot"]
|
||||
|
||||
[tool.uv.workspace]
|
||||
members = ["packages/harness"]
|
||||
|
||||
[tool.uv.sources]
|
||||
deerflow-harness = { workspace = true }
|
||||
```
|
||||
|
||||
## 4. 当前的跨层依赖问题
|
||||
|
||||
在拆分之前,需要先解决 `client.py` 中两处从 harness 到 app 的反向依赖:
|
||||
|
||||
### 4.1 `_validate_skill_frontmatter`
|
||||
|
||||
```python
|
||||
# client.py — harness 导入了 app 层代码
|
||||
from src.gateway.routers.skills import _validate_skill_frontmatter
|
||||
```
|
||||
|
||||
**解决方案**:将该函数提取到 `deerflow/skills/validation.py`。这是一个纯逻辑函数(解析 YAML frontmatter、校验字段),与 FastAPI 无关。
|
||||
|
||||
### 4.2 `CONVERTIBLE_EXTENSIONS` + `convert_file_to_markdown`
|
||||
|
||||
```python
|
||||
# client.py — harness 导入了 app 层代码
|
||||
from src.gateway.routers.uploads import CONVERTIBLE_EXTENSIONS, convert_file_to_markdown
|
||||
```
|
||||
|
||||
**解决方案**:将它们提取到 `deerflow/utils/file_conversion.py`。仅依赖 `markitdown` + `pathlib`,是通用工具函数。
|
||||
|
||||
## 5. 基础设施变更
|
||||
|
||||
### 5.1 LangGraph Server
|
||||
|
||||
LangGraph Server 只需要 harness 包。`langgraph.json` 更新:
|
||||
|
||||
```json
|
||||
{
|
||||
"dependencies": ["./packages/harness"],
|
||||
"graphs": {
|
||||
"lead_agent": "deerflow.agents:make_lead_agent"
|
||||
},
|
||||
"checkpointer": {
|
||||
"path": "./packages/harness/deerflow/agents/checkpointer/async_provider.py:make_checkpointer"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 5.2 Gateway API
|
||||
|
||||
```bash
|
||||
# serve.sh / Makefile
|
||||
# PYTHONPATH 包含 backend/ 根目录,使 app.* 和 deerflow.* 都能被找到
|
||||
PYTHONPATH=. uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001
|
||||
```
|
||||
|
||||
### 5.3 Nginx
|
||||
|
||||
无需变更(只做 URL 路由,不涉及 Python 模块路径)。
|
||||
|
||||
### 5.4 Docker
|
||||
|
||||
Dockerfile 中的 module 引用从 `src.` 改为 `deerflow.` / `app.`,`COPY` 命令需覆盖 `packages/` 和 `app/` 目录。
|
||||
|
||||
## 6. 实施计划
|
||||
|
||||
分 3 个 PR 递进执行:
|
||||
|
||||
### PR 1:提取共享工具函数(Low Risk)
|
||||
|
||||
1. 创建 `src/skills/validation.py`,从 `gateway/routers/skills.py` 提取 `_validate_skill_frontmatter`
|
||||
2. 创建 `src/utils/file_conversion.py`,从 `gateway/routers/uploads.py` 提取文件转换逻辑
|
||||
3. 更新 `client.py`、`gateway/routers/skills.py`、`gateway/routers/uploads.py` 的 import
|
||||
4. 运行全部测试确认无回归
|
||||
|
||||
### PR 2:Rename + 物理拆分(High Risk,原子操作)
|
||||
|
||||
1. 创建 `packages/harness/` 目录,创建 `pyproject.toml`
|
||||
2. `git mv` 将 harness 相关模块从 `src/` 移入 `packages/harness/deerflow/`
|
||||
3. `git mv` 将 app 相关模块从 `src/` 移入 `app/`
|
||||
4. 全局替换 import:
|
||||
- harness 模块:`src.*` → `deerflow.*`(所有 `.py` 文件、`langgraph.json`、测试、文档)
|
||||
- app 模块:`src.gateway.*` → `app.gateway.*`、`src.channels.*` → `app.channels.*`
|
||||
5. 更新 workspace root `pyproject.toml`
|
||||
6. 更新 `langgraph.json`、`Makefile`、`Dockerfile`
|
||||
7. `uv sync` + 全部测试 + 手动验证服务启动
|
||||
|
||||
### PR 3:边界检查 + 文档(Low Risk)
|
||||
|
||||
1. 添加 lint 规则:检查 harness 不 import app 模块
|
||||
2. 更新 `CLAUDE.md`、`README.md`
|
||||
|
||||
## 7. 风险与缓解
|
||||
|
||||
| 风险 | 影响 | 缓解措施 |
|
||||
|------|------|----------|
|
||||
| 全局 rename 误伤 | 字符串中的 `src` 被错误替换 | 正则精确匹配 `\bsrc\.`,review diff |
|
||||
| LangGraph Server 找不到模块 | 服务启动失败 | `langgraph.json` 的 `dependencies` 指向正确的 harness 包路径 |
|
||||
| App 的 `PYTHONPATH` 缺失 | Gateway/Channel 启动 import 报错 | Makefile/Docker 统一设置 `PYTHONPATH=.` |
|
||||
| `config.yaml` 中的 `use` 字段引用旧路径 | 运行时模块解析失败 | `config.yaml` 中的 `use` 字段同步更新为 `deerflow.*` |
|
||||
| 测试中 `sys.path` 混乱 | 测试失败 | 用 editable install(`uv sync`)确保 deerflow 可导入,`conftest.py` 中添加 `app/` 到 `sys.path` |
|
||||
|
||||
## 8. 未来演进
|
||||
|
||||
- **独立发布**:harness 可以发布到内部 PyPI,让其他项目直接 `pip install deerflow-harness`
|
||||
- **插件化 App**:不同的 app(web、CLI、bot)可以各自独立,都依赖同一个 harness
|
||||
- **更细粒度拆分**:如果 harness 内部模块继续增长,可以进一步拆分(如 `deerflow-sandbox`、`deerflow-mcp`)
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user