mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-07-21 10:15:47 +00:00
a028dfd5fb70bd6e26c7dbf9e89543c7c006f9a2
116
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ryker_FengandGitHub
|
a028dfd5fb |
feat(auth): add keep me signed in login option (#4255)
* feat(auth): add keep me signed in login option * fix(auth): address remember-login review feedback |
||
|
Ryker_FengandGitHub
|
f8bef42a04 | fix(wechat): validate timing configuration (#4280) | ||
|
Huixin615andGitHub
|
c9b6131f8f |
fix(skills): reload mounted skills without restarting Gateway (#4264)
* fix(skills): add admin-only reload endpoint * fix(skills): preserve cache when reload fails |
||
|
Huixin615andGitHub
|
65afc9b1d2 |
fix(skills): apply allowed-tools only to active skills (#4098)
* fix(skills): scope allowed-tools to active skills * fix(skills): tolerate stale active skill paths * chore: retrigger CI * fix(skills): document policy activation limits * perf(skills): reuse per-step tool policy decisions * fix(skills): harden runtime tool policy contracts * fix(skills): redact cached policy decisions * fix(skills): make slash tool policy authoritative * fix(skills): preserve policy-safe discovery tools * test(skills): cover explicit task delegation policy |
||
|
Nan GaoandGitHub
|
de55982c5a |
fix(subagents): preserve parent checkpoint namespace (#4215)
* fix(subagents): preserve parent checkpoint namespace * test(subagents): align stream isolation coverage |
||
|
Henry LiandGitHub
|
b68e1c686a | docs: add the link of LLM Space (#4195) | ||
|
|
4af6178358 |
feat(trace): add agent observability with Monocle (#4024)
* Add Monocle tracing Enable Monocle (OpenTelemetry tracing for LLM apps) with one setup call plus the monocle_apptrace dependency. setup_monocle_telemetry auto-instruments the frameworks already in use and writes traces to .monocle/. Additive; no changes to application logic. * Config-gate Monocle telemetry in the Gateway lifespan Addresses review on #4024: moves setup_monocle_telemetry out of agents/__init__ import time into the Gateway lifespan, gated by MonocleTracingConfig (MONOCLE_TRACING env, default off). Warns on the Langfuse/global-OTel-provider conflict and relies on monocle_apptrace's own duplicate-setup guard and existing-provider attach. Pins monocle_apptrace>=0.8.8 (+ uv.lock), adds .monocle/ to .gitignore, adds tests (default-off / toggle-on / no import-time setup), and documents exporters, Okahu, and the VS Code viewer in README, config.example.yaml, and backend/AGENTS.md. * Clarify Monocle/Langfuse single-provider guidance Make the docstring, warning, and AGENTS.md consistent with the README: only one library can own the global OpenTelemetry provider; Monocle initializes at startup before Langfuse's per-run handler, so enabling both drops Langfuse's spans — enable one OTel tracer (LangSmith, a callback, coexists fine). * Address review: optional extra, exporter validation, off-box warning, tests Responds to the second review round. - Make monocle_apptrace an optional extra (deerflow-harness[monocle], re-exposed as deer-flow[monocle]) following the boxlite/tui precedent, so a default install no longer pulls the OpenTelemetry stack. It stays pinned in the dev group for the tracing tests, and enabling MONOCLE_TRACING without the extra raises a clear install error. - Warn loudly at startup whenever any exporter other than `file` is configured, since those move prompts, tool inputs/outputs, and completions beyond the local .monocle/ directory. - Validate MONOCLE_EXPORTERS against the known exporter names and require OKAHU_API_KEY when okahu is selected, mirroring the Langfuse pattern. Validation runs from Monocle's own init (not validate_enabled) so a config typo can never fail agent runs; errors surface at Gateway startup instead. - Grow the tests from 5 to 13: caplog coverage for the Langfuse-conflict and off-box warnings, exporter validation cases, a stronger import-time regression that asserts the global TracerProvider is not replaced, and a subprocess double-invoke test exercising the real check_duplicate_setup. - Docs: config.example.yaml block retitled to a dedicated tracing header; README documents the [monocle] install and scopes tracing to Gateway runs. * docs: align Monocle README section with the other tracing providers Lead with what Monocle is and captures, drop the install step (the dev group already ships monocle_apptrace via uv sync; unusual installs get the RuntimeError), and point the missing-package error at the repo-native command (uv sync --extra monocle / deerflow-harness[monocle]). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address review: verified Langfuse coexistence, lifespan test, scope docs Responds to the third review round. The Langfuse conflict claim was wrong, verified empirically against langfuse 4.5.1 in both init orders: whichever library initializes second reuses the existing global TracerProvider and attaches its own span processor, so neither side loses spans. Dropped the warning and its tests, corrected the README, AGENTS.md, and config.example.yaml statements, and pinned the verified behavior with test_coexists_with_langfuse (real monocle + real langfuse in a subprocess, no mocks). One honest caveat documented: both processors see all spans, so Monocle's exporters also capture Langfuse's spans when both are enabled. Also from the review: - Document the Gateway-only scope in AGENTS.md: the lifespan is the sole call site, so the embedded DeerFlowClient and TUI are not instrumented; embedded users call setup_monocle_tracing_if_enabled() themselves. - Add test_gateway_lifespan_initializes_monocle pinning the lifespan wiring. - Comment why MonocleTracingConfig.is_configured is intentionally coarser than LangSmith/Langfuse (composite validation lives in validate() at startup). - Note that monocle_exporters_list takes the comma-separated string as-is. - Module-level importorskip("monocle_apptrace") so minimal installs collect the test module cleanly. * docs: reword Monocle intro sentence * fix(tests): run the import-time regression in a subprocess test_no_import_time_setup deleted deerflow.agents* from sys.modules and re-imported to force __init__ to re-execute. The re-import creates new module objects, and restoring the old sys.modules entries afterwards leaves the parent package's attribute bindings pointing at the new ones, so any later test that resolves a deerflow.agents.* dotted path (monkeypatch.setattr in test_summarization_middleware, test_thread_data_middleware, and others) failed with "module 'deerflow.agents' has no attribute ...". Run the check in a subprocess instead: the import is genuinely fresh, the assertion is stronger (the provider must still be the SDK-less proxy, proving nothing was installed at any point), and no module identity leaks into the rest of the suite. * Address review: console warning scope, embedded hint, honest naming, doc alignment Responds to the post-approval review round: - Scope the off-box exporter warning to the remote exporters (okahu, s3, blob, gcs): console writes to local stdout and no longer trips it. config.example.yaml's data-handling note now distinguishes file / console / remote likewise. - Rename MonocleTracingConfig.is_configured to is_enabled so the boolean reads as what it checks; the exporter-dependent credential check stays in validate(), run at Gateway startup. - Hint on the embedded path: build_tracing_callbacks() logs a debug line when MONOCLE_TRACING is set but setup never ran in this process, so embedded DeerFlowClient/TUI users are not left with silent no-op tracing. Backed by a process-global setup flag. - Re-export setup_monocle_tracing_if_enabled from deerflow.tracing, matching the package convention. - Note the deliberate fail-open-at-startup contrast with LangSmith/Langfuse in the lifespan, and the OTel SDK-internals dependency in the coexistence test. - Test hygiene: clear MONOCLE_* env in the tracing config/factory fixtures; reset the setup flag in the monocle test fixture; reword the README Langfuse-spans claim as the shared-provider inference it is. - Document that .monocle/ trace files are never rotated or cleaned up. * fix(tests): pin the factory logger level in the embedded-hint tests configure_logging() from earlier tests in the full suite pins an explicit INFO level on the logger hierarchy, so a root-level caplog.at_level(DEBUG) never sees the factory's debug hint. Scope caplog to deerflow.tracing.factory so the test is independent of suite ordering. * Address review: co-export disclosure, lifespan failure test, exporter parse dedup - Off-box warning now notes that Langfuse's spans are exported too when both providers are enabled and share the global OTel provider; pinned both ways by tests. - Pin the lifespan fail-open contract: a raising Monocle setup is logged and the Gateway keeps serving (pragma dropped now that the path is exercised). README notes a config error is reported at startup and tracing stays off until restart. - Hoist exporter parsing into MonocleTracingConfig.exporter_list so validate() and the off-box warning cannot diverge, and note the upstream coupling on the exporter allow-list. - Reduce config.example.yaml's Monocle block to a pointer; the capture, retention, and data-handling detail lives in README's Monocle section. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
4e209827f3 |
feat(agent): Add subagent total delegation cap (#4115)
* fix subagent total delegation cap * fix embedded subagent run cap context * fix subagent cap config consistency * fix resumed subagent run cap boundary * fix legacy resume subagent boundary * address subagent cap review feedback --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Ryker_FengandGitHub
|
41658c5ff4 |
feat(skills): add skill review quality gate (#4037)
* feat(skills): add skill review quality gate * fix(skills): skip review eval fixtures in CI * fix(skills): ignore review eval fixtures in bundled scans * fix(skill-review): harden review gate boundaries * fix(skills): address skill review gate feedback |
||
|
Huixin615andGitHub
|
6389fc03fd |
fix: ensure visible response after tool runs (#4033)
* fix: ensure visible response after tool runs * fix: clean up terminal response recovery state * fix: bound terminal recovery state |
||
|
Nan GaoandGitHub
|
aafd5077b2 |
feat(subagents): show effective model and token usage on task cards (#4049)
* feat(subagents): show runtime metadata on task cards * fix(subagents): stop task-card render loop and dedupe model fetches Address code review on the runtime-metadata cards: - P1 render loop: the terminal ToolMessage is re-parsed on every MessageList render and always carries modelName/usage, so the presence-based setTasks condition fired a fresh state object each render -> "Maximum update depth exceeded". computeNextSubtask now returns a value-compared `changed` flag and a pure subtaskNotification() routes terminal transitions through the deferred after-render path while skipping no-op re-parses. - Per-card useModels refetch: add staleTime: Infinity to the ["models"] query so every subtask card shares one /api/models fetch instead of refetching on each mount. * make format * refactor(subagents): dedupe token-usage validators + tidy event narrowing Address PR review follow-ups: - DRY: extract one shared token-usage validator per side. Backend status_contract.normalize_token_usage() now backs both the terminal ToolMessage metadata and the subagent.step/.end run events (step_events.py), and frontend messages/usage.normalizeTokenUsage() backs both the live task_running event (lifecycle.ts) and the terminal ToolMessage metadata (subtask-result.ts). Prevents the input/output/ total_tokens validation from drifting across the four former copies. - Nit: onCustomEvent narrows event.type once instead of re-checking the object shape per branch; the redundant task_started early-return (already validated by taskEventToSubtaskUpdate) is dropped. |
||
|
|
bbb3deb231 |
fix(subagents): classify LLM error fallbacks as failed (#4042)
* fix(subagents): classify LLM error fallbacks as failed (#4041) * fix(subagents): address #4042 review on LLM error fallback - clarify _extract_llm_error_fallback docstring: tail-only scan is safe because subagents append their own terminal message, so the last AIMessage is never a stale parent-history marker (cross-ref worker.py) - compute final_result and pop stop_reason only on the COMPLETED branch so the guard stop-reason pop no longer fires on the discarded FAILED path - note the error_detail/"LLM request failed" fallbacks are defensive; the middleware always populates a non-empty content - add regression test locking the stale parent-history marker invariant --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Nan GaoandGitHub
|
8fbf101de3 |
fix(subagents): inject durable context before compaction (#4040)
* fix(subagents): inject durable context before compaction * fix(subagents): coalesce system messages after durable-context injection Address #4040 review: - append SystemMessageCoalescingMiddleware innermost on the subagent chain so the SystemMessage(authority) DurableContextMiddleware injects is merged into one leading system_message; otherwise the durable fix trades #4039's assistant-first 400 for a duplicate-system 400 on strict backends - add a two-system regression guard driving the real builder output through a strict model; assert exactly one leading SystemMessage - assert single-leading-system in the compaction integration test too - update the middleware count/last-element assertion (coalescer is now unconditionally last, removing the summarization-dependence ambiguity) - compare _skills_root against posixpath.normpath(container_path) - document the coalescer on the subagent chain in backend/AGENTS.md |
||
|
Ryker_FengandGitHub
|
be637163a6 |
feat(frontend): add voice dictation input (#4036)
* feat(frontend): add voice dictation input * fix(frontend): address voice input review feedback * ci: rerun frontend checks |
||
|
Zhengcy05andGitHub
|
b06372b812 |
feat(backend): queue rapid same-thread messages and preserve topic card previews (#3988)
* feat(backend): queue rapid same-thread messages and preserve topic card previews * fix: checkstyle * refactor(channels): make feishu serialization policy-driven and fix queue cleanup |
||
|
Ryker_FengandGitHub
|
ebc09ce130 |
feat(mcp): auto-promote deferred MCP tools from routing hints (#4019)
* feat(mcp): auto-promote deferred MCP tools from routing hints
When tool_search.enabled=true defers MCP tool schemas, PR1 routing hints
still require the model to spend a tool_search discovery round trip before
it can call the tool the routing metadata already points at. This adds a
McpRoutingMiddleware that matches the latest user message against PR1
routing keywords and promotes the matching deferred schemas before the
model call, removing that round trip.
Design (soft routing, opt-in, additive):
- Matches only the latest real HumanMessage (shared is_real_user_message
helper, reused by SkillActivationMiddleware so the two cannot drift);
case-insensitive substring match, no tokenizer dependency.
- Ordering: priority desc, then tool name asc; capped by the new global
tool_search.auto_promote_top_k (default 3, clamped 1..5). Does not add or
consume a per-tool auto_promote_top_k (PR1 schema unchanged); a per-tool
value is ignored with a DEBUG note.
- Returns a plain {"promoted": ...} state update (not a Command) and relies
on ThreadState.merge_promoted for union/dedupe, so auto-promote and a
model-triggered tool_search converge on the same catalog hash.
- Installed before DeferredToolFilterMiddleware on every deferred-tool path
(lead agent, subagent, embedded client, webhook via shared builders);
a construction-time assert rejects the reversed order. catalog_hash is
None / no routing index is a complete no-op, so bootstrap and ACP skip it.
- Privacy: never executes tools, never promotes policy-filtered tools, adds
no routing keywords or matched tool names to trace metadata or INFO/WARN
logs.
No behavior change when tool_search.enabled=false.
Tests: index construction, matching semantics, middleware state updates,
same-cycle deferred-filter interaction, lead/subagent/embedded-client
builder wiring + order invariant, config clamping, config.example.yaml
parseability, and privacy assertions.
* refactor(mcp): address auto-promote review nits
- executor: access app_config.tool_search.auto_promote_top_k directly to match
the lead-agent and embedded-client paths (drop the over-defensive getattr that
masked missing config); update the subagent test mock to carry tool_search.
- tool_search / mcp_routing_middleware: cross-reference the duplicated routing
priority/keyword normalization between the builder and the middleware's
defensive _normalize_index so they cannot silently drift.
- MCP_SERVER.md: document that auto-promote keyword matching is a case-insensitive
substring test (not word-boundary), advising distinctive keywords.
|
||
|
Huixin615andGitHub
|
66c0c641bf |
fix(gateway): live-tail malformed redis reconnect ids (#4012)
* fix(gateway): live-tail malformed redis reconnect ids * test: avoid redis live-tail heartbeat gaps * fix: handle terminal malformed redis reconnect ids |
||
|
Ryker_FengandGitHub
|
5ba25b06ec |
feat(mcp): add MCP routing hints (#4004)
* feat: add MCP routing hints * test: isolate mcp routing prompt config * fix: address mcp routing review feedback |
||
|
|
01dc067997 |
feat: add composer input polishing (#3986)
* feat: add composer input polishing * Revert "Merge branch 'main' into feat/input-polish" This reverts commit |
||
|
AochenShen99andGitHub
|
658c39ccf7 |
feat(skills): Add native SkillScan phase 1 for skills (#3033)
* Add phase 1 skill static scanning * Rework SkillScan phase 1 as native scanner * refactor(skillscan): align phase 1 with trimmed RFC contract - SecurityFinding: 7 fields (rule_id, severity, file, line, message, remediation, evidence); category/analyzer derive from the rule_id prefix, confidence/column/fingerprint/metadata removed - scan_archive_preflight()/scan_skill_dir() are pure functions: no ScanContext, no policy schema; CRITICAL-blocks is a code constant and skill_scan.enabled is applied by enforce_static_scan()/callers - secret-* evidence is redacted before findings leave the scanner - de-dup keys on (rule_id, file, line) so repeated occurrences keep distinct locations for agent self-correction - cloud-metadata detection consolidated into network-cloud-metadata - nested zip members get a one-level stdlib magic-byte peek; an executable member escalates package-nested-archive to CRITICAL - install metadata sidecar removed (Phase 7 decides if it is needed) - rule specs moved next to their analyzers; skillscan/rules/ removed - tests updated + new anchors: redaction, dedup lines, nested-zip escalation, single cloud-metadata rule, bundled-skill zero-CRITICAL * fix(skillscan): tighten reverse-shell/secret/archive scan rules from review Address PR #3033 review feedback on the native SkillScan analyzers: - Reverse-shell false positives: split shell detection by signal strength (/dev/tcp/, nc -e stay CRITICAL; bash -i, mkfifo -> new HIGH shell-reverse-shell-heuristic, warn->LLM). The Python check is now AST-anchored on real socket.socket/os.dup2/subprocess call sites instead of raw-text substring matching, so prose/docstrings no longer hard-block. - Secret evidence: _redact_secret_evidence returns [redacted] with no secret bytes (was value[:6], which leaked 2 real token bytes past the prefix). - Archive DoS: cap outer archive member count (_MAX_ARCHIVE_MEMBERS=4096); scan_archive_preflight early-aborts with a package-too-many-members CRITICAL finding (routes through the existing blocked->400 fail-closed path). - shell-destructive-command: broaden the rm -rf matcher to sensitive system roots (/home, /usr, /*, --no-preserve-root /) while leaving safe subpaths unflagged. - Dead code: collapse _decode_text_for_analysis to a single decode path and drop the unused _TEXT_SUFFIXES set and _has_text_shebang helper. - local_skill_storage: document why the host_path branch keeps app_config possibly-None (lazy kill-switch resolution; avoids eager get_app_config in config-free environments such as CI). Tests: new negative/positive coverage in test_skillscan_native.py. Full backend suite 6616 passed, 26 skipped. |
||
|
Ryker_FengandGitHub
|
26d7a5970d |
feat: add manual context compaction (#3969)
* feat: add manual context compaction * fix: harden manual context compaction |
||
|
|
186c6ea463 |
feat: add branching support for assistant turns (#3950)
* feat: add assistant turn branching * fix(threads): skip workspace clone when branching from historical turn Workspace files are not checkpointed, so cloning them onto a branch rooted at an older assistant turn leaked files created in a later timeline. Restrict the best-effort workspace copy to branches taken from the latest turn; historical-turn branches now report workspace_clone_mode="skipped_historical_turn" and keep only the restored message history. * style(frontend): fix prettier formatting in e2e mock-api Collapse the branch-title normalization chain onto a single line to satisfy the frontend lint (prettier --check) CI gate. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
00161811bd |
feat: add workspace change review for agent runs (#3945)
* feat: add workspace change review * chore: format workspace change files * fix: optimize workspace change summaries * style: refine workspace change badge scale * fix: restore workspace change user context import * fix(frontend): gate workspace change badge to assistant messages Only pass run_id to assistant MessageListItems so the workspace-change badge can never render under a user's prompt, which carries the same run_id. Assert single badge render in the E2E flow. * fix(workspace-changes): address review feedback on diff parsing and badge - Restrict unified-diff header detection to "+++ "/"--- " (trailing space) in both backend _count_diff_lines and frontend getWorkspaceChangeLineClass so content lines beginning with +++/--- are counted/styled correctly - Gate WorkspaceChangeBadge to ai messages so tool messages folded into an assistant group don't render a duplicate badge - Add regression tests for both diff-classification fixes - Apply prettier formatting to workspace-changes files flagged by CI --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
4fc08b4f15 |
feat: add scheduled tasks MVP (#3898)
* feat: add scheduled tasks MVP
* fix: harden scheduled task execution semantics
* feat(scheduled-tasks): preset-driven schedule form with timezone and live preview
Replace the raw cron input with a preset Select (hourly/daily/weekly/monthly/custom)
plus structured inputs (time picker, weekday toggles, day-of-month), datetime-local
for one-time tasks, a timezone selector defaulting to the browser timezone, and a
live human-readable preview. Reuses one ScheduledTaskScheduleInput for create and
edit; backend contract unchanged; zero new deps (pure Intl + DST-safe offset helpers).
* feat(scheduled-tasks): full-page i18n + recipe templates + E2E locale pin
Localize the rest of the scheduled-tasks page (filters, detail pane, actions,
edit form, run list, enum values) via t.scheduledTasks.* in en/zh. Add four
built-in recipe templates (GitHub Trending, news digest, issue triage, weekly
report) exposed as a chip row that pre-fills title + prompt + schedule. Pin
Playwright locale to en-US so E2E selectors stay stable against i18n. No backend
change, no new deps.
* fix(scheduled-tasks): idempotent 0003 migration, update head constants, future-date once test
Merge with main surfaced three CI failures:
- 0003_scheduled_tasks create_table collided with legacy test seeds that
build from full metadata; guard with inspector.has_table so the revision
no-ops when the table already exists (0004/0005 are already idempotent via
_helpers.py).
- persistence bootstrap concurrency/regression tests pinned HEAD to main's
0002_runs_token_usage; bump to the new head 0005_scheduled_task_thread_nullable.
- once-task router test used a fixed past run_at and tripped the
must-be-in-the-future validation; use a future date.
* address review: ok-check, 502 for trigger failure, mock fields, migration filename, doc fences
- fetchThreadScheduledTasks now checks response.ok like the other fetchers.
- trigger endpoint returns 502 (not 409) when dispatch fails outright, so
clients can distinguish a real conflict from a server-side failure.
- E2E mock normalizes scheduled-task objects with context_mode/last_thread_id
and nullable thread_id, matching the backend contract the UI renders against.
- Rename 0002_scheduled_tasks.py -> 0003_scheduled_tasks.py to match its
revision id (file was renamed in spirit already; filename now follows).
- CONFIGURATION.md: close the Tool Groups yaml fence and drop the stray fence
after the Scheduler notes so the sections render correctly.
* fix(scheduled-tasks): harden lease, poller, config, and frontend UX after review
* fix(scheduled-tasks): harden run lifecycle, overlap skip, non_interactive gating, and DST conversion after review
- defer a once task's terminal status to the run-completion hook; the task
stays running until the real outcome, and a startup sweep cancels once
tasks orphaned by a crash (launch-time 'completed' could stick forever)
- record interrupted runs as a distinct 'interrupted' run status with a
readable message; an interrupted once task ends 'cancelled', not 'failed'
- enforce overlap_policy=skip for fresh_thread_per_run via an active-run
pre-check (same-thread ConflictError can never fire across fresh threads)
- protect terminal run statuses from the late launch-path 'running' write
- honor context.non_interactive only for internally-authenticated callers;
arbitrary clients can no longer strip ask_clarification
- fix DST-stale timezone offset in zonedLocalToUtcIso by re-deriving the
offset at the resolved instant (once tasks fired an hour late around
spring-forward and the create->edit round-trip diverged)
- drop dead ScheduledTaskRunRepository.update_by_run_id; share one Gateway
API error helper between channels and scheduled-tasks frontends
* fix(scheduled-tasks): close review round-3 gaps in guards, concurrency, and API ergonomics
- scrub internal-only context keys (non_interactive) from the assembled run
config for non-internal callers: gating body.context alone left the same
key smuggle-able through the free-form body.config copied verbatim by
build_run_config
- guard update_after_launch with protect_terminal so the launch bookkeeping
write cannot clobber a once task already finalized by a fast-failing run's
completion hook (parent-row sibling of the run-row guard)
- reject a manual trigger while the task has an active run (409) instead of
launching a duplicate concurrent run on fresh_thread_per_run
- re-arm a terminal once task to enabled when PATCH pushes run_at into the
future; previously the endpoint returned 200 with a next_run_at that could
never be claimed
- make max_concurrent_runs a real global cap: each poll claims only into the
remaining budget of active (queued/running) scheduled runs
- paginate GET /scheduled-tasks/{id}/runs (limit<=200, offset) and push the
thread filter of /threads/{id}/scheduled-tasks into SQL
- stamp context.user_id on scheduler-launched runs, matching IM channels, so
user-scoped guardrail providers see the owning user
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
|
38342b15a3 |
fix(redis): stream retention recovery (#3933)
* fix(gateway): retain redis streams safely * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
bjtoloandGitHub
|
c9a5f23e7b |
feat(mcp): add per-server tool_call_timeout for MCP tool calls (#3843)
* feat(mcp): add per-server tool_call_timeout for MCP tool calls Add a configurable timeout for individual MCP tool calls to prevent agent runs from blocking indefinitely when an MCP server becomes unresponsive (e.g., rate-limited HTTP API, hung subprocess). Uses the MCP SDK's built-in read_timeout_seconds parameter on ClientSession.call_tool, which handles the timeout within the session's own task — avoiding cross-task cancellation issues with the session pool (ref #3379, #3203). Config field is named tool_call_timeout (not timeout) to avoid collision with langchain-mcp-adapters' existing timeout field on HTTP/SSE connections. Closes #3840 * fix(mcp): read tool_call_timeout from McpServerConfig, not connection dict The previous implementation put tool_call_timeout into the connection dict returned by build_server_params, which langchain's create_session then passed to _create_stdio_session(), causing TypeError. Now reads the timeout directly from ExtensionsConfig.mcp_servers where the wrapper is built, keeping it out of the connection dict entirely. Fixes P1 bug from review on #3843. * test(mcp): regression test for tool_call_timeout not leaking into connection dict Adds two tests: - test_build_server_params_excludes_tool_call_timeout: verifies the connection dict returned by build_server_params() does NOT contain tool_call_timeout - test_stdio_tool_call_timeout_does_not_raise_typeerror: end-to-end test that get_mcp_tools() with a stdio server having tool_call_timeout configured loads tools without TypeError from _create_stdio_session() Regression for PR #3843 P1 bug. * fix(mcp): only pass read_timeout_seconds when tool_call_timeout is set When tool_call_timeout is None, don't pass read_timeout_seconds=None to session.call_tool(). This avoids breaking existing tests that assert on exact call_tool arguments without the extra kwarg. * docs(mcp): clarify stdio tool timeout |
||
|
|
25ea6970a6 |
feat(runtime): implement goal continuations (#3858)
* implement goal continuations * fix(goal): address review findings for goal continuations - goal: key the no-progress breaker on a signature of the latest visible assistant evidence instead of the evaluator's volatile free-text, so it actually fires on stalled turns; thread the signature through every worker persist / no-progress call site - goal: align _stand_down_reason default caps with should_continue_goal (8 / 2) so the two gate functions agree on goals missing the fields - runtime: offload the synchronous checkpointer fallback via asyncio.to_thread (goal.py + worker.py) to keep blocking IO off the loop - frontend: i18n the GoalStatus "Goal" label (goalLabel in en/zh/types) - frontend: extract pure composer helpers into input-box-helpers.ts with unit tests (parseGoalCommand, readGoalResponseError, skill suggestions) - tests: cover the evidence-based no-progress and default-cap behavior - docs: align backend/AGENTS.md goal paragraph with actual behavior - e2e: prettier-format chat.spec.ts (fixes the lint-frontend CI failure) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(frontend): hide goal continuation counter until the agent continues The goal status bar rendered a raw "0/8" before any auto-continuation, which read as a mysterious score. Now the counter is hidden until continuation_count > 0, then shows "Continuing N/M" with a tooltip explaining the auto-continuation cap. - Extract getGoalContinuationDisplay into a pure helper (hides at 0) + unit tests - Add goalContinuing / goalContinuationTooltip i18n keys (en/zh/types) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(goal): address review findings for goal continuations Frontend correctness - Fix the optimistic /goal result permanently shadowing server goal state: the streamed continuation counter never surfaced for a goal set in-session. Extract a shared useActiveGoal hook (used by both chat pages) that reconciles the optimistic copy with server state via a goalReconciliationKey, de-duping the copy-pasted goal block across the two pages. - Stop /goal status|clear failures from escaping handleSubmit as unhandled rejections (handleGoalCommand now returns success; the run only starts when a goal was actually saved). - Use a function replacer for the goal-status toast so an objective containing $&/$1 isn't treated as a replacement pattern. Backend cleanliness / correctness - De-duplicate four byte-identical helpers (_call_checkpointer_method, _message_type, _additional_kwargs, _is_visible_message) by importing them from runtime.goal instead of re-defining them in the run worker. - Remove the dead `checkpoint_tuple.tasks` durability guard (CheckpointTuple has no tasks field) and document that pending_writes is the durability signal. - Decompose the 176-line _prepare_goal_continuation_input: extract _reread_goal_and_checkpoint and a _persist closure so the thread-unchanged guard and stand-down persistence aren't open-coded three times. Document the last-writer-wins write-window limitation as a follow-up. - Add a shared parse_goal_command helper and use it from the TUI and IM-channel /goal handlers (one place for the status/clear/set semantics). Tests - Restore the 11 command-registry tests dropped by the previous goal change (filter_commands ranking/description, build_registry builtins/skills, resolve cases) alongside the new goal tests. - Add coverage for the IM-channel _handle_goal_command, the TUI _handle_goal handler, parse_goal_command, and goalReconciliationKey. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix goal review feedback * fix goal continuation checkpoint races * prioritize goal commands while streaming Route composer submits through a shared helper so /goal commands can be handled before the streaming stop shortcut, while ordinary streaming submits still stop the active run. Testing: cd frontend && pnpm exec rstest run tests/unit/components/workspace/input-box-helpers.test.ts tests/unit/components/workspace/goal-status-helpers.test.ts; cd frontend && pnpm check * preserve goal status during clarification Keep omitted stream goal fields distinct from explicit null clears so clarification interrupts do not hide an active thread goal that is still present in the checkpoint. Testing: pnpm exec rstest run tests/unit/components/workspace/use-active-goal.test.ts tests/unit/components/workspace/input-box-helpers.test.ts tests/unit/components/workspace/goal-status-helpers.test.ts; pnpm check; git diff --check * style: format active goal hook Run Prettier on use-active-goal.ts to satisfy the frontend lint workflow formatting gate. Testing: pnpm format; pnpm exec rstest run tests/unit/components/workspace/use-active-goal.test.ts tests/unit/components/workspace/input-box-helpers.test.ts tests/unit/components/workspace/goal-status-helpers.test.ts; pnpm check; git diff --check * fix goal review followups --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
AnoobFengandGitHub
|
e3e5c73b03 |
feat(observability): add trace-id correlation and enhanced logging (#3902)
* feat(observability): add trace-id correlation and enhanced logging - add opt-in gateway request trace correlation via X-Trace-Id - enhance logging with configurable trace_id-aware formatting - propagate deerflow_trace_id into runtime context and Langfuse metadata - keep enhanced logging disabled by default to preserve existing behavior * fix: harden trace correlation wiring - Make logging enhancement a restart-required startup snapshot and remove per-request config reads from TraceMiddleware - Restrict trace ids to printable ASCII before writing them to response headers, logs, and Langfuse metadata - Gate implicit DeerFlowClient trace-id creation behind logging.enhance.enabled while preserving explicit caller opt-in - Bind embedded client trace context per stream step to avoid generator ContextVar leaks and cross-context reset errors - Rebind memory update trace ids in Timer/executor worker paths so enhanced logs keep the captured correlation id - Remove unrelated __run_journal context overwrite from the trace-correlation change set * fix(gateway): avoid eager app construction on package import * fix(gateway): avoid config load during app import Keep Gateway app construction import-safe when config.yaml is absent by disabling TraceMiddleware only for that construction-time fallback path. Startup lifespan still performs strict config loading before serving. |
||
|
|
b476c7a18d |
fix(store): honor unified database configuration (#3904)
* fix(store): align store backend with database config * fix(store): preserve no-config memory fallback --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
2e15e3fe0d |
fix: generate fallback title for interrupted first-turn runs (#3874)
* fix: generate title for interrupted first turn * test(title): cover partial-exchange + dict-form messages Harden the interrupted-run fallback path added in |
||
|
Ryker_FengandGitHub
|
a8f950feb6 |
feat(community): add Browserless web_capture screenshot tool (#3881)
* feat(community): add Browserless web_capture screenshot tool Add a web_capture tool that renders a page via Browserless /screenshot and presents it through the artifact system, alongside the existing Browserless web_fetch provider. Hardening: - SSRF guard: reject URLs resolving to private/loopback/link-local (incl. the 169.254.169.254 cloud-metadata endpoint)/reserved/multicast/unspecified addresses; opt out via allow_private_addresses for internal targets. - Surface a warning when Browserless renders a target page that itself responded with a non-2xx/3xx status (X-Response-Code), so an error/anti-bot page is not mistaken for valid visual evidence. - Dedupe colliding output filenames instead of silently overwriting prior captures. Docs: comment out token: $BROWSERLESS_TOKEN in tool examples (an unset $VAR fails AppConfig startup) and document allow_private_addresses. * fix(community): format web_capture guard + document local Browserless startup Address PR #3881 review: fix the lint-backend failure (ruff format on browserless/tools.py) and add local Browserless startup instructions to CONFIGURATION.md so reviewers can run the service to try web_fetch/web_capture. |
||
|
Ryker_FengandGitHub
|
cf02646489 |
feat(scripts): add redacted community support bundle generator (#3886)
* feat(scripts): add redacted community support bundle generator Add `make support-bundle` (scripts/support_bundle.py) to help users file high-signal, privacy-safe GitHub issues for local setup/config/runtime problems. The command produces: - `*-issue-summary.md` to paste into the issue body - `*-issue-draft.md` scaffold for AI-assisted filing (REQUIRED placeholders, never invents repro/expected/summary facts) - an optional evidence zip under `.deer-flow/support-bundles/` containing a stable `triage.json` plus redacted environment/config/extensions/git/doctor evidence Privacy: secrets are redacted across config values, URL userinfo, query strings, CLI flags, custom headers, bearer/sk- tokens, and home paths. The bundle never includes `.env`, raw conversation messages, or user file contents; optional `--thread-id` adds file manifests only. `thread_id` input is validated against path traversal. Wire it into the Makefile, AGENTS.md, README/README_zh, CONTRIBUTING, and the bug-report issue template. Covered by backend/tests/test_support_bundle.py. * fix(scripts): redact MCP env values by default in support bundle Address PR #3886 review (willem-bd, P2): the key-name allowlist let literal secrets under non-standard env keys (e.g. SUPABASE_SERVICE_ROLE_KEY, R2_ACCESS_KEY, hardcoded AIza… keys) leak verbatim into the bundle that users are told is safe to share publicly. Mask all MCP `env` values by default, keeping only `$VAR`/`${VAR}` references visible, and broaden SECRET_KEY_RE (access_key, pwd, private_key). Add tests for non-keyword env secrets, broadened key names, and end-to-end zip redaction. |
||
|
|
fe82552023 |
fix(sandbox): stop blocking bash commands (e.g. servers) from hanging the turn (#3864)
* fix(sandbox): stop blocking bash commands from hanging the turn Starting a server through the host bash tool (e.g. `python -m http.server`) could hang the whole turn for the full 600s timeout. `LocalSandbox.execute_command` used `subprocess.run(capture_output=True)`, whose captured pipes are inherited by any process the command spawns — so a backgrounded long-lived process (`server &`) keeps the read end open and blocks `communicate()` until the timeout fires, even though the foreground command already returned. Commands that read stdin blocked the same way, and on timeout only the direct child was killed, leaving orphaned process groups. Rework the POSIX path to capture stdout/stderr via temp files instead of pipes, take stdin from /dev/null, and run the command in its own session/process group: - Backgrounded long-lived processes (servers) now return immediately while the process keeps running. - A command reading stdin gets immediate EOF instead of blocking. - A genuinely blocking foreground command is bounded by a configurable wall-clock timeout; on timeout the whole process group is killed and the agent gets an explanatory notice telling it to background long-lived processes. The timeout is configurable via `sandbox.bash_command_timeout` (default 600). The Windows path is unchanged. Adds focused regression tests and updates docs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sandbox): instruct the agent to background long-lived processes The code fix bounds a foreground server with a timeout, but the turn still waits the full timeout before the run continues. Add the prompt-side half: the bash tool description now tells the model to ALWAYS start long-lived processes (e.g. web servers) in the background with output redirected, so the tool returns immediately. The timeout notice points at the same readable workspace log path. Pins the guidance with a test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sandbox): make fallback-kill exception explicit and observable Address automated review: the inner `except OSError: pass` in _terminate_process_group silently swallowed the case where the direct-child fallback kill found the process already gone. Make the intent explicit with a comment and a debug log instead of a bare pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sandbox): address bash timeout review feedback * fix(sandbox): document fd cleanup races --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
ef5f54c5bf |
feat(tui): Hermes-like terminal workbench (deerflow) backed by DeerFlowClient (#3760)
* feat(tui): add Hermes-like terminal workbench backed by DeerFlowClient Implements the `deerflow` TUI from RFC #3540: a terminal-native, embedded workbench over the existing harness (no Gateway/frontend/nginx/Docker), built Python-native with Textual and learning UX patterns from tao-pi. Architecture — every layer except the Textual app is pure and unit-tested: - view_state.py: ViewState + reduce(state, action), the testable heart - runtime.py: StreamEvent -> reducer actions (pure translate + threaded driver) - message_format / command_registry / input_history / render / theme: pure - app.py: Textual App; runs the sync DeerFlowClient.stream() on a worker thread and marshals actions back to the UI thread. Slash command palette, model and thread modal pickers, ↑/↓ history, Ctrl+C interrupt, TTY-aware fallback. - cli.py: pure launch-mode planning + headless --print/--json + `deerflow` console script (textual is an optional [tui] extra; degrades to headless help) Web UI visibility (the RFC's key decision): persistence.py writes a threads_meta row under the local default user into the same DB the Gateway reads, so terminal sessions appear in the Web UI sidebar without running the Gateway. Best-effort, no-op on the memory backend; all DB work on one long-lived background loop. Tests: 95 TUI tests — pure layers via pytest, app/palette/overlays via Textual's pilot harness with a fake session, and a threads_meta read/write round-trip. ruff clean; respects the harness->app import boundary. Docs: backend/docs/TUI.md plus CLAUDE.md/README updates and preview screenshots. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tui): de-duplicate streamed assistant text and tool cards; keep Tab in composer Self-test surfaced three issues, all root-caused to consuming non-strict streaming from DeerFlowClient (proven by the client's own test_dedup_requires_messages_before_values_invariant, which shows the client can re-emit a message id's full content twice): - Assistant text was doubled (e.g. "answer answer") because the reducer blindly concatenated same-id deltas. Now merges by content: a re-send or cumulative snapshot replaces; only genuine increments append. - Tool activity showed duplicate and empty "gear" cards from partial/re-emitted tool-call chunks. ToolStarted now de-dupes by tool_call_id, drops id-less noise chunks, and fills the name on a later chunk; a tool result with no prior card still surfaces as a completed card. - Tab moved focus off the composer to the scroll region (felt like broken cursor logic). Tab is now consumed by the composer (completes a command when the palette is open, no-op otherwise). Adds reducer tests for each case plus a Tab-focus test; 102 TUI tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tui): make Esc interrupt an active run (matches the status hint) The status line advertised "esc interrupt" but Esc was only wired to close the slash palette, so it did nothing during a run. Esc now: closes the palette when open, interrupts the active run when streaming, and is a no-op when idle. The interrupt logic is shared with Ctrl+C via _interrupt_run(). Adds a regression test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tui): stop prior answers duplicating on threads with history On a thread with history, DeerFlowClient re-emits every prior message on each new turn (its streamed_ids dedup is per-stream-call), and a re-emitted older message can arrive after a newer message has already started. The reducer only matched the *most recent* assistant row by id and otherwise appended, so each re-emitted older answer was duplicated verbatim at the end of the transcript. Match an assistant row by id anywhere in the transcript and merge in place. Tool cards already de-dupe by call id globally, so they were unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tui): correct CJK cursor drift in the composer Confirmed a Textual Input bug (latest 8.2.7): Input._cursor_offset adds an unconditional +1 at the end of the value, overshooting by one cell after double-width (CJK) characters. That misplaces the hardware/IME cursor — the drift seen when typing Chinese in iTerm2 (the on-screen block cursor, drawn separately in render_line, is fine; English doesn't use an IME so it looks correct). Reproduced with a bare Input, so it's upstream, not our layout. Add ComposerInput(Input) overriding _cursor_offset to the true cell position and use it for the composer. Numeric tests pin the CJK end/mid and ASCII cases. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(tui): render finalized assistant messages as Markdown The transcript showed raw Markdown (literal **bold**, ## headings, - lists, links). Finalized assistant messages now render as Rich Markdown — headings, bold/italic, lists, inline code + code blocks, blockquotes, horizontal rules and links — with the ● speaker marker aligned to the top of the body. The actively-streaming message stays plain text so partial Markdown doesn't reflow/jump, then snaps to its rendered form when the run ends. Transcript re-renders are coalesced on a ~60ms timer (dirty flag) so per-token Markdown re-parsing stays smooth on long threads. Tests cover both the rendered and the streaming-plain paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style(tui): apply ruff format CI lint runs `ruff format --check` via uvx (latest ruff); apply the formatter so the lint-backend job passes. No behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(tui): address code-quality review comments From github-code-quality[bot] on #3760: - runtime.py: give the `_ClientLike` Protocol method a docstring body instead of a bare `...` (flagged as a no-effect statement), matching the harness convention for Protocol stubs (e.g. SafetyTerminationDetector). - test_tui_cli_main.py: drop the unnecessary `lambda: _FakeSession()` wrappers in monkeypatch.setattr; pass `_FakeSession` directly (same behavior). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tui): keep history Markdown-rendered when a follow-up run starts Previously the transcript rendered "the last assistant row" as plain text while streaming. But when a follow-up turn starts, the last assistant row is the *previous, finalized* answer until the new message begins — and the client re-emits prior messages early in the turn — so sending a follow-up reverted the previous answer from rendered Markdown back to raw text. Track the actively-streaming message id in ViewState instead: it's reset on RunStarted, set only when an AssistantDelta actually adds new content (history re-emits are no-ops and don't mark it), and cleared on RunEnded. The renderer keeps only that one message plain; all history stays Markdown. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(readme): add Terminal Workbench (TUI) section to root README Mention the new `deerflow` TUI alongside the Embedded Python Client in the root README.md and README_zh.md (install, launch/headless commands, feature summary, Web UI visibility), with a ToC entry and a preview screenshot. Links to backend/docs/TUI.md for the full guide. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tui): address review feedback (willem-bd) Ten findings from the TUI code review: 1. /resume was dead-ended — registered + in /help + tested as a builtin, but no dispatch branch. Wired it to thread resolution / the switcher. 2. --resume <title> was forwarded raw into the checkpointer (blank thread). Added Session.resolve_ref() to resolve id-or-title via list_threads; used by --resume and /resume. 3. str(get("id","")) returned "None" for an explicit id:None (truthy), defeating the empty-id guard so unrelated null-id tool calls collapsed into one card. Coerce via a None-safe helper. 4. Headless --print/--json no longer spin up the persistence loop/engine/pool (open_session(persistence=False)). 5. _LoopThread + engine are now closed: Session.close() (dispose engine + stop loop) called from a try/finally around app.run(). 6. --cli --continue (and piped --cli) now run headless instead of erroring. 7. Cancelled runs no longer persist a truncated title (guard on _cancelled). 8. Palette highlight resets to the top when the filter set changes. 9. Dropped the never-populated tools count from the header. 10. Documented the `not row.error` merge guard. Adds regression tests for each; 126 TUI tests pass, ruff check + format clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
Nan GaoandGitHub
|
e847d738d3 |
docs: remove broken (404) images from README files (#3716)
The Official Website hero image and the Coding Plan banner referenced deleted GitHub user-attachment assets that now 404. No working replacement exists, so remove them; section headings and prose/bullet links already cover the same destinations. Closes #3715 |
||
|
|
aa015462a7 |
feat(im): Add user-owned IM channel connections (#3487)
* Add user-owned IM channel connections * Fix dev startup and channel connect popup * Use async channel connect flow * Harden dev service daemon startup * Support local IM channel connections * Align IM connections with local channels * Fix safe user id digest algorithm * Address Copilot IM channel feedback * Address IM channel review comments * Support all integrated IM channel connections * Format additional channel connection tests * Keep unavailable channel connect buttons clickable * Fix IM channel provider icons * Add runtime setup for enabled IM channels * Guard global shortcut key handling * Keep configured IM channels editable * Avoid password autofill for channel secrets * Make channel threads visible to connection owners * Persist IM runtime config locally * Allow disconnecting runtime IM channels * Route no-auth channel sessions to local user * Use default user for auth-disabled local mode * Show IM channel source on threads * Prefill IM channel runtime config * Reflect IM channel runtime health * Ignore Feishu message read events * Ignore Feishu non-content message events * Let setup wizard enable IM channels * Fix frontend formatting after merge * Stabilize backend tests without local config * Isolate channel runtime config tests * Address channel connection review comments * Use sha256 user buckets with legacy migration * Ensure runtime IM channels are ready after restart * Persist disconnected IM channel state * Address channel connection review comments * Address channel connection review findings Frontend connect flow: - Open the runtime-config dialog only when a provider still needs credentials; configured providers go straight to the connect flow, so the binding-code/deep-link path is reachable from the UI again. - After saving credentials, continue into the connect flow when a user binding is still required (multi-user mode) instead of stopping at a "Connected" toast. - Extract shared provider-state helpers to core/channels/provider-state and add unit + e2e coverage for the direct-connect and configure-then-connect paths. Provider status semantics: - Report connection_status from the user's newest connection row; with no binding it is not_connected, except in auth-disabled local mode where a configured running channel is effectively connected. Concurrency and event-loop correctness: - Offload ChannelRuntimeConfigStore construction and writes, channel service construction, and Slack connection replies to threads; add a tests/blocking_io/ anchor for the runtime-config handlers. - Consume binding codes with a conditional UPDATE so a code can only be used once under concurrent workers; retry upsert_connection as an update when a concurrent insert wins the unique constraint. - Serialize ensure_channel_ready per channel so concurrent provider polls cannot double-start a channel worker. Config and migration hardening: - Stop mutating the get_app_config()-cached Telegram provider config; the runtime store now owns the UI-entered bot username. - Register channel_connections in STARTUP_ONLY_FIELDS with the standardized startup-only Field description. - Match the legacy unsafe-id bucket by recomputing its exact SHA-1 name so another user's same-prefix bucket can never be migrated. - Remove the unused Telegram process_webhook_update path and document src/core/channels in the frontend docs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address PR review comments on authz scoping and channel runtime Security (review feedback from ShenAC-SAC): - Scope internal-token callers to the connection owner carried in X-DeerFlow-Owner-User-Id instead of bypassing owner checks outright, in both require_permission(owner_check=True) and the stateless run endpoints. Internal callers keep access to their own and shared/legacy threads, and may claim a default-owned channel thread for its real owner, but a leaked internal token no longer grants cross-user thread access. - Require admin privileges for POST/DELETE /api/channels/{provider}/ runtime-config: runtime credentials and channel workers are instance-wide shared state (same model as the MCP config API). Read-only provider listing stays available to all users. Performance (review feedback from willem-bd): - Skip the redundant thread channel-metadata PATCH after the first successful backfill per thread. - Reuse the per-connection Slack WebClient until its token changes instead of constructing one per outbound message. - Reconcile channel readiness for all providers concurrently in GET /api/channels/providers. Also resolve the code-quality unused-import flag in the blocking-io anchor by pre-importing the channel service via importlib. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix prettier formatting in provider-state test Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Reconcile UI runtime channel config with config reload on restart Main now reloads a channel's config.yaml entry on restart_channel() (#3514, issue #3497). Adapt the user-owned connection flow to coexist: - configure_channel() restarts with reload_config=False — the caller just supplied the authoritative config (browser-entered credentials that are never written to config.yaml), so a file reload must not clobber it with the stale on-disk entry. - _load_channel_config() re-applies the UI runtime-store overlay used at startup, so an operator-triggered restart keeps browser-entered credentials for channels without a config.yaml entry and does not resurrect a channel disconnected from the UI. - Offload the reload's disk IO (config.yaml + runtime store) with asyncio.to_thread, matching the blocking-IO policy on this branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Xinmin ZengandGitHub
|
05ae4467ae |
fix(docker): default Gateway to a single worker to prevent multi-worker breakage (#3475)
The default `make up` started the Gateway with `--workers 4`, but run state (RunManager and the stream bridge) is held in-process and nginx uses no sticky sessions. With the default config, same-run requests scatter across workers that each keep their own run state, breaking run cancellation (409), SSE reconnect (hangs on heartbeats), multitask de-duplication, and IM channels (duplicate replies). The shared cross-worker stream bridge does not exist yet. Default GATEWAY_WORKERS to 1 so the out-of-the-box deployment is correct, document the single-worker boundary in the README, and add a regression test pinning the default while keeping it overridable. This is a stop-gap, not a multi-worker implementation; the full fix (shared run state + stream bridge) is tracked in #3191. Refs #3239, #3260 |
||
|
DanielWalnutandGitHub
|
16391e35ab |
fix(skills): harden slash skill activation across chat channels (#3466)
* support slash skill activation * format slash skill activation * Preserve slash skill activation with uploads * Address slash skill review feedback * Address slash skill follow-up review * Fix lazy slash skill storage resolution * Keep slash skill activation out of system prompt * Address slash skill review issues * fix: harden slash skill command handling * feat(frontend): add slash skill autocomplete * fix: address slash skill review feedback * fix: preserve slash skill text for IM uploads |
||
|
|
da41701f87 |
Add static blocking IO inventory (#3208)
* feat(detectors): add static blocking IO inventory * refactor(detectors): drop superseded runtime probe; clarify static report path - Remove the #2924 custom runtime blocking IO probe entirely: backend/tests/support/detectors/blocking_io.py, backend/tests/test_blocking_io_detector.py, backend/tests/test_blocking_io_probe_integration.py, and the pytest_addoption / pytest_runtest_call / pytest_runtest_teardown / pytest_sessionfinish / pytest_terminal_summary hooks plus the blocking_io_detector fixture from backend/tests/conftest.py. Its narrow DEFAULT_BLOCKING_CALL_SPECS (time.sleep, requests, httpx, os.walk, Path.resolve, Path.read_text, Path.write_text) cannot serve as a CI gate; a Blockbuster-backed runtime detector will land in a separate follow-up PR. Leaving the half-coverage probe alongside the static inventory in this PR added a redundant detect path with no production value. - Address Copilot review comments on backend/README.md and backend/CLAUDE.md by stating explicitly that the JSON report writes to .deer-flow/blocking-io-findings.json at the repository root, whether the target is invoked from the repo root or from backend/. Verified: pytest tests/test_detect_blocking_io_static.py (18 passed), ruff check + format on touched files (passed), make detect-blocking-io from both repo root and backend/ produce the same 105-finding report at <repo-root>/.deer-flow/blocking-io-findings.json. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
df95154282 |
fix(tracing): propagate session_id and user_id into Langfuse traces (#2944)
* fix(tracing): propagate session_id and user_id into Langfuse traces
Adds Langfuse v4 reserved trace attributes (langfuse_session_id,
langfuse_user_id, langfuse_trace_name, langfuse_tags) to
RunnableConfig.metadata inside the run worker, so the langchain
CallbackHandler can lift them onto the root trace.
- New deerflow.tracing.metadata.build_langfuse_trace_metadata() returns
the reserved keys when Langfuse is in the enabled providers, else {}.
- worker.run_agent merges them with setdefault so caller-supplied keys
win, allowing per-request overrides from upstream metadata.
- session_id mirrors the LangGraph thread_id; user_id reads
get_effective_user_id() (falls back to "default" in no-auth mode).
- trace_name defaults to "lead-agent"; tags carry env and model name
when DEER_FLOW_ENV (or ENVIRONMENT) and a model name are present.
Closes #2930
* fix(tracing): attach Langfuse callback at graph root so metadata propagates
The first commit injected ``langfuse_session_id`` / ``langfuse_user_id`` /
``langfuse_trace_name`` / ``langfuse_tags`` into ``RunnableConfig.metadata``,
but on ``main`` the Langfuse callback is attached at *model* level
(``models/factory.py``). LangChain still threads ``parent_run_id`` through
the contextvar, so the handler sees the model as a nested observation and
``__on_llm_action`` strips the ``langfuse_*`` keys
(``keep_langfuse_trace_attributes=False``). The trace's top-level
``sessionId`` / ``userId`` therefore stayed empty in deer-flow's LangGraph
runtime — confirmed live against a real Langfuse instance.
This commit moves the callback to the **graph invocation root** so the
handler fires ``on_chain_start(parent_run_id=None)`` and runs the
``propagate_attributes`` path that actually lifts ``session_id`` /
``user_id`` onto the trace:
- ``models/factory.py``: add ``attach_tracing`` keyword (default ``True``)
so standalone callers (``MemoryUpdater``, etc.) keep their direct
model-level tracing.
- ``agents/lead_agent/agent.py``: call ``build_tracing_callbacks()`` once
inside ``_make_lead_agent`` and append the result to
``config["callbacks"]``; the four in-graph ``create_chat_model`` sites
(bootstrap, default agent, sync + async summarization) pass
``attach_tracing=False`` to avoid duplicate spans.
- ``agents/middlewares/title_middleware.py``: same ``attach_tracing=False``
for the title-generation model, since it inherits the graph's
RunnableConfig via ``_get_runnable_config``.
Test updates:
- ``tests/test_lead_agent_model_resolution.py`` and
``tests/test_title_middleware_core_logic.py``: extend the fake
``create_chat_model`` signatures / mock assertions to accept the new
``attach_tracing`` kwarg.
- ``tests/test_worker_langfuse_metadata.py``: switch the no-user fallback
test from direct ContextVar mutation to ``monkeypatch.setattr`` on
``get_effective_user_id`` to avoid pollution across the langfuse OTel
global tracer provider.
- ``tests/conftest.py``: add an autouse fixture that resets
``deerflow.config.title_config._title_config`` to its pristine default
after every test. Any test that loads the real ``config.yaml`` (via
``get_app_config()``) calls ``load_title_config_from_dict`` and mutates
the module-level singleton, which previously poisoned the
title-middleware suite when run after, e.g., the new
``test_worker_langfuse_metadata.py`` cases. The fixture is independent
of this PR's main change but unblocks the cross-file test run.
Live verification (same Langfuse instance as before):
- Drove ``worker.run_agent`` against the real ``make_lead_agent`` +
``gpt-4o-mini`` for three distinct ``user_context`` identities
(``fancy-engineer``, ``alice-pm``, ``bob-designer``).
- Each run produced one ``lead-agent`` trace whose top-level
``sessionId`` / ``userId`` / ``tags`` carry the expected values, e.g.
``session=e2e-2930-8f347c-alice-pm user=alice-pm name='lead-agent'
tags=['model:gpt-4o-mini']``.
Refs #2930.
* fix(tracing): extend root-callback + metadata injection to the embedded client
Addresses Copilot review on PR #2944.
Commit 2 disabled model-level tracing for ``TitleMiddleware`` and
``_create_summarization_middleware`` because ``_make_lead_agent`` now
attaches the tracing callbacks at the graph invocation root. But the
embedded ``DeerFlowClient`` does not call ``_make_lead_agent`` — it
calls ``_build_middlewares`` directly and never appends the tracing
handlers to its ``RunnableConfig``. So under the embedded path,
title-generation and summarization LLM calls were left untraced —
a regression introduced by this PR.
This commit mirrors the gateway worker's injection in
``DeerFlowClient.stream``:
- Append ``build_tracing_callbacks()`` to ``config["callbacks"]`` so
the Langfuse handler sees ``on_chain_start(parent_run_id=None)`` at
the graph root and runs the ``propagate_attributes`` path.
- Merge ``build_langfuse_trace_metadata(...)`` into
``config["metadata"]`` with ``setdefault`` so caller-supplied keys
still win.
- ``_ensure_agent`` now creates its main model with
``attach_tracing=False`` to avoid duplicate spans now that the
callback lives at the graph root.
Docs:
- ``backend/CLAUDE.md`` Tracing section rewritten to describe the
graph-root attachment model (replacing the inaccurate
"at model-creation time" wording).
- ``README.md`` Langfuse section now lists both injection points
(worker + client) instead of only the worker path.
Tests:
- ``tests/test_client_langfuse_metadata.py`` (new, 3 cases):
callbacks + metadata are injected when Langfuse is enabled,
caller-supplied metadata overrides win via ``setdefault``, and the
injection is inert when Langfuse is disabled.
Live verification on the real Langfuse instance:
=== user=fancy-client ===
id=cbd22847.. session=client-2930-6b9491-fancy-client user=fancy-client name='lead-agent'
=== user=alice-client ===
id=b4f6f576.. session=client-2930-6b9491-alice-client user=alice-client name='lead-agent'
Refs #2930.
* refactor(tracing): address maintainer review on PR #2944
Addresses @WillemJiang's 5 comments.
1. Duplicated metadata-injection code between worker.py and client.py
New ``deerflow.tracing.inject_langfuse_metadata(config, ...)`` helper
takes the 10-line build + merge + setdefault logic that was duplicated
in ``runtime/runs/worker.py`` and ``client.py``. Both callers now share
a single source of truth, so the two paths cannot drift.
2. Direct private-attribute mutation in conftest.py and tests
Added public ``reset_tracing_config()`` / ``reset_title_config()``
functions. ``tests/conftest.py`` and every test that previously did
``tracing_module._tracing_config = None`` or
``title_module._title_config = TitleConfig()`` now goes through the
public API. A future internal rename will surface as an ImportError
instead of a silent no-op.
3. client.py reading os.environ directly
``DeerFlowClient.__init__`` grows an optional ``environment`` parameter
so programmatic callers can pass the deployment label explicitly.
``stream()`` consults ``self._environment`` first and only falls back
to ``DEER_FLOW_ENV`` / ``ENVIRONMENT`` env vars when nothing was
passed in. Backwards compatible — env-var behaviour preserved for
callers that opt to keep using it.
4. build_tracing_callbacks() cached on hot path
Not implemented. Inspected the langfuse v4 ``langchain.CallbackHandler``
constructor: it only resolves the module-level singleton client via
``get_client()`` and initialises a few dicts (no I/O, no env parsing
at construction time). The build is essentially free. Caching would
trade a non-measurable speedup for two real risks: handler instances
carry per-run state internally (``_run_states``, ``_root_run_states``,
``last_trace_id``), and tracing config can be reloaded by env-var
changes between runs. Will revisit if profiling ever shows it as
a hot spot.
5. attach_tracing=False easy to forget at new in-graph call sites
- Module docstring at the top of ``lead_agent/agent.py`` documents
the invariant ("every in-graph ``create_chat_model`` MUST pass
``attach_tracing=False``") and enumerates the current sites.
- New regression test
``test_make_lead_agent_attaches_tracing_callbacks_at_graph_root`` in
``tests/test_lead_agent_model_resolution.py`` locks both halves of
the invariant: ``config["callbacks"]`` carries the tracing handler
after ``_make_lead_agent``, AND every ``create_chat_model`` call
captured by the test passes ``attach_tracing=False``. A future
in-graph site that forgets the flag will fail this test.
Lint clean. Full touched-suite bundle: 246 passed.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
|
||
|
|
eab7ae3d62 |
feat: stream subagent token usage to header via terminal task events (#2882)
* feat: real-time subagent token usage display in header and per-turn Backend: - Persist subagent token usage to AIMessage.usage_metadata via TokenUsageMiddleware, so accumulateUsage() naturally includes subagent tokens without frontend state management - Cache subagent usage by tool_call_id in task_tool, write back to the dispatching AIMessage on next model response - Emit subagent token usage on all terminal task events (task_completed, task_failed, task_cancelled, task_timed_out) - Report subagent usage to parent RunJournal for API totals - Search backward from ToolMessage to find dispatching AIMessage for correct multi-tool-call attribution Frontend: - Remove subagentUsage state, custom event handling, and prop threading — subagent tokens are now embedded in message metadata - Simplify selectHeaderTokenUsage (no subagentUsage parameter) - Per-turn inline badges show turn-specific usage via message accumulation - Remove isLoading guard from MessageTokenUsageList for dynamic updates during streaming * fix: prevent header token double counting from baseline reset race onFinish, onError, and thread-switch useEffect all reset pendingUsageBaselineMessageIdsRef to an empty Set. If thread.isLoading is still true on the next render, all messages pass the getMessagesAfterBaseline filter and their tokens are added to backendUsage (which already includes them), causing the header to display up to 2× the actual token count. Capture current message IDs instead of using an empty Set so that getMessagesAfterBaseline correctly returns no pending messages even if thread.isLoading lags behind the stream end. * fix: write back subagent tokens for all concurrent task tool calls TokenUsageMiddleware only processed messages[-2], so when a single model response dispatched multiple task tool calls only the last ToolMessage had its cached subagent usage written back to the dispatch AIMessage.usage_metadata. Earlier tasks' usage stayed in _subagent_usage_cache indefinitely (leak) and never appeared in the per-turn inline token display. Walk backward through all consecutive ToolMessages before the new AIMessage, and accumulate updates targeting the same dispatch message into one state update so overlapping writes don't clobber each other. * fix: clean up subagent usage cache entry on task cancellation When a task_tool invocation is cancelled via CancelledError, any cached subagent usage entry leaked because the TokenUsageMiddleware writeback path never fires after cancellation. Pop the cache entry before re-raising to prevent unbounded growth of the module-level _subagent_usage_cache dict. * fix: address token usage review feedback * fix: handle missing config for subagent usage cache --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
AochenShen99andGitHub
|
c3bc6c7cd5 |
fix(nginx): defer CORS to gateway allowlist (#2861)
* fix(nginx): defer cors to gateway allowlist Remove proxy-level wildcard CORS handling so browser origins are controlled by the Gateway allowlist and stay aligned with CSRF origin checks. * docs: document gateway cors allowlist Clarify that same-origin nginx access needs no CORS headers while split-origin or port-forwarded browser clients must opt in with GATEWAY_CORS_ORIGINS. * docs(gateway): record cors source of truth Document that Gateway CORSMiddleware and CSRFMiddleware share GATEWAY_CORS_ORIGINS as the split-origin source of truth. * fix(gateway): align cors origin normalization * docs: clarify gateway langgraph routing * docs(gateway): update runtime routing note |
||
|
|
c09c334544 |
fix(harness): resolve runtime paths from project root (#2642)
* fix(harness): resolve runtime paths from project root * docs(config): update * fix(config): address runtime path review feedback * test(config): fix skills path e2e root * test(config): cover legacy config fallback when project root lacks config files Verifies that when DEER_FLOW_PROJECT_ROOT is unset and cwd has no config.yaml/extensions_config.json, AppConfig and ExtensionsConfig fall back to the legacy backend/repo-root candidates — the backward-compat path requested in PR #2642 review. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
|
08afdcb907 |
feat(channels): add DingTalk channel integration (#2628)
* feat(channels): add DingTalk channel integration Add a new DingTalk messaging channel using the dingtalk-stream SDK with Stream Push (WebSocket), requiring no public IP. Supports both plain sampleMarkdown replies and optional AI Card streaming for a typewriter effect when card_template_id is configured. - Add DingTalkChannel implementation with token management, message routing, allowed_users filtering, and markdown adaptation - Register dingtalk in channel service registry and capability map - Propagate inbound metadata to outbound messages in ChannelManager for DingTalk sender context (sender_staff_id, conversation_type) - Add dingtalk-stream dependency to pyproject.toml - Add configuration examples in config.example.yaml and .env.example - Update all README translations with setup instructions - Add comprehensive test suite (test_dingtalk_channel.py) and metadata propagation test in test_channels.py - Update backend CLAUDE.md to document DingTalk channel Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(channels): address PR review feedback for DingTalk integration - Replace runtime mutation of CHANNEL_CAPABILITIES with a `supports_streaming` property on the Channel base class, overridden by DingTalkChannel, FeishuChannel, and WeComChannel - Store stream client reference and attempt graceful disconnect in stop(); guard _on_chatbot_message with _running check to prevent post-stop message processing - Use msg.chat_id as the primary routing key in send/send_file via a shared _resolve_routing helper, with metadata as fallback - Fix process() return type annotation from tuple[str, str] to tuple[int, str] to match AckMessage.STATUS_OK - Protect _incoming_messages with threading.Lock for cross-thread safety between the Stream Push thread and the asyncio loop - Re-add Docker Compose URL guidance removed during DingTalk setup docs addition in README.md - Fix incomplete sentence in README_zh.md (missing verb "启用") Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(docs): restore plain paragraph format for Docker Compose note Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(channels): fix isinstance TypeError and add file size guard in DingTalk channel Use tuple syntax for isinstance() type check to avoid runtime TypeError with PEP 604 union types. Add upload size limit (20MB) before reading files into memory. Narrow exception handlers to specific types. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(channels): propagate markdown fallback errors and validate access token response - Re-raise exceptions in _send_markdown_fallback to prevent partial deliveries (files sent without accompanying text) - Validate _get_access_token response: reject non-dict bodies, empty tokens, and coerce invalid expireIn to a safe default - Add tests for both fixes Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(channels): validate upload response and broaden send_file exception handling - Validate _upload_media JSON response: handle JSONDecodeError and non-dict payloads gracefully by returning None - Broaden send_file exception tuple to include TypeError and AttributeError for unexpected JSON shapes Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(channels): fix streaming race on channel registration and slim outbound metadata - Register channel in service before calling start() to avoid race where background receiver publishes inbound before registration, causing manager to fall back to static CHANNEL_CAPABILITIES - Strip known-large metadata keys (raw_message, ref_msg) from outbound messages to prevent memory bloat from propagated inbound payloads Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Update service.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update CLAUDE.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Willem Jiang <willem.jiang@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
JeffJiang
|
da174dfd4d | feat: implement process-local internal authentication for Gateway and enhance CSRF handling | ||
|
JeffJiangandCopilot
|
7bf618de67 |
Refactor DeerFlow to use Gateway's LangGraph-compatible API
- Updated documentation and comments to reflect the transition from LangGraph Server to Gateway. - Changed default URLs in ChannelManager and tests to point to Gateway. - Removed references to LangGraph Server in deployment scripts and configurations. - Updated Nginx configuration to route API traffic to Gateway. - Adjusted frontend configurations to utilize Gateway's API. - Removed LangGraph service from Docker Compose files, consolidating services under Gateway. - Added regression tests to ensure Gateway integration works as expected. Co-authored-by: Copilot <copilot@github.com> |
||
|
|
8a044142cb |
feat(dev): add pre-commit hooks for ruff, eslint, and prettier (#2525)
* feat(dev): add pre-commit hooks for ruff, eslint, and prettier * fix: use local uv-based ruff hooks and uv run for pre-commit install Agent-Logs-Url: https://github.com/bytedance/deer-flow/sessions/a1e34cc5-0d4b-4400-9e6a-e687d964ff1e Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
|
5db71cb68c |
fix(middleware): repair dangling tool-call history after loop interru… (#2035)
* fix(middleware): repair dangling tool-call history after loop interruption (#2029) * docs(backend): fix middleware chain ordering --------- Co-authored-by: luoxiao6645 <luoxiao6645@gmail.com> |
||
|
|
fa96acdf4b |
feat: add WeChat channel integration (#1869)
* feat: add WeChat channel integration * fix(backend): recover stale channel threads and align upload artifact handling * refactor(wechat): reduce scope and restore QR bootstrap * fix(backend): sort manager imports for Ruff lint * fix(tests): add missing patch import in test_channels.py * Update backend/app/channels/wechat.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update backend/app/channels/manager.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix(wechat): streamline allowed file extensions initialization and clean up test file --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
DanielWalnutandGitHub
|
eef0a6e2da | feat(dx): Setup Wizard + doctor command — closes #2030 (#2034) |