* fix(channels): scope IM files and helper commands to owner
* fix(memory): honor bound IM owner for /memory gateway endpoints
The channel manager already attaches X-DeerFlow-Owner-User-Id for /memory
and /models, but the memory router resolved user_id solely from
get_effective_user_id(), which returns the synthetic internal user
(DEFAULT_USER_ID) for channel workers. A bound IM /memory therefore read
the default/internal memory instead of the connection owner's.
Resolve the owner via _resolve_memory_user_id(request) across all
/api/memory* endpoints: trusted internal callers act for the owner header,
browser/API callers fall back to get_effective_user_id(). Mirrors the
threads router's get_trusted_internal_owner_user_id pattern, completing
acceptance criterion #3 of #3539.
Add end-to-end tests asserting the resolved user_id (not just that the
header is sent) and that a spoofed owner header from a browser user is
ignored.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): align memory bucket and reuse cached storage owner
Address PR #3579 review feedback:
- Memory router now sanitizes the trusted owner header via make_safe_user_id
before routing, matching the channel file pipeline
(_safe_user_id_for_run/prepare_user_dir_for_raw_id). A bound owner id needing
sanitization now resolves to the same bucket as its files/uploads instead of
500ing in _validate_user_id.
- _handle_chat reuses the storage_user_id cached at the top of the method for
artifact delivery instead of re-deriving _channel_storage_user_id(msg), so
uploads and outputs cannot drift to different buckets if a channel rewrites
the InboundMessage in receive_file.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): stage unbound IM files under the run's user bucket
Address PR #3579 review feedback (#5): _channel_storage_user_id now mirrors
_resolve_run_params' identity policy, falling back to safe(msg.user_id) instead
of returning None for unbound auth-enabled channels.
Previously an unbound msg ran under safe(platform_user_id) but staged uploads
under get_effective_user_id() in the dispatcher task (unset contextvar ->
"default"), so files landed in users/default/... while the agent read from
users/{safe_platform_user_id}/.... Bound and unbound channels now write where
the agent reads. Returns None only when no identity is available.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): reuse cached storage owner in streaming artifact delivery
Address PR #3579 review feedback (#6): thread the storage_user_id resolved in
_handle_chat into _handle_streaming_chat instead of re-deriving
_channel_storage_user_id(msg) in the finally block. Avoids re-running
_safe_user_id_for_run (and its possible filesystem touch) on the streaming-error
path and guarantees artifact delivery targets the same bucket as the uploads.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(channels): document owner-scoped IM file storage
Address PR #3579 review feedback (#4): the IM Channels and File Upload sections
still described pre-PR default-bucket behaviour. Document that receive_file,
_ingest_inbound_files/ensure_uploads_dir/get_uploads_dir, and
_resolve_attachments/_prepare_artifact_delivery are owner-scoped via the user_id
kwarg, and that the bucket matches the memory bucket from _resolve_memory_user_id.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(channels): unify run identity and storage bucket resolution
Address PR #3579 review feedback (#3): _resolve_run_params no longer duplicates
the owner-resolution rule inline. After the #5 fix the inline block and
_channel_storage_user_id computed the identical sanitized-with-platform-fallback
value, so the run identity now calls the same helper, making it the single
source of truth for run_context["user_id"] and the file/artifact storage bucket.
_owner_headers stays deliberately separate: it sends the raw owner id over HTTP
for the gateway to re-resolve (no sanitize, no platform fallback), documented on
both helpers. test_run_identity_matches_storage_bucket pins the two together so
they cannot drift again.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): harden runtime credential management APIs
* fix(channels): address review feedback on credential hardening
Follow-up to the runtime credential-hardening pass, resolving five review
findings:
- WeChat auth persistence now writes through a 0o600 NamedTemporaryFile +
Path.replace instead of write_text-then-chmod, so the iLink bot_token is
never briefly readable at umask defaults (mirrors ChannelRuntimeConfigStore).
- The post-write chmod is split into its own try/except: a chmod failure on a
filesystem without POSIX perms now logs at debug instead of masquerading as
a "failed to persist" warning.
- Extracted the three near-identical _require_admin_user helpers (mcp,
channel_connections, channels) into a single require_admin_user(request, *,
detail) in app/gateway/deps.py; each router supplies its own detail string.
- Strengthened the runtime-config-store chmod coverage: a new test injects a
temp-file chmod failure and asserts it is logged at debug while the
destination is still owner-only (mutation-verified to fail if the chmod is
dropped), plus a loose-pre-existing-file case.
- Removed the unused _FakeRepo from the blocking-io test: its isinstance gate
routes through the repo-less 503 path, so neither stub was ever invoked.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): make channel connect flow deterministic
* make format
* fix(channels): apply connect-code before allowed_users on telegram and wechat
The bind-bootstrap reorder shipped for slack/dingtalk only. Telegram and
WeChat still gate _check_user/allowed_users before connect-code dispatch, so
a newly allowlisted-but-unbound user is silently rejected when binding via the
browser deep-link / connect-code flow — the same deadlock the PR fixes.
- telegram: consume the /start deep-link token before the allowed_users gate.
- wechat: handle the /connect code before the allowed_users gate, and defer
inbound file extraction + context-token tracking past the gate so blocked
senders no longer trigger CDN downloads or token bookkeeping.
Adds regression tests for both adapters mirroring the slack/dingtalk coverage.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): enforce single-active-owner invariant at the DB layer
_revoke_other_active_owners did a SELECT-then-UPDATE in app code with no row
lock or constraint covering active rows. Under READ COMMITTED, two concurrent
connect-code consumes for the same (provider, external_account_id, workspace_id)
from different owners could each observe "no other active owner" and both commit
a connected row, leaving find_connection_by_external_identity nondeterministic.
- Add a partial unique index on (provider, external_account_id, workspace_id)
WHERE status != 'revoked' (portable to SQLite >= 3.8.0 and PostgreSQL) so the
database guarantees at most one non-revoked row per external identity.
- Reorder upsert_connection to revoke other owners' active rows before the new
connected row is flushed (so the index is satisfied at commit), wrapped in a
bounded rollback-and-retry loop. A losing concurrent writer now retries
against the now-visible state instead of committing a duplicate.
Adds DB-constraint, revoked-slot-reuse, and concurrent-upsert regression tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): harden connect-status polling primitive
pollChannelConnectionUntilResolved was a free-floating recursive setTimeout
started from onSuccess with no cancellation, no per-provider dedup, a redundant
second endpoint per tick, and an unbounded loop on a non-finite expires_in.
- Extract a framework-agnostic, cancellable poller (connect-poll.ts) that polls
only listChannelConnections() and invalidates the providers query once when the
bind resolves, instead of fetching both endpoints every tick.
- Guard expires_in with a finite check + default window so undefined/NaN can no
longer produce a poll loop that runs until the page closes.
- Track one active poll handle per provider in useConnectChannelProvider via a
ref Map: a new connect cancels the prior poll for that provider, and a useEffect
cleanup cancels all polls on unmount.
Adds unit tests for resolve-and-stop, cancellation, and non-finite-expiry.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): stop leaking blocked-sender content in DingTalk INFO log; document bind semantics
Moving the allowed_users gate past _extract_text meant the parsed-message INFO
log (text=%r, first 100 chars) fired for senders that allowed_users would have
rejected, defeating the filter's noise/privacy role. Move that log to after the
allowed_users gate so blocked senders' message text never reaches INFO logs.
Also document the two operator-relevant semantic changes in backend/CLAUDE.md:
connect-code dispatch runs before allowed_users (so allowed_users is no longer a
bind-time defense; the model relies on code confidentiality + 600s TTL + one-time
consumption), and the single-active-owner-per-external-identity transfer semantics
now backed by the partial unique index.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(channels): note connect-code-vs-allowlist and ownership transfer in operator guide
Mirror the backend/CLAUDE.md notes in the operator-facing IM_CHANNEL_CONNECTIONS.md:
connect codes are consumed before allowed_users (so a not-yet-allowlisted user can
still complete a first bind, and allowed_users is not a bind-time defense), and an
external identity has at most one active owner with last-bind-wins transfer enforced
at the DB layer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(channels): lift connect-code dispatch into Channel base class
Each adapter duplicated the ordering-sensitive boilerplate of extracting a
/connect code and guarding on the connection repo before its allowed_users gate.
The duplication is what let telegram/wechat drift and keep the gate ahead of the
bind. Centralize it:
- Move `_connection_repo` onto Channel.__init__ (removing 7 duplicate assignments).
- Add Channel._pending_connect_code(text), which guards on the repo and extracts
the code, documenting that adapters MUST consult it before authorization so a
browser-initiated bind can bootstrap a not-yet-authorized identity.
- Route slack, discord, feishu, dingtalk, wechat, and wecom through the helper.
This also fixes a latent inconsistency where slack dispatched a bind even when
no connection repo was configured.
Pure refactor — the full channel suite stays green; adds a direct unit test for
the base helper's contract.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* make format
* fix(channels): redact DingTalk parsed-message INFO log content
Log text_len instead of the first 100 chars of message text, so message
content never reaches INFO logs (the after-gate move already keeps blocked
senders out entirely). This takes over the redaction from #3584 so only this
PR touches dingtalk.py, letting the two PRs merge in any order conflict-free.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): add operational guardrails
* make format
* fix(channels): converge with #3582 to avoid merge-order conflicts
Drop this PR's DingTalk INFO-log redaction and hand it to #3582, which
already restructures that handler and will redact the same log there. This
PR no longer touches dingtalk.py, so the two PRs can merge to main in any
order without a conflict.
For WeChat, drop the contested thread_ts priority reorder (review #3) and
keep only what inbound dedupe needs: a server-stable message_id in the
inbound metadata (message_id/msg_id, no client_id per review #6). This is a
single added line inside the metadata dict, a region #3582 never touches, so
it auto-merges regardless of order.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): address three correctness review findings
1. Connect-code cap was racy (willem #1): _create_state ran delete-expired,
count, and insert as three separate transactions, so concurrent connect
POSTs from one owner could each see count < cap and all insert past it. Add
ChannelConnectionRepository.create_oauth_state_within_cap which does
delete+count+insert in a single transaction serialized per (owner,
provider) — Postgres via pg_advisory_xact_lock, SQLite via the write lock
the leading DELETE takes — and have the router use it.
2. Inbound dedupe key fell back to "" workspace (willem #3): two workspaces
delivering without team/guild/aibotid would collapse to the same key and
dedupe each other's messages. _inbound_dedupe_key now fails closed
(returns None) when no workspace identifier is present.
3. Dedupe key was recorded on receipt and never released on failure
(ShenAC #1): a transient error (DB blip, Gateway 503) left the key in place
for the full TTL, so a provider redelivery of the same message_id — exactly
the retry dedupe should absorb — was silently dropped. _handle_message now
releases the key in the unexpected-exception branch so redelivery can
recover, while keeping record-on-receipt so retries during handling are
still deduped.
Tests: repo cap enforcement incl. concurrent-issuance non-leak; dedupe
fail-closed; dedupe key release-on-failure redelivery recovery.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): address cleanup/efficiency and test review findings
Efficiency / cleanup:
- Dedupe key set drops client-generated ids (client_msg_id, client_id);
keep only server-stable event_id/message_id/msg_id, which a provider's own
redelivery preserves (ShenAC #6). Every provider already emits message_id.
- TTL/overflow pruning of _recent_inbound_events is now O(k): switch to an
OrderedDict and popitem(last=False) from the front instead of scanning all
4096 entries on every inbound (willem #4).
- Log "received inbound" only after the dedupe check so a provider retrying N
times no longer logs N accepts; document that manager dedupe covers the
agent run/final answer, not provider ack side-effects (willem #5, ShenAC #2).
- Slack drops the redundant `team_id or event.get("team")` fallback the caller
already resolved (willem #6).
- create_oauth_state_within_cap prunes only this owner/provider's expired codes
instead of a global DELETE on every connect POST; global cleanup still runs
on consume_oauth_state (willem #7).
Tests:
- Dedupe test uses tmp_path instead of a leaked mkdtemp, uses distinct objects
per publish, and adds a negative control: a different message_id is still
processed, catching over-dedupe regressions (willem #8, ShenAC #4).
- Slack HTTP-mode rejection test supplies app_token so the missing-token early
return can't mask the guard, giving the state assertions teeth (ShenAC #3).
- count_oauth_states test pins that the active row survives, not just the count
(ShenAC #5).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* make format
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(sandbox): stop flagging string-literal path fragments as unsafe paths
The host-bash absolute-path guard scans the raw command string, so /segment
sequences inside string literals, f-strings, and templates were treated as
absolute path arguments and rejected — e.g. python -c "print(f'/端口{port}')"
or a REST template /devices/{id}/port. Whether a fragment tripped the guard
depended on the character right before the slash (a word char suppressed the
match), so the same literal could pass or fail unpredictably, pushing the model
into retry loops that bloat context and wall-clock time.
Exempt matches carrying non-ASCII characters or format braces: real host paths
a command would open contain neither, so these are text, not paths. The guard
is best-effort (not a security boundary), and plain ASCII host paths like
/etc/passwd — including ones written inside a code string such as
open('/etc/passwd') — stay rejected.
* fix(sandbox): only exempt identifier-template braces, not bash brace expansion
The literal-fragment exemption exempted any path fragment containing { or },
which let bash brace expansion (cat /etc/{passwd,shadow}) and ${VAR} expansion
reconstitute real host paths past validate_local_bash_command_paths. Tighten
the brace branch to only exempt fragments where every {...} block is a single
identifier-like placeholder (/devices/{id}/port, f-string /{port}); reject
${VAR} shell-variable expansion. Add parametrized regression tests for the
brace-expansion and shell-var bypasses.
* feat(community): add Serper Google Images provider for image_search
Add a Serper-backed `image_search` tool alongside the existing Serper
`web_search` provider, so users with a SERPER_API_KEY can pull Google
Images results as reference images for downstream image generation.
- Share request/response handling between web_search and image_search
via `_serper_post` / `_response_items`, with bounded `max_results`
(capped at 10) and query normalization.
- Add a best-effort SSRF guard (`_safe_public_url`) that rejects
non-http(s), localhost and private/non-global IP image URLs; filtered
entries are dropped and never consume the result limit.
- doctor: flag literal `api_key` values in config as a warning and steer
users toward `.env` + `$SERPER_API_KEY`.
- Docs/config: document the Serper image_search provider and SERPER_API_KEY,
and discourage committing literal keys to config.yaml.
- Tests: cover the provider end-to-end (100% line coverage on tools.py)
and the doctor literal-key warning path.
* fix(community): block obfuscated IPv4 literals in Serper image SSRF guard
The image_search SSRF guard only rejected dotted-decimal IP literals; encoded
forms such as decimal (http://2130706433/), hex (0x7f000001) and octal
(0177.0.0.1) raised ValueError in ip_address() and were allowed through, even
though many HTTP clients resolve them to private addresses like 127.0.0.1.
Add _decode_ipv4() to permissively decode these inet_aton-style encodings and
apply the same is_global check; hostnames that do not decode to an IP (e.g.
cafe.com) are still treated as hosts and left to fetch-time re-validation.
Addresses PR review feedback. Tests cover decimal/hex/octal loopback and
private encodings plus non-IP edge cases; tools.py stays at 100% line coverage.
* test(community): cover IPv4-mapped IPv6 URL filtering
* fix(community): address Serper image search review feedback
- Block trailing-dot hostname SSRF bypass (localhost./127.0.0.1.) in
_safe_public_url by stripping the FQDN root label before checks.
- Keep a filtered image/thumbnail URL empty instead of collapsing onto
its counterpart, preserving the high-res/preview contract.
- Evaluate the SSRF guard once per field rather than twice.
- Treat a null-typed organic/images field as "no results" rather than a
malformed payload.
- doctor.py: when a config $VAR is unset, fall through to the default env
var before reporting it as not set.
read_file decodes with UTF-8. Binary uploads (.xlsx, images, ...) raise
UnicodeDecodeError in the sandbox layer. UnicodeDecodeError is a ValueError
subclass, not an OSError, so it bypassed the typed handlers and fell through
to the generic except, surfacing a vague "Unexpected error reading file"
message. The model could not tell the file was binary, so it retried
read_file instead of switching to bash + pandas/openpyxl, burning LLM
round-trips and bloating context with repeated failures.
Add a dedicated UnicodeDecodeError handler that tells the model the file is
binary and to use bash with a suitable library (or view_image for images).
The subagent execution path did not call inject_langfuse_metadata(...)
and built its model with attach_tracing=True, so subagent LLM/tool
spans landed in Langfuse as isolated top-level traces carrying fresh
session ids and the default user. They were findable in the unfiltered
trace list but did not group under the parent thread's session card,
and Langfuse cost attribution for subagent traffic did not line up
with the parent conversation — even though DeerFlow's internal token
accounting (SubagentTokenCollector) was already correct.
Extend the lead-agent tracing wiring to the subagent path so a single
subagent run produces one trace that shares the parent thread's
session_id and user_id, with a subagent:<name> trace name:
- subagents/executor.py: append build_tracing_callbacks() output to
run_config["callbacks"] (preserving SubagentTokenCollector) and
call inject_langfuse_metadata(...) with thread_id, user_id, and
the normalized subagent:<name> trace name. Build the model with
attach_tracing=False so model-level tracing does not double-count
with the graph-root callbacks — the same pairing the lead agent
uses.
- tools/builtins/task_tool.py: resolve user_id via
resolve_runtime_user_id(runtime) at the parent tool layer (before
the background thread starts) and thread it through
SubagentExecutor.__init__, because the _current_user contextvar
is not guaranteed to survive the _execution_pool boundary.
Trace topology is unchanged: subagent traces remain separate top-level
traces in the same session, not nested as child spans under the lead
trace (Plan B follow-up).
Tests: tests/test_subagent_executor.py::TestSubagentTracingWiring
covers the callback append, the session/user/trace-name injection,
the disabled-langfuse no-op, the DEFAULT_USER_ID fallback, the
empty-name trace-name fallback, and the env-tag emission. Existing
test_create_agent_threads_explicit_app_config_to_model_and_middlewares
now also asserts attach_tracing=False.
Docs: CLAUDE.md Tracing System section documents subagents/executor.py
as a third injection point alongside worker.py and client.py.
* fix(sandbox): create shell session before retrying on a fresh id
The AIO sandbox recovery path generated a UUID and passed it straight to
exec_command(id=...). The sandbox image only auto-creates a session when
exec_command is called with *no* id; an exec carrying an unknown id returns
HTTP 404 "Session not found". So every ErrorObservation recovery itself
404'd, turning a transient session lapse into an unrecoverable tool error
that looped the run up to the LangGraph recursion limit.
Explicitly create_session(id=fresh_id) before targeting that id on retry.
create_session is idempotent (returns the existing session if the id already
exists), so this is safe under the serializing lock.
Updated the regression test to assert the retry targets exactly the
created session id rather than a fabricated, uncreated one.
* fix(sandbox): release the one-shot recovery session after retry
The fresh session created on the ErrorObservation recovery path is used for
exactly one command -- the next execute_command runs with no id and returns
to the default session. Under persistent session corruption every command
would create another session that is never reused or released, accumulating
sessions on the container.
Release it best-effort with cleanup_session() in a finally, swallowing any
cleanup error so it never masks a successful retry.
Addresses review feedback on #3577.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(channels): make runtime provider state authoritative
* make format
* fix(channels): close runtime provider config races and status gaps
Address review findings on the runtime-provider-state change:
- configure/disconnect now re-read the live app.state.channels_config
after the worker await and mutate only the affected provider key in
place, so a concurrent mutation for a different provider is no longer
clobbered by a stale pre-await snapshot.
- disconnect revokes DB connection rows before committing the store and
cache, so a repo failure cannot leave the store/cache "disconnected"
while the DB keeps "connected" rows a later re-configure would
silently reactivate.
- _provider_response preserves non-connected statuses (e.g. revoked)
when the provider is unavailable, only masking a stale "connected"
row as not_connected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): require bound identity for user-owned IM messages
* make format
* docs: document bound identity channel config
* refactor: reuse channel connection config
* refactor _requires_bound_identity()
* refactor from_app_config()
* make format
* fix: reject unbound channel chats before semaphore
* security enhancement
* make format
* fix: enforce bound-identity admission at command entry point
The bound-identity gate only ran for non-command messages in
_handle_message() and as a fallback inside _handle_chat(). Commands had
no equivalent boundary, so an unbound platform user could send /new and
reach _create_thread() directly, creating an unowned Gateway thread and
empty checkpoint. Info commands (/status, /models, /memory) likewise
leaked Gateway state to unbound users.
Add the same _requires_bound_identity() check at the top of
_handle_command(), rejecting via _reject_unbound_channel_message() before
any thread creation or Gateway query. The gate is a no-op in legacy
open-bot mode (require_bound_identity=False) and auth-disabled mode.
Provider-level binding flows (/connect, /start) are consumed by the
provider adapter before reaching the manager, so they are unaffected.
Tests:
- unbound auth-enabled /new is rejected before threads.create
- bound auth-enabled /new still creates the thread
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(channels): carry workspace fallback decision on inbound messages
* fix(channels): recheck bound identity by normalized workspace
* fix(channels): avoid duplicate bound identity checks
* fix(channels): preserve verified routing for bound identity rejects
* fix(channels): clarify bound identity upgrade failures
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(subagents): raise general-purpose max_turns to 150 and default timeout to 30min
Deep-research subtasks failed out of the box with GraphRecursionError (Recursion limit of 100 reached): the built-in general-purpose subagent caps at max_turns=100. Raise it to 150 and bump the default subagent timeout from 900s (15min) to 1800s (30min) so the extra turns have time to run instead of shifting the failure to a timeout. The lead agent recursion_limit (100) is unchanged; the failures are subagent-only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(subagents): clarify lead recursion_limit is independent of subagent max_turns
Add comments at both lead recursion_limit=100 sites (gateway services + channel manager) explaining the lead's LangGraph super-step budget is separate from subagent depth, so the two 100s are not conflated. Comment-only, no behavior change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(subagents): clarify built-in vs custom timeout scope; pin bash max_turns in test
Review follow-ups: (1) clarify SubagentConfig docstring + global timeout field/comment that the 1800 default applies to built-in subagents (custom agents keep their own timeout_seconds); (2) pin bash.max_turns==60 in the defaults regression test so the config.example.yaml doc cannot drift; (3) rename test_default_timeout_preserved_when_no_config -> test_explicit_global_timeout_propagates_to_general_purpose since it intentionally exercises an explicit non-default 900. No runtime behavior change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(agents): sync agent_name across context/configurable and reject empty soul (#3549)
Two independent issues caused custom agent creation to silently fail:
1. build_run_config only wrote agent_name into one container (configurable
or context), so setup_agent — which reads ToolRuntime.context exclusively
since LangGraph >=1.1.9 — saw agent_name=None and wrote SOUL.md to the
global base_dir instead of users/{user_id}/agents/{name}/. Mirror the
dual-write pattern already used by merge_run_context_overrides and
naming.py so both containers always carry the same value.
2. setup_agent persisted whatever soul string it received, including empty
or whitespace-only content, and still reported success. The frontend
then surfaced an unusable agent and the global default SOUL.md could be
silently overwritten with empty content. Reject empty soul before any
filesystem operation so the model can retry.
Tests:
- test_gateway_services.py: dual-write regressions for both configurable
and context entry paths, explicit-agent-name precedence on both sides,
and a shape-parity test against merge_run_context_overrides.
- test_setup_agent_tool.py: empty/whitespace soul rejection, plus
no-overwrite guarantees for existing global and per-agent SOUL.md.
* Update services.py
Subagent _create_agent() now passes checkpointer=False to prevent
inheriting the parent run's synchronous checkpointer via copy_context(),
which would cause NotImplementedError when aget_tuple() is called on
the async path. Subagents are one-shot delegations that never resume,
so persistence is unnecessary.
send_file opened the attachment with a bare open() and never closed it,
leaking a file descriptor on every Discord file delivery. The handle was
also leaked on the failure path: when target.send raised, the except
branch logged and returned without closing fp. The "# noqa: SIM115"
suppressed the lint warning instead of fixing it.
Wrap open() in a with statement that stays open for the full upload —
the discord client reads fp while target.send runs on _discord_loop, and
once that future resolves the bytes are consumed, so closing here is
safe. This closes the handle on both the success and exception paths and
matches how telegram and feishu already handle their file uploads.
Adds regression tests asserting the handle is closed after send_file on
both the success and failure paths.
Refs #3544
* feat(community): add Brave Search web search tool
Add a community web_search provider backed by the official Brave Search
API (https://api.search.brave.com/res/v1/web/search). API key is read
from the tool config (inline api_key) or the BRAVE_SEARCH_API_KEY env
var. Output schema (title/url/content) matches existing search tools.
No new dependencies (uses the existing httpx). Also wires up the setup
wizard, doctor health check, config example, and EN/ZH docs.
* refactor(community): drop redundant [:count] slice in Brave search
The Brave API already caps results via the `count` request param, so
client-side slicing was redundant. Tests now simulate the API honoring
`count` instead of relying on the slice. Addresses PR review nit.
* style(tests): apply ruff format to test_doctor.py
Collapse multiline write_text calls onto single lines to satisfy the
CI ruff formatter (lint-backend was failing on format --check).
* fix(channels): surface WeCom WebSocket connection failures (#2000)
The WeCom channel started the SDK connection with a fire-and-forget
asyncio task and never inspected its result, so connection failures
(e.g. the gateway WebSocket handshake to wss://openws.work.weixin.qq.com
failing) were silently swallowed: the channel still logged "started",
SDK error/disconnected events went unobserved, and the connect task
produced "Task exception was never retrieved" noise.
Monitor the connect task with a done-callback that logs a clear,
actionable error (and stays silent on cancellation), and subscribe to
the SDK's error/disconnected events so failures become visible in
DeerFlow's own logs.
* style(channels): apply ruff format to wecom.py
Collapse the multiline log message onto a single line to satisfy the
CI ruff formatter (lint-backend was failing on format --check).
* fix(channels): log WeCom disconnect reason when SDK provides one
Address review feedback: _on_ws_disconnected now includes the first
event arg (e.g. reason/context) in the warning instead of discarding
it, so disconnect causes are visible in logs.
ViewImageMiddleware persists full base64 image payloads in hide_from_ui
human messages inside checkpoints. All REST endpoints that returned
serialize_channel_values(channel_values) sent these multi-megabyte
payloads to the frontend, freezing the UI on threads with images.
Add strip_data_url_image_blocks() to remove data:-scheme image_url
content blocks from hide_from_ui messages, and
serialize_channel_values_for_api() as a convenience wrapper used by all
six affected call sites across threads, runs, and thread_runs routers.
SSE streaming is unaffected (still uses serialize_channel_values).
Fixes#3496
* docs(spec): telegram streaming output design
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(plan): telegram streaming implementation plan
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(telegram): report streaming support for telegram channel
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(channels): use slack as the non-streaming sample channel in manager tests
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(telegram): register running-reply placeholder as stream target
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(telegram): pin last_edit_at sentinel in placeholder registration test
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(telegram): extract _send_new_message from send()
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(telegram): edit streamed message in place for non-final updates
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(telegram): finalize streamed message with overflow splitting
When is_final=True arrives and stream state exists, pop the state, edit
the streamed placeholder with the final text, split overflow into follow-up
send_message calls, update _last_bot_message, and clear stream state.
Falls back to _send_new_message when no stream state is registered.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(telegram): exercise the not-modified handler in final edit path
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: telegram channel now streams replies via message editing
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(telegram): harden final-delivery path with guarded retry and chunk retries
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(channels): accept runtime 'messages' SSE event for streaming text accumulation
The embedded runtime (matching LangGraph Platform semantics) emits SSE
event name 'messages' for the requested 'messages-tuple' stream mode,
so the manager never accumulated token deltas and streaming channels
only updated from end-of-step 'values' snapshots — on Telegram this
looked like 'Working on it...' followed by the full answer in one block.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(telegram): widen stream-edit throttle to 3s in group chats
Telegram caps bots at 20 messages/minute per group, stricter than the
1 msg/s per-chat guideline. Groups have negative chat ids, so pick the
interval by sign.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(telegram): address review findings — thread fallback messages, bound stream registry, share stream-event constants
- Fallback/new stream messages now carry reply_to_message_id parsed from
thread_ts so they stay nested under the user's message (finding 1)
- STREAM_MODES / MESSAGE_STREAM_EVENTS constants link the requested
stream modes to the SSE event names they arrive under (finding 2)
- _register_stream_message bounds the in-flight registry at 256 entries,
evicting oldest, guarding against leaks when a final never arrives (finding 4)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
_ingest_inbound_files ensured the thread uploads dir (mkdir), enumerated it
(iterdir/is_file) to de-duplicate names, and wrote each downloaded attachment
(write_upload_file_no_symlink) directly on the event loop. Offload the directory
prep and every per-file write via asyncio.to_thread; the genuinely async network
read (file_reader) stays on the loop. Externally observable behavior is unchanged.
Found via `make detect-blocking-io` (HIGH: iterdir on an async path).
Add tests/blocking_io/test_channels_ingest.py anchor, verified red->green under
the strict Blockbuster gate.
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
list_messages re-scanned every event in the thread on each call (category
filter + seq filter) — O(total events) per paginated request on the default
run-events backend. Maintain a messages-only, seq-sorted projection of _events
(shared dict refs, no copies) and locate the seq window with bisect:
list_messages drops to O(log m + page) and count_messages to O(1). The index is
kept in lockstep at every mutation site (put / put_batch via _put_one,
delete_by_run, delete_by_thread).
Externally observable behavior is unchanged — the full RunEventStore contract
suite passes across memory/db/jsonl.
Add a test covering pagination over non-contiguous message seqs (messages
interleaved with trace events), including in-gap and exact-boundary cursors.
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(blocking-io): fail-loud repo-root resolution and shared detector CLI shim
The three detectors resolved REPO_ROOT with depth-indexed
Path(__file__).resolve().parents[4]. If a detector file ever moves to a
different directory depth, scan roots resolve under the wrong directory
and the detector reports zero findings with no error — a silent-zero
failure shape for a detection tool.
- Add support/detectors/repo_root.py: resolve the repo root by walking
upward to the .git marker (checked with exists() so git worktrees,
where .git is a file, also resolve), raising RuntimeError when no
marker is found. All three detectors use it at import time, so a
relocated detector fails loudly instead of scanning an empty tree.
- Extract scripts/_detector_cli.py from the three character-identical
CLI shims; the sys.path computation lives in one place and raises
when backend/tests cannot be found.
- tests/test_detector_repo_root.py pins: resolution from an unmarked
location raises instead of returning an empty scan; all three
detectors share the resolved root; each CLI shim delegates to its
detector.
Testing: backend `make test` (4278 passed); smoke-ran
`make detect-blocking-io`, `make detect-thread-boundaries`, and
`scripts/scan_changed_blocking_io.py --base upstream/main`.
Closes#3510 (review follow-up to #3503).
* chore(blocking-io): declare detector modules import-only, drop script-mode residue
Adversarial review caught that blocking_io_static.py and
thread_boundaries.py kept shebangs and __main__ blocks but can no longer
run as plain scripts: the new `from support.detectors.repo_root import`
executes before anything puts backend/tests on sys.path, so direct
invocation dies with ModuleNotFoundError before argparse.
Direct execution was never a documented entry point (Makefile targets,
the scripts/ shims, the blocking-io-guard skill, and tests all go
through the support.detectors package), so converge on import-only
instead of re-adding per-module bootstrap: drop the shebangs and the now
unreachable __main__ blocks (plus the `import sys` they kept alive) and
state the supported entry points in each module docstring. The shim
delegation tests in test_detector_repo_root.py pin the supported CLI
paths.
Testing: backend `make test` (4278 passed); `make detect-blocking-io`
and `make detect-thread-boundaries` smoke-ran.
* perf(runtime): index runs by thread_id to avoid O(n) scans in RunManager
RunManager.list_by_thread, create_or_reject (inflight check), and has_inflight each filtered every in-memory run by thread_id — an O(total in-memory runs) scan that grows with overall gateway traffic rather than the queried thread's depth.
Add a thread_id -> run_ids secondary index (an insertion-ordered dict used as an ordered set) maintained in lockstep with _runs under the existing lock at every add/remove site (create, create_or_reject, both rollbacks, cleanup). The three per-thread queries now run in O(runs-in-thread); insertion order is preserved so list_by_thread keeps stable tie-breaking. Behavior unchanged. Adds 6 regression tests; full RunManager suite 146 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(runtime): cover create_or_reject rollback + clarify thread-index guard docstrings
Address review on #3499 (fancyboi999):
- Reword _thread_records_locked docstring: lockstep under self._lock is the
correctness guarantee; self._runs.get is one-directional defense-in-depth
(drops stale ids, cannot recover index-missing ids), not reconciliation.
- Add test_failed_create_or_reject_unindexes_run covering the create_or_reject
rollback/unindex mutation site (the last untested mutation path).
- Fix _FailingPutRunStore docstring ("initial put" -> "every put").
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add user-owned IM channel connections
* Fix dev startup and channel connect popup
* Use async channel connect flow
* Harden dev service daemon startup
* Support local IM channel connections
* Align IM connections with local channels
* Fix safe user id digest algorithm
* Address Copilot IM channel feedback
* Address IM channel review comments
* Support all integrated IM channel connections
* Format additional channel connection tests
* Keep unavailable channel connect buttons clickable
* Fix IM channel provider icons
* Add runtime setup for enabled IM channels
* Guard global shortcut key handling
* Keep configured IM channels editable
* Avoid password autofill for channel secrets
* Make channel threads visible to connection owners
* Persist IM runtime config locally
* Allow disconnecting runtime IM channels
* Route no-auth channel sessions to local user
* Use default user for auth-disabled local mode
* Show IM channel source on threads
* Prefill IM channel runtime config
* Reflect IM channel runtime health
* Ignore Feishu message read events
* Ignore Feishu non-content message events
* Let setup wizard enable IM channels
* Fix frontend formatting after merge
* Stabilize backend tests without local config
* Isolate channel runtime config tests
* Address channel connection review comments
* Use sha256 user buckets with legacy migration
* Ensure runtime IM channels are ready after restart
* Persist disconnected IM channel state
* Address channel connection review comments
* Address channel connection review findings
Frontend connect flow:
- Open the runtime-config dialog only when a provider still needs
credentials; configured providers go straight to the connect flow, so
the binding-code/deep-link path is reachable from the UI again.
- After saving credentials, continue into the connect flow when a user
binding is still required (multi-user mode) instead of stopping at a
"Connected" toast.
- Extract shared provider-state helpers to core/channels/provider-state
and add unit + e2e coverage for the direct-connect and
configure-then-connect paths.
Provider status semantics:
- Report connection_status from the user's newest connection row;
with no binding it is not_connected, except in auth-disabled local
mode where a configured running channel is effectively connected.
Concurrency and event-loop correctness:
- Offload ChannelRuntimeConfigStore construction and writes, channel
service construction, and Slack connection replies to threads; add a
tests/blocking_io/ anchor for the runtime-config handlers.
- Consume binding codes with a conditional UPDATE so a code can only be
used once under concurrent workers; retry upsert_connection as an
update when a concurrent insert wins the unique constraint.
- Serialize ensure_channel_ready per channel so concurrent provider
polls cannot double-start a channel worker.
Config and migration hardening:
- Stop mutating the get_app_config()-cached Telegram provider config;
the runtime store now owns the UI-entered bot username.
- Register channel_connections in STARTUP_ONLY_FIELDS with the
standardized startup-only Field description.
- Match the legacy unsafe-id bucket by recomputing its exact SHA-1 name
so another user's same-prefix bucket can never be migrated.
- Remove the unused Telegram process_webhook_update path and document
src/core/channels in the frontend docs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address PR review comments on authz scoping and channel runtime
Security (review feedback from ShenAC-SAC):
- Scope internal-token callers to the connection owner carried in
X-DeerFlow-Owner-User-Id instead of bypassing owner checks outright,
in both require_permission(owner_check=True) and the stateless run
endpoints. Internal callers keep access to their own and
shared/legacy threads, and may claim a default-owned channel thread
for its real owner, but a leaked internal token no longer grants
cross-user thread access.
- Require admin privileges for POST/DELETE /api/channels/{provider}/
runtime-config: runtime credentials and channel workers are
instance-wide shared state (same model as the MCP config API).
Read-only provider listing stays available to all users.
Performance (review feedback from willem-bd):
- Skip the redundant thread channel-metadata PATCH after the first
successful backfill per thread.
- Reuse the per-connection Slack WebClient until its token changes
instead of constructing one per outbound message.
- Reconcile channel readiness for all providers concurrently in
GET /api/channels/providers.
Also resolve the code-quality unused-import flag in the blocking-io
anchor by pre-importing the channel service via importlib.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix prettier formatting in provider-state test
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Reconcile UI runtime channel config with config reload on restart
Main now reloads a channel's config.yaml entry on restart_channel()
(#3514, issue #3497). Adapt the user-owned connection flow to coexist:
- configure_channel() restarts with reload_config=False — the caller
just supplied the authoritative config (browser-entered credentials
that are never written to config.yaml), so a file reload must not
clobber it with the stale on-disk entry.
- _load_channel_config() re-applies the UI runtime-store overlay used
at startup, so an operator-triggered restart keeps browser-entered
credentials for channels without a config.yaml entry and does not
resurrect a channel disconnected from the UI.
- Offload the reload's disk IO (config.yaml + runtime store) with
asyncio.to_thread, matching the blocking-IO policy on this branch.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(skills): keep skill archive installation off the event loop
ainstall_skill_from_archive — the async entry point awaited by the gateway
POST /skills/install route — ran its entire filesystem pipeline inline on
the event loop: zip extraction, frontmatter validation, rglob enumeration,
per-file read_text, shutil.copytree staging, and tempdir cleanup.
Restructure into offloaded phases: prepare (extract + validate) and commit
(stage + move) run via asyncio.to_thread, the tempdir lifecycle is
offloaded, and the security scanner's file enumeration and reads move off
the loop — only the per-file LLM scan (genuinely async) stays awaited.
Security decision logic and exception contract are unchanged.
Anchor: tests/blocking_io/test_skills_install.py drives the real install
pipeline (real .skill archive, real FS; only scan_skill_content stubbed)
under the strict Blockbuster gate. Verified red on pre-fix code
(BlockingError: os.stat), green with the fix.
* fix(skills): log temp-dir cleanup failures instead of swallowing them
Review follow-up on the install offload: rmtree(ignore_errors=True) kept
the primary install exception but silently leaked the extraction dir on
cleanup failure. Keep the never-mask behaviour, add a warning log.
* fix(skills): bound install tmp cleanup and pass skill_dir explicitly (review)
- Wrap the best-effort temp-dir cleanup in asyncio.wait_for (5s) so a
hung filesystem in the finally block cannot stall or mask the install
outcome; timeout is logged like the existing OSError path.
- Hoist _collect_scannable_files to module level with skill_dir as an
explicit argument instead of a closure capture.
* feat(blocking-io): add changed-lines blocking-IO scanner (L1)
* feat(blocking-io): add scan-changed CLI wrapper
* feat(skill): add blocking-io-guard developer SOP skill
* docs(blocking-io): point contributors at the blocking-io-guard skill
* style(blocking-io): apply ruff format to scanner and tests
* docs(backend): document changed-lines blocking-IO scanner in CLAUDE.md
* feat(skill): add post-fix re-scan check and PR batching policy
* refactor(skill): fix SOP step ordering, align template with repo conventions
- Move re-scan into an explicit 'apply the fix' step (was wedged after
anchor generation while telling you to go back before the anchor)
- Renumber steps 0-6; drop undefined 'L1' jargon
- Mode A: document that the diff is <base>...HEAD (commit first)
- Mode B: prefer make detect-blocking-io + findings JSON file
- anchor template: module-level pytestmark per tests/blocking_io convention
- CLAUDE.md: fix 'git diff --base' phrasing
* fix(skill): catch findings introduced without touching the blocking line
Review follow-up: changed-line intersection alone misses the case where a
new async caller exposes an old sync helper — the static finding sits on
the untouched blocking line, so Mode A returned empty and the SOP stopped
on a false 'no blocking-IO surface'.
Selection is now a union over the changed files:
- findings on added lines of git diff <base>...HEAD (kept: a second
identical symbol in an already-flagged function collides on the stable
key and only this selection sees it);
- findings new versus the merge base, matched by (path, function,
symbol) — never line numbers.
Base sources are materialized via git show <merge-base>:<path>; files
absent at base count every head finding as new. SKILL.md now states the
residual same-file-only blind spot (cross-file async callers) instead of
treating an empty list as proof of zero exposure, and only requires
reading sop-skeleton.md when generalizing to another detector domain.
* docs(skill): examples teach test-writing, the teeth check defines the rule
All examples in the references/template are filesystem-flavored; make
explicit that they are instances, not the SOP's boundary — the same rules
apply to every detector category (FILE_IO, HTTP, SUBPROCESS, SLEEP) and
acceptance is always red/green teeth, never similarity to an example.
Neutralize the template's arrange comment accordingly.
* fix(blocking-io): harden changed-lines scanner per review
- Dedup the union selection by the stable key (path, function, symbol)
instead of dict identity, so a future selector returning copied dicts
cannot silently empty the result.
- parse_changed_lines now handles any unified diff: context lines advance
the new-file counter, \-markers and deletions do not, and the counter
resets at each +++ header. Previously correct only for --unified=0.
- Add blocking_io_static.scan_source (in-memory scan); base-version
comparison no longer round-trips through temp files.
- Empty Mode A report now prints the same-file-only reachability caveat
at the point of use instead of relying on the SOP text alone.
* docs(skill): bound best-effort cleanup when the offload sits in finally
Lesson from the #3505 review: the SOP routinely drives 'offload the
cleanup branch' transformations, and an awaited cleanup in finally can
mask or stall the primary exception. One sentence in Step 2 closes that
gap at the point where the fix is written.
* feat(community): add SearXNG and Browserless web search/fetch tools
- SearXNG web_search: privacy-focused meta search engine integration
with configurable base_url via config.yaml tool settings
- Browserless web_fetch: headless browser page fetching with
readability article extraction
- Both tools are fully configurable through tool config section
- No external API keys required for basic operation
* fix: address PR review feedback and add unit tests
- Guard config.model_extra against None values (review #1, #2)
- Coerce max_results to int when reading from config (review #2)
- Fix web_fetch_tool to use direct HTTP fetch instead of reusing
the web_search client config (review #3)
- Fix misleading docstring for SearxngClient.fetch (review #4)
- Remove unused target_url variable to pass Ruff lint (review #5)
- Normalize bool config values with _normalize_bool helper to
handle env-resolved string values correctly (review #6)
- Add unit tests for both SearXNG and Browserless client classes
and their tool functions with mocked httpx (review #7, #8)
* fix: convert to async httpx to avoid blocking I/O on event loop
- Replace httpx.Client with httpx.AsyncClient in both client classes
- Convert tool functions to async def
- Wrap readability_extractor calls in asyncio.to_thread()
- Update all tests to use pytest.mark.asyncio and async mocks
- Fix import sorting to pass Ruff lint
* fix(browserless): replace deprecated waitUntil with waitForEvent
The Browserless API has deprecated the waitUntil parameter.
Replace with waitForEvent which accepts values like 'networkidle'.
Default is empty (no wait), configurable via config.yaml.
* fix(browserless): remove deprecated gotoTimeout and bestAttempt params
The Browserless /content API does not accept gotoTimeout or bestAttempt
as top-level payload keys. These were being sent unconditionally,
causing 400 Bad Request errors on current Browserless versions.
Changes:
- Remove goto_timeout_ms parameter and 'gotoTimeout' from payload
- Remove best_attempt parameter and 'bestAttempt' from payload
- Remove _normalize_bool helper (no longer needed)
- Remove goto_timeout_ms and best_attempt config reading in tools.py
- Add tests for waitForSelector and reject params
- Verify no deprecated params are sent in test_fetch_html_success
* refactor(searxng): remove web_fetch_tool, decouple from web_search config
SearXNG is a search engine — it should only provide web_search_tool.
The web_fetch responsibility belongs to Browserless (headless Chrome)
or Jina AI, not SearXNG.
Changes:
- Remove web_fetch_tool from SearXNG tools.py and __init__.py
- Remove SearxngClient.fetch() method (no longer needed)
- Remove unused asyncio/readability imports from SearXNG tools.py
- Add test for max_results string-to-int coercion from config
- Add test for search with categories parameter
- Add test for httpx.RequestError handling
- Apply ruff format fixes to browserless_client.py and test files
* Fix stale AIO sandbox cache reuse
* Address AIO sandbox review feedback
* Distinguish sandbox health check failures
* Keep local discovery recoverable when the runtime check fails
LocalContainerBackend.discover() shares _is_container_running, which now
raises on transient daemon errors instead of returning False. Discovery has
no exception handling in _discover_or_create_with_lock(_async), so a brief
Docker hiccup turned a recoverable "could not verify, create instead" into a
hard acquire failure. Catch the check failure inside discover() and return
None so an unverifiable container is simply not adopted, restoring the
pre-change fall-through while keeping raise-on-unknown semantics protecting
the destroy path.
Reported by fancy-agent on PR #3494.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Narrow the not-found match in container inspect error handling
A bare "not found" substring also matches transient failures like "command
not found" or "context not found", which would misclassify a check error as
"container definitely gone" and bypass the raise-on-unknown contract. Keep
Docker's specific "No such object"/"No such container" phrases, and only
trust a generic "not found" (Apple Container) when the message names the
inspected container or refers to a container/object.
Reported by WillemJiang on PR #3494.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(sandbox): persist lazily-acquired sandbox state via Command
ensure_sandbox_initialized mutates runtime.state in place, which is local
to the current tool invocation and is not picked up by LangGraph's channel
reducer. Subsequent graph steps and downstream consumers (such as
ToolOutputBudgetMiddleware and the sub-agent task_tool) therefore cannot
observe the sandbox id from state.
Wrap tool calls in SandboxMiddleware (wrap_tool_call / awrap_tool_call) to
detect fresh lazy initialization by diffing runtime.state before and after
the handler, and emit a proper state update via Command(update=...):
- ToolMessage results are wrapped into Command(update={sandbox, messages})
- Command results with a dict update are merged on the sandbox key while
preserving messages / goto / graph / resume
- Command results with non-dict updates are left untouched to avoid silent
data loss on unknown update shapes
Tests:
- 7 new unit tests cover lazy-init emit, passthrough, dict-update merge,
non-dict-update passthrough (sync and async)
- Refresh replay golden write_read_file.ultra.events.json: SSE 'values'
events now correctly carry the 'sandbox' key in their keys list, which
is the direct evidence that the fix is effective
Closes#3463
* refactor(sandbox): use dataclasses.replace to preserve Command fields
Address Copilot review on #3464: replace manual field-copy with
dataclasses.replace so any current or future Command fields are
preserved automatically when merging sandbox_update.
Also add a regression test that constructs a Command with non-None
graph/goto/resume to lock this behavior in.
When memory is enabled, the first conversation with a legacy shared agent
creates a per-user agent directory containing only memory.json (no
config.yaml). On the second turn, resolve_agent_dir() returned this
incomplete directory, causing load_agent_config() to fail with
"Agent config not found".
Require config.yaml to exist alongside the directory for both the
per-user and legacy paths, so that memory-only directories fall
through correctly. This aligns resolve_agent_dir with the existing
config.yaml check in list_custom_agents.
Refs: https://github.com/bytedance/deer-flow/issues/3390
* feat(memory): add memory.token_counting config to avoid tiktoken network dependency (#3429)
Add a `memory.token_counting` option (`tiktoken` | `char`) so deployments in
network-restricted environments can opt out of tiktoken entirely. In `char`
mode the memory-injection budget uses a network-free character-based estimate
and never triggers the BPE download from openaipublic.blob.core.windows.net,
which could otherwise block for tens of minutes (see #3402).
Also harden the default `tiktoken` path:
- cache an in-flight LOADING sentinel so concurrent callers fall back
immediately instead of spawning more blocking get_encoding threads when the
first load is still running (e.g. under the 5s startup warm-up timeout);
- cache failures with a timestamp and retry after a cooldown so a transient
network outage self-heals back to accurate counting without a restart;
- skip startup warm-up entirely in char mode.
The new config is surfaced via the memory config API and config.example.yaml
(config_version bumped). Default remains `tiktoken`, so existing deployments
are unaffected.
* fix(memory): use CJK-aware char token estimate and address review feedback
- Replace the flat len(text)//4 fallback with a CJK-aware estimate so
Chinese/Japanese/Korean memory content does not over-fill the injection budget
- Document the internal tiktoken retry cooldown and char-mode escape hatch
- Sync CLAUDE.md / config.example.yaml / MEMORY_IMPROVEMENTS.md wording
- Fix MemoryConfigResponse mocks/assertions and add CJK estimate tests
POST /api/runs/stream and /api/runs/wait accept thread_id in the request
body but performed no owner authorization, letting any authenticated user
start runs on -- and read /wait checkpoint channel_values from -- another
user's thread (cross-user IDOR, #3472).
The @require_permission(owner_check=True) decorator resolves ownership from
the thread_id *path* param, so it cannot cover these body-param endpoints.
Enforce ownership inside start_run() before create_or_reject via
ThreadMetaStore.check_access: missing rows (auto-created temp threads) and
NULL-owner rows stay accessible, while a thread owned by another user
returns 404 (matching thread_runs.py). The internal system role (IM
channels acting for platform users) is exempt.
Closes#3472
The default `make up` started the Gateway with `--workers 4`, but run state
(RunManager and the stream bridge) is held in-process and nginx uses no sticky
sessions. With the default config, same-run requests scatter across workers that
each keep their own run state, breaking run cancellation (409), SSE reconnect
(hangs on heartbeats), multitask de-duplication, and IM channels (duplicate
replies). The shared cross-worker stream bridge does not exist yet.
Default GATEWAY_WORKERS to 1 so the out-of-the-box deployment is correct,
document the single-worker boundary in the README, and add a regression test
pinning the default while keeping it overridable. This is a stop-gap, not a
multi-worker implementation; the full fix (shared run state + stream bridge) is
tracked in #3191.
Refs #3239, #3260
In Docker production deployments, LocalSandboxProvider runs inside the
deer-flow-gateway container, so any `sandbox.mounts[].host_path` from
config.yaml is resolved against the gateway container's filesystem — not
the host machine. When the path isn't also bind-mounted into the gateway
service, the mount was silently dropped with only a WARNING log, leaving
agents reading an empty directory in production while the same config
worked under `make dev`.
Escalate the missing-host_path branch to logger.error with explicit
guidance about Docker bind mounts and docker-compose, so the failure is
hard to miss in default log configurations. Skip behaviour is preserved
to avoid breaking existing deployments.
Also clarify the misleading `VolumeMountConfig.host_path` field
description so it documents reality for both providers:
- LocalSandboxProvider checks host_path from inside the gateway process
(host in `make dev`, container in `make up`).
- AioSandboxProvider (DooD) passes host_path straight to `docker -v`
for the sandbox container, where the host Docker daemon resolves it
from the host machine's perspective.
config.example.yaml's `sandbox.mounts` comment gets a Note: block
pointing operators at the docker-compose bind-mount requirement so the
Docker-mode gotcha is discoverable from the canonical template.
Adds a regression test that:
- confirms missing host_path is still skipped (no behaviour break);
- asserts an ERROR record is emitted referencing the offending paths;
- asserts the message contains actionable Docker/gateway/docker-compose
keywords so future refactors can't quietly downgrade it.
Refs: https://github.com/bytedance/deer-flow/issues/3244
* fix(agents): offload blocking filesystem IO in delete_agent off the event loop
delete_agent is an async route handler but resolved the agent directory (Paths.base_dir -> Path.resolve), probed it (Path.exists), and removed it (shutil.rmtree) directly on the event loop, blocking it for the duration of every delete. Surfaced by 'make detect-blocking-io'.
Move the resolve/exists/rmtree sequence into a sync helper run via asyncio.to_thread, mapping its outcome back to the existing 404/409/500 responses (behavior unchanged). Adds a tests/blocking_io/ regression anchor under the strict Blockbuster gate, mirroring test_skills_load.py (#1917).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(agents): offload blocking filesystem IO in create_agent_endpoint too
Like delete_agent, the async create_agent_endpoint resolved and created the agent directory and wrote config.yaml + SOUL.md (with rmtree cleanup on failure) directly on the event loop. Move the whole create-or-409 sequence into a sync helper run via asyncio.to_thread; behavior is unchanged (201 / 409 / 500). Extends the blocking_io regression anchor to cover create as well as delete and renames it to test_agents_router.py.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Apply suggestions from code review
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* add caller identity in replay e2e
* make format
* fix(replay-e2e): stabilize title caller replay
* fix(replay-e2e): use captured caller without run manager
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Add PatchedChatStepFun adapter for StepFun reasoning models (step-3.7-flash,
step-3.5-flash). Captures reasoning from both streaming and non-streaming
responses and replays it on historical assistant messages for multi-turn
tool-call conversations.
- New: PatchedChatStepFun adapter with streaming/non-streaming reasoning capture
- Support both reasoning and reasoning_content field names
- 17 unit tests covering all response paths
- Updated: config.example.yaml with StepFun configuration example
Copying config.example.yaml to config.yaml and starting DeerFlow crashed with `pydantic ValidationError: models — Input should be a valid list [input_value=None]`, because the example ships every entry under `models:` commented out, so PyYAML parses the key as null. Reported in #1444.
Add a field_validator(mode="before") on AppConfig that coerces null models/tools/tool_groups to [] (matching their default_factory=list), and emit an actionable warning from from_file when no models are configured (pointing to config.example.yaml / make setup). Adds regression tests.
Closes#1444
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Nan Gao