The GraphRecursionError except-block in SubagentExecutor._aexecute derives
usable_partial from the last AIMessage's raw non-empty text, without
checking _extract_llm_error_fallback (#4042) first. A handled provider
failure (LLMErrorHandlingMiddleware's deerflow_error_fallback marker)
always carries non-empty user-facing text, so when it lands on the same
turn that trips max_turns, it is indistinguishable from genuine partial
output and gets misclassified as a completed task instead of the failed
provider error it is.
Consult _extract_llm_error_fallback in this except-block too, same as the
normal-completion path above it, and classify FAILED with
stop_reason=turn_capped when it detects the marker.
* fix(sandbox): use os.sep in reverse-resolve containment check on Windows
Path.resolve() always renders with the native separator (backslash on
Windows), but _reverse_resolve_path's containment check hardcoded a
"/" suffix when testing whether a resolved path is nested under a
mapping's local root. Only the exact-root case (no separator needed)
ever matched; every nested path fell through to the "no mapping
found" branch and returned the raw host path -- leaking the real
username and full directory tree into list_dir/glob/grep results and
bash output masking instead of the virtual /mnt/user-data/... path.
_is_read_only_path already does the equivalent check correctly via
os.sep, so this aligns _reverse_resolve_path with that pattern: the
containment check now compares with os.sep, and the extracted
relative portion is normalized to forward slashes before being
spliced into the (always POSIX-style) container path.
Also fixes a same-file cosmetic bug in list_dir's virtual
sub-directory overlay: it compared a bare child name (e.g.
"workspace") against a set of full container paths, so the
already-listed guard never matched and a mount whose subdirectory the
underlying scan already found (the common case for
/mnt/user-data/workspace, uploads, outputs) was appended a second
time.
Continues the same separator-bug class already fixed in this file by
#3869 (forward-direction command resolution) and #4035 (reverse
regex-boundary matching); neither touched this containment check.
* test(sandbox): add host-OS-independent regression test for the os.sep containment fix
_reverse_resolve_path's os.sep containment check (and the paired
lstrip(os.sep).replace(os.sep, "/") extraction) has no test that would
fail if reverted: backend CI runs only on ubuntu-latest, where
os.sep == "/" makes the pre-fix hardcoded "/" and the current os.sep
form observationally identical, so a plain POSIX-path test can't
discriminate between them.
Add a test that forces the Windows code path independent of host OS by
monkeypatching os.sep to "\" and stubbing both the module's Path name
and the sandbox's cached _resolved_local_paths to return
backslash-joined strings, mirroring what real WindowsPath.resolve()
produces -- without touching the filesystem or requiring an actual
Windows host. Verified this fails with the raw host path leaking
through when the os.sep fix is reverted to the hardcoded "/" form, and
passes with the fix in place.
replace_virtual_paths_in_command matches the virtual root with no
segment-boundary lookahead:
re.compile(rf"{re.escape(VIRTUAL_PATH_PREFIX)}(/[^\s\"';&|<>()]*)?")
The trailing group needs a "/" to consume anything, so when the character after
/mnt/user-data is "-", ".", "_", a digit or a letter, the group matches empty
and the bare root still matches. The substitution then rewrites it to the
thread's host user-data directory and the rest of the sibling name rides along,
so a command naming a prefix sibling of the mount root is pointed at a real host
directory outside the mount contract:
cat /mnt/user-data-backup/secret.txt -> cat <host>/user-data-backup/secret.txt
which reads the host file. This is the same defect as #4035 (reverse patterns)
and #4053 (masking patterns), mirrored into the virtual->host direction; it is
the last unguarded member of that family.
The boundary class mirrors LocalSandbox._content_pattern's rather than
_command_pattern's: a virtual root can legitimately be followed by ":"
(PATH-style concatenation) or ",", which the shell-oriented class rejects, so
narrowing to it would stop translating paths that translate today. "$" covers a
command ending exactly at the root.
* fix(goal): prevent continuation_count regression from racing continuations
* fix(goal): prevent continuation_count regression from racing continuations
* fix(memory): coerce null confidence when ranking stale facts
_build_staleness_section and the per-cycle removal cap in _apply_updates
used fact.get("confidence", 0.0/0), which only defaults when the key is
absent. A fact whose confidence is explicitly null (or otherwise malformed)
returned None, breaking the numeric sort/format. Use
_coerce_source_confidence, which normalizes null/malformed values and clamps
to [0, 1], so null-confidence facts no longer block staleness handling.
* ci: retrigger cancelled CI workflow
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
The parse gate required all of {user, history, newFacts, factsToRemove} to
be present before accepting the model's JSON. A well-formed update that
simply has no facts to remove (the common case) omits the empty
factsToRemove key and was silently rejected. Drop factsToRemove from the
required set so those updates parse and apply.
LoopDetectionMiddleware._get_run_id used a truthiness check that
collapsed a present-but-None run_id to the same "default" key as a
totally absent one. SubagentExecutor sets context["run_id"] =
self.run_id unconditionally, so run_id is genuinely None for an
embedded/TUI-dispatched subagent, and later reads the stop reason back
with that same raw attribute via consume_stop_reason(self.run_id). The
write (keyed "default") and the read (keyed None) disagreed, so a
genuine loop-detection hard-stop's loop_capped reason was silently
dropped instead of reaching the lead.
Align _get_run_id with TokenBudgetMiddleware's key-presence-based
version, which does not have this bug: return the context value as-is
when the key is present (None included), and fall back to a
per-runtime-unique key only when the key is absent.
str_replace returned "OK" whenever the target file was empty, silently
reporting success even when the model asked to replace a non-empty string
that could not possibly be present. Only short-circuit to "OK" when old_str
is also empty; otherwise return the standard not-found error.
search_memory_facts sorted matches by fact.get("confidence", 0), which
returns None for a fact whose confidence key is explicitly null, crashing
the sort comparison. Use _coerce_source_confidence so null/malformed
confidence values are normalized and clamped before ranking.
* feat(frontend): show real project version on About page
The About page heading was hardcoded "About DeerFlow 2.0" while the real
version is 2.1.0, kept in lockstep across backend/pyproject.toml,
frontend/package.json, and Chart.yaml by scripts/verify_versions.sh.
With the nightly workflow shipping unreleased main daily, a hardcoded
version drifts further from reality every day.
Wire the heading to the real version, nightly-aware:
- src/version.ts reads NEXT_PUBLIC_APP_VERSION (nightly override) with a
package.json fallback for release/local builds.
- about-content.ts interpolates APP_VERSION into the heading; the
"DeerFlow 1.0 and 2.0" milestone copy in acknowledgments is left as-is.
- Dockerfile adds ARG APP_VERSION / ENV NEXT_PUBLIC_APP_VERSION in the
builder stage so nightly CI can stamp the version.
- nightly.yaml computes the nightly string once in prepare
(<base>-nightly.<date>-<sha>, mirroring the chart scheme) and feeds it
to both the frontend image build-arg and the chart publish (refactored
to consume the shared output, so the displayed version and chart
version can't drift). Release builds (container.yaml) are untouched
and fall back to package.json.
Tests cover the version branches and the heading interpolation;
RELEASING.md notes the About page as a derived version consumer.
* test(frontend): assert positive About-page version value
Address review feedback on #4126: the unset-env assertion was negative-only
(checked the stale "2.0" literal was gone), so it would still pass if
APP_VERSION interpolated to "" or undefined. Mirror version.test.ts by
asserting the heading carries the real resolved APP_VERSION.
* ci(nightly): fail loudly on empty base version
Address review feedback on #4126: the prepare step's BASE (parsed from
Chart.yaml) had no empty-guard. A malformed/missing version would yield
`-nightly.<date>-<sha>` (invalid semver) that now flows into both the
chart publish and the frontend About-page version. Add `set -euo pipefail`
plus a `test -n "$BASE"` guard mirroring publish-chart. `pipefail` matters
- without it the pipeline's exit code is awk's, so `set -e` alone wouldn't
catch a grep miss.
When _process_queue found another worker already processing, it called
_schedule_timer(0), spawning a fresh Timer thread immediately and looping
tightly (spawn -> busy -> reschedule -> spawn) until the active worker
finished. Replace this with a _reprocess_pending flag: a concurrent caller
sets the flag and returns, and the active worker reschedules exactly once
in its finally block when work remains. Reset the flag in clear().
github.bot_login and trigger.mention_login were read raw in the
require_mention precedence chain, so a whitespace-only value (e.g. " ")
never fell through to the working fallback the chain documents -
Python truthiness lets " " win an `or` chain over agent.name or the
operator default. The third link (channels.github.default_mention_login)
was already correctly normalized and pinned by
test_operator_default_blank_string_treated_as_none; this closes the gap
for the other two by normalizing GitHubTriggerConfig.mention_login and
GitHubAgentConfig.bot_login once, at the config layer, via a
field_validator mirroring the pattern already used elsewhere in the
config package.
* fix(run-events): serialize seq assignment with a per-thread asyncio lock
put() and put_batch() read max(seq) and then INSERT seq+1 in separate awaits.
Two coroutines writing the same thread in one process could interleave
between the read and the insert and assign the same seq, colliding on
SQLite where the DB-level FOR UPDATE lock is weaker than Postgres. Add a
per-thread asyncio.Lock (_write_locks / _get_write_lock) around the
read-assign-insert critical section in both methods.
* address review: evict orphaned per-thread write-lock in delete_by_thread
_write_locks accumulated one asyncio.Lock per thread ever seen and never
released, leaking in the long-lived DbRunEventStore singleton. Evict the
entry after delete_by_thread when no writer holds it (lock recreated
lazily on the next write). Per @willem-bd review on #4077.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
The dual-mode _merge_stream_text used short-circuits chunk==existing
and existing.endswith(chunk) that silently dropped legitimate delta
chunks in messages-tuple mode:
- CJK reduplication: '谢谢' tokenized as ['谢','谢'] -> buffer stays '谢'
- Repeated tokens: 'gogo' as ['go','go'] -> buffer stays 'go'
- Suffix-matching tails: 'hello' as ['hel','l','o'] -> drops 'l'
Delta payloads must always append; only strictly-longer cumulative
snapshots should replace. The fix gates the cumulative-replace branch
with len(chunk) > len(existing) and removes the dropped-shape guards.
Both copies are fixed:
- app/channels/manager.py (IM streaming to Feishu/Telegram/etc.)
- deerflow/tui/view_state.py (TUI client rendering)
The TUI reduce() caller now handles exact re-sends (values snapshot
re-emitting history) before the merge, and the len>1 guard prevents
single-char CJK deltas from being mistaken for re-sends.
Co-authored-by: Claude <noreply@anthropic.com>
get_initial_oauth_headers() now wraps each server's token fetch in
try/except so one broken OAuth server does not abort the entire
multi-server MCP tool load (which returned [] from the outer except,
dropping every server's tools).
_fetch_token() now captures a rotated refresh_token from the token
response and updates the in-memory McpOAuthConfig so subsequent
refreshes use the latest value instead of the stale original, which
fails with invalid_grant on providers that rotate refresh tokens
(Auth0, Okta, Google, etc.).
Adds regression tests:
- One failing OAuth server still yields the healthy server's headers
- Rotated refresh_token is posted on the next refresh attempt
Co-authored-by: Claude <noreply@anthropic.com>
_SubagentEventBuffer.flush() cleared self._pending before the put_batch and
discarded the batch when persistence raised, so a transient store error
silently lost subagent step events. On failure, prepend the failed batch
back onto self._pending (ahead of events queued since) so a later flush can
retry it.
* fix(sandbox): stop glob/grep/ls from surfacing disabled skills' files
The disabled-skill gate checks the path a tool is given, but ls, glob and
grep all descend from it and return other paths, so a root above a disabled
skill still serves its files. glob and grep never called the gate at all;
ls called it only on its own argument and still leaked from a category root.
Add the entry gate to glob/grep, and filter what all three return through
the existing fail-closed _is_disabled_skill_path. The verdict is memoized per
skill because ExtensionsConfig.from_file() is uncached, so a per-match check
would turn a 100-match grep into 100 config reads.
* fix(sandbox): normalize trailing slashes in the disabled-skill path check
Review follow-ups on the disabled-skill gate:
- _extract_skill_name_from_skills_path returned "" instead of None for a
category directory carrying a trailing slash. LocalSandbox.list_dir appends
"/" to directories, so `ls /mnt/skills` yields "/mnt/skills/public/", giving
parts ["public", ""]. The empty name skipped the `skill_name is None`
short-circuit and fell through to a config read, landing on the right outcome
only because unknown skills default to enabled. Drop empty segments so a
trailing-slash category root takes the existing category-root branch.
- ls_tool resolved the runtime user id twice per call; hoist it into a local,
matching glob_tool/grep_tool.
- Cover the CUSTOM path: custom/legacy skills resolve their enabled state
through the per-user _skill_states.json, a different store from the public
skills' extensions_config.json, and no automated test exercised it.
The remote-content allowlist in ToolResultSanitizationMiddleware
(`_REMOTE_CONTENT_TOOL_NAMES`) covered web_fetch / web_search /
image_search but not web_capture, which was added later. The Browserless
web_capture tool embeds the target site's `X-Response-Status` reason
phrase — free-form text controlled by whatever server is being captured
(RFC 7230 §3.1.2) — into its result message via `_target_status_warning`.
A malicious page could therefore forge a `<system-reminder>` block (or a
`--- END USER INPUT ---` boundary marker) through web_capture that would
be escaped for web_fetch, letting attacker-influenced remote content reach
the model as authoritative framework context.
Add "web_capture" to the allowlist so its result is structurally
neutralized for parity with the other remote-content tools. This extends
the same defense introduced in #4002 to the one built-in remote-content
tool it did not yet cover.
Add regression tests that build the web_capture result the way
community/browserless/tools.py does (real `_target_status_warning` +
`BrowserlessScreenshotResult`) and assert the forged tags/boundary markers
are escaped, while a benign status warning is preserved unchanged.
* fix(security): html-escape memory facts rendered into the injection prompt
The lead-agent system prompt declares the <memory> block user-managed and
everything else framework-internal, but the injection renderer _format_fact_line
formats a fact's content, category and correction sourceError raw. Memory is
user-editable via /api/memory, so a fact whose content is
'</memory></system-reminder>...' closes the block and relocates the text after
it out of the user-managed trust zone.
Escape those three fields at render time, mirroring the MEMORY_UPDATE_PROMPT
escaping added for the update-prompt side in #4028/#4060. The fact dict is not
mutated, so stored memory keeps the raw value and the apply path is unaffected.
* fix(memory): stop entity-encoding quotes in injected fact text
The three html.escape() calls in _format_fact_line used the default
quote=True, which also converts " to " and ' to '. These fields
are rendered as element text inside the <memory> block, never inside an
attribute value, so escaping quotes buys no defense here: only <, >, and &
can break out of the surrounding tags, and those are escaped either way.
Ordinary facts ("User's preference", 'Said "use Python"') reached the model
as User's preference / Said "use Python" -- content the
lead-agent prompt declares as user-managed data the model should discuss
freely. Pass quote=False and extend the benign-content test, which used a
string with no quotes and so never exercised this path.
Note the escape in updater.py's consolidation_candidates block renders into
an XML attribute value and correctly keeps quote=True.
`_provider_config` resolved a request-supplied provider name with an
unallowlisted `getattr(config, provider, None)`. `ChannelConnectionsConfig`
carries non-provider attributes -- the `enabled` and `require_bound_identity`
bool fields plus the `provider_status` method -- so a name matching one of
them returned that attribute instead of falling through to the intended 404.
Callers (e.g. `POST /api/channels/{provider}/connect`, reachable by any
authenticated user) then dereferenced the bool/method as a provider config,
crashing with `AttributeError` -> HTTP 500.
Validate `provider` against the `_PROVIDER_META` allowlist before the lookup,
matching how `_credential_fields` / `_connect_instruction` / `_connect_url`
already gate provider names, so unknown providers get a clean 404.
Add a parametrized router regression test covering `enabled`,
`require_bound_identity`, `provider_status`, and an unknown name.
* Make the sync checkpointer honor the unified database config
The sync checkpointer factory (`get_checkpointer` and `checkpointer_context`)
read only the legacy `checkpointer:` config section and fell back to
`InMemorySaver` when it was absent — it never consulted `database:`. The async
`make_checkpointer` factory and both sync/async Store providers already resolve
the unified `database:` section (legacy `checkpointer:` takes precedence,
otherwise `database:` drives the backend), so the sync checkpointer was the
lone outlier.
Consequence: with `database: {backend: sqlite|postgres}` and no legacy
`checkpointer:` section, the sync checkpointer silently returned `InMemorySaver`
while the Store — same process, same config — correctly persisted to
sqlite/postgres. Embedded callers (`DeerFlowClient`) and the TUI hit this: e.g.
the TUI writes `threads_meta` rows to sqlite (thread appears in the Web UI) but
its checkpoints went to memory and were lost on exit. This also contradicts
backend/AGENTS.md ("the unified `database` section selects the Gateway's
LangGraph checkpointer, LangGraph Store, and DeerFlow SQL repositories").
Mirror the sync Store provider: add `_resolve_checkpointer_config` /
`_get_checkpointer_config` (legacy precedence, else translate `database:` into
a CheckpointerConfig) and route both the singleton and the context manager
through them. The Gateway is unaffected — it uses the async path.
Adds a TestCheckpointerDatabaseConfig suite mirroring the Store's database
tests (singleton + context-manager honor `database:`, legacy precedence,
explicit-memory, missing-config fallback). The two `uses_database_config`
tests fail (return InMemorySaver) before the fix and pass after.
* Handle database=None in sync checkpointer resolution
The unified-config change made `checkpointer_context` / `get_checkpointer`
consult `app_config.database` when no legacy `checkpointer:` section is set.
`AppConfig.database` is always a `DatabaseConfig` in production, but the
existing regression test for issue #1016
(`test_checkpointer_none_fix.py::test_sync_checkpointer_context_returns_in_memory_saver_when_not_configured`)
mocks the app config with `database` left unset, exercising a `database=None`
path that now raised `ValueError: Unknown database backend`.
Mirror the async `make_checkpointer` factory, which already tolerates
`database=None` (falls back to memory), by treating a `None` database as the
memory backend in `_resolve_checkpointer_config`. Update the sync test to set
`mock_config.database = None`, matching its async sibling in the same file.
* Address review: mirror None-guard in store resolver, add sqlite coverage
`select:` names its targets explicitly, so capping it silently drops schemas the
model asked for by name -- and picks the survivors by catalog order, not request
order. The model is told the tool it wanted was not returned by nothing at all;
it then tries to call a tool that is still deferred.
The rule is already stated three times in the repo, and this is the one place
that breaks it:
- backend/AGENTS.md:447 -- "select: returns all requested skills without a
result cap; other modes cap at MAX_RESULTS=5"
- skills/catalog.py:71 -- SkillCatalog.search returns select: uncapped; the
ranked modes slice. DeferredToolCatalog shares its query grammar and its
MAX_RESULTS = 5, and is capped.
- tool_search's own docstring -- "select:Read,Edit -- fetch these exact tools
by name" versus "notebook jupyter -- keyword search, up to max_results best
matches". Only the ranked form promises a cap.
The cap is applied twice: once inside `DeferredToolCatalog.search` and again in
the tool closure. `search` already caps the ranked branches internally, so the
closure's slice is redundant for them and is the only thing capping `select:`
once the first is removed -- fixing one site alone changes nothing the model can
observe. Its sibling closure, `skills/describe.py::describe_skill`, calls
`catalog.search(name)` with no slice.
Both slices are removed. The ranked modes keep their cap.
* fix(sandbox): guard the output-masking regex with a segment boundary
`_compiled_mask_patterns` builds the same class of host→virtual matcher as
`LocalSandbox._reverse_output_patterns`, but without the segment-boundary
lookahead that one carries. The trailing group needs a separator to consume
anything, so when the character after a host base is `-`, `.`, `_`, a digit or
a letter, the group matches empty and the regex still matches the bare base.
`replace_match` then takes its `matched_path == base` branch and rewrites the
sibling: with `/mnt/skills` mounted at `.../skills`, output naming a sibling
`.../skills-extra/data.txt` is handed to the model as `/mnt/skills-extra/data.txt`
— a container path forward resolution explicitly refuses to map back, so
reading it raises FileNotFoundError.
This is the sibling site of #4035, which fixed the identical bug in
`local_sandbox.py`. That PR's scope argument enumerated the prefix matchers in
that file and missed this one; `mask_local_paths_in_output` runs on every
glob/grep match and on local bash output.
The boundary class mirrors `_content_pattern`'s, not `_command_pattern`'s: this
runs over arbitrary command output, where a base can legitimately be followed
by `,`, `:` or `\`, all of which the shell-oriented class rejects.
* test(sandbox): anchor the sibling boundary at the ACP source too
The sibling-rejection cases only fed the skills source. `_compiled_mask_patterns`
builds every source's matcher in one loop, so the ACP workspace carried the same
defect: nothing maps its parent, and `/mnt/acp-workspace-backup/hello.py` is
unresolvable in both directions.
User-data is the exception and is now pinned as such: `_thread_virtual_to_actual_mappings`
also maps the virtual root `/mnt/user-data` to the three dirs' common parent, so a
sibling of `outputs` is still inside a mount and has a real virtual path — the output
is byte-identical with and without the boundary. That test is green on main; it guards
the boundary from being narrowed into one that stops translating a mapped path.
Reverting only `boundary` turns the 5 skills + 4 ACP cases red and leaves the 3
user-data cases green.
* fix(sandbox): honor single-bound line ranges in read_file
read_file only sliced content when BOTH start_line and end_line were
provided, so read_file(path, start_line=100) or read_file(path,
end_line=50) silently returned the whole file. Handle one-sided ranges:
default the missing bound (start->1, end->EOF), clamp start to 1, and
return clear messages when start_line exceeds the file length or start_line
> end_line.
* address review: guard one-sided end_line <= 0 in read_file
Add symmetric end_line guard and run the inverted-range check regardless
of which bounds are explicit, so a lone end_line<=0 returns a clean
error instead of a negative-index slice. Per @willem-bd review on #4078.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
`_fallback_title` sliced the user message to `min(max_chars, 50)` and then
appended a three-character ellipsis, so the returned title could be three
characters longer than the configured cap. `_parse_title`, six lines above,
slices the model's answer to `max_chars` exactly -- both read the same
`TitleConfig.max_chars`, only one honoured it.
This is the default path, not an error branch: `config.example.yaml` ships
`title.model_name: null` ("null = fast local fallback"), so every title is
produced here unless the operator opts into a title model. `max_chars` is a
documented key with a pydantic range of 10..200; any value in 10..52 makes a
long first message overshoot its cap.
Reserve room for the ellipsis before slicing. At the shipped `max_chars: 60`
the body is still 50 characters, so default output is unchanged.
The existing `test_sync_generate_title_respects_fallback_truncation` asserted
the shape of the truncation but never its length -- at its own `max_chars=50`
it was passing on a 53-character title. It now asserts the bound it is named
after.
* fix(artifacts): honor trusted owner-user-id header
The artifact endpoint resolved paths only via the effective user, so a
trusted internal caller acting on behalf of an owner (carrying
X-DeerFlow-Owner-User-Id) read the synthetic internal user's storage and
404'd on files the owner's run had written.
Resolve the owner via get_trusted_internal_owner_user_id and pass it
through resolve_thread_virtual_path (now accepting an optional user_id),
matching the memory and threads routers. Browser/API callers send no
such header and fall back to the effective user, so their behavior is
unchanged.
* fix(artifacts): normalize trusted owner id through make_safe_user_id
The owner-user-id header carries the raw platform owner id, while runs
store files under the make_safe_user_id bucket. Resolve artifacts with
the normalized id, mirroring the memory router.
MemoryStreamBridge is the DEFAULT stream backend. Its subscribe() creates a
fresh ended=False stream on every call via _get_or_create_stream(), but the
Gateway's reconnect guard _terminal_record_stream_missing only detects a
missing stream on bridges that expose stream_exists — and MemoryStreamBridge
did NOT define it (only RedisStreamBridge did). After worker cleanup pops the
stream (~60s post-run), a browser SSE reconnect or POST /wait hits
subscribe() -> creates a zombie stream -> yields heartbeat forever without
ever sending END_SENTINEL. The UI spinner never resolves and the coroutine
pins a server-side connection/request until external timeout.
Add the missing stream_exists method, mirroring RedisStreamBridge.
* ci: add nightly build workflow for images + helm chart
Nightly build of backend/frontend/provisioner images and the helm chart,
pushed to GHCR with nightly + nightly-YYYYMMDD tags (latest stays on v*
releases). amd64 only. Gated to the upstream repo (bytedance/deer-flow).
Documented in RELEASING.md.
* ci(nightly): harden chart patch + dedupe config_version check
Address PR #4050 review feedback:
- Gate the in-workflow chart patches with grep assertions so a drifted
Chart.yaml/values.yaml fails loudly instead of silently shipping a chart
that pulls the release `latest` images (sed exits 0 on zero matches).
- Suffix the nightly chart version with the short SHA
(<base>-nightly.<date>-<sha>) so a same-day re-dispatch re-publishes
cleanly; OCI chart versions are immutable and otherwise collide.
- Note in the image-tags comment that :nightly-<date> is mutable within a
day and :sha-<short> is the only truly immutable pin.
- Extract the config_version drift check into scripts/check_config_version.sh,
shared by chart.yaml and nightly.yaml, so the parsing logic lives in one
place.
* feat(runs): cross-process run ownership with lease + reconciliation (#3948)
Implements work items 2 and 3 of the multi-worker P0 plan
(docs/multi_worker.md). Work item 1 (Postgres startup gate, #3960)
already landed; this PR makes run creation race-safe across worker
processes and lets Postgres deployments recover orphaned inflight runs
from crashed workers without mis-marking live runs as orphans.
Work item 2 — cross-process atomic create_or_reject
- Alembic revision 0004_run_ownership adds runs.owner_worker_id,
runs.lease_expires_at, idx_runs_lease, and a partial unique index
uq_runs_thread_active (one pending/running run per thread). The
index is declared on RunRow.__table_args__ with sqlite_where +
postgresql_where (mirroring uq_channel_connection_active_identity)
so the empty-DB bootstrap path — which runs Base.metadata.create_all
+ alembic stamp head without executing any revision's upgrade() —
also lands it on fresh deployments. Migration 0004 additionally
creates it idempotently for legacy/versioned upgrades.
- RunRepository.create_run_atomic is the new atomic primitive:
- reject: INSERT directly; the partial unique index catches
duplicate active runs; the manager surfaces the result as
ConflictError.
- interrupt/rollback: SELECT FOR UPDATE the conflicting rows,
skip rows whose lease is still valid AND owned by another live
worker (raise ConflictError — the INSERT would have failed on
the index anyway, and a retry loop cannot make progress),
cancel the rest in the same transaction, then INSERT the new
row. Rows owned by this worker are interruptible regardless of
lease state.
- RunManager.create_or_reject dispatches to the store under the
existing local lock; same-worker in-memory cancellation runs after
the store commit succeeds. MemoryRunStore mirrors the same
semantics for tests and database.backend=memory.
Work item 3 — lease heartbeat + Postgres reconciliation
- RunOwnershipConfig (lease_seconds=30, grace_seconds=10,
heartbeat_enabled=false by default), registered as startup-only in
reload_boundary.STARTUP_ONLY_FIELDS because the heartbeat background
task is created once in langgraph_runtime() and is not rebuilt on
config.yaml edits.
- When heartbeat_enabled, each worker renewes leases on its own
active runs with interval = lease_seconds / 3. The loop is bounded
and stop-event-cancellable so shutdown is prompt.
- reconcile_orphaned_inflight_runs now runs on every backend — the
sqlite-only gate in app/gateway/deps.py is dropped in the same
commit so there is no window where Postgres would mis-mark live
Worker A runs as orphans. Reconciliation errors only runs whose
lease is NULL (legacy pre-ownership rows) or older than
grace_seconds. In single-worker mode (heartbeat off, NULL leases)
all inflight rows reclaim immediately, preserving the pre-ownership
recovery latency.
- Heartbeat starts AFTER startup reconciliation and stops BEFORE the
in-flight run drain on shutdown so the two cannot race.
GATEWAY_WORKERS=1 with heartbeat_enabled=false keeps current behavior.
Verified: 170 related tests + full backend suite (minus Docker-gated
live tests) green; ruff check + ruff format clean.
* fix(runs): tighten unique-violation handling and document clock-sync budget
Three follow-up fixes to the cross-process run ownership work in #3948,
surfacing during review.
1. _is_unique_violation: detect by driver-native signal, not message text
The previous substring heuristic ("unique" + "violat", or "duplicate")
missed SQLite's actual phrasing "UNIQUE constraint failed: <table>.<index>"
— SQLite says "failed", not "violates", and never "duplicate". On SQLite
the detector returned False, the reject path re-raised the raw
IntegrityError, and clients saw HTTP 500 instead of ConflictError 409.
The conversion is the load-bearing piece of the "store is source of
truth" design but was untested — every atomic test used MemoryRunStore,
which raises ConflictError directly and never reached this branch.
Now prefers driver-native signals: psycopg pgcode/sqlcode "23505" and
sqlite3 sqlite_errorcode SQLITE_CONSTRAINT_UNIQUE (reachable through
SQLAlchemy IntegrityError.orig). Message matching stays as a fallback
with SQLite's exact "unique constraint failed" phrase added.
2. interrupt/rollback: convert exhausted-retry IntegrityError to ConflictError
The reject branch converts unique violations to ConflictError. The
interrupt/rollback retry loop did not — on the 3rd attempt it re-raised
the raw IntegrityError, leaking HTTP 500 for the same race condition
that reject surfaces as 409. Symmetric conversion added after the loop;
callers now see a consistent ConflictError regardless of strategy.
3. Document clock-sync requirement for multi-worker lease reconciliation
reconcile_orphaned_inflight_runs compares another worker's UTC
lease_expires_at against this worker's datetime.now(UTC). The only skew
budget is grace_seconds (default 10s) — worst case, with the owning
worker's heartbeat just about to fire, a peer whose clock is more than
~grace_seconds ahead can mis-reclaim a still-live run as an orphan.
Documented in RunOwnershipConfig's docstring (with the math) and in
config.example.yaml (with operational guidance), so operators in
NTP-poor environments know to raise grace_seconds. Default unchanged:
10s is reasonable for NTP-synced K8s/cloud, and bumping it would slow
recovery of genuinely dead workers (lease_seconds + grace_seconds from
last heartbeat to reclaim).
Tests:
- test_create_run_atomic_reject_propagates_conflict_on_unique_violation:
end-to-end against a real SQLite-backed RunRepository, pre-inserts an
active run, asserts reject-strategy create surfaces as ConflictError
rather than raw IntegrityError.
- test_is_unique_violation_detects_real_sqlite_integrity_error: unit test
for the detector against a real SQLite-raised IntegrityError; asserts
driver-level sqlite_errorcode is SQLITE_CONSTRAINT_UNIQUE.
- test_interrupt_exhausted_retries_surface_as_conflict_error: pins the
symmetric 409 behavior after the retry loop exhausts.
Verified: ruff check + ruff format clean; multi-worker + run_repository
+ owner_isolation + reload_boundary suites green.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(runs): close multi-worker ownership gaps in lease heartbeat and unique-violation detection
Five code-review fixes from docs/multi_worker.md:
1. Drop unused ``claim_inflight_runs`` primitive — no caller anywhere.
``create_run_atomic`` does its own inline claim (SELECT FOR UPDATE +
cancel) inside the INSERT transaction; a separate claim primitive
would split that into two transactions and open a claim→INSERT race.
Removes ~40 lines across base.py / memory.py / sql.py plus the
unused ``now_iso`` parameter, freeing future RunStore implementations
from providing it.
2. Broaden ``_renew_leases`` filter to renew pending/running runs owned
by this worker even when ``record.task is None``. The previous
``task is not None`` requirement skipped the brief window between
``create_run_atomic`` inserting the row and the worker spawning the
agent task; under event-loop load that window can approach
``lease_seconds``, after which peer reconciliation marks the run
``error`` (visible) or a peer's ``create_or_reject("interrupt")``
silently kills the queued run. Filter now:
``task is None or not task.done()``.
3. Document the unsynchronised ``record.lease_expires_at = new_expiry``
write. ``lease_expires_at`` is the only field on an existing record
this path mutates; ``set_status`` / ``_persist_status`` touch other
fields, so there is no concurrent writer to race against. Re-acquiring
``self._lock`` would serialise unrelated run mutations for no gain.
4. Gate ``_is_unique_violation`` message fallbacks on
``isinstance(current, (SAIntegrityError, sqlite3.IntegrityError))``.
The driver-code path (pgcode/sqlite_errorcode) remains load-bearing;
substring fallbacks are now belt-and-suspenders only for cases where
the driver attribute isn't reachable through the cause chain. Without
the gate, any application exception whose ``str()`` happens to contain
"duplicate key" / "unique" + "violat" (CHECK constraint, validation
error) would silently surface as HTTP 409 instead of 500.
5. Route ``update_lease`` through ``_call_store_with_retry`` for
consistency with every other store call, and wrap
``await self._renew_leases()`` in ``_heartbeat_loop`` with
``except Exception: logger.warning(...)``. Previously a transient
error from the snapshot path or an unexpected exception would kill
the heartbeat task silently — after which no lease is ever renewed
again and every active run eventually looks orphaned.
``except Exception`` lets ``CancelledError`` (BaseException since
3.8) propagate so shutdown cancellation still works.
Regression tests:
- ``test_heartbeat_renews_pending_run_before_task_is_spawned``
- ``test_is_unique_violation_does_not_misclassify_application_exception``
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(runs): harden multi-worker migration, memory atomicity, and tz-naive lease comparison
Three follow-up fixes to the multi-worker run ownership work:
- migration 0004 dedupe pass: cancel superseded duplicate active rows per
thread before creating the partial UNIQUE index ``uq_runs_thread_active``
so dirty DBs (Postgres deployments that had reconciliation skipped by the
old sqlite-only gate, or any env that ran GATEWAY_WORKERS>1 before this PR)
do not abort the alembic upgrade and block gateway startup. Keeps the
newest active row per thread, marks the rest as error with an explanatory
message.
- MemoryRunStore.create_run_atomic interrupt/rollback path: split the single-
pass loop into two passes (collect candidates, validate, then mutate) so a
ConflictError raised on a later candidate does not leave earlier candidates
half-interrupted. Mirrors the SQL store's transactional rollback semantics;
the entire test_multi_worker_run_ownership.py suite runs against memory so
this divergence was giving false confidence.
- RunRepository.create_run_atomic interrupt path: coerce tz-naive
``row.lease_expires_at`` to UTC before comparing against the aware
``cutoff``. SQLite drops tzinfo on read despite ``DateTime(timezone=True)``
(this file's own comment acknowledges it), so the Python-side comparison
raised ``TypeError: can't compare offset-naive and offset-aware datetimes``
whenever heartbeat was enabled on SQLite and a lease was non-NULL. Defaults
(heartbeat off -> leases always NULL) masked it, but there was no guard
against the combination. Follows the existing "naive is UTC" convention
from ``coerce_iso``.
Each fix ships with a regression test pinning the behavior.
Co-Authored-By: heart-scalpel <heart-scalpel@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(runs): enforce heartbeat for multi-worker, fix memory-store datetime comparison, lazy-import ConflictError in store layer
Three fixes from code review:
1. Extend the startup gate (GATEWAY_WORKERS>1) to also require
run_ownership.heartbeat_enabled=true. Without heartbeat every run has
a NULL lease, so reconciliation treats all inflight rows as orphans
and Worker B would kill Worker A's live runs on every rolling update
or scale-up.
2. Fix MemoryRunStore.list_inflight_with_expired_lease to parse
created_at as datetime instead of ISO string lexical comparison,
and handle tz-naive lease values uniformly with the SQL store.
3. Store layer (sql.py, memory.py) now lazy-imports ConflictError
inside create_run_atomic instead of importing from the higher
RunManager layer at module level.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(runs): add owner check to update_lease, document create() assumption, restore deleted comment
- update_lease (SQL + memory) now requires owner_worker_id match in WHERE
clause so the primitive is safe by construction against misuse
- create() docstring notes it bypasses atomic create_run_atomic and
assumes no active run exists for the thread
- restore explanatory comment in MemoryRunStore.aggregate_tokens_by_thread
that was dropped in an earlier commit
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(runs): add psycopg3 sqlstate detection and periodic orphan reconciliation
- _is_unique_violation now checks sqlstate attribute (psycopg3 uses this
instead of pgcode). On Postgres, the only supported multi-worker backend,
detection was falling through to the message-substring fallback.
- _heartbeat_loop now runs reconcile_orphaned_inflight_runs every 3rd
cycle (every lease_seconds) to catch orphans whose lease expires between
pod restarts. Single-worker deployments are unaffected (heartbeat off).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: heart-scalpel <heart-scalpel@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat(subagents): show runtime metadata on task cards
* fix(subagents): stop task-card render loop and dedupe model fetches
Address code review on the runtime-metadata cards:
- P1 render loop: the terminal ToolMessage is re-parsed on every
MessageList render and always carries modelName/usage, so the
presence-based setTasks condition fired a fresh state object each
render -> "Maximum update depth exceeded". computeNextSubtask now
returns a value-compared `changed` flag and a pure subtaskNotification()
routes terminal transitions through the deferred after-render path
while skipping no-op re-parses.
- Per-card useModels refetch: add staleTime: Infinity to the ["models"]
query so every subtask card shares one /api/models fetch instead of
refetching on each mount.
* make format
* refactor(subagents): dedupe token-usage validators + tidy event narrowing
Address PR review follow-ups:
- DRY: extract one shared token-usage validator per side. Backend
status_contract.normalize_token_usage() now backs both the terminal
ToolMessage metadata and the subagent.step/.end run events
(step_events.py), and frontend messages/usage.normalizeTokenUsage()
backs both the live task_running event (lifecycle.ts) and the terminal
ToolMessage metadata (subtask-result.ts). Prevents the input/output/
total_tokens validation from drifting across the four former copies.
- Nit: onCustomEvent narrows event.type once instead of re-checking the
object shape per branch; the redundant task_started early-return
(already validated by taskEventToSubtaskUpdate) is dropped.
* fix(subagents): classify LLM error fallbacks as failed (#4041)
* fix(subagents): address #4042 review on LLM error fallback
- clarify _extract_llm_error_fallback docstring: tail-only scan is safe
because subagents append their own terminal message, so the last
AIMessage is never a stale parent-history marker (cross-ref worker.py)
- compute final_result and pop stop_reason only on the COMPLETED branch
so the guard stop-reason pop no longer fires on the discarded FAILED path
- note the error_detail/"LLM request failed" fallbacks are defensive;
the middleware always populates a non-empty content
- add regression test locking the stale parent-history marker invariant
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
