* feat(subagents): show runtime metadata on task cards
* fix(subagents): stop task-card render loop and dedupe model fetches
Address code review on the runtime-metadata cards:
- P1 render loop: the terminal ToolMessage is re-parsed on every
MessageList render and always carries modelName/usage, so the
presence-based setTasks condition fired a fresh state object each
render -> "Maximum update depth exceeded". computeNextSubtask now
returns a value-compared `changed` flag and a pure subtaskNotification()
routes terminal transitions through the deferred after-render path
while skipping no-op re-parses.
- Per-card useModels refetch: add staleTime: Infinity to the ["models"]
query so every subtask card shares one /api/models fetch instead of
refetching on each mount.
* make format
* refactor(subagents): dedupe token-usage validators + tidy event narrowing
Address PR review follow-ups:
- DRY: extract one shared token-usage validator per side. Backend
status_contract.normalize_token_usage() now backs both the terminal
ToolMessage metadata and the subagent.step/.end run events
(step_events.py), and frontend messages/usage.normalizeTokenUsage()
backs both the live task_running event (lifecycle.ts) and the terminal
ToolMessage metadata (subtask-result.ts). Prevents the input/output/
total_tokens validation from drifting across the four former copies.
- Nit: onCustomEvent narrows event.type once instead of re-checking the
object shape per branch; the redundant task_started early-return
(already validated by taskEventToSubtaskUpdate) is dropped.
* fix(subagents): classify LLM error fallbacks as failed (#4041)
* fix(subagents): address #4042 review on LLM error fallback
- clarify _extract_llm_error_fallback docstring: tail-only scan is safe
because subagents append their own terminal message, so the last
AIMessage is never a stale parent-history marker (cross-ref worker.py)
- compute final_result and pop stop_reason only on the COMPLETED branch
so the guard stop-reason pop no longer fires on the discarded FAILED path
- note the error_detail/"LLM request failed" fallbacks are defensive;
the middleware always populates a non-empty content
- add regression test locking the stale parent-history marker invariant
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(security): html-escape fact content in memory prompt sections
Raw memory fact content was injected verbatim into prompt XML — a fact
containing a literal `"` could break the `"..."` delimiter, and a
closing tag like `</consolidation_candidates>` could prematurely end
the XML block, both potentially confusing the model.
Apply `html.escape()` to `content` in `_build_staleness_section` and
`_build_consolidation_section`, and to `cat` in the consolidation
section's XML attribute. Tests added for both sections covering special
characters, XML tag injection, and attribute injection.
Follow-up to #3996 as noted by reviewer willem-bd.
* fix(security): address reviewer follow-ups on html-escaping PR
- Escape `cat` in _build_staleness_section for symmetry with the
consolidation section (both sections now consistently html-escape
all LLM-derived category values that appear in the prompt)
- Add comment at current_memory=json.dumps() documenting the conscious
accept: json.dumps leaves < > & unescaped; lower-risk than
staleness/consolidation (read-only context, not delete/merge
instructions); fix at fact-content insert time if revisited
- Add test for category escaping in the staleness section
* fix(security): reference tracking issue #4044 in conscious-accept comment
* style: compress conscious-accept comment to two lines
* fix(subagents): inject durable context before compaction
* fix(subagents): coalesce system messages after durable-context injection
Address #4040 review:
- append SystemMessageCoalescingMiddleware innermost on the subagent chain
so the SystemMessage(authority) DurableContextMiddleware injects is merged
into one leading system_message; otherwise the durable fix trades #4039's
assistant-first 400 for a duplicate-system 400 on strict backends
- add a two-system regression guard driving the real builder output through
a strict model; assert exactly one leading SystemMessage
- assert single-leading-system in the compaction integration test too
- update the middleware count/last-element assertion (coalescer is now
unconditionally last, removing the summarization-dependence ambiguity)
- compare _skills_root against posixpath.normpath(container_path)
- document the coalescer on the subagent chain in backend/AGENTS.md
* Fix circuit breaker wedging after a non-retriable half-open probe
When the circuit breaker is half-open it admits a single probe call by
setting `_circuit_probe_in_flight = True`. If that probe raised a
*non-retriable* error (e.g. quota/auth), the except block skipped both
`_record_failure()` (correct - business errors must not trip the breaker)
and any probe reset, so the circuit stayed `half_open` with
`_circuit_probe_in_flight = True` permanently. Every later call then
fast-failed in `_check_circuit()` forever, because no call could run the
handler to reach `_record_success` / `_record_failure`.
Release the probe on the non-retriable path (mirroring the existing
GraphBubbleUp handler) so the next call admits a fresh probe. The breaker
still never trips on non-retriable errors. Applied to both the sync and
async paths.
Adds sync + async regression tests asserting the probe is released and the
next `_check_circuit()` re-admits a probe.
* Address review: extract _release_half_open_probe helper
`format_memory_for_injection` bound `facts_header` / `all_fact_lines` only
inside the `if isinstance(facts_data, list) and facts_data:` block, but the
structure-aware overflow-truncation path at the end of the function
references both unconditionally.
When a user's memory has sizeable user-context / history (so `sections` is
non-empty and the assembled output exceeds `max_tokens`) but an empty or
missing `facts` list, that block is skipped, so the truncation branch hits
`UnboundLocalError: cannot access local variable 'all_fact_lines'` and
aborts memory injection entirely.
Hoist the two initializers to function scope, alongside the existing
`guaranteed_line_tokens = 0`, so they are always bound. Behaviour is
unchanged when facts are present.
Adds a regression test (empty facts + oversized user context) that fails
with UnboundLocalError before the fix and truncates gracefully after.
`before_agent` guards `context = runtime.context or {}` at the top, but the
`run_id` stamp on a trailing HumanMessage still read the raw `runtime.context`,
so a None context (thread_id resolved from `config.configurable`) plus a
HumanMessage last message raised `AttributeError: 'NoneType' object has no
attribute 'get'`. Use the guarded local `context` instead.
Adds a regression test.
* fix(sandbox): guard the reverse path-translation regex with a segment boundary
_reverse_output_patterns matches a mount's resolved local root with no
segment-boundary lookahead, unlike every other prefix match in this file: both
forward regexes carry one, and the three string-prefix resolvers compare against
`root + "/"`. Its optional trailing group needs a `/` or `\` to consume more, so
against a sibling that merely shares the prefix (".../skills-extra/data.txt") the
group matches empty and the regex still yields the bare root.
That extracted text then *equals* the mount root, which satisfies
_reverse_resolve_path's own `+ "/"` guard -- so the sibling is rewritten to
"/mnt/skills-extra/data.txt". Forward resolution refuses to map that back, so the
model is handed an absolute-looking container path it can never read. This runs
over every bash stdout/stderr and over read_file content.
Mirror _content_pattern's boundary class rather than _command_pattern's: this
regex sees arbitrary command output, where a root can legitimately be followed by
"," ":" or "\" -- all of which the shell-oriented class would reject, silently
dropping translations that work today. The trailing group keeps [/\\] so Windows
paths still match.
* test(sandbox): pin the end-of-output boundary that stops a host-path leak
The lookahead's `$` alternative had no test: deleting it left all 6866
backend tests green. It is not cosmetic. Output ending exactly at a mount
root -- printf '%s' "$PWD", a stripped last line, a capture truncated at
the buffer limit -- matches neither `/` nor `[^\w./-]`, so the pattern
fails and _reverse_resolve_paths_in_output emits the raw host path to the
model. main's unbounded pattern has no end-of-string problem; the
narrowing added here is what introduced the exposure.
Pin it: three prefixes, asserting both the container path and that the
host root is absent from the output. Deleting only `$` turns exactly
those three red.
Lift the boundary class and trailing group into named locals. The
compiled pattern is byte-identical; correctness hinges entirely on the
boundary class, so it should not sit at column 120 of a comprehension.
The apply-time staleness guardrail built ``candidate_ids`` with a direct
``f["id"]`` access over ``_select_stale_candidates`` output:
candidate_ids = {f["id"] for f in _select_stale_candidates(current_memory, config)}
Every other fact access in ``updater.py`` uses ``f.get("id")``; this was the
lone direct-subscript outlier. An aged, non-protected fact that lacks an
``id`` key — common in legacy / hand-edited / migrated ``memory.json`` — is a
valid staleness candidate, so it reached ``f["id"]`` and raised
``KeyError: 'id'``, aborting the entire background memory-update cycle for
that user. The guardrail runs unconditionally (independent of the
``staleness_review_enabled`` flag), so any id-less aged fact triggers it as
soon as the LLM returns a non-empty ``staleFactsToRemove``.
Skip id-less candidates when building the intersection set. They can never
be targeted by the id-based removal set anyway, so behaviour is otherwise
unchanged.
Adds a regression test with an aged, id-less fact that raises KeyError
before the fix and applies cleanly after.
Two memory-config docs disagreed with the actual defaults/behavior in
`config/memory_config.py` and `agents/memory/storage.py`:
- backend/AGENTS.md: `staleness_age_days` was documented as "default: 180;
range: 1-3650" but the field is `default=90, ge=30, le=365`;
`staleness_max_removals_per_cycle` was documented as "default: 5; range:
1-20" but the field is `default=10, ge=1, le=50`.
- config.example.yaml: the `storage_path` comment said "Path relative to
backend directory", but a relative `storage_path` resolves against the data
`base_dir` (`get_paths().base_dir / p`), not the backend working directory,
and an absolute path is what opts out of per-user isolation (matching the
field's own description in `memory_config.py`).
Doc-only change; no behavior change.
* fix(sandbox): scrub abbreviated *_PASS vars and Postgres PGPASSFILE from skill env
env_policy blocks the full PASSWORD/PASSWD spellings but not the ubiquitous
abbreviated _PASS form (DB_PASS, SMTP_PASS, MYSQL_PASS, ...), whose value is the
plaintext password itself, so every skill subprocess inherited it. Postgres's
file-based credential sources PGPASSFILE (.pgpass locator) and PGSERVICEFILE
(pg_service.conf, may carry a password) were likewise missed -- the psql analog
of the MYSQL_PWD/REDISCLI_AUTH no-flag sources #4018 added.
Collapse *PASSWORD*/*PASSWD* into a single *PASS* pattern (which subsumes both,
covers the abbreviated family, and catches PGPASSFILE) and add PGSERVICEFILE to
the exact-name denylist. *PASS* deliberately does not touch PWD/OLDPWD, which
carry no PASS substring. Capability-handle vars (KUBECONFIG, DOCKER_CONFIG,
GIT_ASKPASS, ...) are a broader, non-value-bearing class and are left out of
this fix.
test_secret_like_names_are_blocked already pins the contract; the new names are
added to its parametrize list (red before, green after) and the benign-names
test guards against over-scrubbing PWD/OLDPWD.
* test(sandbox): pin the *_ASKPASS over-scrub that *PASS* introduces
*PASS* also matches the credential-helper vars GIT_ASKPASS, SSH_ASKPASS and
SUDO_ASKPASS, which main inherits. Each names a program whose purpose is to hand
the caller a credential, so inheriting the pointer is the same leak class as
inheriting the value -- a skill can simply invoke it. Scrubbing them is intended;
pin all three in test_secret_like_names_are_blocked (red on main) so the
behaviour is a decision rather than a side effect of the pattern's shape.
test_benign_names_are_allowed could not have caught this: none of its 14 names
carries a PASS substring, so it passes identically before and after the
broadening and asserts nothing about it. Record that in its docstring -- the
absence of a benign PASS-bearing name is a decision, and PWD/OLDPWD are the
boundary the list does pin.
Also note the incidental COMPASS_*/BYPASS_* over-scrub in the module comment
(over-scrubbing is this module's fail-safe direction; required-secrets is the
escape hatch), and list PGPASSFILE alongside PGSERVICEFILE in AGENTS.md.
* feat(memory): add memory consolidation to synthesize fragmented facts
When a fact category accumulates many individual entries, the LLM
reviews them during the normal memory-update call (same invocation,
no extra API cost) and decides whether groups of related facts can be
synthesized into a single richer fact. This completes the memory
lifecycle: extraction → guaranteed injection → staleness review →
consolidation.
- Select fragmented categories by min-facts threshold, surface the most
fragmented groups first; prompt-layer caps aligned with apply-layer
guardrails so the LLM never sees groups it cannot act on
- Cap consolidated confidence at source maximum to prevent inflation;
reject results below fact_confidence_threshold
- Double-consume protection prevents a fact from being merged into
multiple consolidation targets
- Feature-gated at both prompt and apply time with per-cycle safety caps
- Add 26 tests covering candidate selection, normalization, apply
guardrails, and prompt integration
* fix(memory): address consolidation correctness issues from PR review
Six fixes based on maintainer review of #3996:
1. Deduplicate sourceIds in normalization — ["f1","f1"] previously
bypassed the ≥2-distinct-sources check; dict.fromkeys collapses it
to ["f1"] which is correctly rejected.
2. Run consolidation after max_facts trim — previously, sources were
deleted then the merged fact could be evicted by the trim, leaving
no record of either. Moving consolidation last ensures source facts
exist in the post-trim index before removal.
3. Fix count= attribute in consolidation prompt — advertised the full
category size but listed only max_sources IDs; now uses
min(len(group), max_sources) to match what the LLM can act on.
4. Exempt staleness_protected_categories from consolidation candidates
— mirrors the existing staleness-review contract so correction facts
are never surfaced for merging.
5. Strip and default category in consolidation normalization — " " or
" preference " are now normalised, matching _normalize_memory_update_fact.
6. Propagate sourceError from source facts into consolidated fact —
correction context is no longer silently lost on merge.
* fix(memory): add apply-time guardrails and tests for consolidation
P1: mirror the staleness-pass defense-in-depth pattern — build
allowed_source_ids from _select_consolidation_candidates at apply time
so a protected-category or below-threshold fact proposed by the LLM is
rejected regardless of model behavior.
P2a: test that LLM-returned confidence is capped at max source confidence
and that a capped result below fact_confidence_threshold is rejected.
P2b: test that factsToConsolidate with consolidation_enabled=False is a
no-op at apply time (35 tests, all pass).
* fix(memory): address three correctness issues from second review round
1. Default consolidation_enabled=False — consolidation is lossy (source
content is permanently replaced, only consolidatedFrom IDs preserved);
new lossy features default to off. config.example.yaml updated to match.
2. Unify confidence coercion between prompt and apply — _build_consolidation_section
now calls _coerce_source_confidence(fact) instead of an inline 0.0-default
coercion, so a null-confidence fact renders with 0.50 in the LLM prompt and
is capped at 0.50 at apply time (same value, same function).
3. Preserve staleness clock on merge — consolidated fact now carries the
newest source's createdAt (not now) so aged information does not gain a
fresh staleness-review window just by being consolidated; consolidatedAt
is added as an explicit audit field.
Three regression tests added (default=false, null-confidence consistency,
createdAt policy); all guardrail tests now set consolidation_enabled=True
explicitly so they test the guardrail, not the feature flag. 38 tests pass.
* fix(memory): harden createdAt comparison and confidence handling
1. createdAt max via _parse_fact_datetime — replaces string max() which
crashes on non-string createdAt (numeric unix timestamps) and sorts
Z/+00:00 mixed formats incorrectly. Mirrors how staleness computes age.
2. Remove dead min(..., 1.0) — _coerce_source_confidence already clamps
each source confidence to [0, 1], so max(source_confidences) ≤ 1.0
by contract; the outer min could never bind.
3. Clamp raw_llm_conf to [0, 1] before applying the source cap — out-of-
range values like 1.5 are safe today (pinned by the cap) but defensively
clamped first so the invariant holds even if the cap is ever loosened.
4. Doc: expand the apply-time guardrails comment to call out the protected-
category exclusion via allowed_source_ids — this is the central safety
property ("explicit user feedback is never silently merged away").
5. Test: add test_confidence_fallback_to_max_source_when_llm_omits_field
covering the else-branch (LLM omits confidence → uses max_source_conf).
6. Fix lint: reorder imports in test file (stdlib before third-party).
39 tests, all pass.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Phase 3 of #3875 — subagents previously inherited none of the lead's
context-compaction, so a deep-research subagent (max_turns up to 150)
could accumulate >1M cumulative input before max_turns/timeout/token_budget
engaged, even after Phase 2's budget capped the pathological tail.
- Gate the subagent runtime chain on the SAME ``app_config.summarization.enabled``
switch the lead reads (per maintainer guidance in #3875), via the shared
``create_summarization_middleware`` factory. One config covers both chains;
no separate ``subagents.summarization`` field. No-op when summarization is
off (factory returns None).
- ``skip_memory_flush=True`` on the subagent path: the factory otherwise
attaches ``memory_flush_hook`` (when memory.enabled), which flushes
pre-compaction messages into durable memory keyed by thread_id. Subagents
share the parent's thread_id, so without skipping the hook a subagent's
internal turns would pollute the PARENT thread's durable memory
(#3875 Phase 3 review point).
- Harden ``capture_new_step_messages`` to tolerate history contraction:
summarization rewrites the messages channel via
``RemoveMessage(id=REMOVE_ALL_MESSAGES)``, shrinking len(messages) below
the step-capture cursor. Without a reset, every step appended after the
compaction point was dropped until length overtook the stale cursor (#3845
interaction, maintainer validation point (a)). Cursor now resets to the
new tail; id/content dedup prevents re-emitting pre-compaction steps.
- Couple the DEFAULT token-budget ceiling to ``summarization.enabled``
(#3875 Phase 3 review point): 1M when compaction is on, 2M when off
(preserves Phase 2's deliberate headroom for summarization-off
deep-research runs that can exceed 1M). A user-set budget (global or
per-agent) always wins regardless of the switch. Flagged tunable.
The summarization middleware does not implement ``consume_stop_reason``, so
the Phase 2 guard-cap stop-reason channel is unaffected.
Refs: https://github.com/bytedance/deer-flow/issues/3875
`image_search` set both `image_url` and `thumbnail_url` to the DDGS result's
`thumbnail` field, so the tool never returned the full-resolution image even
though its docstring/usage_hint promise reference-quality images (and a result
with no `thumbnail` returned an empty `image_url`). DDGS `.images()` exposes the
full-res source under the separate `image` key — read that for `image_url`,
matching the serper/brave providers which keep image vs thumbnail distinct.
Adds a regression test.
* fix(agent): snap tool-output tail forward so fallback truncation respects max_chars
_snap_to_line_boundary() moves an offset backwards to the preceding newline.
That is correct for the head's end offset, but _build_fallback() also applied it
to the tail's start offset, where moving backwards lengthens the tail. On output
whose last newline sits in the second half (log lines followed by one long
unbroken line: minified JSON, base64 artifacts), the returned string exceeded
max_chars by up to 17x, defeating the guard that exists to stop a single large
tool result from blowing the model context.
Add _snap_start_to_line_boundary(), the forward-snapping mirror, and use it for
the tail offset. The existing head_end guard becomes redundant because a forward
snap can only move the tail start away from the head.
test_result_never_exceeds_max_chars already asserted this invariant but passed
newline-free content, so the snapping branch was never exercised.
* test(agent): pin the forward-snap direction of the fallback tail
The bound test only asserts that the result fits max_chars. Its content has
no newline inside the tail's snap window, so _snap_start_to_line_boundary
returns pos unchanged and the test stays green even with the snap removed.
Place a newline in the window so the snap has to fire: the tail must begin
after it, which the pre-fix backward snap does not do.
* feat(mcp): auto-promote deferred MCP tools from routing hints
When tool_search.enabled=true defers MCP tool schemas, PR1 routing hints
still require the model to spend a tool_search discovery round trip before
it can call the tool the routing metadata already points at. This adds a
McpRoutingMiddleware that matches the latest user message against PR1
routing keywords and promotes the matching deferred schemas before the
model call, removing that round trip.
Design (soft routing, opt-in, additive):
- Matches only the latest real HumanMessage (shared is_real_user_message
helper, reused by SkillActivationMiddleware so the two cannot drift);
case-insensitive substring match, no tokenizer dependency.
- Ordering: priority desc, then tool name asc; capped by the new global
tool_search.auto_promote_top_k (default 3, clamped 1..5). Does not add or
consume a per-tool auto_promote_top_k (PR1 schema unchanged); a per-tool
value is ignored with a DEBUG note.
- Returns a plain {"promoted": ...} state update (not a Command) and relies
on ThreadState.merge_promoted for union/dedupe, so auto-promote and a
model-triggered tool_search converge on the same catalog hash.
- Installed before DeferredToolFilterMiddleware on every deferred-tool path
(lead agent, subagent, embedded client, webhook via shared builders);
a construction-time assert rejects the reversed order. catalog_hash is
None / no routing index is a complete no-op, so bootstrap and ACP skip it.
- Privacy: never executes tools, never promotes policy-filtered tools, adds
no routing keywords or matched tool names to trace metadata or INFO/WARN
logs.
No behavior change when tool_search.enabled=false.
Tests: index construction, matching semantics, middleware state updates,
same-cycle deferred-filter interaction, lead/subagent/embedded-client
builder wiring + order invariant, config clamping, config.example.yaml
parseability, and privacy assertions.
* refactor(mcp): address auto-promote review nits
- executor: access app_config.tool_search.auto_promote_top_k directly to match
the lead-agent and embedded-client paths (drop the over-defensive getattr that
masked missing config); update the subagent test mock to carry tool_search.
- tool_search / mcp_routing_middleware: cross-reference the duplicated routing
priority/keyword normalization between the builder and the middleware's
defensive _normalize_index so they cannot silently drift.
- MCP_SERVER.md: document that auto-promote keyword matching is a case-insensitive
substring test (not word-boundary), advising distinctive keywords.
* fix(sandbox): scrub MYSQL_PWD and REDISCLI_AUTH from the inherited skill env
env_policy blocks the MySQL and Redis connection strings (MYSQL_URL, REDIS_URL)
because they embed a password, but not the password variables the same clients
read directly: mysql/libmysqlclient takes MYSQL_PWD and redis-cli takes
REDISCLI_AUTH with no further configuration. Both names carry no
KEY/SECRET/TOKEN/PASSWORD/PASSWD substring, so they were inherited by every
skill subprocess -- the leak surface this module exists to close (#3861).
They need exact entries rather than a pattern: *PWD* would also strip PWD and
OLDPWD. REDIS_AUTH is included as the common non-canonical variant.
* Apply suggestions from code review
- Distinguish REDIS_AUTH (defensive, non-canonical) from MYSQL_PWD and
REDISCLI_AUTH (documented no-flag credential sources) in the comment.
- Pin that request-scoped injection still overrides the host value for the
newly blocked names, keeping required-secrets a working escape hatch.
* bench: add provider-agnostic sandbox benchmark with BoxLite warm pool results
- scripts/bench_sandbox_provider.py: CLI for measuring acquire/run/release
across providers, scenarios, workloads, and concurrency levels
- scripts/summarize_bench.py: JSONL aggregation with p50/p95/p99 tables
- bench_results.jsonl: 110 turns across 7 scenarios on real BoxLite 0.9.7
Key findings:
cold acquire: ~860ms
warm reclaim: ~14ms (60x speedup)
release: ~0ms
warm_hit_rate: 95% (warm_same_thread)
* perf(boxlite): skip health check for recently-released warm pool boxes
Boxes released within health_check_skip_seconds (default 5.0s)
are promoted directly without the ~14ms echo-ok round-trip.
A VM alive seconds ago is overwhelmingly likely to still be alive.
Add sandbox.health_check_skip_seconds config option.
Set to 0 to always health-check (old behaviour).
Benchmark (warm_same_thread, noop, 20 iters):
acquire p50: 14.9ms → 0.0ms
total p50: 29.9ms → 14.0ms
* chore: move benchmark scripts into backend/scripts/benchmark/
* fix: address BoxLite benchmark review findings
* fix(boxlite): only skip warm reclaim checks for released boxes
* fix(benchmark): keep BoxLite shim workaround off the event loop
* fix(boxlite): invalidate dead boxes from command path
* test(boxlite): cover skip window and invalidation edge cases
* fix(boxlite): treat sandbox-has-been-closed as terminal in _exec
* fix(boxlite): harden warm-pool reclaim and benchmark accounting
* fix(boxlite): validate warm-pool reclaims by default
* fix(config): expose boxlite health-check skip setting
* fix(boxlite): tighten failure classification and benchmark workaround
* Update config_version to 21 in values.yaml
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
The tag-publish workflow (container.yaml) built backend/Dockerfile with no
build-args, so UV_EXTRAS was empty and the published *-backend image shipped
without the Postgres driver (only --extra redis). Multi-replica deployments
(K8s/Helm) that need shared Postgres persistence instead of file-based SQLite
could not use the release image without rebuilding it.
Pass UV_EXTRAS=postgres so the release image includes
deerflow-harness[postgres]. Additive only: single-replica sqlite/redis setups
keep working; the Postgres driver is added, mirroring how redis is already
always baked in.
* feat(helm): add production-ready Helm chart for Kubernetes deployment
Adds deploy/helm/deer-flow, a native-Kubernetes translation of the
production docker-compose stack, plus CI to publish its images and chart.
* ci(release): gate releases on version-source consistency
Add a reusable verify-versions workflow invoked by both chart.yaml and
container.yaml on v* tags. It runs scripts/verify_versions.sh against the
tag and fails the release — skipping all image and chart publishing — when
Chart.yaml (version + appVersion), backend/pyproject.toml, or
frontend/package.json don't all match the tag.
Add scripts/verify_versions.sh (the check, also runnable locally) and
scripts/bump_version.sh (bumps all four sources in lockstep, then
self-verifies). Document the release flow in RELEASING.md and link it from
AGENTS.md.
* fix(deploy): address Helm chart review feedback (#3987)
Three review items from willem-bd:
1. nginx IPv6 listen strip never matched. The sed pattern required a `;`
immediately after `2026`, but the rendered config emits
`listen [::]:2026 default_server;` (space + `default_server` before the
`;`), so the line was never deleted and nginx crash-looped on pods
without IPv6 (`socket() :::2026 failed (97: Address family not
supported)`). Drop the trailing `;` from the pattern so it matches.
Same latent bug fixed in docker-compose-dev.yaml.
2. Passwords were spliced into DSNs verbatim, so a password containing
URL-special chars (@ : / # ? % [ ] space) produced a malformed DSN and
a confusing parse error. Add a `deer-flow.urlEscape` helper
(replace-based: Sprig lacks urlqueryescape, and regexReplaceAllLiteral
treats the replacement as a regex template so `[`/`]`/`?` break it) and
apply it to the password in the postgres and redis DSNs. The raw
`postgres-password` / `redis-password` keys stay unencoded - they back
POSTGRES_PASSWORD / REDIS_PASSWORD, not a URL segment.
3. NODE_HOST defaulted to "gateway", which can never route: the gateway
Service is ClusterIP:8001 and knows nothing of a sandbox NodePort, so a
user who skips the caveat gets unreachable sandboxes with no error at
install time. Default NODE_HOST to the provisioner pod's node IP via
the downward API (status.hostIP) - a NodePort is exposed on every node,
so <node-IP>:<NodePort> routes from the gateway on most clusters.
`provisioner.nodeHost` remains an override for CNIs/policies that block
pod->node-IP traffic. Updated NOTES.txt, values.yaml, and the chart
README. (#3929 remains the long-term fix - ClusterIP + cluster-DNS URL
removes NODE_HOST and the NodePort exposure entirely.)
Validated with helm lint, helm template (incl. a special-char password
rendering the encoded DSNs), and a sed pattern-match check.
* fix(deploy): address round-2 Helm chart review feedback (#3987)
Three "Medium" items from willem-bd:
1. No helm lint / helm template gate before publish. A template regression
ships as an immutable OCI artifact (GHCR won't overwrite --version), so
gate packaging on `helm lint` + `helm template --include-crds` in
chart.yaml before `helm package`. (ct lint / helm-unittest deferred.)
2. Action pinning inconsistent + PR body overstates it. SHA-pin
actions/checkout (v6.0.3, df4cb1c0) and actions/attest-build-provenance
(v2.4.0, e8998f94) across the publishing workflows (chart.yaml,
container.yaml, verify-versions.yml), matching the existing docker/*
SHA-pin pattern. Resolves the checkout @v4/@v6 mismatch and makes the
"SHA-pinned actions" claim accurate. Other pre-existing workflows left
untouched (out of scope for this PR).
3. Provisioner RBAC broader than needed. Dropped the unused update/patch
verbs and the pods/exec + events rules from the provisioner Role -
audited against docker/provisioner/app.py, which only calls
get/create/delete on pods and get/list/create/delete on services. Fixed
NOTES.txt to accurately describe the grant instead of understating it as
"create Pods and Services". The remaining scope concern - verbs apply to
all Pods in the namespace, not just sandbox Pods - is still deferred
(RBAC can't scope by label; needs a dedicated namespace or admission
control), now noted in NOTES.txt and README.
Validated with helm lint + helm template (narrowed Role renders with
exactly get/list/watch/create/delete).
* feat(helm): enable sandbox+web tools out of the box
The chart's default config loaded zero agent tools (config.tools empty ->
"Total tools loaded: 0"), so a fresh install gave an agent that could do
nothing useful. Add tool_groups + tools to the default config block:
- web: web_search (ddg), web_fetch (jina), image_search - no API key
- file:read: ls, read_file, glob, grep
- file:write: write_file, str_replace
- bash
The file/bash tools run inside the AIO sandbox the chart already
configures; the web tools need outbound internet from the gateway pod
(swap backends or drop entries for air-gapped clusters - see
config.example.yaml).
Also bump config_version 15 -> 19 to match config.example.yaml (the chart
had drifted behind). NOTES.txt and the README example updated to match.
* ci(helm): add chart validation + config_version drift check on PR
Extend the chart workflow with a PR-triggered validate-chart job that runs
helm lint, helm template --include-crds, and a config_version drift check:
it parses config_version from both config.example.yaml and the chart's
values.yaml and fails the build (with a ::error:: naming the files to bump)
if the chart is behind the example. This catches the kind of drift this
PR is fixing - the chart sat at v15 while the example moved to v19 - before
it can merge again.
verify-versions and publish-chart stay tag-only; publish-chart now
needs: [verify-versions, validate-chart]. validate-chart runs on both
PRs and tag pushes: the tag arm is required because a job that `needs`
a skipped job is itself skipped under the default success() check, so
validate-chart must actually run on tag pushes or publish-chart would
never fire.
* Bump config version to 20
* fix(security): neutralize prompt-injection tags in remote tool results
User input is already neutralized for framework/injection tags, but tool
results are not. Remote content fetched by web_fetch/web_search is equally
untrusted and can carry a forged <system-reminder> block that reaches the
model verbatim as authoritative context.
Extract a shared neutralize_untrusted_tags() primitive from
InputSanitizationMiddleware and apply it to remote-content tool results
(web_fetch/web_search/image_search) via a new ToolResultSanitizationMiddleware.
Local tool output (bash/read_file) is left untouched so legitimate code/file
content is never mangled.
* test: update subagent middleware count for tool-result sanitizer
The new ToolResultSanitizationMiddleware adds one entry to the shared runtime
chain (11 -> 12). Update the subagent count assertion, use a lazy import for
neutralize_untrusted_tags so the module loads even when tests stub the
input-sanitization module, and document the new middleware in AGENTS.md.
* fix(security): address review — sanitize bare str list items; document MCP scope
- Neutralize bare str elements inside a ToolMessage content list (previously
only {type:text} dict blocks were rewritten), matching the str-in-list shape
ToolOutputBudgetMiddleware._message_text already anticipates.
- Document the name-based allowlist limitation: MCP remote-content tools
registered under arbitrary names (e.g. fetch_url) are not covered; a name
heuristic is avoided to prevent mangling local tool output, with metadata
tagging tracked as a follow-up. Add a regression test pinning this boundary.
Phase 2 of #3875. Two guardrail axes can end a subagent run early — the turn
budget (GraphRecursionError) and the token budget (TokenBudgetMiddleware) —
and both now surface *why* through one additive `subagent_stop_reason` field
instead of a status enum.
This completes and course-corrects Phase 1 (#3949), which shipped the
turn-budget cap as a `max_turns_reached` status enum. The agreed Phase 2
design replaces that enum with an optional `stop_reason` field
(token_capped | turn_capped | loop_capped): a new enum value would break v1
consumers, while an additive field is ignored by older frontends and ledger
readers. `max_turns_reached` and SubagentStatus.MAX_TURNS_REACHED are removed.
- subagents.token_budget config (default enabled, 2,000,000 tokens, warn 0.7)
with per-agent override; TokenBudgetMiddleware is now attached in
build_subagent_runtime_middlewares so the cost-ceiling backstop engages for
every subagent. The hard-stop does not raise — it strips tool_calls and
lets the run finish with a final answer, recording the cap on a per-run
consume_stop_reason() accessor.
- executor.py: on normal completion it reads consume_stop_reason() and stamps
completed + token_capped when the budget fired; on GraphRecursionError it
recovers the last AIMessage partial (completed + turn_capped) or, if nothing
usable survived, failed + turn_capped. SubagentResult gains stop_reason.
- status_contract.py / contracts/subagent_status_contract.json (v2) /
frontend subtask-result.ts: additive subagent_stop_reason field, pinned by
test_status_values_match_contract / test_stop_reason_values_match_contract.
- task_tool.py + delegation_ledger.py: drop the max_turns_reached paths; the
ledger captures stop_reason and renders model-facing "capped" guidance so
the lead reuses a capped completion knowingly.
The 2,000,000-token default is deliberately loose (tighten to taste) — it
would have roughly halved the reported 4.4M burn while leaving legitimate
deep-research runs (max_turns=150) room. Subagent summarization is a follow-up.
* feat(frontend): render slash-skill activations as inline chips
Show an explicit `/skill` activation as a compact inline chip in both the
composer and the chat transcript instead of raw slash text.
- Composer: selecting a skill suggestion stores it as a removable chip
aligned inline with the textarea; the leading `/skill ` prefix is
reattached only at submit time, so the backend activation protocol is
unchanged. Backspace on an empty input or the chip's close button clears
it; history navigation is disabled while a chip is active.
- Transcript: human messages that begin with `/skill` render the skill as a
read-only chip followed by the task text.
- Add a shared `core/skills/slash.ts` (`parseSlashSkillReference` +
`resolveSlashSkillDisplay`) mirroring the backend `slash.py` gate, so the
transcript only shows a chip when the skill actually exists and is enabled.
This removes a duplicated regex/reserved-name list and keeps display
semantics consistent with backend activation.
Add unit tests for the shared slash parser and extend the chat e2e to assert
the composer still submits `/skill <task>` after showing a chip.
* chore(frontend): format chat e2e test
* refactor(skills): address slash-skill chip review feedback
Follow-up to the inline slash-skill chip PR, resolving three second-order
review findings:
- Drive the reserved-command set and skill-name grammar from a shared
contracts/slash_skill_contract.json instead of a hand-copied
"keep in sync" pair. slash.ts and slash.py now reference the fixture, and
contract tests on both sides fail CI if either drifts.
- Extract a shared SlashSkillChip so the composer and transcript chips stay
in lockstep, and normalize the off-scale /8 and /12 opacity steps to the
standard /10 and /20 tokens.
- Split HumanMessageText into a pure parse gate plus a slash-only subtree
that owns the useSkills() lookup, so a skill-enabled toggle no longer
re-renders every plain-text human turn.
Verified: frontend eslint + tsc clean, pnpm test 572 pass (incl. new
slash-contract test); backend slash contract + slash-skills tests 31 pass.
* style(tests): sort slash skill contract imports
* fix(composer): inline the slash-skill text so the chip aligns with input
Address the "composer body layout change" review on #3981 by rendering the
active skill as an inline chip in the same text flow as the prompt, rather
than a separate flex row that drifted the box model across states.
- Render the chip + prompt inside one leading-6 wrapper and edit the prompt
through a `contentEditable` span, so the chip sits inline with the first
line and long/multi-line input wraps naturally back to the container edge.
- Align the chip with `align-top`: its h-6 (24px) height matches the text
line height, so chip and first-line centers coincide exactly (measured
delta 0), fixing the chip being raised above the baseline.
- Restore the placeholder in chip mode via a `data-empty` CSS `::before`,
which also gives the empty editable span width so it is no longer treated
as hidden.
- Widen the IME helper to `HTMLElement` and route the span's keydown/paste
through the shared skill-suggestion, prompt-history, backspace-to-clear,
and IME-composition handlers so contentEditable behaves like the textarea.
- Extend chat.spec.ts to drive the inline skill editor instead of the
textarea after a chip is shown.
* style(frontend): fix composer class order formatting
* fix(composer): break long unbroken input inside the slash-skill row
The inline slash-skill editor wrapped with `break-words`
(overflow-wrap: break-word), which only moves an over-long token to the
next line before breaking it. A long unbroken string therefore started
on the line below the chip, and when the string contained a break
opportunity such as a hyphen the browser wrapped there and pushed the
remaining run to the next line, leaving a wide gap on the right.
Switch to `break-all` (word-break: break-all) so the text fills each
line from the chip and packs tightly regardless of hyphens or CJK.
* feat: add composer input polishing
* Revert "Merge branch 'main' into feat/input-polish"
This reverts commit 5b6ceccf0d, reversing
changes made to 45fbc57fef.
* Merge main into feat/input-polish
* style(frontend): format input helper polish guard
* fix(input-polish): address composer polish review findings
Frontend
- Add a cancel affordance to the in-flight polish status pill that calls
abortInputPolishRequest(), so a slow/hung provider no longer hard-locks the
composer for up to stream_chunk_timeout with a page reload (and draft loss)
as the only escape.
- Reset promptHistoryIndexRef/promptHistoryDraftRef when a rewrite is applied
(and on undo), so a stale history-browse index can no longer let the next
ArrowDown silently overwrite the polished draft.
- Disable polishing while an open human-input card is present, matching the
frontend/AGENTS.md rule that composer entry points defer to the card so
card-reply metadata is preserved.
- canPolishInput now reuses parseGoalCommand/parseCompactCommand instead of a
third hardcoded reserved-command regex, and drops the phantom /help entry
(no /help parser exists in the composer), so future builtins only need to be
taught to the existing parsers.
Backend
- Extract the non-graph one-shot LLM path (build model + inject Langfuse
metadata + system/user invoke + text extract) into
deerflow.utils.oneshot_llm.run_oneshot_llm, shared by the input-polish and
suggestions routers so tracing-metadata and invocation shape cannot drift
between the two copies.
- strip_think_blocks gains truncate_unclosed (default True, preserving the
suggestions/goal JSON-prep behavior); input polish passes False so a draft
that legitimately contains a literal <think> substring is no longer
truncated into a partial rewrite or a spurious 503.
- Validate the empty-check and max_chars boundary against the same stripped
view of the draft that is sent to the model, so the user-facing length
boundary and the model input can no longer disagree.
Tests / docs
- Backend: literal-<think> preservation, whitespace-only rejection, and
normalized-length/model-input agreement cases; suggestions tests repoint the
create_chat_model patch to the shared helper module.
- Frontend: helper unit tests updated for the /help/reserved-command change; a
new Playwright case covers cancelling an in-flight polish request.
- backend/AGENTS.md documents the shared one-shot helper and the polish
normalization/think-tag behavior.
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* fix(provisioner): gate legacy skills mount by user visibility
* fix(test): aio sandbox provider
* fix: use shared legacy skill visibility helper for sandbox mounts
* fix(mcp): synchronize session pool singleton lifecycle
Parity follow-up to #3778 (skill storage) and #3730 (sandbox provider).
get_session_pool() already serialised creation with _pool_lock, but its
fast-path check and final return read the global separately, and
reset_session_pool() cleared it with no lock. A reset_mcp_tools_cache()
(reachable via the /api/mcp/cache/reset admin endpoint) racing a
concurrent get could null the singleton between the None-check and the
return, handing the caller None despite the -> MCPSessionPool annotation.
Build and return the pool inside _pool_lock with a double check, and
clear it under the same lock in reset_session_pool(). The critical
section is tiny and never awaits, so holding the threading.Lock is safe
from both the async and sync/worker-thread paths. No behavior change for
single-threaded callers.
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Add phase 1 skill static scanning
* Rework SkillScan phase 1 as native scanner
* refactor(skillscan): align phase 1 with trimmed RFC contract
- SecurityFinding: 7 fields (rule_id, severity, file, line, message,
remediation, evidence); category/analyzer derive from the rule_id
prefix, confidence/column/fingerprint/metadata removed
- scan_archive_preflight()/scan_skill_dir() are pure functions: no
ScanContext, no policy schema; CRITICAL-blocks is a code constant and
skill_scan.enabled is applied by enforce_static_scan()/callers
- secret-* evidence is redacted before findings leave the scanner
- de-dup keys on (rule_id, file, line) so repeated occurrences keep
distinct locations for agent self-correction
- cloud-metadata detection consolidated into network-cloud-metadata
- nested zip members get a one-level stdlib magic-byte peek; an
executable member escalates package-nested-archive to CRITICAL
- install metadata sidecar removed (Phase 7 decides if it is needed)
- rule specs moved next to their analyzers; skillscan/rules/ removed
- tests updated + new anchors: redaction, dedup lines, nested-zip
escalation, single cloud-metadata rule, bundled-skill zero-CRITICAL
* fix(skillscan): tighten reverse-shell/secret/archive scan rules from review
Address PR #3033 review feedback on the native SkillScan analyzers:
- Reverse-shell false positives: split shell detection by signal strength
(/dev/tcp/, nc -e stay CRITICAL; bash -i, mkfifo -> new HIGH
shell-reverse-shell-heuristic, warn->LLM). The Python check is now
AST-anchored on real socket.socket/os.dup2/subprocess call sites instead
of raw-text substring matching, so prose/docstrings no longer hard-block.
- Secret evidence: _redact_secret_evidence returns [redacted] with no secret
bytes (was value[:6], which leaked 2 real token bytes past the prefix).
- Archive DoS: cap outer archive member count (_MAX_ARCHIVE_MEMBERS=4096);
scan_archive_preflight early-aborts with a package-too-many-members CRITICAL
finding (routes through the existing blocked->400 fail-closed path).
- shell-destructive-command: broaden the rm -rf matcher to sensitive system
roots (/home, /usr, /*, --no-preserve-root /) while leaving safe subpaths
unflagged.
- Dead code: collapse _decode_text_for_analysis to a single decode path and
drop the unused _TEXT_SUFFIXES set and _has_text_shebang helper.
- local_skill_storage: document why the host_path branch keeps app_config
possibly-None (lazy kill-switch resolution; avoids eager get_app_config in
config-free environments such as CI).
Tests: new negative/positive coverage in test_skillscan_native.py. Full
backend suite 6616 passed, 26 skipped.
* fix(sandbox): stop setup-sandbox from pre-pulling the stale :latest image
scripts/setup-sandbox.sh's fallback (used whenever config.yaml has no
uncommented sandbox.image) pulled the volces mirror's :latest tag. We
confirmed in #3921/#3922 that this tag is frozen on the pre-1.9.3
all-in-one-sandbox digest (1.0.0.156), which lacks the /v1/bash/*
routes required-secrets skills need — so the one script whose entire
job is 'pre-pull a working sandbox image' was pre-pulling a known-broken
one. Pin the fallback to :1.11.0 instead.
Also update config.example.yaml's commented image: example and
'Recommended' line to the same version, so uncommenting the example
doesn't reproduce the same trap.
Out of scope on purpose: aio_sandbox_provider.py's DEFAULT_IMAGE
constant (the harness-level default for AioSandboxProvider itself)
is a separate, broader default-image decision already flagged to
maintainers in #3921 — this PR only fixes the pre-pull helper script.
Reported in #3914 (a real user deleted their stale local image, reran
make setup-sandbox, and got the same broken :latest image back).
* fix(sandbox): make setup-sandbox warn when the pull won't affect the runtime image
Self-review caught a real gap in the previous commit: AioSandboxProvider
resolves its image as `sandbox_config.image or DEFAULT_IMAGE`
(aio_sandbox_provider.py:214), and DEFAULT_IMAGE is deliberately left
untouched (still the frozen :latest, per #3921 — that's a maintainer
decision, not this PR's scope). So when config.yaml has no uncommented
sandbox.image, pre-pulling :1.11.0 alone creates a NEW inconsistency:
the script reports success pulling a modern image, but the sandbox
that actually starts still falls back to the broken :latest — silently
leaving the user's underlying required-secrets/bash.exec problem
unfixed, which is worse than the previous consistent-but-broken
behavior (pre-pull :latest, run :latest).
Make the unconfigured path loud about this instead of silent: print
the exact config.yaml snippet needed to make the pulled image actually
take effect.
* fix(runtime): add final reconciliation for missed tool messages
* fix(gateway): persist hidden human input card responses
Persist allowlisted hidden human_input_response messages in RunJournal so
Human Input Cards can recover answered state from run_events after checkpoint
compaction. Keep generic internal hidden messages filtered and add regression
coverage for ask_clarification responses.
* feat(frontend): add structured human input cards for ask_clarification
Implement a reusable Human Input Card flow for ask_clarification while keeping
the existing text fallback for older clients and IM channels.
Backend:
- Add structured ToolMessage.artifact.human_input payloads for clarification requests.
- Preserve ToolMessage.content as the readable Markdown/text fallback.
- Normalize clarification options from native lists, JSON strings, plain strings,
mixed scalar values, None, and missing options.
- Derive input_mode as choice_with_other when options exist, otherwise free_text.
- Keep disable_clarification non-interactive behavior as a plain ToolMessage with
no human_input artifact.
- Cover artifact persistence and Gateway message metadata preservation in tests.
Frontend:
- Add human input protocol types, runtime guards, extractors, response builders,
and thread-state helpers.
- Add reusable HumanInputCard with option buttons, free-text input, pending,
read-only, disabled, and answered states.
- Render structured clarification cards from artifact.human_input, with Markdown
fallback for malformed or legacy tool messages.
- Preserve line breaks in structured question/context/option text.
- Hide submitted clarification bridge messages from the chat UI via
additional_kwargs.hide_from_ui.
- Send structured human_input_response metadata through the fourth sendMessage
options argument, preserving run context in the third argument.
- Wire submissions for normal chats, custom agent chats, agent bootstrap chats,
and sidecar chats.
- Derive answered state from raw thread.messages so hidden replies still update
the original card.
- Clear pending state when the hidden reply arrives, dispatch is dropped, or a
later async stream failure appears on thread.error.
* perf(frontend): optimize HumanInputCard UI interactions
- Support Enter key to submit text input (Shift+Enter for newline)
- Render question and context fields as Markdown instead of plain text
- Replace deprecated FormEventHandler type with structural typing
* test(frontend): add unit test cover optimize HumanInputCard UI interactions
* feat(frontend): disabled chatbox when has new human-input-card
* fix(style): lint error fix
* fix: sanitize hidden human input replies
- Preserve IME composition safety for human input card Enter submits
- Treat hidden human input responses as genuine user messages for sanitization
- Keep hidden card replies in memory filtering while excluding malformed/internal hidden messages
- Add regression coverage for card IME handling and hidden reply sanitization
* fix: tighten human input response validation
- Reject empty hidden human input response values
- Remove invalid list ARIA role from human input card options
- Add backend coverage for empty response payloads
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
