mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-07-21 02:05:45 +00:00
main
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 qin-chenghanandGitHub
|
e7d7da9e7c |
fix(frontend): strip <memory> tags in Streamdown to avoid React console error (#4209)
* fix(frontend): use code-aware stripLeakedSystemTags for all internal marker tags Replace the ad-hoc /<\/?memory>/g in ClipboardSafeStreamdown with a dedicated stripLeakedSystemTags() function in preprocess.ts that: - Covers all INTERNAL_MARKER_TAGS (<memory>, <system-reminder>, <current_date>, <uploaded_files>, <slash_skill_activation>) instead of only <memory> - Is code-aware: skips fenced code blocks (```) and indented code blocks (4-space indent), so user-written meta-discussions about the memory system are not silently stripped - Handles opening tags, closing tags, self-closing tags, and tags with attributes - Adds 15 unit tests covering all the above cases * fix(frontend): use marker-aware fence tracking in stripLeakedSystemTags The boolean insideFence toggle incorrectly closes a fence on any line matching 3+ backticks or tildes, regardless of the actual delimiter. This causes tags inside a tilde-fenced block containing a backtick sub-fence, or a 4-backtick block containing a 3-backtick sub-fence, to be silently stripped. Track the opening fence marker (character and run length) so that only a matching marker with at least the same length closes the fence. Adds 4 new test cases: - Tilde fence with inner backtick fence - 4-backtick fence with inner 3-backtick fence - Shorter tilde closing inside longer tilde fence - Tags stripped after real closing fence * fix(frontend): fix TypeScript strict errors in fence marker tracking - Use non-null assertion (!) on fenceMatch[1] (guaranteed by regex) - Use charAt(0) instead of [0] to avoid string | undefined type |
||
|
|
ad45f59d66 |
feat(memory): pluggable memory abstraction with self-contained DeerMem backend (#4122)
* feat(memory): pluggable + self-contained memory system (MemoryManager plan phases 1 & 2)
Phase 1 — Pluggable (steps 0-10):
- ABC MemoryManager (9 methods) + singleton factory + drop-in backend discovery
- DeerMem default backend with core/ (storage/queue/updater/prompt/message_processing)
- NoopMemoryManager backend (proves pluggability)
- All call sites (middleware/hook/prompt/gateway/client/app) routed through manager
- hasattr capability probing for DeerMem-internal methods (no hard imports)
- MemoryConfig gains manager_class field; shared vs DeerMem-private annotated
Phase 2 — Self-contained DeerMem (steps 11-18):
- backend_config passthrough + DeerMemConfig (all DeerMem-private fields moved off MemoryConfig)
- DI: DeerMem owns storage/queue/updater/llm as instance attributes (no global singletons)
- Storage independence: core/paths.py with own root (~/.deermem or ),
factory auto-injects deer-flow's runtime_home() as absolute base_dir (zero-config)
- LLM independence: core/llm.py via langchain init_chat_model (no create_chat_model)
- Trace independence: optional tracing_callback replaces inject_langfuse_metadata/request_trace_context
- Message processing independence: hide_from_ui default-skip + optional should_keep_hidden_message hook
- Internal imports → relative (only deer_mem.py ABC import is host-relative)
- Carrier (deer_mem.py adapter) / portable (deermem/ config+core) split
- New tests: test_deermem_self_contained + test_memory_manager_pluggable; all memory tests migrated
- Other-agent demo: samples/other_agent_demo/ + automated portability test
- config.example.yaml memory section updated to phase-2 schema
* feat(memory): port consolidation + staleness fix into self-contained DeerMem; phase-2 host hooks
Port upstream #3996 (memory consolidation) and #3993 (staleness KeyError fix)
from origin/MemoryManager into the pluggable, self-contained DeerMem structure
(backends/deermem/deermem/), adapted to the DI MemoryUpdater (config injected,
not get_memory_config globals):
- DeerMemConfig: add consolidation_enabled (opt-in, default false) /
consolidation_min_facts / consolidation_max_groups_per_cycle /
consolidation_max_sources
- prompt.py: factsToConsolidate JSON field + {consolidation_section} placeholder
+ CONSOLIDATION_PROMPT constant
- updater.py: _coerce_source_confidence / _select_consolidation_candidates /
_build_consolidation_section module helpers (matching the existing
_select_stale_candidates style); consolidation normalization in
_normalize_memory_update_data; consolidation apply in _apply_updates (after
max_facts trim, with apply-time guardrails mirroring staleness); staleness
KeyError fix (f["id"] -> f.get("id") is not None) applied to both the
staleness guardrail and the consolidation allowed_source_ids comprehension
- config.example.yaml: consolidation section under memory.backend_config
- tests/test_memory_consolidation.py: 40 DI-adapted tests (running, not skipped)
incl. the staleness KeyError regression
Also includes in-flight phase-2 host-integration work: storage_path semantics
(any absolute/relative value = root dir) and host-default tracing_callback /
should_keep_hidden_message hooks injected into backend_config by the factory.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(memory): add noop backend template and backends guide
- backends/noop/: complete drop-in template (config.py with zero deer-flow
imports, noop_manager.py with a 6-step new-backend walkthrough in its
docstring, commented optional fact-CRUD capabilities).
- backends/README.md: which files to touch when adding/swapping a backend,
the 5-item backend contract, and common pitfalls.
- manager.py: generalize backend examples in comments (drop mem0-specific
references).
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(frontend): guard formatTimeAgo against invalid timestamps
Return a neutral placeholder when the input date is invalid (e.g. an empty lastUpdated from a backend with no memories) instead of throwing 'Invalid time value' from date-fns.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(memory): wire tool-driven memory mode through the MemoryManager ABC
tools.py (memory_search/add/update/delete) now calls get_memory_manager()
instead of the removed host memory module, so tool mode (memory.mode: tool)
works for any backend. DeerMem.search is implemented (case-insensitive
substring match, ranked by confidence) as a stand-in for the planned
semantic retrieval; noop.search returns [] (unchanged). Fact-CRUD tools
use getattr+callable probing -- backends lacking those ops (noop) get a
clear JSON error instead of crashing.
Tests: test_memory_tools rewired to mock the manager (handler tests) +
TestModeGating retained; test_memory_search now covers DeerMem.search;
pluggable stubs test updated (search no longer a stub).
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: resolve lint errors (import sorting, type annotation quotes, E402 in skipped tests)
* docs: restore explanatory comments in config.example.yaml memory section
* fix(security): port html-escape memory facts fix (#4097) to vendored DeerMem prompt.py
* fix(memory): address review + port dropped upstream memory fixes
Review blockers (vendored DeerMem):
- #4044 restore _escape_memory_for_prompt (current_memory blob in
MEMORY_UPDATE_PROMPT) - prevents </current_memory> breakout
- #4028 html.escape staleness-section cat/content in _build_staleness_section
- #4119 add _escape_summary for injection-path summaries (Work/Personal/
Current Focus/Recent/Earlier/Background)
- default-model silent no-op: factory injects host default chat model via a
new host_llm slot (create_chat_model(name=None)); DeerMem prefers host_llm
over build_llm(model). Zero-config extraction works out of the box again
- MemoryConfigResponse: fix stale docstring (backend-agnostic shape; DeerMem
knobs live under backend_config, not top-level - restoring flat would
re-couple the API to DeerMem). Frontend audited: does not read /memory/config
- _host_default_tracing_callback: restore langfuse assistant_id/environment
- search: push category onto the ABC signature; DeerMem filters BEFORE the
top_k slice (was filtered client-side after slicing -> starved results)
- _do_update_memory_sync: split into wrapper+impl; bind trace_id into the
request-trace ContextVar on the Timer/executor worker via a new
trace_context_manager host hook (None trace_id left unbound - no fabrication)
- client.py fact-CRUD now passes user_id (was writing to the global bucket
while get_memory reads per-user)
- _resolve_manager_class: fail-fast (raise ValueError) on an unresolved
explicit manager_class instead of silently falling back to DeerMem (memory is
persistent state - a wrong store is a silent data-integrity footgun)
Upstream memory fixes dropped by the host->vendored rename conflict, re-ported
to backends/deermem/deermem/core/ (+ deer_mem.py):
- #4073 queue busy-timer-spin -> _reprocess_pending flag (core/queue.py)
- #4074 null source.confidence in staleness -> _coerce_source_confidence
(core/updater.py: _build_staleness_section + _apply_updates stale sort)
- #4075 factsToRemove is optional (drop from _REQUIRED_MEMORY_UPDATE_TOP_LEVEL_KEYS)
- #4076 null confidence in search ranking -> _coerce_source_confidence
(deer_mem.py DeerMem.search)
host_llm + trace_context_manager are host-injected via backend_config (factory
in manager.py), keeping backends/deermem/ at exactly one `from deerflow` line
(the ABC contract) - portability test preserved.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: resolve lint errors (F541 f-string without placeholders, E501 line too long)
* fix(memory): restore hide_from_ui clarification preservation, expose mode
Two memory-system fixes (F541/E501 lint was already fixed on this branch):
- filter_messages_for_memory: restore default preservation of well-formed
human_input_response clarification answers (v2 regression). The
self-containment refactor made the bare function skip ALL hide_from_ui when
no hook was passed, but upstream preserves well-formed clarification
responses by default (test_hide_from_ui_human_input_response_is_preserved).
Inline a host-agnostic _is_human_clarification_response mirror of
read_human_input_response as the default keep-decision; the host-injected
should_keep_hidden_message hook still overrides (production path unchanged).
Portable package stays zero `from deerflow`.
- /memory/config: expose `mode` (middleware|tool) in MemoryConfigResponse +
the config/status endpoints + client.get_memory_config. mode is a host-
shared, behavior-determining field missing from the response projection.
Sync tests (mock .mode; e2e assert mode present).
- Align manager_class field docstring with fail-fast behavior.
Tests: filter/self-contained/portability (35) + memory-config (4) pass;
ruff clean.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(memory): resolve ruff format failures in memory module + tests
`make lint` runs `ruff format --check` in addition to `ruff check`; 8 memory
files had pending format changes -- 7 pre-existing (deer_mem, updater, tools,
test_memory_queue/router/search/tools) + message_processing from the
hide_from_ui fix. Apply `ruff format`: whitespace/wrapping only, no logic
change. 109 memory tests pass; ruff check + format --check both clean.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(memory): address PR review - legacy field migration, fact_id contract, path/docs
Address willem-bd's review on PR head
|
||
|
qin-chenghanandGitHub
|
713ee544b7 |
fix(agents): stop persisting base64 image data in checkpoint state (#4140)
* fix(agents): stop persisting base64 image data in checkpoint state (#4138) The viewed_images state field stored full base64-encoded image data, which was duplicated across every subsequent checkpoint (O(n * steps) growth). A single 1MB image viewed early in a conversation would be re-stored in every checkpoint for the rest of the session. Changes: - ViewedImageData: replace base64 field with lightweight metadata (mime_type, size, actual_path) - view_image_tool: store only metadata in state, no base64 encoding - ViewImageMiddleware: read image files from disk on-demand in before_model and encode base64 temporarily for the model call - Update all tests to use the new metadata-only format This is the first step of #4138. The base64 data is no longer in persistent state, but the injected HumanMessage (with base64 content) still appears in the checkpoint for the step where it was injected. Checkpoint retention policies and large tool result dedup are separate follow-up items. * fix(agents): address review feedback on #4140 - view_image_tool: remove stale 'convert to base64' comment, replace with 'validate contents'; drop redundant image_size reassignment and add a TOCTOU guard that rejects files changed between stat() and read(). - view_image_middleware: extract _read_image_as_data_url helper that re-checks size against the recorded value AND the absolute cap (_MAX_IMAGE_BYTES). Document the trust assumption for actual_path (server-set, not client-settable) in the helper docstring. - view_image_middleware: abefore_model now runs the blocking read+encode via asyncio.to_thread to avoid stalling the event loop on up to 20MB images. - tests: add coverage for OSError during read, file-changed-since-view (TOCTOU), and size-exceeds-cap branches. |
||
|
qin-chenghanandGitHub
|
60e50537f3 |
fix(subagents): prohibit task tool in general-purpose system prompt (#4161)
* fix(subagents): prohibit task tool in general-purpose system prompt (#4159) The general-purpose subagent correctly lists `task` in disallowed_tools to prevent recursive nesting. However, the system prompt did not explicitly tell the LLM that `task` is unavailable. When the subagent sees the parent agent use `task`, it infers the tool is available and attempts to call it, triggering a LangGraph tool validation error. Add an explicit <tool_restrictions> block to the system prompt stating that `task` is NOT available and the subagent must NEVER attempt to call it. This prevents the LLM from attempting the call in the first place, rather than relying on runtime rejection. Add a regression test verifying the prompt contains the prohibition. * fix(security): register tool_restrictions in input sanitization denylist PR #4161 added <tool_restrictions> to general_purpose.py subagent prompt but did not register it in _BLOCKED_TAG_NAMES. The anti-drift test test_denylist_covers_framework_authority_blocks caught this: forging <tool_restrictions> in untrusted input could trick the model into believing it has (or lacks) tool restrictions it does not. Add 'tool_restrictions' to _BLOCKED_TAG_NAMES alongside the other subagent authority blocks (file_editing_workflow / guidelines / output_format / working_directory). |
||
|
qin-chenghanandGitHub
|
a94ea9325b |
fix(agents): load SOUL.md from agent dirs without config.yaml (#4136)
* fix(agents): load SOUL.md from agent dirs without config.yaml (#4135) resolve_agent_dir requires config.yaml to be present in an agent directory (added in #3481 to fix #3390, where memory-only directories were mistaken for agent directories). However, SOUL.md loading does not depend on config.yaml. When an agent is configured externally (e.g. via DEER_FLOW_CONFIG_PATH) and the agent directory contains SOUL.md but no config.yaml, resolve_agent_dir skips the directory and load_agent_soul returns None. Add a fallback in load_agent_soul: if the resolved directory does not contain SOUL.md, check the per-user and legacy directories directly for SOUL.md. This preserves the #3390 fix for resolve_agent_dir (which is also used by load_agent_config) while allowing SOUL.md to load independently of config.yaml. * fix(agents): gate SOUL.md fallback on missing config.yaml per review Address review feedback on PR #4136: 1. Gate the fallback condition on not (agent_dir / config.yaml).exists() so it only fires when resolve_agent_dir returned its default path (no agent dir qualified), not when a properly-resolved per-user agent simply lacks SOUL.md. This preserves the per-user shadowing invariant. 2. Fix test_loads_soul_from_user_dir_without_config_yaml to actually exercise the fallback path: per-user memory-only dir (no config.yaml, no SOUL.md) + legacy dir with SOUL.md (no config.yaml) -> fallback finds legacy SOUL.md. 3. Add test_soul_not_leaked_from_legacy_when_per_user_has_config to verify the gate prevents legacy SOUL.md leaking into a per-user agent. 4. Patch get_effective_user_id in test_loads_soul_without_config_yaml for consistency and resilience. |
||
|
|
68c968f6f3 |
fix(frontend): keep orphan tool messages visible (#3880)
* fix(frontend): keep orphan tool messages visible LangGraph `messages-tuple` stream mode can emit tool-result events out of order or replay them from subagent state (e.g. the bash subagent under LocalSandboxProvider with allow_host_bash: true). When that happens, the tool message arrives after a terminal assistant/human group, so getMessageGroups' lastOpenGroup() returns null. The previous behaviour was console.error + drop, which silently hid the tool result from the UI - the user could not see the tool output or tool-call records even though the agent executed the action correctly (backend + Langfuse trace were both fine). Fallback now attaches the orphan tool message to the most recent group so the UI shows it. Adds a unit test that covers the orphan-tool path and a duplicate-stream regression test. * test(frontend): make orphan-tool replay test actually reach the fallback The previous fixture was `human → ai(tool_calls) → tool → tool(replay)` with no terminal group in between, so both tool messages hit the unchanged happy path (`open.messages.push(...)`). Neither message was an orphan, the new `else if (groups.length > 0)` fallback was never exercised, and the `>= 1` length assertion was satisfied trivially by t-1a alone. Interleave a terminal assistant message between the original tool result and the replayed one, so t-1b arrives when lastOpenGroup() returns null. Now the strict assertion that t-1b is reachable can only pass via the new fallback branch. Review feedback from @willem-bd on PR #3880. * fix(frontend): satisfy noUncheckedIndexedAccess in orphan-tool fallback The fallback branch pushed into `groups[groups.length - 1].messages` directly, which trips `noUncheckedIndexedAccess` under the project's strict TS config and breaks lint-frontend, e2e-tests, and the full-stack render CI layer. Take `groups[groups.length - 1]` into a local `lastGroup` and check for `undefined` explicitly. The `else if (groups.length > 0)` guard becomes redundant once `lastGroup` is checked, so the branches are folded into the outer `else` to keep the structure flat. Behavior is unchanged: orphan tools still attach to the most recent group, and the empty-groups diagnostic still logs at ERROR level. Review feedback from @willem-bd on PR #3880. --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |