mirror of https://github.com/bytedance/deer-flow.git synced 2026-06-10 17:35:57 +00:00

Files

T

Ryker_Feng 167ef4512f feat(memory): add memory.token_counting config to avoid tiktoken network dependency (#3429 ) (#3465 )

* feat(memory): add memory.token_counting config to avoid tiktoken network dependency (#3429)

Add a `memory.token_counting` option (`tiktoken` | `char`) so deployments in
network-restricted environments can opt out of tiktoken entirely. In `char`
mode the memory-injection budget uses a network-free character-based estimate
and never triggers the BPE download from openaipublic.blob.core.windows.net,
which could otherwise block for tens of minutes (see #3402).

Also harden the default `tiktoken` path:
- cache an in-flight LOADING sentinel so concurrent callers fall back
  immediately instead of spawning more blocking get_encoding threads when the
  first load is still running (e.g. under the 5s startup warm-up timeout);
- cache failures with a timestamp and retry after a cooldown so a transient
  network outage self-heals back to accurate counting without a restart;
- skip startup warm-up entirely in char mode.

The new config is surfaced via the memory config API and config.example.yaml
(config_version bumped). Default remains `tiktoken`, so existing deployments
are unaffected.

* fix(memory): use CJK-aware char token estimate and address review feedback

- Replace the flat len(text)//4 fallback with a CJK-aware estimate so
  Chinese/Japanese/Korean memory content does not over-fill the injection budget
- Document the internal tiktoken retry cooldown and char-mode escape hatch
- Sync CLAUDE.md / config.example.yaml / MEMORY_IMPROVEMENTS.md wording
- Fix MemoryConfigResponse mocks/assertions and add CJK estimate tests

2026-06-10 23:26:15 +08:00

2.1 KiB

Raw Blame History

Memory System Improvements

This document tracks memory injection behavior and roadmap status.

Status (As Of 2026-03-10)

Implemented in main:

Accurate token counting via tiktoken in format_memory_for_injection.
Facts are injected into prompt memory context.
Facts are ranked by confidence (descending).
Injection respects max_injection_tokens budget.

Planned / not yet merged:

TF-IDF similarity-based fact retrieval.
current_context input for context-aware scoring.
Configurable similarity/confidence weights (similarity_weight, confidence_weight).
Middleware/runtime wiring for context-aware retrieval before each model call.

Current Behavior

Function today:

def format_memory_for_injection(memory_data: dict[str, Any], max_tokens: int = 2000) -> str:

Current injection format:

User Context section from user.*.summary
History section from history.*.summary
Facts section from facts[], sorted by confidence, appended until token budget is reached

Token counting:

Uses tiktoken (cl100k_base) when available
Falls back to a network-free CJK-aware character estimate if tokenizer import or encoding load fails (CJK characters count as ~2 chars/token, other characters as ~4 chars/token)

Known Gap

Previous versions of this document described TF-IDF/context-aware retrieval as if it were already shipped. That was not accurate for main and caused confusion.

Issue reference: #1059

Roadmap (Planned)

Planned scoring strategy:

final_score = (similarity * 0.6) + (confidence * 0.4)

Planned integration shape:

Extract recent conversational context from filtered user/final-assistant turns.
Compute TF-IDF cosine similarity between each fact and current context.
Rank by weighted score and inject under token budget.
Fall back to confidence-only ranking if context is unavailable.

Validation

Current regression coverage includes:

facts inclusion in memory injection output
confidence ordering
token-budget-limited fact inclusion

Tests:

backend/tests/test_memory_prompt_injection.py

2.1 KiB Raw Blame History