mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-06-10 17:35:57 +00:00
Ryker_Feng
167ef4512f
* feat(memory): add memory.token_counting config to avoid tiktoken network dependency (#3429) Add a `memory.token_counting` option (`tiktoken` | `char`) so deployments in network-restricted environments can opt out of tiktoken entirely. In `char` mode the memory-injection budget uses a network-free character-based estimate and never triggers the BPE download from openaipublic.blob.core.windows.net, which could otherwise block for tens of minutes (see #3402). Also harden the default `tiktoken` path: - cache an in-flight LOADING sentinel so concurrent callers fall back immediately instead of spawning more blocking get_encoding threads when the first load is still running (e.g. under the 5s startup warm-up timeout); - cache failures with a timestamp and retry after a cooldown so a transient network outage self-heals back to accurate counting without a restart; - skip startup warm-up entirely in char mode. The new config is surfaced via the memory config API and config.example.yaml (config_version bumped). Default remains `tiktoken`, so existing deployments are unaffected. * fix(memory): use CJK-aware char token estimate and address review feedback - Replace the flat len(text)//4 fallback with a CJK-aware estimate so Chinese/Japanese/Korean memory content does not over-fill the injection budget - Document the internal tiktoken retry cooldown and char-mode escape hatch - Sync CLAUDE.md / config.example.yaml / MEMORY_IMPROVEMENTS.md wording - Fix MemoryConfigResponse mocks/assertions and add CJK estimate tests
67 lines
2.1 KiB
Markdown
67 lines
2.1 KiB
Markdown
# Memory System Improvements
|
|
|
|
This document tracks memory injection behavior and roadmap status.
|
|
|
|
## Status (As Of 2026-03-10)
|
|
|
|
Implemented in `main`:
|
|
- Accurate token counting via `tiktoken` in `format_memory_for_injection`.
|
|
- Facts are injected into prompt memory context.
|
|
- Facts are ranked by confidence (descending).
|
|
- Injection respects `max_injection_tokens` budget.
|
|
|
|
Planned / not yet merged:
|
|
- TF-IDF similarity-based fact retrieval.
|
|
- `current_context` input for context-aware scoring.
|
|
- Configurable similarity/confidence weights (`similarity_weight`, `confidence_weight`).
|
|
- Middleware/runtime wiring for context-aware retrieval before each model call.
|
|
|
|
## Current Behavior
|
|
|
|
Function today:
|
|
|
|
```python
|
|
def format_memory_for_injection(memory_data: dict[str, Any], max_tokens: int = 2000) -> str:
|
|
```
|
|
|
|
Current injection format:
|
|
- `User Context` section from `user.*.summary`
|
|
- `History` section from `history.*.summary`
|
|
- `Facts` section from `facts[]`, sorted by confidence, appended until token budget is reached
|
|
|
|
Token counting:
|
|
- Uses `tiktoken` (`cl100k_base`) when available
|
|
- Falls back to a network-free CJK-aware character estimate if tokenizer import or encoding load fails
|
|
(CJK characters count as ~2 chars/token, other characters as ~4 chars/token)
|
|
|
|
## Known Gap
|
|
|
|
Previous versions of this document described TF-IDF/context-aware retrieval as if it were already shipped.
|
|
That was not accurate for `main` and caused confusion.
|
|
|
|
Issue reference: `#1059`
|
|
|
|
## Roadmap (Planned)
|
|
|
|
Planned scoring strategy:
|
|
|
|
```text
|
|
final_score = (similarity * 0.6) + (confidence * 0.4)
|
|
```
|
|
|
|
Planned integration shape:
|
|
1. Extract recent conversational context from filtered user/final-assistant turns.
|
|
2. Compute TF-IDF cosine similarity between each fact and current context.
|
|
3. Rank by weighted score and inject under token budget.
|
|
4. Fall back to confidence-only ranking if context is unavailable.
|
|
|
|
## Validation
|
|
|
|
Current regression coverage includes:
|
|
- facts inclusion in memory injection output
|
|
- confidence ordering
|
|
- token-budget-limited fact inclusion
|
|
|
|
Tests:
|
|
- `backend/tests/test_memory_prompt_injection.py`
|