mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-06-10 17:35:57 +00:00
test(e2e): deterministic record/replay front-back contract verification (#3365)
* test(e2e): record/replay front-back contract verification Guards the front-back contract with a deterministic, key-free record/replay harness (mirrors open-design's golden-trace approach): - ReplayChatModel (tests/replay_provider.py): replays recorded LLM turns by a normalized hash of the model input. Strips <system-reminder>/date/uuid/tmp-path so one fixture replays across days and from both the browser and direct-POST paths; a miss raises loudly (no silent divergence). - Recording is record-through-browser (scripts/record_gateway.py + build_fixture_from_jsonl.py + frontend/tests/e2e-record): a real run is driven through the real frontend so captured inputs match exactly what the browser sends; fixtures contain no API key. - Layer 1 — backend golden (tests/test_replay_golden.py): replay through the real gateway, assert the SSE event sequence == committed golden. - Layer 2 — full-stack render (frontend/tests/e2e-real-backend): real Next.js + real gateway (replay model) + Chromium; assert the replayed auto-title and follow-up suggestions render. DOM assertions are the gate; visual regression is a local dev gate (CI uploads the render as an artifact). - CI (.github/workflows/replay-e2e.yml): both layers, triggered on EITHER side of the contract (frontend/** or backend gateway/harness/fixtures). * test(e2e): multi-run render-order cross-stack scenario (#3352) Guards the dangerous front-back class where a backend ordering change silently breaks a frontend assumption while both sides' unit tests stay green. Reproduces issue #3352: backend list_by_thread returns runs newest-first (#2932) and the frontend prepended per-run pages, inverting chronological order once the checkpoint no longer held the older messages. - tests/seed_runs_router.py: test-only seeder, mounted on the replay gateway only when DEERFLOW_ENABLE_TEST_SEED=1 (never in the production app). Seeds a thread with >=2 runs + per-run message events and no checkpoint -- the #3352 precondition -- so the frontend per-run reload path is the sole source of truth and the prepend inversion is observable. - frontend/tests/e2e-real-backend/multi-run-order.spec.ts: drives the real frontend against the real gateway, asserts the first run renders above the second. Reverting the #3354 fix turns it red. - replay-e2e.yml: trigger on the new replay test-infra paths. - docs: REPLAY_E2E.md cross-stack scenario section. * test(e2e): address Copilot review on the replay harness - Fix stale recorder references (scripts/record_traces.py -> scripts/record_gateway.py + scripts/build_fixture_from_jsonl.py) in replay_provider.py, test_replay_golden.py, _replay_fixture.py. - MODE_CONTEXT['ultra']: thinking_enabled False -> True, mirroring the frontend's `context.mode !== 'flash'` (hooks.ts). It did not affect the hashed input (Layer 1 golden still green), but the table now matches the real frontend context it claims to mirror. - replay_provider.py docstring: stop claiming memory is recorded-enabled; the replay config disables memory/summarization for determinism (title stays, as an in-graph deterministic call). - record_gateway.py / run_replay_gateway.py: override DEER_FLOW_HOME instead of setdefault, so an outer value can't leak into the hermetic harness. - record_gateway.py: clear error when DEERFLOW_RECORD_OUT is unset (was a bare KeyError). - playwright.record.config.ts: forward OPENAI_*/DEERFLOW_RECORD_OUT only when set, so the gateway raises a clear 'missing env' error instead of getting ''. * test(e2e): address Copilot review round 2 - seed_runs_router.py: constrain SeedMessage.role to Literal['human','ai'] so a bad value is a clean 422 at the boundary instead of a 500 (KeyError on _EVENT_TYPE). - record-write-read-file.spec.ts: waitForCaptureStable now throws on timeout instead of returning the last count, so a truncated/partial recording can't pass silently. - real-backend-render.spec.ts: guard the suggestions JSON.parse; a bracket-prefixed non-JSON turn falls back to '' so the existing not.toBe('') assertion fails clearly instead of a generic parse throw.
This commit is contained in:
Xinmin Zeng
@@ -0,0 +1,163 @@
|
||||
"""Shared config + gateway-drive helpers for the record/replay e2e.
|
||||
|
||||
Record (``scripts/record_gateway.py`` + ``scripts/build_fixture_from_jsonl.py``)
|
||||
and replay (``tests/test_replay_golden.py``)
|
||||
MUST drive the gateway through an identical, prompt-affecting config — otherwise
|
||||
the system prompt differs and the recorded input hashes never match on replay.
|
||||
Centralising the config builder + drive loop here makes that identity hold by
|
||||
construction; only the ``models[].use`` block differs (real model vs
|
||||
``ReplayChatModel``).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
|
||||
# mode -> (thinking_enabled, is_plan_mode, subagent_enabled). Mirrors the
|
||||
# frontend mapping in core/threads/hooks.ts.
|
||||
MODE_CONTEXT: dict[str, tuple[bool, bool, bool]] = {
|
||||
"flash": (False, False, False),
|
||||
"thinking": (True, False, False),
|
||||
"pro": (True, True, False),
|
||||
# thinking_enabled mirrors the frontend `context.mode !== "flash"` (hooks.ts),
|
||||
# so ultra is thinking-enabled too.
|
||||
"ultra": (True, True, True),
|
||||
}
|
||||
|
||||
# The replay model block: same model NAME as recording (so nothing in the prompt
|
||||
# shifts), only ``use`` swapped to the deterministic replay provider.
|
||||
REPLAY_MODEL_BLOCK = """\
|
||||
- name: scenario-model
|
||||
display_name: Scenario Model
|
||||
use: replay_provider:ReplayChatModel
|
||||
model: replay"""
|
||||
|
||||
|
||||
def real_model_block(model: str) -> str:
|
||||
return f"""\
|
||||
- name: scenario-model
|
||||
display_name: Scenario Model
|
||||
use: langchain_openai:ChatOpenAI
|
||||
model: {model}
|
||||
api_key: $OPENAI_API_KEY
|
||||
base_url: $OPENAI_API_BASE"""
|
||||
|
||||
|
||||
def build_config_yaml(*, model_block: str, home: Path) -> str:
|
||||
"""Full gateway config. Only ``model_block`` varies between record/replay.
|
||||
|
||||
Everything that shapes the system prompt is pinned so record, replay, and CI
|
||||
produce byte-identical prompts regardless of the machine:
|
||||
- sandbox / tool_groups / tools — fixed here
|
||||
- skills — pointed at an empty ``<home>/skills`` so filesystem skills (incl.
|
||||
gitignored custom skills present only on a dev box) never leak into the
|
||||
prompt. Pair with an empty ``extensions_config.json`` (no MCP) via
|
||||
:func:`prepare_hermetic_extras`.
|
||||
- memory / summarization — disabled (background, non-deterministic timing)
|
||||
"""
|
||||
return f"""\
|
||||
log_level: warning
|
||||
models:
|
||||
{model_block}
|
||||
sandbox:
|
||||
use: deerflow.sandbox.local:LocalSandboxProvider
|
||||
skills:
|
||||
path: {home / "skills"}
|
||||
container_path: /mnt/skills
|
||||
tool_groups:
|
||||
- name: file:read
|
||||
- name: file:write
|
||||
tools:
|
||||
- name: ls
|
||||
group: file:read
|
||||
use: deerflow.sandbox.tools:ls_tool
|
||||
- name: read_file
|
||||
group: file:read
|
||||
use: deerflow.sandbox.tools:read_file_tool
|
||||
- name: write_file
|
||||
group: file:write
|
||||
use: deerflow.sandbox.tools:write_file_tool
|
||||
# Memory + summarization make background / debounced model calls whose timing is
|
||||
# non-deterministic; disable them so record and replay see the same model-call
|
||||
# set. (Title stays — it is an in-graph, deterministic call we record.)
|
||||
memory:
|
||||
enabled: false
|
||||
injection_enabled: false
|
||||
summarization:
|
||||
enabled: false
|
||||
agents_api:
|
||||
enabled: true
|
||||
database:
|
||||
backend: sqlite
|
||||
sqlite_dir: {home / "db"}
|
||||
"""
|
||||
|
||||
|
||||
def prepare_hermetic_extras(home: Path) -> Path:
|
||||
"""Create the empty skills tree + an empty extensions_config.json so the
|
||||
system prompt has no environment-dependent skills/MCP content.
|
||||
|
||||
Returns the extensions-config path; the caller must point
|
||||
``DEER_FLOW_EXTENSIONS_CONFIG_PATH`` at it. Call before starting the gateway.
|
||||
"""
|
||||
(home / "skills" / "public").mkdir(parents=True, exist_ok=True)
|
||||
(home / "skills" / "custom").mkdir(parents=True, exist_ok=True)
|
||||
extensions = home / "extensions_config.json"
|
||||
extensions.write_text(json.dumps({"mcpServers": {}, "skills": {}}), encoding="utf-8")
|
||||
return extensions
|
||||
|
||||
|
||||
def sse_event_shapes(resp) -> list[dict]:
|
||||
"""Reduce an SSE stream to (event name, sorted top-level data keys).
|
||||
|
||||
Snapshots the *shape* of the stream, not volatile values, so the golden is
|
||||
stable across runs while still catching event-sequence / payload-shape drift.
|
||||
"""
|
||||
events: list[dict] = []
|
||||
current: str | None = None
|
||||
for line in resp.iter_lines():
|
||||
if line.startswith("event:"):
|
||||
current = line[len("event:") :].strip()
|
||||
elif line.startswith("data:"):
|
||||
raw = line[len("data:") :].strip()
|
||||
try:
|
||||
data = json.loads(raw) if raw else {}
|
||||
except json.JSONDecodeError:
|
||||
data = {"_raw": raw[:200]}
|
||||
events.append({"event": current, "keys": sorted(data.keys()) if isinstance(data, dict) else None})
|
||||
return events
|
||||
|
||||
|
||||
def drive_gateway(app, *, prompt: str, context: dict) -> list[dict]:
|
||||
"""Register -> create thread -> POST /runs/stream; return SSE event shapes.
|
||||
|
||||
This is the exact wire path the React frontend uses (LangGraph SDK), driven
|
||||
in-process via Starlette's TestClient with the real auth flow.
|
||||
"""
|
||||
from starlette.testclient import TestClient
|
||||
|
||||
with TestClient(app) as client:
|
||||
reg = client.post(
|
||||
"/api/v1/auth/register",
|
||||
json={"email": f"e2e-{uuid.uuid4().hex[:8]}@example.com", "password": "very-strong-password-123"},
|
||||
)
|
||||
assert reg.status_code == 201, reg.text
|
||||
csrf = client.cookies.get("csrf_token")
|
||||
assert csrf, "register must set csrf_token cookie"
|
||||
|
||||
thread_id = str(uuid.uuid4())
|
||||
created = client.post("/api/threads", json={"thread_id": thread_id, "metadata": {}}, headers={"X-CSRF-Token": csrf})
|
||||
assert created.status_code == 200, created.text
|
||||
|
||||
body = {
|
||||
"assistant_id": "lead_agent",
|
||||
"input": {"messages": [{"role": "user", "content": prompt}]},
|
||||
"config": {"recursion_limit": 50},
|
||||
"context": context,
|
||||
"stream_mode": ["values"],
|
||||
}
|
||||
with client.stream("POST", f"/api/threads/{thread_id}/runs/stream", json=body, headers={"X-CSRF-Token": csrf}) as resp:
|
||||
assert resp.status_code == 200, resp.read().decode()
|
||||
return sse_event_shapes(resp)
|
||||
@@ -0,0 +1,72 @@
|
||||
{
|
||||
"scenario": "write_read_file",
|
||||
"mode": "ultra",
|
||||
"events": [
|
||||
{
|
||||
"event": "metadata",
|
||||
"keys": [
|
||||
"run_id",
|
||||
"thread_id"
|
||||
]
|
||||
},
|
||||
{
|
||||
"event": "values",
|
||||
"keys": [
|
||||
"artifacts",
|
||||
"messages",
|
||||
"viewed_images"
|
||||
]
|
||||
},
|
||||
{
|
||||
"event": "values",
|
||||
"keys": [
|
||||
"artifacts",
|
||||
"messages",
|
||||
"thread_data",
|
||||
"viewed_images"
|
||||
]
|
||||
},
|
||||
{
|
||||
"event": "values",
|
||||
"keys": [
|
||||
"artifacts",
|
||||
"messages",
|
||||
"thread_data",
|
||||
"viewed_images"
|
||||
]
|
||||
},
|
||||
{
|
||||
"event": "values",
|
||||
"keys": [
|
||||
"artifacts",
|
||||
"messages",
|
||||
"thread_data",
|
||||
"viewed_images"
|
||||
]
|
||||
},
|
||||
{
|
||||
"event": "values",
|
||||
"keys": [
|
||||
"artifacts",
|
||||
"messages",
|
||||
"thread_data",
|
||||
"title",
|
||||
"viewed_images"
|
||||
]
|
||||
},
|
||||
{
|
||||
"event": "values",
|
||||
"keys": [
|
||||
"artifacts",
|
||||
"messages",
|
||||
"thread_data",
|
||||
"title",
|
||||
"viewed_images"
|
||||
]
|
||||
},
|
||||
{
|
||||
"event": "end",
|
||||
"keys": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,224 @@
|
||||
{
|
||||
"scenario": "write_read_file",
|
||||
"mode": "ultra",
|
||||
"model": "gpt-5.5",
|
||||
"prompt": "Using your own file tools directly, create the file /mnt/user-data/outputs/note.txt with exactly this content: hi from replay. Then read that same file back and reply with its exact contents. Do NOT delegate to a subagent and do NOT use the task tool — do it yourself. Do not ask any clarifying questions.",
|
||||
"context": {
|
||||
"is_bootstrap": false,
|
||||
"mode": "ultra",
|
||||
"thinking_enabled": true,
|
||||
"is_plan_mode": true,
|
||||
"subagent_enabled": true
|
||||
},
|
||||
"turns": [
|
||||
{
|
||||
"input_hash": "686cd44a9f17fadc0398768731324f3980480a027593a475fad4583581df677f",
|
||||
"output": {
|
||||
"type": "ai",
|
||||
"data": {
|
||||
"content": "",
|
||||
"additional_kwargs": {},
|
||||
"response_metadata": {
|
||||
"finish_reason": "tool_calls",
|
||||
"model_name": "gpt-5.5",
|
||||
"model_provider": "openai"
|
||||
},
|
||||
"type": "ai",
|
||||
"name": null,
|
||||
"id": "lc_run--019e8c60-8d4b-79a1-8d77-0a67fc360ce4",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "write_file",
|
||||
"args": {
|
||||
"description": "Create requested note file",
|
||||
"path": "/mnt/user-data/outputs/note.txt",
|
||||
"content": "hi from replay"
|
||||
},
|
||||
"id": "call_UdIzq5Vyx7pu1Usnj4wPCC6G",
|
||||
"type": "tool_call"
|
||||
}
|
||||
],
|
||||
"invalid_tool_calls": [],
|
||||
"usage_metadata": {
|
||||
"input_tokens": 3285,
|
||||
"output_tokens": 66,
|
||||
"total_tokens": 3351,
|
||||
"input_token_details": {
|
||||
"audio": 0,
|
||||
"cache_read": 0
|
||||
},
|
||||
"output_token_details": {
|
||||
"audio": 0,
|
||||
"reasoning": 21
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"input_hash": "3598aeb87e221ca8f554e4d61ce6d5e8801754606fa5c95a89c38bd6cb623045",
|
||||
"output": {
|
||||
"type": "ai",
|
||||
"data": {
|
||||
"content": "File Creation and Verification",
|
||||
"additional_kwargs": {},
|
||||
"response_metadata": {
|
||||
"finish_reason": "stop",
|
||||
"model_name": "gpt-5.5",
|
||||
"model_provider": "openai"
|
||||
},
|
||||
"type": "ai",
|
||||
"name": null,
|
||||
"id": "lc_run--019e8c60-9c18-72c1-95e8-f6a240747395",
|
||||
"tool_calls": [],
|
||||
"invalid_tool_calls": [],
|
||||
"usage_metadata": {
|
||||
"input_tokens": 104,
|
||||
"output_tokens": 53,
|
||||
"total_tokens": 157,
|
||||
"input_token_details": {
|
||||
"audio": 0,
|
||||
"cache_read": 0
|
||||
},
|
||||
"output_token_details": {
|
||||
"audio": 0,
|
||||
"reasoning": 39
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"input_hash": "92430ba866abe577c86d2e67eb5158b10f3f19ec306aa9de235bb06736320d70",
|
||||
"output": {
|
||||
"type": "ai",
|
||||
"data": {
|
||||
"content": "",
|
||||
"additional_kwargs": {},
|
||||
"response_metadata": {
|
||||
"finish_reason": "tool_calls",
|
||||
"model_name": "gpt-5.5",
|
||||
"model_provider": "openai"
|
||||
},
|
||||
"type": "ai",
|
||||
"name": null,
|
||||
"id": "lc_run--019e8c60-b036-7710-8db9-717ab54e5805",
|
||||
"tool_calls": [
|
||||
{
|
||||
"name": "read_file",
|
||||
"args": {
|
||||
"description": "Read requested note file",
|
||||
"path": "/mnt/user-data/outputs/note.txt"
|
||||
},
|
||||
"id": "call_0BFNns0FkRb3n2LR0PRrfbIJ",
|
||||
"type": "tool_call"
|
||||
}
|
||||
],
|
||||
"invalid_tool_calls": [],
|
||||
"usage_metadata": {
|
||||
"input_tokens": 3334,
|
||||
"output_tokens": 33,
|
||||
"total_tokens": 3367,
|
||||
"input_token_details": {
|
||||
"audio": 0,
|
||||
"cache_read": 0
|
||||
},
|
||||
"output_token_details": {
|
||||
"audio": 0,
|
||||
"reasoning": 0
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"input_hash": "8ab757aa51f9d556adcea07c0221445a2b791cc882ef11922babf7f2865d1913",
|
||||
"output": {
|
||||
"type": "ai",
|
||||
"data": {
|
||||
"content": "hi from replay",
|
||||
"additional_kwargs": {},
|
||||
"response_metadata": {
|
||||
"finish_reason": "stop",
|
||||
"model_name": "gpt-5.5",
|
||||
"model_provider": "openai"
|
||||
},
|
||||
"type": "ai",
|
||||
"name": null,
|
||||
"id": "lc_run--019e8c60-bef3-7201-a30a-cbc5f45920ba",
|
||||
"tool_calls": [],
|
||||
"invalid_tool_calls": [],
|
||||
"usage_metadata": {
|
||||
"input_tokens": 3380,
|
||||
"output_tokens": 7,
|
||||
"total_tokens": 3387,
|
||||
"input_token_details": {
|
||||
"audio": 0,
|
||||
"cache_read": 0
|
||||
},
|
||||
"output_token_details": {
|
||||
"audio": 0,
|
||||
"reasoning": 0
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"input_hash": "fd67723cc8810ce79b4785fec4c251a272a91d677a216c735b23b5f6d3dec0c3",
|
||||
"output": {
|
||||
"type": "ai",
|
||||
"data": {
|
||||
"content": "[\"Can you append another line to the file?\",\"Can you show the file path again?\",\"Can you delete the file now?\"]",
|
||||
"additional_kwargs": {
|
||||
"refusal": null
|
||||
},
|
||||
"response_metadata": {
|
||||
"token_usage": {
|
||||
"completion_tokens": 71,
|
||||
"prompt_tokens": 224,
|
||||
"total_tokens": 295,
|
||||
"completion_tokens_details": {
|
||||
"accepted_prediction_tokens": 0,
|
||||
"audio_tokens": 0,
|
||||
"reasoning_tokens": 33,
|
||||
"rejected_prediction_tokens": 0
|
||||
},
|
||||
"prompt_tokens_details": {
|
||||
"audio_tokens": 0,
|
||||
"cached_tokens": 0
|
||||
},
|
||||
"input_tokens": 0,
|
||||
"output_tokens": 0,
|
||||
"input_tokens_details": null
|
||||
},
|
||||
"model_provider": "openai",
|
||||
"model_name": "gpt-5.5",
|
||||
"system_fingerprint": null,
|
||||
"id": "chatcmpl-DmaI5yVqQ39LRWyugoCEPalKw0gBR",
|
||||
"finish_reason": "stop",
|
||||
"logprobs": null
|
||||
},
|
||||
"type": "ai",
|
||||
"name": null,
|
||||
"id": "lc_run--019e8c60-d025-7fd2-9cc9-8b4fb8fe1a82-0",
|
||||
"tool_calls": [],
|
||||
"invalid_tool_calls": [],
|
||||
"usage_metadata": {
|
||||
"input_tokens": 224,
|
||||
"output_tokens": 71,
|
||||
"total_tokens": 295,
|
||||
"input_token_details": {
|
||||
"audio": 0,
|
||||
"cache_read": 0
|
||||
},
|
||||
"output_token_details": {
|
||||
"audio": 0,
|
||||
"reasoning": 33
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,230 @@
|
||||
"""Replay a recorded LLM trace deterministically — the "replay" half of
|
||||
record/replay e2e (mirrors open-design's ``mocks/`` golden traces).
|
||||
|
||||
A fixture is a JSON file capturing the *real* model calls of one scenario,
|
||||
keyed by a normalized hash of the **input** each call received::
|
||||
|
||||
{
|
||||
"scenario": "write_read_file",
|
||||
"mode": "ultra",
|
||||
"model": "gpt-5.5",
|
||||
"turns": [
|
||||
{"input_hash": "<sha256>", "input_preview": "...", "output": <message dict>},
|
||||
...
|
||||
]
|
||||
}
|
||||
|
||||
Why hash-by-input (not turn index)
|
||||
----------------------------------
|
||||
A real run makes model calls from several callers — the lead agent's own turns,
|
||||
``TitleMiddleware`` (auto-title), memory, and possibly subagents. They interleave
|
||||
and their count/order is not something we want a replay to depend on. Matching by
|
||||
a normalized hash of the *input messages* means each call gets back exactly the
|
||||
output that was recorded for that input, regardless of order or which middleware
|
||||
issued it. That keeps the in-graph, deterministic title call part of the
|
||||
recording; memory/summarization, by contrast, are disabled in the replay config
|
||||
(``_replay_fixture.py``) because their background, debounced timing is not
|
||||
reproducible across runs.
|
||||
|
||||
Volatile fields (UUID thread/run/user ids, timestamps, dates, tmp/home paths)
|
||||
are normalized out before hashing so a recording replays across processes with
|
||||
different temp dirs. The same ``hash_messages`` is used by the recorder
|
||||
(``scripts/record_gateway.py``) and here, so record and replay agree by
|
||||
construction.
|
||||
|
||||
This lives in ``tests/`` (not in the publishable ``deerflow-harness`` package),
|
||||
matching the repo convention for test-only fakes (cf. ``FakeToolCallingModel`` in
|
||||
``_agent_e2e_helpers.py``). In-process tests get ``tests/`` on ``sys.path`` for
|
||||
free via pytest; a standalone replay gateway just needs ``PYTHONPATH`` to include
|
||||
``backend/tests`` so the config ``use:`` below resolves.
|
||||
|
||||
Point a config model's ``use`` at this class and set the fixture via env::
|
||||
|
||||
models:
|
||||
- name: replay-model
|
||||
use: replay_provider:ReplayChatModel
|
||||
model: gpt-5.5 # placeholder; ignored
|
||||
|
||||
DEERFLOW_REPLAY_FIXTURE=/path/to/write_read_file.ultra.json
|
||||
|
||||
A cache miss raises loudly with a diagnostic — that is the signal that the
|
||||
replayed run diverged from the recording (graph changed, a new volatile field
|
||||
slipped through normalization, or a non-deterministic tool result changed a
|
||||
downstream input). Re-record or extend normalization; never pass silently.
|
||||
|
||||
Recording lives outside production code too (``scripts/record_gateway.py`` +
|
||||
``scripts/build_fixture_from_jsonl.py``); CI consumes the fixtures through this
|
||||
replay side with no API key.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
from collections import deque
|
||||
from collections.abc import Iterator
|
||||
from typing import Any
|
||||
|
||||
from langchain_core.callbacks import CallbackManagerForLLMRun
|
||||
from langchain_core.language_models.chat_models import BaseChatModel
|
||||
from langchain_core.messages import AIMessage, AIMessageChunk, BaseMessage, messages_from_dict
|
||||
from langchain_core.outputs import ChatGeneration, ChatGenerationChunk, ChatResult
|
||||
from langchain_core.runnables import Runnable
|
||||
from pydantic import PrivateAttr
|
||||
|
||||
_FIXTURE_ENV = "DEERFLOW_REPLAY_FIXTURE"
|
||||
|
||||
# Volatile substrings that differ between a recording run and a replay run but
|
||||
# carry no semantic weight for matching. Normalized to stable placeholders
|
||||
# before hashing so the same logical input hashes identically across processes.
|
||||
# The frontend injects a per-request ``<system-reminder>`` (current date, weekday,
|
||||
# dynamic context) that the backend-direct path does not — and its date/weekday
|
||||
# change every day. Strip the whole block before hashing so a fixture replays
|
||||
# (a) across days and (b) from both the browser and direct-POST paths.
|
||||
_SYSTEM_REMINDER_RE = re.compile(r"<system-reminder>.*?</system-reminder>", re.DOTALL)
|
||||
_UUID_RE = re.compile(r"[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}")
|
||||
_ISO_TS_RE = re.compile(r"\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:?\d{2})?")
|
||||
_DATE_RE = re.compile(r"\d{4}-\d{2}-\d{2}")
|
||||
# Absolute temp/home roots used for per-run isolation (macOS + Linux + DEER_FLOW_HOME tmp).
|
||||
_PATH_RE = re.compile(r"(?:/private)?/(?:var/folders|tmp)/[^\s\"']*")
|
||||
|
||||
|
||||
def _normalize_text(text: str) -> str:
|
||||
text = _SYSTEM_REMINDER_RE.sub("", text)
|
||||
text = _UUID_RE.sub("<UUID>", text)
|
||||
text = _ISO_TS_RE.sub("<TS>", text)
|
||||
text = _DATE_RE.sub("<DATE>", text)
|
||||
text = _PATH_RE.sub("<PATH>", text)
|
||||
return text
|
||||
|
||||
|
||||
def _content_to_text(content: Any) -> str:
|
||||
if isinstance(content, str):
|
||||
return content
|
||||
if isinstance(content, list):
|
||||
parts: list[str] = []
|
||||
for block in content:
|
||||
if isinstance(block, dict):
|
||||
parts.append(block.get("text", "") or json.dumps(block, sort_keys=True, ensure_ascii=False))
|
||||
else:
|
||||
parts.append(str(block))
|
||||
return "".join(parts)
|
||||
return str(content)
|
||||
|
||||
|
||||
def _canonical_messages(messages: list[BaseMessage]) -> str:
|
||||
"""Project messages to a stable shape that excludes volatile metadata/ids.
|
||||
|
||||
Keeps only what determines the model's next output: role, text content, and
|
||||
tool-call name+args. Drops ``id``, ``response_metadata``, ``usage_metadata``,
|
||||
and ``tool_call_id`` (all volatile), then normalizes embedded volatile
|
||||
substrings.
|
||||
"""
|
||||
projected: list[dict[str, Any]] = []
|
||||
for message in messages:
|
||||
content = _normalize_text(_content_to_text(message.content))
|
||||
tool_calls = getattr(message, "tool_calls", None)
|
||||
# Drop messages that are empty after normalization — e.g. a turn that was
|
||||
# nothing but a frontend-injected <system-reminder>. They carry no
|
||||
# decision-relevant content and differ between client paths.
|
||||
if not content.strip() and not tool_calls:
|
||||
continue
|
||||
entry: dict[str, Any] = {"type": message.type, "content": content}
|
||||
if tool_calls:
|
||||
entry["tool_calls"] = [{"name": tc.get("name"), "args": tc.get("args")} for tc in tool_calls]
|
||||
name = getattr(message, "name", None)
|
||||
if name:
|
||||
entry["name"] = name
|
||||
projected.append(entry)
|
||||
raw = json.dumps(projected, sort_keys=True, ensure_ascii=False)
|
||||
return _normalize_text(raw)
|
||||
|
||||
|
||||
def hash_messages(messages: list[BaseMessage]) -> str:
|
||||
"""Stable hash of a model call's input. Shared by recorder and replayer."""
|
||||
return hashlib.sha256(_canonical_messages(messages).encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
def _load_fixture(fixture_path: str) -> dict[str, deque[AIMessage]]:
|
||||
with open(fixture_path, encoding="utf-8") as handle:
|
||||
payload = json.load(handle)
|
||||
table: dict[str, deque[AIMessage]] = {}
|
||||
for index, turn in enumerate(payload.get("turns", [])):
|
||||
input_hash = turn["input_hash"]
|
||||
(message,) = messages_from_dict([turn["output"]])
|
||||
if not isinstance(message, AIMessage):
|
||||
raise ValueError(f"replay fixture {fixture_path!r} turn {index} output is {type(message).__name__}, expected AIMessage")
|
||||
table.setdefault(input_hash, deque()).append(message)
|
||||
return table
|
||||
|
||||
|
||||
class ReplayChatModel(BaseChatModel):
|
||||
"""Returns the recorded assistant output whose input matches this call.
|
||||
|
||||
``bind_tools`` is a no-op returning ``self`` — recorded turns already carry
|
||||
the real ``tool_calls``, so the agent dispatches them as if a live model had
|
||||
produced them.
|
||||
"""
|
||||
|
||||
_table: dict[str, deque] = PrivateAttr(default_factory=dict)
|
||||
_fixture_path: str = PrivateAttr(default="")
|
||||
|
||||
def __init__(self, **kwargs: Any) -> None:
|
||||
# Ignore provider noise the factory forwards from config (model, api_key,
|
||||
# base_url, ...). Fixture path comes from the ``fixture`` kwarg or env.
|
||||
fixture_path = kwargs.pop("fixture", None) or os.environ.get(_FIXTURE_ENV)
|
||||
super().__init__()
|
||||
if not fixture_path:
|
||||
raise ValueError(f"ReplayChatModel needs a fixture path via the ``fixture`` kwarg or ${_FIXTURE_ENV}")
|
||||
self._fixture_path = fixture_path
|
||||
self._table = _load_fixture(fixture_path)
|
||||
|
||||
@property
|
||||
def _llm_type(self) -> str:
|
||||
return "deerflow-replay"
|
||||
|
||||
def _match(self, messages: list[BaseMessage]) -> AIMessage:
|
||||
key = hash_messages(messages)
|
||||
bucket = self._table.get(key)
|
||||
if not bucket:
|
||||
preview = _canonical_messages(messages)
|
||||
raise KeyError(
|
||||
f"replay miss: no recorded output for input hash {key} in {self._fixture_path!r}. "
|
||||
"The replayed run diverged from the recording (graph changed, a non-deterministic tool result "
|
||||
"altered a downstream input, or a volatile field slipped past normalization). "
|
||||
f"Known hashes: {sorted(self._table)}. "
|
||||
f"Normalized input (first 800 chars): {preview[:800]!r}"
|
||||
)
|
||||
return bucket.popleft()
|
||||
|
||||
def _generate(
|
||||
self,
|
||||
messages: list[BaseMessage],
|
||||
stop: list[str] | None = None,
|
||||
run_manager: CallbackManagerForLLMRun | None = None,
|
||||
**kwargs: Any,
|
||||
) -> ChatResult:
|
||||
return ChatResult(generations=[ChatGeneration(message=self._match(messages))])
|
||||
|
||||
def _stream(
|
||||
self,
|
||||
messages: list[BaseMessage],
|
||||
stop: list[str] | None = None,
|
||||
run_manager: CallbackManagerForLLMRun | None = None,
|
||||
**kwargs: Any,
|
||||
) -> Iterator[ChatGenerationChunk]:
|
||||
turn = self._match(messages)
|
||||
text = turn.content if isinstance(turn.content, str) else ""
|
||||
chunk = ChatGenerationChunk(message=AIMessageChunk(content=turn.content, tool_calls=turn.tool_calls, additional_kwargs=turn.additional_kwargs, id=turn.id))
|
||||
if run_manager is not None and text:
|
||||
run_manager.on_llm_new_token(text, chunk=chunk)
|
||||
yield chunk
|
||||
|
||||
def bind_tools(self, tools: Any, **kwargs: Any) -> Runnable: # type: ignore[override]
|
||||
return self
|
||||
|
||||
|
||||
# Re-export so the recorder shares the exact hashing logic.
|
||||
__all__ = ["ReplayChatModel", "hash_messages"]
|
||||
@@ -0,0 +1,100 @@
|
||||
"""Test-only run/message seeder for the multi-run render-order e2e (issue #3352).
|
||||
|
||||
Mounted **only** by ``scripts/run_replay_gateway.py`` (the replay e2e gateway)
|
||||
and never by the production app, so it cannot ship. It lets a Playwright spec
|
||||
stand up a thread with >=2 runs whose per-run messages exercise the frontend's
|
||||
reload / history-rebuild ordering path — with no real model, no recording, and
|
||||
no API key.
|
||||
|
||||
Why a seeder instead of recording a conversation: issue #3352 only reproduces
|
||||
when the checkpoint no longer holds the older messages (post-compression), so
|
||||
the frontend rebuilds them from the per-run history endpoints. A seeder lets us
|
||||
create exactly that precondition deterministically — runs in the run store +
|
||||
per-run ``category="message"`` events, and **no checkpoint** — so on reload the
|
||||
buggy ``findLatestUnloadedRunIndex`` + prepend in ``core/threads/hooks.ts`` is
|
||||
the sole source of truth and its reversed order becomes observable.
|
||||
|
||||
It writes through the gateway's OWN ``app.state.run_store`` +
|
||||
``app.state.run_event_store`` using the request's auth context, so the seeded
|
||||
``user_id`` matches the browser session that reads it back. The event shape
|
||||
mirrors exactly what ``runtime/journal.py`` writes for real runs
|
||||
(``event_type`` ``llm.human.input`` / ``llm.ai.response``, ``category``
|
||||
``"message"``, ``content`` = ``message.model_dump()``, ``metadata.caller`` =
|
||||
``"lead_agent"``).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Literal
|
||||
|
||||
from fastapi import APIRouter, Request
|
||||
from pydantic import BaseModel
|
||||
|
||||
router = APIRouter(prefix="/api/test-only", tags=["test-only"])
|
||||
|
||||
# Mirror runtime/journal.py: human prompts are recorded as ``llm.human.input``
|
||||
# and assistant turns as ``llm.ai.response``; both land in ``category="message"``.
|
||||
_EVENT_TYPE = {"human": "llm.human.input", "ai": "llm.ai.response"}
|
||||
|
||||
|
||||
class SeedMessage(BaseModel):
|
||||
role: Literal["human", "ai"]
|
||||
content: str
|
||||
id: str
|
||||
|
||||
|
||||
class SeedRun(BaseModel):
|
||||
run_id: str
|
||||
# ISO timestamp; RunManager.list_by_thread sorts newest-first by created_at,
|
||||
# so a later created_at must mean a later run for the ordering to be faithful.
|
||||
created_at: str
|
||||
messages: list[SeedMessage]
|
||||
|
||||
|
||||
class SeedRunsBody(BaseModel):
|
||||
thread_id: str
|
||||
runs: list[SeedRun]
|
||||
|
||||
|
||||
@router.post("/seed-runs")
|
||||
async def seed_runs(body: SeedRunsBody, request: Request) -> dict:
|
||||
"""Seed runs + per-run message events for the authenticated user.
|
||||
|
||||
No checkpoint is written: that is the whole point — it forces the frontend's
|
||||
reload path to rebuild history from the per-run endpoints (the #3352 bug
|
||||
site) instead of the (correctly ordered) checkpoint snapshot.
|
||||
"""
|
||||
from langchain_core.messages import AIMessage, HumanMessage
|
||||
|
||||
run_store = request.app.state.run_store
|
||||
event_store = request.app.state.run_event_store
|
||||
|
||||
for run in body.runs:
|
||||
# user_id defaults (AUTO) to the request's auth context, matching the
|
||||
# browser session that will read these runs back via GET /runs.
|
||||
await run_store.put(
|
||||
run.run_id,
|
||||
thread_id=body.thread_id,
|
||||
assistant_id="lead_agent",
|
||||
status="success",
|
||||
created_at=run.created_at,
|
||||
)
|
||||
events = []
|
||||
for m in run.messages:
|
||||
msg = (HumanMessage if m.role == "human" else AIMessage)(content=m.content, id=m.id)
|
||||
events.append(
|
||||
{
|
||||
"thread_id": body.thread_id,
|
||||
"run_id": run.run_id,
|
||||
"event_type": _EVENT_TYPE[m.role],
|
||||
"category": "message",
|
||||
"content": msg.model_dump(),
|
||||
"metadata": {"caller": "lead_agent"},
|
||||
"created_at": run.created_at,
|
||||
}
|
||||
)
|
||||
# One batch per run so seq is monotonic and run1's messages precede
|
||||
# run2's; the gateway reads them back per-run anyway.
|
||||
await event_store.put_batch(events)
|
||||
|
||||
return {"ok": True, "thread_id": body.thread_id, "runs": len(body.runs)}
|
||||
@@ -0,0 +1,87 @@
|
||||
"""Layer 1 of the record/replay e2e: replay a recorded trace through the **real
|
||||
gateway** with a deterministic ``ReplayChatModel`` (no API key, no network) and
|
||||
assert the streamed SSE event sequence matches a committed golden.
|
||||
|
||||
This catches backend protocol drift: if a change alters the shape/sequence of
|
||||
SSE the gateway emits for the recorded scenario, this test goes red. The replay
|
||||
model serves the recorded assistant turns by input hash, so the agent graph
|
||||
(write_file -> auto-title -> read_file -> final answer) reproduces offline.
|
||||
|
||||
Fixtures are produced by ``scripts/record_gateway.py`` +
|
||||
``scripts/build_fixture_from_jsonl.py`` (manual, needs a key).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from _replay_fixture import REPLAY_MODEL_BLOCK, build_config_yaml, drive_gateway, prepare_hermetic_extras
|
||||
|
||||
FIXTURE_DIR = Path(__file__).parent / "fixtures" / "replay"
|
||||
|
||||
|
||||
def _reset_process_singletons(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Invalidate process-wide caches so the test-only config/home take effect.
|
||||
|
||||
Same set the real-server e2e resets (see test_setup_agent_http_e2e_real_server).
|
||||
"""
|
||||
from deerflow.config import app_config as app_config_module
|
||||
from deerflow.config import paths as paths_module
|
||||
from deerflow.persistence import engine as engine_module
|
||||
|
||||
for module, attr in (
|
||||
(app_config_module, "_app_config"),
|
||||
(app_config_module, "_app_config_path"),
|
||||
(app_config_module, "_app_config_mtime"),
|
||||
(paths_module, "_paths_singleton"),
|
||||
(engine_module, "_engine"),
|
||||
(engine_module, "_session_factory"),
|
||||
):
|
||||
monkeypatch.setattr(module, attr, None, raising=False)
|
||||
|
||||
|
||||
@pytest.mark.no_auto_user
|
||||
def test_replay_write_read_file_ultra_matches_golden(tmp_path: Path, monkeypatch: pytest.MonkeyPatch):
|
||||
scenario, mode = "write_read_file", "ultra"
|
||||
fixture_path = FIXTURE_DIR / f"{scenario}.{mode}.json"
|
||||
events_path = FIXTURE_DIR / f"{scenario}.{mode}.events.json"
|
||||
fixture = json.loads(fixture_path.read_text(encoding="utf-8"))
|
||||
|
||||
home = tmp_path / "home"
|
||||
home.mkdir()
|
||||
monkeypatch.setenv("DEER_FLOW_HOME", str(home))
|
||||
monkeypatch.setenv("DEERFLOW_REPLAY_FIXTURE", str(fixture_path))
|
||||
|
||||
cfg_path = tmp_path / "config.yaml"
|
||||
cfg_path.write_text(build_config_yaml(model_block=REPLAY_MODEL_BLOCK, home=home), encoding="utf-8")
|
||||
monkeypatch.setenv("DEER_FLOW_CONFIG_PATH", str(cfg_path))
|
||||
monkeypatch.setenv("DEER_FLOW_EXTENSIONS_CONFIG_PATH", str(prepare_hermetic_extras(home)))
|
||||
|
||||
_reset_process_singletons(monkeypatch)
|
||||
from deerflow.config import app_config as app_config_module
|
||||
|
||||
cfg = app_config_module.get_app_config()
|
||||
cfg.database.sqlite_dir = str(home / "db")
|
||||
|
||||
from app.gateway.app import create_app
|
||||
|
||||
events = drive_gateway(create_app(), prompt=fixture["prompt"], context=fixture["context"])
|
||||
|
||||
assert events, "replay produced no SSE events"
|
||||
assert events[0]["event"] == "metadata", f"first event should be metadata, got {events[0]!r}"
|
||||
assert events[-1]["event"] == "end", f"last event should be end (run completed), got {events[-1]!r}"
|
||||
|
||||
# Regenerate the committed golden after re-recording the fixture:
|
||||
# DEERFLOW_WRITE_GOLDEN=1 uv run pytest tests/test_replay_golden.py
|
||||
if os.environ.get("DEERFLOW_WRITE_GOLDEN"):
|
||||
events_path.write_text(json.dumps({"scenario": scenario, "mode": mode, "events": events}, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
return
|
||||
|
||||
golden = json.loads(events_path.read_text(encoding="utf-8"))["events"]
|
||||
# A replay hash-miss surfaces as the run erroring mid-stream -> the event
|
||||
# shape sequence diverges from the golden, so this assertion is the catch-all
|
||||
# for both backend SSE drift and replay divergence.
|
||||
assert events == golden, f"SSE event-shape sequence drifted from the golden.\ngot ({len(events)}): {[e['event'] for e in events]}\nwant ({len(golden)}): {[e['event'] for e in golden]}"
|
||||
Reference in New Issue
Block a user