furyhawk/deer-flow
1
0
Fork 0
mirror of https://github.com/bytedance/deer-flow.git synced 2026-06-10 09:25:57 +00:00
Code Issues Packages Projects Releases Wiki Activity

test(e2e): deterministic record/replay front-back contract verification (#3365)

Browse Source 
* test(e2e): record/replay front-back contract verification

Guards the front-back contract with a deterministic, key-free record/replay
harness (mirrors open-design's golden-trace approach):

- ReplayChatModel (tests/replay_provider.py): replays recorded LLM turns by a
  normalized hash of the model input. Strips <system-reminder>/date/uuid/tmp-path
  so one fixture replays across days and from both the browser and direct-POST
  paths; a miss raises loudly (no silent divergence).
- Recording is record-through-browser (scripts/record_gateway.py +
  build_fixture_from_jsonl.py + frontend/tests/e2e-record): a real run is driven
  through the real frontend so captured inputs match exactly what the browser
  sends; fixtures contain no API key.
- Layer 1 — backend golden (tests/test_replay_golden.py): replay through the real
  gateway, assert the SSE event sequence == committed golden.
- Layer 2 — full-stack render (frontend/tests/e2e-real-backend): real Next.js +
  real gateway (replay model) + Chromium; assert the replayed auto-title and
  follow-up suggestions render. DOM assertions are the gate; visual regression is
  a local dev gate (CI uploads the render as an artifact).
- CI (.github/workflows/replay-e2e.yml): both layers, triggered on EITHER side of
  the contract (frontend/** or backend gateway/harness/fixtures).

* test(e2e): multi-run render-order cross-stack scenario (#3352)

Guards the dangerous front-back class where a backend ordering change
silently breaks a frontend assumption while both sides' unit tests stay
green. Reproduces issue #3352: backend list_by_thread returns runs
newest-first (#2932) and the frontend prepended per-run pages, inverting
chronological order once the checkpoint no longer held the older messages.

- tests/seed_runs_router.py: test-only seeder, mounted on the replay
  gateway only when DEERFLOW_ENABLE_TEST_SEED=1 (never in the production
  app). Seeds a thread with >=2 runs + per-run message events and no
  checkpoint -- the #3352 precondition -- so the frontend per-run reload
  path is the sole source of truth and the prepend inversion is observable.
- frontend/tests/e2e-real-backend/multi-run-order.spec.ts: drives the real
  frontend against the real gateway, asserts the first run renders above
  the second. Reverting the #3354 fix turns it red.
- replay-e2e.yml: trigger on the new replay test-infra paths.
- docs: REPLAY_E2E.md cross-stack scenario section.

* test(e2e): address Copilot review on the replay harness

- Fix stale recorder references (scripts/record_traces.py ->
  scripts/record_gateway.py + scripts/build_fixture_from_jsonl.py) in
  replay_provider.py, test_replay_golden.py, _replay_fixture.py.
- MODE_CONTEXT['ultra']: thinking_enabled False -> True, mirroring the
  frontend's `context.mode !== 'flash'` (hooks.ts). It did not affect the
  hashed input (Layer 1 golden still green), but the table now matches the
  real frontend context it claims to mirror.
- replay_provider.py docstring: stop claiming memory is recorded-enabled;
  the replay config disables memory/summarization for determinism (title
  stays, as an in-graph deterministic call).
- record_gateway.py / run_replay_gateway.py: override DEER_FLOW_HOME instead
  of setdefault, so an outer value can't leak into the hermetic harness.
- record_gateway.py: clear error when DEERFLOW_RECORD_OUT is unset (was a
  bare KeyError).
- playwright.record.config.ts: forward OPENAI_*/DEERFLOW_RECORD_OUT only when
  set, so the gateway raises a clear 'missing env' error instead of getting ''.

* test(e2e): address Copilot review round 2

- seed_runs_router.py: constrain SeedMessage.role to Literal['human','ai']
  so a bad value is a clean 422 at the boundary instead of a 500
  (KeyError on _EVENT_TYPE).
- record-write-read-file.spec.ts: waitForCaptureStable now throws on
  timeout instead of returning the last count, so a truncated/partial
  recording can't pass silently.
- real-backend-render.spec.ts: guard the suggestions JSON.parse; a
  bracket-prefixed non-JSON turn falls back to '' so the existing
  not.toBe('') assertion fails clearly instead of a generic parse throw.
This commit is contained in:
Xinmin Zeng
2026-06-08 12:35:03 +08:00
committed by GitHub
parent 64d923b0fd
commit 88759015e4
17 changed files with 1772 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/tests/fixtures/replay/write_read_file.ultra.events.json
Vendored
+72
View File
@@ -0,0 +1,72 @@
{
"scenario": "write_read_file",
"mode": "ultra",
"events": [
{
"event": "metadata",
"keys": [
"run_id",
"thread_id"
]
},
{
"event": "values",
"keys": [
"artifacts",
"messages",
"viewed_images"
]
},
{
"event": "values",
"keys": [
"artifacts",
"messages",
"thread_data",
"viewed_images"
]
},
{
"event": "values",
"keys": [
"artifacts",
"messages",
"thread_data",
"viewed_images"
]
},
{
"event": "values",
"keys": [
"artifacts",
"messages",
"thread_data",
"viewed_images"
]
},
{
"event": "values",
"keys": [
"artifacts",
"messages",
"thread_data",
"title",
"viewed_images"
]
},
{
"event": "values",
"keys": [
"artifacts",
"messages",
"thread_data",
"title",
"viewed_images"
]
},
{
"event": "end",
"keys": null
}
]
}
backend/tests/fixtures/replay/write_read_file.ultra.json
Vendored
+224
View File
@@ -0,0 +1,224 @@
{
"scenario": "write_read_file",
"mode": "ultra",
"model": "gpt-5.5",
"prompt": "Using your own file tools directly, create the file /mnt/user-data/outputs/note.txt with exactly this content: hi from replay. Then read that same file back and reply with its exact contents. Do NOT delegate to a subagent and do NOT use the task tool — do it yourself. Do not ask any clarifying questions.",
"context": {
"is_bootstrap": false,
"mode": "ultra",
"thinking_enabled": true,
"is_plan_mode": true,
"subagent_enabled": true
},
"turns": [
{
"input_hash": "686cd44a9f17fadc0398768731324f3980480a027593a475fad4583581df677f",
"output": {
"type": "ai",
"data": {
"content": "",
"additional_kwargs": {},
"response_metadata": {
"finish_reason": "tool_calls",
"model_name": "gpt-5.5",
"model_provider": "openai"
},
"type": "ai",
"name": null,
"id": "lc_run--019e8c60-8d4b-79a1-8d77-0a67fc360ce4",
"tool_calls": [
{
"name": "write_file",
"args": {
"description": "Create requested note file",
"path": "/mnt/user-data/outputs/note.txt",
"content": "hi from replay"
},
"id": "call_UdIzq5Vyx7pu1Usnj4wPCC6G",
"type": "tool_call"
}
],
"invalid_tool_calls": [],
"usage_metadata": {
"input_tokens": 3285,
"output_tokens": 66,
"total_tokens": 3351,
"input_token_details": {
"audio": 0,
"cache_read": 0
},
"output_token_details": {
"audio": 0,
"reasoning": 21
}
}
}
}
},
{
"input_hash": "3598aeb87e221ca8f554e4d61ce6d5e8801754606fa5c95a89c38bd6cb623045",
"output": {
"type": "ai",
"data": {
"content": "File Creation and Verification",
"additional_kwargs": {},
"response_metadata": {
"finish_reason": "stop",
"model_name": "gpt-5.5",
"model_provider": "openai"
},
"type": "ai",
"name": null,
"id": "lc_run--019e8c60-9c18-72c1-95e8-f6a240747395",
"tool_calls": [],
"invalid_tool_calls": [],
"usage_metadata": {
"input_tokens": 104,
"output_tokens": 53,
"total_tokens": 157,
"input_token_details": {
"audio": 0,
"cache_read": 0
},
"output_token_details": {
"audio": 0,
"reasoning": 39
}
}
}
}
},
{
"input_hash": "92430ba866abe577c86d2e67eb5158b10f3f19ec306aa9de235bb06736320d70",
"output": {
"type": "ai",
"data": {
"content": "",
"additional_kwargs": {},
"response_metadata": {
"finish_reason": "tool_calls",
"model_name": "gpt-5.5",
"model_provider": "openai"
},
"type": "ai",
"name": null,
"id": "lc_run--019e8c60-b036-7710-8db9-717ab54e5805",
"tool_calls": [
{
"name": "read_file",
"args": {
"description": "Read requested note file",
"path": "/mnt/user-data/outputs/note.txt"
},
"id": "call_0BFNns0FkRb3n2LR0PRrfbIJ",
"type": "tool_call"
}
],
"invalid_tool_calls": [],
"usage_metadata": {
"input_tokens": 3334,
"output_tokens": 33,
"total_tokens": 3367,
"input_token_details": {
"audio": 0,
"cache_read": 0
},
"output_token_details": {
"audio": 0,
"reasoning": 0
}
}
}
}
},
{
"input_hash": "8ab757aa51f9d556adcea07c0221445a2b791cc882ef11922babf7f2865d1913",
"output": {
"type": "ai",
"data": {
"content": "hi from replay",
"additional_kwargs": {},
"response_metadata": {
"finish_reason": "stop",
"model_name": "gpt-5.5",
"model_provider": "openai"
},
"type": "ai",
"name": null,
"id": "lc_run--019e8c60-bef3-7201-a30a-cbc5f45920ba",
"tool_calls": [],
"invalid_tool_calls": [],
"usage_metadata": {
"input_tokens": 3380,
"output_tokens": 7,
"total_tokens": 3387,
"input_token_details": {
"audio": 0,
"cache_read": 0
},
"output_token_details": {
"audio": 0,
"reasoning": 0
}
}
}
}
},
{
"input_hash": "fd67723cc8810ce79b4785fec4c251a272a91d677a216c735b23b5f6d3dec0c3",
"output": {
"type": "ai",
"data": {
"content": "[\"Can you append another line to the file?\",\"Can you show the file path again?\",\"Can you delete the file now?\"]",
"additional_kwargs": {
"refusal": null
},
"response_metadata": {
"token_usage": {
"completion_tokens": 71,
"prompt_tokens": 224,
"total_tokens": 295,
"completion_tokens_details": {
"accepted_prediction_tokens": 0,
"audio_tokens": 0,
"reasoning_tokens": 33,
"rejected_prediction_tokens": 0
},
"prompt_tokens_details": {
"audio_tokens": 0,
"cached_tokens": 0
},
"input_tokens": 0,
"output_tokens": 0,
"input_tokens_details": null
},
"model_provider": "openai",
"model_name": "gpt-5.5",
"system_fingerprint": null,
"id": "chatcmpl-DmaI5yVqQ39LRWyugoCEPalKw0gBR",
"finish_reason": "stop",
"logprobs": null
},
"type": "ai",
"name": null,
"id": "lc_run--019e8c60-d025-7fd2-9cc9-8b4fb8fe1a82-0",
"tool_calls": [],
"invalid_tool_calls": [],
"usage_metadata": {
"input_tokens": 224,
"output_tokens": 71,
"total_tokens": 295,
"input_token_details": {
"audio": 0,
"cache_read": 0
},
"output_token_details": {
"audio": 0,
"reasoning": 33
}
}
}
}
}
]
}