test(e2e): deterministic record/replay front-back contract verification (#3365)

* test(e2e): record/replay front-back contract verification

Guards the front-back contract with a deterministic, key-free record/replay
harness (mirrors open-design's golden-trace approach):

- ReplayChatModel (tests/replay_provider.py): replays recorded LLM turns by a
  normalized hash of the model input. Strips <system-reminder>/date/uuid/tmp-path
  so one fixture replays across days and from both the browser and direct-POST
  paths; a miss raises loudly (no silent divergence).
- Recording is record-through-browser (scripts/record_gateway.py +
  build_fixture_from_jsonl.py + frontend/tests/e2e-record): a real run is driven
  through the real frontend so captured inputs match exactly what the browser
  sends; fixtures contain no API key.
- Layer 1 — backend golden (tests/test_replay_golden.py): replay through the real
  gateway, assert the SSE event sequence == committed golden.
- Layer 2 — full-stack render (frontend/tests/e2e-real-backend): real Next.js +
  real gateway (replay model) + Chromium; assert the replayed auto-title and
  follow-up suggestions render. DOM assertions are the gate; visual regression is
  a local dev gate (CI uploads the render as an artifact).
- CI (.github/workflows/replay-e2e.yml): both layers, triggered on EITHER side of
  the contract (frontend/** or backend gateway/harness/fixtures).

* test(e2e): multi-run render-order cross-stack scenario (#3352)

Guards the dangerous front-back class where a backend ordering change
silently breaks a frontend assumption while both sides' unit tests stay
green. Reproduces issue #3352: backend list_by_thread returns runs
newest-first (#2932) and the frontend prepended per-run pages, inverting
chronological order once the checkpoint no longer held the older messages.

- tests/seed_runs_router.py: test-only seeder, mounted on the replay
  gateway only when DEERFLOW_ENABLE_TEST_SEED=1 (never in the production
  app). Seeds a thread with >=2 runs + per-run message events and no
  checkpoint -- the #3352 precondition -- so the frontend per-run reload
  path is the sole source of truth and the prepend inversion is observable.
- frontend/tests/e2e-real-backend/multi-run-order.spec.ts: drives the real
  frontend against the real gateway, asserts the first run renders above
  the second. Reverting the #3354 fix turns it red.
- replay-e2e.yml: trigger on the new replay test-infra paths.
- docs: REPLAY_E2E.md cross-stack scenario section.

* test(e2e): address Copilot review on the replay harness

- Fix stale recorder references (scripts/record_traces.py ->
  scripts/record_gateway.py + scripts/build_fixture_from_jsonl.py) in
  replay_provider.py, test_replay_golden.py, _replay_fixture.py.
- MODE_CONTEXT['ultra']: thinking_enabled False -> True, mirroring the
  frontend's `context.mode !== 'flash'` (hooks.ts). It did not affect the
  hashed input (Layer 1 golden still green), but the table now matches the
  real frontend context it claims to mirror.
- replay_provider.py docstring: stop claiming memory is recorded-enabled;
  the replay config disables memory/summarization for determinism (title
  stays, as an in-graph deterministic call).
- record_gateway.py / run_replay_gateway.py: override DEER_FLOW_HOME instead
  of setdefault, so an outer value can't leak into the hermetic harness.
- record_gateway.py: clear error when DEERFLOW_RECORD_OUT is unset (was a
  bare KeyError).
- playwright.record.config.ts: forward OPENAI_*/DEERFLOW_RECORD_OUT only when
  set, so the gateway raises a clear 'missing env' error instead of getting ''.

* test(e2e): address Copilot review round 2

- seed_runs_router.py: constrain SeedMessage.role to Literal['human','ai']
  so a bad value is a clean 422 at the boundary instead of a 500
  (KeyError on _EVENT_TYPE).
- record-write-read-file.spec.ts: waitForCaptureStable now throws on
  timeout instead of returning the last count, so a truncated/partial
  recording can't pass silently.
- real-backend-render.spec.ts: guard the suggestions JSON.parse; a
  bracket-prefixed non-JSON turn falls back to '' so the existing
  not.toBe('') assertion fails clearly instead of a generic parse throw.

backend/scripts/build_fixture_from_jsonl.py

@@ -0,0 +1,44 @@
+"""Turn a record-through-browser JSONL capture into a replay fixture.
+
+The recording gateway (``record_gateway.py``) appends ``{input_hash, output}``
+lines as the frontend drives a real run; the record spec writes a ``.meta.json``
+sidecar with ``{scenario, mode, prompt}``. This stitches them into the fixture
+the replay provider + tests consume.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--jsonl", required=True)
+    parser.add_argument("--meta", required=True)
+    parser.add_argument("--out", required=True)
+    parser.add_argument("--model", default="gpt-5.5")
+    args = parser.parse_args()
+
+    turns = [json.loads(line) for line in Path(args.jsonl).read_text(encoding="utf-8").splitlines() if line.strip()]
+    meta = json.loads(Path(args.meta).read_text(encoding="utf-8"))
+    fixture = {
+        "scenario": meta["scenario"],
+        "mode": meta["mode"],
+        "model": args.model,
+        "prompt": meta["prompt"],
+        "context": meta.get("context", {}),
+        "turns": turns,
+    }
+    Path(args.out).write_text(json.dumps(fixture, ensure_ascii=False, indent=2), encoding="utf-8")
+    print(f"wrote {len(turns)} turn(s) -> {args.out}")
+    for index, turn in enumerate(turns):
+        data = turn["output"].get("data", {})
+        tool_calls = [tc.get("name") for tc in (data.get("tool_calls") or [])]
+        print(f"  turn {index}: hash={turn['input_hash'][:12]} tool_calls={tool_calls} content={str(data.get('content'))[:50]!r}")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

backend/scripts/record_gateway.py

@@ -0,0 +1,109 @@
+"""Recording gateway for *record-through-browser* (Plan A).
+
+Runs the gateway with a REAL model and a callback that appends every model
+call's ``(input_hash, output)`` to a JSONL file. Because the run is driven by
+the real frontend (Playwright), the captured inputs are EXACTLY what the
+frontend produces (date system-reminder, suggestions/title calls, ...), so the
+resulting fixture replays cleanly against the browser.
+
+Used by ``frontend/playwright.record.config.ts``. Env:
+  OPENAI_API_KEY / OPENAI_API_BASE  - the real upstream (never committed)
+  DEERFLOW_RECORD_OUT               - JSONL path to append captured turns to
+  RECORD_PORT (default 8012), RECORD_MODEL (default gpt-5.5)
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import sys
+import tempfile
+from pathlib import Path
+
+_BACKEND = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(_BACKEND))
+sys.path.insert(0, str(_BACKEND / "tests"))
+
+
+def _install_capture(out_path: Path) -> None:
+    from langchain_core.callbacks import BaseCallbackHandler
+    from langchain_core.messages import messages_to_dict
+    from replay_provider import hash_messages
+
+    import deerflow.models.factory as factory_mod
+
+    class Capture(BaseCallbackHandler):
+        def __init__(self) -> None:
+            self.inputs: dict[str, list] = {}
+
+        def on_chat_model_start(self, serialized, messages, *, run_id=None, **kwargs):  # noqa: ANN001
+            self.inputs[str(run_id)] = messages[0] if messages else []
+
+        def on_llm_end(self, response, *, run_id=None, **kwargs):  # noqa: ANN001
+            inp = self.inputs.pop(str(run_id), None)
+            if inp is None:
+                return
+            for batch in response.generations:
+                for gen in batch:
+                    message = getattr(gen, "message", None)
+                    if message is None:
+                        continue
+                    record = {"input_hash": hash_messages(inp), "output": messages_to_dict([message])[0]}
+                    with open(out_path, "a", encoding="utf-8") as handle:
+                        handle.write(json.dumps(record, ensure_ascii=False) + "\n")
+                        handle.flush()
+
+    cb = Capture()
+    original = factory_mod.create_chat_model
+
+    def wrapped(*args, **kwargs):
+        model = original(*args, **kwargs)
+        model.callbacks = (model.callbacks or []) + [cb]
+        return model
+
+    factory_mod.create_chat_model = wrapped
+    for module in list(sys.modules.values()):
+        if getattr(module, "create_chat_model", None) is original:
+            module.create_chat_model = wrapped
+
+
+def main() -> int:
+    if not os.environ.get("OPENAI_API_KEY") or not os.environ.get("OPENAI_API_BASE"):
+        print("ERROR: set OPENAI_API_KEY and OPENAI_API_BASE (an OpenAI-compatible /v1 endpoint)", file=sys.stderr)
+        return 2
+
+    record_out = os.environ.get("DEERFLOW_RECORD_OUT")
+    if not record_out:
+        print("ERROR: set DEERFLOW_RECORD_OUT to the JSONL path to append captured turns to", file=sys.stderr)
+        return 2
+
+    port = int(os.environ.get("RECORD_PORT", "8012"))
+    model = os.environ.get("RECORD_MODEL", "gpt-5.5")
+    out = Path(record_out)
+    out.parent.mkdir(parents=True, exist_ok=True)
+    out.write_text("", encoding="utf-8")  # fresh capture per recording run
+
+    from _replay_fixture import build_config_yaml, prepare_hermetic_extras, real_model_block
+
+    home = Path(tempfile.mkdtemp(prefix="record-gw-"))
+    cfg = home / "config.yaml"
+    cfg.write_text(build_config_yaml(model_block=real_model_block(model), home=home), encoding="utf-8")
+    # Override (not setdefault): the recorder must be hermetic, so an outer
+    # DEER_FLOW_HOME can't leak in and shift prompt-affecting paths/skills.
+    os.environ["DEER_FLOW_HOME"] = str(home)
+    os.environ["DEER_FLOW_CONFIG_PATH"] = str(cfg)
+    os.environ["DEER_FLOW_EXTENSIONS_CONFIG_PATH"] = str(prepare_hermetic_extras(home))
+    os.environ.setdefault("AUTH_JWT_SECRET", "record-secret")
+    os.environ["PYTHONPATH"] = os.pathsep.join(p for p in (str(_BACKEND), str(_BACKEND / "tests"), os.environ.get("PYTHONPATH", "")) if p)
+
+    _install_capture(out)
+
+    import uvicorn
+
+    print(f"[record-gw] model={model} out={out} port={port}", flush=True)
+    uvicorn.run("app.gateway.app:app", host="127.0.0.1", port=port, log_level="warning")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

backend/scripts/run_replay_gateway.py

@@ -0,0 +1,73 @@
+"""Start a hermetic *replay* gateway for the full-stack (Layer 2) e2e.
+
+Builds an ephemeral config that points the model at ``ReplayChatModel`` + a
+recorded fixture, then runs uvicorn — no API key, deterministic. Used as a
+Playwright ``webServer`` (see ``frontend/playwright.real-backend.config.ts``) and
+runnable standalone for debugging::
+
+    uv run python scripts/run_replay_gateway.py --port 8011
+
+``tests/`` is put on the path so the config ``use: replay_provider:ReplayChatModel``
+resolves; ``GATEWAY_CORS_ORIGINS`` is set so the frontend on :3000 can talk to it.
+"""
+
+from __future__ import annotations
+
+import argparse
+import os
+import sys
+import tempfile
+from pathlib import Path
+
+_BACKEND = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(_BACKEND))
+sys.path.insert(0, str(_BACKEND / "tests"))  # replay_provider + build_config_yaml live here
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--port", type=int, default=8011)
+    parser.add_argument("--fixture", default=str(_BACKEND / "tests" / "fixtures" / "replay" / "write_read_file.ultra.json"))
+    parser.add_argument("--cors", default="http://localhost:3000")
+    args = parser.parse_args()
+
+    from _replay_fixture import REPLAY_MODEL_BLOCK, build_config_yaml, prepare_hermetic_extras
+
+    home = Path(tempfile.mkdtemp(prefix="replay-gw-"))
+    cfg = home / "config.yaml"
+    cfg.write_text(build_config_yaml(model_block=REPLAY_MODEL_BLOCK, home=home), encoding="utf-8")
+
+    # Override (not setdefault): the replay gateway must be hermetic, so an outer
+    # DEER_FLOW_HOME can't leak in and shift prompt-affecting paths/skills.
+    os.environ["DEER_FLOW_HOME"] = str(home)
+    os.environ["DEER_FLOW_CONFIG_PATH"] = str(cfg)
+    os.environ["DEER_FLOW_EXTENSIONS_CONFIG_PATH"] = str(prepare_hermetic_extras(home))
+    os.environ["DEERFLOW_REPLAY_FIXTURE"] = args.fixture
+    os.environ.setdefault("AUTH_JWT_SECRET", "ci-replay-secret")
+    os.environ["GATEWAY_CORS_ORIGINS"] = args.cors
+    # Child / dynamic imports (resolve_class) search PYTHONPATH too.
+    os.environ["PYTHONPATH"] = os.pathsep.join(p for p in (str(_BACKEND), str(_BACKEND / "tests"), os.environ.get("PYTHONPATH", "")) if p)
+
+    import uvicorn
+
+    target: str | object = "app.gateway.app:app"
+    # Test-only: attach the run/message seeder used by the multi-run render-order
+    # e2e (#3352). Imported from tests/ and mounted here only — never in the
+    # production app. Pass the app object (not the import string) so the extra
+    # router is registered before uvicorn serves it.
+    if os.environ.get("DEERFLOW_ENABLE_TEST_SEED") == "1":
+        from seed_runs_router import router as seed_router
+
+        from app.gateway.app import app as gateway_app
+
+        gateway_app.include_router(seed_router)
+        target = gateway_app
+        print("[replay-gw] test-only seed router mounted at /api/test-only/seed-runs", flush=True)
+
+    print(f"[replay-gw] config={cfg} fixture={args.fixture} cors={args.cors} port={args.port}", flush=True)
+    uvicorn.run(target, host="127.0.0.1", port=args.port, log_level="warning")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())