test(e2e): deterministic record/replay front-back contract verification (#3365)

* test(e2e): record/replay front-back contract verification

Guards the front-back contract with a deterministic, key-free record/replay
harness (mirrors open-design's golden-trace approach):

- ReplayChatModel (tests/replay_provider.py): replays recorded LLM turns by a
  normalized hash of the model input. Strips <system-reminder>/date/uuid/tmp-path
  so one fixture replays across days and from both the browser and direct-POST
  paths; a miss raises loudly (no silent divergence).
- Recording is record-through-browser (scripts/record_gateway.py +
  build_fixture_from_jsonl.py + frontend/tests/e2e-record): a real run is driven
  through the real frontend so captured inputs match exactly what the browser
  sends; fixtures contain no API key.
- Layer 1 — backend golden (tests/test_replay_golden.py): replay through the real
  gateway, assert the SSE event sequence == committed golden.
- Layer 2 — full-stack render (frontend/tests/e2e-real-backend): real Next.js +
  real gateway (replay model) + Chromium; assert the replayed auto-title and
  follow-up suggestions render. DOM assertions are the gate; visual regression is
  a local dev gate (CI uploads the render as an artifact).
- CI (.github/workflows/replay-e2e.yml): both layers, triggered on EITHER side of
  the contract (frontend/** or backend gateway/harness/fixtures).

* test(e2e): multi-run render-order cross-stack scenario (#3352)

Guards the dangerous front-back class where a backend ordering change
silently breaks a frontend assumption while both sides' unit tests stay
green. Reproduces issue #3352: backend list_by_thread returns runs
newest-first (#2932) and the frontend prepended per-run pages, inverting
chronological order once the checkpoint no longer held the older messages.

- tests/seed_runs_router.py: test-only seeder, mounted on the replay
  gateway only when DEERFLOW_ENABLE_TEST_SEED=1 (never in the production
  app). Seeds a thread with >=2 runs + per-run message events and no
  checkpoint -- the #3352 precondition -- so the frontend per-run reload
  path is the sole source of truth and the prepend inversion is observable.
- frontend/tests/e2e-real-backend/multi-run-order.spec.ts: drives the real
  frontend against the real gateway, asserts the first run renders above
  the second. Reverting the #3354 fix turns it red.
- replay-e2e.yml: trigger on the new replay test-infra paths.
- docs: REPLAY_E2E.md cross-stack scenario section.

* test(e2e): address Copilot review on the replay harness

- Fix stale recorder references (scripts/record_traces.py ->
  scripts/record_gateway.py + scripts/build_fixture_from_jsonl.py) in
  replay_provider.py, test_replay_golden.py, _replay_fixture.py.
- MODE_CONTEXT['ultra']: thinking_enabled False -> True, mirroring the
  frontend's `context.mode !== 'flash'` (hooks.ts). It did not affect the
  hashed input (Layer 1 golden still green), but the table now matches the
  real frontend context it claims to mirror.
- replay_provider.py docstring: stop claiming memory is recorded-enabled;
  the replay config disables memory/summarization for determinism (title
  stays, as an in-graph deterministic call).
- record_gateway.py / run_replay_gateway.py: override DEER_FLOW_HOME instead
  of setdefault, so an outer value can't leak into the hermetic harness.
- record_gateway.py: clear error when DEERFLOW_RECORD_OUT is unset (was a
  bare KeyError).
- playwright.record.config.ts: forward OPENAI_*/DEERFLOW_RECORD_OUT only when
  set, so the gateway raises a clear 'missing env' error instead of getting ''.

* test(e2e): address Copilot review round 2

- seed_runs_router.py: constrain SeedMessage.role to Literal['human','ai']
  so a bad value is a clean 422 at the boundary instead of a 500
  (KeyError on _EVENT_TYPE).
- record-write-read-file.spec.ts: waitForCaptureStable now throws on
  timeout instead of returning the last count, so a truncated/partial
  recording can't pass silently.
- real-backend-render.spec.ts: guard the suggestions JSON.parse; a
  bracket-prefixed non-JSON turn falls back to '' so the existing
  not.toBe('') assertion fails clearly instead of a generic parse throw.

backend/tests/seed_runs_router.py

@@ -0,0 +1,100 @@
+"""Test-only run/message seeder for the multi-run render-order e2e (issue #3352).
+
+Mounted **only** by ``scripts/run_replay_gateway.py`` (the replay e2e gateway)
+and never by the production app, so it cannot ship. It lets a Playwright spec
+stand up a thread with >=2 runs whose per-run messages exercise the frontend's
+reload / history-rebuild ordering path — with no real model, no recording, and
+no API key.
+
+Why a seeder instead of recording a conversation: issue #3352 only reproduces
+when the checkpoint no longer holds the older messages (post-compression), so
+the frontend rebuilds them from the per-run history endpoints. A seeder lets us
+create exactly that precondition deterministically — runs in the run store +
+per-run ``category="message"`` events, and **no checkpoint** — so on reload the
+buggy ``findLatestUnloadedRunIndex`` + prepend in ``core/threads/hooks.ts`` is
+the sole source of truth and its reversed order becomes observable.
+
+It writes through the gateway's OWN ``app.state.run_store`` +
+``app.state.run_event_store`` using the request's auth context, so the seeded
+``user_id`` matches the browser session that reads it back. The event shape
+mirrors exactly what ``runtime/journal.py`` writes for real runs
+(``event_type`` ``llm.human.input`` / ``llm.ai.response``, ``category``
+``"message"``, ``content`` = ``message.model_dump()``, ``metadata.caller`` =
+``"lead_agent"``).
+"""
+
+from __future__ import annotations
+
+from typing import Literal
+
+from fastapi import APIRouter, Request
+from pydantic import BaseModel
+
+router = APIRouter(prefix="/api/test-only", tags=["test-only"])
+
+# Mirror runtime/journal.py: human prompts are recorded as ``llm.human.input``
+# and assistant turns as ``llm.ai.response``; both land in ``category="message"``.
+_EVENT_TYPE = {"human": "llm.human.input", "ai": "llm.ai.response"}
+
+
+class SeedMessage(BaseModel):
+    role: Literal["human", "ai"]
+    content: str
+    id: str
+
+
+class SeedRun(BaseModel):
+    run_id: str
+    # ISO timestamp; RunManager.list_by_thread sorts newest-first by created_at,
+    # so a later created_at must mean a later run for the ordering to be faithful.
+    created_at: str
+    messages: list[SeedMessage]
+
+
+class SeedRunsBody(BaseModel):
+    thread_id: str
+    runs: list[SeedRun]
+
+
+@router.post("/seed-runs")
+async def seed_runs(body: SeedRunsBody, request: Request) -> dict:
+    """Seed runs + per-run message events for the authenticated user.
+
+    No checkpoint is written: that is the whole point — it forces the frontend's
+    reload path to rebuild history from the per-run endpoints (the #3352 bug
+    site) instead of the (correctly ordered) checkpoint snapshot.
+    """
+    from langchain_core.messages import AIMessage, HumanMessage
+
+    run_store = request.app.state.run_store
+    event_store = request.app.state.run_event_store
+
+    for run in body.runs:
+        # user_id defaults (AUTO) to the request's auth context, matching the
+        # browser session that will read these runs back via GET /runs.
+        await run_store.put(
+            run.run_id,
+            thread_id=body.thread_id,
+            assistant_id="lead_agent",
+            status="success",
+            created_at=run.created_at,
+        )
+        events = []
+        for m in run.messages:
+            msg = (HumanMessage if m.role == "human" else AIMessage)(content=m.content, id=m.id)
+            events.append(
+                {
+                    "thread_id": body.thread_id,
+                    "run_id": run.run_id,
+                    "event_type": _EVENT_TYPE[m.role],
+                    "category": "message",
+                    "content": msg.model_dump(),
+                    "metadata": {"caller": "lead_agent"},
+                    "created_at": run.created_at,
+                }
+            )
+        # One batch per run so seq is monotonic and run1's messages precede
+        # run2's; the gateway reads them back per-run anyway.
+        await event_store.put_batch(events)
+
+    return {"ok": True, "thread_id": body.thread_id, "runs": len(body.runs)}