test(e2e): deterministic record/replay front-back contract verification (#3365)

* test(e2e): record/replay front-back contract verification Guards the front-back contract with a deterministic, key-free record/replay harness (mirrors open-design's golden-trace approach): - ReplayChatModel (tests/replay_provider.py): replays recorded LLM turns by a normalized hash of the model input. Strips <system-reminder>/date/uuid/tmp-path so one fixture replays across days and from both the browser and direct-POST paths; a miss raises loudly (no silent divergence). - Recording is record-through-browser (scripts/record_gateway.py + build_fixture_from_jsonl.py + frontend/tests/e2e-record): a real run is driven through the real frontend so captured inputs match exactly what the browser sends; fixtures contain no API key. - Layer 1 — backend golden (tests/test_replay_golden.py): replay through the real gateway, assert the SSE event sequence == committed golden. - Layer 2 — full-stack render (frontend/tests/e2e-real-backend): real Next.js + real gateway (replay model) + Chromium; assert the replayed auto-title and follow-up suggestions render. DOM assertions are the gate; visual regression is a local dev gate (CI uploads the render as an artifact). - CI (.github/workflows/replay-e2e.yml): both layers, triggered on EITHER side of the contract (frontend/** or backend gateway/harness/fixtures). * test(e2e): multi-run render-order cross-stack scenario (#3352) Guards the dangerous front-back class where a backend ordering change silently breaks a frontend assumption while both sides' unit tests stay green. Reproduces issue #3352: backend list_by_thread returns runs newest-first (#2932) and the frontend prepended per-run pages, inverting chronological order once the checkpoint no longer held the older messages. - tests/seed_runs_router.py: test-only seeder, mounted on the replay gateway only when DEERFLOW_ENABLE_TEST_SEED=1 (never in the production app). Seeds a thread with >=2 runs + per-run message events and no checkpoint -- the #3352 precondition -- so the frontend per-run reload path is the sole source of truth and the prepend inversion is observable. - frontend/tests/e2e-real-backend/multi-run-order.spec.ts: drives the real frontend against the real gateway, asserts the first run renders above the second. Reverting the #3354 fix turns it red. - replay-e2e.yml: trigger on the new replay test-infra paths. - docs: REPLAY_E2E.md cross-stack scenario section. * test(e2e): address Copilot review on the replay harness - Fix stale recorder references (scripts/record_traces.py -> scripts/record_gateway.py + scripts/build_fixture_from_jsonl.py) in replay_provider.py, test_replay_golden.py, _replay_fixture.py. - MODE_CONTEXT['ultra']: thinking_enabled False -> True, mirroring the frontend's `context.mode !== 'flash'` (hooks.ts). It did not affect the hashed input (Layer 1 golden still green), but the table now matches the real frontend context it claims to mirror. - replay_provider.py docstring: stop claiming memory is recorded-enabled; the replay config disables memory/summarization for determinism (title stays, as an in-graph deterministic call). - record_gateway.py / run_replay_gateway.py: override DEER_FLOW_HOME instead of setdefault, so an outer value can't leak into the hermetic harness. - record_gateway.py: clear error when DEERFLOW_RECORD_OUT is unset (was a bare KeyError). - playwright.record.config.ts: forward OPENAI_*/DEERFLOW_RECORD_OUT only when set, so the gateway raises a clear 'missing env' error instead of getting ''. * test(e2e): address Copilot review round 2 - seed_runs_router.py: constrain SeedMessage.role to Literal['human','ai'] so a bad value is a clean 422 at the boundary instead of a 500 (KeyError on _EVENT_TYPE). - record-write-read-file.spec.ts: waitForCaptureStable now throws on timeout instead of returning the last count, so a truncated/partial recording can't pass silently. - real-backend-render.spec.ts: guard the suggestions JSON.parse; a bracket-prefixed non-JSON turn falls back to '' so the existing not.toBe('') assertion fails clearly instead of a generic parse throw.
2026-06-10 09:25:57 +00:00 · 2026-06-08 12:35:03 +08:00
parent 64d923b0fd
commit 88759015e4
17 changed files with 1772 additions and 0 deletions
@@ -0,0 +1,123 @@
+import { readFileSync } from "node:fs";
+import { dirname, join } from "node:path";
+import { fileURLToPath } from "node:url";
+
+import { expect, test } from "@playwright/test";
+
+const here = dirname(fileURLToPath(import.meta.url));
+
+/**
+ * Layer 2: drive the REAL frontend against the REAL gateway (replay model, no
+ * API key) and assert the browser renders the backend's data correctly.
+ *
+ * The prompt is read from the same fixture the gateway replays, so the input
+ * hash matches and the recorded turns (write_file -> auto-title -> read_file ->
+ * final answer) reproduce deterministically.
+ */
+// Register through the frontend origin (same-origin proxy) so the auth cookies
+// are stored for and sent to localhost:3000 — the gateway is reached via the
+// next.config rewrite, never cross-origin from the browser.
+const APP = "http://localhost:3000";
+const fixture = JSON.parse(
+  readFileSync(
+    join(
+      here,
+      "../../../backend/tests/fixtures/replay/write_read_file.ultra.json",
+    ),
+    "utf-8",
+  ),
+) as {
+  prompt: string;
+  turns: Array<{ output: { data: { content?: unknown } } }>;
+};
+
+const PROMPT = fixture.prompt;
+// Derive the assertions from the fixture so a re-record auto-updates them. Both
+// are model-generated strings absent from the user prompt, so a pass proves the
+// replay drove the render (not a prompt echo): the first plain-text turn is the
+// in-graph auto-title; the JSON-array turn is the follow-up suggestions.
+const textTurns = fixture.turns
+  .map((t) => t.output?.data?.content)
+  .filter((c): c is string => typeof c === "string" && c.trim().length > 0);
+const suggestionsRaw = textTurns.find((c) => c.trim().startsWith("["));
+// Guarded parse: a bracket-prefixed turn that isn't a valid JSON string array
+// falls back to "" so the `not.toBe("")` assertion below fails with a clear
+// message instead of a generic JSON.parse throw.
+const EXPECTED_SUGGESTION = ((): string => {
+  if (!suggestionsRaw) return "";
+  try {
+    const arr: unknown = JSON.parse(suggestionsRaw);
+    return Array.isArray(arr) && typeof arr[0] === "string" ? arr[0] : "";
+  } catch {
+    return "";
+  }
+})();
+const EXPECTED_TITLE = textTurns.find((c) => !c.trim().startsWith("[")) ?? "";
+
+test.describe("real backend render (replay, no API key)", () => {
+  test.beforeEach(async ({ context }) => {
+    // Throwaway test account: register sets access_token + csrf_token cookies in
+    // the browser context (host-scoped to localhost, shared across ports), so
+    // the frontend's SDK (credentials:include + X-CSRF-Token) authenticates.
+    const email = `e2e-${Date.now()}-${Math.floor(Math.random() * 1e6)}@example.com`;
+    const resp = await context.request.post(`${APP}/api/v1/auth/register`, {
+      data: { email, password: "very-strong-password-123" },
+    });
+    expect(resp.status(), await resp.text()).toBe(201);
+  });
+
+  test("renders the replayed auto-title + suggestions from a real backend", async ({
+    page,
+  }) => {
+    // ultra mode so the context the frontend sends (is_plan_mode + subagent_enabled)
+    // matches the recorded fixture; otherwise the replay input hash would miss.
+    await page.addInitScript(() => {
+      window.localStorage.setItem(
+        "deerflow.local-settings",
+        JSON.stringify({ context: { mode: "ultra" } }),
+      );
+    });
+
+    await page.goto("/workspace/chats/new");
+
+    const textarea = page.getByPlaceholder(/how can i assist you/i);
+    await expect(textarea).toBeVisible({ timeout: 30_000 });
+    await textarea.fill(PROMPT);
+    await textarea.press("Enter");
+
+    // Replay-only DOM assertions (derived from the fixture): they render only if
+    // the recorded turns replayed AND the real frontend rendered them — the
+    // in-graph auto-title and the post-answer follow-up suggestion. Together they
+    // prove the whole pipeline (replay backend -> real frontend render).
+    expect(
+      EXPECTED_TITLE,
+      "fixture should contain an auto-title turn",
+    ).not.toBe("");
+    expect(
+      EXPECTED_SUGGESTION,
+      "fixture should contain a suggestions turn",
+    ).not.toBe("");
+    await expect(page.getByText(EXPECTED_TITLE)).toBeVisible({
+      timeout: 60_000,
+    });
+    await expect(page.getByText(EXPECTED_SUGGESTION)).toBeVisible({
+      timeout: 30_000,
+    });
+
+    // Visual regression is OS-sensitive (a macOS baseline won't match CI's
+    // Linux render), so it's a local dev gate only; in CI we capture the render
+    // as an artifact for human review instead of hard-asserting a cross-OS
+    // baseline. The DOM assertions above are the CI gate.
+    if (process.env.CI) {
+      await page.screenshot({
+        path: "test-results/real-backend-render.png",
+        fullPage: true,
+      });
+    } else {
+      await expect(page).toHaveScreenshot("real-backend-render.png", {
+        maxDiffPixelRatio: 0.02,
+        fullPage: true,
+      });
+    }
+  });
+});