test(e2e): deterministic record/replay front-back contract verification (#3365)

* test(e2e): record/replay front-back contract verification Guards the front-back contract with a deterministic, key-free record/replay harness (mirrors open-design's golden-trace approach): - ReplayChatModel (tests/replay_provider.py): replays recorded LLM turns by a normalized hash of the model input. Strips <system-reminder>/date/uuid/tmp-path so one fixture replays across days and from both the browser and direct-POST paths; a miss raises loudly (no silent divergence). - Recording is record-through-browser (scripts/record_gateway.py + build_fixture_from_jsonl.py + frontend/tests/e2e-record): a real run is driven through the real frontend so captured inputs match exactly what the browser sends; fixtures contain no API key. - Layer 1 — backend golden (tests/test_replay_golden.py): replay through the real gateway, assert the SSE event sequence == committed golden. - Layer 2 — full-stack render (frontend/tests/e2e-real-backend): real Next.js + real gateway (replay model) + Chromium; assert the replayed auto-title and follow-up suggestions render. DOM assertions are the gate; visual regression is a local dev gate (CI uploads the render as an artifact). - CI (.github/workflows/replay-e2e.yml): both layers, triggered on EITHER side of the contract (frontend/** or backend gateway/harness/fixtures). * test(e2e): multi-run render-order cross-stack scenario (#3352) Guards the dangerous front-back class where a backend ordering change silently breaks a frontend assumption while both sides' unit tests stay green. Reproduces issue #3352: backend list_by_thread returns runs newest-first (#2932) and the frontend prepended per-run pages, inverting chronological order once the checkpoint no longer held the older messages. - tests/seed_runs_router.py: test-only seeder, mounted on the replay gateway only when DEERFLOW_ENABLE_TEST_SEED=1 (never in the production app). Seeds a thread with >=2 runs + per-run message events and no checkpoint -- the #3352 precondition -- so the frontend per-run reload path is the sole source of truth and the prepend inversion is observable. - frontend/tests/e2e-real-backend/multi-run-order.spec.ts: drives the real frontend against the real gateway, asserts the first run renders above the second. Reverting the #3354 fix turns it red. - replay-e2e.yml: trigger on the new replay test-infra paths. - docs: REPLAY_E2E.md cross-stack scenario section. * test(e2e): address Copilot review on the replay harness - Fix stale recorder references (scripts/record_traces.py -> scripts/record_gateway.py + scripts/build_fixture_from_jsonl.py) in replay_provider.py, test_replay_golden.py, _replay_fixture.py. - MODE_CONTEXT['ultra']: thinking_enabled False -> True, mirroring the frontend's `context.mode !== 'flash'` (hooks.ts). It did not affect the hashed input (Layer 1 golden still green), but the table now matches the real frontend context it claims to mirror. - replay_provider.py docstring: stop claiming memory is recorded-enabled; the replay config disables memory/summarization for determinism (title stays, as an in-graph deterministic call). - record_gateway.py / run_replay_gateway.py: override DEER_FLOW_HOME instead of setdefault, so an outer value can't leak into the hermetic harness. - record_gateway.py: clear error when DEERFLOW_RECORD_OUT is unset (was a bare KeyError). - playwright.record.config.ts: forward OPENAI_*/DEERFLOW_RECORD_OUT only when set, so the gateway raises a clear 'missing env' error instead of getting ''. * test(e2e): address Copilot review round 2 - seed_runs_router.py: constrain SeedMessage.role to Literal['human','ai'] so a bad value is a clean 422 at the boundary instead of a 500 (KeyError on _EVENT_TYPE). - record-write-read-file.spec.ts: waitForCaptureStable now throws on timeout instead of returning the last count, so a truncated/partial recording can't pass silently. - real-backend-render.spec.ts: guard the suggestions JSON.parse; a bracket-prefixed non-JSON turn falls back to '' so the existing not.toBe('') assertion fails clearly instead of a generic parse throw.
2026-06-10 09:25:57 +00:00 · 2026-06-08 12:35:03 +08:00
parent 64d923b0fd
commit 88759015e4
17 changed files with 1772 additions and 0 deletions
@@ -0,0 +1,60 @@
+import { defineConfig, devices } from "@playwright/test";
+
+/**
+ * Layer 2 of the record/replay e2e: the REAL Next.js frontend rendering data
+ * from a REAL gateway whose LLM is the deterministic `ReplayChatModel` (no API
+ * key). This is separate from `playwright.config.ts` (which mocks the backend)
+ * so the mock-based suite is untouched.
+ *
+ * Two webServers are started: the replay gateway (:8011) and the frontend
+ * (:3000, pointed at the gateway). Auth uses a throwaway test account the spec
+ * registers at runtime — no secrets.
+ */
+export default defineConfig({
+  testDir: "./tests/e2e-real-backend",
+  fullyParallel: false,
+  forbidOnly: !!process.env.CI,
+  retries: process.env.CI ? 1 : 0,
+  workers: 1,
+  reporter: process.env.CI ? "github" : "html",
+  timeout: 90_000,
+
+  use: {
+    baseURL: "http://localhost:3000",
+    trace: "on-first-retry",
+  },
+
+  projects: [{ name: "chromium", use: { ...devices["Desktop Chrome"] } }],
+
+  webServer: [
+    {
+      command: "uv run python scripts/run_replay_gateway.py --port 8011",
+      cwd: "../backend",
+      url: "http://localhost:8011/health",
+      reuseExistingServer: !process.env.CI,
+      timeout: 180_000,
+      stdout: "pipe",
+      stderr: "pipe",
+      // Mount the test-only run/message seeder used by multi-run-order.spec.ts
+      // (#3352). The endpoint exists only on this replay gateway, never in the
+      // production app.
+      env: { DEERFLOW_ENABLE_TEST_SEED: "1" },
+    },
+    {
+      command: "pnpm build && pnpm start",
+      url: "http://localhost:3000",
+      reuseExistingServer: !process.env.CI,
+      timeout: 240_000,
+      env: {
+        SKIP_ENV_VALIDATION: "1",
+        DEER_FLOW_AUTH_DISABLED: "1",
+        BETTER_AUTH_SECRET: "local-dev-secret",
+        // Leave NEXT_PUBLIC_* unset so the frontend uses its built-in
+        // next.config rewrites (same-origin proxy) instead of talking to the
+        // gateway cross-origin — cross-origin fetches drop the auth cookies.
+        // Just point that proxy at the replay gateway.
+        DEER_FLOW_INTERNAL_GATEWAY_BASE_URL: "http://127.0.0.1:8011",
+      },
+    },
+  ],
+});
@@ -0,0 +1,58 @@
+import { defineConfig, devices } from "@playwright/test";
+
+/**
+ * RECORD-through-browser config (Plan A): drive the REAL frontend against a
+ * REAL-model gateway and capture every model call so the fixture's inputs match
+ * exactly what the frontend produces. Manual, needs OPENAI_API_KEY/OPENAI_API_BASE
+ * + DEERFLOW_RECORD_OUT in the environment — never run in CI.
+ *
+ * Not committed as a test run; `tests/e2e-record/` holds the driver spec.
+ */
+export default defineConfig({
+  testDir: "./tests/e2e-record",
+  fullyParallel: false,
+  workers: 1,
+  reporter: "list",
+  timeout: 200_000,
+  use: { baseURL: "http://localhost:3000", trace: "off" },
+  projects: [{ name: "chromium", use: { ...devices["Desktop Chrome"] } }],
+  webServer: [
+    {
+      command: "uv run python scripts/record_gateway.py",
+      cwd: "../backend",
+      url: "http://localhost:8012/health",
+      reuseExistingServer: false,
+      timeout: 180_000,
+      stdout: "pipe",
+      stderr: "pipe",
+      env: {
+        RECORD_PORT: "8012",
+        RECORD_MODEL: process.env.RECORD_MODEL ?? "gpt-5.5",
+        // Forwarded from the invoking shell; never hardcoded. Passed through only
+        // when actually set, so record_gateway.py raises a clear "missing env"
+        // error instead of receiving "" (which would write to Path("")).
+        ...(process.env.DEERFLOW_RECORD_OUT
+          ? { DEERFLOW_RECORD_OUT: process.env.DEERFLOW_RECORD_OUT }
+          : {}),
+        ...(process.env.OPENAI_API_KEY
+          ? { OPENAI_API_KEY: process.env.OPENAI_API_KEY }
+          : {}),
+        ...(process.env.OPENAI_API_BASE
+          ? { OPENAI_API_BASE: process.env.OPENAI_API_BASE }
+          : {}),
+      },
+    },
+    {
+      command: "pnpm build && pnpm start",
+      url: "http://localhost:3000",
+      reuseExistingServer: false,
+      timeout: 240_000,
+      env: {
+        SKIP_ENV_VALIDATION: "1",
+        DEER_FLOW_AUTH_DISABLED: "1",
+        BETTER_AUTH_SECRET: "local-dev-secret",
+        DEER_FLOW_INTERNAL_GATEWAY_BASE_URL: "http://127.0.0.1:8012",
+      },
+    },
+  ],
+});
@@ -0,0 +1,2 @@
+# OS-specific Playwright visual baselines — generated locally, not committed
+*-snapshots/
@@ -0,0 +1,101 @@
+import { expect, test } from "@playwright/test";
+
+/**
+ * Layer 2 (cross-stack contract): reproduces upstream issue #3352 — after the
+ * checkpoint no longer holds the older messages (post context-compression), the
+ * frontend rebuilds thread history from the per-run endpoints, and the order it
+ * rebuilds them in must stay chronological.
+ *
+ * The dangerous class this guards: a BACKEND change to run ordering silently
+ * breaks a FRONTEND assumption. Backend `list_by_thread` returns runs
+ * NEWEST-FIRST (PR #2932); the pre-#3354 frontend iterated runs from the end and
+ * PREPENDED each loaded page (`core/threads/hooks.ts`), which inverts order. A
+ * backend-only ordering test was green the whole time #3352 was live, and the
+ * frontend regression unit test hardcodes "backend returns newest-first" in a
+ * mock — so only a real frontend against a real backend catches the desync.
+ *
+ * This drives the REAL frontend against a REAL gateway with two seeded runs and
+ * NO checkpoint (the seeder forces the per-run reload path to be the sole source
+ * of truth), then asserts the first run's message renders ABOVE the second's.
+ * No model, no recording, no API key — the runs are seeded via a test-only
+ * endpoint mounted only on the replay gateway.
+ */
+const APP = "http://localhost:3000";
+
+// Distinctive markers so getByText can't collide with UI chrome.
+const ALPHA = "ALPHA-FIRST-QUESTION-7f3a2c";
+const OMEGA = "OMEGA-SECOND-QUESTION-9b21d4";
+
+test.describe("multi-run thread renders chronologically (replay, no API key)", () => {
+  test("first run renders above second run after history rebuild (#3352)", async ({
+    page,
+    context,
+  }) => {
+    const uniq = `${Date.now()}-${Math.floor(Math.random() * 1e6)}`;
+    const threadId = `e2e-multi-run-${uniq}`;
+    const email = `e2e-${uniq}@example.com`;
+
+    // Register through the frontend origin (same-origin proxy) so the auth
+    // cookies are stored for localhost and forwarded to the gateway via the
+    // next.config rewrite — never cross-origin from the browser.
+    const reg = await context.request.post(`${APP}/api/v1/auth/register`, {
+      data: { email, password: "very-strong-password-123" },
+    });
+    expect(reg.status(), await reg.text()).toBe(201);
+
+    const cookies = await context.cookies();
+    const csrf = cookies.find((c) => c.name === "csrf_token")?.value;
+    expect(csrf, "register must set csrf_token cookie").toBeTruthy();
+
+    // Seed two runs in one thread: run-1 (ALPHA) older, run-2 (OMEGA) newer, so
+    // the real backend's list_by_thread returns them newest-first. No checkpoint
+    // is seeded — that is the #3352 precondition.
+    const seed = await context.request.post(`${APP}/api/test-only/seed-runs`, {
+      headers: { "X-CSRF-Token": csrf! },
+      data: {
+        thread_id: threadId,
+        runs: [
+          {
+            run_id: `${threadId}-r1`,
+            created_at: "2026-01-01T00:00:00+00:00",
+            messages: [
+              { role: "human", content: ALPHA, id: `${threadId}-a-h` },
+              { role: "ai", content: "ALPHA reply", id: `${threadId}-a-a` },
+            ],
+          },
+          {
+            run_id: `${threadId}-r2`,
+            created_at: "2026-01-01T00:01:00+00:00",
+            messages: [
+              { role: "human", content: OMEGA, id: `${threadId}-o-h` },
+              { role: "ai", content: "OMEGA reply", id: `${threadId}-o-a` },
+            ],
+          },
+        ],
+      },
+    });
+    expect(seed.status(), await seed.text()).toBe(200);
+
+    // Load the thread fresh — triggers useThreadHistory's per-run reload path.
+    await page.goto(`/workspace/chats/${threadId}`);
+
+    const alpha = page.getByText(ALPHA, { exact: false });
+    const omega = page.getByText(OMEGA, { exact: false });
+    await expect(alpha).toBeVisible({ timeout: 60_000 });
+    await expect(omega).toBeVisible({ timeout: 30_000 });
+    // Each marker renders exactly once (guards against accidental duplicate matches).
+    expect(await alpha.count(), "ALPHA should render exactly once").toBe(1);
+    expect(await omega.count(), "OMEGA should render exactly once").toBe(1);
+
+    // The contract: ALPHA (first run) must render ABOVE OMEGA (second run). With
+    // the #3352 bug the per-run rebuild inverts this and OMEGA renders first.
+    const alphaBox = await alpha.first().boundingBox();
+    const omegaBox = await omega.first().boundingBox();
+    expect(alphaBox, "ALPHA must have a layout box").toBeTruthy();
+    expect(omegaBox, "OMEGA must have a layout box").toBeTruthy();
+    expect(
+      alphaBox!.y,
+      `chronological order broken: ALPHA(first run) rendered at y=${alphaBox!.y}, OMEGA(second run) at y=${omegaBox!.y} — backend list_by_thread ordering and frontend history rebuild are out of sync (#3352)`,
+    ).toBeLessThan(omegaBox!.y);
+  });
+});
@@ -0,0 +1,123 @@
+import { readFileSync } from "node:fs";
+import { dirname, join } from "node:path";
+import { fileURLToPath } from "node:url";
+
+import { expect, test } from "@playwright/test";
+
+const here = dirname(fileURLToPath(import.meta.url));
+
+/**
+ * Layer 2: drive the REAL frontend against the REAL gateway (replay model, no
+ * API key) and assert the browser renders the backend's data correctly.
+ *
+ * The prompt is read from the same fixture the gateway replays, so the input
+ * hash matches and the recorded turns (write_file -> auto-title -> read_file ->
+ * final answer) reproduce deterministically.
+ */
+// Register through the frontend origin (same-origin proxy) so the auth cookies
+// are stored for and sent to localhost:3000 — the gateway is reached via the
+// next.config rewrite, never cross-origin from the browser.
+const APP = "http://localhost:3000";
+const fixture = JSON.parse(
+  readFileSync(
+    join(
+      here,
+      "../../../backend/tests/fixtures/replay/write_read_file.ultra.json",
+    ),
+    "utf-8",
+  ),
+) as {
+  prompt: string;
+  turns: Array<{ output: { data: { content?: unknown } } }>;
+};
+
+const PROMPT = fixture.prompt;
+// Derive the assertions from the fixture so a re-record auto-updates them. Both
+// are model-generated strings absent from the user prompt, so a pass proves the
+// replay drove the render (not a prompt echo): the first plain-text turn is the
+// in-graph auto-title; the JSON-array turn is the follow-up suggestions.
+const textTurns = fixture.turns
+  .map((t) => t.output?.data?.content)
+  .filter((c): c is string => typeof c === "string" && c.trim().length > 0);
+const suggestionsRaw = textTurns.find((c) => c.trim().startsWith("["));
+// Guarded parse: a bracket-prefixed turn that isn't a valid JSON string array
+// falls back to "" so the `not.toBe("")` assertion below fails with a clear
+// message instead of a generic JSON.parse throw.
+const EXPECTED_SUGGESTION = ((): string => {
+  if (!suggestionsRaw) return "";
+  try {
+    const arr: unknown = JSON.parse(suggestionsRaw);
+    return Array.isArray(arr) && typeof arr[0] === "string" ? arr[0] : "";
+  } catch {
+    return "";
+  }
+})();
+const EXPECTED_TITLE = textTurns.find((c) => !c.trim().startsWith("[")) ?? "";
+
+test.describe("real backend render (replay, no API key)", () => {
+  test.beforeEach(async ({ context }) => {
+    // Throwaway test account: register sets access_token + csrf_token cookies in
+    // the browser context (host-scoped to localhost, shared across ports), so
+    // the frontend's SDK (credentials:include + X-CSRF-Token) authenticates.
+    const email = `e2e-${Date.now()}-${Math.floor(Math.random() * 1e6)}@example.com`;
+    const resp = await context.request.post(`${APP}/api/v1/auth/register`, {
+      data: { email, password: "very-strong-password-123" },
+    });
+    expect(resp.status(), await resp.text()).toBe(201);
+  });
+
+  test("renders the replayed auto-title + suggestions from a real backend", async ({
+    page,
+  }) => {
+    // ultra mode so the context the frontend sends (is_plan_mode + subagent_enabled)
+    // matches the recorded fixture; otherwise the replay input hash would miss.
+    await page.addInitScript(() => {
+      window.localStorage.setItem(
+        "deerflow.local-settings",
+        JSON.stringify({ context: { mode: "ultra" } }),
+      );
+    });
+
+    await page.goto("/workspace/chats/new");
+
+    const textarea = page.getByPlaceholder(/how can i assist you/i);
+    await expect(textarea).toBeVisible({ timeout: 30_000 });
+    await textarea.fill(PROMPT);
+    await textarea.press("Enter");
+
+    // Replay-only DOM assertions (derived from the fixture): they render only if
+    // the recorded turns replayed AND the real frontend rendered them — the
+    // in-graph auto-title and the post-answer follow-up suggestion. Together they
+    // prove the whole pipeline (replay backend -> real frontend render).
+    expect(
+      EXPECTED_TITLE,
+      "fixture should contain an auto-title turn",
+    ).not.toBe("");
+    expect(
+      EXPECTED_SUGGESTION,
+      "fixture should contain a suggestions turn",
+    ).not.toBe("");
+    await expect(page.getByText(EXPECTED_TITLE)).toBeVisible({
+      timeout: 60_000,
+    });
+    await expect(page.getByText(EXPECTED_SUGGESTION)).toBeVisible({
+      timeout: 30_000,
+    });
+
+    // Visual regression is OS-sensitive (a macOS baseline won't match CI's
+    // Linux render), so it's a local dev gate only; in CI we capture the render
+    // as an artifact for human review instead of hard-asserting a cross-OS
+    // baseline. The DOM assertions above are the CI gate.
+    if (process.env.CI) {
+      await page.screenshot({
+        path: "test-results/real-backend-render.png",
+        fullPage: true,
+      });
+    } else {
+      await expect(page).toHaveScreenshot("real-backend-render.png", {
+        maxDiffPixelRatio: 0.02,
+        fullPage: true,
+      });
+    }
+  });
+});
@@ -0,0 +1,115 @@
+import { existsSync, readFileSync, writeFileSync } from "node:fs";
+
+import { expect, test } from "@playwright/test";
+
+/**
+ * RECORD driver (Plan A): drive the real frontend through the write/read-file
+ * scenario against the real-model gateway. The gateway captures every model
+ * call to DEERFLOW_RECORD_OUT; this just needs to drive the flow and wait until
+ * the captures stop arriving (main turns + in-graph title + follow-up
+ * suggestions all fired). It asserts nothing about content — it produces the
+ * fixture, it doesn't verify it.
+ */
+const APP = "http://localhost:3000";
+const SCENARIO = "write_read_file";
+const MODE = "ultra";
+const PROMPT =
+  "Using your own file tools directly, create the file /mnt/user-data/outputs/note.txt " +
+  "with exactly this content: hi from replay. Then read that same file back and reply with its " +
+  "exact contents. Do NOT delegate to a subagent and do NOT use the task tool — do it yourself. " +
+  "Do not ask any clarifying questions.";
+
+function countLines(path: string): number {
+  return existsSync(path)
+    ? readFileSync(path, "utf-8")
+        .split("\n")
+        .filter((l) => l.trim()).length
+    : 0;
+}
+
+async function waitForCaptureStable(
+  path: string,
+  { stableMs = 12_000, maxMs = 160_000 } = {},
+): Promise<number> {
+  const start = Date.now();
+  let last = -1;
+  let lastChange = Date.now();
+  while (Date.now() - start < maxMs) {
+    const n = countLines(path);
+    if (n !== last) {
+      last = n;
+      lastChange = Date.now();
+    } else if (n > 0 && Date.now() - lastChange > stableMs) {
+      return n;
+    }
+    await new Promise((r) => setTimeout(r, 1000));
+  }
+  // Hard failure on timeout: returning the last count here would let a
+  // truncated/partial recording pass silently (captured > 0). A recording must
+  // stabilize, or it is not trustworthy.
+  throw new Error(
+    `[record] captures never stabilized within ${maxMs}ms (last count=${last}); ` +
+      `the recording may be truncated — raise maxMs or check the record gateway.`,
+  );
+}
+
+test.describe.configure({ timeout: 220_000 });
+
+test("record write/read-file run through the real frontend", async ({
+  page,
+  context,
+}) => {
+  const out = process.env.DEERFLOW_RECORD_OUT;
+  expect(out, "DEERFLOW_RECORD_OUT must be set").toBeTruthy();
+  // The context the frontend derives for ultra mode (core/threads/hooks.ts). The
+  // backend-direct golden test (Layer 1) POSTs this so its prompt — hence the
+  // recorded input hashes — matches the browser run. thinking/reasoning don't
+  // affect the prompt; is_plan_mode + subagent_enabled add the todo/task tools.
+  const CONTEXT = {
+    is_bootstrap: false,
+    mode: MODE,
+    thinking_enabled: true,
+    is_plan_mode: true,
+    subagent_enabled: true,
+  };
+  writeFileSync(
+    `${out}.meta.json`,
+    JSON.stringify({
+      scenario: SCENARIO,
+      mode: MODE,
+      prompt: PROMPT,
+      context: CONTEXT,
+    }),
+    "utf-8",
+  );
+
+  const reg = await context.request.post(`${APP}/api/v1/auth/register`, {
+    data: {
+      email: `rec-${Date.now()}@example.com`,
+      password: "very-strong-password-123",
+    },
+  });
+  expect(reg.status(), await reg.text()).toBe(201);
+
+  await page.addInitScript(() => {
+    window.localStorage.setItem(
+      "deerflow.local-settings",
+      JSON.stringify({ context: { mode: "ultra" } }),
+    );
+  });
+  await page.goto("/workspace/chats/new");
+
+  const textarea = page.getByPlaceholder(/how can i assist you/i);
+  await expect(textarea).toBeVisible({ timeout: 30_000 });
+  await textarea.fill(PROMPT);
+  await textarea.press("Enter");
+
+  const captured = await waitForCaptureStable(out!);
+  console.log(
+    `[record] captures stabilized at ${captured} model call(s) -> ${out}`,
+  );
+  expect(
+    captured,
+    "expected at least the agent turns to be captured",
+  ).toBeGreaterThan(0);
+});