mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-06-10 09:25:57 +00:00
Xinmin Zeng
88759015e4
* test(e2e): record/replay front-back contract verification Guards the front-back contract with a deterministic, key-free record/replay harness (mirrors open-design's golden-trace approach): - ReplayChatModel (tests/replay_provider.py): replays recorded LLM turns by a normalized hash of the model input. Strips <system-reminder>/date/uuid/tmp-path so one fixture replays across days and from both the browser and direct-POST paths; a miss raises loudly (no silent divergence). - Recording is record-through-browser (scripts/record_gateway.py + build_fixture_from_jsonl.py + frontend/tests/e2e-record): a real run is driven through the real frontend so captured inputs match exactly what the browser sends; fixtures contain no API key. - Layer 1 — backend golden (tests/test_replay_golden.py): replay through the real gateway, assert the SSE event sequence == committed golden. - Layer 2 — full-stack render (frontend/tests/e2e-real-backend): real Next.js + real gateway (replay model) + Chromium; assert the replayed auto-title and follow-up suggestions render. DOM assertions are the gate; visual regression is a local dev gate (CI uploads the render as an artifact). - CI (.github/workflows/replay-e2e.yml): both layers, triggered on EITHER side of the contract (frontend/** or backend gateway/harness/fixtures). * test(e2e): multi-run render-order cross-stack scenario (#3352) Guards the dangerous front-back class where a backend ordering change silently breaks a frontend assumption while both sides' unit tests stay green. Reproduces issue #3352: backend list_by_thread returns runs newest-first (#2932) and the frontend prepended per-run pages, inverting chronological order once the checkpoint no longer held the older messages. - tests/seed_runs_router.py: test-only seeder, mounted on the replay gateway only when DEERFLOW_ENABLE_TEST_SEED=1 (never in the production app). Seeds a thread with >=2 runs + per-run message events and no checkpoint -- the #3352 precondition -- so the frontend per-run reload path is the sole source of truth and the prepend inversion is observable. - frontend/tests/e2e-real-backend/multi-run-order.spec.ts: drives the real frontend against the real gateway, asserts the first run renders above the second. Reverting the #3354 fix turns it red. - replay-e2e.yml: trigger on the new replay test-infra paths. - docs: REPLAY_E2E.md cross-stack scenario section. * test(e2e): address Copilot review on the replay harness - Fix stale recorder references (scripts/record_traces.py -> scripts/record_gateway.py + scripts/build_fixture_from_jsonl.py) in replay_provider.py, test_replay_golden.py, _replay_fixture.py. - MODE_CONTEXT['ultra']: thinking_enabled False -> True, mirroring the frontend's `context.mode !== 'flash'` (hooks.ts). It did not affect the hashed input (Layer 1 golden still green), but the table now matches the real frontend context it claims to mirror. - replay_provider.py docstring: stop claiming memory is recorded-enabled; the replay config disables memory/summarization for determinism (title stays, as an in-graph deterministic call). - record_gateway.py / run_replay_gateway.py: override DEER_FLOW_HOME instead of setdefault, so an outer value can't leak into the hermetic harness. - record_gateway.py: clear error when DEERFLOW_RECORD_OUT is unset (was a bare KeyError). - playwright.record.config.ts: forward OPENAI_*/DEERFLOW_RECORD_OUT only when set, so the gateway raises a clear 'missing env' error instead of getting ''. * test(e2e): address Copilot review round 2 - seed_runs_router.py: constrain SeedMessage.role to Literal['human','ai'] so a bad value is a clean 422 at the boundary instead of a 500 (KeyError on _EVENT_TYPE). - record-write-read-file.spec.ts: waitForCaptureStable now throws on timeout instead of returning the last count, so a truncated/partial recording can't pass silently. - real-backend-render.spec.ts: guard the suggestions JSON.parse; a bracket-prefixed non-JSON turn falls back to '' so the existing not.toBe('') assertion fails clearly instead of a generic parse throw.
104 lines
5.0 KiB
Markdown
104 lines
5.0 KiB
Markdown
# Record/Replay E2E — front-back contract verification
|
|
|
|
Deterministic, **key-free** end-to-end checks that a backend change can't
|
|
silently break the frontend (and vice-versa). Two complementary layers, fed by a
|
|
single recording.
|
|
|
|
## Why
|
|
|
|
The mock-based frontend e2e hand-writes the backend's JSON/SSE, so a backend
|
|
schema or SSE change passes green ("fake green"). These layers replay a recorded
|
|
**real** run against the **real** backend (and, for Layer 2, the real frontend),
|
|
so contract drift turns the build red instead.
|
|
|
|
## The two layers
|
|
|
|
- **Layer 1 — backend golden** (`tests/test_replay_golden.py`): replays a fixture
|
|
through the real FastAPI gateway with `ReplayChatModel` and asserts the streamed
|
|
SSE event sequence equals a committed golden. Fast, no browser. Guards protocol
|
|
*shape*.
|
|
- **Layer 2 — full-stack render** (`frontend/tests/e2e-real-backend/`): real
|
|
Next.js + real gateway (replay model) + Chromium; asserts the replayed
|
|
auto-title and a follow-up suggestion render in the browser. Guards semantic
|
|
*render*. (Complementary to Layer 1 — neither subsumes the other.)
|
|
|
|
Layer 2 also hosts **cross-stack contract scenarios** — the dangerous class
|
|
where a backend change silently breaks a frontend assumption and *both sides'
|
|
unit tests stay green*. See below.
|
|
|
|
## Cross-stack scenario: multi-run render order (`multi-run-order.spec.ts`)
|
|
|
|
Regression guard for issue **#3352** (after context compression, refreshing a
|
|
thread rendered history out of order). Root cause was a front-back desync:
|
|
backend `RunManager.list_by_thread` returns runs **newest-first** (PR #2932),
|
|
while the frontend (`core/threads/hooks.ts`) iterated runs and **prepended** each
|
|
loaded page — inverting chronological order once the checkpoint no longer held
|
|
the older messages. The backend ordering test was green throughout, and the
|
|
frontend regression unit test hardcodes "backend returns newest-first" in a mock,
|
|
so only a *real frontend against a real backend* catches the desync.
|
|
|
|
This scenario does **not** record a conversation. It uses a **test-only seeder**
|
|
(`tests/seed_runs_router.py`, mounted on the replay gateway only when
|
|
`DEERFLOW_ENABLE_TEST_SEED=1`) to stand up a thread with ≥2 runs and per-run
|
|
message events — and deliberately **no checkpoint**, which is the #3352
|
|
precondition: it forces the frontend's per-run reload path to be the sole source
|
|
of truth so the ordering bug becomes observable. The seeder writes through the
|
|
gateway's own run/event stores using the request's auth context, so the real
|
|
`list_by_thread` → `/runs/{id}/messages` → prepend path runs live. Reverting the
|
|
#3354 frontend fix turns this spec red.
|
|
|
|
## How replay works
|
|
|
|
`tests/replay_provider.py::ReplayChatModel` returns recorded assistant turns keyed
|
|
by a **normalized hash** of the model input (strips `<system-reminder>`, dates,
|
|
UUIDs, tmp paths). A miss raises loudly rather than passing silently. The system
|
|
prompt is made environment-independent by pinning skills + extensions empty and
|
|
disabling memory/summarization (`tests/_replay_fixture.py::build_config_yaml`), so
|
|
a fixture replays the same across machines, days, and CI. Replaying needs **no
|
|
API key**.
|
|
|
|
## Record a new scenario (needs a real key — dev machine only)
|
|
|
|
Recording drives the **real frontend** so captured inputs match exactly what the
|
|
browser sends; fixtures contain no API key.
|
|
|
|
```bash
|
|
# 1. drive the real frontend against a real-model gateway, capturing model calls
|
|
OPENAI_API_KEY=... OPENAI_API_BASE=<openai-compatible-endpoint>/v1 \
|
|
DEERFLOW_RECORD_OUT=/tmp/rec/turns.jsonl RECORD_MODEL=<model> \
|
|
bash -c 'cd frontend && pnpm exec playwright test -c playwright.record.config.ts'
|
|
|
|
# 2. stitch the capture into a fixture
|
|
cd backend && uv run python scripts/build_fixture_from_jsonl.py \
|
|
--jsonl /tmp/rec/turns.jsonl --meta /tmp/rec/turns.jsonl.meta.json \
|
|
--out tests/fixtures/replay/<scenario>.<mode>.json --model <model>
|
|
|
|
# 3. regenerate the committed golden
|
|
DEERFLOW_WRITE_GOLDEN=1 PYTHONPATH=. uv run pytest tests/test_replay_golden.py
|
|
```
|
|
|
|
## Run (no key)
|
|
|
|
```bash
|
|
cd backend && PYTHONPATH=. uv run pytest tests/test_replay_golden.py # Layer 1
|
|
cd frontend && pnpm exec playwright test -c playwright.real-backend.config.ts # Layer 2
|
|
```
|
|
|
|
## CI
|
|
|
|
`.github/workflows/replay-e2e.yml` runs both layers on changes to **either** side
|
|
of the contract (`frontend/**`, `backend/app/gateway/**`,
|
|
`backend/packages/harness/**`, fixtures). DOM assertions are the gate; the rendered
|
|
screenshot + Playwright HTML report are uploaded as a CI artifact.
|
|
|
|
## Known limitations
|
|
|
|
- Visual regression baselines are OS-specific, so they are a **local dev gate
|
|
only** (gitignored); CI uploads the render as an artifact for human review
|
|
instead of hard-asserting a cross-OS baseline.
|
|
- Fixtures are coupled to the recording-time prompt; if new
|
|
environment-dependent content enters the system prompt, extend the
|
|
normalization in `replay_provider.py` (or pin it in `build_config_yaml`).
|
|
- Re-record a scenario if the agent graph changes how many model calls it makes
|
|
— the replay raises loudly on a hash miss pointing at the divergence.
|