fix(replay-e2e): match by conversation, not the living system prompt (#3436)

* fix(replay-e2e): match by conversation, not the living system prompt

The model-replay match key hashed the full input including the lead-agent
system prompt. That prompt is edited frequently (e.g. #3195 added a "File
Editing Workflow" section), so the committed fixture went stale the moment
the prompt changed on main — turning the Layer-2 render gate RED on every
unrelated PR (#3430, #3432, ...). This was a self-inflicted false positive.

Root-cause fix:
- replay_provider._canonical_messages now EXCLUDES the system message from
  the hash. The conversation (human/ai/tool) is the stable contract that
  identifies a recorded turn; the system prompt is an internal detail not
  part of the front-back contract under test. (Mirrors how open-design keys
  its mock picker on the user prompt, not the system internals.) Proven
  robust: injecting a prompt edit no longer causes a replay miss.
- Layer-1 golden was BLIND to replay misses: the gateway swallows a miss
  into an assistant error message, so the shape-only golden stayed green on
  a stale fixture. It now inspects replay_provider.replay_misses() and fails
  loud. (Layer-2 already fails on a miss.)
- Re-recorded write_read_file.ultra fixture + regenerated golden under the
  new conversation-only hash.
- Layer-2 render spec: assert the in-graph auto-title (deterministic); the
  follow-up suggestion is fired async and depends on a clean JSON model
  output, so assert it only when the fixture captured one — never gate on
  its absence (recording flakiness must not block CI).
- docs: REPLAY_E2E.md updated.

Verified: Layer-1 golden green (no miss), Layer-2 both specs green,
CI=true make test 4033 passed / 0 failed, frontend pnpm check clean.

* test(replay-e2e): restore suggestions coverage with a reliable capture

Addresses review feedback (the suggestion path was dropped from Layer-2):

- record spec now waits for the `/suggestions` response before checking
  capture stability, so the recorded fixture reliably includes the
  frontend-fired suggestions turn (previously the stability window could
  return before suggestions fired, yielding a fixture without it).
- Re-recorded write_read_file.ultra: 5 turns (write_file, auto-title,
  read_file, answer, suggestions). Golden unchanged — suggestions is a
  separate /suggestions call, not part of the /runs/stream SSE sequence.
- Layer-2 spec: restore the hard `EXPECTED_SUGGESTION` assertion. With the
  record spec now waiting for /suggestions, a fixture missing the suggestion
  turn means a broken recording and must fail loud, not pass silently.

Verified: Layer-1 golden green (no miss), Layer-2 both specs green
(auto-title + suggestion render), frontend pnpm check clean.

* ci: re-trigger (flaky Docker Hub image pull in sandbox e2e, unrelated)

backend-unit-tests failed only in test_sandbox_orphan_reconciliation_e2e.py
with 'docker pull busybox:latest ... context deadline exceeded' — a CI-runner
network flake reaching Docker Hub, not related to this docs/tests-only change.
Empty commit to re-run CI.

---------

Co-authored-by: DanielWalnut <45447813+hetaoBackend@users.noreply.github.com>

backend/tests/test_replay_golden.py

@@ -66,14 +66,24 @@ def test_replay_write_read_file_ultra_matches_golden(tmp_path: Path, monkeypatch
     cfg = app_config_module.get_app_config()
     cfg.database.sqlite_dir = str(home / "db")
 
+    # Fail loud on a replay miss. The gateway swallows a hash-miss into a normal
+    # assistant error message, so the SSE *shapes* below stay green on a stale
+    # fixture — the miss list is the only reliable signal at this layer.
+    import replay_provider
+
     from app.gateway.app import create_app
 
+    replay_provider.reset_replay_misses()
+
     events = drive_gateway(create_app(), prompt=fixture["prompt"], context=fixture["context"])
 
     assert events, "replay produced no SSE events"
     assert events[0]["event"] == "metadata", f"first event should be metadata, got {events[0]!r}"
     assert events[-1]["event"] == "end", f"last event should be end (run completed), got {events[-1]!r}"
 
+    misses = replay_provider.replay_misses()
+    assert not misses, f"replay miss ({len(misses)}): the fixture is stale vs the current system prompt or agent graph. Re-record it (see backend/docs/REPLAY_E2E.md). Missed hashes: {misses}"
+
     # Regenerate the committed golden after re-recording the fixture:
     #   DEERFLOW_WRITE_GOLDEN=1 uv run pytest tests/test_replay_golden.py
     if os.environ.get("DEERFLOW_WRITE_GOLDEN"):
@@ -81,7 +91,7 @@ def test_replay_write_read_file_ultra_matches_golden(tmp_path: Path, monkeypatch
         return
 
     golden = json.loads(events_path.read_text(encoding="utf-8"))["events"]
-    # A replay hash-miss surfaces as the run erroring mid-stream -> the event
-    # shape sequence diverges from the golden, so this assertion is the catch-all
-    # for both backend SSE drift and replay divergence.
+    # Guards backend SSE protocol drift: the event name + payload-key sequence
+    # must match the committed golden. (Replay divergence is caught by the miss
+    # assertion above, not here — a swallowed miss keeps the shapes identical.)
     assert events == golden, f"SSE event-shape sequence drifted from the golden.\ngot  ({len(events)}): {[e['event'] for e in events]}\nwant ({len(golden)}): {[e['event'] for e in golden]}"