fix(replay-e2e): match by conversation, not the living system prompt (#3436)

* fix(replay-e2e): match by conversation, not the living system prompt

The model-replay match key hashed the full input including the lead-agent
system prompt. That prompt is edited frequently (e.g. #3195 added a "File
Editing Workflow" section), so the committed fixture went stale the moment
the prompt changed on main — turning the Layer-2 render gate RED on every
unrelated PR (#3430, #3432, ...). This was a self-inflicted false positive.

Root-cause fix:
- replay_provider._canonical_messages now EXCLUDES the system message from
  the hash. The conversation (human/ai/tool) is the stable contract that
  identifies a recorded turn; the system prompt is an internal detail not
  part of the front-back contract under test. (Mirrors how open-design keys
  its mock picker on the user prompt, not the system internals.) Proven
  robust: injecting a prompt edit no longer causes a replay miss.
- Layer-1 golden was BLIND to replay misses: the gateway swallows a miss
  into an assistant error message, so the shape-only golden stayed green on
  a stale fixture. It now inspects replay_provider.replay_misses() and fails
  loud. (Layer-2 already fails on a miss.)
- Re-recorded write_read_file.ultra fixture + regenerated golden under the
  new conversation-only hash.
- Layer-2 render spec: assert the in-graph auto-title (deterministic); the
  follow-up suggestion is fired async and depends on a clean JSON model
  output, so assert it only when the fixture captured one — never gate on
  its absence (recording flakiness must not block CI).
- docs: REPLAY_E2E.md updated.

Verified: Layer-1 golden green (no miss), Layer-2 both specs green,
CI=true make test 4033 passed / 0 failed, frontend pnpm check clean.

* test(replay-e2e): restore suggestions coverage with a reliable capture

Addresses review feedback (the suggestion path was dropped from Layer-2):

- record spec now waits for the `/suggestions` response before checking
  capture stability, so the recorded fixture reliably includes the
  frontend-fired suggestions turn (previously the stability window could
  return before suggestions fired, yielding a fixture without it).
- Re-recorded write_read_file.ultra: 5 turns (write_file, auto-title,
  read_file, answer, suggestions). Golden unchanged — suggestions is a
  separate /suggestions call, not part of the /runs/stream SSE sequence.
- Layer-2 spec: restore the hard `EXPECTED_SUGGESTION` assertion. With the
  record spec now waiting for /suggestions, a fixture missing the suggestion
  turn means a broken recording and must fail loud, not pass silently.

Verified: Layer-1 golden green (no miss), Layer-2 both specs green
(auto-title + suggestion render), frontend pnpm check clean.

* ci: re-trigger (flaky Docker Hub image pull in sandbox e2e, unrelated)

backend-unit-tests failed only in test_sandbox_orphan_reconciliation_e2e.py
with 'docker pull busybox:latest ... context deadline exceeded' — a CI-runner
network flake reaching Docker Hub, not related to this docs/tests-only change.
Empty commit to re-run CI.

---------

Co-authored-by: DanielWalnut <45447813+hetaoBackend@users.noreply.github.com>

backend/tests/replay_provider.py

@@ -76,6 +76,24 @@ from pydantic import PrivateAttr
 
 _FIXTURE_ENV = "DEERFLOW_REPLAY_FIXTURE"
 
+# Process-wide record of replay misses. A miss raises inside the model, but the
+# gateway's LLMErrorHandlingMiddleware swallows it into a normal assistant error
+# message — so the SSE *event shapes* are unchanged and a shape-only golden stays
+# green on a stale fixture. The in-process Layer-1 test inspects this list to fail
+# loud on a miss instead. (Layer-2 already fails on a miss: the recorded turns
+# never render.)
+_replay_misses: list[str] = []
+
+
+def replay_misses() -> list[str]:
+    """Hashes that missed the fixture since the last reset (see ``_replay_misses``)."""
+    return list(_replay_misses)
+
+
+def reset_replay_misses() -> None:
+    _replay_misses.clear()
+
+
 # Volatile substrings that differ between a recording run and a replay run but
 # carry no semantic weight for matching. Normalized to stable placeholders
 # before hashing so the same logical input hashes identically across processes.
@@ -117,13 +135,24 @@ def _content_to_text(content: Any) -> str:
 def _canonical_messages(messages: list[BaseMessage]) -> str:
     """Project messages to a stable shape that excludes volatile metadata/ids.
 
-    Keeps only what determines the model's next output: role, text content, and
-    tool-call name+args. Drops ``id``, ``response_metadata``, ``usage_metadata``,
-    and ``tool_call_id`` (all volatile), then normalizes embedded volatile
-    substrings.
+    Keeps only what determines which recorded turn to replay: the conversation
+    (human / ai / tool messages — role, text content, tool-call name+args). Drops
+    ``id``, ``response_metadata``, ``usage_metadata``, ``tool_call_id`` (all
+    volatile), then normalizes embedded volatile substrings.
+
+    **The system message is excluded entirely.** The lead-agent system prompt is
+    a living, frequently-edited implementation detail (its wording changes across
+    PRs), not part of the front-back contract this harness verifies. Hashing it
+    would make every fixture go stale — and red-fail on unrelated PRs — the moment
+    anyone edits the prompt. The conversation flow (user input -> tool calls ->
+    results -> answer) is the stable key that identifies a recorded turn.
     """
     projected: list[dict[str, Any]] = []
     for message in messages:
+        # Exclude the system prompt from the match key — see docstring. It is the
+        # most-edited part of the prompt and not part of the contract under test.
+        if message.type == "system":
+            continue
         content = _normalize_text(_content_to_text(message.content))
         tool_calls = getattr(message, "tool_calls", None)
         # Drop messages that are empty after normalization — e.g. a turn that was
@@ -189,6 +218,7 @@ class ReplayChatModel(BaseChatModel):
         key = hash_messages(messages)
         bucket = self._table.get(key)
         if not bucket:
+            _replay_misses.append(key)
             preview = _canonical_messages(messages)
             raise KeyError(
                 f"replay miss: no recorded output for input hash {key} in {self._fixture_path!r}. "
@@ -227,4 +257,4 @@ class ReplayChatModel(BaseChatModel):
 
 
 # Re-export so the recorder shares the exact hashing logic.
-__all__ = ["ReplayChatModel", "hash_messages"]
+__all__ = ["ReplayChatModel", "hash_messages", "replay_misses", "reset_replay_misses"]