fix(dreaming): drop heartbeat assistant responses from dream corpus (#109403)

* fix(dreaming): drop heartbeat assistant responses from dream corpus

The dreaming ingestion pipeline filters heartbeat user messages
(containing [OpenClaw heartbeat poll]) via sanitizeSessionText, but the
paired assistant response is only dropped when it is the exact string
HEARTBEAT_OK. Local models frequently respond with natural-language
acknowledgments (e.g. "Heartbeat received. Main is active.") which pass
through the sanitizer unchanged and enter the dream corpus as
low-confidence (0.58) memory snippets.

Fix: track heartbeat user message drops in buildSessionEntry and skip
the immediately following assistant response. This cross-message coupling
is safe because the heartbeat prompt pattern is injected by the runtime
and cannot be spoofed by user input, unlike [cron:...] or
System (untrusted): ... patterns (see PR #70737).

Refs: #103720

* test(dreaming): add heartbeat assistant response filter test

* fix(dreaming): use provenance-based heartbeat detection instead of text matching

Replace text-based heartbeat detection (isGeneratedHeartbeatPromptMessage)
with provenance-based authentication. The heartbeat turn now carries
provenance: { kind: "heartbeat" } when persisted to the transcript,
and buildSessionEntry checks for this provenance instead of matching
user-spoofable text content.

Changes:
- src/sessions/input-provenance.ts: Add "heartbeat" to
  INPUT_PROVENANCE_KIND_VALUES
- src/auto-reply/reply/get-reply-run.ts: Attach heartbeat provenance
  to user turn input when isHeartbeat is true
- packages/memory-host-sdk/src/host/session-files.ts: Check
  message.provenance.kind === "heartbeat" instead of text matching
- packages/memory-host-sdk/src/host/session-files.test.ts: Update
  heartbeat test to include provenance; add lookalike regression test

This addresses the ClawSweeper review finding [P1] about user-spoofable
text matching. The cross-message coupling is now authenticated by
runtime provenance, not user-typed content.

Refs: #103720

* fix(dreaming): add targeted heartbeat-derived corpus repair

* fix(dreaming): fix TS return type for clearScopedLegacySessionIngestionJson catch

* fix(dreaming): match chunked SQLite seen-state keys by stored scope

* fix(dreaming): repair SQLite-backed session transcript lookup in doctor

The findHeartbeatContaminatedCorpusLines function could not read
SQLite-backed session transcripts. When a corpus ref had no .jsonl
extension (SQLite logical path format), it appended .jsonl, read an
empty file, and silently skipped — leaving pre-fix heartbeat
contamination intact for SQLite users.

Fix:
- Add loadTranscriptLinesFromSqlite helper that reads events via
  the canonical loadTranscriptEventsSync API
- Serialize all events to preserve original event positions for
  corpus #L<n> line-number references
- Fall back to SQLite reader when filesystem read fails on a
  non-.jsonl corpus ref path

Test: add SQLite-backed session regression test that seeds a
session, writes corpus refs without .jsonl extension, and
verifies both audit detection and targeted repair.

* fix(dreaming): address autoreview P1 and P2 for checkpoint clearing and early return

P1: Remove clearScopedSessionIngestionState call after heartbeat line
removal. Clearing the cursor causes the next ingestion to re-process the
transcript from the beginning, duplicating normal corpus lines that were
deliberately retained.

P2: Remove early return after heartbeat cleanup so the self-ingestion
narrative content check and archiveDiary request still run.

* fix(dreaming): resolve corpus ref session path without duplicate agent ID

The session path in corpus refs is already in the format
'sessions/main/abc.jsonl' (includes agent subdirectory). The
previous code prepended source.agentId again, creating a path
like 'agents/main/sessions/main/abc.jsonl' — which silently
returned empty in production workspaces.

Fix: use path.basename to extract just the filename from the
session path, then construct the transcript path correctly.

* refactor(dreaming): split repair utils into separate file to satisfy LOC ratchet

The dreaming-repair.ts file grew from 337 to 831 lines, exceeding the
500-line LOC ratchet limit. Split helper utilities into a new
dreaming-repair-utils.ts file:

- dreaming-repair.ts (365 lines): imports, types, public API functions
  (auditDreamingArtifacts, repairDreamingArtifacts)
- dreaming-repair-utils.ts (498 lines): all helper functions, constants,
  and internal types

* fix(ts): add missing type imports for return types in dreaming-repair.ts

* fix(deadcode): remove unused exports from dreaming-repair-utils.ts

* fix: resolve merge conflict marker in get-reply-run.ts

* fix: remove unnecessary export from INPUT_PROVENANCE_KIND_VALUES

* fix(dreaming): converge targeted repair path with wholesale archive+clear

When heartbeat contamination is found, archive the entire session-corpus directory and clear all SQLite checkpoints instead of doing targeted line-by-line rewrite. This ensures the cleaned corpus gets re-ingested under the new provenance-based forward filter.

* fix(dreaming): authenticate heartbeat transcript turns

Co-authored-by: Erick Kinnee <1707617+ekinnee@users.noreply.github.com>

---------

Co-authored-by: Erick Kinnee <erick@ekinnee.dev>
Co-authored-by: Erick Kinnee <ekinnee@gmail.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>

packages/memory-host-sdk/src/host/session-files.test.ts

@@ -904,6 +904,118 @@ describe("buildSessionEntry", () => {
     expect(entry.lineMap).toStrictEqual([2, 3]);
   });
 
+  it("drops every assistant response in a provenance-marked heartbeat turn", async () => {
+    const jsonlLines = [
+      JSON.stringify({
+        type: "message",
+        message: {
+          role: "user",
+          content: "[OpenClaw heartbeat poll]",
+          provenance: { kind: "internal_system", sourceTool: "heartbeat" },
+        },
+      }),
+      JSON.stringify({
+        type: "message",
+        message: {
+          role: "assistant",
+          content: "Heartbeat received. Main is active. No pending user request in this cron poll.",
+        },
+      }),
+      JSON.stringify({
+        type: "message",
+        message: { role: "toolResult", content: "Background check complete." },
+      }),
+      JSON.stringify({
+        type: "message",
+        message: { role: "assistant", content: "One maintenance task was also completed." },
+      }),
+      JSON.stringify({
+        type: "message",
+        message: {
+          role: "user",
+          content: "Internal handoff.",
+          provenance: { kind: "inter_session", sourceTool: "sessions_send" },
+        },
+      }),
+      JSON.stringify({
+        type: "message",
+        message: { role: "assistant", content: "Cross-session response." },
+      }),
+      JSON.stringify({
+        type: "message",
+        message: { role: "user", content: "What is the weather today?" },
+      }),
+      JSON.stringify({
+        type: "message",
+        message: { role: "assistant", content: "The weather is sunny." },
+      }),
+    ];
+    const filePath = path.join(tmpDir, "heartbeat-session.jsonl");
+    fsSync.writeFileSync(filePath, jsonlLines.join("\n"));
+
+    const entry = requireSessionEntry(await buildSessionEntry(filePath));
+    expect(entry.content).toBe(
+      "Assistant: Cross-session response.\nUser: What is the weather today?\nAssistant: The weather is sunny.",
+    );
+    expect(entry.lineMap).toStrictEqual([6, 7, 8]);
+  });
+
+  it("does not couple user-spoofed heartbeat text to the next assistant response", async () => {
+    const jsonlLines = [
+      JSON.stringify({
+        type: "message",
+        message: {
+          role: "user",
+          content: "[OpenClaw heartbeat poll]",
+        },
+      }),
+      JSON.stringify({
+        type: "message",
+        message: {
+          role: "assistant",
+          content: "This reply belongs to a real user turn.",
+        },
+      }),
+    ];
+    const filePath = path.join(tmpDir, "normal-session.jsonl");
+    fsSync.writeFileSync(filePath, jsonlLines.join("\n"));
+
+    const entry = requireSessionEntry(await buildSessionEntry(filePath));
+    expect(entry.content).toBe("Assistant: This reply belongs to a real user turn.");
+    expect(entry.lineMap).toStrictEqual([2]);
+  });
+
+  it("ends a heartbeat turn when the next real user message has no text", async () => {
+    const jsonlLines = [
+      JSON.stringify({
+        type: "message",
+        message: {
+          role: "user",
+          content: "[OpenClaw heartbeat poll]",
+          provenance: { kind: "internal_system", sourceTool: "heartbeat" },
+        },
+      }),
+      JSON.stringify({
+        type: "message",
+        message: { role: "assistant", content: "Heartbeat received." },
+      }),
+      JSON.stringify({
+        type: "message",
+        message: { role: "user", content: [{ type: "image", source: "photo.jpg" }] },
+      }),
+      JSON.stringify({
+        type: "message",
+        message: { role: "assistant", content: "I can see the photo." },
+      }),
+    ];
+    const filePath = path.join(tmpDir, "heartbeat-before-media-session.jsonl");
+    fsSync.writeFileSync(filePath, jsonlLines.join("\n"));
+
+    const entry = requireSessionEntry(await buildSessionEntry(filePath));
+    expect(entry.content).toBe("Assistant: I can see the photo.");
+    expect(entry.lineMap).toStrictEqual([4]);
+  });
+
   it("drops Date-invalid numeric message timestamps", async () => {
     const jsonlLines = [
       JSON.stringify({

packages/memory-host-sdk/src/host/session-files.ts

@@ -802,6 +802,9 @@ export async function buildSessionEntry(
       false;
     const allowArchiveRecordCronClassification =
       isUsageCountedSessionArchiveTranscriptPath(absPath);
+    // A heartbeat owns every generated response until the next user turn. The
+    // persisted runtime provenance makes this coupling safe from text spoofing.
+    let insideHeartbeatTurn = false;
     for (let jsonlIdx = 0, lineStart = 0; lineStart <= raw.length; jsonlIdx++) {
       await yieldSessionEntryParseIfNeeded(jsonlIdx, parseYieldEveryLines);
       const newlineIndex = raw.indexOf("\n", lineStart);
@@ -846,6 +849,14 @@ export async function buildSessionEntry(
       if (message.role !== "user" && message.role !== "assistant") {
         continue;
       }
+      const provenance = message.provenance as { kind?: unknown; sourceTool?: unknown } | undefined;
+      const isHeartbeatUser =
+        message.role === "user" &&
+        provenance?.kind === "internal_system" &&
+        provenance.sourceTool === "heartbeat";
+      if (message.role === "user") {
+        insideHeartbeatTurn = isHeartbeatUser;
+      }
       if (message.role === "user" && hasInterSessionUserProvenance(message)) {
         continue;
       }
@@ -853,17 +864,14 @@ export async function buildSessionEntry(
       if (rawText === null) {
         continue;
       }
+
       // User text is not trusted archive-wide provenance. Per-message sanitization
       // drops cron prompts without clearing unrelated content from the archive.
       const text = sanitizeSessionText(rawText, message.role);
       if (!text) {
-        // Assistant-side machinery (silent replies, system wrappers) is already
-        // dropped by sanitizeSessionText. We deliberately do NOT use the prior
-        // user message's pattern-match to drop the next assistant message:
-        // user-typed text can match those same patterns (`[cron:...]`,
-        // `System (untrusted): ...`) and a cross-message drop would let users
-        // exfiltrate real assistant replies from the dreaming corpus by
-        // prefixing their own prompt. See PR #70737 review (aisle-research-bot).
+        continue;
+      }
+      if (insideHeartbeatTurn) {
         continue;
       }
       if (generatedByDreamingNarrative || generatedByCronRun) {

src/auto-reply/reply/get-reply-run.media-only.test.ts

@@ -2801,6 +2801,9 @@ describe("runPreparedReply media-only handling", () => {
     expect(call?.followupRun.prompt).toContain(heartbeatPrompt);
     expect(call?.transcriptCommandBody).toBe("[OpenClaw heartbeat poll]");
     expect(call?.followupRun.transcriptPrompt).toBe("[OpenClaw heartbeat poll]");
+    expect(call?.followupRun.userTurnTranscriptRecorder?.message).toMatchObject({
+      provenance: { kind: "internal_system", sourceTool: "heartbeat" },
+    });
   });
 
   it("keeps active goal context out of background heartbeat turns", async () => {

src/auto-reply/reply/get-reply-run.ts

@@ -1471,7 +1471,10 @@ export async function runPreparedReply(
           text: userTurnTranscriptText,
           senderIsOwner: command.senderIsOwner,
           ...(sourceTurnId ? { idempotencyKey: sourceTurnId } : {}),
-          ...(inputProvenance ? { provenance: inputProvenance } : {}),
+          ...(inputProvenance && !isHeartbeat ? { provenance: inputProvenance } : {}),
+          ...(isHeartbeat
+            ? { provenance: { kind: "internal_system" as const, sourceTool: "heartbeat" } }
+            : {}),
           ...(userTurnMediaForPersistence.length > 0
             ? {
                 media: userTurnMediaForPersistence,