fix(harness)!: hydrate runs from RunStore and persist interrupted status (#2932)

* fix(harness): hydrate run history from RunStore and persist cancellation status

fix:
- Make RunManager.get() async and hydrate from RunStore when in-memory record is missing
- Merge store rows into list_by_thread() with in-memory precedence for active runs
- Persist interrupted status to RunStore in cancel() and create_or_reject(interrupt|rollback)
- Extract _persist_status() to reuse the best-effort store update pattern
- Await run_mgr.get() in all gateway endpoints
- Return 409 with distinct message for store-only runs not active on current worker

Closes #2812, Closes #2813

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(harness): consistent sort and guarded hydration in RunManager

fix:
- list_by_thread() now sorts by created_at desc (newest first) even when
  no RunStore is configured, matching the store-backed code path
- guard _record_from_store() call sites in get() and list_by_thread()
  with best-effort error handling so a single malformed store row cannot
  turn read paths into 500s

test:
- update test_list_by_thread assertion to expect newest-first order
- seed MemoryRunStore via public put() API instead of writing to _runs

* fix(harness): guard store-only runs from streaming and fix get() TOCTOU

Add RunRecord.store_only flag set by _record_from_store so callers can
distinguish hydrated history from live in-memory runs.  join_run and
stream_existing_run (action=None) now return 409 instead of hanging
forever on an empty MemoryStreamBridge channel.

Re-check _runs under lock after the store await in RunManager.get() so a
concurrent create() that lands between the two checks returns the
authoritative in-memory record rather than a stale store-hydrated copy.

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

* fix(harness): reorder bridge fetch in join_run and make list_by_thread limit explicit

Move get_stream_bridge() after the store_only guard in join_run so a
missing bridge cannot produce 503 for historical runs before the 409
guard fires.

Add limit parameter to RunManager.list_by_thread (default 100, matching
the store's page size) and pass it explicitly to the store call.
Update docstring to document the limit instead of claiming all runs are
returned.

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

* fix(harness): cap list_by_thread result to limit after merge

Apply [:limit] to all return paths in list_by_thread so the method
consistently returns at most limit records regardless of how many
in-memory runs exist, making the limit parameter a true upper bound
on the response size rather than just a store-query hint.

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

* fix `list_by_thread` docstring

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* fix(runtime): add update_model_name to RunStore to prevent SQL integrity errors

RunManager.update_model_name() was calling _persist_to_store() which uses
RunStore.put(), but RunRepository.put() is insert-only. This caused integrity
errors when updating model_name for existing runs in SQL-backed stores.

fix:
- Add abstract update_model_name method to RunStore base class
- Implement update_model_name in MemoryRunStore
- Implement update_model_name in RunRepository with proper normalization
- Add _persist_model_name helper in RunManager
- Update RunManager.update_model_name to use the new method

test:
- Add tests for update_model_name functionality
- Add integration tests for RunManager with SQL-backed store

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(runtime): handle NULL status/on_disconnect in _record_from_store

`dict.get(key, default)` only uses the default when the key is absent,
so a SQL row with an explicit NULL status would pass `None` to
`RunStatus(None)` and raise, breaking hydration for otherwise valid rows.
Switch to `row.get(...) or fallback` so both missing and NULL values
get a safe default. Add tests for get() and list_by_thread() with a
NULL status row to prevent regression.

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

* fix(runs): address PR review feedback on store consistency changes

- Fix list_by_thread limit semantics: pass store_limit = max(0, limit - len(memory_records)) to store so newer store records are not crowded out by in-memory records
- Remove dead code: cancelled guard after raise is always True, simplify to if wait and record.task
- Document _record_from_store NULL fallback policy (status→pending, on_disconnect→cancel) in docstring

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

backend/packages/harness/deerflow/persistence/run/sql.py

@@ -151,6 +151,11 @@ class RunRepository(RunStore):
             await session.execute(update(RunRow).where(RunRow.run_id == run_id).values(**values))
             await session.commit()
 
+    async def update_model_name(self, run_id, model_name):
+        async with self._sf() as session:
+            await session.execute(update(RunRow).where(RunRow.run_id == run_id).values(model_name=self._normalize_model_name(model_name), updated_at=datetime.now(UTC)))
+            await session.commit()
+
     async def delete(
         self,
         run_id,

backend/packages/harness/deerflow/runtime/runs/manager.py

@@ -6,7 +6,7 @@ import asyncio
 import logging
 import uuid
 from dataclasses import dataclass, field
-from typing import TYPE_CHECKING
+from typing import TYPE_CHECKING, Any
 
 from deerflow.utils.time import now_iso as _now_iso
 
@@ -37,6 +37,7 @@ class RunRecord:
     abort_action: str = "interrupt"
     error: str | None = None
     model_name: str | None = None
+    store_only: bool = False
 
 
 class RunManager:
@@ -71,6 +72,38 @@ class RunManager:
         except Exception:
             logger.warning("Failed to persist run %s to store", record.run_id, exc_info=True)
 
+    async def _persist_status(self, run_id: str, status: RunStatus, *, error: str | None = None) -> None:
+        """Best-effort persist a status transition to the backing store."""
+        if self._store is None:
+            return
+        try:
+            await self._store.update_status(run_id, status.value, error=error)
+        except Exception:
+            logger.warning("Failed to persist status update for run %s", run_id, exc_info=True)
+
+    @staticmethod
+    def _record_from_store(row: dict[str, Any]) -> RunRecord:
+        """Build a read-only runtime record from a serialized store row.
+
+        NULL status/on_disconnect columns (e.g. from rows written before those
+        columns were added) default to ``pending`` and ``cancel`` respectively.
+        """
+        return RunRecord(
+            run_id=row["run_id"],
+            thread_id=row["thread_id"],
+            assistant_id=row.get("assistant_id"),
+            status=RunStatus(row.get("status") or RunStatus.pending.value),
+            on_disconnect=DisconnectMode(row.get("on_disconnect") or DisconnectMode.cancel.value),
+            multitask_strategy=row.get("multitask_strategy") or "reject",
+            metadata=row.get("metadata") or {},
+            kwargs=row.get("kwargs") or {},
+            created_at=row.get("created_at") or "",
+            updated_at=row.get("updated_at") or "",
+            error=row.get("error"),
+            model_name=row.get("model_name"),
+            store_only=True,
+        )
+
     async def update_run_completion(self, run_id: str, **kwargs) -> None:
         """Persist token usage and completion data to the backing store."""
         if self._store is not None:
@@ -110,61 +143,77 @@ class RunManager:
         logger.info("Run created: run_id=%s thread_id=%s", run_id, thread_id)
         return record
 
-    def get(self, run_id: str) -> RunRecord | None:
-        """Return an in-memory run record by ID, or ``None``."""
-        return self._runs.get(run_id)
+    async def get(self, run_id: str, *, user_id: str | None = None) -> RunRecord | None:
+        """Return a run record by ID, or ``None``.
 
-    async def aget(self, run_id: str, *, user_id: str | None = None) -> RunRecord | None:
-        """Return a run record by ID, checking the persistent store as fallback."""
-        record = self._runs.get(run_id)
+        Args:
+            run_id: The run ID to look up.
+            user_id: Optional user ID for permission filtering when hydrating from store.
+        """
+        async with self._lock:
+            record = self._runs.get(run_id)
         if record is not None:
             return record
-        if self._store is not None:
-            try:
-                d = await self._store.get(run_id, user_id=user_id)
-                if d is not None:
-                    return self._store_dict_to_record(d)
-            except Exception:
-                logger.warning("Failed to query store for run %s", run_id, exc_info=True)
-        return None
-
-    def _store_dict_to_record(self, d: dict) -> RunRecord:
-        """Convert a store dict back to a RunRecord for read-only use."""
-        return RunRecord(
-            run_id=d["run_id"],
-            thread_id=d["thread_id"],
-            assistant_id=d.get("assistant_id"),
-            status=RunStatus(d.get("status", RunStatus.error.value)),
-            on_disconnect=DisconnectMode.cancel,
-            multitask_strategy=d.get("multitask_strategy", "reject"),
-            metadata=d.get("metadata", {}),
-            kwargs=d.get("kwargs", {}),
-            created_at=d.get("created_at", ""),
-            updated_at=d.get("updated_at", ""),
-            model_name=d.get("model_name"),
-            error=d.get("error"),
-        )
-
-    async def list_by_thread(self, thread_id: str, *, user_id: str | None = None) -> list[RunRecord]:
-        """Return all runs for a given thread, oldest first."""
+        if self._store is None:
+            return None
+        try:
+            row = await self._store.get(run_id, user_id=user_id)
+        except Exception:
+            logger.warning("Failed to hydrate run %s from store", run_id, exc_info=True)
+            return None
+        # Re-check after store await: a concurrent create() may have inserted the
+        # in-memory record while the store call was in flight.
         async with self._lock:
-            in_memory = [r for r in self._runs.values() if r.thread_id == thread_id]
-            in_memory_ids = {r.run_id for r in in_memory}
+            record = self._runs.get(run_id)
+        if record is not None:
+            return record
+        if row is None:
+            return None
+        try:
+            return self._record_from_store(row)
+        except Exception:
+            logger.warning("Failed to map store row for run %s", run_id, exc_info=True)
+            return None
 
-        store_records: list[RunRecord] = []
-        if self._store is not None:
-            try:
-                store_dicts = await self._store.list_by_thread(thread_id, user_id=user_id)
-                for d in store_dicts:
-                    if d["run_id"] not in in_memory_ids:
-                        store_records.append(self._store_dict_to_record(d))
-            except Exception:
-                logger.warning("Failed to query store for thread %s runs", thread_id, exc_info=True)
+    async def aget(self, run_id: str, *, user_id: str | None = None) -> RunRecord | None:
+        """Return a run record by ID, checking the persistent store as fallback.
 
-        return sorted(
-            in_memory + store_records,
-            key=lambda record: record.created_at or "",
-        )
+        Alias for :meth:`get` for backward compatibility.
+        """
+        return await self.get(run_id, user_id=user_id)
+
+    async def list_by_thread(self, thread_id: str, *, user_id: str | None = None, limit: int = 100) -> list[RunRecord]:
+        """Return runs for a given thread, newest first, at most ``limit`` records.
+
+        In-memory runs take precedence only when the same ``run_id`` exists in both
+        memory and the backing store. The merged result is then sorted newest-first
+        by ``created_at`` and trimmed to ``limit`` (default 100).
+
+        Args:
+            thread_id: The thread ID to filter by.
+            user_id: Optional user ID for permission filtering when hydrating from store.
+            limit: Maximum number of runs to return.
+        """
+        async with self._lock:
+            # Dict insertion order gives deterministic results when timestamps tie.
+            memory_records = [r for r in self._runs.values() if r.thread_id == thread_id]
+        if self._store is None:
+            return sorted(memory_records, key=lambda r: r.created_at, reverse=True)[:limit]
+        records_by_id = {record.run_id: record for record in memory_records}
+        store_limit = max(0, limit - len(memory_records))
+        try:
+            rows = await self._store.list_by_thread(thread_id, user_id=user_id, limit=store_limit)
+        except Exception:
+            logger.warning("Failed to hydrate runs for thread %s from store", thread_id, exc_info=True)
+            return sorted(memory_records, key=lambda r: r.created_at, reverse=True)[:limit]
+        for row in rows:
+            run_id = row.get("run_id")
+            if run_id and run_id not in records_by_id:
+                try:
+                    records_by_id[run_id] = self._record_from_store(row)
+                except Exception:
+                    logger.warning("Failed to map store row for run %s", run_id, exc_info=True)
+        return sorted(records_by_id.values(), key=lambda record: record.created_at, reverse=True)[:limit]
 
     async def set_status(self, run_id: str, status: RunStatus, *, error: str | None = None) -> None:
         """Transition a run to a new status."""
@@ -177,13 +226,18 @@ class RunManager:
             record.updated_at = _now_iso()
             if error is not None:
                 record.error = error
-        if self._store is not None:
-            try:
-                await self._store.update_status(run_id, status.value, error=error)
-            except Exception:
-                logger.warning("Failed to persist status update for run %s", run_id, exc_info=True)
+        await self._persist_status(run_id, status, error=error)
         logger.info("Run %s -> %s", run_id, status.value)
 
+    async def _persist_model_name(self, run_id: str, model_name: str | None) -> None:
+        """Best-effort persist model_name update to the backing store."""
+        if self._store is None:
+            return
+        try:
+            await self._store.update_model_name(run_id, model_name)
+        except Exception:
+            logger.warning("Failed to persist model_name update for run %s", run_id, exc_info=True)
+
     async def update_model_name(self, run_id: str, model_name: str | None) -> None:
         """Update the model name for a run."""
         async with self._lock:
@@ -193,7 +247,7 @@ class RunManager:
                 return
             record.model_name = model_name
             record.updated_at = _now_iso()
-        await self._persist_to_store(record)
+        await self._persist_model_name(run_id, model_name)
         logger.info("Run %s model_name=%s", run_id, model_name)
 
     async def cancel(self, run_id: str, *, action: str = "interrupt") -> bool:
@@ -218,6 +272,7 @@ class RunManager:
                 record.task.cancel()
             record.status = RunStatus.interrupted
             record.updated_at = _now_iso()
+        await self._persist_status(run_id, RunStatus.interrupted)
         logger.info("Run %s cancelled (action=%s)", run_id, action)
         return True
 
@@ -245,6 +300,7 @@ class RunManager:
         now = _now_iso()
 
         _supported_strategies = ("reject", "interrupt", "rollback")
+        interrupted_run_ids: list[str] = []
 
         async with self._lock:
             if multitask_strategy not in _supported_strategies:
@@ -263,6 +319,7 @@ class RunManager:
                         r.task.cancel()
                     r.status = RunStatus.interrupted
                     r.updated_at = now
+                    interrupted_run_ids.append(r.run_id)
                 logger.info(
                     "Cancelled %d inflight run(s) on thread %s (strategy=%s)",
                     len(inflight),
@@ -285,6 +342,8 @@ class RunManager:
             )
             self._runs[run_id] = record
 
+        for interrupted_run_id in interrupted_run_ids:
+            await self._persist_status(interrupted_run_id, RunStatus.interrupted)
         await self._persist_to_store(record)
         logger.info("Run created: run_id=%s thread_id=%s", run_id, thread_id)
         return record

backend/packages/harness/deerflow/runtime/runs/store/base.py

@@ -66,6 +66,15 @@ class RunStore(abc.ABC):
     async def delete(self, run_id: str) -> None:
         pass
 
+    @abc.abstractmethod
+    async def update_model_name(
+        self,
+        run_id: str,
+        model_name: str | None,
+    ) -> None:
+        """Update the model_name field for an existing run."""
+        pass
+
     @abc.abstractmethod
     async def update_run_completion(
         self,

backend/packages/harness/deerflow/runtime/runs/store/memory.py

@@ -66,6 +66,11 @@ class MemoryRunStore(RunStore):
                 self._runs[run_id]["error"] = error
             self._runs[run_id]["updated_at"] = datetime.now(UTC).isoformat()
 
+    async def update_model_name(self, run_id, model_name):
+        if run_id in self._runs:
+            self._runs[run_id]["model_name"] = model_name
+            self._runs[run_id]["updated_at"] = datetime.now(UTC).isoformat()
+
     async def delete(self, run_id):
         self._runs.pop(run_id, None)