fix(harness)!: hydrate runs from RunStore and persist interrupted status (#2932)

* fix(harness): hydrate run history from RunStore and persist cancellation status fix: - Make RunManager.get() async and hydrate from RunStore when in-memory record is missing - Merge store rows into list_by_thread() with in-memory precedence for active runs - Persist interrupted status to RunStore in cancel() and create_or_reject(interrupt|rollback) - Extract _persist_status() to reuse the best-effort store update pattern - Await run_mgr.get() in all gateway endpoints - Return 409 with distinct message for store-only runs not active on current worker Closes #2812, Closes #2813 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(harness): consistent sort and guarded hydration in RunManager fix: - list_by_thread() now sorts by created_at desc (newest first) even when no RunStore is configured, matching the store-backed code path - guard _record_from_store() call sites in get() and list_by_thread() with best-effort error handling so a single malformed store row cannot turn read paths into 500s test: - update test_list_by_thread assertion to expect newest-first order - seed MemoryRunStore via public put() API instead of writing to _runs * fix(harness): guard store-only runs from streaming and fix get() TOCTOU Add RunRecord.store_only flag set by _record_from_store so callers can distinguish hydrated history from live in-memory runs. join_run and stream_existing_run (action=None) now return 409 instead of hanging forever on an empty MemoryStreamBridge channel. Re-check _runs under lock after the store await in RunManager.get() so a concurrent create() that lands between the two checks returns the authoritative in-memory record rather than a stale store-hydrated copy. Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com> * fix(harness): reorder bridge fetch in join_run and make list_by_thread limit explicit Move get_stream_bridge() after the store_only guard in join_run so a missing bridge cannot produce 503 for historical runs before the 409 guard fires. Add limit parameter to RunManager.list_by_thread (default 100, matching the store's page size) and pass it explicitly to the store call. Update docstring to document the limit instead of claiming all runs are returned. Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com> * fix(harness): cap list_by_thread result to limit after merge Apply [:limit] to all return paths in list_by_thread so the method consistently returns at most limit records regardless of how many in-memory runs exist, making the limit parameter a true upper bound on the response size rather than just a store-query hint. Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com> * fix `list_by_thread` docstring Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * fix(runtime): add update_model_name to RunStore to prevent SQL integrity errors RunManager.update_model_name() was calling _persist_to_store() which uses RunStore.put(), but RunRepository.put() is insert-only. This caused integrity errors when updating model_name for existing runs in SQL-backed stores. fix: - Add abstract update_model_name method to RunStore base class - Implement update_model_name in MemoryRunStore - Implement update_model_name in RunRepository with proper normalization - Add _persist_model_name helper in RunManager - Update RunManager.update_model_name to use the new method test: - Add tests for update_model_name functionality - Add integration tests for RunManager with SQL-backed store Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(runtime): handle NULL status/on_disconnect in _record_from_store `dict.get(key, default)` only uses the default when the key is absent, so a SQL row with an explicit NULL status would pass `None` to `RunStatus(None)` and raise, breaking hydration for otherwise valid rows. Switch to `row.get(...) or fallback` so both missing and NULL values get a safe default. Add tests for get() and list_by_thread() with a NULL status row to prevent regression. Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com> * fix(runs): address PR review feedback on store consistency changes - Fix list_by_thread limit semantics: pass store_limit = max(0, limit - len(memory_records)) to store so newer store records are not crowded out by in-memory records - Remove dead code: cancelled guard after raise is always True, simplify to if wait and record.task - Document _record_from_store NULL fallback policy (status→pending, on_disconnect→cancel) in docstring Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-05-18 22:25:02 +08:00
parent 3acca12614
commit c810e9f809
10 changed files with 499 additions and 77 deletions
@@ -4,7 +4,7 @@ import re

 import pytest

-from deerflow.runtime import RunManager, RunStatus
+from deerflow.runtime import DisconnectMode, RunManager, RunStatus
 from deerflow.runtime.runs.store.memory import MemoryRunStore

 ISO_RE = re.compile(r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}")
@@ -34,7 +34,7 @@ async def test_create_and_get(manager: RunManager):
    assert ISO_RE.match(record.created_at)
    assert ISO_RE.match(record.updated_at)

-    fetched = manager.get(record.run_id)
+    fetched = await manager.get(record.run_id)
    assert fetched is record


@@ -64,6 +64,22 @@ async def test_cancel(manager: RunManager):
    assert record.status == RunStatus.interrupted


+@pytest.mark.anyio
+async def test_cancel_persists_interrupted_status_to_store():
+    """Cancel should persist interrupted status to the backing store."""
+    store = MemoryRunStore()
+    manager = RunManager(store=store)
+    record = await manager.create("thread-1")
+    await manager.set_status(record.run_id, RunStatus.running)
+
+    cancelled = await manager.cancel(record.run_id)
+
+    stored = await store.get(record.run_id)
+    assert cancelled is True
+    assert stored is not None
+    assert stored["status"] == "interrupted"
+
+
@pytest.mark.anyio
 async def test_cancel_not_inflight(manager: RunManager):
    """Cancelling a completed run should return False."""
@@ -83,9 +99,9 @@ async def test_list_by_thread(manager: RunManager):

    runs = await manager.list_by_thread("thread-1")
    assert len(runs) == 2
-    # list_by_thread returns oldest-first (ascending created_at).
-    assert runs[0].run_id == r1.run_id
-    assert runs[1].run_id == r2.run_id
+    # Newest first: r2 was created after r1.
+    assert runs[0].run_id == r2.run_id
+    assert runs[1].run_id == r1.run_id


@pytest.mark.anyio
@@ -117,7 +133,7 @@ async def test_cleanup(manager: RunManager):
    run_id = record.run_id

    await manager.cleanup(run_id, delay=0)
-    assert manager.get(run_id) is None
+    assert await manager.get(run_id) is None


@pytest.mark.anyio
@@ -132,7 +148,116 @@ async def test_set_status_with_error(manager: RunManager):
@pytest.mark.anyio
 async def test_get_nonexistent(manager: RunManager):
    """Getting a nonexistent run should return None."""
-    assert manager.get("does-not-exist") is None
+    assert await manager.get("does-not-exist") is None
+
+
+@pytest.mark.anyio
+async def test_get_hydrates_store_only_run():
+    """Store-only runs should be readable after process restart."""
+    store = MemoryRunStore()
+    await store.put(
+        "run-store-only",
+        thread_id="thread-1",
+        assistant_id="lead_agent",
+        status="success",
+        multitask_strategy="reject",
+        metadata={"source": "store"},
+        kwargs={"input": "value"},
+        created_at="2026-01-01T00:00:00+00:00",
+        model_name="model-a",
+    )
+    manager = RunManager(store=store)
+
+    record = await manager.get("run-store-only")
+
+    assert record is not None
+    assert record.run_id == "run-store-only"
+    assert record.thread_id == "thread-1"
+    assert record.assistant_id == "lead_agent"
+    assert record.status == RunStatus.success
+    assert record.on_disconnect == DisconnectMode.cancel
+    assert record.metadata == {"source": "store"}
+    assert record.kwargs == {"input": "value"}
+    assert record.model_name == "model-a"
+    assert record.task is None
+    assert record.store_only is True
+
+
+@pytest.mark.anyio
+async def test_get_hydrates_run_with_null_enum_fields():
+    """Rows with NULL status/on_disconnect must hydrate with safe defaults, not raise."""
+    store = MemoryRunStore()
+    # Simulate a SQL row where the nullable status column is NULL
+    await store.put(
+        "run-null-status",
+        thread_id="thread-1",
+        status=None,
+        created_at="2026-01-01T00:00:00+00:00",
+    )
+    manager = RunManager(store=store)
+
+    record = await manager.get("run-null-status")
+
+    assert record is not None
+    assert record.status == RunStatus.pending
+    assert record.on_disconnect == DisconnectMode.cancel
+    assert record.store_only is True
+
+
+@pytest.mark.anyio
+async def test_list_by_thread_hydrates_run_with_null_enum_fields():
+    """list_by_thread must not skip rows with NULL status; applies safe defaults."""
+    store = MemoryRunStore()
+    await store.put(
+        "run-null-status-list",
+        thread_id="thread-null",
+        status=None,
+        created_at="2026-01-01T00:00:00+00:00",
+    )
+    manager = RunManager(store=store)
+
+    runs = await manager.list_by_thread("thread-null")
+
+    assert len(runs) == 1
+    assert runs[0].run_id == "run-null-status-list"
+    assert runs[0].status == RunStatus.pending
+    assert runs[0].on_disconnect == DisconnectMode.cancel
+
+
+@pytest.mark.anyio
+async def test_create_record_is_not_store_only(manager: RunManager):
+    """In-memory records created via create() must have store_only=False."""
+    record = await manager.create("thread-1")
+    assert record.store_only is False
+
+
+@pytest.mark.anyio
+async def test_get_prefers_in_memory_record_over_store():
+    """In-memory records retain task/control state when store has same run."""
+    store = MemoryRunStore()
+    manager = RunManager(store=store)
+    record = await manager.create("thread-1")
+    await store.update_status(record.run_id, "success")
+
+    fetched = await manager.get(record.run_id)
+
+    assert fetched is record
+    assert fetched.status == RunStatus.pending
+
+
+@pytest.mark.anyio
+async def test_list_by_thread_merges_store_runs_newest_first():
+    """list_by_thread should merge memory and store rows with memory precedence."""
+    store = MemoryRunStore()
+    await store.put("old-store", thread_id="thread-1", status="success", created_at="2026-01-01T00:00:00+00:00")
+    await store.put("other-thread", thread_id="thread-2", status="success", created_at="2026-01-03T00:00:00+00:00")
+    manager = RunManager(store=store)
+    memory_record = await manager.create("thread-1")
+
+    runs = await manager.list_by_thread("thread-1")
+
+    assert [run.run_id for run in runs] == [memory_record.run_id, "old-store"]
+    assert runs[0] is memory_record


@pytest.mark.anyio
@@ -171,11 +296,45 @@ async def test_model_name_create_or_reject():
    assert stored["model_name"] == "anthropic.claude-sonnet-4-20250514-v1:0"

    # Verify retrieval returns the model_name via in-memory record
-    fetched = mgr.get(record.run_id)
+    fetched = await mgr.get(record.run_id)
    assert fetched is not None
    assert fetched.model_name == "anthropic.claude-sonnet-4-20250514-v1:0"


+@pytest.mark.anyio
+async def test_create_or_reject_interrupt_persists_interrupted_status_to_store():
+    """interrupt strategy should persist interrupted status for old runs."""
+    store = MemoryRunStore()
+    manager = RunManager(store=store)
+    old = await manager.create("thread-1")
+    await manager.set_status(old.run_id, RunStatus.running)
+
+    new = await manager.create_or_reject("thread-1", multitask_strategy="interrupt")
+
+    stored_old = await store.get(old.run_id)
+    assert new.run_id != old.run_id
+    assert old.status == RunStatus.interrupted
+    assert stored_old is not None
+    assert stored_old["status"] == "interrupted"
+
+
+@pytest.mark.anyio
+async def test_create_or_reject_rollback_persists_interrupted_status_to_store():
+    """rollback strategy should persist interrupted status for old runs."""
+    store = MemoryRunStore()
+    manager = RunManager(store=store)
+    old = await manager.create("thread-1")
+    await manager.set_status(old.run_id, RunStatus.running)
+
+    new = await manager.create_or_reject("thread-1", multitask_strategy="rollback")
+
+    stored_old = await store.get(old.run_id)
+    assert new.run_id != old.run_id
+    assert old.status == RunStatus.interrupted
+    assert stored_old is not None
+    assert stored_old["status"] == "interrupted"
+
+
@pytest.mark.anyio
 async def test_model_name_default_is_none():
    """create_or_reject without model_name should default to None."""