mirror of
https://github.com/bytedance/deer-flow.git
synced 2026-07-21 10:15:47 +00:00
* fix(runs): cancel degrades to lease takeover for multi-worker Work item 4 of the multi-worker ownership epic (https://github.com/bytedance/deer-flow/issues/3948). Problem: POST /runs/{run_id}/cancel landing on a non-owning worker returns 409 — the cancel button silently fails under GATEWAY_WORKERS>1 with no sticky routing. cancel() required the current worker to hold the in-memory task/abort_event, which any non-owner pod cannot satisfy. Changes: - RunManager.cancel() returns CancelOutcome enum (cancelled / taken_over / lease_valid_elsewhere / not_active_locally / not_cancellable / unknown) instead of bool, so the router can map each outcome to the right HTTP response. - New store primitive claim_for_takeover(): a single atomic conditional UPDATE that marks a run as error only when status IN (pending, running) AND (lease IS NULL OR lease < now - grace). Closes the stale-read / concurrent-heartbeat race — if the owner renews between our read and write, the UPDATE matches 0 rows and we surface lease_valid_elsewhere. - HTTP cancel + stream-join endpoints route on CancelOutcome: cancelled -> 202 (or 204 with wait=true); taken_over -> 202 immediately (no SSE streaming — the run is terminal on another worker, streaming would hang); lease_valid_elsewhere -> 409 + Retry-After header computed from lease_expires_at + grace_seconds. - RunManager.grace_seconds exposed as a public property; the router no longer reaches into _run_ownership_config. - _is_lease_expired extracted to a module-level function, shared by RunManager.cancel() and MemoryRunStore.claim_for_takeover(). - GATEWAY_WORKERS=1 + heartbeat_enabled=false is zero-regression: the non-local path short-circuits to not_active_locally, preserving the original 409 behaviour the existing tests pin. Tests: 12 new (5 store primitive + 4 cancel-takeover unit + 3 HTTP including a regression guard verifying POST /stream?action=interrupt on a dead-owner run returns 202 instead of hanging on SSE). 244 directly-related tests pass; 36/36 blocking-IO gate pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(runs): guard update_status and self-terminate on takeover Two defenses close a split-brain window where the original owner could overwrite a peer's takeover status: - update_status (SQL + memory store) now guards on status IN ('pending','running'). When takeover already set the row to 'error', the owner's final status write matches 0 rows and is dropped. - _persist_status: when update_status returns False, check whether the row exists before attempting recovery via put(). If the row exists (takeover by another worker), skip recovery instead of blindly upserting over the takeover. - Heartbeat _renew_leases: when update_lease returns False (row no longer pending/running or owner changed), cancel the local task so wasted CPU is bounded to the next heartbeat tick (~10s) instead of the full task lifetime. Also fix three reviewer feedback items: - Re-fetch the store row when cancel() returns lease_valid_elsewhere, so Retry-After uses the owner's freshly-renewed lease instead of a stale value from request start. - Fallback 'unknown' in takeover error message when owner_worker_id is NULL (pre-ownership data). - Remove dead else-10 branch from grace_seconds property (unreachable — all callers are downstream of the heartbeat_enabled guard). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test(runs): pin split-brain defences from update_status guard + heartbeat Three tests lock down the takeover authoritativeness so a late-running owner cannot overwrite a peer's claim: - update_status must reject writes when the store row is already terminal (taken over by another worker). - _persist_status must skip row-recovery via put() when the row exists but has been taken over. - Heartbeat _renew_leases must cancel the local task when update_lease returns False (row claimed by another worker). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(runs): precise outcome + log when local cancel loses to peer takeover Two reviewer precision nits on the split-brain defence: - _persist_status: branch the skip-reason log on existing["status"]. error → WARNING "peer takeover" (anomalous); interrupted/success → INFO "local cancel/completion race" (expected when user hits stop as the run finishes). Stops noisy false-positive takeover warnings in operator logs. - cancel() local path: when _persist_status returns False, re-check the store. If a peer's claim_for_takeover flipped the row to error between our in-memory cancel and the guarded update_status, surface taken_over instead of cancelled so the client sees a status consistent with the store. Test: test_cancel_returns_taken_over_when_peer_claims_during_local_cancel pins the race outcome. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(runs): widen update_status guard, de-duplicate lease helpers, add coverage Round 3 of reviewer feedback: - Widen update_status guard to status IN ('pending','running','interrupted'). The original guard blocked interrupted→error (the rollback finalize path), losing the "Rolled back by user" message. interrupted is now permitted while error/success stay locked — takeover protection unchanged. - claim_for_takeover False now re-reads the store row to distinguish causes: owner renewed lease → lease_valid_elsewhere; row went terminal → not_cancellable; another worker already took it over → taken_over. - Extract _raise_lease_valid_elsewhere() helper to de-duplicate the 409+Retry-After block shared across cancel_run and stream_existing_run. - Extract _lease_expired_or_null() in persistence/run/sql.py to de-duplicate the lease-expiry SQL WHERE clause shared by claim_for_takeover and list_inflight_with_expired_lease. - 11 new tests: 5 SQL-layer claim_for_takeover (expired/valid/NULL/ terminal/nonexistent), 3 _compute_retry_after unit (NULL/unparseable/ normal), 2 claim re-read precision (terminal/takeover), 1 stream endpoint 409+Retry-After. Not addressed (non-blocking, reviewer agreed): - The 2–3 store.gets in the takeover cold path: optimizing the API to accept a pre-fetched record would couple the router to the manager more tightly than justified by the perf gain. - The lease-expiry inline loop in MemoryRunStore.list_inflight_with_- expired_lease pre-computes cutoff once for all rows; switching to the shared _is_lease_expired helper would recompute datetime.now() per row with no real benefit. 260 related tests pass; 36/36 blocking-IO gate pass; ruff clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(runs): de-duplicate lease-expiry helper, restore defensive fallback Address final round of review feedback: - Extract is_lease_expired to deerflow.utils.time (no _ prefix, public utility). Manager and MemoryRunStore now import from the same place instead of the store reaching backward into the manager for a private function. - Restore defensive else-10 fallback in grace_seconds property (removed in an earlier round). The guard is unreachable for current callers but protects future ones from AttributeError. - Comment the transient in-memory interrupted vs store error state when a local cancel is superseded by a peer takeover. - Comment the max(1, ...) floor in _compute_retry_after — the floor is a lower bound, not a poll interval; clients should apply jitter. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: rayhpeng <rayhpeng@gmail.com>
143 lines
5.3 KiB
Python
143 lines
5.3 KiB
Python
"""Tests for idempotent run cancellation (issue #3055).
|
|
|
|
RunManager.cancel() returns True when a run is already interrupted so that
|
|
a second cancel request from the same worker is treated as a no-op success
|
|
(202) rather than a conflict (409). Both the POST cancel endpoint and the
|
|
POST stream endpoint share this behaviour through the same cancel() call.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
|
|
from _router_auth_helpers import make_authed_test_app
|
|
from fastapi.testclient import TestClient
|
|
|
|
from app.gateway.routers import thread_runs
|
|
from deerflow.runtime import CancelOutcome, RunManager, RunStatus
|
|
|
|
THREAD_ID = "thread-cancel-test"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Helpers
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def _make_app(mgr: RunManager) -> TestClient:
|
|
app = make_authed_test_app()
|
|
app.include_router(thread_runs.router)
|
|
app.state.run_manager = mgr
|
|
return TestClient(app, raise_server_exceptions=False)
|
|
|
|
|
|
def _create_interrupted_run(mgr: RunManager) -> str:
|
|
"""Create a run and cancel it, returning its run_id."""
|
|
|
|
async def _setup():
|
|
record = await mgr.create(THREAD_ID)
|
|
await mgr.set_status(record.run_id, RunStatus.running)
|
|
await mgr.cancel(record.run_id)
|
|
return record.run_id
|
|
|
|
return asyncio.run(_setup())
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# RunManager.cancel() unit tests
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestRunManagerCancelIdempotency:
|
|
def test_cancel_returns_cancelled_for_already_interrupted_run(self):
|
|
"""cancel() must return CancelledOutcome.cancelled when the run is already interrupted."""
|
|
|
|
async def run():
|
|
mgr = RunManager()
|
|
record = await mgr.create(THREAD_ID)
|
|
await mgr.set_status(record.run_id, RunStatus.running)
|
|
first = await mgr.cancel(record.run_id)
|
|
assert first == CancelOutcome.cancelled
|
|
second = await mgr.cancel(record.run_id)
|
|
assert second == CancelOutcome.cancelled # idempotent
|
|
|
|
asyncio.run(run())
|
|
|
|
def test_cancel_returns_not_cancellable_for_successful_run(self):
|
|
"""cancel() must return not_cancellable for runs that completed successfully."""
|
|
|
|
async def run():
|
|
mgr = RunManager()
|
|
record = await mgr.create(THREAD_ID)
|
|
await mgr.set_status(record.run_id, RunStatus.running)
|
|
await mgr.set_status(record.run_id, RunStatus.success)
|
|
result = await mgr.cancel(record.run_id)
|
|
assert result == CancelOutcome.not_cancellable
|
|
|
|
asyncio.run(run())
|
|
|
|
def test_cancel_returns_not_active_locally_for_unknown_run(self):
|
|
async def run():
|
|
mgr = RunManager()
|
|
result = await mgr.cancel("nonexistent-run-id")
|
|
assert result == CancelOutcome.not_active_locally
|
|
|
|
asyncio.run(run())
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# POST /cancel endpoint — idempotent 202
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestCancelRunEndpointIdempotency:
|
|
def test_double_cancel_returns_202_not_409(self):
|
|
"""Second cancel on an already-interrupted run must return 202, not 409."""
|
|
mgr = RunManager()
|
|
run_id = _create_interrupted_run(mgr)
|
|
client = _make_app(mgr)
|
|
|
|
resp = client.post(f"/api/threads/{THREAD_ID}/runs/{run_id}/cancel")
|
|
assert resp.status_code == 202, f"Expected 202, got {resp.status_code}: {resp.text}"
|
|
|
|
def test_cancel_unknown_run_returns_404(self):
|
|
mgr = RunManager()
|
|
client = _make_app(mgr)
|
|
resp = client.post(f"/api/threads/{THREAD_ID}/runs/no-such-run/cancel")
|
|
assert resp.status_code == 404
|
|
|
|
def test_cancel_successful_run_returns_409(self):
|
|
"""Successfully-completed runs cannot be cancelled — must return 409."""
|
|
|
|
async def _setup():
|
|
mgr = RunManager()
|
|
record = await mgr.create(THREAD_ID)
|
|
await mgr.set_status(record.run_id, RunStatus.running)
|
|
await mgr.set_status(record.run_id, RunStatus.success)
|
|
return mgr, record.run_id
|
|
|
|
mgr, run_id = asyncio.run(_setup())
|
|
client = _make_app(mgr)
|
|
resp = client.post(f"/api/threads/{THREAD_ID}/runs/{run_id}/cancel")
|
|
assert resp.status_code == 409
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# POST /{thread_id}/runs/{run_id}/join (stream_existing_run) — idempotent cancel
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestStreamExistingRunIdempotentCancel:
|
|
def test_stream_cancel_already_interrupted_returns_not_409(self):
|
|
"""stream_existing_run with action=interrupt on an already-interrupted run
|
|
must not raise 409 — the idempotent cancel path returns 202/SSE."""
|
|
mgr = RunManager()
|
|
run_id = _create_interrupted_run(mgr)
|
|
client = _make_app(mgr)
|
|
|
|
resp = client.post(
|
|
f"/api/threads/{THREAD_ID}/runs/{run_id}/join",
|
|
params={"action": "interrupt"},
|
|
)
|
|
assert resp.status_code != 409, f"Should not 409 on idempotent cancel, got {resp.status_code}"
|