Files
deer-flow/backend/tests/test_cancel_run_idempotent.py
b53c1ae0e0 fix(runs): cancel degrades to lease takeover for multi-worker (#4064)
* fix(runs): cancel degrades to lease takeover for multi-worker

Work item 4 of the multi-worker ownership epic
(https://github.com/bytedance/deer-flow/issues/3948).

Problem: POST /runs/{run_id}/cancel landing on a non-owning worker
returns 409 — the cancel button silently fails under GATEWAY_WORKERS>1
with no sticky routing. cancel() required the current worker to hold
the in-memory task/abort_event, which any non-owner pod cannot satisfy.

Changes:
- RunManager.cancel() returns CancelOutcome enum (cancelled /
  taken_over / lease_valid_elsewhere / not_active_locally /
  not_cancellable / unknown) instead of bool, so the router can map
  each outcome to the right HTTP response.
- New store primitive claim_for_takeover(): a single atomic
  conditional UPDATE that marks a run as error only when
  status IN (pending, running) AND (lease IS NULL OR
  lease < now - grace). Closes the stale-read / concurrent-heartbeat
  race — if the owner renews between our read and write, the UPDATE
  matches 0 rows and we surface lease_valid_elsewhere.
- HTTP cancel + stream-join endpoints route on CancelOutcome:
  cancelled -> 202 (or 204 with wait=true); taken_over -> 202
  immediately (no SSE streaming — the run is terminal on another
  worker, streaming would hang); lease_valid_elsewhere -> 409 +
  Retry-After header computed from lease_expires_at + grace_seconds.
- RunManager.grace_seconds exposed as a public property; the router
  no longer reaches into _run_ownership_config.
- _is_lease_expired extracted to a module-level function, shared by
  RunManager.cancel() and MemoryRunStore.claim_for_takeover().
- GATEWAY_WORKERS=1 + heartbeat_enabled=false is zero-regression:
  the non-local path short-circuits to not_active_locally, preserving
  the original 409 behaviour the existing tests pin.

Tests: 12 new (5 store primitive + 4 cancel-takeover unit + 3 HTTP
including a regression guard verifying POST /stream?action=interrupt
on a dead-owner run returns 202 instead of hanging on SSE).
244 directly-related tests pass; 36/36 blocking-IO gate pass.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(runs): guard update_status and self-terminate on takeover

Two defenses close a split-brain window where the original owner
could overwrite a peer's takeover status:

- update_status (SQL + memory store) now guards on
  status IN ('pending','running'). When takeover already set
  the row to 'error', the owner's final status write matches
  0 rows and is dropped.

- _persist_status: when update_status returns False, check
  whether the row exists before attempting recovery via put().
  If the row exists (takeover by another worker), skip recovery
  instead of blindly upserting over the takeover.

- Heartbeat _renew_leases: when update_lease returns False
  (row no longer pending/running or owner changed), cancel the
  local task so wasted CPU is bounded to the next heartbeat
  tick (~10s) instead of the full task lifetime.

Also fix three reviewer feedback items:

- Re-fetch the store row when cancel() returns
  lease_valid_elsewhere, so Retry-After uses the owner's
  freshly-renewed lease instead of a stale value from
  request start.

- Fallback 'unknown' in takeover error message when
  owner_worker_id is NULL (pre-ownership data).

- Remove dead else-10 branch from grace_seconds property
  (unreachable — all callers are downstream of the
  heartbeat_enabled guard).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* test(runs): pin split-brain defences from update_status guard + heartbeat

Three tests lock down the takeover authoritativeness so a
late-running owner cannot overwrite a peer's claim:

- update_status must reject writes when the store row is already
  terminal (taken over by another worker).
- _persist_status must skip row-recovery via put() when the row
  exists but has been taken over.
- Heartbeat _renew_leases must cancel the local task when
  update_lease returns False (row claimed by another worker).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(runs): precise outcome + log when local cancel loses to peer takeover

Two reviewer precision nits on the split-brain defence:

- _persist_status: branch the skip-reason log on existing["status"].
  error → WARNING "peer takeover" (anomalous); interrupted/success →
  INFO "local cancel/completion race" (expected when user hits stop
  as the run finishes). Stops noisy false-positive takeover warnings
  in operator logs.

- cancel() local path: when _persist_status returns False, re-check
  the store. If a peer's claim_for_takeover flipped the row to error
  between our in-memory cancel and the guarded update_status, surface
  taken_over instead of cancelled so the client sees a status
  consistent with the store.

Test: test_cancel_returns_taken_over_when_peer_claims_during_local_cancel
pins the race outcome.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(runs): widen update_status guard, de-duplicate lease helpers, add coverage

Round 3 of reviewer feedback:

- Widen update_status guard to status IN ('pending','running','interrupted').
  The original guard blocked interrupted→error (the rollback finalize path),
  losing the "Rolled back by user" message. interrupted is now permitted
  while error/success stay locked — takeover protection unchanged.

- claim_for_takeover False now re-reads the store row to distinguish causes:
  owner renewed lease → lease_valid_elsewhere; row went terminal →
  not_cancellable; another worker already took it over → taken_over.

- Extract _raise_lease_valid_elsewhere() helper to de-duplicate the
  409+Retry-After block shared across cancel_run and stream_existing_run.

- Extract _lease_expired_or_null() in persistence/run/sql.py to
  de-duplicate the lease-expiry SQL WHERE clause shared by
  claim_for_takeover and list_inflight_with_expired_lease.

- 11 new tests: 5 SQL-layer claim_for_takeover (expired/valid/NULL/
  terminal/nonexistent), 3 _compute_retry_after unit (NULL/unparseable/
  normal), 2 claim re-read precision (terminal/takeover), 1 stream
  endpoint 409+Retry-After.

Not addressed (non-blocking, reviewer agreed):
- The 2–3 store.gets in the takeover cold path: optimizing the API to
  accept a pre-fetched record would couple the router to the manager
  more tightly than justified by the perf gain.
- The lease-expiry inline loop in MemoryRunStore.list_inflight_with_-
  expired_lease pre-computes cutoff once for all rows; switching to the
  shared _is_lease_expired helper would recompute datetime.now() per row
  with no real benefit.

260 related tests pass; 36/36 blocking-IO gate pass; ruff clean.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(runs): de-duplicate lease-expiry helper, restore defensive fallback

Address final round of review feedback:

- Extract is_lease_expired to deerflow.utils.time (no _ prefix, public
  utility). Manager and MemoryRunStore now import from the same place
  instead of the store reaching backward into the manager for a private
  function.

- Restore defensive else-10 fallback in grace_seconds property (removed
  in an earlier round). The guard is unreachable for current callers but
  protects future ones from AttributeError.

- Comment the transient in-memory interrupted vs store error state when
  a local cancel is superseded by a peer takeover.

- Comment the max(1, ...) floor in _compute_retry_after — the floor is
  a lower bound, not a poll interval; clients should apply jitter.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: rayhpeng <rayhpeng@gmail.com>
2026-07-14 07:37:59 +08:00

143 lines
5.3 KiB
Python

"""Tests for idempotent run cancellation (issue #3055).
RunManager.cancel() returns True when a run is already interrupted so that
a second cancel request from the same worker is treated as a no-op success
(202) rather than a conflict (409). Both the POST cancel endpoint and the
POST stream endpoint share this behaviour through the same cancel() call.
"""
from __future__ import annotations
import asyncio
from _router_auth_helpers import make_authed_test_app
from fastapi.testclient import TestClient
from app.gateway.routers import thread_runs
from deerflow.runtime import CancelOutcome, RunManager, RunStatus
THREAD_ID = "thread-cancel-test"
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _make_app(mgr: RunManager) -> TestClient:
app = make_authed_test_app()
app.include_router(thread_runs.router)
app.state.run_manager = mgr
return TestClient(app, raise_server_exceptions=False)
def _create_interrupted_run(mgr: RunManager) -> str:
"""Create a run and cancel it, returning its run_id."""
async def _setup():
record = await mgr.create(THREAD_ID)
await mgr.set_status(record.run_id, RunStatus.running)
await mgr.cancel(record.run_id)
return record.run_id
return asyncio.run(_setup())
# ---------------------------------------------------------------------------
# RunManager.cancel() unit tests
# ---------------------------------------------------------------------------
class TestRunManagerCancelIdempotency:
def test_cancel_returns_cancelled_for_already_interrupted_run(self):
"""cancel() must return CancelledOutcome.cancelled when the run is already interrupted."""
async def run():
mgr = RunManager()
record = await mgr.create(THREAD_ID)
await mgr.set_status(record.run_id, RunStatus.running)
first = await mgr.cancel(record.run_id)
assert first == CancelOutcome.cancelled
second = await mgr.cancel(record.run_id)
assert second == CancelOutcome.cancelled # idempotent
asyncio.run(run())
def test_cancel_returns_not_cancellable_for_successful_run(self):
"""cancel() must return not_cancellable for runs that completed successfully."""
async def run():
mgr = RunManager()
record = await mgr.create(THREAD_ID)
await mgr.set_status(record.run_id, RunStatus.running)
await mgr.set_status(record.run_id, RunStatus.success)
result = await mgr.cancel(record.run_id)
assert result == CancelOutcome.not_cancellable
asyncio.run(run())
def test_cancel_returns_not_active_locally_for_unknown_run(self):
async def run():
mgr = RunManager()
result = await mgr.cancel("nonexistent-run-id")
assert result == CancelOutcome.not_active_locally
asyncio.run(run())
# ---------------------------------------------------------------------------
# POST /cancel endpoint — idempotent 202
# ---------------------------------------------------------------------------
class TestCancelRunEndpointIdempotency:
def test_double_cancel_returns_202_not_409(self):
"""Second cancel on an already-interrupted run must return 202, not 409."""
mgr = RunManager()
run_id = _create_interrupted_run(mgr)
client = _make_app(mgr)
resp = client.post(f"/api/threads/{THREAD_ID}/runs/{run_id}/cancel")
assert resp.status_code == 202, f"Expected 202, got {resp.status_code}: {resp.text}"
def test_cancel_unknown_run_returns_404(self):
mgr = RunManager()
client = _make_app(mgr)
resp = client.post(f"/api/threads/{THREAD_ID}/runs/no-such-run/cancel")
assert resp.status_code == 404
def test_cancel_successful_run_returns_409(self):
"""Successfully-completed runs cannot be cancelled — must return 409."""
async def _setup():
mgr = RunManager()
record = await mgr.create(THREAD_ID)
await mgr.set_status(record.run_id, RunStatus.running)
await mgr.set_status(record.run_id, RunStatus.success)
return mgr, record.run_id
mgr, run_id = asyncio.run(_setup())
client = _make_app(mgr)
resp = client.post(f"/api/threads/{THREAD_ID}/runs/{run_id}/cancel")
assert resp.status_code == 409
# ---------------------------------------------------------------------------
# POST /{thread_id}/runs/{run_id}/join (stream_existing_run) — idempotent cancel
# ---------------------------------------------------------------------------
class TestStreamExistingRunIdempotentCancel:
def test_stream_cancel_already_interrupted_returns_not_409(self):
"""stream_existing_run with action=interrupt on an already-interrupted run
must not raise 409 — the idempotent cancel path returns 202/SSE."""
mgr = RunManager()
run_id = _create_interrupted_run(mgr)
client = _make_app(mgr)
resp = client.post(
f"/api/threads/{THREAD_ID}/runs/{run_id}/join",
params={"action": "interrupt"},
)
assert resp.status_code != 409, f"Should not 409 on idempotent cancel, got {resp.status_code}"