mirror of
https://github.com/milvus-io/milvus.git
synced 2026-07-21 10:15:43 +00:00
10cd30c66e
## Summary In CDC replication mode, a 2PC bulk import (`auto_commit=false`) runs **independently** on the primary (source) and standby clusters against the same files. Only the source can broadcast `Commit`/`Rollback`; the standby replays them via CDC. When the import fails on **one** cluster but the other holds a replicated `Uncommitted` job, that job had no recovery path: - **Mode 1 (source fails, standby `Uncommitted`):** the failure never broadcast a `RollbackImport`, and `AbortImport` on a terminal-`Failed` source was rejected — so the standby stayed `Uncommitted` indefinitely, holding invisible imported segments. (Observed in production: standby stuck 2+ days.) - **Mode 2 (standby fails, source `Uncommitted`):** if the platform commits the source before confirming the standby is prepared, the source commits while the standby's commit ack no-ops on its `Failed` job → silent divergence. issue: #51070 ## What this changes (datacoord) - **Abort a failed source (`AbortImport`):** a `Failed` source (its own import failed) is now abortable — it broadcasts `RollbackImport` to release the peer's `Uncommitted` job. `Committing`/`Completed` stay rejected. Lets an operator recover **immediately**, instead of waiting for GC. - **GC self-heal (`importChecker.checkGC`):** before GC removes a `Failed` replicate-source job, it broadcasts `RollbackImport` so the peer is released **automatically at retention expiry**. Transient broadcast failure keeps the job and retries on the next tick; a standby's `ErrNotPrimary` proceeds; **job removal itself is the idempotency guard** (once gone, never re-broadcast). This closes the recovery-window gap without a persisted flag or proto change. - **Divergence signal (`CommitImport` ack):** a commit ack landing on a `Failed` replica now logs at WARN (Mode 2). - `isReplicatingClusterNow`: the GC decision now **propagates** replication-check errors instead of tolerating them — an indeterminate status (transient balancer error, `OnShutdownError`) retains the job and retries next tick, since a false "not replicating" would irreversibly strand the replicating peer; only permanent broadcast errors (`ErrNotPrimary` from a standby, `ErrCollectionNotFound` after a replicated drop) proceed to GC. The balancer access is bounded by a 10s timeout so an unregistered balancer cannot park the checker loop. - Minor: the coordinator callbacks the checker needs are bundled into an `importCheckerHooks` struct instead of a growing positional-arg list. ## Recovery guarantee The peer's `Uncommitted` job is released either **immediately** via `AbortImport`, or **automatically** when the failed source reaches GC retention — no permanent stranding, and no dependence on a manual abort within the 3h retention window. ## Not in scope (follow-ups) - **Idempotent re-broadcast dedup:** repeated `AbortImport` on a real-failure source re-broadcasts each time (harmless — flusher/standby-ack no-op). A persisted `rollback_broadcasted` flag would dedup this. - **Eager self-heal:** broadcasting the rollback at *failure* time (in `checkFailedJob`) rather than at GC time, to shorten peer-stranding to seconds. - **Mode 2 prevention:** a standby→coordinator prepared-vote so the source refuses to commit unless all replicas are prepared. ## Test plan - `checkGC` self-heal: failed replicate-source broadcasts-then-removes with transient-retry; `ErrNotPrimary` (standby) proceeds; non-replicating skips the broadcast. - `AbortImport`: failed-source broadcasts rollback; `Committing` and `Completed` rejected; previously-user-aborted is idempotent. - CI `ci-v2/ut-go` compiles and runs these tests. (Local datacoord UT was not run for this revision due to cgo core drift in the worktree.) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>