mirror of
https://github.com/milvus-io/milvus.git
synced 2026-07-21 10:15:43 +00:00
b66f55fe67
issue: #51191 Fixes three robustness defects in the copy-segment restore pipeline (`internal/datacoord/copy_segment_*.go`), each with a unit test covering the failure mode. ### 1. `checkPendingJob` cannot resume a partially created job (high) Task creation is a multi-step sequence (per-group `AllocID` + `AddTask`, each persisted individually, then the `Pending → Executing` transition). If any step fails mid-way (etcd hiccup, DataCoord restart), the job stayed `Pending` with only a subset of tasks persisted, and the old early-return `if len(tasks) > 0 { return }` fired on every subsequent round — the restore hung until the job timeout with no way to progress. Now the transition is resume-safe: it computes the source segments already covered by persisted tasks, creates tasks only for the uncovered mappings, and always (re-)applies the idempotent `Executing` transition (so a round that persisted all tasks but failed the job update also recovers). ### 2. Transient `QueryCopySegment` RPC error failed the whole restore (medium) A single network blip or DataNode restart during a status poll permanently failed the task and its parent restore job. This was inconsistent with the import pipeline, where a `QueryImport` RPC error resets the task for retry instead of failing the job. Now a query RPC error is logged and the task is kept `InProgress` for re-query on the next tick. The job-level timeout still bounds a permanently unreachable node; worker-reported task failures keep the fail-fast behavior. ### 3. `copySegmentTask.Clone` mutated shared state before persisting (medium) `Clone` copied the `atomic.Pointer` value, not the protobuf, so the "clone" used by `UpdateTask` shared the same `*datapb.CopySegmentTask` as the cached entry. Update actions mutate the proto in place, so the in-memory task was mutated *before* `SaveCopySegmentTask` — a failed catalog save left the coordinator cache reflecting the new state while etcd kept the old one. Now `Clone` deep-copies with `proto.Clone`, matching `copySegmentJob.Clone`. ### Testing - `TestCheckPendingJob_ResumesPartialTaskCreation`, `TestCheckPendingJob_AllTasksExistTransitionsToExecuting` - `TestQueryTaskOnWorker_RPCErrorKeepsTaskInProgress` - `TestClone_DeepCopiesProto`, `TestUpdateTask_SaveFailureLeavesCacheUnchanged` - Full `./internal/datacoord/` package: PASS 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: cai.zhang <cai.zhang@zilliz.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>