mirror of
https://github.com/milvus-io/milvus.git
synced 2026-07-21 18:25:44 +00:00
8275d10583
## Summary Implements Two-Phase Commit (2PC) for Import in primary/secondary replication clusters. Data stays invisible (`is_importing=true`) until an explicit commit signal is delivered via WAL, ensuring primary and secondary clusters reach the same visible state at the same logical position. - **New proto**: `CommitImport=44`, `RollbackImport=45` WAL message types; `Uncommitted=8`, `Committing=9` `ImportJobState` values; `CommitImport`, `AbortImport`, `HandleCommitVchannel` RPCs on DataCoord; `committed_vchannels` + `auto_commit` fields on `ImportJob` - **WAL broadcast**: DataCoord broadcasts `CommitImportMessage`/`RollbackImportMessage` to all vchannels via DDL broadcast; CDC replicates to secondary clusters verbatim - **DDL ack callbacks**: `commitImportV2AckCallback` CAS `Uncommitted→Committing`; `rollbackImportV2AckCallback` CAS `*→Failed` + segment drop - **WAL flusher**: `wal_flusher.dispatch()` intercepts `CommitImportMessage` per-vchannel, calls `wbMgr.FlushChannel` + `DataCoord.HandleCommitVchannel`; no-op handler for `RollbackImportMessage` - **ImportChecker**: new `Uncommitted` case (auto-commits when `auto_commit=true`); new `Committing` case (transitions to `Completed` when all vchannels confirmed) - **auto_commit option**: default `true` (backward compatible); `false` lets replication platform control commit timing - **RESTful API**: `POST /v2/vectordb/jobs/import/commit` and `POST /v2/vectordb/jobs/import/abort`; `GetImportProgress` surfaces `Uncommitted` and `Committing` states **Out of scope**: `commit_timestamp` propagation (handled in companion PR) ## Test Plan - [ ] Unit tests for `IsAutoCommit` helper - [ ] Unit tests for `HandleCommitVchannel` idempotency - [ ] Unit tests for `CommitImport`/`AbortImport` RPC handlers (state validation, mutex, broadcast) - [ ] Unit tests for DDL ack callbacks (CAS races for commit/abort) - [ ] Unit tests for `ImportChecker` `Uncommitted`/`Committing` cases - [ ] Unit tests for WAL flusher dispatch (CommitImport/RollbackImport cases) - [ ] Unit tests for RESTful commit/abort handlers (valid jobId, invalid jobId error path) - [ ] `GetImportProgress` surfaces `Uncommitted` and `Committing` states - [ ] E2E: non-replication cluster with `auto_commit=true` (default) behaves identically to pre-2PC issue: #48525 design doc: https://github.com/milvus-io/milvus-design-docs/pull/29 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Yihao Dai <yihao.dai@zilliz.com> Signed-off-by: bigsheeper <yihao.dai@zilliz.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>