mirror of
https://github.com/milvus-io/milvus.git
synced 2026-07-21 18:25:44 +00:00
wei liuandGitHub
4d24af7063
issue: #45881 ## Summary Add the `milvus-table` external format so external collection refresh can consume Milvus snapshot metadata and StorageV3 segment manifests directly. This change lets refresh resolve source manifest row counts, rebuild target StorageV3 manifests from source column groups, copy source deltalogs with deterministic preallocated IDs, and keep real external primary-key semantics for milvus-table segments. ## Why Snapshot-backed external tables are not generic file tables. The refresh path needs to preserve Milvus StorageV3 manifest structure, deltalogs, and real PK behavior instead of falling back to virtual PK semantics or format-agnostic fragment handling. ## Details - Add milvus-table external spec parsing and snapshot manifest resolution. - Add StorageV3 manifest translation for source Milvus segment manifests. - Copy milvus-table deltalogs into target manifests with deterministic LogIDs. - Wire QueryNode real-PK lookup/load handling for milvus-table segments. - Add unit and Go client coverage for spec parsing, manifest resolution, refresh, and deltalog handling. ## Validation - `git diff --cached --check` - No proto changes detected - No mockery patterns in changed test files - `source ~/.profile && source scripts/setenv.sh && make milvus` - Reset Milvus local standalone before Go tests - `go test -tags dynamic,test -gcflags="all=-N -l" -ldflags="-r ${RPATH}" github.com/milvus-io/milvus/internal/datanode/external github.com/milvus-io/milvus/internal/storagev2/packed -count=1 -timeout 300s` - `source ~/.profile && source scripts/setenv.sh && make lint-fix` Related: [#45881](https://github.com/milvus-io/milvus/issues/45881) --------- Signed-off-by: Wei Liu <wei.liu@zilliz.com>