mirror of
https://github.com/milvus-io/milvus.git
synced 2026-07-21 10:15:43 +00:00
yihao.daiandGitHub
ddf3e4d558
## Summary Fix L0 import storage version selection when `UseLoonFFI` is enabled: - Force DataCoord to assign StorageV2 for L0 import segments. - Force L0 import requests to disable Loon FFI. - Defensively keep DataNode L0 sync tasks on StorageV2 even if the request carries StorageV3. - Add regression coverage for L0 import with `UseLoonFFI=true`. Fixes #51148 ## Known mixed-version window (documented, not fixed here) The `AssignSegments` metadata fix only applies to jobs created on the new version. A job created by a **pre-upgrade** DataCoord with `common.storage.useLoonFFI=true` has already persisted its L0 segments with `StorageVersion=3` in etcd, and segment IDs are reused (not reallocated) when the task is recovered. If such an in-flight job completes **after** the upgrade — either the upgraded DataCoord recovers the task and reassembles the request as V2, or an upgraded DataNode serves the old coordinator's V3 request (`syncDelete` now hard-codes V2) — the deltalogs are written as V2 but the segment stays tagged `StorageVersion=3` with an empty `ManifestPath`: the import completion path does not reconcile `StorageVersion`, and L0 segments are excluded from the storage-version compaction policy, so nothing self-heals the tag. Impact: harmless for delta load and L0 compaction (both branch on the manifest, not the tag), but snapshot restore trusts the tag — `collectSegmentFiles` fails on `storage_version>=3` with an empty manifest. Scope: requires a deployment running the experimental `useLoonFFI=true` with an L0 import job in flight at the moment of the upgrade, later restored via snapshot. Since the pre-upgrade V3 L0 write path always fails (#51148), no successfully-written V3 L0 data exists — only the metadata tag can be stale. If this window ever materializes in practice, the backfill is a one-operator change in the completion path (`UpdateStorageVersionOperator(segID, StorageV2)` for L0). ## Test Plan - [x] `make static-check` - [x] `go test -tags dynamic,test -gcflags="all=-N -l" -count=1 ./internal/datacoord -run TestImportUtil_L0ImportUsesStorageV2WhenLoonFFIEnabled` - [x] `go test -tags dynamic,test -gcflags="all=-N -l" -count=1 ./internal/datanode/importv2 -run 'TestL0Import/TestL0ImportForcesStorageV2ForSyncTask'` Note: after rebasing to latest master, the focused local tests could not be rerun with the reused local C++ output because the local `libmilvus_core` is older than current headers and misses `_SegcoreSetPrefetchThreadPoolNum`. The same focused tests passed before the rebase; CI should build with matching artifacts. Signed-off-by: Yihao Dai <yihao.dai@zilliz.com>