mirror of
https://github.com/milvus-io/milvus.git
synced 2026-07-21 10:15:43 +00:00
enhance: revert index_store_path full-path plumbing to montage assembly (#49805)
## Summary PR #49325 introduced collection-rooted index storage and also added a precomputed `index_store_path` string flowing through the build path. This PR scopes that back down: the collection-rooted layout stays, but the precomputed `index_store_path` string is removed from the write/build path. Build/write path: - DataCoord sends `IndexStorePathVersion` with the build job. - DataNode forwards the version to C++ indexbuilder. - C++ `FileManager` assembles the index object prefix locally from `(path_version, collID, partID, segID, buildID, indexVersion)`. Read/load path: - Go keeps passing full `IndexFilePaths` through DataCoord -> QueryCoord -> QueryNode -> cgo/segcore. - Load-side protos stay on full paths; no `index_file_keys` / `index_store_path_version` split is added there. - C++ load keeps PR #49325's direct full-path open behavior for `index_files/...` and `index_v1/...` paths. The v1 root is intentionally named `index_v1` before release. v0 remains `index_files`. issue: #49086 ## Compatibility note This PR is wire-incompatible with the short-lived #49325-era internal schema that carried `index_store_path`. That is intentional and acceptable because no release has shipped between the merge of #49325 and this rework. In other words, there are no released binaries or released persisted wire data that rely on the #49325-era `index_store_path` field semantics. This PR treats the proto changes as a pre-release schema correction rather than a compatibility migration. Because of that: - `BuildIndexInfo.index_store_path = 18` is replaced by `index_store_path_version = 18`. - `CreateJobRequest.index_store_path = 21` is removed. - `CreateJobRequest.index_store_path_version = 36` is kept. - We do not reserve the removed field numbers/names in this PR; if this schema had crossed a release boundary, we would reserve them and use fresh field numbers instead. ## What changes **Protos** - `worker.proto`: drop `CreateJobRequest.index_store_path`; keep `CreateJobRequest.index_store_path_version`. - `index_cgo_msg.proto`: replace `BuildIndexInfo.index_store_path` with `BuildIndexInfo.index_store_path_version`. - Read/load-side protos keep their full-path shape: no `IndexFilePaths -> IndexFileKeys` rename and no load-side `index_store_path_version` propagation. **C++** - `storage::IndexMeta`: replace `std::string index_store_path` with `IndexStorePathVersion index_store_path_version`. - `FileManager::GetRemoteIndexObjectPrefix()`: assemble v0/v1 prefixes from path version and IDs. - `FileManager::OpenInputStream()`: keep direct full-path open support for read/load paths. - `indexbuilder/index_c.cpp`: pass `index_store_path_version` into `IndexMeta`. - `DiskFileManagerTest`: cover v0 and v1 prefix assembly. **Go** - DataCoord `task_index.go`: stop building/sending `IndexStorePath`; continue sending `IndexStorePathVersion`. - DataNode `index/task_index.go`: forward `IndexStorePathVersion` to C++ build info. - DataNode `index_services.go`: drop logs for the removed `index_store_path` field. - DataCoord FinishTask validation and `SegmentIndex.IndexStorePathVersion` metadata remain unchanged. - GC/copy/import/snapshot/read path keep full-path behavior; only expected v1 root strings are updated from `index_files_v1` to `index_v1`. ## Notes for reviewers - The read/load path intentionally remains full-path based. QueryNode should not convert full paths to basenames or parse path version for load. - `IndexFilePathInfo.IndexStorePathVersion` remains as DataCoord metadata, but QueryCoord does not forward it for loading. - `metautil.IndexPathBuilder` remains needed for Go-side object storage operations such as GC, snapshot, copy-segment, and DataCoord `GetIndexInfos` path construction. - Mixed-version concerns are limited to unreleased #49325-era binaries, which is why this PR can remove/retype `index_store_path` without compatibility scaffolding. ## Test plan - [x] C++ build: `ninja milvus_core` clean. - [x] `DiskAnnFileManagerTest.GetRemoteIndexObjectPrefix_{V0BuildRooted,V1CollectionRooted}` covers v0/v1 prefix assembly. - [x] Updated Go/C++ tests for the `index_v1` root rename. - [ ] CI on this PR: code-check, UT, E2E. Signed-off-by: bigsheeper <yihao.dai@zilliz.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.7
parent
dd6a411e5e
commit
66fa592752
@@ -139,6 +139,14 @@ ${protoc_opt} --cpp_out=$CPP_SRC_DIR/src/pb clustering.proto|| { echo 'generate
|
||||
${protoc_opt} --cpp_out=$CPP_SRC_DIR/src/pb index_cgo_msg.proto|| { echo 'generate index_cgo_msg.proto failed'; exit 1; }
|
||||
${protoc_opt} --cpp_out=$CPP_SRC_DIR/src/pb cgo_msg.proto|| { echo 'generate cgo_msg.proto failed'; exit 1; }
|
||||
${protoc_opt} --cpp_out=$CPP_SRC_DIR/src/pb plan.proto|| { echo 'generate plan.proto failed'; exit 1; }
|
||||
# index_coord.proto and its transitive deps are needed in C++ because segcore.proto,
|
||||
# index_cgo_msg.proto and cgo_msg.proto now reference milvus.proto.index.IndexStorePathVersion.
|
||||
${protoc_opt} --cpp_out=$CPP_SRC_DIR/src/pb index_coord.proto|| { echo 'generate index_coord.proto failed'; exit 1; }
|
||||
${protoc_opt} --cpp_out=$CPP_SRC_DIR/src/pb internal.proto|| { echo 'generate internal.proto failed'; exit 1; }
|
||||
${protoc_opt} --cpp_out=$CPP_SRC_DIR/src/pb milvus.proto|| { echo 'generate milvus.proto failed'; exit 1; }
|
||||
${protoc_opt} --cpp_out=$CPP_SRC_DIR/src/pb rg.proto|| { echo 'generate rg.proto failed'; exit 1; }
|
||||
${protoc_opt} --cpp_out=$CPP_SRC_DIR/src/pb feder.proto|| { echo 'generate feder.proto failed'; exit 1; }
|
||||
${protoc_opt} --cpp_out=$CPP_SRC_DIR/src/pb msg.proto|| { echo 'generate msg.proto failed'; exit 1; }
|
||||
|
||||
restore_unchanged_generated_files
|
||||
|
||||
|
||||
Reference in New Issue
Block a user