enhance: revert index_store_path full-path plumbing to montage assembly (#49805)

## Summary

PR #49325 introduced collection-rooted index storage and also added a
precomputed `index_store_path` string flowing through the build path.
This PR scopes that back down: the collection-rooted layout stays, but
the precomputed `index_store_path` string is removed from the
write/build path.

Build/write path:
- DataCoord sends `IndexStorePathVersion` with the build job.
- DataNode forwards the version to C++ indexbuilder.
- C++ `FileManager` assembles the index object prefix locally from
`(path_version, collID, partID, segID, buildID, indexVersion)`.

Read/load path:
- Go keeps passing full `IndexFilePaths` through DataCoord -> QueryCoord
-> QueryNode -> cgo/segcore.
- Load-side protos stay on full paths; no `index_file_keys` /
`index_store_path_version` split is added there.
- C++ load keeps PR #49325's direct full-path open behavior for
`index_files/...` and `index_v1/...` paths.

The v1 root is intentionally named `index_v1` before release. v0 remains
`index_files`.

issue: #49086

## Compatibility note

This PR is wire-incompatible with the short-lived #49325-era internal
schema that carried `index_store_path`.

That is intentional and acceptable because no release has shipped
between the merge of #49325 and this rework. In other words, there are
no released binaries or released persisted wire data that rely on the
#49325-era `index_store_path` field semantics. This PR treats the proto
changes as a pre-release schema correction rather than a compatibility
migration.

Because of that:
- `BuildIndexInfo.index_store_path = 18` is replaced by
`index_store_path_version = 18`.
- `CreateJobRequest.index_store_path = 21` is removed.
- `CreateJobRequest.index_store_path_version = 36` is kept.
- We do not reserve the removed field numbers/names in this PR; if this
schema had crossed a release boundary, we would reserve them and use
fresh field numbers instead.

## What changes

**Protos**
- `worker.proto`: drop `CreateJobRequest.index_store_path`; keep
`CreateJobRequest.index_store_path_version`.
- `index_cgo_msg.proto`: replace `BuildIndexInfo.index_store_path` with
`BuildIndexInfo.index_store_path_version`.
- Read/load-side protos keep their full-path shape: no `IndexFilePaths
-> IndexFileKeys` rename and no load-side `index_store_path_version`
propagation.

**C++**
- `storage::IndexMeta`: replace `std::string index_store_path` with
`IndexStorePathVersion index_store_path_version`.
- `FileManager::GetRemoteIndexObjectPrefix()`: assemble v0/v1 prefixes
from path version and IDs.
- `FileManager::OpenInputStream()`: keep direct full-path open support
for read/load paths.
- `indexbuilder/index_c.cpp`: pass `index_store_path_version` into
`IndexMeta`.
- `DiskFileManagerTest`: cover v0 and v1 prefix assembly.

**Go**
- DataCoord `task_index.go`: stop building/sending `IndexStorePath`;
continue sending `IndexStorePathVersion`.
- DataNode `index/task_index.go`: forward `IndexStorePathVersion` to C++
build info.
- DataNode `index_services.go`: drop logs for the removed
`index_store_path` field.
- DataCoord FinishTask validation and
`SegmentIndex.IndexStorePathVersion` metadata remain unchanged.
- GC/copy/import/snapshot/read path keep full-path behavior; only
expected v1 root strings are updated from `index_files_v1` to
`index_v1`.

## Notes for reviewers

- The read/load path intentionally remains full-path based. QueryNode
should not convert full paths to basenames or parse path version for
load.
- `IndexFilePathInfo.IndexStorePathVersion` remains as DataCoord
metadata, but QueryCoord does not forward it for loading.
- `metautil.IndexPathBuilder` remains needed for Go-side object storage
operations such as GC, snapshot, copy-segment, and DataCoord
`GetIndexInfos` path construction.
- Mixed-version concerns are limited to unreleased #49325-era binaries,
which is why this PR can remove/retype `index_store_path` without
compatibility scaffolding.

## Test plan

- [x] C++ build: `ninja milvus_core` clean.
- [x]
`DiskAnnFileManagerTest.GetRemoteIndexObjectPrefix_{V0BuildRooted,V1CollectionRooted}`
covers v0/v1 prefix assembly.
- [x] Updated Go/C++ tests for the `index_v1` root rename.
- [ ] CI on this PR: code-check, UT, E2E.

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
yihao.dai
2026-05-16 00:02:29 +08:00
committed by GitHub
co-authored by Claude Opus 4.7
parent dd6a411e5e
commit 66fa592752
24 changed files with 963 additions and 914 deletions
+8
View File
@@ -139,6 +139,14 @@ ${protoc_opt} --cpp_out=$CPP_SRC_DIR/src/pb clustering.proto|| { echo 'generate
${protoc_opt} --cpp_out=$CPP_SRC_DIR/src/pb index_cgo_msg.proto|| { echo 'generate index_cgo_msg.proto failed'; exit 1; }
${protoc_opt} --cpp_out=$CPP_SRC_DIR/src/pb cgo_msg.proto|| { echo 'generate cgo_msg.proto failed'; exit 1; }
${protoc_opt} --cpp_out=$CPP_SRC_DIR/src/pb plan.proto|| { echo 'generate plan.proto failed'; exit 1; }
# index_coord.proto and its transitive deps are needed in C++ because segcore.proto,
# index_cgo_msg.proto and cgo_msg.proto now reference milvus.proto.index.IndexStorePathVersion.
${protoc_opt} --cpp_out=$CPP_SRC_DIR/src/pb index_coord.proto|| { echo 'generate index_coord.proto failed'; exit 1; }
${protoc_opt} --cpp_out=$CPP_SRC_DIR/src/pb internal.proto|| { echo 'generate internal.proto failed'; exit 1; }
${protoc_opt} --cpp_out=$CPP_SRC_DIR/src/pb milvus.proto|| { echo 'generate milvus.proto failed'; exit 1; }
${protoc_opt} --cpp_out=$CPP_SRC_DIR/src/pb rg.proto|| { echo 'generate rg.proto failed'; exit 1; }
${protoc_opt} --cpp_out=$CPP_SRC_DIR/src/pb feder.proto|| { echo 'generate feder.proto failed'; exit 1; }
${protoc_opt} --cpp_out=$CPP_SRC_DIR/src/pb msg.proto|| { echo 'generate msg.proto failed'; exit 1; }
restore_unchanged_generated_files