mirror of
https://github.com/milvus-io/milvus.git
synced 2026-07-21 18:25:44 +00:00
66fa592752
## Summary PR #49325 introduced collection-rooted index storage and also added a precomputed `index_store_path` string flowing through the build path. This PR scopes that back down: the collection-rooted layout stays, but the precomputed `index_store_path` string is removed from the write/build path. Build/write path: - DataCoord sends `IndexStorePathVersion` with the build job. - DataNode forwards the version to C++ indexbuilder. - C++ `FileManager` assembles the index object prefix locally from `(path_version, collID, partID, segID, buildID, indexVersion)`. Read/load path: - Go keeps passing full `IndexFilePaths` through DataCoord -> QueryCoord -> QueryNode -> cgo/segcore. - Load-side protos stay on full paths; no `index_file_keys` / `index_store_path_version` split is added there. - C++ load keeps PR #49325's direct full-path open behavior for `index_files/...` and `index_v1/...` paths. The v1 root is intentionally named `index_v1` before release. v0 remains `index_files`. issue: #49086 ## Compatibility note This PR is wire-incompatible with the short-lived #49325-era internal schema that carried `index_store_path`. That is intentional and acceptable because no release has shipped between the merge of #49325 and this rework. In other words, there are no released binaries or released persisted wire data that rely on the #49325-era `index_store_path` field semantics. This PR treats the proto changes as a pre-release schema correction rather than a compatibility migration. Because of that: - `BuildIndexInfo.index_store_path = 18` is replaced by `index_store_path_version = 18`. - `CreateJobRequest.index_store_path = 21` is removed. - `CreateJobRequest.index_store_path_version = 36` is kept. - We do not reserve the removed field numbers/names in this PR; if this schema had crossed a release boundary, we would reserve them and use fresh field numbers instead. ## What changes **Protos** - `worker.proto`: drop `CreateJobRequest.index_store_path`; keep `CreateJobRequest.index_store_path_version`. - `index_cgo_msg.proto`: replace `BuildIndexInfo.index_store_path` with `BuildIndexInfo.index_store_path_version`. - Read/load-side protos keep their full-path shape: no `IndexFilePaths -> IndexFileKeys` rename and no load-side `index_store_path_version` propagation. **C++** - `storage::IndexMeta`: replace `std::string index_store_path` with `IndexStorePathVersion index_store_path_version`. - `FileManager::GetRemoteIndexObjectPrefix()`: assemble v0/v1 prefixes from path version and IDs. - `FileManager::OpenInputStream()`: keep direct full-path open support for read/load paths. - `indexbuilder/index_c.cpp`: pass `index_store_path_version` into `IndexMeta`. - `DiskFileManagerTest`: cover v0 and v1 prefix assembly. **Go** - DataCoord `task_index.go`: stop building/sending `IndexStorePath`; continue sending `IndexStorePathVersion`. - DataNode `index/task_index.go`: forward `IndexStorePathVersion` to C++ build info. - DataNode `index_services.go`: drop logs for the removed `index_store_path` field. - DataCoord FinishTask validation and `SegmentIndex.IndexStorePathVersion` metadata remain unchanged. - GC/copy/import/snapshot/read path keep full-path behavior; only expected v1 root strings are updated from `index_files_v1` to `index_v1`. ## Notes for reviewers - The read/load path intentionally remains full-path based. QueryNode should not convert full paths to basenames or parse path version for load. - `IndexFilePathInfo.IndexStorePathVersion` remains as DataCoord metadata, but QueryCoord does not forward it for loading. - `metautil.IndexPathBuilder` remains needed for Go-side object storage operations such as GC, snapshot, copy-segment, and DataCoord `GetIndexInfos` path construction. - Mixed-version concerns are limited to unreleased #49325-era binaries, which is why this PR can remove/retype `index_store_path` without compatibility scaffolding. ## Test plan - [x] C++ build: `ninja milvus_core` clean. - [x] `DiskAnnFileManagerTest.GetRemoteIndexObjectPrefix_{V0BuildRooted,V1CollectionRooted}` covers v0/v1 prefix assembly. - [x] Updated Go/C++ tests for the `index_v1` root rename. - [ ] CI on this PR: code-check, UT, E2E. Signed-off-by: bigsheeper <yihao.dai@zilliz.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Compile and install milvus cluster
Environment
OS: Ubuntu 20.04
go：1.21
cmake: >=3.18
gcc: >= 11
Install dependencies
Install compile dependencies
$ sudo apt install -y g++ gcc make libssl-dev zlib1g-dev libboost-regex-dev \
libboost-program-options-dev libboost-system-dev libboost-filesystem-dev \
libboost-serialization-dev python3-dev libboost-python-dev libcurl4-openssl-dev gfortran libtbb-dev
$ export GO111MODULE=on
$ go get github.com/golang/protobuf/protoc-gen-go@v1.3.2
Install OpenBlas library
install using apt
sudo apt install -y libopenblas-dev
or build from source code
$ wget https://github.com/xianyi/OpenBLAS/archive/v0.3.9.tar.gz && \
$ tar zxvf v0.3.9.tar.gz && cd OpenBLAS-0.3.9 && \
$ make TARGET=CORE2 DYNAMIC_ARCH=1 DYNAMIC_OLDER=1 USE_THREAD=0 USE_OPENMP=0 FC=gfortran CC=gcc COMMON_OPT="-O3 -g -fPIC" FCOMMON_OPT="-O3 -g -fPIC -frecursive" NMAX="NUM_THREADS=128" LIBPREFIX="libopenblas" INTERFACE64=0 NO_STATIC=1 && \
$ make PREFIX=/usr install
Compile
Generate the go files from proto file
$ make check-proto-product
Check code specifications
$ make verifiers
Compile milvus
$ make milvus
Install docker-compose
refer: https://docs.docker.com/compose/install/
$ sudo curl -L "https://github.com/docker/compose/releases/download/1.27.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
$ sudo chmod +x /usr/local/bin/docker-compose
$ sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
$ docker-compose --version
$ docker compose --version
Start service
Start third-party service:
$ cd [milvus project path]/deployments/docker/cluster
$ docker-compose up -d
$ docker compose up -d
Start milvus cluster:
$ cd [milvus project path]
$ ./scripts/start_cluster.sh
Run unittest
Run all unittest including go and cpp cases:
$ make unittest
You also can run go unittest only:
$ make test-go
Run cpp unittest only:
$ make test-cpp
Run code coverage
Run code coverage including go and cpp:
$ make codecov
You also can run go code coverage only:
$ make codecov-go
Run cpp code coverage only:
$ make codecov-cpp