mirror of
https://github.com/milvus-io/milvus.git
synced 2026-07-21 10:15:43 +00:00
Buqian ZhengandGitHub
99097abea8
this PR allows to create Sort/Bitmap/Hybrid index for json path index, and also removed the `JsonInvertedIndex` class, to use `InvertedIndexTantivy` directly for Inverted index type. C++ changes: - Add ConvertJsonToTypedFieldData<T>() to extract typed values from JSON field data with separate tracking of non_exist_offsets for EXISTS semantics - Add JsonScalarIndexWrapper<T, BaseIndex> template for Sort/Bitmap with dual file-manager pattern (original JSON schema for reading, cast-type schema for base index dispatching) - Add JsonHybridScalarIndex<T> with validity-aware cardinality counting - Add IndexBase::Exists() virtual method; override in Sort/Bitmap/Hybrid wrappers using non_exist_offsets (serialized via WriteEntries/LoadEntries) - Simplify ExistsExpr to use index->Exists() uniformly - Extend IndexFactory::CreateJsonIndex() to route STL_SORT/BITMAP/HYBRID Go changes: - Update STL_SORT/Bitmap/Hybrid checkers to accept JSON with cast_type and json_path validation - Change AUTOINDEX default for JSON from INVERTED to HYBRID - Bump ScalarIndexEngineVersion to 4 - Add version gate in DataCoord CreateIndex and snapshot RestoreIndexes - Update test fixtures to cover new (index_type, cast_type) combinations <img width="2380" height="708" alt="image" src="https://github.com/user-attachments/assets/8b3923a0-2cb3-4af7-b73a-b76d1d1ec2d0" /> issue: https://github.com/milvus-io/milvus/issues/48954 design-doc: https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260410-json_path_index_multi_type.md --------- Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
Compile and install milvus cluster
Environment
OS: Ubuntu 20.04
go：1.21
cmake: >=3.18
gcc: >= 11
Install dependencies
Install compile dependencies
$ sudo apt install -y g++ gcc make libssl-dev zlib1g-dev libboost-regex-dev \
libboost-program-options-dev libboost-system-dev libboost-filesystem-dev \
libboost-serialization-dev python3-dev libboost-python-dev libcurl4-openssl-dev gfortran libtbb-dev
$ export GO111MODULE=on
$ go get github.com/golang/protobuf/protoc-gen-go@v1.3.2
Install OpenBlas library
install using apt
sudo apt install -y libopenblas-dev
or build from source code
$ wget https://github.com/xianyi/OpenBLAS/archive/v0.3.9.tar.gz && \
$ tar zxvf v0.3.9.tar.gz && cd OpenBLAS-0.3.9 && \
$ make TARGET=CORE2 DYNAMIC_ARCH=1 DYNAMIC_OLDER=1 USE_THREAD=0 USE_OPENMP=0 FC=gfortran CC=gcc COMMON_OPT="-O3 -g -fPIC" FCOMMON_OPT="-O3 -g -fPIC -frecursive" NMAX="NUM_THREADS=128" LIBPREFIX="libopenblas" INTERFACE64=0 NO_STATIC=1 && \
$ make PREFIX=/usr install
Compile
Generate the go files from proto file
$ make check-proto-product
Check code specifications
$ make verifiers
Compile milvus
$ make milvus
Install docker-compose
refer: https://docs.docker.com/compose/install/
$ sudo curl -L "https://github.com/docker/compose/releases/download/1.27.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
$ sudo chmod +x /usr/local/bin/docker-compose
$ sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
$ docker-compose --version
$ docker compose --version
Start service
Start third-party service:
$ cd [milvus project path]/deployments/docker/cluster
$ docker-compose up -d
$ docker compose up -d
Start milvus cluster:
$ cd [milvus project path]
$ ./scripts/start_cluster.sh
Run unittest
Run all unittest including go and cpp cases:
$ make unittest
You also can run go unittest only:
$ make test-go
Run cpp unittest only:
$ make test-cpp
Run code coverage
Run code coverage including go and cpp:
$ make codecov
You also can run go code coverage only:
$ make codecov-go
Run cpp code coverage only:
$ make codecov-cpp