mirror of
https://github.com/milvus-io/milvus.git
synced 2026-07-21 02:05:41 +00:00
Buqian ZhengandGitHub
99097abea8
this PR allows to create Sort/Bitmap/Hybrid index for json path index, and also removed the `JsonInvertedIndex` class, to use `InvertedIndexTantivy` directly for Inverted index type. C++ changes: - Add ConvertJsonToTypedFieldData<T>() to extract typed values from JSON field data with separate tracking of non_exist_offsets for EXISTS semantics - Add JsonScalarIndexWrapper<T, BaseIndex> template for Sort/Bitmap with dual file-manager pattern (original JSON schema for reading, cast-type schema for base index dispatching) - Add JsonHybridScalarIndex<T> with validity-aware cardinality counting - Add IndexBase::Exists() virtual method; override in Sort/Bitmap/Hybrid wrappers using non_exist_offsets (serialized via WriteEntries/LoadEntries) - Simplify ExistsExpr to use index->Exists() uniformly - Extend IndexFactory::CreateJsonIndex() to route STL_SORT/BITMAP/HYBRID Go changes: - Update STL_SORT/Bitmap/Hybrid checkers to accept JSON with cast_type and json_path validation - Change AUTOINDEX default for JSON from INVERTED to HYBRID - Bump ScalarIndexEngineVersion to 4 - Add version gate in DataCoord CreateIndex and snapshot RestoreIndexes - Update test fixtures to cover new (index_type, cast_type) combinations <img width="2380" height="708" alt="image" src="https://github.com/user-attachments/assets/8b3923a0-2cb3-4af7-b73a-b76d1d1ec2d0" /> issue: https://github.com/milvus-io/milvus/issues/48954 design-doc: https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260410-json_path_index_multi_type.md --------- Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
31 lines
924 B
Bash
Executable File
31 lines
924 B
Bash
Executable File
#!/usr/bin/env bash
|
|
|
|
SCRIPTS_DIR=$(dirname "$0")
|
|
THIRD_PARTY_DIR=$SCRIPTS_DIR/../cmake_build/thirdparty
|
|
API_VERSION=$(go list -m github.com/milvus-io/milvus-proto/go-api/v3 | awk -F' ' '{print $2}')
|
|
PROTO_REPO=https://github.com/milvus-io/milvus-proto.git
|
|
|
|
# Try tagged version first.
|
|
COMMIT_ID=$(git ls-remote "$PROTO_REPO" refs/tags/${API_VERSION} | cut -f 1)
|
|
if [[ -z $COMMIT_ID ]]; then
|
|
# Parse commit from pseudo version (eg v0.0.0-20230608062631-c453ef1b870a => c453ef1b870a).
|
|
COMMIT_ID=$(echo $API_VERSION | awk -F'-' '{print $3}')
|
|
fi
|
|
|
|
if [ ! -d "$THIRD_PARTY_DIR/milvus-proto" ]; then
|
|
mkdir -p $THIRD_PARTY_DIR
|
|
pushd $THIRD_PARTY_DIR
|
|
git clone "$PROTO_REPO"
|
|
popd
|
|
fi
|
|
|
|
pushd "$THIRD_PARTY_DIR/milvus-proto"
|
|
git fetch --tags origin
|
|
echo "version: $API_VERSION, commitID: $COMMIT_ID"
|
|
if [ -z $COMMIT_ID ]; then
|
|
git checkout -B "$API_VERSION" "$API_VERSION"
|
|
else
|
|
git reset --hard "$COMMIT_ID"
|
|
fi
|
|
popd
|