issue: https://github.com/milvus-io/milvus/issues/51466
## What changed
`PhyUnaryRangeFilterExpr::DetermineExecPath()` previously used a
JSON-specific hard-coded denylist for string predicates:
- `Match`
- `PostfixMatch`
- `InnerMatch`
This bypassed the concrete scalar index's `ShouldUseOp()` policy. As a
result:
- indexes capable of evaluating these predicates, such as `STL_SORT`,
were
forced to scan and parse raw JSON;
- JSON string predicates could use a different execution policy from
ordinary
VARCHAR predicates;
- `RegexMatch` remained on the index path even for indexes whose planner
policy
explicitly preferred raw-data execution.
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
## Summary
- Reuse the existing `ExprResCacheManager/ExprCacheHelper` from the
expression layer.
- Cache JSON-flat exact-path validity bitmaps by segment + field/full
JSON pointer + `Any|Numeric|String|Bool` canonical key.
- Exclude literal/operator from the key so validity can be reused across
predicates.
- Do not add a `JsonFlatIndex`-owned LRU or change the index format.
This is based on the latest `master`, including merged PR #51235.
## Verification
Verification is pending and will continue after the PR is created.
issue: #51234
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
## Summary
- add `json_subpaths` support to Tantivy `json_exist_query`
- expose exact-path existence through `ExistsQuery(..., false)`
- use a contains-specific `JsonExactPath` validity mode for actual
JSON-flat executors
- keep contains hit lookup through `In()` while using exact-path
non-null-leaf validity
## Semantics
- NULL, missing paths, JSON null, and object-only paths remain UNKNOWN
- `NOT UNKNOWN` remains UNKNOWN
- scalar values and non-empty arrays preserve current flattened contains
behavior
- empty arrays remain the documented limitation
## Verification
- `JsonFlatIndex*`: 25/25 passed
- `*JsonContains*`: 597/597 passed
- `cargo fmt --check`
- clang-format dry-run on changed non-generated C++ files
- `git diff --check`
## Benchmark
For scalar-benchmark
`json_array_operations/base/A01_contains_flat_array/json_flat` with 1M
rows and 2 matches:
- first query: 24.945 ms
- average: 24.625 ms
- P50/P99: 24.525/24.607 ms
- current local master: approximately 1114/1119 ms P50/P99
issue: #51234
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
## Summary
issue: #50976
This PR fixes NULL / UNKNOWN propagation gaps in filter expressions for
missing nested values:
- Treat missing JSON arithmetic operands and JSON `array_length`
extraction failures as UNKNOWN instead of definite booleans.
- Preserve UNKNOWN during parser/RBO contradiction folds for nested JSON
paths, array subscripts, and struct-array subscripts.
- Treat missing array / struct-array elements as UNKNOWN during
execution.
- Propagate JSON path/cast index typed-known masks so missing paths and
cast failures are not matched by `!=`, `not in`, or outer `not(...)`.
- Propagate JSON flat index comparable-value masks for comparison
predicates.
The commits are split by issue area:
- `4f15d6fe79` `fix: treat missing json arithmetic operands as unknown`
- `69d6e95917` `fix: preserve unknown for missing nested predicates in
rbo`
- `51b58edf05` `fix: treat missing array elements as unknown`
- `d9dd4c9334` `fix: honor json path index unknown values`
- `a9d813b322` `fix: honor json flat index unknown values`
## Verification
- `LOCALSTORAGE_PATH=/tmp/milvus-test-localstorage go test
-buildvcs=false -count=1 ./internal/parser/planparserv2/rewriter`
- `ninja -C /tmp/milvus-50976-build all_tests -j32`
- focused C++ regressions, including
`JsonFlatIndexTest.*:JsonFlatIndexExprTest.*:JsonIndexTest.*:JsonPathIndexTest.*`
(49 tests)
- focused array regressions including
`Expr.TestArraySubscriptMissingElementIsUnknown` and `Expr.TestArray*`
- `git diff --check origin/master...HEAD`
## Note
While adding extra JsonFlatIndex null-expression coverage, a separate
pre-existing `PhyNullExpr::DetermineExecPath()` / `std::call_once`
deadlock was found. That hanging test is not included here; this PR
keeps JsonFlatIndex coverage scoped to comparison UNKNOWN behavior.
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
issue: #51069
This PR fixes ARRAY<FLOAT> brute-force `array_contains` execution by
dispatching FLOAT arrays through `ExecArrayContains<float>` /
`ExecArrayContainsAll<float>` instead of sharing the DOUBLE path.
Summary:
- Split ARRAY<FLOAT> contains dispatch from DOUBLE in
`JsonContainsExpr.cpp`.
- Added a regression covering the reported float literals for
`contains`, `contains_any`, and `contains_all`.
Verification from branch author:
- `clang-format-15` on touched files.
- `git diff --check --
internal/core/src/exec/expression/JsonContainsExpr.cpp
internal/core/src/exec/expression/ExprArrayTest.cpp`.
- Targeted C++ object compile was blocked by unrelated stale local
Conan/generated-header issues.
Verification before PR creation:
- `git diff --check origin/master...buqian/bob/fix-51069-array-float`.
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
## Summary
- Adds a `FilterBitsNode` predicate-to-filtered-bitset fast path for
all-valid predicate results: when the live validity bitmap is all true,
flip the predicate data bitmap directly and skip invalid/UNKNOWN bitmap
materialization.
- Preserves NULL/UNKNOWN semantics for mixed-validity predicate results
by keeping the invalid-row OR path, so only definitely TRUE predicate
rows pass.
- Adds focused `FilterBitsNodeTest` coverage for the all-valid fast path
and the mixed-validity path where invalid/UNKNOWN rows must still be
filtered out.
## Scope note
The fix is intentionally at the `FilterBitsNode` conversion boundary. It
does not add `ColumnVector` null-count metadata or producer-side
validity propagation. Producers that return an all-true validity bitmap
still use the fast path through the runtime `valid.all()` check;
mixed-validity results keep the existing three-valued-logic handling.
issue: #51034
## Benchmark context
Local scalar-benchmark config:
`/home/zilliz/.slock/agents/dde3be7c-41b3-4d6c-9efc-449ebfa3d026/bench_configs/array_contains_0624_min.yaml`
with 8 tests, upload disabled.
| Build | Bundle | Avg QPS | Avg p50 latency |
|---|---:|---:|---:|
| pre-`f0bab30fb3` baseline | `1783071127408` | 13722.301 | 0.077 ms |
| `f0bab30fb3` regression | `1783073484488` | 10277.493 | 0.098 ms |
| current branch with this fix | `1783090388177` | 11078.635 | 0.088 ms
|
Note: the current-branch benchmark is not a perfect apples-to-apples
comparison with the old baseline/regressed commits because later
scalar-expression changes are also included. The local run is meant to
validate the `FilterBitsNode` fast-path direction on the same minimal
`array_contains` workload.
## Verification
- [x] `git diff --check origin/master...HEAD`
- [x]
`PROTOC=/home/zilliz/scalar-benchmark/milvus-worktree-filterbits-fastpath/cmake_build/bin/protoc
ninja -C cmake_build all_tests`
- [x] `source ./scripts/setenv.sh && ./cmake_build/unittest/all_tests
--gtest_filter='FilterBitsNodeTest.*:DetermineExecPathTest.FilterBitsNode_*:ArrayBitmapE2ECheck*.CountFuncTest:ArrayBitmapE2ECheck*.INFuncTest:ArrayBitmapE2ECheck*.NotINFuncTest:Naive/ArrayInvertedIndexTest/*.ArrayContainsAny:Naive/ArrayInvertedIndexTest/*.ArrayContainsAll:ConjunctExprTest.AndKeepsUnknownRowsActiveForFollowingFalse:ColumnVectorTest.*'`
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
## Summary
issue: #51030
Add a sealed-segment fast path for filling VARCHAR primary keys from
chunked/StorageV2 sealed segments. The change bulk reads the VARCHAR
primary-key column by segment offsets and avoids the generic raw-data
path while preserving correctness and storage cost accounting.
This branch is cherry-picked from
`00261e8b48127401db6ac1ba4e39003a849d0d76` onto `origin/master`.
## Verification
- `git diff --check origin/master...HEAD`
Local `cmake --build cmake_build --target all_tests -j32` did not reach
compilation because the existing build directory fails master
reconfigure with missing `milvus-commonConfig.cmake`; this appears to be
a local build-dir/dependency configuration issue rather than a patch
compile failure.
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
## Summary
issue: #50910
issue: #50945
Growing StorageV3 recovery could load all rows from the manifest even
when `SegmentLoadInfo.num_of_rows` represented an earlier checkpoint.
That allowed the growing text-match index to advance beyond the
checkpoint row count; the next WAL replay insert then reserved the
checkpoint offset and Tantivy rejected the lower doc id.
This PR:
- Caps recovered column-group rows to the segment checkpoint row count
before raw fields, PKs, and text indexes are populated.
- Adds `Growing.LoadStorageV3ManifestCapsRowsAtCheckpoint`, which writes
a 6-row StorageV3 manifest, loads it with a 4-row checkpoint, then
inserts the next row at offset 4.
## Verification
- `clang-format-12` on changed C++ files
- `git diff --check`
- Fresh C++ configure in `/tmp/milvus-50910-build` was blocked before
compilation because generated `internal/core/src/pb` sources were
missing: `No SOURCES given to target: milvus_pb`
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
## Summary
issue: #50980
Fix ARRAY `IS NULL` hang when a scalar BITMAP index is present by
avoiding recursive `std::call_once` re-entry in
`PhyNullExpr::DetermineExecPath()`.
This PR also adds a sealed ARRAY + BITMAP null-expression regression
test.
## Verification
- `clang-format-12 -i internal/core/src/exec/expression/NullExpr.cpp
internal/core/src/exec/expression/ExprArrayTest.cpp`
- `git diff --check`
- built `all_tests`
- focused C++ tests passed:
`Expr.TestArrayNullExprWithBitmapIndex:Expr.TestArrayNullExpr:Expr.TestStructArrayParentNullExprUsesRepresentativeSubField:Expr.TestVectorArrayNullExpr`
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
issue: #50698Fixes#50698
## What this PR does
This PR fixes NULL/UNKNOWN validity handling in boolean filter execution
paths:
- Makes conjunctive filter short-circuit decisions preserve rows that
are still UNKNOWN and may be resolved by later predicates.
- Treats UNKNOWN predicate results as filtered out when converting
predicate results to filter bitsets.
- Preserves nullable parent JSON validity for `!=` instead of producing
`data=true, valid=false` that can leak through raw-bit consumers.
- Fixes chunked sealed variable-length offset filtering so skipped NULL
rows keep invalid validity instead of becoming definite false.
## Verification
- `ninja unittest/all_tests` linked successfully.
- Focused C++ test run passed 109/109:
- `ConjunctExprTest.*`
- `*TestUnaryRangeWithJSONNullable*`
- `*TestUnaryRangeJsonNullable*`
- `git diff --check` passed.
- Staged diff check passed.
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
## What
Fix several scalar index V3 load-path issues found during the 2.6
backport review:
- pass collection id when loading V3 text indexes so encrypted entries
use the correct decryptor key
- propagate `CurrentScalarIndexVersion` through text index stats
conversion
- clean up encrypted V3 writer temp files and upload read fds on
exception paths
- close the Marisa trie temp fd on exception paths
- validate string sort V3 valid-bitset entry size before reading
- capture encrypted entry slices by value in async reads
## Validation
- `git diff --check`
- `make fmt`
Local `go test ./internal/util/segcore -count=1` did not reach tests
because the local cgo environment could not resolve `C.executor_set_*`
symbols. Local C++ configure for `cmake --build cmake_build --target
milvus_core -j96` was blocked by missing local Azure CMake package
(`AzureConfig.cmake`).
issue: https://github.com/milvus-io/milvus/issues/47417
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
this PR allows to create Sort/Bitmap/Hybrid index for json path index,
and also removed the `JsonInvertedIndex` class, to use
`InvertedIndexTantivy` directly for Inverted index type.
C++ changes:
- Add ConvertJsonToTypedFieldData<T>() to extract typed values from JSON
field data with separate tracking of non_exist_offsets for EXISTS
semantics
- Add JsonScalarIndexWrapper<T, BaseIndex> template for Sort/Bitmap with
dual file-manager pattern (original JSON schema for reading, cast-type
schema for base index dispatching)
- Add JsonHybridScalarIndex<T> with validity-aware cardinality counting
- Add IndexBase::Exists() virtual method; override in Sort/Bitmap/Hybrid
wrappers using non_exist_offsets (serialized via
WriteEntries/LoadEntries)
- Simplify ExistsExpr to use index->Exists() uniformly
- Extend IndexFactory::CreateJsonIndex() to route STL_SORT/BITMAP/HYBRID
Go changes:
- Update STL_SORT/Bitmap/Hybrid checkers to accept JSON with cast_type
and json_path validation
- Change AUTOINDEX default for JSON from INVERTED to HYBRID
- Bump ScalarIndexEngineVersion to 4
- Add version gate in DataCoord CreateIndex and snapshot RestoreIndexes
- Update test fixtures to cover new (index_type, cast_type) combinations
<img width="2380" height="708" alt="image"
src="https://github.com/user-attachments/assets/8b3923a0-2cb3-4af7-b73a-b76d1d1ec2d0"
/>
issue: https://github.com/milvus-io/milvus/issues/48954
design-doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260410-json_path_index_multi_type.md
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
Batch JSON document ingestion across the Tantivy FFI boundary and add
JSON terms query APIs for flat index IN predicates. Use TermSetQuery for
larger JSON IN lists while preserving per-term execution for small lists
and bool terms.
issue: #48609
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
* Optimize metadata and request handling hot paths
* Make replication pending-message queue capacity and max size
configurable instead of hard-coded.
* Batch QueryCoord collection metadata saves with MultiSave to avoid
oversized single writes while still reducing per-key etcd operations.
* Reduce avoidable allocations in DataCoord catalog tests, access log
list formatting, HTTP array joining, and HTTP JSON field extraction.
* Use WAL-specific message ID decoding during flusher recovery.
* Use set-based RootCoord collection-name filtering and structured
collection-not-found errors.
issue: #44452
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
## Summary
This PR adds validation to reject array element access expressions used
with `IS NULL` and `IS NOT NULL` operators at parse time, preventing
internal errors that would occur during execution.
## Key Changes
- Added validation in `VisitIsNull()` and `VisitIsNotNull()` methods in
`parser_visitor.go` to detect and reject array element access patterns
(e.g., `ArrayField[0] IS NULL`)
- The validation checks if a column has a nested path and is of array
type, returning a clear error message instead of allowing invalid
expressions to proceed
- Extended test coverage in `plan_parser_v2_test.go` to verify:
- Valid expressions: direct array field null checks (e.g., `ArrayField
IS NULL`)
- Invalid expressions: array element access with null checks (e.g.,
`ArrayField[0] IS NULL`)
- Both `IS NULL` and `IS NOT NULL` operators with various array field
types
## Implementation Details
- The fix leverages the existing `typeutil.IsArrayType()` utility to
identify array-typed columns
- Error messages are consistent and reference the issue (#48904) for
context
- The validation mirrors the existing pattern used for JSON field
handling, ensuring consistency with the codebase
issue: #48904
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
## Summary
Fixed a bug where templated expressions wrapped by unary NOT operators
were not properly propagating the `IsTemplate` flag to the outer
expression, causing `FillExpressionValue` to skip template placeholder
substitution.
## Changes
- **parser_visitor.go**: Added `IsTemplate:
childExpr.expr.GetIsTemplate()` to the UnaryExpr construction in
`VisitUnary()` to ensure the template flag is propagated from child
expressions to the parent unary expression.
- **fill_expression_value_test.go**: Added comprehensive regression test
`TestUnaryNotWithTemplate()` covering four scenarios:
- NOT wrapping templated term expressions with array values
- NOT wrapping templated unary range expressions
- NOT wrapping compound expressions with templates
- NOT wrapping templated JSON contains expressions
## Implementation Details
The fix ensures that when a unary NOT operator wraps a templated
expression, the `IsTemplate` flag is correctly set on the outer
UnaryExpr. This allows the expression filler to properly process and
substitute template placeholders in all nested expressions, preventing
VAL_NOT_SET errors and silent incorrect results.
issue: #49141
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
rename LoadV3/UploadV3 to LoadUnified/UploadUnified
Engine version (CurrentScalarIndexEngineVersion) tracks a node's feature
capability and is what gets aggregated across the cluster for rolling-
upgrade safety. File format version (MILVUS_V3_FORMAT_VERSION in
IndexEntryWriter.h, matched by the on-disk ".v3" filename suffix) tracks
the on-disk layout. These are deliberately independent concepts: an
engine version bump does not necessarily imply a format change.
- pkg/common/common.go: add a comment block above the engine version
constants explaining the distinction and the specific meaning of engine
version 3.
- Rename LoadV3 / UploadV3 / CreateIndexEntryWriterV3 to LoadUnified /
UploadUnified / CreateIndexEntryWriterUnified. These methods are the
format-agnostic packed-file entry points; keeping "V3" in the name would
wrongly suggest they are tied to the V3 on-disk format only. Future file
format versions can dispatch from the same entry point by reading the
file header.
issue: https://github.com/milvus-io/milvus/issues/47417
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
## Summary
Refactored the `IsInteger()` utility function to use explicit character
validation instead of relying on exception handling for parsing
validation.
## Key Changes
- Replaced try-catch block with `std::stoi()` with manual
character-by-character validation
- Added explicit handling for optional leading '+' or '-' signs
- Validates that all remaining characters are digits ('0'-'9')
- Improved performance by avoiding exception overhead for invalid inputs
## Implementation Details
- The function now iterates through the string starting from index 0 (or
1 if a sign is present)
- Validates that if a sign character exists, it's not the only character
in the string
- Returns false immediately upon encountering any non-digit character
- Maintains the same validation semantics: returns true only if the
entire string represents a valid integer
issue: #44452
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
Eliminate redundant string copies and memory allocations across the
search/query execution pipeline from Go through CGO to C++ segcore:
- Add std::string_view to data_access_type variant, avoiding per-element
std::string construction in expression evaluation hot paths
- Use std::move in MergeDataArray for VARCHAR/JSON/ARRAY/Sparse types
- Fix StrPKVisitor to take const ref instead of by-value parameter
- Replace intermediate vector<string> with direct protobuf Reserve
- Use std::move in GroupReduceHelper::RefreshSingleSearchResult
- Change ExtractSubJson to accept string_view parameter
- Support move semantics in SetLoadInfo across segment interface
- Convert AssertInfo string concatenation to fmt::format parameters
- Hoist indices vector outside loop in SortEqualScoresOneNQ
- Move strings in ParsePksFromFieldData instead of copying
- Fix auto -> auto& in SegmentChunkReader lambda
- Remove unused GetIterators() dead code
issue: https://github.com/milvus-io/milvus/issues/44452
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Remove kScalarIndexUseV3 from Meta.h/Meta.cpp and init_gtest.cpp
- Remove if(kScalarIndexUseV3) branches from 8 index source files,
keeping only the V2 Upload/Load code paths for production use
- Migrate all C++ UT scalar index calls from Upload/Load to
UploadV3/LoadV3 (44 call sites across 7 test files)
- Remove V2-only compat test (ScalarIndexV2Compat.ForceV2Upload)
- Clean up RTreeIndexTest V2 dead code paths
- Remove kScalarIndexUseV3 ternary from FileManager.h
GetRemoteIndexObjectPrefix/GetRemoteTextLogPrefix
issue: https://github.com/milvus-io/milvus/issues/47417
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
Set CurrentScalarIndexEngineVersion and MaximumScalarIndexEngineVersion
to 3, enabling V3 scalar index format by default. Update corresponding
test expectations in SegmentLoadInfoTest.
This PR also fixes the issue of snapshot not persisting scalar index
version, as verified by go-sdk e2e tests.
This PR also removes the `V2` series method of
`GetRemoteIndexObjectPrefixV2`, which are the same as their V1
counterparts, except for not prepending root path.
issue: #47417
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
StringIndexMarisa (Trie index) was missing SupportPatternMatch() and
PatternMatch() overrides, causing the expression evaluation framework to
bypass the trie's efficient predictive_search-based PrefixMatch and fall
back to O(n) brute-force scan via Reverse_Lookup for every row.
This regression was introduced when the expression framework was
refactored to use UnaryIndexFunc/UnaryIndexFuncForMatch instead of the
old StringIndex::Query(dataset) path. Other index types
(StringIndexSort, InvertedIndexTantivy, BitmapIndex) were updated to
implement these interfaces, but StringIndexMarisa was missed.
issue: https://github.com/milvus-io/milvus/issues/48685
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
issue: https://github.com/milvus-io/milvus/issues/45525
## Summary
This PR adds optimizations for boolean IN and NOT IN expressions, with
special handling for nullable boolean fields. The changes improve query
performance by rewriting certain bool IN expressions to simpler, faster
operations.
## Key Changes
- **Bool IN expression optimizations:**
- `BoolField IN [true, false]` → `AlwaysTrueExpr` (non-nullable) or `IS
NOT NULL` (nullable)
- `BoolField IN [true]` → `BoolField == true` (uses fast SIMD path)
- `BoolField IN [false]` → `BoolField == false`
- Single-value deduplication: `BoolField IN [true, true]` → `BoolField
== true`
- **Bool NOT IN expression optimizations:**
- `BoolField NOT IN [true, false]` → `AlwaysFalseExpr` (non-nullable) or
`IS NULL` (nullable)
- `BoolField NOT IN [true]` → `BoolField != true`
- `BoolField NOT IN [false]` → `BoolField != false`
- **Unary expression optimizations:**
- `NOT AlwaysTrueExpr` → `AlwaysFalseExpr`
- `NOT (IS NOT NULL)` → `IS NULL` (and vice versa)
- `NOT (col == val)` → `col != val`
- **Helper functions and test infrastructure:**
- Added `newNullExpr()` utility to create NULL check expressions
- Added `allBoolVals()` helper to validate boolean value lists
- Added `buildSchemaHelperForRewriteNullableT()` for testing nullable
fields
- Comprehensive test coverage for all bool IN/NOT IN optimization paths
## Implementation Details
The optimization logic is implemented in the `visitTermExpr()` method,
which:
1. Detects when a TermExpr operates on a boolean column
2. Analyzes the values to determine if both true and false are present
3. Rewrites to appropriate expressions based on nullability and value
set
4. Handles single-value cases by converting to equality comparisons
The unary expression visitor was enhanced to handle NOT operations on
the newly created expressions, enabling proper composition of
optimizations (e.g., NOT IN becomes NOT of an optimized expression).
https://claude.ai/code/session_01GkYztNzb75zgjuevzqVcwb
---------
Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Replace the fixed threshold (150) for merging == or into in[] with
type-specific thresholds based on benchmark data:
- INT types: use in[] when N >= 10 (== or is faster below due to simpler
execution path)
- FLOAT types: use in[] when N >= 15 (float hash is more expensive)
- Other types (varchar, bool): use in[] when N >= 3
The rewriting is now bidirectional:
- == or → in[]: when shouldUseInExpr returns true (existing direction)
- in[] → == or: when shouldUseInExpr returns false (new direction, via
visitTermExpr)
- not in → != and: when shouldUseInExpr returns false (new, via
visitUnaryExpr)
Both directions use the same shouldUseInExpr function to ensure
consistency.
issue: https://github.com/milvus-io/milvus/issues/45525
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
issue: #47417
Consolidate TextMatchIndex build/upload from standalone BuildTextIndex
CGO function into the unified CreateIndex → ScalarIndexCreator path.
Remove the deprecated BuildTextIndex CGO entry point and add V3 Load
support for text match indexes.
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
issue: https://github.com/milvus-io/milvus/issues/47417
- Fix TOCTOU: snapshot current/minimal/maximum under single lock in
Resolve*IndexVersion()
- Use log.RatedWarn in clampVersion() to avoid log flooding on hot path
- Remove dead code getIndexEngineVersion() from segments/utils.go
- Add explicit empty-map checks in
getMaximumVersion/getMaximumScalarVersion
- Remove verbose per-call log.Info from internal version methods
- Add test case: target < current with force=false
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Introduce full scalar index version management parity with vector
indexes:
- Add MaximumScalarIndexEngineVersion constant and MaximumIndexVersion
to session
- Add targetScalarIndexVersion and forceRebuildScalarSegmentIndex config
params
- Add GetMaximum*Version() and Resolve*IndexVersion() to
IndexEngineVersionManager
- Propagate MaximumIndexVersion from knowhere C++ to Go layer via
session
- Add scalar forceRebuild path in compaction trigger
- Fix DataNode scalar version clamp to use Maximum instead of Current
- Fix sort_compaction and stats task to use request-carried version
- Use resolved (clamped) target version in compaction trigger to prevent
infinite rebuild loops
issue: https://github.com/milvus-io/milvus/issues/47417
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260313-scalar_index_version_management.md
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Unified log.level configuration to manage all logging subsystems in
Milvus (Go zap, C++ glog, Rust tantivy). Added Rust FFI function to
control Tantivy log level via log::set_max_level() and call it from C++
SetLogLevel() to sync levels across all subsystems. Removed MY_LOG_LEVEL
environment variable dependency from Tantivy initialization.
issue: #44452
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
issue: #44452
Previously, unit tests could not run in parallel because:
1. Tests used hardcoded /tmp paths (e.g., /tmp/test-bitmap-index/,
/tmp/milvus/rtree-index/), causing file conflicts between shards.
2. Multiple tests independently initialized and released global
singletons (ArrowFileSystemSingleton, LocalChunkManagerSingleton),
leading to use-after-release crashes when shards ran concurrently.
This PR fixes that by:
- Add centralized init_gtest.cpp that generates shard-aware random test
paths (TestLocalPath/TestRemotePath/TestMmapPath) and initializes all
singletons once.
- Replace hardcoded /tmp paths across ~40 test files with TestLocalPath.
- Remove redundant per-test singleton init/release that conflict across
shards.
This PR also fixes 2 other issues:
- Fix `ConvertToArrowSchema` to use `field_ids_` (insertion-ordered
vector) instead of `fields_` (`unordered_map`) for deterministic column
ordering.
- Add thread barrier in PreparedGeometryTest to prevent TLS address
reuse.
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
issue: #47083
Changes:
- Add configurable low/high cardinality index types for hybrid index
- Default high cardinality: STL_SORT
- Add float/double support for hybrid index
Compatibility:
- Version <= 2: Uses legacy behavior (STLSORT for int, INVERTED for
string/float)
- Version >= 3: Uses configurable index types from config
This PR does not bump scalar version, so indexes will continue use
version 2, until we bump the version.
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
issue: #44452
This PR includes mainly `#include` updates, to ensure each file only and
always `#include`s the headers it actually references, avoiding `if A
uses a symbol in C, and B includes C, then A includes B but not C`.
This PR does not yet fix all IWYU issues, also does not enforce this in
CI. Just an one time batch fix.
This PR also removes `unittest/bench` as it is no longer used.
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
issue: #44452
This is a following up pr for #47384.
This pr also splits the huge `ExprTest.cpp` into several test files for
better readability.
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
issue: #47083
The scalar index version was correctly passed and stored in ETCD during
index build, but the load path was broken. After this PR:
- QueryCoord now passes CurrentScalarIndexVersion from etcd metadata to
QueryNode
- QueryNode forwards the version to C++ via LoadIndexInfo proto
- C++ injects the version into index_params for scalar index
implementations
This fixes the version chain: etcd -> QueryCoord -> QueryNode -> C++ ->
Index::Load()
This PR also:
- Fixes `ShouldRebuildSegmentIndex()` to check index type and use
scalar/vec index version
- DescribeIndex now returns scalar index version for scalar index
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
issue: #44452
added the ability to create SearchPlan to plan parser so; updated all
c++ ut to use plan parser so to create retrieve/search plan, instead of
writting literal text protos
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
issue: #46820
- Add AllTrue() and AllFalse() as instance methods of ColumnVector
- These methods use early termination for efficiency
- Methods assert that the ColumnVector is a bitmap type
- Update ConjunctExpr::CanSkipFollowingExprs to use the new methods
- Add VectorTest.cpp with comprehensive tests for the new methods
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
issue: https://github.com/milvus-io/milvus/issues/46820
This commit addresses multiple issues identified in PR #46939:
1. **Correctness Issues (High Priority)**:
- Fix `CanSkipFollowingExprs` short-circuit logic: Previously used
`TrueCount(vec) == 0` for AND, which incorrectly skips when NULL values
are present. Now correctly uses `FalseCount(vec) == vec->size()` to only
skip when ALL rows are definitely FALSE.
- Fix `SetNextExprBitmapInput` to consider NULL values: For AND, both
TRUE and NULL rows need evaluation; for OR, both FALSE and NULL rows
need evaluation. Previously only considered data bitmap.
2. **Performance Improvements (Medium Priority)**:
- Optimize `TrueCount` with zero-copy SIMD popcount using
`__builtin_popcountll` instead of creating temporary bitmap copy.
- Add `FalseCount` function with same SIMD optimization.
- Change function parameters from `ColumnVectorPtr` (by value) to `const
ColumnVectorPtr&` (by reference) to avoid atomic ref count ops.
3. **Code Cleanup (Low Priority)**:
- Remove unused `ConjunctElementFunc` template class (dead code from old
two-valued logic implementation).
4. **Tests**:
- Add comprehensive unit tests for `TrueCount` and `FalseCount`
including boundary conditions for non-64-aligned sizes.
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
Emit config events on paramtable Save/SaveGroup/Reset/Remove so
tracing/log watchers can react without restart.
Fix EventDispatcher to dispatch keyPrefix handlers Dispatch prefix
handlers in addition to exact key handlers to restore tracing hot-reload
and global cache eviction after ParamItem handler registration.
issue: https://github.com/milvus-io/milvus/issues/47142
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
test: add unit tests for mixed int64/float types in BinaryRangeExpr
When processing binary range expressions (e.g., `x > 499 && x <= 512.0`)
on JSON/dynamic fields with expression templates, the lower and upper
bounds could have different numeric types (int64 vs float64). This
caused an assertion failure in GetValueFromProto when the template type
didn't match the actual proto value type.
Fixes:
1. Go side (fill_expression_value.go): Normalize numeric types for JSON
fields - if either bound is float and the other is int, convert the int
to float.
2. C++ side (BinaryRangeExpr.cpp):
- Check both lower_val and upper_val types when dispatching
- Use double template when either bound is float
- Use GetValueWithCastNumber instead of GetValueFromProto to safely
handle int64->double conversion
issue: #46588
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
- Core invariant: JSON field binary-range expressions must present
numeric bounds to the evaluator with a consistent numeric type; if
either bound is floating-point, both bounds must be treated as double to
avoid proto-type mismatches during template instantiation.
- Bug fix (issue #46588 & concrete change): mixed int64/float bounds
could dispatch the wrong template (e.g.,
ExecRangeVisitorImplForJson<int64_t>) and trigger assertions in
GetValueFromProto. Fixes: (1) Go parser (FillBinaryRangeExpressionValue
in fill_expression_value.go) normalizes mixed JSON numeric bounds by
promoting the int bound to float; (2) C++ evaluator
(PhyBinaryRangeFilterExpr::Eval in BinaryRangeExpr.cpp) inspects both
lower_type and upper_type, sets use_double when either is float, selects
ExecRangeVisitorImplForJson<double> for mixed numeric cases, and
replaces GetValueFromProto with GetValueWithCastNumber so int64→double
conversions are handled safely.
- Removed / simplified logic: the previous evaluator branched on only
the lower bound's proto type and had separate index/non-index handling
for int64 vs float; that per-bound branching is replaced by unified
numeric handling (convert to double when needed) and a single numeric
path for index use — eliminating redundant, error-prone branches that
assumed homogeneous bound types.
- No data loss or regression: changes only promote int→double for
JSON-range comparisons when the other bound is float; integer-only and
float-only paths remain unchanged. Promotion uses IEEE double (C++
double and Go float64) and only affects template dispatch and
value-extraction paths; GetValueWithCastNumber safely converts int64 to
double and index/non-index code paths both normalize consistently,
preserving semantics for comparisons and avoiding assertion failures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
issue: https://github.com/milvus-io/milvus/issues/44452
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
**Scalar Bench Decoupled from Milvus Build System**
- **Core assumption**: Scalar-bench is now managed as an independent
build artifact outside the milvus repository, eliminating the need for
conditional compilation integration within milvus's Makefile and
CMakeLists.txt.
- **Build infrastructure simplified**: Removed `scalar-bench` and
`scalar-bench-ui` targets from Makefile and deleted the entire
`ENABLE_SCALAR_BENCH` conditional block in
`internal/core/unittest/CMakeLists.txt` (which handled FetchContent,
cache variables, and subdirectory integration)—this eliminates optional,
redundant build-time coupling that is no longer necessary.
- **No regression introduced**: The removal only affects optional
build-time integration paths. Core C++ builds continue functioning as
before, and unit tests remain unaffected since `ENABLE_SCALAR_BENCH` was
always optional (not a required dependency); the newly added
`plan-parser-so` dependency on core build targets appears to be a
separate, required component.
- **Decoupling benefit**: Scalar-benchmark can now evolve and release on
its own schedule independent of milvus release cycles, while maintaining
clean separation of concerns between the two projects.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
issue: #44452
## Summary
Reduce test combinations in `TestSparsePlaceholderGroupSize` to decrease
test execution time:
- `nqs`: from `[1, 10, 100, 1000, 10000]` to `[1, 100, 10000]`
- `averageNNZs`: from `[1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024,
2048]` to `[1, 4, 16, 64, 256, 1024]`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## TestSparsePlaceholderGroupSize Test Reduction
**Core Invariant:** The sparse vector NNZ estimation algorithm
(`EstimateSparseVectorNNZFromPlaceholderGroup`) must maintain accuracy
within bounded error thresholds—individual cases < 10% error and no more
than 2% of cases exceeding 5% error—across representative parameter
ranges.
**Test Coverage Optimized, Not Removed:** Test combinations reduced from
60 to 18 by pruning redundant parameter points while retaining critical
coverage: nqs now tests [1, 100, 10000] (min, mid, max) and averageNNZs
tests [1, 4, 16, 64, 256, 1024] (exponential spacing). Variant
generation logic (powers of 2 scaling) remains unchanged, ensuring error
scenarios are still exercised.
**No Behavioral Regression:** The algorithm under test is untouched;
only test case frequency decreases. The same assertions validate error
bounds are satisfied—individual assertions (`assert.Less(errorRatio,
10.0)`) and statistical assertions (`assert.Less(largeErrorRatio, 2.0)`)
remain identical, confirming that estimation quality is still verified.
**Why Safe:** Exponential spacing of removed parameters (e.g., nqs: 10,
1000 removed; averageNNZs: 2, 8, 32, 128, 512, 2048 removed) addresses
diminishing returns—intermediate values provide no new error scenarios
beyond what surrounding powers-of-2 values expose, while keeping test
execution time proportional to coverage value gained.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
issue: https://github.com/milvus-io/milvus/issues/44399
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **New Features**
* Enhanced pattern matching for string indexes with support for prefix,
postfix, inner, and regex-based matching operations.
* Optimized pattern matching performance through prefix-based filtering
and range-based lookups.
* **Tests**
* Added comprehensive test coverage for pattern matching functionality
across multiple index implementations.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
issue: https://github.com/milvus-io/milvus/issues/45525
see added README.md for added optimizations
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added query expression optimization feature with a new `optimizeExpr`
configuration flag to enable automatic simplification of filter
predicates, including range predicate optimization, merging of IN/NOT IN
conditions, and flattening of nested logical operators.
* **Bug Fixes**
* Adjusted delete operation behavior to correctly handle expression
evaluation.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
issue: https://github.com/milvus-io/milvus/issues/44399
this PR also adds `ByteSize()` methods for scalar indexes. currently not
used in milvus code, but used in scalar benchmark. may be used by
cachinglayer in the future.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Refactor**
* Improved and standardized memory-size computation and caching across
index types so reported index footprints are more accurate and
consistent.
* **Chores**
* Ensured byte-size metrics are refreshed immediately after index
build/load operations to keep memory accounting in sync with runtime
state.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
generated a library that wraps the go expr parser, and embedded that
into libmilvus-core.so
issue: https://github.com/milvus-io/milvus/issues/45702
see `internal/core/src/plan/milvus_plan_parser.h` for the exposed
interface
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Introduced C++ API for plan parsing with schema registration and
expression parsing capabilities.
* Plan parser now available as shared libraries instead of a standalone
binary tool.
* **Refactor**
* Reorganized build system to produce shared library artifacts instead
of executable binaries.
* Build outputs relocated to standardized library and include
directories.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
issue: #45511
our tantivy inverted index currently does not include item index if the
value is an array, thus we can't do `a[0] == 'b'` type of look up in the
inverted index. for such, we need to skip the index and use brute force
search.
we may improve our index in the future, so this is a temp solution
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
This commit optimizes std::vector usage across segcore by adding
reserve() calls where the size is known in advance, reducing memory
reallocations during push_back operations.
Changes:
- TimestampIndex.cpp: Reserve space for prefix_sums and
timestamp_barriers
- SegmentGrowingImpl.cpp: Reserve space for binlog info vectors
- ChunkedSegmentSealedImpl.cpp: Reserve space for futures and field data
vectors
- storagev2translator/GroupChunkTranslator.cpp: Reserve space for
metadata vectors
This improves performance by avoiding multiple memory reallocations when
the vector size is predictable.
issue: https://github.com/milvus-io/milvus/issues/45679
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
1. Array.h: Add output_data(ScalarFieldProto&) overload for both Array
and ArrayView classes
2. Use std::string_view instead of std::string for VARCHAR and GEOMETRY
types to avoid extra string copies
3. Call Reserve(length_) before writing to proto objects to reduce
memory reallocations
a simple test shows those optimizations improve the Array of Varchar
bulk_subscript performance by 20%
issue: https://github.com/milvus-io/milvus/issues/45679
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
Remove redundant `VectorPlanNode` subclasses and simplify the visitor
pattern by consolidating to a single `VectorPlanNode`.
The previous design used distinct `VectorPlanNode` subclasses and a
templated `VectorVisitorImpl` for type-directed dispatch. However, the
template parameter was not functionally used to implement different
logic for each vector type, making the subclasses redundant for their
intended purpose.
This PR is created by Cursor Agent and manually moved from
https://github.com/zhengbuqian/milvus/pull/14.
Signed-off-by: zhengbuqian <zhengbuqian@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: buqian.zheng <buqian.zheng@zilliz.com>
