issue: https://github.com/milvus-io/milvus/issues/42053
This PR fixes ngram index eligibility checks for UTF-8 literals by using
character count instead of byte length in the C++ gate and phase1
validation. It also adds regression coverage for short multibyte
literals to ensure they correctly fall back instead of being sent to
Tantivy ngram queries.
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
issue: https://github.com/milvus-io/milvus/issues/42148
1. Support range search in hybrid search
Enable element-level vector fields to participate in hybrid range search
while preserving element identity and offsets.
2. Support group-by in hybrid search, with constraints
Enable element-level hybrid search with PK group-by and preserve element
offsets. For group-by, sub-searches that need element identity merging
must stay within the same struct array semantic scope.
Also, it fixes 2 related bugs:
1. Fix group-by related issues
Fix lost element indices and incorrect merging behavior in query/search
group-by paths, so multiple matching elements under the same PK can
still be returned with correct offsets.
2. Fix aggregation output issues
Fix aggregation/count output corruption when a collection contains
struct array fields. Proxy struct-field reconstruction now preserves
virtual aggregation fields such as `count(*)` instead of treating
`field_id=0` as a struct sub-field. As a result, `id + count(*)`,
element-level `count(*)`, and group-by `count(*)` return correctly named
fields.
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
issue: https://github.com/milvus-io/milvus/issues/50556https://github.com/milvus-io/milvus/pull/50436 uses BulkIsValid to
replace get_batch_views in hoping to avoid cost of constructing views
such as StringViews. However, BulkIsValid is random access where the
cost of other staff exceed the benefits which leads to performance
regression.
Constructing any kind of views is still unnecessary, and this PR uses
valid data in Chunk directly. Here benchmarks:
1 million rows of string data, `is null` expression
StringView: ~4300us
BulkIsValid: ~8340us
This PR: 1440us
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
issue: https://github.com/milvus-io/milvus/issues/50461
ref: https://github.com/milvus-io/milvus/issues/42148
Issue: nullable VectorArray chunks store non-null rows in a compact
physical layout, so logical row indexes can diverge from physical row
indexes when missing/null rows exist. Some access paths use
logical-to-physical translation when reading the vector-array data, but
the offsets returned by VectorArrayOffsets() were still indexed by the
compact physical layout. Sealed segment vector search used logical row
indexes to read those offsets directly, without applying the same
logical-to-physical translation. This could make it read the wrong row
span or read past the valid offsets range, resulting in empty search
results or std::bad_alloc.
Fix: build VectorArrayChunk offsets by logical row count instead of
physical row count. The offsets array now has row_nums + 1 entries and
can be indexed directly by logical row id. Null rows and empty rows both
have a zero-length span, while the existing Chunk valid bitmap preserves
the distinction between null and non-null empty rows.
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
issue: https://github.com/milvus-io/milvus/issues/42148
This PR remove `element_row_ids_` in ArrayOffsets to reduce memory usage
whose memory is O(total element count). Instead, use binary search on
`row_to_element_start_` to calculate the element id.
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
issue: https://github.com/milvus-io/milvus/issues/48602
ref: https://github.com/milvus-io/milvus/issues/42148
This PR fixes issue: When performing element_filter search across
multiple segments, if one segment returns 0 hits, C++ reduce creates an
empty `LongArray` for `ElementIndices` which proto3 serializes as absent
(nil). The Go-side reduce then sees inconsistent `ElementIndices` (nil
vs non-nil) across segment results and returns an "inconsistent
element-level flag" error.
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
issue: https://github.com/milvus-io/milvus/issues/42148
Correctness:
- JSON/CSV import: reject struct array elements with mismatched field
count
- Parquet import: error on type assertion failure in scalar and vector
paths instead of silent zero-fill/data loss
- JSON import: validate per-vector dimension in ArrayOfVector
FloatVector path
- Move element_level inference from pymilvus into C++
ParsePlaceholderGroup: pymilvus cannot reliably infer if it's
elelment_level as which kinds of search on ArrayOfVector are supported
are determined by metric type but pymilvus does not have this info.
- Fix element-level search returning wrong row IDs on growing segments
with multiple chunks by using cumulative element offset instead of row
offset as begin_id in brute-force search.
https://github.com/milvus-io/milvus/issues/48617
Performance:
- Replace proto.Clone with in-place FieldName mutation in
reconstructStructFieldData
Nested index:
- field name is missing in CreateIndexInfo which is needed to determine
whether the index should be nested or not
Tests: added for all above fixes
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
issue: https://github.com/milvus-io/milvus/issues/46813
After this PR, if there are some LIKE exprs connected by `and` and ngram
index is enabled, all phase 1 of them will be executed before phase 2.
This PR significantly improve the performance of ngram index in the
situation of multiple LIKE.
Before this PR, LIKE in ngram index is under performed in the case of
multiple LIKE connected with `and` when the intersection of these
results are small. This is because brute force can take advantage of
rows that are filtered while ngram always do all the things which is
solved by this PR.
The following test shows the effect where the dataset is wiki with 20k
rows, and there are 4, 6, 8 LIKE with `and`:
Before this PR
<img width="805" height="678" alt="image"
src="https://github.com/user-attachments/assets/202ea67d-2ed0-400a-b128-529695e438ab"
/>
After this PR
<img width="720" height="707" alt="image"
src="https://github.com/user-attachments/assets/1865e2ad-f1d6-498c-aaf3-652445e7fc6e"
/>
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
issue: https://github.com/milvus-io/milvus/issues/42148
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
- Core invariant: StructArray now accepts only fixed-dimension vector
element types (typeutil.IsFixDimVectorType) for array/struct vector
fields — i.e., dense/quantized fixed-dim vectors (FloatVector,
Float16Vector, BFloat16Vector, Int8Vector, BinaryVector) are supported
while sparse vectors remain rejected at validation
(internal/proxy/util.go → ValidateFieldsInStruct).
- Logic removed / simplified: the previous FloatVector-only special-case
was removed and replaced by a single, generic fixed-dimension check;
numerous duplicated test branches were collapsed into parameterized
fixtures (internal/core/src/common/ChunkWriterTest.cpp: GetByteWidth +
TEST_P/INSTANTIATE_TEST_SUITE_P) and ElementFilter tests were unified
under CreatePlaceholderGroupForType()/ElementFilterSealedParam,
eliminating repetitive type-specific code.
- No data loss / regression rationale (concrete code paths): existing
VECTOR_FLOAT code paths and behavior are preserved — tests explicitly
instantiate VECTOR_FLOAT cases in ChunkWriterTest and ElementFilter;
deserialization now explicitly handles empty ArrayOfVector cases and
returns the correct VectorField by elementType
(internal/storage/serde.go: deserializeArrayOfVector), and StructArray
readers were extended with byte-based branches for
float16/bfloat16/int8/binary without altering FloatVector logic
(internal/util/importutilv2/parquet/struct_field_reader.go), making all
additions additive and failing unsupported sparse vectors early at proxy
validation rather than silently.
- New capability added: full StructArray support for additional
fixed-dimension vector element types (Float16Vector, BFloat16Vector,
Int8Vector, BinaryVector) across import, serde, parquet struct readers,
test generators, and end-to-end tests (files touched include
internal/util/importutilv2/parquet/struct_field_reader.go,
internal/storage/serde.go, pkg/util/testutils/gen_data.go, various
tests), enabling non-float embedding-list and byte-based vector
workflows.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
issue: https://github.com/milvus-io/milvus/issues/42148
This PR enable struct sub scalar fields to create nested index. The
nested means treating elements in array (we know that all fields in
array of struct are actually array) as separate documents.
The index is used by Struct to speed up Match expression as well as
Element-filter expression. This PR only enable Match expression to be
optmized by index and leave Element-filter expression in the following
PR.
Now, MatchExpr executes sub-expr by using offset inputs. To make it
support index, it also needs to support brute force without offset
inputs, so this PR:
1. enable struct scalar fields to create nested index
2. enable match expr to support brute force without offset input
3. enable match expr to support index
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
issue: https://github.com/milvus-io/milvus/issues/42053
Process ngram in batch rather than all by once.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Batch Processing for N-gram Queries
**Core Invariant:** All data iteration is now driven by `batch_size_` as
the fundamental unit; for sealed chunked segments processing string/JSON
data, processing is strictly stateless to allow specialized batched
algorithms.
**Simplified Logic:**
- Removed the `process_all_chunks` boolean flag from
`ProcessMultipleChunksCommon` (renamed to
`ProcessDataChunksForMultipleChunk`) as it was redundant—all iteration
paths now converge on the same `batch_size_`-driven chunking strategy
with unified data size clamping (`std::min(chunk_size, batch_size_ -
processed_size)`).
- Eliminated wrapper delegation methods
(`ProcessDataChunksForMultipleChunk` and
`ProcessAllChunksForMultipleChunk` old wrappers) that pointed to a
single common implementation with a conditional flag.
**No Data Loss or Behavior Regression:**
- The new `ProcessAllDataChunkBatched<T>` is an additional stateless
public path (requires sealed + chunked segments, type constraints:
`std::string_view|Json|ArrayView`) that iterates all `num_data_chunk_`
chunks in `batch_size_` granularity without mutating cursor state
(`current_data_chunk_`, `current_data_chunk_pos_`), ensuring
deterministic re-entrant processing.
- Existing cursor-based APIs (`ProcessDataChunksForMultipleChunk`,
`ProcessChunkForSealedSeg`) remain unchanged for standard expression
evaluation—no segment state is corrupted.
- N-gram query execution now routes through
`ExecuteQueryWithPredicate<T, Predicate>(literal, segment, predicate,
need_post_filter)` which forwards generic predicates and delegates to
`segment->ProcessAllDataChunkBatched<T>(execute_batch, res)` for
post-filtering, avoiding per-chunk single-pass traversal.
**Enhancement:** Generic predicate template `template <typename T,
typename Predicate>` with perfect forwarding (`Predicate&& predicate`)
replaces the fixed `std::function<bool(const T&)>` signature,
eliminating function wrapper overhead for n-gram matcher closures and
enabling efficient batch processing callbacks.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
issue: https://github.com/milvus-io/milvus/issues/46517
ref: https://github.com/milvus-io/milvus/issues/42148
This PR supports match operator family with struct array and brute force
search only.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
- Core invariant: match operators only target struct-array element-level
predicates and assume callers provide a correct row_start so element
indices form a contiguous range; IArrayOffsets implementations convert
row-level bitmaps/rows (starting at row_start) into element-level
bitmaps or a contiguous element-offset vector used by brute-force
evaluation.
- New capability added: end-to-end support for MATCH_* semantics
(match_any, match_all, match_least, match_most, match_exact) — parser
(grammar + proto), planner (ParseMatchExprs), expr model
(expr::MatchExpr), compilation (Expr→PhyMatchFilterExpr), execution
(PhyMatchFilterExpr::Eval uses element offsets/bitmaps), and unit tests
(MatchExprTest + parser tests). Implementation currently works for
struct-array inputs and uses brute-force element counting via
RowBitsetToElementOffsets/RowBitsetToElementBitset.
- Logic removed or simplified and why: removed the ad-hoc
DocBitsetToElementOffsets helper and consolidated offset/bitset
derivation into IArrayOffsets::RowBitsetToElementOffsets and a
row_start-aware RowBitsetToElementBitset, and removed EvalCtx overloads
that embedded ExprSet (now EvalCtx(exec_ctx, offset_input)). This
centralizes array-layout logic in ArrayOffsets and removes duplicated
offset conversion and EvalCtx variants that were redundant for
element-level evaluation.
- No data loss / no behavior regression: persistent formats are
unchanged (no proto storage or on-disk layout changed); callers were
updated to supply row_start and now route through the centralized
ArrayOffsets APIs which still use the authoritative
row_to_element_start_ mapping, preserving exact element index mappings.
Eval logic changes are limited to in-memory plumbing (how
offsets/bitmaps are produced and how EvalCtx is constructed); expression
evaluation still invokes exprs_->Eval where needed, so existing behavior
and stored data remain intact.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
issue: https://github.com/milvus-io/milvus/issues/45890
ComputePhraseMatchSlop accepts three pararms:
1. A string: query text
2. Some trings: data texts
3. Analyzer params,
Slop will be calculated for the query text with each data text in the
context of phrase match where they are tokenized with tokenizer with
analyzer params.
So two array will be returned:
1. is_match: is phrase match can sucess
2. slop: the related slop if phrase match can sucess, or -1 is cannot.
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
issue: https://github.com/milvus-io/milvus/issues/42148
For a vector field inside a STRUCT, since a STRUCT can only appear as
the element type of an ARRAY field, the vector field in STRUCT is
effectively an array of vectors, i.e. an embedding list.
Milvus already supports searching embedding lists with metrics whose
names start with the prefix MAX_SIM_.
This PR allows Milvus to search embeddings inside an embedding list
using the same metrics as normal embedding fields. Each embedding in the
list is treated as an independent vector and participates in ANN search.
Further, since STRUCT may contain scalar fields that are highly related
to the embedding field, this PR introduces an element-level filter
expression to refine search results.
The grammar of the element-level filter is:
element_filter(structFieldName, $[subFieldName] == 3)
where $[subFieldName] refers to the value of subFieldName in each
element of the STRUCT array structFieldName.
It can be combined with existing filter expressions, for example:
"varcharField == 'aaa' && element_filter(struct_field, $[struct_int] ==
3)"
A full example:
```
struct_schema = milvus_client.create_struct_field_schema()
struct_schema.add_field("struct_str", DataType.VARCHAR, max_length=65535)
struct_schema.add_field("struct_int", DataType.INT32)
struct_schema.add_field("struct_float_vec", DataType.FLOAT_VECTOR, dim=EMBEDDING_DIM)
schema.add_field(
"struct_field",
datatype=DataType.ARRAY,
element_type=DataType.STRUCT,
struct_schema=struct_schema,
max_capacity=1000,
)
...
filter = "varcharField == 'aaa' && element_filter(struct_field, $[struct_int] == 3 && $[struct_str] == 'abc')"
res = milvus_client.search(
COLLECTION_NAME,
data=query_embeddings,
limit=10,
anns_field="struct_field[struct_float_vec]",
filter=filter,
output_fields=["struct_field[struct_int]", "varcharField"],
)
```
TODO:
1. When an `element_filter` expression is used, a regular filter
expression must also be present. Remove this restriction.
2. Implement `element_filter` expressions in the `query`.
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
issue: https://github.com/milvus-io/milvus/issues/45006
ref: https://github.com/milvus-io/milvus/issues/42148
Previsouly, the parquet import is implemented based on that the STRUCT
in the parquet files is hanlded in the way that each field in struct is
stored in a single column.
However, in the user's perspective, the array of STRUCT contains data is
something like STRUCT_A:
for one row, [struct{field1_1, field2_1, field3_1}, struct{field1_2,
field2_2, field3_2}, ...], rather than {[field1_1, field1_2, ...],
[field2_1, field2_2, ...], [field3_1, field3_2, field3_3, ...]}.
This PR fixes this.
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
issue: https://github.com/milvus-io/milvus/issues/44399
This PR implements STL_SORT for VARCHAR data type for both RAM and MMAP
mode.
The general idea is that we deduplicate field values and maintains a
posting list for each unique value.
The serialization format of the index is:
```
[unique_count][string_offsets][string_data][post_list_offsets][post_list_data][magic_code]
string_offsets: array of offsets into string_data section
string_data: str_len1, str1, str_len2, str2, ...
post_list_offsets: array of offsets into post_list_data section
post_list_data: post_list_len1, row_id1, row_id2, ..., post_list_len2, row_id1, row_id2, ...
```
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
Ref https://github.com/milvus-io/milvus/issues/42148
This PR supports create index for vector array (now, only for
`DataType.FLOAT_VECTOR`) and search on it.
The index type supported in this PR is `EMB_LIST_HNSW` and the metric
type is `MAX_SIM` only.
The way to use it:
```python
milvus_client = MilvusClient("xxx:19530")
schema = milvus_client.create_schema(enable_dynamic_field=True, auto_id=True)
...
struct_schema = milvus_client.create_struct_array_field_schema("struct_array_field")
...
struct_schema.add_field("struct_float_vec", DataType.ARRAY_OF_VECTOR, element_type=DataType.FLOAT_VECTOR, dim=128, max_capacity=1000)
...
schema.add_struct_array_field(struct_schema)
index_params = milvus_client.prepare_index_params()
index_params.add_index(field_name="struct_float_vec", index_type="EMB_LIST_HNSW", metric_type="MAX_SIM", index_params={"nlist": 128})
...
milvus_client.create_index(COLLECTION_NAME, schema=schema, index_params=index_params)
```
Note: This PR uses `Lims` to convey offsets of the vector array to
knowhere where vectors of multiple vector arrays are concatenated and we
need offsets to specify which vectors belong to which vector array.
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
