issue: #51538
## What changed
- Treat nullable TIMESTAMPTZ rows as SQL UNKNOWN on the manual
year/month interval arithmetic path.
- Add focused regression coverage for growing and sealed segments, both
sequential scans and offset-input evaluation.
## Root cause
`PhyTimestamptzArithCompareExpr` evaluated the storage placeholder for
NULL rows without consulting `valid_data`. This could return NULL rows,
especially after logical negation, and disagreed with the fixed-duration
arithmetic path.
## Validation
- `clang++ -fsyntax-only` passed for the new test translation unit using
the repository-generated compile flags.
- `git diff --check` passed.
- The full C++ test binary could not be linked locally because the
existing Tantivy Rust dependency build cannot find macOS system headers
(`string.h`, `stdio.h`); this failure occurs before compiling this
change.
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
issue: #51379
Part of the stack tracked in #51385 (layer 1, no dependencies).
## What
`BitsetBase::read(idx, nbits)` validated `nbits <= sizeof(data_type)`
instead of `8 * sizeof(data_type)`, capping legal reads at 8 bits on a
64-bit word and never checking the end bound. The vectorized
element-wise policy also lacked the `op_read` forwarding, so `read()`
was unusable on production bitset types. The sibling `write()` had the
same broken check and zero callers repo-wide — deleted.
## Review guide
~60 lines. Check the new range checks and the op_read forwarding;
`ReadTest` covers widths 1–64 across offsets for all four policies
(bit-wise, element-wise, vectorized-ref, vectorized-dynamic).
Word-wise consumers land in the next stack layer.
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Related to #51068
- move row count, mmap markers, average field sizes, and skip index into
runtime state
- remove duplicated live resource containers from chunked sealed
segments
- preserve old and new resource lifetimes through COW publication
- reuse one published snapshot capture within each internal method
- add runtime lifetime and skip index regression coverage
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
The `if (!nullable_)` branch had no `return`, so a non-nullable column
fell through into the nullable block and invoked the callback a second
time per row (latent — currently masked by callers that guard with `if
(!IsNullable()) return`). Added the missing `return`.
issue: #51385🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`SealedDataGetter::Get`'s indexed branch (`from_data_==false`) threw
`AssertInfo("field data not found")` on a null row instead of returning
`std::nullopt` like the from-data branches, failing the whole group-by
query. Now returns nullopt so the NULL forms a distinct null group. Adds
a nullable group-by regression test (the index-only branch is
CI-verified).
issue: #51385🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
issue: #51299
## Problem
storage-v3 (loon/vortex) sealed segments load the raw vector column
resident **on top of** a vector index that already carries the raw data
(`HasRawData=true`), roughly **doubling** the per-segment disk footprint
of DiskANN / IVF_FLAT / HNSW(with-raw) segments. On the 100M / 128-dim
DiskANN benchmark (1 replica, 100Gi querynode ephemeral-storage) this
pushes a querynode over the cap and `load_collection` fails with
`segment request resource failed[resourceType=Disk]`.
The v2 binlog path already skips the raw fielddata when the index has
raw data (`ChunkedSegmentSealedImpl`: *"skip fielddata because index has
raw data"*); the v3 manifest path in
`SegmentLoadInfo::ComputeDiffColumnGroups` only marked the column
**lazy**, and `LoadColumnGroup(eager_load=false)` still materialized a
`ProxyChunkColumn` and set `field_data_ready`, so the raw column stayed
resident anyway.
## Fix
In `ComputeDiffColumnGroups`, when a field's index carries raw data and
`prefer_field_data_when_index_has_raw_data == false`, drop the raw
column from the load set entirely (mirroring the v2 binlog path) instead
of lazy-loading it. This covers vector fields and other indexed scalars
(retrieve/filter read the raw value from the index).
The **primary key is excluded**: sorted segments navigate rows through
the pk column (`num_rows_until_chunk` / `get_chunk_by_offset`) and a pk
scalar index is not registered in `scalar_indexings`, so the pk raw
column must stay resident — otherwise a `ColumnExpr` on the pk falls
back to a column scan (`use_index_data_ == false`) and crashes (`field
100 must exist when getting rows until chunk`, regressed go-sdk
`TestCreateSortedScalarIndex`).
The stale-copy drop is scoped to the real no-raw-index -> raw-index
reopen transition. `prefer_field_data` remains the explicit opt-in to
keep both resident.
---------
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Li Liu <li.liu@zilliz.com>
`insert_helper` in `PhyIterativeFilterNode` indexed the per-query result
window by the **relative** count instead of the **absolute** window end
(`base_idx + count`). For `nq_index >= 1` the shift guard `count > pos`
is always false, so an out-of-order candidate (a closer distance
arriving after farther ones — which approximate vector iterators do
emit) overwrote an existing top-k entry instead of shifting it,
corrupting the top-k of **every query vector after the first** in a
multi-query filtered search.
Extracted the insertion into `Utils.h::topk_binsert` with the correct
absolute-window math, matching the already-correct math in
`PhyIterativeElementFilterNode::CollectResults`.
Deterministic regression test
`IterativeFilterInsert.MultiQueryWindowOrdering` inserts an out-of-order
stream into the second query window and asserts the correct sorted
result.
issue: #51385🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Avoid direct captures of structured bindings in
`ChunkedSegmentSealedImpl.cpp`, which is compiled with OpenMP enabled.
- Use an init-capture for the field ID in `LoadBatchTextIndexes`.
- Use an init-capture for the field ID in `FinalizeLoadDiffForReopen`
when dropping JSON indexes.
- Keep the inner text-index commit lambda as a normal value capture
because it captures the ordinary init-capture from the outer lambda, not
a structured binding.
issue: #51231
## Test Plan
- [x] `make`
- [x] `ninja -C cmake_build
src/segcore/CMakeFiles/milvus_segcore.dir/ChunkedSegmentSealedImpl.cpp.o`
- [x] `git diff --check`
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
issue: #50834
Fix no-index sealed vector raw data warmup selection. When a sealed
segment has no usable built or interim vector index, segcore now treats
raw vector field data as the vector search representation and resolves
its warmup from vector-index policy. Once a built or interim index
exists, raw vector data keeps normal vector-field warmup semantics.
Go load paths continue to pass field warmup unchanged; segcore resolves
the effective policy from current segment state.
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
Optimize the k-way merge by advancing the heap root in place instead of
popping and pushing it for every candidate. Use typed deduplication keys
for INT64 and VARCHAR primary keys, including element-level and group-by
search paths.
Skip late materialization when the search plan has no non-primary target
fields, while preserving the primary field for mixed output fields
required by proxy reranking.
Ensure submitted segcore output-field tasks are joined on exceptional
paths and add correctness and benchmark coverage for the optimized merge
implementations.
https://github.com/milvus-io/milvus/issues/51315
Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
pr: https://github.com/milvus-io/milvus/pull/49922
## Summary
Route generic LIKE Match operations to raw-data scan when a varchar
inverted index is present, while keeping PrefixMatch on the
inverted-index path. Update planner policy and sealed-segment coverage
for a complex LIKE pattern.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
issue: https://github.com/milvus-io/milvus/issues/51466
## What changed
`PhyUnaryRangeFilterExpr::DetermineExecPath()` previously used a
JSON-specific hard-coded denylist for string predicates:
- `Match`
- `PostfixMatch`
- `InnerMatch`
This bypassed the concrete scalar index's `ShouldUseOp()` policy. As a
result:
- indexes capable of evaluating these predicates, such as `STL_SORT`,
were
forced to scan and parse raw JSON;
- JSON string predicates could use a different execution policy from
ordinary
VARCHAR predicates;
- `RegexMatch` remained on the index path even for indexes whose planner
policy
explicitly preferred raw-data execution.
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
## Summary
Keep raw search distances through segment search, iterator output, index
query, and chunk merging so ranking/reduce order is not affected by
`round_decimal`.
Apply `round_decimal` only when producing the final search response,
after reduce/rerank/order-by has already determined result order.
Returned scores keep the requested precision without changing TopK
selection.
issue: #50347
---------
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
issue: #51250
Keep TEXT segments on StorageV3 while making growing-source flush an
explicit allowed mode instead of a mandatory TEXT path.
Propagate create-segment storage versions through WAL flusher and write
buffer, reject storage-version mismatches in DataCoord, and keep raw
segcore chunks sticky for segments that may be flushed from growing
source.
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
relate: #50304
## Summary
Add the local format design doc, use the storage writer format constant,
and introduce local_format metadata propagation with column-group split
policy support.
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Related to #51068
Store PK index slots and virtual PK offset maps in RuntimeResourceState.
Build Storage V1 and V2 PK indexes through the unified translator and
stage PK replacements until the final state publication.
Make PK lookup, range search, bulk subscript, and delete filtering use a
single published snapshot. Remove the sealed InsertRecord fallback and
clear stale PK resources when external segments become empty.
Add regression coverage for atomic PK replacement and zero-row external
segment cleanup.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #51100
## Summary
The C++ persistent-storage `get` failure counter was registered without
the `status="fail"` label, unlike the other operation failure counters.
The counter was still incremented only on failed `get` operations, but
Prometheus exported it as a no-status `get` series, so status-based
dashboards and queries could not identify it as a failure series.
This PR adds `status="fail"` to `getFailMap`, making
`internal_storage_op_count{persistent_data_op_type="get",
status="fail"}` consistent with `put`, `stat`, `list`, and `remove`
failures.
## Note
This changes the Prometheus time series shape: the old no-status `get`
failure series will stop, and the new `status="fail"` series will start
from zero.
Signed-off-by: xaxys <tpnnghd@163.com>
issue: #51114
JsonStats shredding now loads through
`ManifestGroupTranslator`/`ChunkReader`, but JsonStats files do not have
the normal sealed-segment manifest metadata. The load path reconstructs
the needed column-group view from the existing parquet footer/schema:
column names, Arrow field ids, row counts, and file ordering. This keeps
the current `meta.json` contract unchanged; parquet key-value metadata
is only a fallback for old files that do not have the separate meta
file.
The main follow-up after switching to the manifest translator was
projection behavior. Lazy loading now builds one projected reader per
shredding column, so a predicate on one JSON path does not pull the
whole group; when warmup is enabled it still uses the full column group
so warmup can preload everything together. The added tests cover parquet
files without packed field-list metadata, multi-file ordering, lazy
single-column projection, and warmup full-group projection.
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
Related to #51068
- manage JSON index load, replace, and drop through reopen COW snapshots
- remove LoadScope_Index and the legacy QueryCoord-to-segcore update
path
- drop JSON indexes by field and nested path to preserve sibling indexes
- regenerate query proto and QueryNode mocks
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: https://github.com/milvus-io/milvus/issues/42053
This PR fixes ngram index eligibility checks for UTF-8 literals by using
character count instead of byte length in the C++ gate and phase1
validation. It also adds regression coverage for short multibyte
literals to ensure they correctly fall back instead of being sent to
Tantivy ngram queries.
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
issue: #36214
issue: #51324
## Summary
gcp-native-storage transitively depends on
`boost/align/aligned_allocator.hpp` and
`google/cloud/storage/retry_policy.h` headers but did not declare
milvus_conan_deps. As a result, building with ENABLE_GCP_NATIVE=ON
failed because the required Conan include directories were not
propagated to the object target during compilation.
## Changes
Link milvus_conan_deps directly to gcp-native-storage in
`internal/core/src/storage/gcp-native-storage/CMakeLists.txt` so its
usage requirements are available when compiling the target.
## Verification
Verified locally.
### Reproduce
```bash
ENABLE_GCP_NATIVE=ON make build-cpp
```
Before this change, compilation failed with:
```text
fatal error: boost/align/aligned_allocator.hpp: No such file or directory
fatal error: google/cloud/storage/retry_policy.h: No such file or directory
```
After this change:
- `ENABLE_GCP_NATIVE=ON make build-cpp` succeeds.
- `make verifiers` passes, including:
- build-cpp
- getdeps
- cppcheck
- rustcheck
- fmt
- static-check
Signed-off-by: Jeremiah Xing <dev.jrmh@gmail.com>
## Summary
issue: #51052
This PR reduces QueryNode memory pressure by avoiding duplicated or
long-lived segment metadata after sealed segment load/reopen and by
caching frequently reused schema conversion state.
## Storage Mode Scope
- For manifest-backed Storage V3 segments, both Go-side runtime load
info and segcore-side runtime load info are compacted after load/reopen.
- For Storage V2 binlog-mode segments, Go-side sealed runtime load info
is compacted, while segcore keeps the full load info because binlog diff
still depends on binlog/index metadata.
- The optimization primarily targets Storage V3, which is the forward
path.
## Commit Breakdown
1. `9d4dbdf065 enhance: cache text LOB loon arrow schema`
- Cache the Loon Arrow schema generated for the TEXT LOB binary
projection used by Storage V3 load paths.
- Invalidate the cache when schema fields change and clear mutable cache
state on schema assignment.
2. `4f0bb58ef6 enhance: avoid duplicate segment index load info`
- Remove duplicated converted index load info storage.
- Reuse the existing field-index cache for current index identity and
diff/drop decisions.
3. `89e8a15b42 enhance: compact runtime segment load info`
- Compact Go QueryNode sealed segment load info after load/reopen.
- Keep runtime-required lightweight fields and cache related data size
before dropping large metadata.
4. `8a38b2fdbf enhance: skip real count scan without deletes`
- Return row count directly when a segment has no deleted rows.
- Avoid scanning the deleted bitmap on the real-count fast path.
5. `d041837d33 enhance: compact segcore runtime load info`
- Compact manifest-backed segcore runtime load info after load/reopen.
- Preserve lightweight diff identity and local runtime state needed by
schema-only reopen and later diffs.
## Test Plan
- [x] `git diff --check upstream/master...HEAD`
- [x] `gofumpt -l internal/querynodev2/segments/segment.go
internal/querynodev2/segments/segment_loader.go
internal/querynodev2/segments/segment_test.go
internal/querynodev2/segments/utils.go
internal/querynodev2/segments/utils_test.go`
- [x] `/opt/homebrew/opt/llvm@18/bin/clang-format --dry-run --Werror
--style=file <changed C++ files>`
- [x] `ninja -C cmake_build all_tests`
- [x] `./cmake_build/unittest/all_tests
--gtest_filter='SchemaTest.ConvertToLoonArrowSchemaCachesTextLobBinarySchema:SegmentLoadInfoTest.CompactRuntimeInfoForManifest:SegmentLoadInfoTest.CompactRuntimeInfoForManifestKeepsIndexDiffIdentity:SegmentLoadInfoTest.CompactRuntimeInfoForManifestSchemaCopyKeepsIndexIdentity:SegmentLoadInfoTest.ReplaceSchemaForReopenPrunesDroppedFieldRuntimeState:SealedSegmentLoad.LoadPublishesDefaultFilledStateAfterCompact:SealedSegmentReopen.SchemaOnlyReopenPublishesDefaultFilledState:SealedSegmentReopen.SchemaAwareReopenDiscardsOlderSchema:SealedSegmentReopen.SchemaOnlyReopenPreservesCreatedTextIndexState:SealedSegmentReopen.TextIndexCreatedWipedByReopen'`
- [x] `go test -v -count=1 -tags dynamic,test -gcflags="all=-N -l"
-ldflags="-extldflags \"-Wl,-rpath,${MILVUS_WORK_DIR}/cmake_build/lib
-Wl,-rpath,${MILVUS_WORK_DIR}/internal/core/output/lib
-Wl,-rpath,${MILVUS_WORK_DIR}/cmake_build/thirdparty/boost_ext\""
./internal/querynodev2/segments -run
"Test(CompactSegmentLoadInfoForRuntime|GetSegmentRelatedDataSize)$"
-timeout 300s`
- [x] `go test -v -count=1 -tags dynamic,test -gcflags="all=-N -l"
-ldflags="-extldflags \"-Wl,-rpath,${MILVUS_WORK_DIR}/cmake_build/lib
-Wl,-rpath,${MILVUS_WORK_DIR}/internal/core/output/lib
-Wl,-rpath,${MILVUS_WORK_DIR}/cmake_build/thirdparty/boost_ext\""
./internal/querynodev2/segments -timeout 300s` timed out locally after
303.862s while still running MinIO-backed retrieve tests
(`TestRetrieveWithFilter`).
---------
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
issue: #51275
### Problem
Under StorageV3 growing-source flush, a manifest legally keeps column
groups whose fields were later dropped: drop does not rewrite flushed
data, and compaction removes the leftover lazily. Loaders however
projected such groups against the **current** schema and treated the
mismatch as an error:
- **Growing recovery** (`SegmentGrowingImpl::LoadColumnsGroups`)
submitted every manifest group with a full-schema projection; a group
whose fields were all dropped has an empty projection, which
milvus-storage rejects by contract (`ChunkReaderImpl::open`: "No needed
columns found in column group"). The channel-recovery level retried the
deterministic failure every second — observed locally: **4140 identical
failures over ~3 minutes**, the channel unavailable (`no available shard
leaders`) the whole time, until a compaction rewrite happened to remove
the group.
- **Sealed load** (`ChunkedSegmentSealedImpl::LoadColumnGroups`)
resolved every manifest column to a field id without a membership check
and tripped the schema-lookup assert on any group still carrying a
dropped column — mixed groups included.
### Fix
Treat a dropped-field column as a legal leftover at the loaders:
- Growing: skip a column group before submitting its load task when none
of its columns exist in the segment schema (logged).
- Sealed: filter resolved field ids absent from the schema snapshot
while building `cg_field_ids` (logged); a group filtered to empty
produces no load task. Mixed groups keep loading their live columns via
projection, unchanged.
### Verification
- Local A/B with a schema-evolution chaos workload (concurrent add/drop
ordinary + BM25 function field, DML/query/search, `useLoonFFI` +
growing-source flush enabled, StreamingNode SIGKILL/restart injections):
before the fix, recovery wedged in the retry loop (4140 errors / ~3 min
/ final query 503); after the fix the same scenario logs 2 group skips,
the channel recovers in seconds, and the full serial consistency
validation passes.
- New UT `LoadGrowingSegmentSkipsDroppedFieldColumnGroup`: flushes a
manifest with a field-exclusive column group (writer pattern
`"0|1|100,101"`), reloads the segment under a schema without that field,
and asserts the load succeeds with the group skipped (plus a guard
assertion that the manifest really contains the exclusive group).
- The sealed-path filter is exercised by code inspection and mirrors the
growing-side semantics; on current master the external-table load path
derives its projection from schema-owned column names and is not exposed
to this class.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## Summary
- Upgrade Knowhere dependency to 49507ef3 to pick up the fix that
removes the num_build_thread upper-bound validation.
issue: #42937
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
- Reuse the existing `ExprResCacheManager/ExprCacheHelper` from the
expression layer.
- Cache JSON-flat exact-path validity bitmaps by segment + field/full
JSON pointer + `Any|Numeric|String|Bool` canonical key.
- Exclude literal/operator from the key so validity can be reused across
predicates.
- Do not add a `JsonFlatIndex`-owned LRU or change the index format.
This is based on the latest `master`, including merged PR #51235.
## Verification
Verification is pending and will continue after the PR is created.
issue: #51234
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
Related to #51068
Store sealed-segment text index holders and cache slots in the published
runtime resource snapshot. Stage create, load, and reopen updates
against cloned state before publishing them atomically.
Keep text index resources alive through captured snapshots, prune
indexes removed by schema reopen, and preserve created-index markers
only for fields present in the target schema.
Remove the obsolete direct LoadTextIndex path, legacy batch overloads,
and marker wrappers. Add coverage for staged visibility and old snapshot
resource lifetime.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #51117
Supersedes #51207 per team decision to ship the skipped-field tolerance
for 3.0.0; the era-consistent replay consume path (#51146) remains a
follow-up.
### Problem
A growing segment created while a field was absent from the schema never
materializes that column: the InsertRecord ctor only allocates fields
present at creation, Reopen fills later-added ordinary fields but skips
function outputs by design, and a field dropped before segment creation
is skipped at consume. Flushing such a segment with a schema snapshot
that still carries the field hit `Assert "data_.find(field_id) !=
data_.end()"` in the growing-source flush path and crash-looped the
StreamingNode (exit 134 / CrashLoopBackOff in the schema-evolution L3
runs).
### Design
Principle: the flush writes exactly `flushSchema ∩ materialized`. A
non-materialized column is legally absent — a field dropped from the
segment's own schema, or a function output recomputed by bump-schema
compaction backfill. Real schema/data inconsistency is segcore's own
concern and is verified inside the flush.
- **Go (layout source of truth)**: trim the column groups to the source
segment's materialized field ids (plus system fields) right after layout
derivation — `Fields` and the parallel `Columns` array in lockstep — so
the writer pattern, binlog meta and metacache current split all describe
the same layout. An empty materialized report is refused. On a
committed-flush ack retry the layout is trimmed to the committed binlogs
(`ChildFields` union), the persisted truth.
- **C++ (segment-internal judgement)**: a column missing from the insert
record (`HasFieldData` semantics: allocated-but-empty counts as missing)
is skipped when the field is gone from the segment's own schema
(dropped) or is a function output; a regular field still present in the
segment schema is materialized by ctor/Reopen by construction, so that
case hard-errors instead of asserting.
- Skipped columns are backfilled by bump-schema compaction via
`RecordMaterializer`.
### Review responses (@liliu-z)
- stale `Columns` / phantom column in writer pattern →
`filterColumnGroupFields` trims `Fields`+`Columns` in lockstep at all
three trim sites
- committed ack-retry currentSplit divergence → layout trimmed to
committed binlogs' `ChildFields` union
- allocated-but-empty function-output column wedging the flush →
materialized report and the flush skip both gate on `HasFieldData`
semantics (empty = not materialized)
- silent skip of a missing regular field → three-way judgement: dropped
from segment schema → skip; function output → skip;
present-in-segment-schema regular field → hard error (`has no field data
in growing segment ... but is present in the segment schema`)
- retryability routing of the missing-field error: the legally-absent
cases no longer produce an error at all; the remaining hard error
indicates real data loss and is construction-unreachable via public APIs
(the consume path asserts regular fields carry data)
### Verification
- Go UT (`growing_source_test.go`): trim semantics — dropped ordinary
field trimmed, `Columns` kept in lockstep, empty materialized report
refused, committed-retry trim by `ChildFields`
- C++ UT (`test_storage.cpp`): flush skips a non-materialized function
output; skips a dropped field carried by a staler flush schema; skips an
allocated-but-empty function-output column
- Local schema-evolution chaos runs (concurrent add/drop field + BM25
function field + DML/query/search with `useLoonFFI` + growing-source
flush enabled, incl. StreamingNode force-kill/restart injections): no
panic, no crash loop, final serial consistency validation passes. The
exact original assert requires an era-mismatch replay not reproduced
locally; that path is covered by the unit tests above.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## Summary
- add `json_subpaths` support to Tantivy `json_exist_query`
- expose exact-path existence through `ExistsQuery(..., false)`
- use a contains-specific `JsonExactPath` validity mode for actual
JSON-flat executors
- keep contains hit lookup through `In()` while using exact-path
non-null-leaf validity
## Semantics
- NULL, missing paths, JSON null, and object-only paths remain UNKNOWN
- `NOT UNKNOWN` remains UNKNOWN
- scalar values and non-empty arrays preserve current flattened contains
behavior
- empty arrays remain the documented limitation
## Verification
- `JsonFlatIndex*`: 25/25 passed
- `*JsonContains*`: 597/597 passed
- `cargo fmt --check`
- clang-format dry-run on changed non-generated C++ files
- `git diff --check`
## Benchmark
For scalar-benchmark
`json_array_operations/base/A01_contains_flat_array/json_flat` with 1M
rows and 2 matches:
- first query: 24.945 ms
- average: 24.625 ms
- P50/P99: 24.525/24.607 ms
- current local master: approximately 1114/1119 ms P50/P99
issue: #51234
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
Related to #51068
Migrate sealed segment vector index runtime state from live members into
RuntimeResourceState, including vector index entries and binlog vector
index configs. Update vector search, vector retrieval, raw-data checks,
refine, prefetch, load/drop/clear, and staged reopen paths to read and
publish through runtime snapshots.
Also update SearchOnSealedIndex to consume a pinned vector index entry
directly, and make vector index config accessors const-compatible for
snapshot usage.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
## Summary
After schema evolution adds an `enable_match` field (e.g.
`drop_collection_field` + `add_collection_field` of a same-name text
field), `text_match` could fail with `text index not found for field
<id>` on a growing segment even though `describe_index` reported
`Finished`.
## Root cause
A growing segment builds its per-segment text indexes exactly once, in
the constructor (`CreateTextIndexes`), against the construction-time
schema — `text_indexes_` has a single writer, reachable only from that
path. When a query carrying the new schema drives `LazyCheckSchema ->
Reopen(SchemaPtr)`, Reopen only backfilled empty column data
(`fill_empty_field`) and never built the text index for the newly added
field — unlike the sealed path, where both load and schema-only reopen
go through `ComputeDiff -> text_indexes_to_create -> CreateTextIndex`.
So an old growing segment upgraded by Reopen had no
`text_indexes_[new_field]`, and `text_match` hit `GetTextIndex()` and
threw `TextIndexNotFound`. The window closes once the segment is
flushed/sealed, loaded (sealed load rebuilds the index) and released
from the delegator, which is why the failure is intermittent and
self-healing but hard, non-retryable while it lasts.
## Fix
In `SegmentGrowingImpl::Reopen(SchemaPtr)`, for newly added
`IsStringDataType && enable_match` fields, **before publishing
`schema_`** (LazyCheckSchema readers don't take `sch_mutex_`, so a
concurrent query observing the new schema skips Reopen and queries
immediately — the index must already exist by then):
1. Build the text index via the new `BuildTextIndexForMeta(const
FieldMeta&)` (the field is not yet in the published `schema_`, so the
meta is passed in; mirrors sealed's `CreateTextIndexWithSchema`). The
index is **staged as a local object** — it is not visible in
`text_indexes_` yet.
2. **Index the pre-existing rows with the same default-value-or-nulls
fill the column received** (constructed directly — the content is
knowable without reading the column back, since the exclusive
`sch_mutex_` blocks concurrent inserts). This keeps the index consistent
with the column, matching sealed's create-from-raw behavior
(`BulkRawStringAt -> AddTextSealed`): when the added field carries a
`default_value`, old rows must match text queries against the default
text and report not-null. Backfilling every row (nulls included) also
keeps text-index doc offsets dense and aligned with `insert_record_`
rows, which query-side bitmap sizing relies on.
3. Force `Commit()+Reload()` on each staged index; after **every** new
field succeeds, publish all staged indexes into `text_indexes_` together
(an O(1) map insert per field under one `mutex_` acquisition).
Publish-after-success makes the failure path sound across fields as well
as within one: a present `text_indexes_` entry is always a fully
backfilled, committed index; a throw anywhere during any field's
build/backfill/commit publishes nothing and leaves `schema_`
un-advanced, so the next query rebuilds everything from scratch (no
per-field partial publication that a retry's presence guard would skip
over). It also shrinks the `mutex_` hold during Reopen from the O(N)
backfill to the final map insert, so concurrent `GetTextIndex` readers
(text_match on existing fields) are not stalled by the backfill.
Function-output fields (e.g. BM25 sparse output) remain skipped: they
are non-string and never carry a text index. The related-but-distinct
`field index meta not found` symptom for BM25 function-output fields
(follow-up of #50783: `index_meta_` is not updated on growing reopen) is
a separate fix and not covered here.
## Test
`test_schema_reopen.cpp`:
- `ReopenBuildsTextIndexForNewEnableMatchField`: growing segment on
schema v1, old rows inserted, Reopen to v2 adding a nullable
`enable_match` VARCHAR (no default) — `GetTextIndex` throws before
Reopen; after Reopen it returns the index, `MatchQuery` finds nothing
and `IsNotNull` reports all rows null, consistent with the column.
- `ReopenTextIndexIndexesDefaultValueForOldRows`: same flow with a
`default_value` on the added field — after Reopen
`MatchQuery("default")` matches all N pre-existing rows and `IsNotNull`
reports them non-null, matching sealed semantics.
- `ReopenBuildsTextIndexesForMultipleNewFields`: one Reopen adds two
enable_match fields (one nullable-null, one with default) — both indexes
come out complete through the batched staged publish.
All tests query immediately after Reopen with no explicit
`Commit()/Reload()`, asserting the forced-visibility behavior.
## Known limitation (pre-existing, follow-up)
`EnsureArrayOffsetsForStructField` still runs after `schema_` is
published, so a throw there can leave a schema-advanced segment without
array offsets — pre-existing behavior, unchanged by this PR; reworking
Reopen's publication order for non-text state is left as a follow-up.
(The text-index half of this concern is resolved: the index is now built
before publication and a build failure leaves the reopen retryable.)
issue: #50484🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
issue: #51252
## What
- Add `common.enableDriverPrefetch` with default `false`.
- Wire the config into segcore via initcore and a refresh callback.
- Skip `Driver::PrefetchAsync()` when the switch is disabled.
## Why
This provides a runtime mitigation for regressions caused by query
driver operator prefetch on short scalar queries, while keeping prefetch
configurable for follow-up validation.
## Notes
When disabled, expression execution path determination still runs lazily
on the query thread; this only disables driver-level async prefetch
submission/waiting.
## Tests
- `make generate-yaml`
- `CLANG_FORMAT=/opt/homebrew/opt/llvm@15/bin/clang-format
internal/core/run_clang_format.sh internal/core`
- `make lint-fix`
Signed-off-by: sunby <sunbingyi1992@gmail.com>
issue: #51192
design doc:
docs/design-docs/design_docs/20260708-xgboost-function-chain.md
Add native xgboost FunctionChain expression support for L0 rerank with
FileResource-backed UBJ models.
This change includes:
- xgboost FunctionChain expression registration, parameter validation,
and execution
- FileResource-based UBJ model discovery and local path resolution
- lazy model loading with singleflight, lease/refcount lifecycle
protection, and stale eviction on FileResource sync
- cgo bridge for Arrow C Data based batch prediction
- native C++ UBJ model parser and predictor for supported tree models
- runtime-disabled stub for builds without cgo and with_xgboost
- validation for unsupported params, output modes, feature count
mismatch, invalid models, unsupported objectives, unsupported boosters,
multiclass models, multi-target leaf vectors, and unsupported input
column types
- C++ unit tests, Go tests, native parity tests, and Python client L0
E2E tests
- xgboost FunctionChain design document
L2 rerank support is intentionally deferred because Proxy does not yet
support FileResource sync and local resolution.
Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
## Summary
This PR avoids directly capturing a structured binding in
`ChunkedSegmentSealedImpl::LoadBatchJsonKeyIndexes`.
Clang rejects direct structured-binding captures when compiling this
code path with OpenMP enabled. The lambda now uses a generalized
init-capture for `field_id`, preserving the existing value-capture
behavior while avoiding the unsupported capture form.
issue: #51231
## Test Plan
- [x] `git diff --check`
- [x] `/tmp/milvus-clang-format-15-bin/clang-format-15 --dry-run
--Werror --style=file
internal/core/src/segcore/ChunkedSegmentSealedImpl.cpp`
- [x] `make SKIP_3RDPARTY=1`
- [x] `PATH="/tmp/milvus-clang-format-15-bin:$PATH" make verifiers`
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
issue: #51181
The 3VL null-semantics rework correctly made conjunctions keep UNKNOWN
rows active (under `NOT`, `UNKNOWN AND FALSE = FALSE` must stay
reachable), but this disabled the AND early-exit everywhere — including
at the top-level filter, which folds UNKNOWN into the excluded set
exactly like FALSE. For filters like `json["a"] == 1 and heavy_expr and
...` over data where most rows lack the key, every following conjunct
ran over rows whose outcome was already decided.
## Changes
**Null-rejecting propagation (compile-time, zero hot-path cost)**
- New virtual `Expr::MarkNullRejecting()`: no-op default stops the
propagation at any non-conjunct node (in particular `NOT`, where FALSE ≠
UNKNOWN); `PhyConjunctFilterExpr` overrides it to set `null_rejecting_`
and recurse into its inputs. The property legally crosses nested AND/OR:
their combination cannot turn an operand's UNKNOWN into TRUE when the
root folds UNKNOWN into FALSE.
- `ExprSet` gains a `null_rejecting` ctor flag; `PhyFilterBitsNode`
(folds UNKNOWN in `ConvertPredicateToFilteredBitset`) and
`PhyIterativeFilterNode` (includes rows on data bits) pass true.
Element-level filter operators keep the default and merely miss the
optimization for now.
- A null-rejecting AND uses `data & valid` as its active set: the
zero-TRUE batch early exit is restored, and UNKNOWN rows are dropped
from the next expression's row-level bitmap input. OR is untouched
(`UNKNOWN OR TRUE = TRUE` can still flip a row in). Included row set is
provably unchanged — pinned by `NullRejectingMatchesDefaultIncludedSet`;
the existing NOT-context test keeps guarding strict 3VL behavior in
unmarked conjuncts.
**Single active-bitmap build per input per batch**
- Previously the identical `data | ~valid` bitmap was built twice per
non-first input (once in `UpdateResult` only to be counted and
discarded, once in `SetNextExprBitmapInput`), plus once more for the
final input where `ClearBitmapInput` immediately discarded it. Now
`BuildActiveBitmap` runs once: `none()` (early-terminating, no full
popcount) decides the batch-level exit and the same bitmap is
`std::move`d into the eval context. The build is skipped for the last
actually-evaluated position (`last_eval_pos`), so trailing batch-ngram
entries don't force a useless build.
**Pre-existing OOB crash fixed at its root**
- `ReorderConjunctExpr` reserves `input_order_` slot `inputs_.size()`
for a runtime `PhyLikeConjunctExpr`, but the block that materializes or
erases it was gated on `!has_input_offset`. On
`PhyIterativeFilterNode`'s native offset path (reachable: LIKE supports
offset input) the reserved slot never got a backing expression, and both
the Eval loop and `SkipFollowingExprs` indexed `inputs_[inputs_.size()]`
— reproduced as a SIGSEGV under gtest with the production-reachable
state. The init block now decides materialize-vs-erase in every mode
(the batch-ngram path cannot be used with an offset input anyway, and an
instance's offset mode is fixed for its lifetime), restoring the
invariant "every `input_order_` entry has a backing expression" at its
single owner rather than guarding each consumer.
**IterativeFilterNode: null-rejecting by construction**
- Both consumption paths now fold `data &= valid` explicitly instead of
relying on the unenforced convention that UNKNOWN rows carry `data=0`
(the fallback path collected `valid_bitset` but used it only in
asserts).
Note: `ColumnVector::AllTrue/AllFalse` lose their last production caller
here but are kept as general-purpose public API with existing tests.
## Verification
- Unit tests (Linux, `make build-cpp-with-unittest`): `ConjunctExprTest`
7/7 (6 new: null-rejecting early exit / TRUE rows still evaluated /
included-set equivalence marked-vs-unmarked / OR unaffected /
propagation stops at non-conjunct nodes / offset-input reserved-slot
erasure incl. the previously-crashing early-exit path), `*Conjunct*`
9/9, `*Iterative*` 4/4, `TaskTest*` + `FilterOnlySearchTest*` 37/37,
`LikeConjunctExpr` pass.
- Two rounds of multi-agent adversarial review; all confirmed findings
addressed.
- clang-format (19) clean on all changed lines.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
issue: https://github.com/milvus-io/milvus/issues/50416
## What
This PR moves external field checks into the C++ Search/Retrieve path
after `LazyCheckSchema` has refreshed the segment schema and load info.
Instead of probing `HasFieldData` or `FieldAccessible`, the guard checks
whether every referenced external field exists in the currently loaded
external manifest:
- `CheckExternalFieldsInLoadedSnapshot` is renamed to
`CheckExternalFieldsInLoadedManifest` to match the real readiness
boundary.
- Plan `access_entries_` now includes expression/execution fields and
requested output fields, so Search and Retrieve use one field-reference
list for external manifest checks.
- Search filter-only skips the vector field and output target fields
because that path does not access them.
- Non-manifest segments keep the existing behavior.
The loaded manifest column-group cache is also preserved across
schema-only reopen when the manifest path is unchanged, so the query
path can check manifest membership without parsing remote manifests or
falsely reporting missing columns after schema refresh.
## Why
For external collections, a newly added field can appear in the Go
collection schema before the loaded C++ segment has refreshed to a
manifest that contains the corresponding external column.
If Search/Retrieve touches that field against the stale loaded manifest,
segcore can surface an internal field-state failure instead of a clear
external-field-not-ready error. The real readiness condition here is
whether the loaded manifest contains the external column, not whether
the field is loaded into memory.
## Validation
- `ninja -C cmake_build milvus_core all_tests`
- `DYLD_LIBRARY_PATH=cmake_build/src cmake_build/unittest/all_tests
--gtest_filter='PlanProto.SearchPlanCollectsFieldAccessInfo:PlanProto.RetrievePlanCollectsFieldAccessInfo:SegmentLoadInfoTest.HasManifestColumnUsesManifestColumnNames:SegmentLoadInfoTest.CachedManifestColumnsCanBeInherited'`
- `/opt/homebrew/opt/llvm@15/bin/clang-format --dry-run -Werror
internal/core/src/query/PlanImpl.h internal/core/src/query/PlanProto.cpp
internal/core/src/query/PlanProtoTest.cpp
internal/core/src/segcore/ChunkedSegmentSealedImpl.cpp
internal/core/src/segcore/ChunkedSegmentSealedImpl.h
internal/core/src/segcore/SegmentInterface.h
internal/core/src/segcore/SegmentLoadInfo.cpp
internal/core/src/segcore/SegmentLoadInfo.h
internal/core/src/segcore/SegmentLoadInfoTest.cpp
internal/core/src/segcore/segment_c.cpp`
- `CLANG_FORMAT=/opt/homebrew/opt/llvm@15/bin/clang-format make
cppcheck`
- `git diff --check milvus/master...HEAD`
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #51068
Keep sealed JSON stats in RuntimeResourceState so published runtime
snapshots carry them through clone and freeze transitions.
Load and replace JSON stats through the staged reopen committer, and
drop them by mutating the staged runtime state. This removes the old
direct LoadJsonKeyIndex CGO path and the unused RemoveJsonStats
interface.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #51179
`ThreadPool::Submit` returns `std::packaged_task` futures, which do not
block on destruction. Sites that submit tasks capturing stack state by
reference and wait with a bare `future.get()` loop let the first
exception unwind the frame while remaining tasks are still
queued/running — turning a recoverable I/O error into use-after-scope /
use-after-free. Same defect class as the `IndexEntryReader` fix (#46958
/ 2.6's #50937); an audit found four remaining sites.
## Changes
- **`storage/Util.h`**: add noexcept `DrainFutures` / `DrainFuture`
cleanup helpers (wait for every still-valid future, swallow its
exception), complementing the existing `WaitAllFutures`. For use in
catch blocks / scope guards where another exception is already
propagating.
- **`DiskFileManagerImpl::AddBatchIndexFiles`**: capture
`local_chunk_manager` by value instead of `[&]`; collect slice buffers
via the existing `WaitAllFutures`, which drains all tasks before
rethrowing the first error.
- **`GetFieldDatasFromStorageV2`** (`storage/Util.cpp`): wrap the
channel-consumption loop; on exception, drain the channel first (the
producer may be blocked pushing into the bounded channel), then wait for
the load task, then rethrow — the pattern `GroupChunkTranslator` already
uses.
- **`SegmentGrowingImpl::load_column_group_data_internal`**: same
treatment via a `folly::makeGuard` drain guard (dismissed on the success
path).
- **`IndexEntryEncryptedLocalWriter::WriteEntry` /
`EncryptAndWriteSlices`**: drain the sliding window of pending
encryption tasks (which capture `this`) with a scope guard before
unwinding, so no task outlives the writer.
## Semantics
The primary exception always propagates to the caller unchanged (bare
`throw;` / `WaitAllFutures` rethrows the first task error after all
tasks complete). Only secondary background-task errors during cleanup
are swallowed — C++ permits a single in-flight exception, and letting
another escape from unwind-time cleanup would `std::terminate`.
## Verification
- All five touched files compile cleanly (clang 19 `-fsyntax-only` with
the project's exact compile flags), including a collision check against
the file-local `DrainFutures(futures, first_error)` in
`IndexEntryReader.cpp`.
- Changed lines pass clang-format 19 with the repo `.clang-format` (zero
diffs).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## Summary
issue: #50976
This PR fixes NULL / UNKNOWN propagation gaps in filter expressions for
missing nested values:
- Treat missing JSON arithmetic operands and JSON `array_length`
extraction failures as UNKNOWN instead of definite booleans.
- Preserve UNKNOWN during parser/RBO contradiction folds for nested JSON
paths, array subscripts, and struct-array subscripts.
- Treat missing array / struct-array elements as UNKNOWN during
execution.
- Propagate JSON path/cast index typed-known masks so missing paths and
cast failures are not matched by `!=`, `not in`, or outer `not(...)`.
- Propagate JSON flat index comparable-value masks for comparison
predicates.
The commits are split by issue area:
- `4f15d6fe79` `fix: treat missing json arithmetic operands as unknown`
- `69d6e95917` `fix: preserve unknown for missing nested predicates in
rbo`
- `51b58edf05` `fix: treat missing array elements as unknown`
- `d9dd4c9334` `fix: honor json path index unknown values`
- `a9d813b322` `fix: honor json flat index unknown values`
## Verification
- `LOCALSTORAGE_PATH=/tmp/milvus-test-localstorage go test
-buildvcs=false -count=1 ./internal/parser/planparserv2/rewriter`
- `ninja -C /tmp/milvus-50976-build all_tests -j32`
- focused C++ regressions, including
`JsonFlatIndexTest.*:JsonFlatIndexExprTest.*:JsonIndexTest.*:JsonPathIndexTest.*`
(49 tests)
- focused array regressions including
`Expr.TestArraySubscriptMissingElementIsUnknown` and `Expr.TestArray*`
- `git diff --check origin/master...HEAD`
## Note
While adding extra JsonFlatIndex null-expression coverage, a separate
pre-existing `PhyNullExpr::DetermineExecPath()` / `std::call_once`
deadlock was found. That hanging test is not included here; this PR
keeps JsonFlatIndex coverage scoped to comparison UNKNOWN behavior.
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
issue: #51054
## What
Sealed tantivy index builds (inverted scalar, nested array, text match,
ngram, json key stats — V5 and V7 writers alike) now deterministically
produce **one tantivy segment**:
- Build-mode writers set `NoMergePolicy` at creation, so background
policy merges never exist during a build — the historical merge-all race
("segments could not be found in the SegmentManager") is impossible by
construction rather than merely handled. The growing text index passes
`enable_background_merge=true` and keeps the default policy (long-lived
writer, periodic commits, needs bounded segment count).
- `finish()` becomes: commit → **when more than one segment remains,
merge all searchable segment ids on the same writer and propagate any
merge error** → garbage-collect → wait_merging_threads → log the
resulting segment ids. V5's `let _ = merge(...)` error swallow is
replaced with the same error-propagating shape (its best-effort merge
could silently leave multi-segment indexes when the race fired). Note:
with `NoMergePolicy` the merge deterministically yields one segment, but
`finish()` does not itself re-assert `len == 1` after the merge — the
single-segment guarantee is covered by the
`test_sealed_build_finishes_single_segment` unit test rather than a
runtime check.
## Why
The V7 writer lost V5's finish-time merge (commented out with a TODO
referencing the long-closed #45590), so every sealed V7 index ships as
~15 MB-arena-sized segments: 19 segments for a 10M-row scalar index,
~400 for a 100M-element nested array index. Multi-segment hits
range-style queries — each segment pays its own term-dict streaming +
posting decode + doc-bitmap pass. Measured (10M rows, Apple M-series,
median of 3):
| query | multi-seg | single-seg | speedup |
|---|---|---|---|
| int64 `> v` @1% (19 seg) | 10.6 ms | 4.2 ms | 2.5x |
| varchar `> v` @1% (19 seg) | 11.5 ms | 4.1 ms | 2.8x |
| array MATCH_ANY @1% (261 seg, 60M elems) | 15.9 ms | 5.1 ms | 3.1x |
| array MATCH_ANY @50% | 379.7 ms | 55.3 ms | 6.9x |
Equality/term queries are unchanged (multi ≈ single within noise). Index
size shrinks (int64 10M: 51→21 MB; array 60M elems: 698→174 MB). Cost:
build wall time +16–30% at the 100M-element worst case (seconds at
typical segment scale); total build IO is lower since intermediate
policy merges no longer write discarded segments.
## Verification
- 292 segcore element/array/match tests green on the development branch;
text/ngram/json suites' failure sets byte-identical to pre-change
baseline.
- ~15 merge-all executions up to 60M docs with `RUST_BACKTRACE=full`:
zero panics (#45590 did not reproduce), exactly 1 segment in every
finish log (36/36 log lines on the final validation run).
- Growing text index path exercised: growing never calls `Finish()`
(periodic commits, background merges retained); the sealed interim text
index does call `Finish()` and gets single-segment.
- Cross-engine benchmark (inverted multi/single-seg vs sort index vs
brute force) asserted identical result sets across all engines; the
bench lands separately with the element-level query work it depends on.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
issue: #51069
This PR fixes ARRAY<FLOAT> brute-force `array_contains` execution by
dispatching FLOAT arrays through `ExecArrayContains<float>` /
`ExecArrayContainsAll<float>` instead of sharing the DOUBLE path.
Summary:
- Split ARRAY<FLOAT> contains dispatch from DOUBLE in
`JsonContainsExpr.cpp`.
- Added a regression covering the reported float literals for
`contains`, `contains_any`, and `contains_all`.
Verification from branch author:
- `clang-format-15` on touched files.
- `git diff --check --
internal/core/src/exec/expression/JsonContainsExpr.cpp
internal/core/src/exec/expression/ExprArrayTest.cpp`.
- Targeted C++ object compile was blocked by unrelated stale local
Conan/generated-header issues.
Verification before PR creation:
- `git diff --check origin/master...buqian/bob/fix-51069-array-float`.
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
Related to #51068
Follow up for #50580
Route Reopen and ApplyLoadDiff through a staged runtime/state committer
so batch load paths can keep IO-heavy work concurrent while serializing
only the final mutation into staged state.
Fix staged index readiness handling by reading/writing staged bitsets
during load, carrying index raw-data facts into the final published
delta, and avoiding mid-reopen runtime publication from JSON ngram index
loading. This also lets field-data loading observe indexes loaded
earlier in the same batch.
Add coverage for loading a vector index for a newly added schema field
to ensure the staged bitsets are resized before use.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #50891
## What changed
- Reserve `IndexEntryReader` future vectors before submitting
read/download tasks so submitted tasks cannot be lost to vector
reallocation failures.
- Drain already stored futures if submission/setup fails before
releasing result buffers, download state, file descriptors, or
positioned writers.
- Cover memory reads, fd downloads, and stream-to-files download paths.
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
## Summary
- Adds a `FilterBitsNode` predicate-to-filtered-bitset fast path for
all-valid predicate results: when the live validity bitmap is all true,
flip the predicate data bitmap directly and skip invalid/UNKNOWN bitmap
materialization.
- Preserves NULL/UNKNOWN semantics for mixed-validity predicate results
by keeping the invalid-row OR path, so only definitely TRUE predicate
rows pass.
- Adds focused `FilterBitsNodeTest` coverage for the all-valid fast path
and the mixed-validity path where invalid/UNKNOWN rows must still be
filtered out.
## Scope note
The fix is intentionally at the `FilterBitsNode` conversion boundary. It
does not add `ColumnVector` null-count metadata or producer-side
validity propagation. Producers that return an all-true validity bitmap
still use the fast path through the runtime `valid.all()` check;
mixed-validity results keep the existing three-valued-logic handling.
issue: #51034
## Benchmark context
Local scalar-benchmark config:
`/home/zilliz/.slock/agents/dde3be7c-41b3-4d6c-9efc-449ebfa3d026/bench_configs/array_contains_0624_min.yaml`
with 8 tests, upload disabled.
| Build | Bundle | Avg QPS | Avg p50 latency |
|---|---:|---:|---:|
| pre-`f0bab30fb3` baseline | `1783071127408` | 13722.301 | 0.077 ms |
| `f0bab30fb3` regression | `1783073484488` | 10277.493 | 0.098 ms |
| current branch with this fix | `1783090388177` | 11078.635 | 0.088 ms
|
Note: the current-branch benchmark is not a perfect apples-to-apples
comparison with the old baseline/regressed commits because later
scalar-expression changes are also included. The local run is meant to
validate the `FilterBitsNode` fast-path direction on the same minimal
`array_contains` workload.
## Verification
- [x] `git diff --check origin/master...HEAD`
- [x]
`PROTOC=/home/zilliz/scalar-benchmark/milvus-worktree-filterbits-fastpath/cmake_build/bin/protoc
ninja -C cmake_build all_tests`
- [x] `source ./scripts/setenv.sh && ./cmake_build/unittest/all_tests
--gtest_filter='FilterBitsNodeTest.*:DetermineExecPathTest.FilterBitsNode_*:ArrayBitmapE2ECheck*.CountFuncTest:ArrayBitmapE2ECheck*.INFuncTest:ArrayBitmapE2ECheck*.NotINFuncTest:Naive/ArrayInvertedIndexTest/*.ArrayContainsAny:Naive/ArrayInvertedIndexTest/*.ArrayContainsAll:ConjunctExprTest.AndKeepsUnknownRowsActiveForFollowingFalse:ColumnVectorTest.*'`
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
