issue: #50834
Fix no-index sealed vector raw data warmup selection. When a sealed
segment has no usable built or interim vector index, segcore now treats
raw vector field data as the vector search representation and resolves
its warmup from vector-index policy. Once a built or interim index
exists, raw vector data keeps normal vector-field warmup semantics.
Go load paths continue to pass field warmup unchanged; segcore resolves
the effective policy from current segment state.
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
issue: #51114
JsonStats shredding now loads through
`ManifestGroupTranslator`/`ChunkReader`, but JsonStats files do not have
the normal sealed-segment manifest metadata. The load path reconstructs
the needed column-group view from the existing parquet footer/schema:
column names, Arrow field ids, row counts, and file ordering. This keeps
the current `meta.json` contract unchanged; parquet key-value metadata
is only a fallback for old files that do not have the separate meta
file.
The main follow-up after switching to the manifest translator was
projection behavior. Lazy loading now builds one projected reader per
shredding column, so a predicate on one JSON path does not pull the
whole group; when warmup is enabled it still uses the full column group
so warmup can preload everything together. The added tests cover parquet
files without packed field-list metadata, multi-file ordering, lazy
single-column projection, and warmup full-group projection.
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
issue: #50891
## What changed
- Reserve `IndexEntryReader` future vectors before submitting
read/download tasks so submitted tasks cannot be lost to vector
reallocation failures.
- Drain already stored futures if submission/setup fails before
releasing result buffers, download state, file descriptors, or
positioned writers.
- Cover memory reads, fd downloads, and stream-to-files download paths.
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
issue: #48792
Local index cache directories were derived from index metadata. When
QueryNode releases a segment and then loads the same segment/index
again, the old and new `DiskFileManagerImpl` instances can point to the
same local directory.
That makes two races possible:
- a late writer from the old load can write into the directory used by
the new load;
- cleanup can remove a local directory while async writers for that
directory are still active.
Both races can leave the local disk index state inconsistent during
load/release concurrency.
This PR fixes the problem in two layers:
- Add a process-local generation to each `DiskFileManagerImpl` and
append it to all local index/text/json/ngram/raw-data cache prefixes.
Old and new file managers no longer share the same local directory,
while remote paths stay unchanged.
- Add per-directory RAII write leases. Cleanup closes the directory,
rejects new writers, and defers `RemoveDir` until active writers release
their leases.
The disk index raw-data scratch paths and cleanup are routed through
`DiskFileManagerImpl` so they use the same generation and lease rules.
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
- Replace `common.entryStream.streamBudgetRatio` with refreshable
`common.loadTransientBudgetBytes`, a process-wide transient load budget
shared by scalar index V3 entry streaming and storage v2/v3 field-data
loading. `0` keeps the limit disabled.
- Gate field-data batch reads on the shared budget, split batches by
loading overhead, scale read parallelism from the effective batch
budget, and finalize `GroupChunk` cells before pushing them so Arrow
tables are released promptly.
- Align MCL loading-overhead reservations for storage v1 scalar indexes
and storage v2/v3 field data under one load-transient overhead group,
with array-field overhead and mmap file usage accounted separately.
- Propagate load cancellation into budget waits, field-data batch tasks,
and V3 scalar `IndexEntryReader` stream/file reads so canceled loads do
not stay blocked behind transient memory pressure.
- Update segcore init/config watchers and add C++/Go coverage for budget
sizing, cancellation, field-data batching/finalization, V3 entry
streaming, and the new config.
issue: #49499
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
issue: https://github.com/milvus-io/milvus/issues/48957
## What changed
- Add streaming V3 `LoadEntries` paths for scalar indexes to avoid
reading full index entries into memory before materialization.
- Persist and load auxiliary metadata for sort/string indexes, including
offset maps, valid bitsets, and CSR arrays, so mmap paths can map
metadata directly and memory paths can preallocate buffers.
- Update scalar index resource estimates and related loading tests for
the streaming/mmap load behavior.
## Why
- Reduce peak memory during sealed scalar index load, especially for
large scalar sort, string, and bitmap indexes.
- Avoid recomputing persisted metadata at load time when V3 entries
already contain it.
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
issue: #48957
- Add `IndexEntryReader::ReadEntryStream`, `GetEntrySize`, and
`HasEntry` for V3 entries.
- Stream plain and encrypted entries through ordered slices with
incremental CRC32c verification.
- Bound transient stream memory with a process-wide entry stream budget.
- Add dynamically refreshable `common.entryStream.streamBudgetRatio` to
tune the shared entry stream budget for current scalar index loading and
later field data loading.
- Use 16MB stream slices by default, aligned with the encrypted slice
size and existing V3 range size.
- Derive the entry stream budget from CPU core count: `CPU cores *
streamBudgetRatio * 16MB`.
- Harden stream error handling:
- keep active futures bounded by the thread pool size
- reject zero or too-small explicit slice sizes
- use overflow-safe slice count calculation
- verify CRC for empty entries
- release budget when callbacks throw or task submission fails
- wait for active workers before returning on abnormal exits
- validate decrypted slice length and include actual/expected CRC in
stream errors
- Document V3 streaming read behavior and shared budget configuration.
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
issue: #49616
- Expose Arrow reader range coalescing options through
`common.arrow.reader.holeSizeLimitBytes` and
`common.arrow.reader.rangeSizeLimitBytes`.
- Pass the configured Arrow reader properties into StorageV2
`FileRowGroupReader` paths.
- Propagate the same config into StorageV3 milvus-storage reader
properties (`reader.parquet.prebuffer.*`).
- Update milvus-storage to the official main revision that supports the
reader hole and range size options.
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
issue: #49735
- Preserve upstream scalar filter bitsets when
`common.visibilityFilterEnabled=false`.
- Only mark all rows visible in the disabled visibility-filter path when
`MvccNode` has no upstream filter input.
- Add C++ and Go regression coverage for preserving predicates/filter
bitsets in this path.
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
issue: https://github.com/milvus-io/milvus/issues/47218
Two-stage search evaluates the same sealed-segment filter in Stage 1 for
selectivity and again in Stage 2 before vector search. Cache the Stage 1
full-filter bitmap in ExprResCacheManager so Stage 2 can reuse it
instead of re-evaluating the predicate.
Build full-filter cache keys from the FilterBitsNode currently being
executed instead of carrying an outer vector-plan cache key. Include
entity TTL physical time when TTL filtering is active so cached bitsets
are reused only for equivalent effective predicates.
Text/Phrase match already store sub-expression bitmaps in the same
manager. When full-filter caching is enabled, disable sub-expression
writes so one request path does not cache both the child text bitmap and
the full-filter bitmap. Existing child entries may still be read.
Keep request merging from mixing EnableExprCache modes, otherwise a
Stage 2 two-stage request can merge with a normal search and run with
the wrong cache behavior. Restore FilterOnly and EnableExprCache after
the filter stage so fallback continues with the caller original request
mode.
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
## Summary
- New QueryNode toggle `queryNode.preferFieldDataWhenIndexHasRawData`
(default `false`) makes sealed retrieve serve scalar and vector fields
from resident column data instead of index-backed raw data, and the
loader keeps the column resident alongside the index on load/reload.
- Hot-path short-circuit: when the toggle is off, retrieve does not call
`HasFieldData` or take the segment's `shared_lock`, so the default
retrieve path is unchanged in cost.
- Observability: log at `InitQueryNode` when the toggle is on, calling
out the memory trade-off (field data + index both resident).
## Test plan
- [x] C++ unit:
`RetrieveScalarFieldFromColumnWhenIndexHasRawDataAndPreferFieldDataEnabled`
— seeds the scalar index with values deliberately desynchronized from
the column, then asserts the retrieved value equals the column value
under toggle=true. Positively proves the index branch is skipped rather
than just checking the API returns something.
- [x] Go unit (positive):
`TestLoadWithIndexPreferFieldDataWhenIndexHasRawData` — toggle=on,
asserts `HasFieldData=true`.
- [x] Go unit (negative): `TestLoadWithIndexSkipsFieldDataByDefault` —
toggle=off default, asserts `HasFieldData=false`. Guards the
memory-saving behavior existing deployments rely on.
- [ ] Integration smoke on local standalone.
## Trade-off
Enabling the toggle keeps both the raw field data and the index resident
in memory, roughly doubling the memory footprint for fields whose index
normally supplies raw data. The toggle is off by default, so no existing
deployment is affected.
issue: #49126
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
- Parallelize per-segment `FillTargetEntry` on the MIDDLE thread pool
(matching `FillPrimaryKeys`) and plumb `OpContext*` through the reduce
path for cancellation support.
- Add `AsyncReduceSearchResultsAndFillData` returning `CFuture`,
propagating the folly cancellation token into a per-call `OpContext`.
Switch the Go wrapper to `cgo.Async` so a cancelled query context aborts
the in-flight reduce instead of blocking until completion.
- Use a per-call `OpContext` for `storage_usage` in `FillPrimaryKeys` /
`FillTargetEntry` — sharing `op_ctx` across segments made each segment's
`search_storage_cost_` accumulate every prior segment's bytes, inflating
`GetTotalStorageCost()`.
- Move preflight logic (AssertInfo, input copying) inside the async
lambda so exceptions become failed `CFuture` results rather than
escaping across the C ABI.
issue: #49116
## Test plan
- [x] Add `TestReduceAsyncSuccess` — end-to-end async reduce with valid
results
- [x] Add `TestReduceAsyncCancelled` — pre-cancelled context surfaces
`context.Canceled`
- [ ] Existing reduce tests pass (`go test -tags dynamic,test
./internal/util/segcore/...`)
- [ ] C++ unit tests pass (synchronous C entry point preserved)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
This PR ships two related commits:
1. **`enhance: fold calculate_size and write_to_target passes for
variable-length chunk writers`** — refactor String/JSON/Geometry chunk
writers from two Arrow passes (count + cache) to one Arrow pass that
fills a shared `offsets_` member. Eliminates per-row heap allocation (no
more `simdjson::padded_string` copies). `AssertInfo` guards `cursor <=
UINT32_MAX` before each narrowing cast so oversize chunks fail loudly
instead of silently wrapping.
2. **`enhance: configurable byte-size threshold for storage v2 cell
packing`** — replace the hardcoded `4 row groups per cell` constant with
a runtime-configurable byte-size target driven by
`queryNode.segcore.storageV2.cellTargetSizeBytes` (default `4 MiB`,
refreshable).
- At cell-build time, `rgs_per_cell = max(1, target /
avg_row_group_size)`, with cells never crossing file boundaries.
- Paramtable `Formatter` rejects non-positive values and falls back to 4
MiB with a warn log.
- Value pushed into segcore via a new CGO setter
(`SetStorageV2CellTargetSizeBytes`) on QueryNode startup and on
paramtable change, backed by an inline atomic in `GroupCTMeta.h`.
## Why
The fixed `4` constant does not scale across workloads: large row groups
make each cell too fat (coarse eviction, unpredictable memory), small
row groups waste cache slots and increase bookkeeping overhead. A
byte-target keeps cache cell footprint predictable regardless of parquet
row group layout, and gives operators a runtime tuning knob. Default 4
MiB preserves the prior `4 rgs × ~1 MiB/rg` footprint.
## Test plan
- [x] Dedicated `ComputeRowGroupsPerCell` unit tests with hardcoded
expected outputs pin helper behavior independently of the translator
integration tests.
- [x] `GroupChunkTranslatorTest` / `ManifestGroupTranslatorTest` read
`GetCellTargetSizeBytes()` so assertions track production.
- [ ] CI: `ci-v2/build-ut-cov`, `ci-v2/e2e-amd`
- [ ] Manual: toggle `queryNode.segcore.storageV2.cellTargetSizeBytes`
at runtime and confirm newly loaded segments observe the new pack size.
issue: #49204
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
issue: #49237
## Summary
`SegmentExpr::InitSegmentExpr()` was unconditionally pinning the scalar
index via `PinIndex()` in its ctor. On a tiered-storage
`ChunkedSegmentSealedImpl`, that triggers `CacheSlot::PinCells()` and a
cold fetch even for expressions whose exec path is `TextIndex` /
`PkIndex` / `JsonStats` / `RawData` — none of which ever touch
`pinned_index_`.
Observed impact: `TEXT_MATCH(field, "…")` on an `enable_match=true`
VARCHAR field slows down after an `INVERTED` AUTOINDEX is added on the
same field, because every expr construction cold-loads the scalar index
cell that `TEXT_MATCH` never reads.
This PR restructures `SegmentExpr::DetermineExecPath()` so that
`EnsurePinnedIndex()` runs only once we've fully committed to
`ExprExecPath::ScalarIndex`. Pin happens **iff** `exec_path_ ==
ScalarIndex`.
## Changes
- `InitSegmentExpr()` sets `num_index_chunk_` speculatively from
`segment_->HasIndex()` (cheap bitset, no pin), so
`HasCompatibleScalarIndex()` can decide existence without triggering a
cold fetch.
- New `SegmentInterface::GetJsonFlatIndexNestedPath()` exposes the
`JsonFlatIndex` nested path already stored in segment JSON-index
metadata; `SegmentSealed` overrides it with a mirror of `PinJsonIndex`'s
prefix-matching loop, but without pinning.
- `IsJsonPathCompatible()` is extracted from
`HasCompatibleScalarIndex()` and rewritten to use the new segment API,
removing its dependency on `pinned_index_`.
- Base `DetermineExecPath()` short-circuits to `RawData` when either
check fails and only then calls `EnsurePinnedIndex()`. A post-pin
reconciliation guard also falls back to `RawData` if `HasIndex()` was
speculatively true but `scalar_indexings_` had no entry (rare edge case:
vector/binlog-index-only field or a load transient).
All subclass `DetermineExecPath` short-circuits (`TextMatch` /
`PhraseMatch` / `PkIndex` / `JsonStats`) already return before reaching
the base, so they still never pin. Downstream `pinned_index_[i]`
accessors are gated on `exec_path_ == ScalarIndex` (either explicitly or
via `UseIndexCursor()` / `ProcessIndexChunks` helpers), so the invariant
is preserved without changing dispatch sites.
Observable behavior of `HasCompatibleScalarIndex()` and
`CanUseNestedIndex()` is unchanged: the JSON-path narrowing moves into
`IsJsonPathCompatible()` but the combined decision in
`DetermineExecPath` is identical, and `CanUseNestedIndex()`'s
`pinned_index_.empty()` guard keeps the external return value the same
for every valid state.
## Test plan
- [ ] `make core` + segcore C++ UT (`tests/cpp_test/`), especially
expression-related suites
- [ ] `make test-querynode` / `make test-proxy`
- [ ] Manual: with tiered storage / lazy loading on QueryNode, compare
`TEXT_MATCH(field, "...")` P99 latency on a collection that has
`enable_match=true` VARCHAR + AUTOINDEX `INVERTED` on the same field —
expect no cold-fetch of the scalar INVERTED cell per query
- [ ] Regression: scalar `==` / `IN` / range / `LIKE` / JSON-path / PK
queries still use the correct index path (`exec_path_ == ScalarIndex` →
`pinned_index_` populated)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Expose Arrow's internal IO thread pool capacity as a refreshable
paramtable knob so operators can raise it beyond Arrow's fixed default
of 8, independently of segcore `HIGH/MIDDLE` pools and
`minio.maxConnections`.
- New keys (both refreshable, `common` scope):
- `common.arrow.ioThreadPoolCoefficient` (default `0` → keep Arrow's
built-in default)
- `common.arrow.ioThreadPoolMaxCapacity` (default `0` → no cap)
- Effective size = `coefficient × CPU cores`, clamped by `maxCapacity`
when `> 0`.
- Wired through `init_c.{h,cpp}` + `initcore` + QueryNode paramtable
watcher; hot reloads apply without restart.
## Why
`GetIOThreadPool` runs the async range reads behind `ReadRangeCache`,
which is what actually issues S3 `GetObject` for storage v2 reads.
Milvus never calls `SetIOThreadPoolCapacity`, so this pool sticks at
Arrow's fixed default (`kDefaultNumIoThreads = 8` in arrow 17). Off-CPU
profiling on a QueryNode under heavy storage v2 load shows ~62% of
`HIGH`-pool wait time sitting on `curl_easy_perform -> poll` —
`HIGH`-pool threads blocked inside `ReadRangeCache::Wait` because
Arrow's IO pool cannot dispatch their range reads fast enough.
## Test plan
- [ ] CI: `ci-v2/build-ut-cov`, `ci-v2/e2e-amd`
- [ ] Manual: set `common.arrow.ioThreadPoolCoefficient=4` on a
QueryNode, confirm the `arrow io thread pool capacity set to N` log at
startup and that the pool is resized on subsequent config edits.
- [ ] Manual: re-run the storage-v2-heavy workload that produced the
off-CPU profile and confirm `curl_easy_perform -> poll` share drops from
~62%.
issue: #49207
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
issue: #49025
## Summary
- Extract parquet metadata loading out of `GroupChunkTranslator`'s
constructor into a free `LoadGroupChunkMetadata` function declared in
`segcore/ChunkedSegmentSealedImpl.h`. Returns `{ row_group_meta_list,
parquet_stats_by_field }`.
- When `ENABLE_PARQUET_STATS_SKIP_INDEX` is off (default), the loader
only reads the lightweight `RowGroupMetadataVector` (24 B per row group)
and skips `GetParquetMetadata()` entirely — no `parquet::FileMetaData`
is loaded or retained.
- When on, stats are extracted during the same parallel file pass and
the `FileMetaData` is released before the translator is constructed.
- `GroupChunkTranslator` now accepts
`std::vector<milvus_storage::RowGroupMetadataVector>&&` directly; the
`parquet_file_metas()` / `field_id_mapping()` getters are removed.
- Updated both call sites
(`ChunkedSegmentSealedImpl::load_column_group_data_internal`,
`JsonKeyStats`) and the translator unit test.
## Test plan
- [x] Local build passes (`milvus_segcore`, `milvus_index`)
- [x] Existing `GroupChunkTranslatorTest` updated and compiles
- [ ] Full CI — C++ unit tests + integration tests
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
## Summary
- Await background load futures in `GroupChunkTranslator` and
`ManifestGroupTranslator` cancellation cleanup paths to prevent
use-after-free / dangling futures
- Add `CheckCancellation` calls in `LoadCellBatchAsync` loop for prompt
cancellation
- Fix `tieredStorage.loadingTimeoutMs` default from `0` (immediate
failure) to `-1` (no timeout)
issue: #48854
## Test plan
- [x] Added unit test `CancellationStopsMidBatchPush` verifying
mid-batch cancellation stops loading and raises `FollyCancel`
- [ ] CI passes
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
## Summary
- Add per-pool Prometheus metrics (`pool=search`/`pool=load`) for the
folly `CPUThreadPoolExecutor` pools split in #48675, covering pool_size,
inflight tasks, executing tasks, queue duration, execute duration, and
cancellation counters.
- Add Prometheus metrics for the custom storage `ThreadPool`
(high/middle/low priority): capacity, active threads, idle threads,
queue depth, submitted/completed task counters.
- Fix stale active gauge on Worker thread shutdown/shrink paths and data
race in `SetMetrics()` initial value seeding.
issue: #33132
## Test plan
- [ ] Verify metrics are exposed via `/metrics` endpoint on a running
QueryNode
- [ ] Confirm `internal_cgo_pool_size{pool="search"}` and
`internal_cgo_pool_size{pool="load"}` report correct thread counts
- [ ] Confirm `internal_storage_pool_*` metrics report for
high/middle/low priority pools
- [ ] Verify gauge values stay consistent after pool shrink (idle
timeout)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary
- Add bitpacking-based CompressedInt64PkArray for int64 PK compression
(50-75% memory reduction)
- Defer timestamp/PK index construction to first access via lazy
SystemIndexTranslator (zero S3 I/O on load path)
- Parallelize StorageV2 row group metadata fetching across files
- Skip OffsetOrderedArray for sorted-by-pk segments (save ~12 bytes/row)
issue: #48770
## Changes
### PK Compressed Storage (4 commits)
- `CompressedInt64PkArray`: block-based encoding with per-block min +
variable-width bitpacked deltas, O(1) random access via block index
- `build_offset2pk()` on `InsertRecord`: builds compressed offset→pk for
both sorted and unsorted segments, enabling fast `FillPrimaryKeys` path
- Unit tests covering edge cases: empty, single element, large ranges,
INT64_MAX boundaries
### Segment Loading Optimization (2 commits)
- Prefetch pk and timestamp column chunks before sequential access
- Parallelize row group metadata fetching using HIGH thread pool (fix:
capture loop variable by value to avoid dangling reference)
### Lazy System Index - SystemIndexTranslator (2 commits)
- `TimestampIndexTranslator`: builds timestamp index lazily on first
`mask_with_timestamps` / `bulk_subscript` call
- `PkIndexTranslator`: builds pk2offset / offset2pk lazily on first
`Contain` / `search` call
- V1 compatibility: fallback to `insert_record_` when translator slot is
empty
- Sorted-by-pk: skip OffsetOrderedArray (pk2offset) construction, only
build offset2pk
### Bug Fixes (4 commits)
- Fix `Contain()` returning false for sorted-by-pk segments (missing
binary search path)
- Remove `TimestampData` from `TimestampIndexCell` to prevent DList
deadlock (read timestamp from column directly)
- Remove stale `system_ready_count_++` references; init empty timestamps
for zero-row segments so `is_system_field_ready()` returns true
- Fix dangling reference in parallel metadata fetch lambda
(GroupChunkTranslator + JsonKeyStats)
- Fix same-timestamp delete check in StorageV2 lazy-init path: when
`insert_record_.timestamps_` is empty, use a callback to read insert
timestamps from the timestamp column instead of skipping the check
(which incorrectly allowed deletes with matching timestamps to take
effect)
## Test plan
- [ ] CompressedInt64PkArray unit tests pass
- [ ] StorageV2 lazy system index tests pass
(TestLazySystemIndexesOnUnsortedSegment,
TestLazySystemIndexesOnSortedSegment)
- [ ] TestSameTimestampDeleteNotEffective passes (verifies delete_ts ==
insert_ts is correctly rejected in StorageV2 lazy path)
- [ ] Load collection with int64 PK fields, verify reduced RSS
- [ ] Load sorted-by-pk segments, verify Contain() and delete work
correctly
- [ ] Concurrent segment loading does not deadlock on TimestampIndex
access
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
Co-authored-by: Buqian Zheng <zhengbuqian@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary
- Add `common.visibilityFilterEnabled` toggle to bypass row visibility
filtering (timestamp, delete, and TTL) on querynode. When disabled, all
rows are returned regardless of insert/delete timestamps, improving
performance for workloads that don't need timestamp consistency.
- Add `common.bloomFilterEnabled` toggle to skip bloom filter/PK stats
loading on querynode, reducing memory for workloads without delete or
search-by-id. Delete forwarding falls back to broadcast mode when
disabled.
issue: #48703
## Test plan
- [ ] Unit tests for bloom filter disabled path
(`TestLoadSegmentsWithoutBloomFilter`, `TestLoadWithoutBloomFilter`)
- [ ] Unit tests for nil candidate broadcast in delete forwarding
- [ ] Unit tests for visibility filter bypass with
`visibilityFilterEnabled=false`
- [ ] Verify no nil pointer panics when bloom filter is disabled
- [ ] Integration test with `bloomFilterEnabled=false` and
`visibilityFilterEnabled=false`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
issue: https://github.com/milvus-io/milvus/issues/47902
When indexes are replaced (e.g., during reload) or cleared (e.g., during
segment release), any in-flight async warmup tasks on the old index must
be cancelled first. Without this, warmup tasks could race with the index
teardown, leading to wasted work.
This change:
- Adds cancel_warmup/cancel_and_erase/cancel_and_clear helpers for
scalar, ngram, and json indexes
- Calls CancelWarmup() on old indexes before erase/clear in
LoadScalarIndex, DropIndex, DropJSONIndex, and ClearData
- Deduplicates ngram and json index replacement logic
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
issue: #48507
Update milvus-common to 5983a18 which adds warmup loading timeout,
runtime config hot-reload via TieredStorageConfig, and DList reservation
fixes. Wire through the new warmup_loading_timeout parameter and add
UpdateTieredStorageConfig CGO binding with config change callbacks for
runtime updates.
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
issue: #47902
Integrate milvus-common commit 7b54b6e which adds
CacheWarmupPolicy_Async and a prefetch thread pool for background cache
warmup. This enables segments to be marked as loaded immediately while
cache warming happens asynchronously, reducing load latency.
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
issue: #47872
Replace `ConcurrentVector<Timestamp>` with a lightweight `TimestampData`
class that pins column chunks directly (zero-copy) for StorageV2, and
falls back to owned `std::vector` for StorageV1.
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
issue: #43040
Implement intelligent LoadPriority assignment based on operation
scenarios,
allowing QueryNodes to prioritize critical operations over background
tasks.
**Background:**
Previously, all segment load operations used the same priority, which
could
cause issues when urgent operations (like search requests) competed with
background tasks (like balance) for resources.
**Solution:**
Centralize LoadPriority decisions in Checkers, with each Checker
understanding
its operational context and assigning appropriate priorities:
| Scenario | LoadPriority | Rationale |
|-----------------------|------------------------|------------------------------------|
| Recovery | HIGH | Restore data availability ASAP |
| User Load / Refresh | replica.LoadPriority() | Respect user's
configured priority |
| Stopping Balance | HIGH | Drain nodes quickly for shutdown |
| Normal Balance | LOW | Background optimization |
| Manual Balance | LOW | User maintenance, not urgent |
| Handoff | LOW | Growing→sealed, can wait |
| Index Update | LOW | Background maintenance |
| Stats Update | LOW | Background maintenance |
**Implementation:**
1. SegmentChecker: Determine priority by collection state
- Check if segment exists in currentTarget (recovery vs new segment)
- Check `collection.IsRefreshed()` to distinguish handoff from import
2. BalanceChecker: Add `isStoppingBalance` flag
- Stopping balance uses HIGH to accelerate node draining
- Normal balance uses LOW to avoid interfering with user operations
3. IndexChecker: Always use LOW for index/stats updates
4. Manual Balance: Manual balance uses LOW directly
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
issue: #45999fix: #46000
1. Variable-length types (String, JSON, Array) buffer sharing bug:
StringChunk constructor calculates null bitmap size and offsets position
based on row_nums parameter. When sharing a buffer built for N rows with
a chunk claiming M rows (M != N), the offsets pointer would be wrong.
Fix: use separate buffers for primary cells and tail cell when they have
different row counts.
2. Vector types using wrong Arrow builder: Used BinaryBuilder
(variable-length) but ChunkWriter expects FixedSizeBinaryArray. Fix: use
CreateArrowBuilder(data_type, element_type, dim) which creates
FixedSizeBinaryBuilder.
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
issue: #45999
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
- Core invariant: DefaultValueChunkTranslator assumes a constant
per-value byte size (value_size()) for a field and therefore models
default-value storage as multiple deterministic, fixed-size cells
(num_cells(), cell_id_of(uid) == uid). A single contiguous
primary_buffer_ is built (optionally persisted to mmap) and sliced
per-cell; this invariant enables safe sharing of the same underlying
bytes across multiple Chunk views.
- Removed/simplified logic: per-field direct file_path writes and
duplicated make_chunk write paths were collapsed into a buffer-first
flow (ChunkBuffer + create_chunk_buffer + make_chunk_from_buffer). This
removes redundant per-field mmap/write code and allows multiple Chunk
instances to reuse a shared ChunkBuffer/ChunkMmapGuard instead of
performing independent writes.
- Why no data loss or regression: byte layout, alignment, padding and
per-row content are preserved because create_chunk_buffer implements the
same alignment/padding and writes identical bytes previously produced by
the direct make_chunk path; make_chunk_from_buffer constructs Chunk
views over those same bytes. DefaultValueChunkTranslator uses
deterministic math (total_rows_, num_rows_until_chunk_,
primary_cell_rows_) to slice rows—no rows are dropped, reordered, or
altered. Concrete code paths: callers that used create_group_chunk →
per-field make_chunk now use create_group_chunk → create_chunk_buffer →
make_chunk_from_buffer; DefaultValueChunkTranslator’s get_cells(),
estimated_byte_size_of_cell(), and value_size() produce the same
observable outputs as before for single-cell cases and correct per-cell
outputs for multi-cell cases.
- Enhancement / scope: adds multi-cell default-value chunks and optional
mmap-backed persistence plus shared-memory ChunkBuffer support. Changes
touch DefaultValueChunkTranslator (multi-cell behavior, value_size(),
primary_buffer_, mmap_dir_path_), ChunkWriter (ChunkBuffer,
create_chunk_buffer, make_chunk_from_buffer), and
ChunkedSegmentSealedImpl (per-field mmap decision and mmap_dir_path
propagation); comprehensive tests (mmap/non-mmap, nullable,
multi/single-cell) validate correctness.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
issue: #41435
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added segment resource tracking and automatic memory/disk accounting
during inserts, deletes, loads and reopen.
* Exposed a configuration to set interim index memory expansion rate.
* Added explicit loaded-resource charge/refund operations and Bloom
filter resource lifecycle management.
* **Bug Fixes**
* Ensured consistent memory-size vs. row-count calculations across
segment operations.
* Improved resource refunding and cleanup when segments are released or
closed.
* **Tests**
* Added comprehensive resource-tracking and concurrency tests, plus
Bloom filter accounting tests.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
issue: #41435
- fix: avoid double destruction with placement new
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Pull Request Summary
**Bug Fix: milvus-common dependency update to address double destruction
with placement new**
- **Root Cause Resolution:** Updates the milvus-common library
dependency from commit 4d7781d to fbe5cf7 to fix a double destruction
issue that occurs when placement new is used for object construction.
The upstream fix ensures proper lifecycle management of objects
constructed with placement new semantics, preventing accidental
re-destruction of objects allocated in pre-allocated memory regions.
- **No Logic Changes in Milvus Core:** This PR contains only a
CMakeLists.txt version bump in
`internal/core/thirdparty/milvus-common/CMakeLists.txt`; no Milvus
codebase logic is modified, removed, or simplified. The fix is entirely
contained within the milvus-common library dependency fetched during the
build process.
- **Data Integrity and Behavior Preservation:** No behavior regression
or data loss is introduced. The change is a pure dependency update to
pull in an upstream bug fix. All memory management and object lifecycle
handling improvements are confined to the milvus-common library,
remaining transparent to Milvus core operations.
- **Issue Resolution:** Addresses issue #41435 by integrating the
corrected milvus-common version that prevents double destruction bugs
occurring in code paths that use placement new for memory-efficient
object construction.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
issue: #45486
Introduce row group batching to reduce cache cell granularity and
improve
memory&disk efficiency. Previously, each parquet row group mapped 1:1 to
a cache
cell. Now, up to `kRowGroupsPerCell` (4) row groups are merged into one
cell.
This reduces the number of cache cells (and associated overhead) by ~4x
while
maintaining the same data granularity for loading.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Refactor**
* Switched to cell-based grouping that merges multiple row groups for
more efficient multi-file aggregation and reads.
* Chunk loading now combines multiple source batches/tables per cell and
better supports mmap-backed storage.
* **New Features**
* Exposed helpers to query row-group ranges and global row-group offsets
for diagnostics and testing.
* Translators now accept chunk-type and mmap/load hints to control
on-disk vs in-memory behavior.
* **Bug Fixes**
* Improved bounds checks and clearer error messages for out-of-range
cell requests.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
issue: #46443
Add `Forbidden: true` to all tiered storage related parameters to
prevent runtime configuration changes via etcd. These parameters are
marked as refreshable:"false" but that tag was only documentation - the
actual prevention requires the Forbidden field.
Without this fix, if tiered storage parameters are modified at runtime:
- Go side would read the new values dynamically
- C++ caching layer would still use the old values (set at InitQueryNode
time)
- This mismatch could cause resource tracking issues and anomalies
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
issue: #45486
This commit refactors the chunk writing system by introducing a
two-phase
approach: size calculation followed by writing to a target. This enables
efficient group chunk creation where multiple fields share a single mmap
region, significantly reducing the number of mmap system calls and VMAs.
- Optimize `mmap` usage: single `mmap` per group chunk instead of per
field
- Split ChunkWriter into two phases:
- `calculate_size()`: Pre-compute required memory without allocation
- `write_to_target()`: Write data to a provided ChunkTarget
- Implement `ChunkMmapGuard` for unified mmap region lifecycle
management
- Handles `munmap` and file cleanup via RAII
- Shared via `std::shared_ptr` across multiple chunks in a group
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
issue: #41435
After introducing the caching layer's lazy loading and eviction
mechanisms, most parts of a segment won't be loaded into memory or disk
immediately, even if the segment is marked as LOADED. This means
physical resource usage may be very low. However, we still need to
reserve enough resources for the segments marked as LOADED. Thus, the
logic of resource usage estimation during segment loading, which based
on physcial resource usage only for now, should be changed.
To address this issue, we introduced the concept of logical resource
usage in this patch. This can be thought of as the base reserved
resource for each LOADED segment.
A segment’s logical resource usage is derived from its final evictable
and inevictable resource usage and calculated as follows:
```
SLR = SFPIER + evitable_cache_ratio * SFPER
```
it also equals to
```
SLR = (SFPIER + SFPER) - (1.0 - evitable_cache_ratio) * SFPER
```
`SLR`: The logical resource usage of a segment.
`SFPIER`: The final physical inevictable resource usage of a segment.
`SFPER`: The final physical evictable resource usage of a segment.
`evitable_cache_ratio`: The ratio of a segment's evictable resources
that can be cached locally. The higher the ratio, the more physical
memory is reserved for evictable memory.
When loading a segment, two types of resource usage are taken into
account.
First is the estimated maximum physical resource usage:
```
PPR = HPR + CPR + SMPR - SFPER
```
`PPR`: The predicted physical resource usage after the current segment
is allowed to load.
`HPR`: The physical resource usage obtained from hardware information.
`CPR`: The total physical resource usage of segments that have been
committed but not yet loaded. When one new segment is allow to load,
`CPR' = CPR + (SMR - SER)`. When one of the committed segments is
loaded, `CPR' = CPR - (SMR - SER)`.
`SMPR`: The maximum physical resource usage of the current segment.
`SFPER`: The final physical evictable resource usage of the current
segment.
Second is the estimated logical resource usage, this check is only valid
when eviction is enabled:
```
PLR = LLR + CLR + SLR
```
`PLR`: The predicted logical resource usage after the current segment is
allowed to load.
`LLR`: The total logical resource usage of all loaded segments. When a
new segment is loaded, `LLR` should be updated to `LLR' = LLR + SLR`.
`CLR`: The total logical resource usage of segments that have been
committed but not yet loaded. When one new segment is allow to load,
`CLR' = CLR + SLR`. When one of the committed segments is loaded, `CLR'
= CLR - SLR`.
`SLR`: The logical resource usage of the current segment.
Only when `PPR < PRL && PLR < PRL` (`PRL`: Physical resource limit of
the querynode), the segment is allowed to be loaded.
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
issue: #43040
This patch introduces a disk file writer that supports Direct IO.
Currently, it is exclusively utilized during the QueryNode load process.
Below is its parameters:
1. `common.diskWriteMode`
This parameter controls the write mode of the local disk, which is used
to write temporary data downloaded from remote storage.
Currently, only QueryNode uses 'common.diskWrite*' parameters. Support
for other components will be added in the future.
The options include 'direct' and 'buffered'. The default value is
'buffered'.
2. `common.diskWriteBufferSizeKb`
Disk write buffer size in KB, only used when disk write mode is
'direct', default is 64KB.
Current valid range is [4, 65536]. If the value is not aligned to 4KB,
it will be rounded up to the nearest multiple of 4KB.
3. `common.diskWriteNumThreads`
This parameter controls the number of writer threads used for disk write
operations. The valid range is [0, hardware_concurrency].
It is designed to limit the maximum concurrency of disk write operations
to reduce the impact on disk read performance.
For example, if you want to limit the maximum concurrency of disk write
operations to 1, you can set this parameter to 1.
The default value is 0, which means the caller will perform write
operations directly without using an additional writer thread pool.
In this case, the maximum concurrency of disk write operations is
determined by the caller's thread pool size.
Both parameters can be updated during runtime.
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
issue: #40942
Add simde package, which can make porting SIMD code to other
architectures much easier.
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
