issue: #51252
## What
- Add `common.enableDriverPrefetch` with default `false`.
- Wire the config into segcore via initcore and a refresh callback.
- Skip `Driver::PrefetchAsync()` when the switch is disabled.
## Why
This provides a runtime mitigation for regressions caused by query
driver operator prefetch on short scalar queries, while keeping prefetch
configurable for follow-up validation.
## Notes
When disabled, expression execution path determination still runs lazily
on the query thread; this only disables driver-level async prefetch
submission/waiting.
## Tests
- `make generate-yaml`
- `CLANG_FORMAT=/opt/homebrew/opt/llvm@15/bin/clang-format
internal/core/run_clang_format.sh internal/core`
- `make lint-fix`
Signed-off-by: sunby <sunbingyi1992@gmail.com>
- Replace `common.entryStream.streamBudgetRatio` with refreshable
`common.loadTransientBudgetBytes`, a process-wide transient load budget
shared by scalar index V3 entry streaming and storage v2/v3 field-data
loading. `0` keeps the limit disabled.
- Gate field-data batch reads on the shared budget, split batches by
loading overhead, scale read parallelism from the effective batch
budget, and finalize `GroupChunk` cells before pushing them so Arrow
tables are released promptly.
- Align MCL loading-overhead reservations for storage v1 scalar indexes
and storage v2/v3 field data under one load-transient overhead group,
with array-field overhead and mmap file usage accounted separately.
- Propagate load cancellation into budget waits, field-data batch tasks,
and V3 scalar `IndexEntryReader` stream/file reads so canceled loads do
not stay blocked behind transient memory pressure.
- Update segcore init/config watchers and add C++/Go coverage for budget
sizing, cancellation, field-data batching/finalization, V3 entry
streaming, and the new config.
issue: #49499
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
issue: #48957
- Add `IndexEntryReader::ReadEntryStream`, `GetEntrySize`, and
`HasEntry` for V3 entries.
- Stream plain and encrypted entries through ordered slices with
incremental CRC32c verification.
- Bound transient stream memory with a process-wide entry stream budget.
- Add dynamically refreshable `common.entryStream.streamBudgetRatio` to
tune the shared entry stream budget for current scalar index loading and
later field data loading.
- Use 16MB stream slices by default, aligned with the encrypted slice
size and existing V3 range size.
- Derive the entry stream budget from CPU core count: `CPU cores *
streamBudgetRatio * 16MB`.
- Harden stream error handling:
- keep active futures bounded by the thread pool size
- reject zero or too-small explicit slice sizes
- use overflow-safe slice count calculation
- verify CRC for empty entries
- release budget when callbacks throw or task submission fails
- wait for active workers before returning on abnormal exits
- validate decrypted slice length and include actual/expected CRC in
stream errors
- Document V3 streaming read behavior and shared budget configuration.
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
issue: #49616
- Expose Arrow reader range coalescing options through
`common.arrow.reader.holeSizeLimitBytes` and
`common.arrow.reader.rangeSizeLimitBytes`.
- Pass the configured Arrow reader properties into StorageV2
`FileRowGroupReader` paths.
- Propagate the same config into StorageV3 milvus-storage reader
properties (`reader.parquet.prebuffer.*`).
- Update milvus-storage to the official main revision that supports the
reader hole and range size options.
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
Related to #46358
Route sealed segment JSON key stats loading through the LoadDiff-based
load/reopen path so initial loads and stats updates share the same
self-managed segment workflow.
Add segcore diff support for JSON stats load, replace, and drop
operations, including conversion to LoadJsonKeyIndexInfo, mmap
configuration, feature flag checks, and LoadDiff application in
ChunkedSegmentSealedImpl. Update QueryCoord/QueryNode to trigger reopen
for JSON stats updates and keep LocalSegment JSON stats metadata
synchronized after load/reopen.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #49398
Bump Go module references from pkg/v2 and milvus-proto/go-api/v2 to
pkg/v3 and milvus-proto/go-api/v3 so the client tracks the Milvus 3.x
release line.
This prepares the repository for the upcoming 3.x.y release by aligning
imports, module dependencies, and proto API references with the new
major-version module paths.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
## Summary
- New QueryNode toggle `queryNode.preferFieldDataWhenIndexHasRawData`
(default `false`) makes sealed retrieve serve scalar and vector fields
from resident column data instead of index-backed raw data, and the
loader keeps the column resident alongside the index on load/reload.
- Hot-path short-circuit: when the toggle is off, retrieve does not call
`HasFieldData` or take the segment's `shared_lock`, so the default
retrieve path is unchanged in cost.
- Observability: log at `InitQueryNode` when the toggle is on, calling
out the memory trade-off (field data + index both resident).
## Test plan
- [x] C++ unit:
`RetrieveScalarFieldFromColumnWhenIndexHasRawDataAndPreferFieldDataEnabled`
— seeds the scalar index with values deliberately desynchronized from
the column, then asserts the retrieved value equals the column value
under toggle=true. Positively proves the index branch is skipped rather
than just checking the API returns something.
- [x] Go unit (positive):
`TestLoadWithIndexPreferFieldDataWhenIndexHasRawData` — toggle=on,
asserts `HasFieldData=true`.
- [x] Go unit (negative): `TestLoadWithIndexSkipsFieldDataByDefault` —
toggle=off default, asserts `HasFieldData=false`. Guards the
memory-saving behavior existing deployments rely on.
- [ ] Integration smoke on local standalone.
## Trade-off
Enabling the toggle keeps both the raw field data and the index resident
in memory, roughly doubling the memory footprint for fields whose index
normally supplies raw data. The toggle is off by default, so no existing
deployment is affected.
issue: #49126
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
This PR ships two related commits:
1. **`enhance: fold calculate_size and write_to_target passes for
variable-length chunk writers`** — refactor String/JSON/Geometry chunk
writers from two Arrow passes (count + cache) to one Arrow pass that
fills a shared `offsets_` member. Eliminates per-row heap allocation (no
more `simdjson::padded_string` copies). `AssertInfo` guards `cursor <=
UINT32_MAX` before each narrowing cast so oversize chunks fail loudly
instead of silently wrapping.
2. **`enhance: configurable byte-size threshold for storage v2 cell
packing`** — replace the hardcoded `4 row groups per cell` constant with
a runtime-configurable byte-size target driven by
`queryNode.segcore.storageV2.cellTargetSizeBytes` (default `4 MiB`,
refreshable).
- At cell-build time, `rgs_per_cell = max(1, target /
avg_row_group_size)`, with cells never crossing file boundaries.
- Paramtable `Formatter` rejects non-positive values and falls back to 4
MiB with a warn log.
- Value pushed into segcore via a new CGO setter
(`SetStorageV2CellTargetSizeBytes`) on QueryNode startup and on
paramtable change, backed by an inline atomic in `GroupCTMeta.h`.
## Why
The fixed `4` constant does not scale across workloads: large row groups
make each cell too fat (coarse eviction, unpredictable memory), small
row groups waste cache slots and increase bookkeeping overhead. A
byte-target keeps cache cell footprint predictable regardless of parquet
row group layout, and gives operators a runtime tuning knob. Default 4
MiB preserves the prior `4 rgs × ~1 MiB/rg` footprint.
## Test plan
- [x] Dedicated `ComputeRowGroupsPerCell` unit tests with hardcoded
expected outputs pin helper behavior independently of the translator
integration tests.
- [x] `GroupChunkTranslatorTest` / `ManifestGroupTranslatorTest` read
`GetCellTargetSizeBytes()` so assertions track production.
- [ ] CI: `ci-v2/build-ut-cov`, `ci-v2/e2e-amd`
- [ ] Manual: toggle `queryNode.segcore.storageV2.cellTargetSizeBytes`
at runtime and confirm newly loaded segments observe the new pack size.
issue: #49204
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Expose Arrow's internal IO thread pool capacity as a refreshable
paramtable knob so operators can raise it beyond Arrow's fixed default
of 8, independently of segcore `HIGH/MIDDLE` pools and
`minio.maxConnections`.
- New keys (both refreshable, `common` scope):
- `common.arrow.ioThreadPoolCoefficient` (default `0` → keep Arrow's
built-in default)
- `common.arrow.ioThreadPoolMaxCapacity` (default `0` → no cap)
- Effective size = `coefficient × CPU cores`, clamped by `maxCapacity`
when `> 0`.
- Wired through `init_c.{h,cpp}` + `initcore` + QueryNode paramtable
watcher; hot reloads apply without restart.
## Why
`GetIOThreadPool` runs the async range reads behind `ReadRangeCache`,
which is what actually issues S3 `GetObject` for storage v2 reads.
Milvus never calls `SetIOThreadPoolCapacity`, so this pool sticks at
Arrow's fixed default (`kDefaultNumIoThreads = 8` in arrow 17). Off-CPU
profiling on a QueryNode under heavy storage v2 load shows ~62% of
`HIGH`-pool wait time sitting on `curl_easy_perform -> poll` —
`HIGH`-pool threads blocked inside `ReadRangeCache::Wait` because
Arrow's IO pool cannot dispatch their range reads fast enough.
## Test plan
- [ ] CI: `ci-v2/build-ut-cov`, `ci-v2/e2e-amd`
- [ ] Manual: set `common.arrow.ioThreadPoolCoefficient=4` on a
QueryNode, confirm the `arrow io thread pool capacity set to N` log at
startup and that the pool is resized on subsequent config edits.
- [ ] Manual: re-run the storage-v2-heavy workload that produced the
off-CPU profile and confirm `curl_easy_perform -> poll` share drops from
~62%.
issue: #49207
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
- Add `common.visibilityFilterEnabled` toggle to bypass row visibility
filtering (timestamp, delete, and TTL) on querynode. When disabled, all
rows are returned regardless of insert/delete timestamps, improving
performance for workloads that don't need timestamp consistency.
- Add `common.bloomFilterEnabled` toggle to skip bloom filter/PK stats
loading on querynode, reducing memory for workloads without delete or
search-by-id. Delete forwarding falls back to broadcast mode when
disabled.
issue: #48703
## Test plan
- [ ] Unit tests for bloom filter disabled path
(`TestLoadSegmentsWithoutBloomFilter`, `TestLoadWithoutBloomFilter`)
- [ ] Unit tests for nil candidate broadcast in delete forwarding
- [ ] Unit tests for visibility filter bypass with
`visibilityFilterEnabled=false`
- [ ] Verify no nil pointer panics when bloom filter is disabled
- [ ] Integration test with `bloomFilterEnabled=false` and
`visibilityFilterEnabled=false`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Check CStatus return value from InitLocalChunkManagerSingleton to
propagate initialization errors instead of silently ignoring them.
issue: #48549
Signed-off-by: sunby <sunbingyi1992@gmail.com>
## Summary
- Replace the hardcoded thread pool max threads limit (16) with a
configurable parameter `common.threadCoreCoefficient.maxThreadsSize`,
default 16, only effective when greater than 0
- Add missing Resize watchers for middle and low priority thread pools
- When `maxThreadsSize` changes dynamically, update the limit first then
resize all pools to ensure correct ordering
issue: #48378
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
## Summary
- HashTable now dynamically rehashes (doubles capacity) when load factor
exceeds 7/8, fixing crash when GROUP BY cardinality > ~1792
- Added configurable `queryNode.segcore.maxGroupByGroups` (default 100K)
to cap total groups and prevent OOM on both C++ (per-segment HashTable)
and Go (cross-segment agg reducer) layers
- Added 4 C++ unit tests covering rehash basic/correctness, max groups
limit, and multiple rehash rounds
issue: #47569
## Test plan
- [ ] C++ unit tests: `--gtest_filter="*HashTableRehash*:*MaxGroups*"`
- [ ] E2E: GROUP BY aggregation with >2K unique values should succeed
- [ ] E2E: Set `queryNode.segcore.maxGroupByGroups` to small value,
verify clear error message
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Layer 2 (OpenSSL FIPS) changes:
- Add programmatic FIPS activation via OSSL_LIB_CTX_load_config in
boring_enabled.go (gated by //go:build boringcrypto)
- Add openssl-fips.cnf with fips + default providers and
default_properties = fips=yes
- Use absolute .include path for fipsmodule.cnf — OpenSSL resolves
relative .include from the process working directory, not the config
file's directory, causing silent FIPS provider load failure
- Add RAND_bytes probe after config load to verify the FIPS provider is
truly functional (EVP_default_properties_is_fips_enabled only checks the
property string, not whether the provider loaded)
- Dockerfiles: add openssl fipsinstall + OPENSSL_MODULES env var
- Log OpenSSL FIPS status from C++ via
EVP_default_properties_is_fips_enabled
Layer 1 (Go BoringCrypto) changes:
- Add GOEXPERIMENT=boringcrypto build flag (conditional on
MILVUS_FIPS_ENABLED=ON)
- Add boringEnabled() build-tagged functions for startup logging
s2n-tls upgrade:
- Override s2n 1.4.1 (from aws-c-io) to 1.6.0 in conanfile.py. s2n 1.4.1
only detects FIPS via the legacy OPENSSL_FIPS define (not set by OpenSSL
3.x). s2n 1.6.0 adds EVP_default_properties_is_fips_enabled() detection
so s2n enters FIPS mode and uses RAND_bytes() through the FIPS provider.
See also: #48202, #48301
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
### **User description**
issue: https://github.com/milvus-io/milvus/issues/44011
Updated the DescribeCollection functionality to exclude the namespace
field from the response schema. This change ensures that these fields
are not returned in the collection description.
___
### **PR Type**
Bug fix, Enhancement
___
### **Description**
- Filter namespace field from DescribeCollection response schema
- Add support for PartitionKeySortCompaction in compaction task handling
- Fix lambda capture issues in C++ expression evaluation code
- Add comprehensive test coverage for namespace field filtering
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["DescribeCollection Request"] --> B["Filter Fields"]
B --> C["Exclude Dynamic Fields"]
B --> D["Exclude Namespace Field"]
C --> E["Response Schema"]
D --> E
F["Compaction Task"] --> G["Create/Complete Task"]
G --> H["Support PartitionKeySortCompaction"]
H --> I["Task Execution"]
```
<details><summary><h3>File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Bug
fix</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>service_provider.go</strong><dd><code>Filter namespace
field from DescribeCollection response</code>
</dd></summary>
<hr>
internal/proxy/service_provider.go
<ul><li>Added import for <code>common</code> package to access
<code>NamespaceFieldName</code> constant<br> <li> Modified field
filtering logic in <code>DescribeCollection</code> to exclude
<br>namespace field alongside dynamic fields<br> <li> Updated filter
condition to check both <code>IsDynamic</code> flag and field name
<br>equality</ul>
</details>
</td>
<td><a
href="https://github.com/milvus-io/milvus/pull/46512/files#diff-e72e29bf5e62a9c5a797c0045e0d6f427d5c49e587c848f68e81ceabaa3e2a0c">+2/-1</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>task.go</strong><dd><code>Exclude namespace field from
task execution</code>
</dd></summary>
<hr>
internal/proxy/task.go
<ul><li>Updated <code>describeCollectionTask.Execute</code> to filter
out namespace field <br>from response schema<br> <li> Modified condition
to skip fields that are either dynamic or have <br>namespace field
name<br> <li> Ensures namespace field is not included in collection
description <br>results</ul>
</details>
</td>
<td><a
href="https://github.com/milvus-io/milvus/pull/46512/files#diff-b4aff1bcd223cde92858085e1be028292b0899b08c9664aa590a952f01106e3d">+1/-1</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>Expr.cpp</strong><dd><code>Fix lambda capture and
remove duplicate includes</code>
</dd></summary>
<hr>
internal/core/src/exec/expression/Expr.cpp
<ul><li>Removed duplicate include of <code>expr/ITypeExpr.h</code><br>
<li> Removed unused include of <code>fmt/format.h</code><br> <li> Fixed
lambda capture issues in <code>SetNamespaceSkipIndex</code> function by
using <br>pointer capture<br> <li> Changed from reference capture
<code>[&]</code> to explicit pointer and value <br>captures to avoid
dangling references<br> <li> Extracted namespace field ID and value as
const variables for safer <br>lambda capture</ul>
</details>
</td>
<td><a
href="https://github.com/milvus-io/milvus/pull/46512/files#diff-1051ba05ef883a88cf621aa1ca7ba4f3b433be44f574877c1888f39eed42cb50">+16/-16</a>
</td>
</tr>
</table></td></tr><tr><td><strong>Tests</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>service_provider_test.go</strong><dd><code>Add test for
namespace field filtering</code>
</dd></summary>
<hr>
internal/proxy/service_provider_test.go
<ul><li>Added imports for <code>schemapb</code> and <code>common</code>
packages<br> <li> Created new test
<br><code>TestCachedProxyServiceProvider_DescribeCollection_FilterNamespaceField</code><br>
<li> Test verifies namespace and dynamic fields are filtered while user
<br>fields are preserved<br> <li> Uses mock cache to simulate collection
metadata with namespace field</ul>
</details>
</td>
<td><a
href="https://github.com/milvus-io/milvus/pull/46512/files#diff-47cdd329529238ca17e2a5c9f66f548cf941e19d332203a099d0fa8a9e67dd47">+65/-0</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>task_test.go</strong><dd><code>Add namespace field
filtering test case</code>
</dd></summary>
<hr>
internal/proxy/task_test.go
<ul><li>Added comprehensive test
<br><code>TestDescribeCollectionTask_FilterNamespaceField</code><br>
<li> Test creates schema with namespace field as partition key and
dynamic <br>metadata field<br> <li> Verifies both namespace and dynamic
fields are excluded from describe <br>result<br> <li> Confirms user
fields like id and fvec are properly included</ul>
</details>
</td>
<td><a
href="https://github.com/milvus-io/milvus/pull/46512/files#diff-ab14021f2a0170ad1c610628871f98c5cd4386ce95efe7af94b4509bdea4ccf7">+90/-0</a>
</td>
</tr>
</table></td></tr><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>compaction_inspector.go</strong><dd><code>Support
PartitionKeySortCompaction in task creation</code>
</dd></summary>
<hr>
internal/datacoord/compaction_inspector.go
<ul><li>Added
<code>datapb.CompactionType_PartitionKeySortCompaction</code> to switch
case <br>in <code>createCompactTask</code><br> <li> Now handles
PartitionKeySortCompaction alongside MixCompaction and
<br>SortCompaction<br> <li> Routes PartitionKeySortCompaction to
<code>newMixCompactionTask</code> handler</ul>
</details>
</td>
<td><a
href="https://github.com/milvus-io/milvus/pull/46512/files#diff-1c884001f2e84de177fea22b584f3de70a6e73695dbffa34031be9890d17da6d">+1/-1</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>meta.go</strong><dd><code>Handle
PartitionKeySortCompaction in mutation completion</code>
</dd></summary>
<hr>
internal/datacoord/meta.go
<ul><li>Added
<code>datapb.CompactionType_PartitionKeySortCompaction</code> to switch
case <br>in <code>CompleteCompactionMutation</code><br> <li> Routes
PartitionKeySortCompaction to
<code>completeSortCompactionMutation</code> <br>handler<br> <li> Ensures
proper mutation completion for partition key sort compaction
<br>type</ul>
</details>
</td>
<td><a
href="https://github.com/milvus-io/milvus/pull/46512/files#diff-1b1c74d883d233d2457813f0708bd3a3418102555615d6abca5e04b9815e6c37">+1/-1</a>
</td>
</tr>
</table></td></tr></tbody></table>
</details>
___
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
- Core invariant: schemapb.CollectionSchema.EnableNamespace is the
authoritative per-collection flag for namespace behavior — it is
persisted on Collection models, copied into DescribeCollection
responses, and used at runtime to derive
datapb.SegmentInfo.IsNamespaceSorted and
datapb.CompactionSegmentBinlogs/CompactionSegment.IsNamespaceSorted
(i.e., per-collection EnableNamespace → per-segment IsNamespaceSorted).
- Removed/simplified logic: eliminated property-based namespace toggles
and partition-key-sort special-cases (TriggerTypePartitionKeySort /
TriggerTypeClusteringPartitionKeySort and
IsPartitionKeySortCompactionEnabled) and removed the dual namespace-skip
API in expression evaluation (SetNamespaceSkipFunc /
SetNamespaceSkipIndex); routing now reuses existing Mix/Sort/Clustering
handlers and uniformly treats IsNamespaceSorted alongside IsSorted in
compaction, index, and inspection checks.
- Why no data loss or behavior regression: wire compatibility preserved
(SegmentInfo field number unchanged when renaming
is_partition_key_sorted → is_namespace_sorted), binlog/segment IDs,
manifest paths and storage versions are unchanged, and compaction
requests only add/pass an extra boolean flag — no binlog content,
identifiers, or storage layout are modified; DescribeCollection now only
changes user-facing filtering of schema fields (internal stored schema
unchanged) and unit tests were added for filtering to prevent
regressions.
- Bug fix (milvus-io/milvus#44011): fixes leakage of internal
namespace/meta fields in DescribeCollection by filtering out fields with
IsDynamic==true and the Namespace field name (common.NamespaceFieldName)
and by propagating EnableNamespace through
DescribeCollectionTask.Execute; tests added in
internal/proxy/service_provider_test.go and internal/proxy/task_test.go
validate the fix.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
Related to #46595
Remove the EnableStorageV2 config option and enforce StorageV2 format
across all write paths including compaction, import, write buffer, and
streaming segment allocation. V1 format write tests are now skipped as
writing V1 format is no longer supported.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #45597
The streaming node currently cannot use GPU resources and does not need
to perform initialization.
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
issue: #44596
- querynode already init the storage v2 and segcore, so streamingnode
should not do this again.
- It also fix the gcp object storage access denied.
Signed-off-by: chyezh <chyezh@outlook.com>
