issue: #51253
> Rebased on master after #51254 merged. The `db_name`/`db_id` part of
the original PR is now covered by #51254; this PR is reduced to the two
remaining gaps on the same query-by-id path.
## Problem
When `DescribeCollection` is called with **only a `collectionID`** — no
collection name, no db name (e.g. the HTTP management API on `:9091`) —
two gaps remain after #51254:
1. **Top-level `collection_name` is empty** on both the cached provider
path and the `describeCollectionTask` path. The response is initialized
from the (empty) `request.CollectionName` before the name is resolved,
and never backfilled.
2. **The cached provider discards the caller-provided collection id.**
After deriving the name from the id, it re-resolves the id from that
name via `GetCollectionID(db, name)`. Id-only lookups are cached under
the request db name — empty here — so with same-name collections in
different databases, a concurrent cache refresh can rebind the entry
under the `""` key and the request would silently describe the wrong
collection (the derived name matches, the id does not).
## Fix
- `service_provider.go`:
- Backfill `resp.CollectionName` from the resolved cached schema when
the request carried no name. Requests that pass a name (including
aliases) still echo it unchanged.
- Skip the name→id re-resolution when the caller already supplied the
id. `GetCollectionInfo` already validates the cache entry against the id
(`collInfo.collID != collectionID` → refresh by id), so the
caller-provided id is authoritative end to end. Name+id requests keep
the existing name-precedence behavior.
- Resolve identifiers on local copies instead of rewriting
`request.CollectionName`/`CollectionID` in place — the access log
interceptor and the success metric labels serialize the request after
the handler returns, and previously recorded the resolved values instead
of what the client sent.
- `task.go`: backfill `t.result.CollectionName` from the coordinator
result on the non-cached path.
- `meta_cache.go`: fix a stale comment on `ResolveCollectionAlias` —
`update()` now always keys `collInfo` by the real collection name
(aliases live in the separate alias map), the old comment described
pre-refactor behavior.
- Tests:
-
`TestCachedProxyServiceProvider_DescribeCollection_ByIDFillsNameAndUsesRequestID`:
drives the id-only path; `GetCollectionID` is deliberately not mocked,
so any regression back to re-resolving the id panics the test.
- `TestDescribeCollectionTask_FillsNameFromResultWhenQueriedByID`: same
assertion for the non-cached path.
- Removed the now-unused `GetCollectionID` expectation from the test
added in #51254 (that call no longer happens on the id-only path).
## Follow-up commit: entity-keyed meta cache with a cluster-wide id
index
Reviewing this path surfaced a deeper issue in the proxy meta cache,
fixed in the second commit:
- By-id lookups (`GetCollectionName`, `GetCollectionInfo` with an empty
name) linearly scanned a whole db bucket under the read lock, on every
request even on cache hits.
- The bucket scanned/filled was keyed by the *request* db name — empty
for id-only calls — so entries landed in a bogus `""` bucket: a
collection also cached under its real db was duplicated, and same-name
collections of different databases evicted each other from the single
`""`-keyed slot.
The cache is now entity-keyed instead of request-keyed:
- Fills land under the collection's **actual database** (carried in the
describe response); the `""` bucket no longer exists.
- A cluster-wide `collectionID → entry` index serves by-id lookups in
**O(1)** regardless of the request db — matching rootcoord, which
resolves by-id describes straight from `collID2Meta` without consulting
the db name.
- Name lookups normalize an empty db name to `default`, mirroring
rootcoord's backward-compat normalization
(`meta_table.getCollectionByNameInternal`). This also closes a latent
cross-database mis-hit: a name lookup with an empty db could previously
return whichever same-name collection was last id-cached under the `""`
bucket.
- All removal paths maintain the index (pointer-identity-guarded
unindexing); invalidation sweeps stay exhaustive.
## Verification
- Ran locally against a current master core build: the full
`TestMetaCache*`, `TestCachedProxyServiceProvider*`,
`TestDescribeCollectionTask*`, alias and partition test sets all pass,
plus the **full `internal/proxy` suite** (only the pre-existing
etcd-dependent `TestProxyRpcLimit` fails locally — it needs a live etcd,
unrelated to this change).
- New tests:
`TestCachedProxyServiceProvider_DescribeCollection_ByIDFillsNameAndUsesRequestID`
(id-only path; also asserts the request is not rewritten),
`TestDescribeCollectionTask_FillsNameFromResultWhenQueriedByID`,
`TestMetaCache_ByIDIndexRealDBBucket` (same-name collections in two
databases filled by id: no eviction, entries under real dbs, by-name
reuses by-id fill, every removal path cleans the index),
`TestMetaCache_EmptyDBNameSharesDefaultEntry`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---
## Update: proxy meta cache rebuilt around an id-primary store (2 new
commits)
Following review discussions on invalidation cost and the alias-DDL
races, this PR now also rebuilds the cache (issue: #51533, design doc:
`docs/design-docs/design_docs/20260716-proxy-metacache-id-inverted-index.md`
in this PR):
1. **`fix: forward the pre-alter alias target id in the AlterAlias
expiration`** — rootcoord resolves the pre-alter target under its
database lock and carries it in the new additive
`AlterAliasMessageHeader.old_collection_id`; the ack callback emits a
second expiration entry so proxies evict BOTH AlterAlias targets by id.
Closes the race where a concurrent Describe re-points the proxy's alias
resolution before the expiration arrives, leaving the old target
permanently stale. Older rootcoords (field 0) fall back to the proxy's
hint resolution.
2. **`enhance: rebuild the proxy meta cache around an id-primary
store`** — the primary store is keyed by the cluster-unique collection
id (single source of truth, with a per-database generation); name/alias
resolution become hints validated against the primary on read; partition
caches are keyed by id; fills are ordered against invalidations by a
fill RWMutex (drain in-flight describes before evicting), replacing the
per-collection timestamp floor. Every invalidation becomes O(1) — no
more full-cache scans under the write lock on
Load/Release/Drop/Rename/Alter broadcasts, and
DropDatabase/AlterDatabase becomes a generation bump. Also removes a
latent stale read (alias-keyed partition entries surviving
DropCollection).
Verification: targeted cache suites green (rename, alias re-point under
concurrent describe, drop+recreate with a reused name, db-generation
invalidation, cross-db isolation, gated-mock fill/invalidation
ordering); three negative controls (hint validation / dbGen check / fill
drain disabled) each fail exactly the test guarding them; full-package
`-race` delegated to CI.
---
## Update: simplification series (final form)
Following design review, the cache was iteratively simplified to an
**id-primary store with declared hints** — every mechanism that could be
replaced by an invariant was deleted (net-negative diffs throughout):
- Primary store keyed by the cluster-unique collection id; liveness IS
presence. Name/alias resolution are hints written ONLY together with
their entry (declared aliases) and validated against the primary on
every read — a stale hint can only cost one extra describe, never a
stale read.
- Fill/invalidation ordering via a fill RWMutex (drain in-flight
describes before evicting); the per-collection timestamp floor, database
generations, background GC, alias negative cache, per-partition cache,
and the resolve-and-forget DescribeAlias path are all **deleted**.
Evictions clean everything the entry owns synchronously; **no
invalidation path performs any RPC**.
- Old-rootcoord fallbacks: id-describes without DbName are served
uncached; alias-DDL broadcasts without ids trigger hint resolution plus
a holder scan **gated on that exact fingerprint** (dead code once
rootcoords are upgraded), closing the ghost-alias case with no healing
event.
- **Accepted gaps (WONT-FIX)** are recorded in the design doc §6 —
notably the upgrade-window new-target `Aliases` display lag (the
association exists only at rootcoord; self-heals on first use;
display-only).
See
`docs/design-docs/design_docs/20260716-proxy-metacache-id-inverted-index.md`
for the full design, invariants and trade-offs.
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What
Add `dataCoord.import.l0ImportDisabled` (default **true**) and reject
`ImportV2` requests carrying `l0_import=true` when it is set.
- `pkg/util/paramtable/component_param.go`: new `L0ImportDisabled` param
(default true).
- `internal/datacoord/services.go`: in `ImportV2`, reject `l0_import`
requests when disabled (fail-closed, before resource allocation).
- `configs/milvus.yaml`: documented config entry.
## Why
Importing L0 (delete-only) segments as separate L0 segments during a
binlog/backup import is incompatible with `commit_timestamp`
(two-phase-commit / replication imports). L0 deletes carry original
timestamps while data segments are stamped with `commit_ts`, so
`delete_ts < insert_ts` and the deletes are **silently dropped** — rows
that should be deleted survive. Because this is silent data corruption,
a fail-closed default is safer than relying on callers/operators to
avoid it. See #51247.
The intended replacement is to fold a backup's L0 deletes into
per-segment deltalogs beforehand (offline L0 compaction), after which a
plain restore drops the rows via the existing per-segment delta path.
Operators who still need the legacy behavior can set the config to
`false`.
## Rollout note
This changes the default so that L0 import is rejected. Deployments
relying on legacy L0 import must either (a) migrate their backups to the
deltalog-folded form first, or (b) set
`dataCoord.import.l0ImportDisabled=false`. The default flip is intended
to be coordinated with the availability of the offline-compaction (wash)
tooling.
## Testing
- `paramtable` builds and its unit tests pass locally (pure-Go module).
- Added `TestImportV2_L0ImportDisabledReturnsError` (rejects `l0_import`
when disabled). The datacoord package requires the C++ core to build;
local core is stale in this dev env, so the datacoord test is left to
CI.
issue: #51247
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
issue: #51252
## What
- Add `common.enableDriverPrefetch` with default `false`.
- Wire the config into segcore via initcore and a refresh callback.
- Skip `Driver::PrefetchAsync()` when the switch is disabled.
## Why
This provides a runtime mitigation for regressions caused by query
driver operator prefetch on short scalar queries, while keeping prefetch
configurable for follow-up validation.
## Notes
When disabled, expression execution path determination still runs lazily
on the query thread; this only disables driver-level async prefetch
submission/waiting.
## Tests
- `make generate-yaml`
- `CLANG_FORMAT=/opt/homebrew/opt/llvm@15/bin/clang-format
internal/core/run_clang_format.sh internal/core`
- `make lint-fix`
Signed-off-by: sunby <sunbingyi1992@gmail.com>
issue: https://github.com/milvus-io/milvus/issues/51065
## What changed
This PR switches the default QueryCoord balancer from
`ScoreBasedBalancer` to
`ChannelLevelScoreBalancer` and keeps unknown balancer names aligned
with the
new default fallback.
It also sets the default `queryCoord.channelExclusiveNodeFactor` to `3`,
so
channel exclusive mode is enabled by default only when every channel can
average
at least three RW QueryNodes.
## Why
`ChannelLevelScoreBalancer` improves channel placement, but channel
exclusive
mode constrains segment balancing inside each channel's assigned node
group. If
a replica has too many channels for its QueryNode count, enabling
exclusive mode
can prevent global node-level balancing from smoothing out skewed
channel data.
The default factor of `3` keeps smaller or high-channel-count replicas
on the
score-based node-level balancing path until there is enough QueryNode
capacity
per channel.
## Behavior
With the default configuration, QueryCoord enables channel exclusive
mode only
when:
```text
replica.RWNodesCount() >= len(channels) * 3
```
For example, in an 8 QueryNode replica:
- 2 channels can enable channel exclusive mode.
- 3 channels cannot enable channel exclusive mode and continue using the
score-based balancing fallback.
## Details
- Update `configs/milvus.yaml` balancer and exclusive node factor
defaults
- Update paramtable defaults and default assertions
- Make unknown balancer names fall back to `ChannelLevelScoreBalancer`
- Add balancer factory coverage for default, fallback, explicit
score-based,
and explicit round-robin cases
- Add replica coverage for the default three-nodes-per-channel threshold
- Pin existing tests to legacy `ScoreBasedBalancer` or explicit factors
where
the old behavior is the scenario under test
- Update replica observer tests to provide target channel metadata
explicitly
- Add and refresh the channel exclusive mode design document under
`docs/design-docs`
## Validation
- `git diff --check`
- `source ~/.profile && source scripts/setenv.sh && GOTOOLCHAIN=local go
test -tags dynamic,test -gcflags="all=-N -l"
github.com/milvus-io/milvus/pkg/v3/util/paramtable -run
'TestComponentParam' -count=1 -v`
- Attempted reset-milvus; Docker daemon was unavailable and this
worktree has no
`bin/milvus`.
- Added `internal/querycoordv2/meta` threshold coverage; local execution
is
blocked before assertions by stale Cgo output missing
`C.SegcoreSetPrefetchThreadPoolNum`.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: #51019
## What
A task that fails on a worker (compaction / stats / index / import) is
reset to `Init`/`Retry` and re-dispatched by the DataCoord global task
scheduler on the next ~100ms tick, with no backoff and no cap. When the
failure is persistent — e.g. the task's object-storage reads are
throttled with 429 — one bad task becomes a dispatch storm that keeps
the store saturated so the task can never succeed (observed in
production: ~435k dispatches concentrated on 3 segments in one day,
~330k failing with 429, on a store with a daily GET quota).
This PR delays re-dispatch of a failed task with exponential backoff.
## How
- The scheduler records a per-task-id failure count when
`CreateTaskOnWorker` leaves the task in `Init`/`Retry`, or
`QueryTaskOnWorker` resets it to `Init`/`Retry`.
- The next dispatch is delayed by `dataCoord.taskRetryBackoffInterval`
(default 1s) doubling per consecutive failure up to
`dataCoord.taskRetryBackoffMaxInterval` (default 60s). Both are
dynamically updatable; `0` restores the legacy behavior.
- While a task waits out its backoff it yields its scheduling slot to
other pending tasks (it is skipped and re-queued, not blocking the queue
head).
- The backoff state is cleared when the task finishes, fails terminally
(dropped), or is aborted — so a task that recovers pays nothing on its
next run.
- Returning a task to the queue because no worker has free slots is
*not* counted as a failure.
Applies to every task type the global scheduler handles; no change to
the `Task` interface.
## Tests
- Deterministic unit test for the backoff math: 1s → 2s → 4s, capped at
the configured max; cleared entry ends the backoff; interval=0 disables
the mechanism.
- Scheduling test: a task whose `CreateTaskOnWorker` always fails is
dispatched once, NOT re-dispatched by the ~100ms ticks during its 1s
backoff window (previously ~5 more dispatches), and re-dispatched after
the window elapses.
- Full `internal/datacoord/task` suite green.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
issue: https://github.com/milvus-io/milvus/issues/50816
Add Hugging Face Inference Providers client support for feature
extraction and sentence similarity APIs, and wire it into text embedding
and rerank model providers.
The new provider supports:
- text embedding via feature-extraction
- rerank scoring via sentence-similarity
- Hugging Face router provider selection with hf_provider
- MILVUS_HUGGINGFACE_API_KEY credential fallback
- provider config entries for text embedding and rerank
Also add focused tests for the Hugging Face client, rerank provider, and
paramtable provider docs.
Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
issue: #50811
related: #49435
## What
On the QueryNode, the load-time delta-log replay (`LoadDeltaData`,
`UseLoad=true`) and the online forward delete (`segment.Delete`) both
submit their CGO onto the **same `DynamicPool`** (size `CPUNum`). After
a compaction, the multi-partition replay burst exhausts the pool,
starving online `Insert`/`Delete`. Because `deleteNode.Operate` runs
`ProcessDelete` before `UpdateTSafe`, channel **tSafe freezes for tens
of seconds**, and Strong-consistency queries fail with `channel tsafe
stalled` (#49435).
Evidence from the failing master run (traceID
`b31d6e1626bd4d368961c89e6d4a5652`): `update tsafe` / `start to process
delete` had a ~19–27s gap exactly overlapping a post-compaction segment
load; `milvus_querynode_pool_active_threads{pool_name="DynamicPool"}`
was pegged at **8/8 for ~27s** while `LoadPool` sat idle at **0/40**.
## Changes
Pool isolation + cleanup (`internal/querynodev2/segments`):
- Add a dedicated **MutatePool** for the online-write CGO path
(`segment.Insert`/`segment.Delete`), isolated from load/management work.
- Route load-time **`LoadDeltaData`** replay and **BM25 stats** load
onto **LoadPool**.
- Remove orphaned **WarmupPool**; merge away **BM25LoadPool**.
- Keep **DeletePool** (it is the `DeleteBatch` dispatch pool, not
orphaned).
- Make **DynamicPool**/**DeletePool** size-configurable + hot-reloadable
(`CPUNum * factor`, default `CPUNum`); add `MutatePoolSizeFactor` /
`DynamicPoolSizeFactor` / `DeletePoolSizeFactor`.
Observability:
- Export `capacity / active_threads / queue_depth` for all QueryNode
pools (`milvus_querynode_pool_*`, MutatePool added) and, newly, for the
important DataNode pools (`milvus_datanode_pool_*`: compaction-exec,
index-build, import-exec, io, stats).
## Notes
- This fixes the **resource-exhaustion** half of #49435. The `deleteMut`
lock decoupling (moving the worker RPC out of the lock so it only guards
the in-memory catch-up→publish boundary) is tracked as follow-up.
- The pool "priority" coefficients are pool-size only (no OS scheduling
priority); isolation guarantees online-write admission, not CPU
preemption under full saturation.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What this PR does
Makes the etcd client dial timeout configurable via `etcd.dialTimeout`
(milliseconds, default `5000` — unchanged behavior).
## Why
The dial timeout was hardcoded to 5s in `GetRemoteEtcdClient` /
`GetRemoteEtcdClientWithAuth` / `GetRemoteEtcdSSLClientWithCfg`. On
newly scaled-out nodes, transient network conditions during pod startup
(sidecar proxy not fully ready, DNS propagation delay, or etcd
connection pressure from many nodes starting at once) can push the
blocking dial past 5s, causing the second etcd client created during
`streaming.Init()` to panic with `context deadline exceeded`. Operators
had no way to tune it.
## Changes
- `pkg/util/etcd/etcd_util.go`: add `WithDialTimeout` client option (a
non-positive value keeps the default).
- `pkg/util/paramtable/service_param.go`: add `etcd.dialTimeout` param
and wire it through `EtcdConfig.ClientOptions()`.
- `configs/milvus.yaml`: document the new option.
- `pkg/config/etcd_source.go` + `pkg/config/source.go` +
`pkg/util/paramtable/base_table.go`: thread `dialTimeout` into the
bootstrap config-source client as well (see below).
Every paramtable-based etcd client honors it: meta kv, coordinator/node
services, and the woodpecker WAL builder (the `streaming.Init()` path
from the issue, which already passes `ClientOptions()...`).
## Bootstrap config-source client
The bootstrap config-source client (`pkg/config/etcd_source.go`) is
created before paramtable is fully ready, so it cannot read
`etcd.dialTimeout` from etcd itself. But it does not need to:
`initConfigsFromLocal()` loads the file/env config source *before*
`initConfigsFromRemote()` builds the `EtcdInfo`, and the client's
endpoints/auth/TLS are already read from there. `dialTimeout` is
threaded through the same `EtcdInfo` path, so a value set in
`milvus.yaml`/env applies to this client too. A zero/non-positive value
is a no-op, so unconfigured behavior stays at the 5s default.
The only remaining paths still on the fixed default are CLI/offline
tools and the etcd `HealthCheck` (which uses a short fixed 3s context) —
intended.
issue: #50867
---------
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
issue: #50808
### What this PR does
The datanode object-storage IO pool (`GetOrCreateIOPool`) bounds how
many binlog **download/upload** operations run concurrently. It is
shared by all **compaction** (mix / L0 / sort / clustering) and
**stats** (sort / text index / json key) tasks on a DataNode.
Its size was hard-capped at **32** with a default of only **16**,
regardless of node size:
```go
capacity := paramtable.Get().DataNodeCfg.IOConcurrency.GetAsInt() // default 16
if capacity > 32 { capacity = 32 } // hard-coded since #19892 (2022)
```
On large nodes this pool — not object-storage bandwidth — became the
throughput ceiling for compaction/stats IO (notably after bulk import,
when many sort/compaction tasks contend for it), and an explicit
`ioConcurrency > 32` was silently clamped back to 32.
### Changes
- `dataNode.dataSync.ioConcurrency` now defaults to **auto (`0` →
`CPU*2`)** instead of `16`.
- Removed the hard-coded `> 32` clamp — an explicit value is honored
as-is.
- Extracted `ioPoolCapacity()` and added a unit test.
`CPU*2` is safe because these goroutines are network-IO bound and mostly
blocked on object storage, so concurrency can exceed the CPU count (same
rationale as the existing stats/multiRead pools in this file).
### Notes / follow-up
Downloads currently materialize whole objects in memory, so concurrency
must still be raised with memory in mind — the original `32` cap traces
back to an OOM fix (#33554). Making downloads streaming / range-based
(so concurrency can be raised further without memory blow-up) is a
follow-up. Related: #50452 (storage v3 import slowdown).
### Tests
- `TestIOPoolCapacity` (new): explicit config honored; `0` → `CPU*2`.
- Existing `TestGetOrCreateIOPool`, `TestResizePools` pass.
---------
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
issue: #43638
update woodpecker to [release
v0.1.31](https://github.com/zilliztech/woodpecker/releases/tag/v0.1.31)
* fix(dashboard): add server runtime metrics
* fix(meta): honor configured etcd metadata prefix
* fix: make condition write auto mode fail on verification errors
* feat: split cluster, region, and az topology metadata
update woodpecker to [release
v0.1.32](https://github.com/zilliztech/woodpecker/releases/tag/v0.1.32)
* enhance: gate bucket check/create on createBucket for service mode
* enhance(docker): run server images as root by default
* enhance(docker): bundle wp (wpcli) into server images for zero-config
in-pod ops
* fix(objectstorage): silence idle-spin DEBUG logs in MinioFileWriter
sync loop , **related issue:** #48976
* fix: cut read-path DEBUG log volume (per-read verbosity + idle
tail-read spin)
* feat(quorum): make client-side quorum config runtime-tunable via
Dynamic[T]
* feat(metrics): active-segment node metric + client dashboard panels
* feat: delete-log lifecycle with async local reclaim
update woodpecker to [release
v0.1.33](https://github.com/zilliztech/woodpecker/releases/tag/v0.1.33)
* fix(reader): allow reopening at a truncated-but-not-yet-GC'd position
instead of force-advancing to the truncation point
---------
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
- Replace `common.entryStream.streamBudgetRatio` with refreshable
`common.loadTransientBudgetBytes`, a process-wide transient load budget
shared by scalar index V3 entry streaming and storage v2/v3 field-data
loading. `0` keeps the limit disabled.
- Gate field-data batch reads on the shared budget, split batches by
loading overhead, scale read parallelism from the effective batch
budget, and finalize `GroupChunk` cells before pushing them so Arrow
tables are released promptly.
- Align MCL loading-overhead reservations for storage v1 scalar indexes
and storage v2/v3 field data under one load-transient overhead group,
with array-field overhead and mmap file usage accounted separately.
- Propagate load cancellation into budget waits, field-data batch tasks,
and V3 scalar `IndexEntryReader` stream/file reads so canceled loads do
not stay blocked behind transient memory pressure.
- Update segcore init/config watchers and add C++/Go coverage for budget
sizing, cancellation, field-data batching/finalization, V3 entry
streaming, and the new config.
issue: #49499
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
This reverts PR #50711 (commit ce6cc762fd), which added a `cubefs` note
to `configs/milvus.yaml` documenting CubeFS as a supported cloud storage
provider.
Reverting per maintainer request.
Reverts #50711
issue: #26189
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
## What
Add CubeFS as a supported `cloudProvider` option in Milvus object
storage configuration.
## Changes
- Add `CloudProviderCubeFS = "cubefs"` constant to
`pkg/objectstorage/util.go`
- Add CubeFS case in `NewMinioClient` switch: forces path-style bucket
lookup (`BucketLookupPath`) and static V4 credentials (IAM not supported
by CubeFS ObjectNode)
- Document `cubefs` as a valid `cloudProvider` value in
`configs/milvus.yaml`
- Add integration test `TestNewMinioClientCubeFS` (skipped unless
`CLOUD_PROVIDER=cubefs` env is set, consistent with all other provider
tests)
## Why path-style?
CubeFS ObjectNode requires `S3ForcePathStyle` — it does not support
virtual-host style bucket addressing (`bucket.host/key`).
issue: #26189
Signed-off-by: Ayush KAshyap <kashyap11ayush02@gmail.com>
related: #46442
This commit adds proper RBAC (Role-Based Access Control) support for the
/expr HTTP endpoint, replacing the previous root-only authentication.
Changes:
- Add new rbac.go with CheckPrivilege function for HTTP endpoints
- Support HTTP Basic Auth only (removed non-standard Bearer token
format)
- Integrate with existing Casbin RBAC framework
- Add PrivilegeExpr to GlobalLevelPrivileges and ClusterAdminPrivileges
- Register GetUserRoleFunc callback in meta_cache.go
- Update tests for new RBAC behavior
Security features:
- Authentication required when authorization is enabled
- Root user bypass when RootShouldBindRole is false
- Proper 401/403 status code differentiation
- Integration with privilege result cache
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
## Summary
Decouple QueryCoord task dispatch from the dist pull loop.
When a QueryNode holds more and more segments, `GetDataDistribution`
returns a larger distribution and `pullDist` can take longer. If
dispatch only happens after dist pull finishes, QueryCoord task dispatch
frequency drops together with pull latency, which can hurt scheduling
stability. This PR keeps dispatch cadence independent from dist pull
latency.
issue: #50158
## Changes
- Add `queryCoord.dispatchInterval` with a default value of 500ms.
- Start separate dist pull and task dispatch loops in each dist handler.
- Keep `pullDist` and `handleDistResp` focused on updating distribution
state.
- Remove the old dispatch flag from dist response handling.
## Tests
- `gofumpt -l internal/querycoordv2/dist/dist_controller.go
internal/querycoordv2/dist/dist_controller_test.go
internal/querycoordv2/dist/dist_handler.go
internal/querycoordv2/dist/dist_handler_test.go
pkg/util/paramtable/component_param.go`
- `git diff upstream/master..HEAD --check`
- `go test -count=1 ./util/paramtable`
- `go test -v -count=1 -tags dynamic,test -gcflags="all=-N -l"
-ldflags="-r ${MILVUS_WORK_DIR}/cmake_build/lib -r
${MILVUS_WORK_DIR}/internal/core/output/lib"
./internal/querycoordv2/dist -run
"TestDistControllerSuite/TestSyncAll|TestDispatchLoopUsesDispatchInterval"
-timeout 300s` blocked locally because this worktree has no
`internal/core/output/lib/pkgconfig/milvus_core.pc`.
---------
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
## What this PR does
Removes two unused C++ third-party dependencies from `internal/core` to
reduce dependency count, comprehension cost, and link/build surface.
### rapidjson
Linked into `milvus_core` but has **zero `#include` usage** in the C++
sources (only a stale comment in `common/Json.h`). knowhere v3.0.3 and
milvus-storage do not depend on it. Removes the conan requirement,
`find_package(RapidJSON)`, and the link target.
### OpenDAL backend
Experimental `ChunkManager` backend gated behind `USE_OPENDAL`
(**default OFF**) — released binaries never built it. Removes
`OpenDALChunkManager`, the vendored OpenDAL Rust dependency, and all
`USE_OPENDAL` build plumbing (Makefile / core_build.sh /
3rdparty_build.sh / CMake). The Go-side `"opendal"` storageType value
remains a backward-compatible alias to the standard remote chunk
manager.
### Kept
`simde` is **not** removed — knowhere v3.0.3 requires it
(`find_package(simde REQUIRED)`, used in
`src/index/sparse/codec/varintdecode.c`).
issue: #50551🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
issue: #50229
## What this PR does
Adds configurable request timeouts for external function model providers
(embedding / rerank / highlight), and standardizes the timeout unit from
**seconds → milliseconds** across all providers and clients.
### Changes
- New global config `function.model.timeout_ms` (default `30000`,
refreshable, exported to `milvus.yaml`).
- New per-function param `timeout_ms` which **overrides** the global
setting (`models.ResolveTimeoutMs`). Invalid / non-positive values fall
back to the `30000` default.
- Converted every provider/client timeout argument from `timeoutSec` to
`timeoutMs`; `PostRequest` now applies the value as milliseconds.
- Zilliz gRPC client now applies a per-request deadline via
`requestContext` (previously had no request timeout).
- Bedrock embedding now derives its call context from the request `ctx`
with the configured timeout (was `context.Background()`).
## Tests
- `models`: `TestParseTimeoutMs` (defaults, override, invalid,
zero/negative, case-insensitive key), `TestResolveTimeoutMs` (global
default + per-function override), and `TestZillizClient_RequestContext`
(per-request deadline + metadata).
- `rerank`: provider param validation for invalid / non-positive
`timeout_ms`.
- `paramtable`: `TestFunctionConfig` asserts the new default.
- Updated zilliz client / rerank / embedding / highlight tests for the
ms unit and the new test constructor.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
- Persist role descriptions when creating roles, expose them in role
list/describe results, and support updating descriptions through the
AlterRole path without changing role names.
- Store role descriptions in the role value body while keeping role
names in keys, and tolerate legacy empty role values plus undecodable
stored values by returning an empty description for that role row.
- Reject built-in/default roles and over-limit descriptions before WAL
writes, and share the role-description length validator between proxy
and rootcoord.
related: #50183
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
issue: #49944
## What this PR does
Makes the proxy HTTP API server timeouts configurable so deployments can
tune request, response, and keep-alive behavior without changing code.
### Changes
- New config under `proxy.http`: `readHeaderTimeout` (5s), `readTimeout`
(30s), `writeTimeout` (30s), `idleTimeout` (300s), `maxHeaderBytes`
(1048576). All exported to `milvus.yaml`.
- Apply the settings to the proxy HTTP API server; the management HTTP
server is left unchanged.
## Tests
- `paramtable`: `TestHTTPConfig_Init` (defaults) and
`TestHTTPConfig_TimeoutOverrides` (override behavior) — verified
locally.
- `proxy`: `Test_NewServer_HTTPServer_TimeoutDefaults` /
`Test_NewServer_HTTPServer_TimeoutConfigOverrides` assert the values are
wired into the running `http.Server`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
issue: #47420
## What this PR does
Project-wide migration of raw `fmt.Errorf` / `errors.New` in function
bodies onto
the `merr` framework, plus the Sys-vs-Input error classification and the
machinery it drives (retriability, fine-grained metrics, segcore
unification),
plus the convention docs and a linter that keeps it from regressing.
Scope: storage, proxy, coordinators (root/data/query), query node, data
node,
`pkg/util` & `internal/util`, expression parser, message queue,
streaming, and
misc packages. Bare raw-error usages went from ~3000 to a ~340 allowlist
(package-level sentinels / build-tag / test sites).
---
## How to review this PR
It is large but the vast majority is mechanical. Changes fall into three
tiers;
spend review budget on Part 2 and Part 3.
### Part 1 — Mechanical standardization (low risk, verify by rule)
Each converted call follows one of a small fixed set of rules. To
review, check
that each site obeys the matching rule rather than reading every line:
| Pattern | Rule |
|---|---|
| `fmt.Errorf("...")` originating a new error | →
`merr.WrapErrXxxMsg("...")` with a code matching the failure's meaning |
| Adding context to an existing typed error | → `merr.Wrap(err, "...")`
/ `merr.Wrapf(...)` — **preserves** the inner code (never `WrapErr*Err`,
which overwrites it) |
| Errors inside the streaming subsystem | → `status.New*` factories
(StreamingError), **not** merr — this is the component-internal dialect
(see `docs/dev/error_handling_guide.md`) |
| Low-level / control-flow signal caught by `errors.Is` | → kept as a
package-level `errors.New` sentinel (lowercase, same-package) |
Conventions are documented in `docs/dev/error_handling_guide.md`
(how-to) and
`docs/dev/error_sentinel_convention.md` (rules + audit). A
`gocritic`/`ruleguard`
rule (`rawmerrerror`, in `rules.go`) enforces "no raw `return
errors.New/fmt.Errorf`"
under `make verifiers`.
### Part 2 — Behavior changes (review these closely)
These are the sites where the wire contract or runtime behavior changes,
not just
the source text. Listed by category; representative locations given,
full set in
the diff.
**A. gRPC wire-code shifts: `UnexpectedError(1)/Code 65535` → typed
code.**
Where a handler previously returned a raw error (collapsed to
`Code=65535` on the
wire), it now returns a typed merr, so the client sees a real code. The
most
common shift is to `IllegalArgument(5)/Code 1100` (ParameterInvalid).
Touch
points include datanode task handlers (CreateTask/Query/Drop), proxy
Upsert,
querynode GetMetrics, datacoord CreateIndex, httpserver query-response
builder,
and typeutil schema validation. One code refinement: an index-param
validation
moved `1100` → `1101` (ParameterMissing). **Client/SDK assertions and
any code
that switched on `Code=65535` for these paths must be re-checked** (the
go_client
e2e assertions were already aligned in this PR).
**B. Prometheus `status` label contract change (externally visible).**
The proxy metric's coarse `fail` / `rejected` values are split into
`fail_input` / `fail_system` and `rejected_user` / `rejected_system` (in
`requestutil.ParseMetricLabel`; auth/privilege rejections count as
`rejected_user`), so dashboards can attribute a failure to caller vs
operator.
**Dashboards/alerts querying `status="fail"` must migrate to
`status=~"fail_.*"`, and `status="rejected"` to
`status=~"rejected_.*"`.** The
in-repo Grafana dashboard is already migrated; external dashboards built
on the
old values silently go empty after upgrade. This is the one change that
requires an ops-side migration.
**C. Retriability semantics.**
- C1: `merr.Status(err)` now forces `Retriable=false` when the error is
an
`InputError` — a malformed request can never succeed on blind retry, so
clients
never get the self-contradictory "your input is wrong but you may
retry".
- C2: `retry.Do` short-circuits an `InputError` (non-retriable) — **but
only when
the caller did not pass a `RetryErr` predicate**. The check is an
`if c.isRetryErr != nil { ... } else if InputError { ... }` *mutually
exclusive*
branch (`pkg/util/retry/retry.go`): an explicit `RetryErr` takes
precedence and
bypasses the InputError abort. `retry.Handle` deliberately does **not**
apply
the InputError abort (its callers signal abort via `shouldRetry=false`).
Four
flusher startup callsites that must retry through transient "not ready"
errors
were given explicit `RetryErr` escape hatches.
**D. segcore (C++→Go) error classification.**
A single shared Go-side table (`pkg/util/merr/segcore.go`) maps each
segcore code
to a merr sentinel + InputError/signal category, replacing scattered
hand-written
`if errorCode == ...` switches in the cgo wrappers. **Wire `Code` values
change
for every segcore pass-through error, not just the remapped ones.**
Named
sentinels remap (C++ `2003` → merr `2001`, `2033` → `2002`,
Folly/Knowhere codes
likewise); **all remaining pass-through codes (`2004`–`2043`, previously
surfaced to clients as raw C++ enum values) now serialize as `2000`**
(`ErrSegcore`), with the original C++ code preserved in the `Reason`
text
(`segcoreCode=...`); unknown/future codes collapse to `2000` as well
(pinned by
the `wire_code_projection` test). Transient segcore classes (object
storage /
file IO / OOM / mmap / FieldNotLoaded — 11 codes) now report
`Retriable=true`.
**Any client switching on raw segcore codes in the `2004`–`2043` range
must be
re-checked**; the in-Reason code remains available for diagnostics.
Signal
codes (PretendFinished / FollyCancel) are recognized centrally.
`errors.Is`-based
control flow on these (e.g. scheduler skip/retry) is preserved.
**E. InputError classification (25 sentinels + dynamic marks).**
25 sentinels in `errors.go` carry `WithErrorType(InputError)` (the
Collection /
ResourceGroup / Database families, `ErrIndexDuplicate`,
`ErrParameterInvalid`,
`ErrPrivilegeNotAuthenticated`, `ErrImportFailed`, `ErrQueryPlan`, ...),
plus dynamic
marks for the 8 segcore input codes (ExprInvalid, DimNotMatch,
MetricTypeInvalid, FieldIDInvalid, ...) and
`WrapErrAsInputError`. The widest blast radius is `ErrParameterInvalid`
(1100):
~2335 `WrapErrParameterInvalid*` callsites now classify as input /
non-retriable. Because of C1/C2 this changes retriability for
any path that returns these. **The audit to confirm no transient path
was
mis-marked is the single most important review item** (see Part 3). One
reverse
correction: storage field-stats parsing moved from `ErrParameterInvalid`
(input)
to `ErrDataIntegrity` — a corrupted stored stat is data corruption, not
user
input.
### Part 3 — Known risks & traps (called out proactively)
1. **`merr.Wrap` vs `WrapErr*Err` (code-masking).** `WrapErr*Err` builds
a
`wrappedMilvusError{sentinel: ErrServiceInternal}` whose `code()`
returns the
*outer* sentinel — it overwrites the inner typed code and hides the
`errors.Is`
chain. This is intentional (use it to *deliberately* downgrade), but it
was a
recurring conversion defect; the rule "add context with `merr.Wrap`,
downgrade
with `WrapErr*Err`" is enforced by convention and reviewed across the
diff.
2. **InputError × `retry.Do` blast radius.** Marking a sentinel
`InputError` makes
any `retry.Do(...)` without a `RetryErr` predicate stop retrying it.
Reviewers
should sanity-check that no transient use of the 19 newly-marked
sentinels
(especially `ErrParameterInvalid`) sits inside a retry loop that needed
to keep
spinning. The known flusher cases were handled (see C2).
3. **The ~340 raw-error allowlist.** What remains as bare `errors.New`
is, by
design: package-level sentinels (caught by `errors.Is`), `//go:build
test`
sites, and out-of-band trees (`cmd/`, `tests/`, codegen, walimpls). The
linter
only bans the *direct-return* form; assignment-then-return escapes and
the full
no-exceptions ban are deferred to an AST-based linter (Tier 2,
documented).
4. **segcore C++ second step deferred.** This PR unifies classification
on the Go
side; splitting the dual-semantic C++ codes at the source is a
follow-up.
---
## Validation
- `make verifiers`: Go side clean (gofmt + static-check across modules,
including
the new `rawmerrerror` rule with a 0-hit baseline repo-wide).
- `make test-go`: passing; the one real regression introduced (a
datanode
`invalid_task_type` assertion shifting `1` → `5` from a ParameterInvalid
conversion) was fixed in-tree.
- go_client e2e CreateIndex assertions aligned to the new merr messages.
---------
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add a refreshable proxy.maxCollectionDescriptionLength setting with a
default 1024-byte limit and document it in milvus.yaml.
- Validate collection descriptions during CreateCollection and
AlterCollection PreExecute, including duplicate alter properties, while
leaving existing metadata loading unchanged.
- Document that restore or replication flows that recreate collections
through CreateCollection must satisfy the configured limit or raise it
first.
- Cover byte-length boundaries, CJK byte overflow, create rejection,
alter rejection, refreshability, duplicate alter-property rejection, and
generated YAML consistency in tests.
related: #50173
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
issue: #50206
## Summary
- Support dynamically updating access log configuration (formatters,
methods, writer settings) without restarting, by watching all
`proxy.accessLog` config keys instead of only the enable switch.
- Add RESTful API paths (`/search`, `/query`, `/v2/vectordb/entities/*`)
to the default search/query formatter methods, so RESTful requests also
get detailed access log formatting.
- Fix resource leak: close old `RotateWriter` before re-initializing
when config changes.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
## What was fixed
This fixes import two-phase commit timestamp handling so imported
segments keep the commit timestamp assigned by the import transaction
instead of being made visible at an unrelated later timestamp.
Changes included:
- Preserve import commit timestamps in DataCoord segment metadata.
- Broadcast import commit and rollback through DML channels so
downstream consumers observe the 2PC result consistently.
- Apply delete visibility in sealed segments based on the imported
segment commit timestamp.
- Carry the import transaction DML position needed for commit/rollback
propagation.
- Guard import in replicating clusters behind a disabled-by-default
config switch, and only allow `auto_commit=false` when that switch is
enabled.
- Downgrade only unsupported `GetSalvageCheckpoint` errors for
compatibility with older StreamingCoord implementations.
- Move import jobs to `Committing` before handling per-vchannel commit
acknowledgements, including the race where the vchannel commit is
handled before the DDL ack callback.
- Sync the generated `configs/milvus.yaml` entry for the new import
replication guard.
Fixes#48525
## Test plan
- `source ./scripts/setenv.sh && make SKIP_3RDPARTY=1
build-cpp-with-unittest`
- `make build-go`
- `make static-check`
-
`CLANG_FORMAT=/opt/homebrew/Caskroom/miniconda/base/envs/milvus2/bin/clang-format
make cppcheck`
- `make rustcheck`
- `make fmt`
- `go test ./cmd/tools/config -run TestYamlFile -count=1`
- `go test -tags dynamic,test -gcflags="all=-N -l" ./internal/datacoord
-run 'TestHandleCommitVchannel|TestCommitImportCallback' -count=1`
- Local import commit timestamp tests under
`~/workspace/snippets/tests/test_import_commit_ts`:
- `test_import_delete.py`
- `test_import_auto_commit.py`
- `test_import_basic.py`
- `test_import_mvcc.py`
- `test_import_insert_delete.py`
- `test_import_insert_delete_auto_id.py`
- `test_import_ttl.py`
- Local import + replication tests under
`~/workspace/snippets/tests/test_cdc/import`.
Local Python coverage used StorageV2. StorageV3 needs a separate run
with `common.storage.useLoonFFI=true`.
---------
Signed-off-by: Yihao Dai <yihao.dai@zilliz.com>
relate: #50113
## Summary
Make function runner text tokenization concurrency configurable through
function.analyzer.runner_concurrency, preserving the default value of 8.
BM25 and multi-analyzer BM25 now read the parameter for each
BatchRun/BatchAnalyze call so runtime config changes can take effect
without recreating runners. Also register a config event handler for
function.analyzer.concurrency_per_cpu_core so the shared analyzer pool
resizes dynamically when that setting changes.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
issue: #50098
## Summary
- Apply `common.arrow.ioThreadPoolCoefficient` /
`ioThreadPoolMaxCapacity` to DataNode at startup, mirroring the
QueryNode wiring from #49208/#49623. Without this, the paramtable keys
exist but have no effect on DataNode and the Arrow IO pool stays at
Arrow's hard-coded default of 8.
- Also call `initcore.InitArrowReaderConfig` so the parquet reader
range-coalescing limits (`hole`/`range` size) apply.
- Register paramtable watchers so capacity changes hot-reload without
restart, matching QueryNode behavior.
- Raise `dataCoord.import.fileNumPerSlot` default from 1 to 4 to reduce
per-import slot demand by 4x and leave more arrow IO pool headroom for
concurrent compactions on the same worker.
## Why
See #50098 for the full reproducer and analysis. Briefly: production
observed a 1.5M-row sort compaction taking >1 hour while the worker's
CPU sat at 12% and network at 14% of baseline — all 5 large
SortCompactions on different pods finished at nearly identical wall
times (variance < 1%), the classic shared-pool saturation signature.
Local bench at 50ms RTT with 100 concurrent readers:
| ARROW_IO_THREADS | conc=100 wall_ms | per-task p50_ms |
|---|---|---|
| 8 (default) | 36,917 | 35,908 |
| 32 | 18,174 | 16,074 |
| 64 | 17,320 | 15,702 |
## Test plan
- [x] `go test -count=1 -run TestComponentParam ./util/paramtable/` (pkg
module)
- [x] `go test -tags dynamic,test -gcflags="all=-N -l" -count=1 -run
TestRegisterArrowIOThreadPoolWatchers ./internal/datanode/index/`
- [x] `go test -tags dynamic,test -gcflags="all=-N -l" -count=1 -run
"TestImport" ./internal/datacoord/`
- [ ] CI
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
issue: #48957
- Add `IndexEntryReader::ReadEntryStream`, `GetEntrySize`, and
`HasEntry` for V3 entries.
- Stream plain and encrypted entries through ordered slices with
incremental CRC32c verification.
- Bound transient stream memory with a process-wide entry stream budget.
- Add dynamically refreshable `common.entryStream.streamBudgetRatio` to
tune the shared entry stream budget for current scalar index loading and
later field data loading.
- Use 16MB stream slices by default, aligned with the encrypted slice
size and existing V3 range size.
- Derive the entry stream budget from CPU core count: `CPU cores *
streamBudgetRatio * 16MB`.
- Harden stream error handling:
- keep active futures bounded by the thread pool size
- reject zero or too-small explicit slice sizes
- use overflow-safe slice count calculation
- verify CRC for empty entries
- release budget when callbacks throw or task submission fails
- wait for active workers before returning on abnormal exits
- validate decrypted slice length and include actual/expected CRC in
stream errors
- Document V3 streaming read behavior and shared budget configuration.
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
issue: #50147
This change pulls the Vortex writer-format switch into its own focused
update. The goal is to make the insert data storage format configurable
from Milvus config, so we can turn Vortex on for CI validation without
mixing it with the unrelated AssertInfo cleanup from the previous PR.
The implementation adds dataNode.storage.format and wires it through
paramtable as DataNodeCfg.StorageFormat, with vortex as the current
default for this branch. When storagev2 builds milvus-storage
properties, it now passes the selected value through writer.format,
which matches the updated milvus-storage property contract.
This also updates the C++ writer path to use PROPERTY_WRITER_FORMAT
instead of the old PROPERTY_FORMAT key. That keeps the growing-segment
flush path aligned with the new storage property naming while still
forcing the TEXT flush writer to parquet where that path expects parquet
output.
---------
Signed-off-by: jiaqizho <jiaqi.zhou@zilliz.com>
issue: #49707
- query node scheduler: make read tasks context-aware, use an unbuffered
add path, and cleanup canceled or near-deadline queued tasks before
queue limit checks
- query grouping: cap grouped NQ at 16 and add an NQ merge ratio guard
to avoid merging small queries into much larger groups
- REST timeout: propagate request timeout as a context deadline so
downstream query tasks receive timeout timestamps
- scheduler metrics: add ready NQ plus queue and execution duration
metrics while removing obsolete receive queue configuration
- proxy config: reduce the proxy task queue default to 256
---------
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #49767
## What changed
This patch adds a request-level CPU-factor limiter for QueryNode
delegator post-load work after worker `LoadSegments` returns.
The worker `LoadSegments` fan-out remains unchanged. The limiter covers
the delegator-side follow-up stages before the segment becomes visible:
- `LoadBloomFilterSet`
- `loadBM25Stats`
- `loadStreamDelete`
- distribution update
A refreshable config is added:
```text
queryNode.delegatorPostLoadConcurrencyFactor
```
Default factor is `1`, so the effective concurrency is CPU cores * 1.
## Why
Delegator post-load work can spike CPU when many load requests complete
around the same time. This is especially expensive when delete buffer
rows, loading segments, and historical BF entries multiply the amount of
BF work.
## Validation
- `git diff --check`
- `CONAN_CMD=/Users/wei.liu/Library/Python/3.9/bin/conan make build-cpp`
- `go test -tags dynamic,test
github.com/milvus-io/milvus/pkg/v3/util/paramtable -run
'TestComponentParam' -count=1 -v`
- `go test -gcflags="all=-N -l" -tags dynamic,test
github.com/milvus-io/milvus/internal/querynodev2/delegator -run
'TestDelegatorDataSuite/TestPostLoadLimiter' -count=1 -v -ldflags="-r
${RPATH}"`
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
## Summary
Make the StorageV2 system PK index use the configurable scalar index
warmup policy instead of a hardcoded sync policy. The default remains
sync, while deployments can set
`queryNode.segcore.tieredStorage.warmup.scalarIndex` to disable or async
for the PK index as well.
## Issue
issue: #49665
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
issue: https://github.com/milvus-io/milvus/issues/47403
this pr contains performance optimization for concurrent
CreatePartition:
1. add a version cache on proxy for partition level cache
2. avoid repeated updates of target when there're many CreatePartition
requests.
3. avoid unnecessary copy of meta data.
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
## Summary
- Keep QueryCoord waitQueue as the global priority queue while using
node-bucketed indexing for processQueue so Dispatch(node) scans only
related tasks.
- Index multi-node process tasks under all related nodes, including
MoveTask endpoints and LeaderAction worker/leader nodes.
- Dispatch LeaderAction through the leader executor to match the
SyncDistribution RPC target while preserving action.Node() as the worker
payload node.
- Remove the dist handler checkExecutedFlag ticker and avoid eager
task.String() formatting on non-error stale checks.
- Serialize RemoveByNode with scheduling to avoid re-adding tasks to
processQueue during node-down cleanup races.
Fixes#49512
## Test Plan
- git diff --check upstream/master...HEAD
- gofumpt -l internal/querycoordv2/dist/dist_handler.go
internal/querycoordv2/task/scheduler.go
internal/querycoordv2/task/task_test.go
- go test -v -count=1 -tags dynamic,test -gcflags="all=-N -l"
-ldflags="-r ${MILVUS_WORK_DIR}/cmake_build/lib -r
${MILVUS_WORK_DIR}/internal/core/output/lib"
./internal/querycoordv2/task -run
"TestTask/(TestRemoveByNodeWaitsForSchedule|TestLeaderTaskUsesLeaderExecutor|TestNoExecutor|TestTaskQueueRangePriority|TestNodeTaskQueueNodeBucketing|TestNodeTaskQueueMoveTaskDualNode|TestNodeTaskQueueLeaderActionDualNode|TestNodeTaskQueueRangeByNodePriority)"
-timeout 300s
- make static-check
---------
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
* Optimize metadata and request handling hot paths
* Make replication pending-message queue capacity and max size
configurable instead of hard-coded.
* Batch QueryCoord collection metadata saves with MultiSave to avoid
oversized single writes while still reducing per-key etcd operations.
* Reduce avoidable allocations in DataCoord catalog tests, access log
list formatting, HTTP array joining, and HTTP JSON field extraction.
* Use WAL-specific message ID decoding during flusher recovery.
* Use set-based RootCoord collection-name filtering and structured
collection-not-found errors.
issue: #44452
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
## Summary
- add `search_agg` package in proxy for context build, recursive
aggregation compute, and bucket ordering
- wire `group_by` aggregation through `task_search` and built-in
pipeline (`searchWithAggPipe` + aggregate operator serialization)
- upgrade querynode reduce key extraction for agg multi-field grouping,
and update related proxy/querynode tests
- keep local proto iteration path and local core build job defaults
aligned for this development workflow
## Test plan
- [x] bash
/home/zilliz/hc-claude-projects/hc-milvus-projects/search-aggregation/scripts/verify_baseline.sh
- [x] direct association tests passed
- [x] indirect association tests passed
issue: #49046🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
issue: #49437
Make minio.ssl.tlsCACert default to an empty value instead of the
placeholder /path/to/public.crt. This way Milvus will not set
SSL_CERT_FILE by default, and the process env only gets touched when the
user actually configures a custom CA file. The generated milvus.yaml and
the related paramtable test are updated to match the new default.
Signed-off-by: jiaqizho <jiaqi.zhou@zilliz.com>
## Summary
- Add `proxy.maxIndexParamsSize` to cap create index parameter payloads
before coordinator persistence.
- Reject oversized raw `ExtraParams`, expanded JSON params, and final
merged index params in proxy.
- Cover normal-sized params and oversized payload regressions in
`parseIndexParams` tests.
issue: #48330
## Test plan
- [ ] Not run locally: `go test -tags dynamic,test -gcflags=\"all=-N
-l\" -count=1 ./internal/proxy -run
'Test_parseIndexParams(Rejects|Allows)'` requires generated segcore
C/C++ headers not available in this local checkout.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
UpsertRequest carries `bool partial_update`, and task_upsert may
implicitly promote a request to partial_update=true when any field
carries a FieldPartialUpdateOp != REPLACE. Today the proxy access log
exposes neither — every Upsert line falls through to the generic `base`
formatter and operators triaging a partial-update incident cannot tell
from logs alone whether a given Upsert was full or partial.
Add `$partial_update` to the access-log formatter vocabulary alongside
the existing 27 metrics:
- info.go: register $partial_update in MetricFuncMap; add
PartialUpdate() string to the AccessInfo interface; add the
getPartialUpdate delegator.
- grpc_info.go / restful_info.go: implement PartialUpdate() — type-
assert *milvuspb.UpsertRequest and return fmt.Sprint of
GetPartialUpdate(); return NotAny ("n/a") for any other type.
- configs/milvus.yaml: add a dedicated `upsert` formatter bound to
methods: "Upsert" so the new field shows up out of the box without
polluting the shared `base` formatter.
Implicit promotion (task_upsert.go:1308-1309) is reflected for free:
UnaryAccessLogInterceptor writes the log after handler() returns, and
the *UpsertRequest pointer on GrpcAccessInfo is the same instance the
task mutates, so the type-asserted GetPartialUpdate() reads the promoted
value without any SetPartialUpdate(ctx, …) writeback.
Tests: TestPartialUpdate added to both GrpcAccessInfoSuite and
RestfulAccessInfoSuite (non-Upsert -> "n/a"; PartialUpdate=false ->
"false"; PartialUpdate=true -> "true"). Both suites stay green
end-to-end.
issue: #49347
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
## Summary
This PR ships two related commits:
1. **`enhance: fold calculate_size and write_to_target passes for
variable-length chunk writers`** — refactor String/JSON/Geometry chunk
writers from two Arrow passes (count + cache) to one Arrow pass that
fills a shared `offsets_` member. Eliminates per-row heap allocation (no
more `simdjson::padded_string` copies). `AssertInfo` guards `cursor <=
UINT32_MAX` before each narrowing cast so oversize chunks fail loudly
instead of silently wrapping.
2. **`enhance: configurable byte-size threshold for storage v2 cell
packing`** — replace the hardcoded `4 row groups per cell` constant with
a runtime-configurable byte-size target driven by
`queryNode.segcore.storageV2.cellTargetSizeBytes` (default `4 MiB`,
refreshable).
- At cell-build time, `rgs_per_cell = max(1, target /
avg_row_group_size)`, with cells never crossing file boundaries.
- Paramtable `Formatter` rejects non-positive values and falls back to 4
MiB with a warn log.
- Value pushed into segcore via a new CGO setter
(`SetStorageV2CellTargetSizeBytes`) on QueryNode startup and on
paramtable change, backed by an inline atomic in `GroupCTMeta.h`.
## Why
The fixed `4` constant does not scale across workloads: large row groups
make each cell too fat (coarse eviction, unpredictable memory), small
row groups waste cache slots and increase bookkeeping overhead. A
byte-target keeps cache cell footprint predictable regardless of parquet
row group layout, and gives operators a runtime tuning knob. Default 4
MiB preserves the prior `4 rgs × ~1 MiB/rg` footprint.
## Test plan
- [x] Dedicated `ComputeRowGroupsPerCell` unit tests with hardcoded
expected outputs pin helper behavior independently of the translator
integration tests.
- [x] `GroupChunkTranslatorTest` / `ManifestGroupTranslatorTest` read
`GetCellTargetSizeBytes()` so assertions track production.
- [ ] CI: `ci-v2/build-ut-cov`, `ci-v2/e2e-amd`
- [ ] Manual: toggle `queryNode.segcore.storageV2.cellTargetSizeBytes`
at runtime and confirm newly loaded segments observe the new pack size.
issue: #49204
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ate drift
Built-in RBAC privilege groups were shipping both in
pkg/util/constant.go Go
arrays and in configs/milvus.yaml. Whenever a new privilege was added to
the
Go arrays, the yaml was frequently forgotten; stale yaml then overrode
the fresh Go defaults at
runtime, returning PERMISSION_DENIED on built-in group roles.
This change flips Export=false on the 9 privilege group ParamItems so
generated milvus.yaml no longer ships them as defaults. Runtime falls
through
to DefaultValue (Go constants). Deployments that need to override still
work
unchanged — FileSource still wins over DefaultValue when the user sets
the
key in their own yaml.
Also removes the deprecated
common.security.rbac.overrideBuiltInPrivilegeGroups.enabled
flag (zero production references).
issue: #49167
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
## Summary
Two improvements to BM25 stats memory accounting in idfOracle:
### 1. Per-entry estimate: 80 → 20 bytes/entry
The previous hardcoded `bm25StatsPerEntryBytes = 80` overestimates
`map[uint32]int32` memory by 4-7x. Empirical measurement on Go 1.24
(median of 5 runs, n ≥ 10K to avoid GC noise):
| n | heap delta | bytes/entry | overestimate vs 80 |
|---|---|---|---|
| 10K | 152 KB | 15.22 | 5.2× |
| 100K | 1.22 MB | 12.18 | 6.6× |
| 1M | 19.4 MB | 19.43 | 4.1× |
| 5M | 77.9 MB | 15.58 | 5.1× |
| 10M | 156 MB | 15.58 | 5.1× |
| 50M | 623 MB | 12.47 | 6.4× |
Steady-state cost oscillates between **12 and 19.5 bytes/entry**
depending on the swiss table fill ratio (peak ~19.5 right after a 2×
capacity grow, trough ~12 when near load factor 7/8).
Root cause: the original comment claimed "bucket overhead ~72B per
entry", but Go map buckets/groups hold 8 entries each, so amortized
overhead is ~10 bytes/entry, not 72.
**Fix**: lower the default to 20 (covers measured upper bound 19.5 with
small safety margin) and expose as
`queryNode.idfOracle.bm25StatsBytesPerEntry` (refreshable) for runtime
tuning.
### 2. Skip cgo Charge/Refund when tiered eviction is disabled
When `queryNode.segcore.tieredStorage.evictionEnabled = false` (the
default), the C++ caching layer's resource accounting is inert — no
eviction will be driven by it. The per-load `C.ChargeLoadedResource` /
`C.RefundLoadedResource` cgo calls are pure overhead in this case.
**Fix**: gate `syncResource` and `checkMemoryResource` on the eviction
flag. Skip entirely when eviction is off.
## Impact
- ~75% reduction in cache budget consumed by BM25 stats (when eviction
enabled)
- Eliminates per-segment cgo overhead in default deployments (eviction
off)
- Tunable per-entry estimate without redeploying
## Test plan
- [x] `TestIDFOracle` passes
- [x] `TestBM25Stats_MemSize` passes with the new configurable value
- [x] Verified memory measurement methodology with multiple sample sizes
issue: #46468🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
related: #48587
Provide default userName/password (etcdadmin) for etcd authentication
and add fail-fast validation that panics with a clear message when
etcd.auth.enabled=true but credentials are empty, instead of failing
with an opaque etcd connection error.
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
