issue: #51497
After a successful WAL switch via `POST /management/wal/alter`, `GET
/management/config/get?keys=mq.type` kept returning the pre-switch value
with `"source":"RuntimeSource"` until the coordinator restarted, even
though the ack callback had already persisted the new mq.type into etcd
(and linearizably refreshed the local EtcdSource). External operators
polling that endpoint to confirm switch completion never saw the new
value.
### Root cause
`InitAndSelectWALName()` saved the startup-resolved WAL name into the
config manager's **runtime overlay**, and `Manager.GetConfig` serves the
overlay with top priority — permanently shadowing the EtcdSource for
every reader in the process. The overlay write existed only to feed
`ProcessImmutableConfigs`, which pins `GetConfig`'s value into etcd
create-if-absent at first boot (without it, the literal `"default"`
would be persisted).
### Fix
Replace the overlay bridge with an explicit per-key **renderer** on the
persistence path, so mq.type reads fall through to the etcd source and
naturally follow a WAL switch:
- `ProcessImmutableConfigs(renderers map[string]func(raw string)
string)`: a renderer converts a placeholder raw value (mq.type's literal
`"default"`) into the concrete value at create-if-absent persist time.
It never runs when etcd already holds the key (a completed switch
survives restart), and it also covers the key-absent-from-all-sources
case (raw is passed as `""`). Generic: any future immutable config with
a placeholder value can plug in its own renderer.
- `InitAndSelectWALName()` no longer writes the runtime overlay and
returns the resolved `message.WALName`; the coordinator startup passes a
mq.type renderer backed by that name.
- `dependency.HealthCheck` resolves the literal `"default"` to the
actually-enabled MQ before probing (this gap predates the overlay Save:
since #36822, deployments running with `mq.type: default` probed nothing
and always reported unhealthy).
This also fixes two other readers of the shadowed value:
- the `HandleAlterWAL` same-type short-circuit: re-posting the completed
target no longer re-broadcasts, and a rollback to the original MQ is no
longer silently swallowed with "already configured";
- the proxy MQ health check (`/_cluster/dependencies`) no longer keeps
probing the old MQ forever after a switch.
### Behavior note
`MQCfg.Type.GetValue()` now follows etcd at runtime instead of being
frozen at process start: the coordinator sees a switch immediately
(linearizable refresh in the ack callback); other nodes converge within
one EtcdSource poll cycle (default `refreshInterval` 5s). Audited
readers: `mustSelectMQType` / `MustSelectWALName` resolve `"default"`
themselves and are startup-scoped; the AlterWAL idempotency check and
the proxy health check are exactly the readers this change fixes.
### Verification
- New tests (all watched failing first): renderer converts placeholder
before first persist / existing etcd value never overwritten nor
re-rendered / no-renderer keys unchanged / key absent from all sources
still pinned via renderer (`pkg/config`); `InitAndSelectWALName` leaves
mq.type served from its original source, not `RuntimeSource`
(`streamingutil/util`); `HealthCheck("default")` resolves to the enabled
MQ (`dependency`).
- Full package runs green locally: `pkg/config`,
`internal/util/streamingutil/util`, `internal/util/dependency` (with
`-tags dynamic,test -gcflags="all=-N -l"`).
- `cmd/roles`, `internal/coordinator`, `internal/distributed/streaming`
type-checked via `go vet` locally (no current-master segcore build on
this machine); relying on CI for full compilation. A live-cluster alter
→ config/get end-to-end pass is still pending — the switch-visibility
chain (etcd write → `GetConfig` returns the new value) is covered at
unit level by the existing `TestAlterConfigsInEtcd`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## What
Enforce a function–field binding principle for function DDL (add / drop
/ alter function on an existing collection), so a function and its
output field stay coupled and already-stored vectors can never be
silently invalidated by DDL.
BM25 and MinHash follow the strict binding: add/drop only via the
unified `add_function_field` / `drop_function_field`, output field
always coupled. **TextEmbedding is a temporary carve-out** — its legacy
`AddCollectionFunction` / `DropCollectionFunction` paths are retained,
so attach-over-existing-field and detach are still possible for it,
because `add_function_field` embedding backfill is not yet implemented.
It will be folded into the unified path in a follow-up. So the "always
coupled" invariant currently holds for BM25/MinHash, not TextEmbedding.
## Changes
- **AlterCollectionSchema invariant**: reject standalone add-function (a
function must be added together with its new output field), reject
detaching a function without dropping its output field, and reject
function cascade (a new function's input being another function's
output).
- **Legacy RPCs**: `AddCollectionFunction` / `DropCollectionFunction`
are rejected for BM25/MinHash (must use `add_function_field` /
`drop_function_field`) and retained only for TextEmbedding (see
carve-out). `AlterCollectionFunction` is kept and gains the whitelist
below. In the alter and legacy-add paths, function field IDs are always
re-derived from field names (never trusted from the request), so a
request cannot inject an unrelated field ID that a later
`drop_function_field` would delete.
- **alter_function whitelist**: only connection/runtime params may
change (TextEmbedding: `url`, `credential`, `timeout_ms`,
`max_client_batch_size`, `region`, `location`, `projectid`, `user`);
BM25/MinHash have no alterable params. Function identity (type, name,
input/output fields) and output-shaping params (`dim`, `model_name`,
`endpoint` — the TEI model identity, `normalize`, `truncate*`, prompts,
...) are immutable. Params are normalized (keys lowercased, duplicate
keys rejected) so a crafted duplicate key cannot bypass the diff.
- **drop_function_field**: allow any output-producing function (dropping
needs no backfill), removing the previous BM25/MinHash-only restriction.
## Not in scope / follow-up
- Extending `add_function_field` to TextEmbedding (post-creation add
would require backfilling every existing row through the external
embedding model) — deferred; folding the TextEmbedding legacy paths into
the unified binding follows this.
- MinHash input-field analyzer immutability lives on a different DDL
path (`AlterCollectionField`); tracked as a separate follow-up.
## Breaking changes
- `DropCollectionFunction` on a **MinHash** function (detach — remove
the function but keep its output field) is no longer supported; use
`drop_function_field` instead (drops the function together with its
output field). If a released version supported MinHash detach, migrate
any such call to `drop_function_field`.
issue: #51348🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Optimize the k-way merge by advancing the heap root in place instead of
popping and pushing it for every candidate. Use typed deduplication keys
for INT64 and VARCHAR primary keys, including element-level and group-by
search paths.
Skip late materialization when the search plan has no non-primary target
fields, while preserving the primary field for mixed output fields
required by proxy reranking.
Ensure submitted segcore output-field tasks are joined on exceptional
paths and add correctness and benchmark coverage for the optimized merge
implementations.
https://github.com/milvus-io/milvus/issues/51315
Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
issue: #51493
### What
#50221 split the `status` label values of `milvus_proxy_req_count` /
`milvus_proxy_grpc_latency` in place. Released in **v2.6.19**, that
silently zeroes every existing dashboard panel and alert rule matching
`status="fail"` / `status="rejected"` — an alert that stops firing
rather than erroring.
This keeps the classification but moves it onto its own dimension,
restoring the status domain to what <= v2.6.18 emitted:
```
status: success | fail | rejected | retry | total | abandon (as before v2.6.19)
cause: user | system | cancel | na (new)
```
| v2.6.19 / v2.6.20 | this PR |
|---|---|
| `status="fail_input"` | `status="fail", cause="user"` |
| `status="fail_system"` | `status="fail", cause="system"` |
| `status="rejected_user"` | `status="rejected", cause="user"` |
| `status="rejected_system"` | `status="rejected", cause="system"` |
| `status="cancel"` | `status="fail"` or `"rejected"`, `cause="cancel"`
|
| `success` / `retry` / `total` / `abandon` | unchanged, `cause="na"` |
### Why a label instead of new status values
- **Old queries work unchanged and count each request once.**
`status="fail"` is again every hard failure; Prometheus aggregates over
`cause` for free. That rollup is what a label dimension *is* — the split
forces every consumer who just wants "how many failures" to hand-write
`status=~"fail_.*"`.
- **Cardinality is unchanged.** `cause` is functionally dependent on the
outcome, so the realized `(status, cause)` pairs are exactly the series
the split already produces. Additive, not multiplicative.
- **New consumers lose nothing**: alert on `status="fail",
cause="system"`; route `cause="user"` to the owning application team.
The alternative — emitting old and new `status` values side by side
during a transition — was rejected: it defers the break rather than
removing it (the old values must still be dropped eventually, forcing
the same migration later), and meanwhile double-counts every request in
unfiltered aggregations.
`cancel` is folded into `cause` for the same reason: before v2.6.19
client cancellations were counted as `fail` (cancel in the response
status) or `rejected` (cancel at the interceptor), so promoting it to a
`status` value quietly makes `status="fail"` under-count versus older
releases. As a cause it stays excludable via `cause!="cancel"` without
redefining `status`.
### Also in this PR
- The 7 `snapshot_impl` sites that emitted a bare `fail` with no
classification now report their real cause via `failMetricLabel(err)`
(e.g. `snapshot_metadata_uri is required` is `cause="user"`, not a
system fault). Their `status` is unchanged.
- `deployments/monitor/grafana/milvus-dashboard.json`: the "Faild
Request Rate" panel goes back to `status="fail"` — a verbatim revert of
what #50221 changed, which is the compatibility claim demonstrated.
- Dev docs and stale comments that named the old label values.
### Verification
- `TestParseMetricLabelStatusDomainIsStable` pins the status domain, so
re-splitting it fails CI rather than shipping.
- Adding a label makes every `WithLabelValues` site arity-sensitive **at
runtime, not compile time** — a missed site panics in production. Unit
tests do not reach all of them (coverage shows
`GetRestoreSnapshotState`, `ListRestoreSnapshotJobs`, `PinSnapshotData`,
`UnpinSnapshotData` at 0%), so all **58 emit sites were verified
statically via AST**, not only the ones tests happen to hit.
- Passing: `pkg/metrics`, `pkg/util/requestutil`, `pkg/util/merr`,
`internal/distributed/proxy` (incl. `httpserver`, which covers the REST
emit path), and the `internal/proxy` snapshot / metric-label tests.
`golangci-lint` clean on every touched package.
- Pre-existing and unrelated: `service_test.go` bind failures (`listen
tcp :19529: address already in use`) reproduce identically on unmodified
master — a local process holds the port.
### Rollout
Should be cherry-picked to 2.6 so the window in which the fine-grained
`status` values exist stays confined to v2.6.19–v2.6.20. Users who
already adopted the new values on those two releases need a one-time
query change (`status="fail_system"` -> `status="fail",
cause="system"`); that is a known, bounded set, and preferable to
leaving every pre-2.6.19 dashboard silently broken.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Problem
Milvus supports online collection-schema mutations — add/drop field,
add/drop function, alter field, enable dynamic field — through several
RootCoord DDL paths. Existing segments are written under the committed
schema, so a mutation that reinterprets an existing field (its data
type, nullability, or structural role) or relabels existing data as a
function output makes already-written binlogs read incorrectly. Before
this change there was no single authoritative check that a proposed
schema is a **safe structural successor** of the committed one;
validation was scattered and incomplete across DDL paths, so an unsafe
mutation could reach the WAL and silently corrupt or misinterpret
existing data.
## What this PR does
Adds an authoritative, validation-only, fail-closed gate
`ValidateSchemaEvolution(oldSchema, newSchema)`, invoked in every
RootCoord schema-mutating callback **before** any side effect (analyzer
reservation, bound-index allocation, WAL broadcast). It rejects:
- in-place field reinterpretation — name, data type, element type,
nullability, or structural role (primary / partition / clustering key,
autoID, dynamic);
- relabeling an existing field as a function output — function outputs
must use brand-new fields, never existing data;
- unsafe field additions — a newly added field may not arrive already
marked primary / partition / clustering key, function output, or
dynamic;
- unsafe drops of protected fields;
- more than one clustering key;
- a dynamic field that is also a function output;
- corruption of the reserved `max_field_id` watermark.
Property-only / TTL updates, analyzer rollback, BM25 / MinHash
`add_function_field`, and legacy function Drop / detach paths are
preserved unchanged.
## Scope
Non-function structural invariants only. Function-graph validation
(single producer, output binding, alter-time immutability, cascade) is
intentionally **out of scope here and owned by #51360**. This PR does
not include schema propagation, Proxy barriers, DML draining, backfill /
readiness, index promotion, TextEmbedding enablement, protobuf,
configuration, or C++ changes.
## Validation
- `go test -tags dynamic,test -gcflags='all=-N -l'
./internal/util/schemautil -count=1` — PASS (the gate's own unit tests)
- `git diff --check origin/master...HEAD` — PASS
issue: #51340
---------
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Keep raw search distances through segment search, iterator output, index
query, and chunk merging so ranking/reduce order is not affected by
`round_decimal`.
Apply `round_decimal` only when producing the final search response,
after reduce/rerank/order-by has already determined result order.
Returned scores keep the requested precision without changing TopK
selection.
issue: #50347
---------
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
issue: #51372
### Problem
`MergeOp.buildResultArrays` builds a zero-hit chunk's `$id` array with a
hardcoded Int64 builder
(`operator_merge.go:713-721` before this change), while chunks that do
have hits are typed from the
first actual ID (`buildInt64Results` / `buildStringResults`). All
per-query chunks then go into
`AddColumnFromChunks`, which builds one `arrow.NewChunked` column out of
them.
For a **VarChar-PK collection** where **one query of an nq >= 2 batch
returns zero hits** while
another returns hits, those chunks mix `utf8` and `int64` and arrow
panics:
```
panic: invalid: arrow/array: mismatch data type int64 vs utf8
```
`MergeOp` sits at chain index 0, so every request that goes through a
function rerank chain — plain
search with a rerank function as well as hybrid search — can hit it. The
proxy has no recovery
interceptor on its gRPC chains
(`internal/distributed/proxy/service.go:295-307`, `416-424`) and no
`recover()` in `internal/proxy/`, so the panic takes the proxy process
down.
The input side was never affected: `collectRRFScores` /
`collectWeightedScores` read IDs row by row,
so an empty chunk is simply iterated zero times. Only the output side
was broken.
### Fix
Resolve the `$id` arrow type once per merge, from the first input that
actually carries IDs
(`resolveIDType`), and build the zero-hit chunk with that type.
Zero-row inputs are deliberately skipped when resolving: an empty result
carries no ID type of its
own and is materialized as an empty Int64 column regardless of the
collection's PK type
(`importEmptyIDs`, converter.go). When *every* input is empty, Int64 is
kept — every chunk is then
empty as well, so the column is self-consistent and the rerank
operator's `allEmpty` early-return
short-circuits before it anyway.
An unexpected arrow ID type now returns a `merr` error instead of
silently producing an Int64 array.
### Testing
- `TestMergeWithEmptyChunkVarCharIDs` — single input, VarChar IDs,
`topks=[2,0]`; asserts the merged
`$id` column stays `utf8` with an empty second chunk. Written first and
observed panicking with
`mismatch data type int64 vs utf8` on unmodified master.
- `TestMergeMultiInputEmptyLegVarCharIDs` — hybrid-search shape: one leg
entirely empty, the other
with a zero-hit query.
- The existing `TestMergeWithEmptyChunk` (Int64) still passes, pinning
the unchanged Int64 path.
- `./internal/util/function/...` green, including `-race`; `gofmt`
clean.
### Note
Found while reviewing #51156 (fix for #50969). This panic is **not**
introduced by that PR —
`operator_merge.go` is byte-identical between master and its head, and
the repro above does not touch
the lines it changes. #51156 does widen the reachable surface, though:
before it, a single-shard
zero-hit result failed earlier in the converter with `unsupported ID
type` and never reached
`MergeOp`; after it, that input builds a DataFrame and walks into this
panic.
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
issue: #51252
## What
- Add `common.enableDriverPrefetch` with default `false`.
- Wire the config into segcore via initcore and a refresh callback.
- Skip `Driver::PrefetchAsync()` when the switch is disabled.
## Why
This provides a runtime mitigation for regressions caused by query
driver operator prefetch on short scalar queries, while keeping prefetch
configurable for follow-up validation.
## Notes
When disabled, expression execution path determination still runs lazily
on the query thread; this only disables driver-level async prefetch
submission/waiting.
## Tests
- `make generate-yaml`
- `CLANG_FORMAT=/opt/homebrew/opt/llvm@15/bin/clang-format
internal/core/run_clang_format.sh internal/core`
- `make lint-fix`
Signed-off-by: sunby <sunbingyi1992@gmail.com>
issue: #51192
design doc:
docs/design-docs/design_docs/20260708-xgboost-function-chain.md
Add native xgboost FunctionChain expression support for L0 rerank with
FileResource-backed UBJ models.
This change includes:
- xgboost FunctionChain expression registration, parameter validation,
and execution
- FileResource-based UBJ model discovery and local path resolution
- lazy model loading with singleflight, lease/refcount lifecycle
protection, and stale eviction on FileResource sync
- cgo bridge for Arrow C Data based batch prediction
- native C++ UBJ model parser and predictor for supported tree models
- runtime-disabled stub for builds without cgo and with_xgboost
- validation for unsupported params, output modes, feature count
mismatch, invalid models, unsupported objectives, unsupported boosters,
multiclass models, multi-target leaf vectors, and unsupported input
column types
- C++ unit tests, Go tests, native parity tests, and Python client L0
E2E tests
- xgboost FunctionChain design document
L2 rerank support is intentionally deferred because Proxy does not yet
support FileResource sync and local resolution.
Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
issue: #51104
companion SDK PR (merge first): milvus-io/pymilvus#3670
## Summary
`add_function_field` previously accepted a vector output field without
any index meta; bump-schema-version compaction then stalled forever (no
segment index task can be created for a field with no index defined, and
`GetQueryVChanPositions` never hands off unindexed compacted-to segments
— the fail-closed chain is silent). This PR makes the index meta a
mandatory, atomically-bound part of the DDL:
- **Mandatory at every layer**: SDK requires `index_params`; proxy and
rootcoord reject vector function-output fields without an explicit
`index_type` strictly before the broadcast (no silent AUTOINDEX).
- **create_index-aligned materialization**: proxy normalizes the params
with the same logic as `create_index` (name rules via
`validateIndexName`, checker existence, `CheckTrain` incl. dimension
filling, function-type defaults such as `bm25_k1/bm25_b/bm25_avgdl`) and
writes them back into the request; rootcoord prepare allocates index
id/name, rejects name conflicts and unknown index types, and serializes
a complete `indexpb.FieldIndex` into the new
`AlterCollectionMessageUpdates.bound_field_indexes` WAL field.
- **Atomic apply**: the alter-collection ack callback commits the schema
and then applies the bound index by dispatching a synthetic
`CreateIndexMessage` to datacoord's existing `createIndexV2AckCallback`
(same pattern as `cascadeDropFieldIndexesInline`). The callback is
replayed until success across crashes and is fully idempotent (all ids
come from the message body), so `DDL success ⇒ index meta exists` holds
with no partial terminal state.
- **Reuse promotion**: `ValidateIndexParams`/`CheckDuplidateKey` moved
from datacoord to `internal/util/indexparamcheck`; shared
`ExpandIndexParams` / `FillFunctionOutputIndexParams` /
`PrepareFunctionOutputIndexParams` helpers now back both the
create_index path and this DDL.
- **Dead fan-out removed**: the ordinary `CreateIndex` broadcast is
reverted to control-channel-only; the vchannel fan-out had no consumer
and wrote one dead WAL entry per vchannel per index creation.
- **No milvus-proto change**: the request already carried
`FieldInfo.index_name`/`extra_params` (previously unused); this feature
activates them.
- e2e suites updated to the new semantics:
`test_add_function_field_feature.py` rewritten (explicit `index_params`
everywhere, post-DDL `create_index` blocks removed since the bound index
conflicts with a second distinct index per existing semantics, new
negative cases for the mandatory checks); the external-table negative
case updated as well.
## Verification
- New/updated unit tests: rootcoord DDL callbacks
(mandatory/AUTOINDEX/unknown-index-type rejections, bound-index
materialization + ack-callback application via a recording fake), proxy
task validation (invalid index name, normalization write-back), shared
helper tests.
- End-to-end on a live cluster (StorageV3 + bump compaction enabled):
request without params rejected; index meta visible with
create_index-aligned params immediately after the DDL returns; bump
compaction rewrote 2000-row sealed segments and the bound index built on
the results (the exact chain that previously stalled); BM25 search over
pre-DDL rows succeeds once the compacted segments are loaded.
## Limitations (follow-ups)
- Live propagation of the new index meta to already-loaded QueryNode
delegators is intentionally out of scope; a loaded collection observes
the bound index through the segment load path (e.g. reload /
compacted-segment load). Tracked as a separate follow-up PR.
- The python e2e suites require the companion pymilvus change:
milvus-io/pymilvus#3670 is merged and
`tests/python_client/requirements.txt` is bumped to
`pymilvus==3.1.0rc61` in this PR (all 22 rewritten cases + the
external-table case verified locally against the released rc61).
- During a rolling-upgrade window an old coordinator replaying the new
message ignores `bound_field_indexes` (proto3 unknown field), i.e.
pre-existing behavior; avoid the new argument in mixed-version clusters.
- `AddCollectionFunction` on an existing field and plain
add-vector-field keep their current behavior (the WAL carrier is generic
for future extension).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Make ComposeTSByTime the logical-zero time-based timestamp API, add
ComposeTSByTimeWithLogical for explicit logical composition in tests,
and remove the duplicate GetCurrentTime wrapper.
See also: #51119
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
issue: #50931
## Summary
Clean analyzer option update helpers on master to match the 2.6
follow-up.
Remove the exported UpdateParams wrapper and return C status handling
directly during option initialization.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
issue: #50905
- streamingcoord recovery: include pchannels recovered from RootCoord
collection metadata through ConfigChannelProvider so missing WAL topics
are added after stats initialization
- dml channel compatibility: expand configured DML-style pchannels up to
the recovered index while preserving pre-created-topic constraints
- channel stats: expose active pchannels from PChannelStatsManager for
metadata-driven channel recovery
---------
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #50931
## Summary
Initialize analyzer runtime options with Lindera download URLs and the
default dictionary path. Register runtime config callbacks so analyzer
yaml updates are propagated to the Rust analyzer layer.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
issue: https://github.com/milvus-io/milvus/issues/50571
design doc: docs/design-docs/design_docs/20260624-function-chain-api.md
Add the FunctionChain proto-to-runtime path for ordinary Search L2
rerank. The change converts public FunctionChain protos into ChainRepr,
derives caller-independent read/write metadata, validates
Search-specific inputs in Proxy, and executes public chains through the
existing rerank pipeline.
Add request validation for duplicate stages, unsupported stages,
unsupported system inputs/outputs, unknown schema fields, unsupported
input field types, and hybrid-search usage. Extend REST v2 request
conversion to accept function_chains.
Replace score_combine with num_combine and add typed parameter readers,
repr-based FuncChain construction, chain optimization/pruning helpers,
and model-rerank parameter handling. Add coverage for chain repr
conversion, function/operator behavior, Proxy planning, search pipeline
integration, REST conversion, and REST API cases.
Add the Function Chain API design document and update milvus-proto to
the latest upstream pseudo-version.
Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
issue: https://github.com/milvus-io/milvus/issues/50816
Add Hugging Face Inference Providers client support for feature
extraction and sentence similarity APIs, and wire it into text embedding
and rerank model providers.
The new provider supports:
- text embedding via feature-extraction
- rerank scoring via sentence-similarity
- Hugging Face router provider selection with hf_provider
- MILVUS_HUGGINGFACE_API_KEY credential fallback
- provider config entries for text embedding and rerank
Also add focused tests for the Hugging Face client, rerank provider, and
paramtable provider docs.
Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
## Summary
- Thread the insert-channel shard through field, index, JSON stats, text
index, storage v1, and storage v2 load metadata into cachinglayer
translator metadata.
- Add cgo shard setters so Go-side load requests can pass shard into C++
load info.
- Keep the actual cachinglayer metric definition and CacheSlot
accounting in companion `milvus-common` PR:
https://github.com/zilliztech/milvus-common/pull/98
issue: #50941
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
issue: #50583
## Summary
- prefetch scalar expression chunks, vector data, and MVCC timestamp
chunks before driver execution
- wait for prefetch completion before consuming prefetched
MVCC/vector/scalar data
- size the cgo caller pool from queryNode MaxReadConcurrency and expose
queryNode.segcore.cgoPoolSizeRatio in milvus.yaml
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
- Replace `common.entryStream.streamBudgetRatio` with refreshable
`common.loadTransientBudgetBytes`, a process-wide transient load budget
shared by scalar index V3 entry streaming and storage v2/v3 field-data
loading. `0` keeps the limit disabled.
- Gate field-data batch reads on the shared budget, split batches by
loading overhead, scale read parallelism from the effective batch
budget, and finalize `GroupChunk` cells before pushing them so Arrow
tables are released promptly.
- Align MCL loading-overhead reservations for storage v1 scalar indexes
and storage v2/v3 field data under one load-transient overhead group,
with array-field overhead and mmap file usage accounted separately.
- Propagate load cancellation into budget waits, field-data batch tasks,
and V3 scalar `IndexEntryReader` stream/file reads so canceled loads do
not stay blocked behind transient memory pressure.
- Update segcore init/config watchers and add C++/Go coverage for budget
sizing, cancellation, field-data batching/finalization, V3 entry
streaming, and the new config.
issue: #49499
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
issue: #50749
Depends on #50527.
Route proxy PK/channel and partition-key hash helpers through the
routing table while preserving the existing hash modulo behavior.
Query/search are covered through assignPartitionKeys, which delegates to
internal/util/typeutil.HashKey2Partitions.
Signed-off-by: sunby <sunbingyi1992@gmail.com>
## Issue
issue: #50102
## Problem
With etcd auth enabled and `mixCoord.enableActiveStandby: true`, a
long-running **standby** MixCoord panics when promoted to active:
```
panic: rpc error: code = Unauthenticated desc = etcdserver: invalid auth token
ProxyWatcher.startWatchEtcd (proxy_watcher.go:146)
```
This is **not** a credential misconfiguration (the same config works for
the active node and all other components, and a restart of the same pod
recovers).
Root cause:
- etcd's default `simple` auth token is stateful and server-local: GC'd
after idle TTL (`--auth-token-ttl`, default 300s), member-local, and
lost on member restart. A long-idle standby's token gets invalidated
server-side without the client knowing.
- When an **already-established watch stream** then hits
`Unauthenticated`, clientv3 classifies it as a halt error (`isHaltErr` —
anything that isn't `Unavailable`/`Internal`) and tears the stream down
**without** refreshing the token. clientv3's token refresh only covers
unary calls and a stream's first recv, not a live watch stream.
- `ProxyWatcher.startWatchEtcd` and `sessionutil.handleWatchErr` only
handled `ErrCompacted` and `panic()`ed on every other watch error.
## Fix
Treat auth-token watch errors as recoverable and re-establish the watch
with a bounded retry, instead of panicking. Re-watching issues a unary
request first (`getSessionsOnEtcd` / `GetSessions`), which refreshes the
auth token via clientv3's unary retry interceptor, so the new watch
stream recovers.
- New shared predicate `etcd.IsRetriableWatchErr` classifies recoverable
watch errors (`ErrCompacted` + auth-token errors: `ErrInvalidAuthToken`
/ `ErrUserEmpty` / `ErrAuthOldRevision` / raw `Unauthenticated`).
- `ProxyWatcher.startWatchEtcd` and `sessionutil.handleWatchErr` now
re-watch on recoverable errors via `retry.Do(...,
retry.RetryErr(etcd.IsRetriableWatchErr))`. The retry is bounded by
`retry.Do`'s default attempt count / backoff; non-recoverable errors
still fail fast.
The CDC controller and streaming session discoverer already self-heal
(return-and-re-list on watch error), so they are unchanged.
> Operational note for affected users: configuring etcd with **JWT**
auth tokens (`--auth-token jwt,...`) instead of the default `simple`
token avoids the unexpected server-side invalidation entirely (JWT is
stateless — valid across members, survives restart, no idle GC).
## Test
- Added `TestIsRetriableWatchErr` (pkg/util/etcd) — predicate
classification.
- Added subtests in `TestWatcherHandleWatchResp` (sessionutil) —
auth-token error triggers re-watch (not channel close), non-retriable
error closes the channel.
- Existing `TestProxyManager_ErrCompacted` still passes (non-retriable
bad-session-data fails fast).
> Note: the watcher integration tests link the C++ core; they were not
runnable in my local/remote env at PR time and will be validated in CI.
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
relate: #49716
pr: https://github.com/milvus-io/milvus/pull/49717
## Summary
Split FunctionRunnerManager lifecycle references between WAL and
delegator keys while still sharing runners by function signature. Add
latest-version lookup support for schemaVersion 0 or nil schema in the
manager; master call sites still pass the current schema version
explicitly.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
- Allow AlterCollectionSchema AddRequest to carry a field-only add
request while keeping the added field as a non-function output.
- Support function-only add requests against existing output fields, and
validate the merged function schema through the shared function
validator.
- Keep the BM25 field-plus-function path explicit, reject invalid BM25
add forms, and normalize TIMESTAMPTZ default values before broadcasting
field-only schema changes.
issue: #50119
## Test Plan
- [x] source scripts/setenv.sh && go test -v -count=1 -tags dynamic,test
-gcflags="all=-N -l" -ldflags="-r ${MILVUS_WORK_DIR}/cmake_build/lib -r
${MILVUS_WORK_DIR}/internal/core/output/lib" ./internal/proxy -run
'TestAlterCollectionSchemaTask($|_)' -timeout 300s
- [x] gofumpt -l internal/proxy/function_task.go
internal/proxy/function_task_test.go internal/proxy/task.go
internal/proxy/task_test.go internal/proxy/util.go
internal/proxy/util_test.go
internal/rootcoord/ddl_callbacks_alter_collection_schema.go
internal/rootcoord/ddl_callbacks_alter_collection_schema_test.go
internal/util/function/validator/validator.go
- [x] git diff --check upstream/master..HEAD
- [ ] source scripts/setenv.sh && go test -v -count=1 -tags dynamic,test
-gcflags="all=-N -l" -ldflags="-r ${MILVUS_WORK_DIR}/cmake_build/lib -r
${MILVUS_WORK_DIR}/internal/core/output/lib" ./internal/rootcoord -run
'TestDDLCallbacksBroadcastAlterCollectionSchema|TestDDLCallbacksAlterCollectionSchemaAddSkipsSchemaDropReady'
-timeout 300s (blocked locally at link time by stale C++/Loon symbols
such as _FreeCFieldMemSizeList and _loon_segment_writer_* in local core
libraries)
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
issue: https://github.com/milvus-io/milvus/issues/46565
Implement boost score evaluation for the Go search reduce pipeline,
including
QueryNode task integration, segcore C API bindings, and score expression
combination support.
Add boost score runner logic in core, expose segment-level boost scoring
to Go,
and cover the new behavior with C++, Go, and Python client tests.
Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
issue: #50229
## What this PR does
Adds configurable request timeouts for external function model providers
(embedding / rerank / highlight), and standardizes the timeout unit from
**seconds → milliseconds** across all providers and clients.
### Changes
- New global config `function.model.timeout_ms` (default `30000`,
refreshable, exported to `milvus.yaml`).
- New per-function param `timeout_ms` which **overrides** the global
setting (`models.ResolveTimeoutMs`). Invalid / non-positive values fall
back to the `30000` default.
- Converted every provider/client timeout argument from `timeoutSec` to
`timeoutMs`; `PostRequest` now applies the value as milliseconds.
- Zilliz gRPC client now applies a per-request deadline via
`requestContext` (previously had no request timeout).
- Bedrock embedding now derives its call context from the request `ctx`
with the configured timeout (was `context.Background()`).
## Tests
- `models`: `TestParseTimeoutMs` (defaults, override, invalid,
zero/negative, case-insensitive key), `TestResolveTimeoutMs` (global
default + per-function override), and `TestZillizClient_RequestContext`
(per-request deadline + metadata).
- `rerank`: provider param validation for invalid / non-positive
`timeout_ms`.
- `paramtable`: `TestFunctionConfig` asserts the new default.
- Updated zilliz client / rerank / embedding / highlight tests for the
ms unit and the new test constructor.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
- Persist role descriptions when creating roles, expose them in role
list/describe results, and support updating descriptions through the
AlterRole path without changing role names.
- Store role descriptions in the role value body while keeping role
names in keys, and tolerate legacy empty role values plus undecodable
stored values by returning an empty description for that role row.
- Reject built-in/default roles and over-limit descriptions before WAL
writes, and share the role-description length validator between proxy
and rootcoord.
related: #50183
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
## What this PR does and why
Restores FileManager stream reads to a single local-path contract.
Stream readers now pass a local full path or filename, and FileManager
derives the remote index object path from index metadata plus the
basename.
The PR also propagates `index_store_path_version` through QueryCoord,
QueryNode, and C++ load metadata so collection-rooted index paths
continue to resolve correctly, including the StorageV3 segment-load
path.
Fixes#50451
## Test plan
- [x] `GOTOOLCHAIN=auto make generated-proto-without-cpp`
- [x] `git diff --check`
- [ ] `make static-check` (will be run manually before merge)
- [ ] C++ unit tests (blocked locally by Conan/C++ output environment)
Signed-off-by: Yihao Dai <yihao.dai@zilliz.com>
issue: #47420
## What this PR does
Project-wide migration of raw `fmt.Errorf` / `errors.New` in function
bodies onto
the `merr` framework, plus the Sys-vs-Input error classification and the
machinery it drives (retriability, fine-grained metrics, segcore
unification),
plus the convention docs and a linter that keeps it from regressing.
Scope: storage, proxy, coordinators (root/data/query), query node, data
node,
`pkg/util` & `internal/util`, expression parser, message queue,
streaming, and
misc packages. Bare raw-error usages went from ~3000 to a ~340 allowlist
(package-level sentinels / build-tag / test sites).
---
## How to review this PR
It is large but the vast majority is mechanical. Changes fall into three
tiers;
spend review budget on Part 2 and Part 3.
### Part 1 — Mechanical standardization (low risk, verify by rule)
Each converted call follows one of a small fixed set of rules. To
review, check
that each site obeys the matching rule rather than reading every line:
| Pattern | Rule |
|---|---|
| `fmt.Errorf("...")` originating a new error | →
`merr.WrapErrXxxMsg("...")` with a code matching the failure's meaning |
| Adding context to an existing typed error | → `merr.Wrap(err, "...")`
/ `merr.Wrapf(...)` — **preserves** the inner code (never `WrapErr*Err`,
which overwrites it) |
| Errors inside the streaming subsystem | → `status.New*` factories
(StreamingError), **not** merr — this is the component-internal dialect
(see `docs/dev/error_handling_guide.md`) |
| Low-level / control-flow signal caught by `errors.Is` | → kept as a
package-level `errors.New` sentinel (lowercase, same-package) |
Conventions are documented in `docs/dev/error_handling_guide.md`
(how-to) and
`docs/dev/error_sentinel_convention.md` (rules + audit). A
`gocritic`/`ruleguard`
rule (`rawmerrerror`, in `rules.go`) enforces "no raw `return
errors.New/fmt.Errorf`"
under `make verifiers`.
### Part 2 — Behavior changes (review these closely)
These are the sites where the wire contract or runtime behavior changes,
not just
the source text. Listed by category; representative locations given,
full set in
the diff.
**A. gRPC wire-code shifts: `UnexpectedError(1)/Code 65535` → typed
code.**
Where a handler previously returned a raw error (collapsed to
`Code=65535` on the
wire), it now returns a typed merr, so the client sees a real code. The
most
common shift is to `IllegalArgument(5)/Code 1100` (ParameterInvalid).
Touch
points include datanode task handlers (CreateTask/Query/Drop), proxy
Upsert,
querynode GetMetrics, datacoord CreateIndex, httpserver query-response
builder,
and typeutil schema validation. One code refinement: an index-param
validation
moved `1100` → `1101` (ParameterMissing). **Client/SDK assertions and
any code
that switched on `Code=65535` for these paths must be re-checked** (the
go_client
e2e assertions were already aligned in this PR).
**B. Prometheus `status` label contract change (externally visible).**
The proxy metric's coarse `fail` / `rejected` values are split into
`fail_input` / `fail_system` and `rejected_user` / `rejected_system` (in
`requestutil.ParseMetricLabel`; auth/privilege rejections count as
`rejected_user`), so dashboards can attribute a failure to caller vs
operator.
**Dashboards/alerts querying `status="fail"` must migrate to
`status=~"fail_.*"`, and `status="rejected"` to
`status=~"rejected_.*"`.** The
in-repo Grafana dashboard is already migrated; external dashboards built
on the
old values silently go empty after upgrade. This is the one change that
requires an ops-side migration.
**C. Retriability semantics.**
- C1: `merr.Status(err)` now forces `Retriable=false` when the error is
an
`InputError` — a malformed request can never succeed on blind retry, so
clients
never get the self-contradictory "your input is wrong but you may
retry".
- C2: `retry.Do` short-circuits an `InputError` (non-retriable) — **but
only when
the caller did not pass a `RetryErr` predicate**. The check is an
`if c.isRetryErr != nil { ... } else if InputError { ... }` *mutually
exclusive*
branch (`pkg/util/retry/retry.go`): an explicit `RetryErr` takes
precedence and
bypasses the InputError abort. `retry.Handle` deliberately does **not**
apply
the InputError abort (its callers signal abort via `shouldRetry=false`).
Four
flusher startup callsites that must retry through transient "not ready"
errors
were given explicit `RetryErr` escape hatches.
**D. segcore (C++→Go) error classification.**
A single shared Go-side table (`pkg/util/merr/segcore.go`) maps each
segcore code
to a merr sentinel + InputError/signal category, replacing scattered
hand-written
`if errorCode == ...` switches in the cgo wrappers. **Wire `Code` values
change
for every segcore pass-through error, not just the remapped ones.**
Named
sentinels remap (C++ `2003` → merr `2001`, `2033` → `2002`,
Folly/Knowhere codes
likewise); **all remaining pass-through codes (`2004`–`2043`, previously
surfaced to clients as raw C++ enum values) now serialize as `2000`**
(`ErrSegcore`), with the original C++ code preserved in the `Reason`
text
(`segcoreCode=...`); unknown/future codes collapse to `2000` as well
(pinned by
the `wire_code_projection` test). Transient segcore classes (object
storage /
file IO / OOM / mmap / FieldNotLoaded — 11 codes) now report
`Retriable=true`.
**Any client switching on raw segcore codes in the `2004`–`2043` range
must be
re-checked**; the in-Reason code remains available for diagnostics.
Signal
codes (PretendFinished / FollyCancel) are recognized centrally.
`errors.Is`-based
control flow on these (e.g. scheduler skip/retry) is preserved.
**E. InputError classification (25 sentinels + dynamic marks).**
25 sentinels in `errors.go` carry `WithErrorType(InputError)` (the
Collection /
ResourceGroup / Database families, `ErrIndexDuplicate`,
`ErrParameterInvalid`,
`ErrPrivilegeNotAuthenticated`, `ErrImportFailed`, `ErrQueryPlan`, ...),
plus dynamic
marks for the 8 segcore input codes (ExprInvalid, DimNotMatch,
MetricTypeInvalid, FieldIDInvalid, ...) and
`WrapErrAsInputError`. The widest blast radius is `ErrParameterInvalid`
(1100):
~2335 `WrapErrParameterInvalid*` callsites now classify as input /
non-retriable. Because of C1/C2 this changes retriability for
any path that returns these. **The audit to confirm no transient path
was
mis-marked is the single most important review item** (see Part 3). One
reverse
correction: storage field-stats parsing moved from `ErrParameterInvalid`
(input)
to `ErrDataIntegrity` — a corrupted stored stat is data corruption, not
user
input.
### Part 3 — Known risks & traps (called out proactively)
1. **`merr.Wrap` vs `WrapErr*Err` (code-masking).** `WrapErr*Err` builds
a
`wrappedMilvusError{sentinel: ErrServiceInternal}` whose `code()`
returns the
*outer* sentinel — it overwrites the inner typed code and hides the
`errors.Is`
chain. This is intentional (use it to *deliberately* downgrade), but it
was a
recurring conversion defect; the rule "add context with `merr.Wrap`,
downgrade
with `WrapErr*Err`" is enforced by convention and reviewed across the
diff.
2. **InputError × `retry.Do` blast radius.** Marking a sentinel
`InputError` makes
any `retry.Do(...)` without a `RetryErr` predicate stop retrying it.
Reviewers
should sanity-check that no transient use of the 19 newly-marked
sentinels
(especially `ErrParameterInvalid`) sits inside a retry loop that needed
to keep
spinning. The known flusher cases were handled (see C2).
3. **The ~340 raw-error allowlist.** What remains as bare `errors.New`
is, by
design: package-level sentinels (caught by `errors.Is`), `//go:build
test`
sites, and out-of-band trees (`cmd/`, `tests/`, codegen, walimpls). The
linter
only bans the *direct-return* form; assignment-then-return escapes and
the full
no-exceptions ban are deferred to an AST-based linter (Tier 2,
documented).
4. **segcore C++ second step deferred.** This PR unifies classification
on the Go
side; splitting the dual-semantic C++ codes at the source is a
follow-up.
---
## Validation
- `make verifiers`: Go side clean (gofmt + static-check across modules,
including
the new `rawmerrerror` rule with a 0-hit baseline repo-wide).
- `make test-go`: passing; the one real regression introduced (a
datanode
`invalid_task_type` assertion shifting `1` → `5` from a ParameterInvalid
conversion) was fixed in-tree.
- go_client e2e CreateIndex assertions aligned to the new merr messages.
---------
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
relate: #49716
## Summary
Move BM25/MinHash function execution from DN/QN pipeline embedding nodes
to the write-before path before WAL append. Add collection-scoped
function runner caching and keep pipeline-side compatibility fill for
old insert messages without generated function output fields.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
https://github.com/milvus-io/milvus/issues/48794
Normalize credential names before building config keys so API key,
AK/SK, and GCP credential lookups work when names contain mixed-case
characters.
Add tests covering mixed-case credential names for all supported
credential types.
Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
#48986
Replace C++ ReduceSearchResultsAndFillData with a Go reduce pipeline:
- Export per-segment search results as Arrow RecordBatch via C Data
Interface
- HeapMergeReduce: k-way heap merge with PK dedup and GroupBy support
- Late Materialization: single CGO call (FillOutputFieldsOrdered) for
output fields
- ExportSearchResultAsArrow supports extra field IDs for future L0
rerank
Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
## Issue
issue: #50188
## Problem
For external collections, `query_iterator` returns hundreds of thousands
of rows and then fails with `MilvusException: (code=65535,
message=fieldID(1) not found)`. Rebuilding the iterator from the last
primary-key cursor reproduces the error immediately and
deterministically.
## Root Cause
The proxy unconditionally appends the reserved system Timestamp field
(`common.TimeStampField`, fieldID=1) to `OutputFieldsId` for the
iterator MVCC dedup (`internal/proxy/task_query.go:610`).
On the QueryNode side, `FillRetrieveResultIfEmpty` is invoked when a
query produces a **fully empty** result. It iterates `OutputFieldsId`
and resolves each field id through the collection `SchemaHelper` to
synthesize empty columns. System fields are never part of the
user/collection schema, so for external collections (whose schema
contains no system fields, and whose segcore synthesizes the timestamp
as a constant Int64 column via `SynthesizeExternalSystemFields` /
`init_timestamps_constant`) the lookup `GetFieldFromID(1)` fails with
`fieldID(1) not found`.
This surfaces when the iterator paginates past the last matching row so
the next batch matches zero rows, and reproduces deterministically when
the iterator is rebuilt from the last cursor (the query is again empty,
hence the ~20ms fail-fast).
Internal collections do not hit this because segcore returns
empty-but-present field columns for zero-match queries, bypassing the
empty-fill path; the underlying bug is that the Go empty-fill path does
not special-case system fields the way the C++ retrieve path does.
## Fix
Special-case reserved system fields (RowID / Timestamp) in
`FillRetrieveResultIfEmpty` and emit them as empty Int64 columns —
matching the non-empty retrieve path (segcore emits system fields as
Int64) — instead of resolving them through the schema. The proxy already
strips system fields via `filterSystemFields`, so the user-facing result
is unchanged.
## Test
- Added table-driven cases to `TestFillIfEmpty` covering an
external-collection schema (no system fields) with `OutputFieldsId`
carrying `TimeStampField`, and a RowID+Timestamp case. Both assert no
error and that empty Int64 columns are emitted.
- Changed code covered at 100%.
- `go test -tags dynamic,test -gcflags="all=-N -l"
./internal/util/typeutil/` passes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
relate: #50113
## Summary
Make function runner text tokenization concurrency configurable through
function.analyzer.runner_concurrency, preserving the default value of 8.
BM25 and multi-analyzer BM25 now read the parameter for each
BatchRun/BatchAnalyze call so runtime config changes can take effect
without recreating runners. Also register a config event handler for
function.analyzer.concurrency_per_cpu_core so the shared analyzer pool
resizes dynamically when that setting changes.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
issue: https://github.com/milvus-io/milvus/issues/47425
# Plan: upgrade Milvus to Knowhere with Conan `milvus-common` and Conan
OpenBLAS
## Context
Milvus currently fetches both `knowhere` and `milvus-common` from source
with CMake `FetchContent`. The requested Knowhere commit
`0e53e57cd1f67f74decbdf9af42e74d0e3850155` expects `milvus-common` and
Linux OpenBLAS from Conan 2.x instead:
-
`milvus-common/1.0.0-9ca5ea6@milvus/dev#274d428d85f1d3d996e1092f0c9c7144`
- `find_package(milvus-common REQUIRED)` and target
`milvus-common::milvus-common`
- Linux `openblas/0.3.30` with CMake target `OpenBLAS::OpenBLAS`
- macOS Apple BLAS/LAPACK via `BLA_VENDOR Apple`, not Conan OpenBLAS
Milvus also currently installs system OpenBLAS in builder Dockerfiles,
runtime Dockerfiles, dependency scripts, and packaging scripts. The
intended outcome is one build-time OpenBLAS source on Linux: Conan.
Runtime packaging should use the Conan-built shared libraries instead of
relying on distro `libopenblas-dev`/`openblas-devel` where possible.
## Implementation
1. **Update Conan dependency pins in `internal/core/conanfile.py`.**
- Add direct requirement
`milvus-common/1.0.0-9ca5ea6@milvus/dev#274d428d85f1d3d996e1092f0c9c7144`.
- Align dependencies required by Knowhere `0e53e57c`:
- `folly/2026.04.20.00@milvus/dev#06852bea5b6449f0c4eb0df002b5779c`
- `fmt/11.2.0#eb98daa559c7c59d591f4720dde4cd5c`
- `lz4/1.10.0#982d9b673900f665a1da109e09c17cab`
- `fast_float/8.0.0@milvus/dev#c7802833c74c5a86ffed70e4af1a795e`
- Linux-only `openblas/0.3.30`
- Add `openblas/*:dynamic_arch = True` to default options to match
Knowhere.
2. **Move Milvus core build settings to C++20.**
- Change `internal/core/CMakeLists.txt` from `CMAKE_CXX_STANDARD 17` to
`20`.
- Keep `OPENTELEMETRY_STL_VERSION=2017` unchanged; it controls
OpenTelemetry ABI mode, not the language standard.
- Change all `compiler.cppstd=17` Conan install settings in
`scripts/3rdparty_build.sh` to `compiler.cppstd=20`.
- Review `internal/core/src/storage/gcp-native-storage/CMakeLists.txt`;
if it inherits incompatible flags or includes C++20-dependent headers,
change its local `CMAKE_CXX_STANDARD 17` to `20` too.
3. **Expose Conan packages to Milvus CMake.**
- Add `find_package(milvus-common REQUIRED)` in
`internal/core/CMakeLists.txt` near the existing Conan
`find_package(...)` block.
- Add Linux-only `find_package(OpenBLAS CONFIG REQUIRED)` in
`internal/core/CMakeLists.txt` so missing Conan OpenBLAS fails before
Knowhere configures.
- Remove `-DOpenBLAS_SOURCE=AUTO` from `internal/core/build.sh`; the
requested Knowhere commit no longer consumes that flag and directly
calls `find_package(OpenBLAS CONFIG REQUIRED)` on non-Apple platforms.
4. **Stop executing source FetchContent for `milvus-common`.**
- Remove `add_subdirectory(milvus-common)` from
`internal/core/thirdparty/CMakeLists.txt`.
- Keep `add_subdirectory(knowhere)`.
- Do not keep a compatibility alias to the old source target; all
consumers should use the Conan target.
- Leave `internal/core/thirdparty/milvus-common/` files in place for the
first pass unless cleanup is explicitly desired; they will no longer be
active once the subdirectory is removed.
5. **Pin Knowhere to the requested commit.**
- Update `internal/core/thirdparty/knowhere/CMakeLists.txt`:
- `KNOWHERE_VERSION` -> `0e53e57cd1f67f74decbdf9af42e74d0e3850155`.
- Use the full SHA for deterministic FetchContent checkout.
6. **Migrate CMake links/includes from source target to Conan target.**
- In `internal/core/src/CMakeLists.txt`:
- Remove `${MILVUS_COMMON_INCLUDE_DIR}` from include directories.
- Add `milvus-common::milvus-common` to `CONAN_TARGETS` so
`milvus_conan_deps` propagates include directories to object libraries.
- Remove plain `milvus-common` from `LINK_TARGETS`.
- In `internal/core/unittest/CMakeLists.txt`:
- Remove `${MILVUS_COMMON_INCLUDE_DIR}` from include directories.
- Remove plain `milvus-common` from `all_tests` and `test_json_uint64`
link lists; both already link `milvus_conan_deps`, which will carry
`milvus-common::milvus-common`.
7. **Adjust build-time system OpenBLAS installs.**
- Remove Linux build-time OpenBLAS package installs that can shadow or
duplicate Conan OpenBLAS:
- `scripts/install_deps.sh`: remove Ubuntu `libopenblas-dev`; remove
Rocky/Amazon/CentOS `openblas-devel` where only needed for C++ core
build.
- `build/docker/builder/cpu/ubuntu20.04/Dockerfile`,
`ubuntu22.04/Dockerfile`, `ubuntu24.04/Dockerfile`: remove
`libopenblas-dev`.
- `build/docker/builder/gpu/ubuntu20.04/Dockerfile`,
`gpu/ubuntu22.04/Dockerfile`: remove `libopenblas-dev`.
- `build/docker/builder/cpu/amazonlinux2023/Dockerfile`,
`rockylinux9/Dockerfile`: remove `openblas-devel` and any OpenBLAS
header symlink workaround that only supported system OpenBLAS.
- `build/deb/build_deb.sh`: remove build-time `libopenblas-dev` install
if no longer needed after Conan generation.
- Keep Fortran/toolchain packages needed to build Conan OpenBLAS from
source when binary packages are missing.
- Do not change `scripts/install_deps_embd.sh` or
`scripts/install_deps_msys.sh` in the first pass unless the target build
path uses them; they are separate embedded/MSYS dependency paths and may
still require manual/system OpenBLAS.
- On macOS, remove `openblas` from `.github/workflows/mac.yaml` only
after confirming the new Knowhere path uses Apple BLAS/LAPACK and Milvus
has no other macOS OpenBLAS consumer.
8. **Adjust runtime/package OpenBLAS handling carefully.**
- The runtime should still include `libopenblas.so*` if the Milvus
binary or Conan-built Knowhere/Faiss links it dynamically.
- Prefer copying the Conan-provided OpenBLAS shared library from
`cmake_build/lib` or the Conan package output instead of installing
distro development packages in runtime images.
- Update packaging sites that currently copy or depend on system
OpenBLAS:
- `build/deb/build_deb.sh`: change the copied source from
`/usr/lib/x86_64-linux-gnu/libopenblas.so.0` to the Conan/build output
path if present.
- `build/rpm/milvus.spec`: stop assuming
`/usr/lib/libopenblas-r0.3.9.so`; use the packaged Conan/build output
library.
- Runtime Dockerfiles under `build/docker/milvus/**`: remove
`libopenblas-dev`/`openblas-devel` only after confirming the image gets
the Conan OpenBLAS `.so` with Milvus libs.
- `build/docker/milvus/gpu/ubuntu20.04/Dockerfile.base`: review the
explicit `milvusdb/openblas` stage; replace it with the Conan-produced
library if that image path is still active.
- Keep runtime `libgomp1`/`libgomp` and `libaio` if required by
OpenMP/libaio-linked dependencies.
9. **Clean stale build state before verification.**
- Use a fresh `cmake_build` or remove the old FetchContent
checkout/cache before rebuilding; stale `MILVUS_COMMON_INCLUDE_DIR`, old
`knowhere-src`, or system OpenBLAS paths can hide integration errors.
Signed-off-by: yhmo <yihua.mo@zilliz.com>
This PR extends the named OS thread active metrics added in #50064.
- classify RocksDB background threads into `rocksdb_high`,
`rocksdb_low`, and `rocksdb_bottom`
- count active named OS threads that do not match a known classification
as `unclassified`
- keep `classifyThreadName` returning whether a known rule matched,
while callers explicitly map unmatched threads to `unclassified`
- add focused coverage for RocksDB and unclassified active-thread
accounting
issue: #50066
Signed-off-by: CLiQing <2208529306@qq.com>
