## What
Enforce a function–field binding principle for function DDL (add / drop
/ alter function on an existing collection), so a function and its
output field stay coupled and already-stored vectors can never be
silently invalidated by DDL.
BM25 and MinHash follow the strict binding: add/drop only via the
unified `add_function_field` / `drop_function_field`, output field
always coupled. **TextEmbedding is a temporary carve-out** — its legacy
`AddCollectionFunction` / `DropCollectionFunction` paths are retained,
so attach-over-existing-field and detach are still possible for it,
because `add_function_field` embedding backfill is not yet implemented.
It will be folded into the unified path in a follow-up. So the "always
coupled" invariant currently holds for BM25/MinHash, not TextEmbedding.
## Changes
- **AlterCollectionSchema invariant**: reject standalone add-function (a
function must be added together with its new output field), reject
detaching a function without dropping its output field, and reject
function cascade (a new function's input being another function's
output).
- **Legacy RPCs**: `AddCollectionFunction` / `DropCollectionFunction`
are rejected for BM25/MinHash (must use `add_function_field` /
`drop_function_field`) and retained only for TextEmbedding (see
carve-out). `AlterCollectionFunction` is kept and gains the whitelist
below. In the alter and legacy-add paths, function field IDs are always
re-derived from field names (never trusted from the request), so a
request cannot inject an unrelated field ID that a later
`drop_function_field` would delete.
- **alter_function whitelist**: only connection/runtime params may
change (TextEmbedding: `url`, `credential`, `timeout_ms`,
`max_client_batch_size`, `region`, `location`, `projectid`, `user`);
BM25/MinHash have no alterable params. Function identity (type, name,
input/output fields) and output-shaping params (`dim`, `model_name`,
`endpoint` — the TEI model identity, `normalize`, `truncate*`, prompts,
...) are immutable. Params are normalized (keys lowercased, duplicate
keys rejected) so a crafted duplicate key cannot bypass the diff.
- **drop_function_field**: allow any output-producing function (dropping
needs no backfill), removing the previous BM25/MinHash-only restriction.
## Not in scope / follow-up
- Extending `add_function_field` to TextEmbedding (post-creation add
would require backfilling every existing row through the external
embedding model) — deferred; folding the TextEmbedding legacy paths into
the unified binding follows this.
- MinHash input-field analyzer immutability lives on a different DDL
path (`AlterCollectionField`); tracked as a separate follow-up.
## Breaking changes
- `DropCollectionFunction` on a **MinHash** function (detach — remove
the function but keep its output field) is no longer supported; use
`drop_function_field` instead (drops the function together with its
output field). If a released version supported MinHash detach, migrate
any such call to `drop_function_field`.
issue: #51348🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
issue: #51493
### What
#50221 split the `status` label values of `milvus_proxy_req_count` /
`milvus_proxy_grpc_latency` in place. Released in **v2.6.19**, that
silently zeroes every existing dashboard panel and alert rule matching
`status="fail"` / `status="rejected"` — an alert that stops firing
rather than erroring.
This keeps the classification but moves it onto its own dimension,
restoring the status domain to what <= v2.6.18 emitted:
```
status: success | fail | rejected | retry | total | abandon (as before v2.6.19)
cause: user | system | cancel | na (new)
```
| v2.6.19 / v2.6.20 | this PR |
|---|---|
| `status="fail_input"` | `status="fail", cause="user"` |
| `status="fail_system"` | `status="fail", cause="system"` |
| `status="rejected_user"` | `status="rejected", cause="user"` |
| `status="rejected_system"` | `status="rejected", cause="system"` |
| `status="cancel"` | `status="fail"` or `"rejected"`, `cause="cancel"`
|
| `success` / `retry` / `total` / `abandon` | unchanged, `cause="na"` |
### Why a label instead of new status values
- **Old queries work unchanged and count each request once.**
`status="fail"` is again every hard failure; Prometheus aggregates over
`cause` for free. That rollup is what a label dimension *is* — the split
forces every consumer who just wants "how many failures" to hand-write
`status=~"fail_.*"`.
- **Cardinality is unchanged.** `cause` is functionally dependent on the
outcome, so the realized `(status, cause)` pairs are exactly the series
the split already produces. Additive, not multiplicative.
- **New consumers lose nothing**: alert on `status="fail",
cause="system"`; route `cause="user"` to the owning application team.
The alternative — emitting old and new `status` values side by side
during a transition — was rejected: it defers the break rather than
removing it (the old values must still be dropped eventually, forcing
the same migration later), and meanwhile double-counts every request in
unfiltered aggregations.
`cancel` is folded into `cause` for the same reason: before v2.6.19
client cancellations were counted as `fail` (cancel in the response
status) or `rejected` (cancel at the interceptor), so promoting it to a
`status` value quietly makes `status="fail"` under-count versus older
releases. As a cause it stays excludable via `cause!="cancel"` without
redefining `status`.
### Also in this PR
- The 7 `snapshot_impl` sites that emitted a bare `fail` with no
classification now report their real cause via `failMetricLabel(err)`
(e.g. `snapshot_metadata_uri is required` is `cause="user"`, not a
system fault). Their `status` is unchanged.
- `deployments/monitor/grafana/milvus-dashboard.json`: the "Faild
Request Rate" panel goes back to `status="fail"` — a verbatim revert of
what #50221 changed, which is the compatibility claim demonstrated.
- Dev docs and stale comments that named the old label values.
### Verification
- `TestParseMetricLabelStatusDomainIsStable` pins the status domain, so
re-splitting it fails CI rather than shipping.
- Adding a label makes every `WithLabelValues` site arity-sensitive **at
runtime, not compile time** — a missed site panics in production. Unit
tests do not reach all of them (coverage shows
`GetRestoreSnapshotState`, `ListRestoreSnapshotJobs`, `PinSnapshotData`,
`UnpinSnapshotData` at 0%), so all **58 emit sites were verified
statically via AST**, not only the ones tests happen to hit.
- Passing: `pkg/metrics`, `pkg/util/requestutil`, `pkg/util/merr`,
`internal/distributed/proxy` (incl. `httpserver`, which covers the REST
emit path), and the `internal/proxy` snapshot / metric-label tests.
`golangci-lint` clean on every touched package.
- Pre-existing and unrelated: `service_test.go` bind failures (`listen
tcp :19529: address already in use`) reproduce identically on unmodified
master — a local process holds the port.
### Rollout
Should be cherry-picked to 2.6 so the window in which the fine-grained
`status` values exist stays confined to v2.6.19–v2.6.20. Users who
already adopted the new values on those two releases need a one-time
query change (`status="fail_system"` -> `status="fail",
cause="system"`); that is a known, bounded set, and preferable to
leaving every pre-2.6.19 dashboard silently broken.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
issue: #51170
Expose the existing server-side query ORDER BY capability (query param
key `order_by_fields`, already consumed by proxy `task_query.go`)
through the two remaining client surfaces:
- **Go SDK**: `queryOption.WithOrderByFields(fields ...string)` — each
spec is `"field"`, `"field:asc"` or `"field:desc"`, joined into the
`order_by_fields` query param.
- **RESTful v2**: `QueryReqV2` gains `orderByFields` (`[]string`, same
spec format), forwarded by the query handler as the `order_by_fields`
query param.
Validation (sortable type, explicit-limit requirement, iterator
exclusion) stays server-side in the proxy, consistent with how
limit/offset/group-by are handled at these layers. Interface shape
mirrors pymilvus (`order_by=["price:desc"]`).
Tests: Go SDK option unit test, RESTful v2 handler unit test (asserts
the KV pair is forwarded, and absent when not requested), and a
go_client e2e case (desc / asc-default / no-limit rejection).
Follow-up (out of scope here): search-side `order_by` first-class parity
for RESTful v2 / Go SDK (currently reachable via raw search params
only).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## Summary
- validate REST quick-create BinaryVector metricType against the
configured binary auto-index support set
- keep default BIN_IVF_FLAT quick-create limited to HAMMING/JACCARD so
SUBSTRUCTURE/SUPERSTRUCTURE/MHJACCARD fail before collection creation
- add Go handler coverage and RESTful e2e params for the previously
missed binary metrics
Follow-up for review comment on #51088.
related issue: https://github.com/milvus-io/milvus/issues/51084
## Test Plan
- [x] git diff --check
- [x] python3 -m py_compile
tests/restful_client_v2/testcases/test_collection_operations.py
- [ ] go test ./internal/distributed/proxy/httpserver -run
TestCreateCollectionQuickVectorFieldType -count=1 (blocked locally:
missing rocksdb.pc and milvus_core.pc pkg-config files)
Signed-off-by: Yanliang Qiao <yanliang.qiao@zilliz.com>
## What
Validate REST v2 quick-create enum-like fields before creating the
collection.
This PR fixes `vectorFieldType` handling for quick collection creation:
- rejects invalid `vectorFieldType` values instead of silently using
`FloatVector`
- supports `FloatVector`, `BinaryVector`, `Float16Vector`,
`BFloat16Vector`, and `SparseFloatVector`
- defaults quick-create index metric by vector type: dense `COSINE`,
binary `HAMMING`, sparse `IP`
- validates explicit binary/sparse `metricType` before collection
creation
- rejects `SparseFloatVector` quick-create with `dimension`
It also fixes top-level quick-create `consistencyLevel` handling:
- parses and validates top-level `consistencyLevel`
- preserves existing `params.consistencyLevel` behavior
- rejects conflicting top-level and `params.consistencyLevel` values
Fixes#51085Fixes#51084
## Why
Previously, `/v2/vectordb/collections/create` ignored top-level
`vectorFieldType` and always generated a `FloatVector` schema in
quick-create mode. A typo such as `InvalidVectorType` could return
success and mask client-side bugs.
Binary and sparse quick-create requests without `metricType` could also
create the collection first and then fail index creation because the old
fallback metric was `COSINE`, which is incompatible with those vector
types.
For #51084, top-level `consistencyLevel` was not present in
`CollectionReq`, so JSON decoding silently dropped it. Invalid values
such as `"consistencyLevel": "Invalid"` were ignored and the collection
could be created with the default consistency level.
## Tests
- `./bin/gofumpt -w internal/distributed/proxy/httpserver/request_v2.go
internal/distributed/proxy/httpserver/handler_v2.go
internal/distributed/proxy/httpserver/handler_v2_test.go`
- `./bin/gci write internal/distributed/proxy/httpserver/request_v2.go
internal/distributed/proxy/httpserver/handler_v2.go
internal/distributed/proxy/httpserver/handler_v2_test.go
--skip-generated -s standard -s default -s
"prefix(github.com/milvus-io)" --custom-order`
- `git diff --check`
- `python3 -m py_compile
tests/restful_client_v2/testcases/test_collection_operations.py`
- `rg -n
"string\\(paramtable\\.(BinaryVectorDefaultMetricType|SparseFloatVectorDefaultMetricType)\\)|interface\\("
internal/distributed/proxy/httpserver/handler_v2.go
internal/distributed/proxy/httpserver/handler_v2_test.go`
Attempted locally but blocked by missing native dependencies:
```
go test ./internal/distributed/proxy/httpserver -run 'TestCreateCollection(QuickVectorFieldType|TopLevelConsistencyLevel)$' -count=1
# missing rocksdb.pc and milvus_core.pc
```
CI investigation before this push:
```
# build-ut-cov/code-check failed on c6 predecessor because of unconvert:
# internal/distributed/proxy/httpserver/handler_v2.go:2054:16 unnecessary conversion
# internal/distributed/proxy/httpserver/handler_v2.go:2056:16 unnecessary conversion
# internal/distributed/proxy/httpserver/handler_v2_test.go:2185:28 unnecessary conversion
# internal/distributed/proxy/httpserver/handler_v2_test.go:2217:28 unnecessary conversion
# Fixed by removing string(...) around paramtable default metric constants.
# e2e-default failed in job 40810 on:
# testcases/test_geometry_operations.py::TestGeometryCollection::test_spatial_query_and_search[False-True-sealed-ST_CROSSES]
# error: Connection refused to /v2/vectordb/collections/flush on port 19530
# the same job showed querynode/streamingnode restarts and proxy ContainerCreating during cleanup.
# REST collection tests in that job had already passed, so this looked unrelated to this PR.
# e2e-default failed again in dispatcher 13914 / job-1 32439 on:
# testcases/test_collection_operations.py::TestCreateCollectionNegative::test_create_collection_quick_setup_with_invalid_consistency_level
# root cause: the test used collection_create(), whose wrapper auto-injects params.consistencyLevel=Strong.
# That made the request hit the new conflict branch instead of the intended top-level invalid consistencyLevel branch.
# Fixed in 3e190eee56 by sending the request through raw client.post() to avoid wrapper mutation.
```
Signed-off-by: Yanliang Qiao <yanliang.qiao@zilliz.com>
## Summary
Move the RBAC check before the hook interceptor.
Write the resolved roles for the current request into the request
context.
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
issue: https://github.com/milvus-io/milvus/issues/44358
design doc:
docs/design-docs/design_docs/20260609-external-snapshot-export-restore.md
Part 1/3 of the external snapshot cross-bucket restore stack.
This PR defines the API contract and entry surfaces for external
snapshot export and restore:
- Adds the consolidated design document for cross-bucket external
snapshot restore.
- Adds public gRPC, REST, and Go SDK API surfaces for
RestoreExternalSnapshot and ExportSnapshot.
- Adds internal DataCoord proto plumbing and generated code needed by
later implementation PRs.
- Wires Proxy RBAC grouping and database interceptor behavior for the
new APIs.
- Keeps the request contract on a single external_spec field and keeps
db_name for namespace routing rather than permission scoping.
Validation copied from the commit:
- GOTOOLCHAIN=go1.25.10 go test -c -tags dynamic,test -gcflags="all=-N
-l" -ldflags="-r ${RPATH}" -o /tmp/datacoord-commit1.test
github.com/milvus-io/milvus/internal/datacoord
- cd client && GOTOOLCHAIN=go1.25.10 go test -c -o
/tmp/client-milvusclient-commit1.test ./milvusclient
- internal/proxy package compile was blocked locally by missing C++
header internal/core/output/include/segcore/search_result_export_c.h
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: https://github.com/milvus-io/milvus/issues/50571
design doc: docs/design-docs/design_docs/20260624-function-chain-api.md
Add the FunctionChain proto-to-runtime path for ordinary Search L2
rerank. The change converts public FunctionChain protos into ChainRepr,
derives caller-independent read/write metadata, validates
Search-specific inputs in Proxy, and executes public chains through the
existing rerank pipeline.
Add request validation for duplicate stages, unsupported stages,
unsupported system inputs/outputs, unknown schema fields, unsupported
input field types, and hybrid-search usage. Extend REST v2 request
conversion to accept function_chains.
Replace score_combine with num_combine and add typed parameter readers,
repr-based FuncChain construction, chain optimization/pruning helpers,
and model-rerank parameter handling. Add coverage for chain repr
conversion, function/operator behavior, Proxy planning, search pipeline
integration, REST conversion, and REST API cases.
Add the Function Chain API design document and update milvus-proto to
the latest upstream pseudo-version.
Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
## Summary
Reject REST import create requests when `options.auto_commit` is present
but not `"true"` or `"false"`. The omitted option remains accepted and
keeps the default auto-commit behavior.
Fixes#50460
## Changes
- Add strict REST JSON validation for `options.auto_commit`.
- Add handler coverage for omitted, valid, null, empty, and invalid
values.
## Test Plan
- `GOTOOLCHAIN=auto make static-check`
- `GOTOOLCHAIN=auto go test ./util/merr` from `pkg/`
- `GOTOOLCHAIN=auto go test ./internal/json`
Note: local `static-check` used `milvus-cpp-share` to reuse the main
checkout C++ artifacts and a temporary ignored
`cmd/tools/migration/legacy/legacypb/legacy.pb.go` copied from the main
checkout, because this generated file is ignored but required by the
migration package during local lint loading.
---------
Signed-off-by: Yihao Dai <yihao.dai@zilliz.com>
## Summary
- Add authorization for REST v2 import commit and abort endpoints.
- Resolve the import job collection via GetImportProgress before
checking PrivilegeImport.
- Cover denied non-privileged users, allowed root users, and
authorization-disabled behavior.
Fixes#50458
## Test Plan
- [x] make build-cpp-with-unittest
- [x] go test -tags dynamic,test -gcflags=\"all=-N -l\" -count=1
./internal/distributed/proxy/httpserver/... -run
'TestCommitImportJob|TestAbortImportJob'
- [x] go test -tags dynamic,test -gcflags=\"all=-N -l\" -count=1
./internal/distributed/proxy/httpserver/... -skip '^TestSearchV2$'
- [x] GOTOOLCHAIN=auto make static-check
## Notes
- Full httpserver package test currently has an unrelated existing
TestSearchV2/search#09 expectation mismatch: expected `Mismatch type
uint8`, actual contains `Mismatch type []uint8`.
---------
Signed-off-by: Yihao Dai <yihao.dai@zilliz.com>
Honor top-level partitionNames for REST v2 hybrid_search so it only
searches requested partitions.
issue: #50396
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
- Persist role descriptions when creating roles, expose them in role
list/describe results, and support updating descriptions through the
AlterRole path without changing role names.
- Store role descriptions in the role value body while keeping role
names in keys, and tolerate legacy empty role values plus undecodable
stored values by returning an empty description for that role row.
- Reject built-in/default roles and over-limit descriptions before WAL
writes, and share the role-description length validator between proxy
and rootcoord.
related: #50183
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
- Wire credential descriptions through create/update/read paths,
including proxy, HTTP v2, Go SDK options, length validation, optional
update presence handling, and internal credential proto/model
persistence.
- Preserve existing encrypted passwords and descriptions during
RootCoord credential updates when either field is omitted, reject
all-empty or partial direct password updates, and keep sha256 passwords
cache-only.
- Skip auth-cache refresh for description-only credential updates so
existing authentication state is not blanked, while still refreshing
caches for password updates.
- Return persisted user descriptions from select_user and HTTP v2
describe_user, including include_role_info=false, and reuse
already-loaded credential rows in user listing and RBAC backup to avoid
duplicate etcd reads while ignoring malformed credential keys.
- Use the upstream milvus-proto go-api/v3 version that includes the
credential description proto change; the temporary fork replaces in
root, pkg, client, and tests/go_client modules have been removed.
design doc:
docs/design-docs/design_docs/20260601-rbac-user-description.md
related: #50179
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
issue: #47420
## What this PR does
Project-wide migration of raw `fmt.Errorf` / `errors.New` in function
bodies onto
the `merr` framework, plus the Sys-vs-Input error classification and the
machinery it drives (retriability, fine-grained metrics, segcore
unification),
plus the convention docs and a linter that keeps it from regressing.
Scope: storage, proxy, coordinators (root/data/query), query node, data
node,
`pkg/util` & `internal/util`, expression parser, message queue,
streaming, and
misc packages. Bare raw-error usages went from ~3000 to a ~340 allowlist
(package-level sentinels / build-tag / test sites).
---
## How to review this PR
It is large but the vast majority is mechanical. Changes fall into three
tiers;
spend review budget on Part 2 and Part 3.
### Part 1 — Mechanical standardization (low risk, verify by rule)
Each converted call follows one of a small fixed set of rules. To
review, check
that each site obeys the matching rule rather than reading every line:
| Pattern | Rule |
|---|---|
| `fmt.Errorf("...")` originating a new error | →
`merr.WrapErrXxxMsg("...")` with a code matching the failure's meaning |
| Adding context to an existing typed error | → `merr.Wrap(err, "...")`
/ `merr.Wrapf(...)` — **preserves** the inner code (never `WrapErr*Err`,
which overwrites it) |
| Errors inside the streaming subsystem | → `status.New*` factories
(StreamingError), **not** merr — this is the component-internal dialect
(see `docs/dev/error_handling_guide.md`) |
| Low-level / control-flow signal caught by `errors.Is` | → kept as a
package-level `errors.New` sentinel (lowercase, same-package) |
Conventions are documented in `docs/dev/error_handling_guide.md`
(how-to) and
`docs/dev/error_sentinel_convention.md` (rules + audit). A
`gocritic`/`ruleguard`
rule (`rawmerrerror`, in `rules.go`) enforces "no raw `return
errors.New/fmt.Errorf`"
under `make verifiers`.
### Part 2 — Behavior changes (review these closely)
These are the sites where the wire contract or runtime behavior changes,
not just
the source text. Listed by category; representative locations given,
full set in
the diff.
**A. gRPC wire-code shifts: `UnexpectedError(1)/Code 65535` → typed
code.**
Where a handler previously returned a raw error (collapsed to
`Code=65535` on the
wire), it now returns a typed merr, so the client sees a real code. The
most
common shift is to `IllegalArgument(5)/Code 1100` (ParameterInvalid).
Touch
points include datanode task handlers (CreateTask/Query/Drop), proxy
Upsert,
querynode GetMetrics, datacoord CreateIndex, httpserver query-response
builder,
and typeutil schema validation. One code refinement: an index-param
validation
moved `1100` → `1101` (ParameterMissing). **Client/SDK assertions and
any code
that switched on `Code=65535` for these paths must be re-checked** (the
go_client
e2e assertions were already aligned in this PR).
**B. Prometheus `status` label contract change (externally visible).**
The proxy metric's coarse `fail` / `rejected` values are split into
`fail_input` / `fail_system` and `rejected_user` / `rejected_system` (in
`requestutil.ParseMetricLabel`; auth/privilege rejections count as
`rejected_user`), so dashboards can attribute a failure to caller vs
operator.
**Dashboards/alerts querying `status="fail"` must migrate to
`status=~"fail_.*"`, and `status="rejected"` to
`status=~"rejected_.*"`.** The
in-repo Grafana dashboard is already migrated; external dashboards built
on the
old values silently go empty after upgrade. This is the one change that
requires an ops-side migration.
**C. Retriability semantics.**
- C1: `merr.Status(err)` now forces `Retriable=false` when the error is
an
`InputError` — a malformed request can never succeed on blind retry, so
clients
never get the self-contradictory "your input is wrong but you may
retry".
- C2: `retry.Do` short-circuits an `InputError` (non-retriable) — **but
only when
the caller did not pass a `RetryErr` predicate**. The check is an
`if c.isRetryErr != nil { ... } else if InputError { ... }` *mutually
exclusive*
branch (`pkg/util/retry/retry.go`): an explicit `RetryErr` takes
precedence and
bypasses the InputError abort. `retry.Handle` deliberately does **not**
apply
the InputError abort (its callers signal abort via `shouldRetry=false`).
Four
flusher startup callsites that must retry through transient "not ready"
errors
were given explicit `RetryErr` escape hatches.
**D. segcore (C++→Go) error classification.**
A single shared Go-side table (`pkg/util/merr/segcore.go`) maps each
segcore code
to a merr sentinel + InputError/signal category, replacing scattered
hand-written
`if errorCode == ...` switches in the cgo wrappers. **Wire `Code` values
change
for every segcore pass-through error, not just the remapped ones.**
Named
sentinels remap (C++ `2003` → merr `2001`, `2033` → `2002`,
Folly/Knowhere codes
likewise); **all remaining pass-through codes (`2004`–`2043`, previously
surfaced to clients as raw C++ enum values) now serialize as `2000`**
(`ErrSegcore`), with the original C++ code preserved in the `Reason`
text
(`segcoreCode=...`); unknown/future codes collapse to `2000` as well
(pinned by
the `wire_code_projection` test). Transient segcore classes (object
storage /
file IO / OOM / mmap / FieldNotLoaded — 11 codes) now report
`Retriable=true`.
**Any client switching on raw segcore codes in the `2004`–`2043` range
must be
re-checked**; the in-Reason code remains available for diagnostics.
Signal
codes (PretendFinished / FollyCancel) are recognized centrally.
`errors.Is`-based
control flow on these (e.g. scheduler skip/retry) is preserved.
**E. InputError classification (25 sentinels + dynamic marks).**
25 sentinels in `errors.go` carry `WithErrorType(InputError)` (the
Collection /
ResourceGroup / Database families, `ErrIndexDuplicate`,
`ErrParameterInvalid`,
`ErrPrivilegeNotAuthenticated`, `ErrImportFailed`, `ErrQueryPlan`, ...),
plus dynamic
marks for the 8 segcore input codes (ExprInvalid, DimNotMatch,
MetricTypeInvalid, FieldIDInvalid, ...) and
`WrapErrAsInputError`. The widest blast radius is `ErrParameterInvalid`
(1100):
~2335 `WrapErrParameterInvalid*` callsites now classify as input /
non-retriable. Because of C1/C2 this changes retriability for
any path that returns these. **The audit to confirm no transient path
was
mis-marked is the single most important review item** (see Part 3). One
reverse
correction: storage field-stats parsing moved from `ErrParameterInvalid`
(input)
to `ErrDataIntegrity` — a corrupted stored stat is data corruption, not
user
input.
### Part 3 — Known risks & traps (called out proactively)
1. **`merr.Wrap` vs `WrapErr*Err` (code-masking).** `WrapErr*Err` builds
a
`wrappedMilvusError{sentinel: ErrServiceInternal}` whose `code()`
returns the
*outer* sentinel — it overwrites the inner typed code and hides the
`errors.Is`
chain. This is intentional (use it to *deliberately* downgrade), but it
was a
recurring conversion defect; the rule "add context with `merr.Wrap`,
downgrade
with `WrapErr*Err`" is enforced by convention and reviewed across the
diff.
2. **InputError × `retry.Do` blast radius.** Marking a sentinel
`InputError` makes
any `retry.Do(...)` without a `RetryErr` predicate stop retrying it.
Reviewers
should sanity-check that no transient use of the 19 newly-marked
sentinels
(especially `ErrParameterInvalid`) sits inside a retry loop that needed
to keep
spinning. The known flusher cases were handled (see C2).
3. **The ~340 raw-error allowlist.** What remains as bare `errors.New`
is, by
design: package-level sentinels (caught by `errors.Is`), `//go:build
test`
sites, and out-of-band trees (`cmd/`, `tests/`, codegen, walimpls). The
linter
only bans the *direct-return* form; assignment-then-return escapes and
the full
no-exceptions ban are deferred to an AST-based linter (Tier 2,
documented).
4. **segcore C++ second step deferred.** This PR unifies classification
on the Go
side; splitting the dual-semantic C++ codes at the source is a
follow-up.
---
## Validation
- `make verifiers`: Go side clean (gofmt + static-check across modules,
including
the new `rawmerrerror` rule with a 0-hit baseline repo-wide).
- `make test-go`: passing; the one real regression introduced (a
datanode
`invalid_task_type` assertion shifting `1` → `5` from a ParameterInvalid
conversion) was fixed in-tree.
- go_client e2e CreateIndex assertions aligned to the new merr messages.
---------
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fixes: #49890
This PR validates the REST `Request-Timeout` header instead of silently
ignoring invalid values.
Changes:
- Return `ErrParameterInvalid` when `Request-Timeout` is present but
cannot be parsed as an integer.
- Preserve the underlying `strconv.ParseInt` error in the response
message.
- Add timeout middleware coverage for invalid values such as `3.5` and
`abc`.
Test:
- `go test -tags dynamic,test -gcflags="all=-N -l" -count=1
./internal/distributed/proxy/httpserver -run 'TestTimeoutMiddleware'`
*(fails locally because generated C/CGo headers are unavailable:
`storage/loon_ffi/external_spec_c.h`, `C.CArrowReaderConfig`)*
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
issue: #49967
## What was changed
- Add `X-Milvus-Trace-Id` to REST responses for `/v1/vector/...` and
`/v2/vectordb/...` routes.
- Create a REST entry trace span before access logging so response
headers and REST logs use the same request trace ID.
- Keep the v2 wrapper from overwriting the entry trace ID when one
already exists.
- Add timeout and early-return coverage for the trace ID header.
## Verification
- `gofumpt -l internal/distributed/proxy/httpserver/handler_v2.go
internal/distributed/proxy/httpserver/traceid_test.go
internal/distributed/proxy/httpserver/utils.go
internal/distributed/proxy/httpserver/constant.go
internal/distributed/proxy/httpserver/timeout_middleware.go
internal/distributed/proxy/service.go`
- `git diff --check`
- `zsh -c 'source scripts/setenv.sh && go test -v -count=1 -tags
dynamic,test -gcflags="all=-N -l" -ldflags="-r
${MILVUS_WORK_DIR}/cmake_build/lib -r
${MILVUS_WORK_DIR}/internal/core/output/lib"
./internal/distributed/proxy/httpserver -run
"TestHTTPReturnWritesTraceIDHeader|TestHTTPReturnSkipsTraceIDHeaderWhenMissing|TestTraceIDHandlerFuncCreatesRequestTraceID|TestTraceIDHandlerFuncCoversV1AndV2RestRoutes|TestTraceIDHandlerFuncCreatesTraceIDWhenUnsampled|TestTraceIDHandlerFuncReturnsUniqueTraceID|TestTraceIDHandlerFuncCoversEarlyAbort|TestWrapperPostBadJSONReturnsTraceIDHeader|TestWrapperPostKeepsRequestTraceID|TestTimeoutMiddlewareReturnsTraceIDHeader"
-timeout 300s'`
- Manual curl verification against local standalone: REST response
header and REST access log trace IDs matched; 10 repeated search
requests returned 10 valid and unique 32-hex trace IDs.
## Note
Full `go test ./internal/distributed/proxy/httpserver` currently hits an
existing unrelated `TestSearchV2/search#09` assertion mismatch: expected
`Mismatch type uint8`, actual contains `Mismatch type []uint8 ...`.
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
## Summary
Implements Two-Phase Commit (2PC) for Import in primary/secondary
replication clusters. Data stays invisible (`is_importing=true`) until
an explicit commit signal is delivered via WAL, ensuring primary and
secondary clusters reach the same visible state at the same logical
position.
- **New proto**: `CommitImport=44`, `RollbackImport=45` WAL message
types; `Uncommitted=8`, `Committing=9` `ImportJobState` values;
`CommitImport`, `AbortImport`, `HandleCommitVchannel` RPCs on DataCoord;
`committed_vchannels` + `auto_commit` fields on `ImportJob`
- **WAL broadcast**: DataCoord broadcasts
`CommitImportMessage`/`RollbackImportMessage` to all vchannels via DDL
broadcast; CDC replicates to secondary clusters verbatim
- **DDL ack callbacks**: `commitImportV2AckCallback` CAS
`Uncommitted→Committing`; `rollbackImportV2AckCallback` CAS `*→Failed` +
segment drop
- **WAL flusher**: `wal_flusher.dispatch()` intercepts
`CommitImportMessage` per-vchannel, calls `wbMgr.FlushChannel` +
`DataCoord.HandleCommitVchannel`; no-op handler for
`RollbackImportMessage`
- **ImportChecker**: new `Uncommitted` case (auto-commits when
`auto_commit=true`); new `Committing` case (transitions to `Completed`
when all vchannels confirmed)
- **auto_commit option**: default `true` (backward compatible); `false`
lets replication platform control commit timing
- **RESTful API**: `POST /v2/vectordb/jobs/import/commit` and `POST
/v2/vectordb/jobs/import/abort`; `GetImportProgress` surfaces
`Uncommitted` and `Committing` states
**Out of scope**: `commit_timestamp` propagation (handled in companion
PR)
## Test Plan
- [ ] Unit tests for `IsAutoCommit` helper
- [ ] Unit tests for `HandleCommitVchannel` idempotency
- [ ] Unit tests for `CommitImport`/`AbortImport` RPC handlers (state
validation, mutex, broadcast)
- [ ] Unit tests for DDL ack callbacks (CAS races for commit/abort)
- [ ] Unit tests for `ImportChecker` `Uncommitted`/`Committing` cases
- [ ] Unit tests for WAL flusher dispatch (CommitImport/RollbackImport
cases)
- [ ] Unit tests for RESTful commit/abort handlers (valid jobId, invalid
jobId error path)
- [ ] `GetImportProgress` surfaces `Uncommitted` and `Committing` states
- [ ] E2E: non-replication cluster with `auto_commit=true` (default)
behaves identically to pre-2PC
issue: #48525
design doc: https://github.com/milvus-io/milvus-design-docs/pull/29🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Yihao Dai <yihao.dai@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
- Enforce the nullable vector compact-storage contract across query,
storage, import, segcore, and index build paths.
- Keep `ValidData` as the logical row source and only read/write
physical vector payload for valid rows.
- Preserve master nullable `ArrayOfVector` behavior: it is supported as
row-dense vector-array data with null-row placeholders, but it is
intentionally excluded from compact nullable-vector helpers.
issue: #49881
issue: #49974
issue: #49992
## What changed
- Add shared nullable vector helpers for supported compact vector types
and logical-to-physical compact row mapping.
- Fix proxy search result reorder/order_by when output fields include
nullable compact vectors.
- Fix queryutil row count, row-size accounting, slice/order/merge paths
for nullable compact vector payloads.
- Fix storage payload reader/writer, row serde, record builder metadata,
data sorter, and insert-data merge for nullable compact vectors.
- Fix parquet sparse vector import and numpy nullable dense vector
`ValidData` generation.
- Fix C++ growing flush, external take output, result merge, and bitset
transform for nullable compact vector rows.
- Skip 0-row Knowhere build for all-null nullable vectors while
preserving valid-data-only index metadata.
- Keep nullable `ArrayOfVector` on master on its existing row-dense
path; only V1 storage rejects nullable `ArrayOfVector`.
- Add regression coverage in Go unit tests, C++ gtests, and go-client
nullable vector e2e scenarios.
## Review notes
- Review 3 / 4 - master supports nullable `ArrayOfVector`. This PR no
longer rejects it at schema, proxy, JSON/CSV/Parquet import, or
add-field validation. `ArrayOfVector` remains excluded from
`IsSupportedNullableVectorType` because that helper means compact
nullable vector payloads; `ArrayOfVector` uses row-dense data with empty
placeholder rows for null logical rows.
- Review 10 - nullable sparse vector paths were audited around the
compact layout: `ValidData` is logical-row sized, while sparse
`Contents` only stores valid physical rows.
Reader/writer/serde/import/query reorder paths derive
logical-to-physical offsets from `ValidData` instead of indexing sparse
contents by logical row.
- Storage V1 note - nullable `ArrayOfVector` is still rejected only in
V1 storage-format paths. That does not remove master V2 nullable
`ArrayOfVector` support.
## Test plan
- [x] `git diff --check HEAD`
- [x] `make static-check`
- [x] `cmake --build cmake_build --target all_tests -j8`
- [x] C++ `all_tests` targeted nullable vector / nullable VectorArray
cases
- [x] `go test -count=1 ./internal/util/queryutil`
- [x] `cd pkg && go test -count=1 ./util/typeutil`
- [x] `cd pkg && go test -count=1 ./util/funcutil`
- [x] `go test -tags dynamic,test -gcflags="all=-N -l" -count=1
./internal/util/importutilv2/parquet`
- [x] `go test -tags dynamic,test -gcflags="all=-N -l" -count=1
./internal/proxy -run
'Test_validateUtil_checkArrayOfVectorFieldData|TestFillWithNullValue_Geometry|Test_validateUtil_checkAligned|Test_validateUtil_Validate'`
- [x] `go test -tags dynamic,test -gcflags="all=-N -l" -count=1
./internal/datanode/importv2`
- [x] `go test -tags dynamic,test -gcflags="all=-N -l" -count=1
./internal/storage -run '<nullable-vector targeted tests>'`
Signed-off-by: marcelo-cjl <marcelo.chen@zilliz.com>
issue: #49707
- query node scheduler: make read tasks context-aware, use an unbuffered
add path, and cleanup canceled or near-deadline queued tasks before
queue limit checks
- query grouping: cap grouped NQ at 16 and add an NQ merge ratio guard
to avoid merging small queries into much larger groups
- REST timeout: propagate request timeout as a context deadline so
downstream query tasks receive timeout timestamps
- scheduler metrics: add ready NQ plus queue and execution duration
metrics while removing obsolete receive queue configuration
- proxy config: reduce the proxy task queue default to 256
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Related to #49981
The REST timeout middleware could return a timeout response while the
handler goroutine continued using the original Gin context. Once Gin
recycled that context, late handler writes could race with a later
request and trigger concurrent map writes.
Run timeout-wrapped handlers with a copied Gin context and a fully
buffered response recorder instead. The original context and real
response writer now remain owned by the middleware goroutine, while
normal completion explicitly propagates selected metadata and commits
the recorder. Timeout closes the recorder and writes the 408 response
through the real writer, so late handler writes are discarded safely.
Add tests for recorder isolation, normal buffered commits, metadata and
abort propagation, late writes after timeout, and race validation of the
late-write timeout path.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
## Summary
Add struct array (ArrayOfVector) support to the RESTful v2 API:
- Schema: declare struct fields with sub-fields in `createCollection`
- Data: ingest struct array rows in `insert` / `upsert`, auto-detecting
per-element-object vs. parallel-column row shapes
- Search: query a struct sub-vector via `annsField:
"<struct>.<subField>"`,
with dim/type validated against the schema
- Reject `nullable` / `defaultValue` on sub-fields (not supported by
engine)
## Test plan
- [x] Unit tests in `struct_array_v2_test.go` cover schema validation,
row parsing for both shapes, and search request building
- [ ] Manual e2e via curl against a local standalone
issue: #45496
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Related to #49894
restfulSizeMiddleware wrapped metrics stats around the Gin context,
which can drop the original HTTP request cancellation signal before
timeoutMiddleware for v2 entity RESTful APIs.
Use the original request context as the metrics parent so client
disconnects continue propagating through the timeout and proxy task
chain. Add regression coverage for the
restfulSizeMiddleware(timeoutMiddleware(...)) chain.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #49241
This PR exposes partial update field operations through REST upsert so
ARRAY_APPEND and ARRAY_REMOVE requests can reach proxy validation.
Changes:
- Accept fieldOps in v1 and v2 REST upsert request payloads.
- Convert REST op strings into FieldPartialUpdateOp on UpsertRequest.
- Add request helper coverage for supported and unsupported ops.
Validation:
- make milvus
- Manual REST validation on standalone:
- v2 ARRAY_APPEND updated tags from [1,2] to [1,2,3,4]
- v2 ARRAY_REMOVE updated tags to [1,3]
- v1 ARRAY_APPEND updated tags to [1,3,8]
- unsupported op ARRAY_EXTEND returned parameter error
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
* Optimize metadata and request handling hot paths
* Make replication pending-message queue capacity and max size
configurable instead of hard-coded.
* Batch QueryCoord collection metadata saves with MultiSave to avoid
oversized single writes while still reducing per-key etcd operations.
* Reduce avoidable allocations in DataCoord catalog tests, access log
list formatting, HTTP array joining, and HTTP JSON field extraction.
* Use WAL-specific message ID decoding during flusher recovery.
* Use set-based RootCoord collection-name filtering and structured
collection-not-found errors.
issue: #44452
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
issue: milvus-io/milvus#45881
This change returns the external spec used by refresh external
collection jobs through the public refresh job info paths.
Changes:
- Adds external spec to client refresh job info conversion.
- Returns redacted external spec from proxy refresh job responses.
- Exposes externalSpec in REST v2 refresh job response maps.
- Bumps milvus-proto go-api/v3 to the version containing
RefreshExternalCollectionJobInfo.external_spec.
Validation:
- go test -tags dynamic,test
github.com/milvus-io/milvus/client/v2/milvusclient -run
'Test.*External|Test.*Refresh' -count=1 -v
Note:
- internal/proxy and httpserver targeted tests were blocked locally by
missing C++ pkg-config artifacts: milvus_core, rocksdb, and
milvus-storage.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #49398
Bump Go module references from pkg/v2 and milvus-proto/go-api/v2 to
pkg/v3 and milvus-proto/go-api/v3 so the client tracks the Milvus 3.x
release line.
This prepares the repository for the upcoming 3.x.y release by aligning
imports, module dependencies, and proto API references with the new
major-version module paths.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #45881
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260105-external_table.md
## What
Support external collection operations in REST v2.
- Add schema-level external source/spec handling for REST v2 create and
describe collection.
- Add REST v2 external collection refresh, describe-job, and list-job
routes.
- Reject top-level external config and quick-create external collection
requests.
- Reject add-field requests that try to add external field mappings.
- Support listing all refresh jobs when collection name is empty.
## Tests
- `go build -tags dynamic,test github.com/milvus-io/milvus/internal/...
github.com/milvus-io/milvus/pkg/...`
- `go test -tags dynamic,test -gcflags="all=-N -l" -ldflags="-r
${RPATH}" github.com/milvus-io/milvus/internal/proxy -run
'TestProxy_ListRefreshExternalCollectionJobs_(ListAll|ByCollection)'
-count=1 -v`
Known local limitation:
- Full `internal/datacoord` package coverage run is blocked by an
existing compaction test failure:
`TestCompactionTriggerManagerSuite/TestGetExpectedSegmentSize/all_DISKANN`
expects `209715200` but gets `104857600`.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
## Summary
- Inject Milvus zap logger into hook plugin via `SetZapLogger` type
assertion after plugin load, enabling the plugin to use Milvus's
centralized logging
- Inject connection manager into hook plugin via `SetClientInfoProvider`
type assertion, allowing the plugin to look up SDK info (sdk_type,
sdk_version, host) for non-Connect requests by connection identifier
- Pass `*gin.Context` directly via `GinParamsKey` instead of
`ginCtx.Keys` map, enabling the hook plugin to access `ClientIP()`,
`FullPath()`, `GetHeader()` and other gin context methods for RESTful
requests
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
- REST handlers for `/roles/grant_privilege_v2` and
`/roles/revoke_privilege_v2` pass V1 fullMethod `OperatePrivilege`
instead of `OperatePrivilegeV2` to `wrapperProxy`, causing Hook
interceptors to reject the request as a denied API
- Also fix `routeToMethod` metrics map entries for the same endpoints
issue: #48115
Signed-off-by: sunby <sunbingyi1992@gmail.com>
## Summary
- Add `GetCollectionName()` method to 12 RESTful API request types that
were missing it
- Ensures `ProxyFunctionCall` metrics report the correct collection name
label for all RESTful endpoints
- Affected types: `RenameCollectionReq`, `QueryReqV2`,
`CollectionIDReq`, `CollectionFilterReq`, `CollectionDataReq`,
`SearchReqV2`, `HybridSearchReq`, `PartitionsReq`, `GrantV2Req`,
`IndexParamReq`, `CollectionReq`, `RunAnalyzerReq`
issue: #48461
## Test plan
- [ ] Verify existing unit tests pass
- [ ] Confirm RESTful API metrics now include collection name for
search/query/hybrid_search endpoints
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
timeoutMiddleware runs handlers in a separate goroutine, which writes to
gCtx.Keys directly (e.g. gCtx.Keys["traceID"] = traceID). Meanwhile
AccessLogMiddleware reads from the same map via RestfulInfo in the
original goroutine. Both bypass gin.Context's built-in RWMutex, causing
"concurrent map read and map write" panics.
Fix: store a *gin.Context reference in RestfulInfo and replace all
direct map reads (i.params.Keys[...]) with i.ctx.Get(), and replace the
direct map write with gCtx.Set(). These methods use gin's internal
RWMutex for safe concurrent access.
issue: #48316
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
The timeout path swapped gCtx.Writer to the raw ResponseWriter while the
handler goroutine was still running, causing concurrent writes to
http.Header and nil pointer dereferences on the freed buffer.
Three fixes applied:
- Write timeout response directly to the original ResponseWriter without
swapping gCtx.Writer, so the handler always sees the buffered Writer
that safely no-ops after timeout
- Move timeout/body-nil guard checks inside the mutex in Write() and
WriteHeader() to eliminate TOCTOU race on buffer free
- Use atomic.Bool for the timeout flag so Status() reads are safe
without the mutex; add WriteHeaderNow() override to prevent bypassing
the mutex
issue: #48316
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When ResponseWriter.Write fails in timeoutMiddleware (e.g., client
disconnected), the code panics. While gin.Recovery() catches it, mass
concurrent failures cause excessive panic/recover overhead (stack
traces, goroutine scheduling), leading to high system load.
Replace panic(err) with log.Warn + proper resource cleanup + return,
consistent with error handling in the rest of the HTTP handler layer.
issue: #48269
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #43966
Previously all grpc metrics update moved to interceptor while restful
metrics logic is not updated.
This PR add similar logic to unified entry of restful v2: `wrapperPost`
handling metrics update logic.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue : https://github.com/milvus-io/milvus/issues/41746
This PR adds MinHash "DIDO" (Data In, Data Out) support to Milvus, which
allows computing MinHash signatures on-the-fly during search operations
instead of requiring pre-stored vectors.
Key changes:
- Implemented SIMD-optimized C++ MinHash computation (AVX2/AVX512 for
x86, NEON/SVE for ARM)
- Added runtime CPU detection and function hooks to automatically select
the best SIMD implementation
- Integrated MinHash computation into search pipeline (brute force
search, growing segment search)
- Added support for LSH-based MinHash search with configurable band
width and bit width parameters
- Enabled direct text-to-signature conversion during query execution,
reducing storage overhead
This enables efficient text deduplication and similarity search without
storing pre-computed MinHash vectors.
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
Related to #46818
When a collection has autoID enabled and `allow_insert_auto_id` property
set to true, the RESTful v2 insert API was incorrectly rejecting
requests that included the primary key field. This fix adds proper
checking of the `allow_insert_auto_id` flag in the `anyToColumns`
function.
Changes:
- Read `allow_insert_auto_id` property from collection schema
- Skip PK field only when autoID is enabled AND allow_insert_auto_id is
false
- Allow PK field in insert request when allow_insert_auto_id is true
- Filter out empty PK column when autoID is enabled and user didn't
provide PK
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #46635
## Summary
- Fix spelling error in constant name: `CredentialSeperator` ->
`CredentialSeparator`
- Updated all usages across the codebase to use the correct spelling
## Changes
- `pkg/util/constant.go`: Renamed the constant
- `pkg/util/contextutil/context_util.go`: Updated usage
- `pkg/util/contextutil/context_util_test.go`: Updated usage
- `internal/proxy/authentication_interceptor.go`: Updated usage
- `internal/proxy/util.go`: Updated usage
- `internal/proxy/util_test.go`: Updated usage
- `internal/proxy/trace_log_interceptor_test.go`: Updated usage
- `internal/proxy/accesslog/info/util.go`: Updated usage
- `internal/distributed/proxy/service.go`: Updated usage
- `internal/distributed/proxy/httpserver/utils.go`: Updated usage
## Test Plan
- [x] All references updated consistently
- [x] No functional changes - only constant name spelling correction
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
- Core invariant: the separator character for credentials remains ":"
everywhere — only the exported identifier was renamed from
CredentialSeperator → CredentialSeparator; the constant value and
split/join semantics are unchanged.
- Change (bug fix): corrected the misspelled exported constant in
pkg/util/constant.go and updated all references across the codebase
(parsing, token construction, header handling and tests) to use the new
identifier; this is an identifier rename that removes an inconsistent
symbol and prevents compile-time/reference errors.
- Logic simplified/redundant work removed: no runtime logic was removed;
the simplification is purely maintenance-focused — eliminating a
misspelled exported name that could cause developers to introduce
duplicate or incorrect constants.
- No data loss or behavior regression: runtime code paths are unchanged
— e.g., GetAuthInfoFromContext, ParseUsernamePassword,
AuthenticationInterceptor, proxy service token construction and
access-log extraction still use ":" to split/join credentials; updated
and added unit tests (parsing and metadata extraction) exercise these
paths and validate identical semantics.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: majiayu000 <1835304752@qq.com>
Signed-off-by: lif <1835304752@qq.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
issue： #46033
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Pull Request Summary: Entity-Level TTL Field Support
### Core Invariant and Design
This PR introduces **per-entity TTL (time-to-live) expiration** via a
dedicated TIMESTAMPTZ field as a fine-grained alternative to
collection-level TTL. The key invariant is **mutual exclusivity**:
collection-level TTL and entity-level TTL field cannot coexist on the
same collection. Validation is enforced at the proxy layer during
collection creation/alteration (`validateTTL()` prevents both being set
simultaneously).
### What Is Removed and Why
- **Global `EntityExpirationTTL` parameter** removed from config
(`configs/milvus.yaml`, `pkg/util/paramtable/component_param.go`). This
was the only mechanism for collection-level expiration. The removal is
safe because:
- The collection-level TTL path (`isEntityExpired(ts)` check) remains
intact in the codebase for backward compatibility
- TTL field check (`isEntityExpiredByTTLField()`) is a secondary path
invoked only when a TTL field is configured
- Existing deployments using collection TTL can continue without
modification
The global parameter was removed specifically because entity-level TTL
makes per-entity control redundant with a collection-wide setting, and
the PR chooses one mechanism per collection rather than layering both.
### No Data Loss or Behavior Regression
**TTL filtering logic is additive and safe:**
1. **Collection-level TTL unaffected**: The `isEntityExpired(ts)` check
still applies when no TTL field is configured; callers of
`EntityFilter.Filtered()` pass `-1` as the TTL expiration timestamp when
no field exists, causing `isEntityExpiredByTTLField()` to return false
immediately
2. **Null/invalid TTL values treated safely**: Rows with null TTL or TTL
≤ 0 are marked as "never expire" (using sentinel value `int64(^uint64(0)
>> 1)`) and are preserved across compactions; percentile calculations
only include positive TTL values
3. **Query-time filtering automatic**: TTL filtering is transparently
added to expression compilation via `AddTTLFieldFilterExpressions()`,
which appends `(ttl_field IS NULL OR ttl_field > current_time)` to the
filter pipeline. Entities with null TTL always pass the filter
4. **Compaction triggering granular**: Percentile-based expiration (20%,
40%, 60%, 80%, 100%) allows configurable compaction thresholds via
`SingleCompactionRatioThreshold`, preventing premature data deletion
### Capability Added: Per-Entity Expiration with Data Distribution
Awareness
Users can now specify a TIMESTAMPTZ collection property `ttl_field`
naming a schema field. During data writes, TTL values are collected per
segment and percentile quantiles (5-value array) are computed and stored
in segment metadata. At query time, the TTL field is automatically
filtered. At compaction time, segment-level percentiles drive
expiration-based compaction decisions, enabling intelligent compaction
of segments where a configurable fraction of data has expired (e.g.,
compact when 40% of rows are expired, controlled by threshold ratio).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
issue: #39157
Overview:
Support search by PK by resolving IDs to vectors on Proxy side. Upgrade
go-api to adapt to new proto definitions.
Design:
- Upgrade milvus-proto/go-api to latest master.
- Implement handleIfSearchByPK in Proxy: resolve IDs to vectors via
internal Query, then rewrite SearchRequest.
- Adapt to 'SearchInput' oneof field in SearchRequest across client and
handlers.
- Fix binary vector stride calculation bug in placeholder utils.
Compatibility:
- Old Pymilvus can still work w/o this feature
What is included:
- Dense and Sparse
- Multi vector fields
- Rejection on BM25
What is **not** include:
- Hybrid Search
- EmbeddingList
- Restful API
Signed-off-by: Li Liu <li.liu@zilliz.com>
