related: #46442
This commit adds proper RBAC (Role-Based Access Control) support for the
/expr HTTP endpoint, replacing the previous root-only authentication.
Changes:
- Add new rbac.go with CheckPrivilege function for HTTP endpoints
- Support HTTP Basic Auth only (removed non-standard Bearer token
format)
- Integrate with existing Casbin RBAC framework
- Add PrivilegeExpr to GlobalLevelPrivileges and ClusterAdminPrivileges
- Register GetUserRoleFunc callback in meta_cache.go
- Update tests for new RBAC behavior
Security features:
- Authentication required when authorization is enabled
- Root user bypass when RootShouldBindRole is false
- Proper 401/403 status code differentiation
- Integration with privilege result cache
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
issue: #47420
## What this PR does
Project-wide migration of raw `fmt.Errorf` / `errors.New` in function
bodies onto
the `merr` framework, plus the Sys-vs-Input error classification and the
machinery it drives (retriability, fine-grained metrics, segcore
unification),
plus the convention docs and a linter that keeps it from regressing.
Scope: storage, proxy, coordinators (root/data/query), query node, data
node,
`pkg/util` & `internal/util`, expression parser, message queue,
streaming, and
misc packages. Bare raw-error usages went from ~3000 to a ~340 allowlist
(package-level sentinels / build-tag / test sites).
---
## How to review this PR
It is large but the vast majority is mechanical. Changes fall into three
tiers;
spend review budget on Part 2 and Part 3.
### Part 1 — Mechanical standardization (low risk, verify by rule)
Each converted call follows one of a small fixed set of rules. To
review, check
that each site obeys the matching rule rather than reading every line:
| Pattern | Rule |
|---|---|
| `fmt.Errorf("...")` originating a new error | →
`merr.WrapErrXxxMsg("...")` with a code matching the failure's meaning |
| Adding context to an existing typed error | → `merr.Wrap(err, "...")`
/ `merr.Wrapf(...)` — **preserves** the inner code (never `WrapErr*Err`,
which overwrites it) |
| Errors inside the streaming subsystem | → `status.New*` factories
(StreamingError), **not** merr — this is the component-internal dialect
(see `docs/dev/error_handling_guide.md`) |
| Low-level / control-flow signal caught by `errors.Is` | → kept as a
package-level `errors.New` sentinel (lowercase, same-package) |
Conventions are documented in `docs/dev/error_handling_guide.md`
(how-to) and
`docs/dev/error_sentinel_convention.md` (rules + audit). A
`gocritic`/`ruleguard`
rule (`rawmerrerror`, in `rules.go`) enforces "no raw `return
errors.New/fmt.Errorf`"
under `make verifiers`.
### Part 2 — Behavior changes (review these closely)
These are the sites where the wire contract or runtime behavior changes,
not just
the source text. Listed by category; representative locations given,
full set in
the diff.
**A. gRPC wire-code shifts: `UnexpectedError(1)/Code 65535` → typed
code.**
Where a handler previously returned a raw error (collapsed to
`Code=65535` on the
wire), it now returns a typed merr, so the client sees a real code. The
most
common shift is to `IllegalArgument(5)/Code 1100` (ParameterInvalid).
Touch
points include datanode task handlers (CreateTask/Query/Drop), proxy
Upsert,
querynode GetMetrics, datacoord CreateIndex, httpserver query-response
builder,
and typeutil schema validation. One code refinement: an index-param
validation
moved `1100` → `1101` (ParameterMissing). **Client/SDK assertions and
any code
that switched on `Code=65535` for these paths must be re-checked** (the
go_client
e2e assertions were already aligned in this PR).
**B. Prometheus `status` label contract change (externally visible).**
The proxy metric's coarse `fail` / `rejected` values are split into
`fail_input` / `fail_system` and `rejected_user` / `rejected_system` (in
`requestutil.ParseMetricLabel`; auth/privilege rejections count as
`rejected_user`), so dashboards can attribute a failure to caller vs
operator.
**Dashboards/alerts querying `status="fail"` must migrate to
`status=~"fail_.*"`, and `status="rejected"` to
`status=~"rejected_.*"`.** The
in-repo Grafana dashboard is already migrated; external dashboards built
on the
old values silently go empty after upgrade. This is the one change that
requires an ops-side migration.
**C. Retriability semantics.**
- C1: `merr.Status(err)` now forces `Retriable=false` when the error is
an
`InputError` — a malformed request can never succeed on blind retry, so
clients
never get the self-contradictory "your input is wrong but you may
retry".
- C2: `retry.Do` short-circuits an `InputError` (non-retriable) — **but
only when
the caller did not pass a `RetryErr` predicate**. The check is an
`if c.isRetryErr != nil { ... } else if InputError { ... }` *mutually
exclusive*
branch (`pkg/util/retry/retry.go`): an explicit `RetryErr` takes
precedence and
bypasses the InputError abort. `retry.Handle` deliberately does **not**
apply
the InputError abort (its callers signal abort via `shouldRetry=false`).
Four
flusher startup callsites that must retry through transient "not ready"
errors
were given explicit `RetryErr` escape hatches.
**D. segcore (C++→Go) error classification.**
A single shared Go-side table (`pkg/util/merr/segcore.go`) maps each
segcore code
to a merr sentinel + InputError/signal category, replacing scattered
hand-written
`if errorCode == ...` switches in the cgo wrappers. **Wire `Code` values
change
for every segcore pass-through error, not just the remapped ones.**
Named
sentinels remap (C++ `2003` → merr `2001`, `2033` → `2002`,
Folly/Knowhere codes
likewise); **all remaining pass-through codes (`2004`–`2043`, previously
surfaced to clients as raw C++ enum values) now serialize as `2000`**
(`ErrSegcore`), with the original C++ code preserved in the `Reason`
text
(`segcoreCode=...`); unknown/future codes collapse to `2000` as well
(pinned by
the `wire_code_projection` test). Transient segcore classes (object
storage /
file IO / OOM / mmap / FieldNotLoaded — 11 codes) now report
`Retriable=true`.
**Any client switching on raw segcore codes in the `2004`–`2043` range
must be
re-checked**; the in-Reason code remains available for diagnostics.
Signal
codes (PretendFinished / FollyCancel) are recognized centrally.
`errors.Is`-based
control flow on these (e.g. scheduler skip/retry) is preserved.
**E. InputError classification (25 sentinels + dynamic marks).**
25 sentinels in `errors.go` carry `WithErrorType(InputError)` (the
Collection /
ResourceGroup / Database families, `ErrIndexDuplicate`,
`ErrParameterInvalid`,
`ErrPrivilegeNotAuthenticated`, `ErrImportFailed`, `ErrQueryPlan`, ...),
plus dynamic
marks for the 8 segcore input codes (ExprInvalid, DimNotMatch,
MetricTypeInvalid, FieldIDInvalid, ...) and
`WrapErrAsInputError`. The widest blast radius is `ErrParameterInvalid`
(1100):
~2335 `WrapErrParameterInvalid*` callsites now classify as input /
non-retriable. Because of C1/C2 this changes retriability for
any path that returns these. **The audit to confirm no transient path
was
mis-marked is the single most important review item** (see Part 3). One
reverse
correction: storage field-stats parsing moved from `ErrParameterInvalid`
(input)
to `ErrDataIntegrity` — a corrupted stored stat is data corruption, not
user
input.
### Part 3 — Known risks & traps (called out proactively)
1. **`merr.Wrap` vs `WrapErr*Err` (code-masking).** `WrapErr*Err` builds
a
`wrappedMilvusError{sentinel: ErrServiceInternal}` whose `code()`
returns the
*outer* sentinel — it overwrites the inner typed code and hides the
`errors.Is`
chain. This is intentional (use it to *deliberately* downgrade), but it
was a
recurring conversion defect; the rule "add context with `merr.Wrap`,
downgrade
with `WrapErr*Err`" is enforced by convention and reviewed across the
diff.
2. **InputError × `retry.Do` blast radius.** Marking a sentinel
`InputError` makes
any `retry.Do(...)` without a `RetryErr` predicate stop retrying it.
Reviewers
should sanity-check that no transient use of the 19 newly-marked
sentinels
(especially `ErrParameterInvalid`) sits inside a retry loop that needed
to keep
spinning. The known flusher cases were handled (see C2).
3. **The ~340 raw-error allowlist.** What remains as bare `errors.New`
is, by
design: package-level sentinels (caught by `errors.Is`), `//go:build
test`
sites, and out-of-band trees (`cmd/`, `tests/`, codegen, walimpls). The
linter
only bans the *direct-return* form; assignment-then-return escapes and
the full
no-exceptions ban are deferred to an AST-based linter (Tier 2,
documented).
4. **segcore C++ second step deferred.** This PR unifies classification
on the Go
side; splitting the dual-semantic C++ codes at the source is a
follow-up.
---
## Validation
- `make verifiers`: Go side clean (gofmt + static-check across modules,
including
the new `rawmerrerror` rule with a 0-hit baseline repo-wide).
- `make test-go`: passing; the one real regression introduced (a
datanode
`invalid_task_type` assertion shifting `1` → `5` from a ParameterInvalid
conversion) was fixed in-tree.
- go_client e2e CreateIndex assertions aligned to the new merr messages.
---------
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Related to #50089
Add a Proxy management entry point that routes through MixCoord to clear
queued read tasks across all Proxy and QueryNode instances. Queued tasks
are failed with canceled semantics and per-component counts are returned
so operators can recover from read-task backlog without attempting
unsafe active-task preemption.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #49398
Bump Go module references from pkg/v2 and milvus-proto/go-api/v2 to
pkg/v3 and milvus-proto/go-api/v3 so the client tracks the Milvus 3.x
release line.
This prepares the repository for the upcoming 3.x.y release by aligning
imports, module dependencies, and proto API references with the new
major-version module paths.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: zilliztech/spark-milvus#74
Add a new proxy management HTTP endpoint
/management/datacoord/backfill/commit that takes an object-storage path
to a BackfillResult JSON produced by the Spark offline backfill job,
lets DataCoord download and parse it, then dispatches per-segment
metadata updates through the existing BatchUpdateManifest broadcast
pipeline.
V2 (StorageV2 packed-parquet) and V3 (Loon manifest) segments share the
broadcast, so the broadcaster's SharedDBName + SharedCollectionName
resource keys serialise the commit against compaction and other DDL-like
operations on the same collection. The ack callback classifies each item
and applies either UpdateManifestVersion (V3) or the ported
UpdateSegmentColumnGroupsOperator (V2) in one UpdateSegmentsInfo batch.
The V2 path trusts the group-level row_count in the result JSON rather
than reading parquet footers; row_count is split evenly across the
group's binlog files (remainder added to the last file).
Changes:
- proto: new CommitBackfillResult RPC on DataCoord; extend
BatchUpdateManifestItem with a v2_column_groups payload + new
BatchUpdateManifestV2ColumnGroups message.
- datacoord: new services_commit_backfill.go handler; new
backfill_result.go (JSON decoding, object-URI normaliser, bucket
validation, V2 group constructor). Port
UpdateSegmentColumnGroupsOperator from stash/update_columngroup_restful
into meta.go. Extend the BatchUpdateManifest broadcast callback to
dispatch V2/V3 items.
- proxy: new /management/datacoord/backfill/commit HTTP handler.
- mixcoord: grpc server/client/wrapper delegators.
- storage: expose RemoteChunkManager.BucketName() for bucket-mismatch
rejection in the URI normaliser.
Tests:
- backfill_result_test.go: JSON parsing, normaliser, V2 group builder.
- services_test.go: TestServer_CommitBackfillResult (V3 happy, V2 happy,
mixed/partial failure, success=false, bad JSON, pre-validation-all-fail,
unhealthy). Extend BatchUpdateManifest callback tests with V2 and
mixed-item scenarios.
- meta_test.go: TestUpdateSegmentColumnGroupsOperator (add, replace
in-place, child-field stripping, not-found, DataVersion monotonicity).
- management_test.go: proxy handler contract (missing param, success,
rpc error, downstream status error).
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #47281
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260131-client_side_telemetry.md
Implement comprehensive client-side telemetry system that includes:
- Metrics collection for Search, Query, Insert, Delete, Upsert
operations
- Automatic heartbeat reporting to server with configurable intervals
- Per-collection metrics tracking with wildcard support
- P99 latency calculation using ring buffer sampling
- Server-push command handling (push_config, collection_metrics,
show_errors)
- Historical snapshot storage for latency history queries
- Error tracking with circular buffer for recent errors
- WebUI for telemetry visualization and command management
- HTTP API endpoints for telemetry data access and command push
- RootCoord telemetry manager for centralized command routing
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
related: https://github.com/milvus-io/milvus/issues/46442
core changes:
- Add config (default: false) to disable /expr endpoint by default
- On Proxy nodes, require root user authentication via HTTP Basic Auth
when enabled
- On non-Proxy nodes, keep original auth parameter behavior for backward
compatibility
- Add HasRegistered() and AuthBypass to expr package for node type
detection
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
The restful API default timeout was hard-coded. This PR make this
timeout value configurable via paramtable.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #39083
/kind improvement
Three new functions of the static web page:
1. The input box can be expanded and scrolled if it exceeds the maximum
size
2. Input history
3. It will simply check whether the quotation marks and brackets appear
in pairs
Signed-off-by: SimFG <bang.fu@zilliz.com>
issue: #36621
- add filters to most of lists
- add more options to the existing list filter
- fix the error list column in channels and resource group list
Signed-off-by: mimo.oyn <ning.ouyang@zilliz.com>
issue: #36621
- fix the empty segment source
- remove http request timeout limit
- support more filters in seg and channel table
Signed-off-by: mimo.oyn <ning.ouyang@zilliz.com>
issue: #36621
* The front-end code of the management interface has been refactored
using React.
* The interaction and display of the front-end interface have been
optimized.
Signed-off-by: mimo.oyn <ning.ouyang@zilliz.com>
Co-authored-by: zilliz <zilliz@zillizdeMacBook-Pro.local>
issue: ##36621
- For simple types in a struct, add "string" to the JSON tag for
automatic string conversion during JSON encoding.
- For complex types in a struct, replace "int64" with "string."
Signed-off-by: jaime <yun.zhang@zilliz.com>
issue: #36621
1. Add API to access task runtime metrics, including:
- build index task
- compaction task
- import task
- balance (including load/release of segments/channels and some leader
tasks on querycoord)
- sync task
2. Add a debug model to the webpage by using debug=true or debug=false
in the URL query parameters to enable or disable debug mode.
Signed-off-by: jaime <yun.zhang@zilliz.com>
after manual stop component by management restful api, `healthz` may
return unhealthy state. k8s may restart the pod to save the unhealthy
sate, and the manual stop operation will got unexpected result.
to solve this, we make `healthz` API skip the manual stopped component.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: #32698
This PR add two rest api for component stop and status check:
1. `/management/stop?role=querynode` can stop the specified component
2. `/management/check/ready?role=rootcoord` can check whether the target
component is serviceable
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
