enhance: standardize error handling on merr + Sys/Input classification (#50221)

issue: #47420

## What this PR does

Project-wide migration of raw `fmt.Errorf` / `errors.New` in function
bodies onto
the `merr` framework, plus the Sys-vs-Input error classification and the
machinery it drives (retriability, fine-grained metrics, segcore
unification),
plus the convention docs and a linter that keeps it from regressing.

Scope: storage, proxy, coordinators (root/data/query), query node, data
node,
`pkg/util` & `internal/util`, expression parser, message queue,
streaming, and
misc packages. Bare raw-error usages went from ~3000 to a ~340 allowlist
(package-level sentinels / build-tag / test sites).

---

## How to review this PR

It is large but the vast majority is mechanical. Changes fall into three
tiers;
spend review budget on Part 2 and Part 3.

### Part 1 — Mechanical standardization (low risk, verify by rule)

Each converted call follows one of a small fixed set of rules. To
review, check
that each site obeys the matching rule rather than reading every line:

| Pattern | Rule |
|---|---|
| `fmt.Errorf("...")` originating a new error | →
`merr.WrapErrXxxMsg("...")` with a code matching the failure's meaning |
| Adding context to an existing typed error | → `merr.Wrap(err, "...")`
/ `merr.Wrapf(...)` — **preserves** the inner code (never `WrapErr*Err`,
which overwrites it) |
| Errors inside the streaming subsystem | → `status.New*` factories
(StreamingError), **not** merr — this is the component-internal dialect
(see `docs/dev/error_handling_guide.md`) |
| Low-level / control-flow signal caught by `errors.Is` | → kept as a
package-level `errors.New` sentinel (lowercase, same-package) |

Conventions are documented in `docs/dev/error_handling_guide.md`
(how-to) and
`docs/dev/error_sentinel_convention.md` (rules + audit). A
`gocritic`/`ruleguard`
rule (`rawmerrerror`, in `rules.go`) enforces "no raw `return
errors.New/fmt.Errorf`"
under `make verifiers`.

### Part 2 — Behavior changes (review these closely)

These are the sites where the wire contract or runtime behavior changes,
not just
the source text. Listed by category; representative locations given,
full set in
the diff.

**A. gRPC wire-code shifts: `UnexpectedError(1)/Code 65535` → typed
code.**
Where a handler previously returned a raw error (collapsed to
`Code=65535` on the
wire), it now returns a typed merr, so the client sees a real code. The
most
common shift is to `IllegalArgument(5)/Code 1100` (ParameterInvalid).
Touch
points include datanode task handlers (CreateTask/Query/Drop), proxy
Upsert,
querynode GetMetrics, datacoord CreateIndex, httpserver query-response
builder,
and typeutil schema validation. One code refinement: an index-param
validation
moved `1100` → `1101` (ParameterMissing). **Client/SDK assertions and
any code
that switched on `Code=65535` for these paths must be re-checked** (the
go_client
e2e assertions were already aligned in this PR).

**B. Prometheus `status` label contract change (externally visible).**
The proxy metric's coarse `fail` / `rejected` values are split into
`fail_input` / `fail_system` and `rejected_user` / `rejected_system` (in
`requestutil.ParseMetricLabel`; auth/privilege rejections count as
`rejected_user`), so dashboards can attribute a failure to caller vs
operator.
**Dashboards/alerts querying `status="fail"` must migrate to
`status=~"fail_.*"`, and `status="rejected"` to
`status=~"rejected_.*"`.** The
in-repo Grafana dashboard is already migrated; external dashboards built
on the
old values silently go empty after upgrade. This is the one change that
requires an ops-side migration.

**C. Retriability semantics.**
- C1: `merr.Status(err)` now forces `Retriable=false` when the error is
an
`InputError` — a malformed request can never succeed on blind retry, so
clients
never get the self-contradictory "your input is wrong but you may
retry".
- C2: `retry.Do` short-circuits an `InputError` (non-retriable) — **but
only when
  the caller did not pass a `RetryErr` predicate**. The check is an
`if c.isRetryErr != nil { ... } else if InputError { ... }` *mutually
exclusive*
branch (`pkg/util/retry/retry.go`): an explicit `RetryErr` takes
precedence and
bypasses the InputError abort. `retry.Handle` deliberately does **not**
apply
the InputError abort (its callers signal abort via `shouldRetry=false`).
Four
flusher startup callsites that must retry through transient "not ready"
errors
  were given explicit `RetryErr` escape hatches.

**D. segcore (C++→Go) error classification.**
A single shared Go-side table (`pkg/util/merr/segcore.go`) maps each
segcore code
to a merr sentinel + InputError/signal category, replacing scattered
hand-written
`if errorCode == ...` switches in the cgo wrappers. **Wire `Code` values
change
for every segcore pass-through error, not just the remapped ones.**
Named
sentinels remap (C++ `2003` → merr `2001`, `2033` → `2002`,
Folly/Knowhere codes
likewise); **all remaining pass-through codes (`2004`–`2043`, previously
surfaced to clients as raw C++ enum values) now serialize as `2000`**
(`ErrSegcore`), with the original C++ code preserved in the `Reason`
text
(`segcoreCode=...`); unknown/future codes collapse to `2000` as well
(pinned by
the `wire_code_projection` test). Transient segcore classes (object
storage /
file IO / OOM / mmap / FieldNotLoaded — 11 codes) now report
`Retriable=true`.
**Any client switching on raw segcore codes in the `2004`–`2043` range
must be
re-checked**; the in-Reason code remains available for diagnostics.
Signal
codes (PretendFinished / FollyCancel) are recognized centrally.
`errors.Is`-based
control flow on these (e.g. scheduler skip/retry) is preserved.

**E. InputError classification (25 sentinels + dynamic marks).**
25 sentinels in `errors.go` carry `WithErrorType(InputError)` (the
Collection /
ResourceGroup / Database families, `ErrIndexDuplicate`,
`ErrParameterInvalid`,
`ErrPrivilegeNotAuthenticated`, `ErrImportFailed`, `ErrQueryPlan`, ...),
plus dynamic
marks for the 8 segcore input codes (ExprInvalid, DimNotMatch,
MetricTypeInvalid, FieldIDInvalid, ...) and
`WrapErrAsInputError`. The widest blast radius is `ErrParameterInvalid`
(1100):
~2335 `WrapErrParameterInvalid*` callsites now classify as input /
non-retriable. Because of C1/C2 this changes retriability for
any path that returns these. **The audit to confirm no transient path
was
mis-marked is the single most important review item** (see Part 3). One
reverse
correction: storage field-stats parsing moved from `ErrParameterInvalid`
(input)
to `ErrDataIntegrity` — a corrupted stored stat is data corruption, not
user
input.

### Part 3 — Known risks & traps (called out proactively)

1. **`merr.Wrap` vs `WrapErr*Err` (code-masking).** `WrapErr*Err` builds
a
`wrappedMilvusError{sentinel: ErrServiceInternal}` whose `code()`
returns the
*outer* sentinel — it overwrites the inner typed code and hides the
`errors.Is`
chain. This is intentional (use it to *deliberately* downgrade), but it
was a
recurring conversion defect; the rule "add context with `merr.Wrap`,
downgrade
with `WrapErr*Err`" is enforced by convention and reviewed across the
diff.
2. **InputError × `retry.Do` blast radius.** Marking a sentinel
`InputError` makes
any `retry.Do(...)` without a `RetryErr` predicate stop retrying it.
Reviewers
should sanity-check that no transient use of the 19 newly-marked
sentinels
(especially `ErrParameterInvalid`) sits inside a retry loop that needed
to keep
   spinning. The known flusher cases were handled (see C2).
3. **The ~340 raw-error allowlist.** What remains as bare `errors.New`
is, by
design: package-level sentinels (caught by `errors.Is`), `//go:build
test`
sites, and out-of-band trees (`cmd/`, `tests/`, codegen, walimpls). The
linter
only bans the *direct-return* form; assignment-then-return escapes and
the full
no-exceptions ban are deferred to an AST-based linter (Tier 2,
documented).
4. **segcore C++ second step deferred.** This PR unifies classification
on the Go
side; splitting the dual-semantic C++ codes at the source is a
follow-up.

---

## Validation

- `make verifiers`: Go side clean (gofmt + static-check across modules,
including
  the new `rawmerrerror` rule with a 0-hit baseline repo-wide).
- `make test-go`: passing; the one real regression introduced (a
datanode
`invalid_task_type` assertion shifting `1` → `5` from a ParameterInvalid
  conversion) was fixed in-tree.
- go_client e2e CreateIndex assertions aligned to the new merr messages.

---------

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
zhenshan.cao
2026-06-12 15:04:51 -07:00
committed by GitHub
co-authored by Claude Opus 4.7
parent 14c4cf65fc
commit e2787d3981
629 changed files with 7465 additions and 4797 deletions
+4 -4
View File
@@ -359,15 +359,15 @@ proxy:
remoteMaxTime: 0 # The time interval allowed for uploading access log files. If the upload time of a log file exceeds this interval, the file will be deleted. Setting the value to 0 disables this feature.
formatters:
base:
format: "[$time_now] [ACCESS] <$user_name: $user_addr> $method_name [status: $method_status] [code: $error_code] [sdk: $sdk_version] [msg: $error_msg] [traceID: $trace_id] [timeCost: $time_cost]"
format: "[$time_now] [ACCESS] <$user_name: $user_addr> $method_name [status: $method_status] [code: $error_code] [sdk: $sdk_version] [msg: $error_msg] [traceID: $trace_id] [timeCost: $time_cost] [error_type: $error_type]"
query:
format: "[$time_now] [ACCESS] <$user_name: $user_addr> $method_name [status: $method_status] [code: $error_code] [sdk: $sdk_version] [msg: $error_msg] [traceID: $trace_id] [timeCost: $time_cost] [database: $database_name] [collection: $collection_name] [partitions: $partition_name] [expr: $method_expr] [params: $query_params]"
format: "[$time_now] [ACCESS] <$user_name: $user_addr> $method_name [status: $method_status] [code: $error_code] [sdk: $sdk_version] [msg: $error_msg] [traceID: $trace_id] [timeCost: $time_cost] [database: $database_name] [collection: $collection_name] [partitions: $partition_name] [expr: $method_expr] [params: $query_params] [error_type: $error_type]"
methods: "Query, Delete, /query, /v2/vectordb/entities/query, /v2/vectordb/entities/get, /v2/vectordb/entities/delete"
search:
format: "[$time_now] [ACCESS] <$user_name: $user_addr> $method_name [status: $method_status] [code: $error_code] [sdk: $sdk_version] [msg: $error_msg] [traceID: $trace_id] [timeCost: $time_cost] [database: $database_name] [collection: $collection_name] [partitions: $partition_name] [expr: $method_expr] [nq: $nq] [params: $search_params]"
format: "[$time_now] [ACCESS] <$user_name: $user_addr> $method_name [status: $method_status] [code: $error_code] [sdk: $sdk_version] [msg: $error_msg] [traceID: $trace_id] [timeCost: $time_cost] [database: $database_name] [collection: $collection_name] [partitions: $partition_name] [expr: $method_expr] [nq: $nq] [params: $search_params] [error_type: $error_type]"
methods: "HybridSearch, Search, /search, /v2/vectordb/entities/search, /v2/vectordb/entities/advanced_search, /v2/vectordb/entities/hybrid_search"
upsert:
format: "[$time_now] [ACCESS] <$user_name: $user_addr> $method_name [status: $method_status] [code: $error_code] [sdk: $sdk_version] [msg: $error_msg] [traceID: $trace_id] [timeCost: $time_cost] [database: $database_name] [collection: $collection_name] [partitions: $partition_name] [partial_update: $partial_update]"
format: "[$time_now] [ACCESS] <$user_name: $user_addr> $method_name [status: $method_status] [code: $error_code] [sdk: $sdk_version] [msg: $error_msg] [traceID: $trace_id] [timeCost: $time_cost] [database: $database_name] [collection: $collection_name] [partitions: $partition_name] [partial_update: $partial_update] [error_type: $error_type]"
methods: Upsert
cacheSize: 0 # Size of log of write cache, in byte. (Close write cache if size was 0)
cacheFlushInterval: 3 # time interval of auto flush write cache, in seconds. (Close auto flush if interval was 0)
@@ -4794,7 +4794,7 @@
"uid": "${datasource}"
},
"exemplar": true,
"expr": "sum(increase(milvus_proxy_req_count{app_kubernetes_io_instance=~\"$instance\", app_kubernetes_io_name=\"$app_name\", namespace=\"$namespace\", status=\"fail\"}[2m])/120) by(function_name, pod, node_id)",
"expr": "sum(increase(milvus_proxy_req_count{app_kubernetes_io_instance=~\"$instance\", app_kubernetes_io_name=\"$app_name\", namespace=\"$namespace\", status=~\"fail_.*\"}[2m])/120) by(function_name, pod, node_id)",
"interval": "",
"legendFormat": "{{function_name}}-{{pod}}-{{node_id}}",
"queryType": "randomWalk",
+314
View File
@@ -0,0 +1,314 @@
# Error Handling Guide
How to produce and return errors in Milvus server code — the day-to-day how-to.
For the underlying rules, the sentinel naming convention, and the enforcement
roadmap, see [error_sentinel_convention.md](./error_sentinel_convention.md). For
the canonical numeric code list, see the sentinel definitions in
[`pkg/util/merr/errors.go`](../../pkg/util/merr/errors.go). (The
[appendix_d_error_code.md](../developer_guides/appendix_d_error_code.md)
appendix predates merr and lists the **deprecated** `commonpb.ErrorCode` enum,
not the merr codes.)
## TL;DR
1. **Never** `return errors.New(...)` or `return fmt.Errorf(...)`. A linter
rejects it (see [The one rule](#the-one-rule-never-return-a-bare-error)).
2. An error that leaves your component must be a **typed** error carrying a code.
The lingua franca across Milvus components is **merr**.
3. Pick one of three forms: originate a typed merr, add context with `merr.Wrap`,
or — only if a caller branches on it by identity — a package-level sentinel.
## The mental model: three kinds of error, one boundary rule
Milvus has three legitimate kinds of error, distinguished by **how far they
travel**, not by syntax:
| Kind | Scope | Carrier | Example |
|---|---|---|---|
| **① merr** | across Milvus components & to clients | numeric code on the main gRPC wire | `merr.ErrCollectionNotFound` |
| **② component-internal dialect** | between sub-modules of one big component | that component's own typed error + its own wire | `streamingutil/status.StreamingError` |
| **③ internal sentinel** | within a single Go process | `errors.New` pointer identity, caught by `errors.Is` | `errSessionVersionCheckFailure` |
The single rule that ties them together:
> **The error you must return follows the interface's promise** — it is decided
> by *who is on the other side of the boundary you are crossing*, not by where
> the error was born.
- Crossing into **another Milvus component** (proxy → rootcoord), or returning to
a **client**: the promise is **merr**. Translate to a typed merr at that
boundary.
- Staying **inside one component** (e.g. the streaming sub-modules talking to
each other): the component may speak its own typed dialect (`StreamingError`).
It is **not** required to be merr — but it must still be *typed*, and it gets
translated to merr at the component's outer edge.
- On **no** wire, ever: a **bare** `errors.New` / `fmt.Errorf`. Internal
sentinels (kind ③) are bare, but they never reach a wire — they are caught by
`errors.Is` and translated first.
### Why component-internal dialects are allowed (the StreamingError case)
Streaming is one big component whose sub-modules (streamingnode, streamingcoord,
the streaming client) talk to each other constantly. They use
`streamingutil/status.StreamingError`, which has its **own** error codes and its
**own** gRPC encoding. That is deliberate: it is a *bounded context* with its own
vocabulary. There is intentionally **no** global "StreamingError → merr"
auto-converter — that would erase the dialect. Instead the conversion happens
**once, at the consumer's boundary**, and the consumer decides how, based on
what its own interface promises:
```go
// rootcoord consumes a streaming service inside CreateCollection.
err := s.streamingService.DoSomething(ctx, ...) // may return *StreamingError
if err != nil {
// (a) You care about the code the client sees → translate explicitly:
if se := status.AsStreamingError(err); se != nil && se.IsRateLimitRejected() {
return merr.WrapErrServiceRateLimit("streaming backpressure")
}
// (b) You don't care about a precise code → let it fall back at the
// boundary (see "The safety net"); the client gets a generic
// internal-class error. Prefer being explicit, but this is allowed.
return err
}
```
## The one rule: never return a bare error
```go
return errors.New("segment not loaded") // ❌ linter rejects
return fmt.Errorf("segment %d not loaded", id) // ❌ linter rejects
```
Why it is banned, even though the boundary would "fix it up" anyway: a bare error
that escapes to a gRPC boundary becomes `Code=65535 (Unexpected)` — visually
indistinguishable from "the server hit an unhandled bug". A wall of
`errors.New("reason 1")`, `errors.New("reason 2")` is a sign nobody planned the
error taxonomy: the caller cannot program against it, and it all collapses into
one opaque code on the wire. A typed error costs one extra word and makes the
failure *addressable*. The linter exists to build the habit: **the thing I
return is always typed.**
## Decision tree: what should I return?
```
Am I crossing into another component / returning to a client?
├─ No (staying inside my own component)
│ ├─ My component has its own typed dialect (e.g. streaming)?
│ │ → use that dialect's factory (status.New*), not merr, not errors.New
│ └─ Otherwise → a typed merr (rules below); or, only if a caller in THIS
│ process branches on the outcome by identity, a package-level sentinel (§3.3)
└─ Yes → I must return a typed merr:
├─ Brand-new failure, no underlying error worth carrying?
│ → merr.WrapErrXxxMsg("detail %s", v) (§3.1 originate)
│ and pick Input vs System deliberately (next section)
├─ I hold an underlying error and want to KEEP its code, just add context?
│ → merr.Wrap(err, "while doing X") (§3.2 add context)
└─ I hold an underlying error and want to DOWNGRADE it to a generic class?
→ merr.WrapErrServiceInternalErr(err, "...") (§3.2 — deliberate override)
```
## Input vs System: who is to blame?
Every merr is classified as **InputError** (the request author's fault) or
**SystemError** (Milvus's fault, the default). Choosing a factory chooses the
classification, so when you originate an error, ask one question first:
> **Would a correctly implemented Milvus ever hit this branch, given this
> request?** If the request content itself triggers it → InputError. If
> reaching this branch means a Milvus bug or internal failure → SystemError.
Quick rules for the cases that get misclassified in practice:
- A **plan / task type / request produced by a coordinator** is not user input.
An unrecognized task type or malformed compaction plan is an internal
protocol violation (think mixed-version rolling upgrade) →
`WrapErrServiceInternalMsg`, even though the check looks like validation.
- **Data produced by segcore or another internal component** is not user
input. A violated data-shape contract (ValidData length, truncated vectors)
is a Milvus bug → `WrapErrServiceInternalMsg`.
- A **TOCTOU race** (state was valid at check time, changed by execution time)
is not user input → keep it a system error.
### How classification is attached
Two mechanisms, used in different situations:
1. **Baked-in sentinels.** ~25 sentinels are declared
`WithErrorType(InputError)` in `errors.go` (`ErrParameterInvalid/Missing/
TooLarge`, `ErrPrivilegeNotPermitted`, `ErrDatabaseInvalidName`, ...).
Using their factory *is* the classification — which is exactly why reaching
for `WrapErrParameterInvalidMsg` to express an internal assertion is wrong.
2. **Boundary marking** for dual-use sentinels. `ErrCollectionNotFound` stays
SystemError (internal refresh/retry paths depend on that), and the proxy
boundary stamps it InputError only where the name came from the user:
```go
// proxy meta cache, the central chokepoint for user-supplied names:
return collection, merr.WrapErrAsInputErrorWhen(err,
merr.ErrCollectionNotFound, merr.ErrDatabaseNotFound)
```
`WrapErrAsInputError(err)` marks unconditionally;
`WrapErrAsInputErrorWhen(err, targets...)` marks only if the error's code
matches a target. Both preserve `errors.Is` and the code — they relabel the
classification, nothing else.
### What the classification drives
| Surface | InputError behavior |
|---|---|
| `commonpb.Status` | `ExtraInfo["is_input_error"]="true"`, `Retriable` forced `false` |
| Prometheus | request counted as `fail_input` / `rejected_user` (vs `fail_system` / `rejected_system`) |
| Access log / failure log | `error_type` field set accordingly |
| proxy lb_policy | **no cross-replica failover** — retrying a bad request elsewhere can't help |
| `retry.Do` | aborts immediately instead of retrying |
The last two rows are why misclassification is not cosmetic: marking an
internal failure as InputError disables the retry/failover machinery that
would have healed it, and a dashboard blames users for Milvus bugs.
### Pitfalls (each of these happened)
- **Don't mark a shared sentinel InputError globally** to fix one callsite —
every internal `retry.Do` loop waiting on that error stops retrying. Use
boundary marking instead.
- **Don't classify in a helper** what only the boundary can know. The same
not-found is the user's fault when the name came from a request, and a
system fault when it came from internal state — stamp at the chokepoint
where the origin is known.
- **"Looks like validation" is not the test.** Coordinator-to-node protocol
checks, segcore output checks, and cgo boundary checks all look like
validation; none of them are user input.
## The three correct ways
### 3.1 Originate a typed error — `WrapErrXxxMsg` / `WrapErrXxx`
When the failure starts here and there is no inner error worth preserving:
```go
return merr.WrapErrParameterInvalidMsg("nq (%d) exceeds the limit (%d)", nq, max)
return merr.WrapErrCollectionNotFound(collectionName)
```
Pick the sentinel whose **code** matches the failure's meaning (see the sentinel
definitions in [`pkg/util/merr/errors.go`](../../pkg/util/merr/errors.go)). This is the common
case: most of the time you only need to attach a message to a well-chosen code,
and the framework does the rest.
### 3.2 Add context but keep the code — `merr.Wrap`, never `WrapErr*Err`
When you already hold a typed error and only want to add a breadcrumb:
```go
if err := s.loadSegment(ctx, id); err != nil {
return merr.Wrap(err, "while loading sealed segment") // ✅ keeps the inner code
}
```
`merr.Wrap` / `merr.Wrapf` is a thin wrapper (like `errors.Wrap`): it prepends
context and **preserves** the inner error's code and its `errors.Is` chain.
⚠️ Do **not** reach for `merr.WrapErrServiceInternalErr(err, ...)` (or any
`merr.WrapErrXxxErr`) just to add context. Those build a
`wrappedMilvusError{sentinel: ErrServiceInternal}` whose `code()` returns the
**outer** sentinel's code — they **overwrite** the inner typed code with
ServiceInternal (5) and force it non-retriable (`As()` resolves to the outer
sentinel). The `errors.Is` chain itself survives via `Unwrap()`, but the typed
code and retriability are masked. `WrapErrXxxErr` is *only* for
when you **deliberately** want to relabel the inner error to a new code (e.g.
collapse a noisy internal failure into one ServiceInternal for the client).
This split is intentional, and the framework deliberately does **not** try to be
clever: *keep the code* versus *downgrade the code* is a decision you state, not
one the framework guesses. `merr.Wrap` **always** keeps the inner code;
`WrapErrXxxErr` **always** relabels to the outer one. A helper that "smartly"
preserved the inner code whenever it recognized a typed merr would blur the
intent — a reader could no longer tell from the call site whether the author
meant to preserve or to downgrade. The choice of helper *is* the statement of
intent.
### 3.2.1 The base-package case: pass through, wrap, or relabel?
Low-level packages (`pkg/...`, `internal/util/...`) sit under many callers and
usually receive an error from something even lower — etcd, S3, a third-party
library, another util. For every such error you must consciously pick one of
three. Getting this wrong in a base package is expensive: it is multiplied across
every caller.
| Choice | Use it when | How |
|---|---|---|
| **Pass through** the original `err` | the inner err is already a typed error meaningful to your caller, or your package has no business classifying it — let the caller decide | `return err` |
| **Wrap, keep the code** | you want to add a breadcrumb (which key/path/op failed) without changing what the error *means* | `merr.Wrapf(err, "etcd txn on key %s", k)` |
| **Relabel / downgrade** | the inner err is a leaky implementation / third-party detail your caller should not see; you translate it into the typed merr your package's interface promises | `merr.WrapErrXxxErr(err, "...")` |
Deciding questions, in order:
1. **Is the inner err already typed and meaningful to my caller?** → pass through.
2. **Does any caller `errors.Is` the inner err's identity?** → pass through or
`merr.Wrap` (both preserve the chain); **never relabel** — it hides the chain.
3. **Is the inner err a third-party / implementation detail I promise to hide?**
→ relabel to the typed merr my interface promises.
A base package that relabels too eagerly destroys codes the upper layers needed;
one that passes a raw third-party error straight through leaks an untyped error
toward the boundary. Neither is acceptable — the choice must be deliberate, and it
follows your package's interface promise, not convenience.
### 3.3 Need identity branching or reuse — a package-level sentinel
Define a sentinel when a caller **in the same process** branches on the outcome
by identity, via `errors.Is`:
```go
// internal/util/sessionutil/session_util.go — caught by isNotSessionVersionCheckFailure
// and used as a retry.Do predicate; the identity must survive, so it stays a
// bare sentinel rather than a merr error.
var errSessionVersionCheckFailure = errors.New("session version check failure")
```
Rules for sentinels (full version in
[error_sentinel_convention.md](./error_sentinel_convention.md)):
- **Package-level, never function-local.** A local `x := errors.New(...)` is a
refactor hazard — tomorrow it gets hoisted and silently crosses a boundary.
Lift it to a `var` at package scope.
- **Lowercase / unexported** when it lives in `internal/...`. If a
*cross-package* caller needs the signal, do **not** export the sentinel —
redesign the API to carry the signal in a return value (e.g.
`(ignored bool, err error)`).
- It must be `errors.Is`-caught and translated to a typed merr (or
`merr.Success()`) **before** crossing any gRPC boundary. A sentinel that
reaches the wire is just an opaque `Code=65535`.
When unsure between 3.1 and 3.3: if nobody does `errors.Is` on it, you don't need
a sentinel — just originate a typed merr with a message (3.1).
## The safety net: boundary fallback (and why not panic)
If a non-typed error does reach a gRPC handler, `merr.Status(err)` falls back to
`Code=65535 (Unexpected)`. This is a **backstop, not a feature**: it keeps the
server from leaking internals or crashing, but the client gets an opaque code.
Treat any `Code=65535` in logs as "someone forgot to type their error".
Why the boundary **falls back instead of panicking**: third-party libraries and
deep call chains produce errors on paths that tests cannot fully cover. Panicking
on "not a typed merr" would turn a stray untyped error into an outage. The
contract is therefore: **fall back to a generic code, never panic.** The goal of
the linter and this guide is to make that fallback path *empty in practice* — so
that every error a client sees was deliberately typed at its source.
## Enforcement
- A `gocritic`/`ruleguard` rule (`rawmerrerror` in `rules.go`) rejects
`return errors.New / fmt.Errorf / errors.Errorf` from function bodies. It runs
under `make verifiers` (via `static-check`), so no extra command is needed.
Exempt paths (run outside the request path): `*_test.go`, `cmd/`, `tests/`,
codegen, the walimpls harness, and `/mocks/` (generated mock helpers are
test infrastructure even though some lack a "Code generated" header).
- It catches the **direct-return** form — the one that lets a raw error escape to
a boundary. Assignment-then-return escapes (`e := errors.New(); return e`) and
the full no-exceptions ban require the AST-based linter described as "Tier 2"
in [error_sentinel_convention.md](./error_sentinel_convention.md).
+324
View File
@@ -0,0 +1,324 @@
# Error Sentinel Convention
Milvus error handling has two distinct layers. This document describes the rules
that separate them, the rationale, and an audit of where the codebase currently
violates them.
## The two layers
### 1. Typed merr (wire-protocol errors)
Defined in `pkg/util/merr/errors.go` (`ErrCollectionNotFound`,
`ErrParameterInvalid`, etc.). These carry a numeric error code that is
serialized into `commonpb.Status{ErrorCode, Reason}` and shipped to the
client over gRPC. They are the only thing a client (or another Milvus
component on the receiving end of an RPC) sees.
Creation: `merr.WrapErrXxxMsg(...)` / `merr.WrapErrXxxErr(cause, ...)`
(origination only — never to add context to a typed merr that already exists,
see `merr.Wrap`).
### 2. Internal sentinels (single-process control flow)
Created with `errors.New(...)` at package scope (e.g. `errIgnoredAlterAlias`,
`errReleaseCollectionNotLoaded`, `errNodeNotEnough`). These are signaling
vocabulary inside a single Go process: a callee tells its caller "this is an
idempotent no-op" or "the queue is empty" so the caller can branch / retry /
ignore. They are **not** part of the wire protocol.
The catcher is always `errors.Is(err, sentinelX)` at some boundary in the
calling stack, and the boundary either:
- translates to `merr.Success()` (idempotency: e.g. drop something that
doesn't exist → success), or
- translates to a typed merr `merr.WrapErrXxxMsg(...)` (e.g. user already
exists → `WrapErrParameterInvalid`).
## The hard invariant
> **Any internal sentinel must be `errors.Is`-caught and translated to a
> typed merr (or `Success`) before crossing any gRPC handler boundary —
> client-facing or component-to-component.**
Why "any gRPC boundary, not just client-facing": gRPC serializes errors to
`commonpb.Status{ErrorCode, Reason}`. The Go-level pointer identity that
`errors.New` sentinels rely on does not survive the wire. The peer's
`merr.Error(status)` reconstructs a typed merr from the numeric code; the
sentinel chain is gone forever. So a sentinel that escapes an internal coord
RPC is just as broken as one that escapes a user-facing RPC — only quieter,
because no customer sees the resulting `Code=65535 (unexpected)`.
### What breaks the invariant
The `errors.Is` chain survives `return err`, `errors.Wrap(err, "...")` /
`merr.Wrap(err, "...")` (cockroachdb thin wrap), **and**
`merr.WrapErrServiceInternalErr(err, "...")``milvusError.Unwrap()`
returns the inner error, so `errors.Is(outer, innerSentinel)` stays true through any
of them. It is **destroyed** only by:
- Putting the cause in a format argument instead of the chain:
`merr.WrapErrXxxMsg("...: %s", err)`, or the `%w` mistake (`WrapErr*Msg`
formats with `fmt.Sprintf`, which does **not** honor `%w` and renders
`%!w(...)`). The inner error reaches the message text but is unreachable via
`Unwrap()`, so `errors.Is(outer, innerSentinel)` returns false.
- Any custom wrapper that doesn't implement `Unwrap()`.
Do **not** confuse this with the separate code/retriability rule.
`merr.Wrap(err, ...)` and `merr.WrapErrXxxErr(err, ...)` both keep `errors.Is`
intact, but they differ in what the result reports at the boundary:
- `merr.Wrap(err, ...)` preserves the inner's `Code()` and `IsRetryable` — the
chain still resolves to the inner's `*milvusError`.
- `merr.WrapErrXxxErr(err, ...)` reports the **outer** sentinel's code and
retriability: `As()` resolves to `ErrServiceInternal` (Code 5,
non-retriable), **masking** the inner's classification.
So there are two distinct rules, often conflated:
1. To keep `errors.Is` working: never stuff the cause into a format string;
pass it as the error argument.
2. To add context **without changing the classification** (preserve the inner
code + retriability): use `merr.Wrap`, not `merr.WrapErr*Err`. Reserve
`merr.WrapErrXxxErr` for when you intend to *assert* a new classification
(e.g. this genuinely is a service-internal error). (See also
[feedback rule](#related-rules) on `merr.Wrap` vs `merr.WrapErr*Err`.)
## Naming convention
The convention has two layers, matched to the two error categories:
### Wire-protocol layer (typed merr) — uppercase `Err*` in `pkg/util/merr` only
All errors that may cross any gRPC boundary (client-facing or
component-to-component) must be `*merr.milvusError` defined in
`pkg/util/merr/errors.go`. They have:
- A numeric code passed to `newMilvusError(...)`; uniqueness is enforced by
the init-time code registry (defining a second sentinel on an occupied code
panics at package init, since `milvusError.Is` matches by code alone)
- A `var ErrXxx = newMilvusError(...)` declaration in `pkg/util/merr/errors.go`
- An exported `WrapErrXxxMsg` / `WrapErrXxxErr` helper
If an error needs to be visible to the wire, it lives here. No exceptions.
Every sentinel also carries an Input-vs-System classification (who is to
blame) that drives `Status.Retriable`, the `fail_input`/`fail_system` metric
labels, lb_policy failover and `retry.Do`; see "Input vs System: who is to
blame?" in [error_handling_guide.md](error_handling_guide.md).
### Internal-sentinel layer — lowercase `errXxx`, same-package only
Internal sentinels live in `internal/...` packages, are created with
`errors.New(...)`, and are **lowercase / unexported**. The rule:
> **A `var err* = errors.New(...)` declared in `internal/...` may only be
> referenced inside the same Go package. Cross-package consumers must not
> see it.**
This makes Go visibility do the enforcement: if you need a signal across
package boundaries, you either (a) lift it into the wire layer as a typed
merr, or (b) redesign the API so the signal flows via a return value
(e.g. `(ignored bool, err error)`), not via the error type.
Example (current code, after the 04-coord cleanup):
```go
// errFull / errNoSuchElement are INTERNAL sentinels: caught by errors.Is
// inside the compaction inspector / scheduler loop and never serialized
// across any gRPC boundary.
var (
errFull = errors.New("compaction queue is full")
errNoSuchElement = errors.New("compaction queue has no element")
)
```
### Cross-package idempotency: use a return-value flag, not an exported sentinel
The previous code exported `meta.ErrResourceGroupOperationIgnored` so that
the parent package `querycoordv2` could catch it via `errors.Is` and
translate to `merr.Success()`. This was an exported `Err*` in
`internal/...` — visually indistinguishable from a `merr.ErrXxx` typed
wire error, easy to misuse.
The current code instead encodes the signal in the return value:
```go
// meta/resource_manager.go
func (rm *ResourceManager) CheckIfResourceGroupAddable(...) (ignored bool, err error) {
if proto.Equal(rm.groups[rgName].GetConfig(), cfg) {
return true, nil // idempotent no-op
}
...
}
// querycoordv2/ddl_callbacks_alter_resource_group.go (broadcaster)
func (s *Server) broadcastCreateResourceGroup(...) (ignored bool, err error) {
if ignored, err := s.meta.CheckIfResourceGroupAddable(...); err != nil || ignored {
return ignored, err
}
...
}
// querycoordv2/services.go (RPC handler)
ignored, err := s.broadcastCreateResourceGroup(ctx, req)
if err != nil { return merr.Status(err), nil }
if ignored { return merr.Success(), nil }
```
No sentinel crosses the package boundary; the signal travels via a
structured return value. This is the preferred pattern for any new
cross-package idempotency case.
## Current state (audit done on err-std-04-coord branch, 2026-05-19)
Across `internal/{datacoord,rootcoord,querycoordv2}` there are 28
`errors.New(...)` sentinels. Their fates:
| Kind | Count | Examples | Status |
|---|---|---|---|
| Idempotency: caught → `merr.Success()` | 13 catch sites, ~12 distinct sentinels | `errIgnoredAlterAlias`, `errIgnoredCreateCollection`, `errReleaseCollectionNotLoaded`, `errUserNotFound`, ... | ✅ compliant |
| Caught → translated to typed merr | 3 catch sites (`errUserAlreadyExists`, `errRoleAlreadyExists`, `errRoleNotExists`) | client gets `WrapErrParameterInvalidMsg(...)` or `WrapErrServiceInternalMsg(...)` | ✅ compliant (1100 / 5) |
| Background-only (never enter an RPC handler) | 5 (`errFull`, `errNoSuchElement`, `errNodeNotEnough`, `errDisposed`, `errTypeNotFound`) | compaction queue / resource observer / session lifecycle / checker registry | ✅ compliant |
| Cross-package idempotency via `(ignored bool, err error)` signature | 1 (resource group create/drop) | meta layer returns `ignored=true`; querycoordv2 RPC handler translates to `merr.Success()` — no sentinel crosses package | ✅ compliant (refactored from exported `ErrResourceGroupOperationIgnored` in this branch) |
| Dead code (function with 0 callers) | 3 (`errNilResponse`, `errNilStatusResponse`, `errUnknownResponseType` — all only used by `VerifyResponse` in `datacoord/util.go`) | safe to delete | 🪦 cleanup candidate |
| ~~**Violation**: escapes RPC handler with no catch~~ (resolved) | 3 (`errEmptyUsername`, `errEmptyRoleName`, `errEmptyPrivilegeGroupName`) | origin sites in `meta_table.go` now emit `WrapErrParameterInvalidMsg` directly; the bare sentinels are gone | ✅ resolved |
| ~~**Semantic miscategorization**: caught and wrapped with the wrong typed merr code~~ (resolved) | 1 (`errTypeNotFound` in `ops_services.go:87,101`) | invalid `CheckerID` from the client now wrapped as `WrapErrParameterInvalidMsg` (code 1100), was `WrapErrServiceInternal` (code 5) | ✅ resolved |
### Cleanup status
1. ✅ Done — `errEmptyUsername` / `errEmptyRoleName` / `errEmptyPrivilegeGroupName`
at the origin sites in `internal/rootcoord/meta_table.go` now emit
`merr.WrapErrParameterInvalidMsg("username is empty")` etc. directly; the bare
sentinels no longer exist.
2. ✅ Done — `errTypeNotFound` at the catcher in
`internal/querycoordv2/ops_services.go:87,101` is now wrapped as
`merr.WrapErrParameterInvalidMsg("invalid checker type %d: %v", req.CheckerID, err)`.
3. Not done — delete `VerifyResponse` and its three dead-code sentinels
(`errNilResponse`, `errNilStatusResponse`, `errUnknownResponseType`).
## Future linter ideas
Three candidates, in order of how cheap they are to implement and how
hard the enforcement is. **Tier 1.5's return form is now implemented (see
below); Tier 1 and Tier 2 remain a design queue.**
### Tier 1 — exported-sentinel ban (1 hour to write)
The simplest rule: **`internal/...` packages may not declare exported
`var Err\w+ = errors.New(...)`.** Scan all `internal/...` *.go files,
fail CI if any match. Two paths to fix a violation:
1. Lowercase it (`var errXxx = errors.New(...)`) — only callable inside the
same package. If the lint fails because a cross-package caller needs the
signal, see fix 2.
2. Refactor the API so the signal travels via a return value
(e.g. add `ignored bool` to the return tuple) and delete the sentinel.
This makes Go visibility itself the enforcement mechanism: anything that
needs to look like `merr.ErrXxx` to a reviewer can only exist in
`pkg/util/merr`. Internal sentinels stay quietly lowercase in their owning
package.
Lowercase sentinels (`var errXxx = errors.New(...)`) inside `internal/...`
are still encouraged to carry an `INTERNAL: ...` doc comment for reviewer
context, but it's not enforced — the visibility rule already prevents the
worst-case (`Err*` collision with `merr.ErrXxx`).
### Tier 1.5 — bare-usage ban (1 hour grep, half-day AST for 100% precision)
> **Status — return form implemented.** The **return** form of this ban is now
> enforced by a `gocritic`/`ruleguard` rule (`rawmerrerror` in `rules.go`), run
> under `make verifiers`: it rejects `return errors.New / fmt.Errorf /
> errors.Errorf` from function bodies (package-level sentinels, `cmd/`, `tests/`,
> codegen and the walimpls harness exempt). The day-to-day guide is
> [error_handling_guide.md](./error_handling_guide.md). The **no-exceptions**
> form below (local `:=`, `panic(...)`, function argument) is *not* covered:
> ruleguard's DSL cannot match "a call anywhere in a function body but not in a
> `ValueSpec`", so the full ban still needs the AST-based Tier 2 linter.
**Hardest enforcement, no exceptions.** `internal/...` packages may not
use `errors.New(...)` or `errors.Errorf(...)` **inline inside a function
body**. The only legal site for these calls is a package-level
`var <Name> = errors.New(...)` (sentinel declaration).
Allow/deny matrix:
| Form | Location | Verdict |
|---|---|---|
| `var errInvalid = errors.New("invalid")` | package-level (file top / `var` block) | ✅ allowed |
| `var ( errA = ...; errB = ... )` | package-level `var` block | ✅ allowed |
| `return errors.New(...)` | function body | ❌ banned |
| `x := errors.New(...)` | function body local | ❌ banned |
| `panic(errors.New(...))` | function body | ❌ banned |
| `foo(errors.New(...))` | function body argument | ❌ banned |
**Why no exceptions** (even for "local break signal" / "log-only" /
"panic-bound" cases that look harmless today):
- Today's local var can be hoisted to package-level by tomorrow's refactor
and silently start crossing boundaries.
- A linter with exceptions needs AST-level wire-reachability analysis
(expensive); a no-exception linter is one grep.
- Forces authors to use the right primitive instead of `errors.New` as
a universal escape hatch:
- **break signal from a callback** → define a `type doneSignal struct{}`
that implements `error` and use `errors.As`. Intent is now in the type,
not in a string-keyed sentinel.
- **"unreachable" assertion** → just `panic(...)`. If caller already
does `if err != nil { panic(err) }`, fold it into the callee.
- **input validation / config validation** → `merr.WrapErrParameterInvalidMsg(...)`
or `status.NewInvalidArgument(...)` depending on layer.
Implementation: grep version covers ~95% true violations in ~1 hour.
AST version (go/analysis) covers the edge cases (e.g. `init()` body
assigning to a package var) but needs ~half a day. Start with grep,
upgrade if false-positive rate exceeds 5%.
```bash
# grep skeleton
grep -rnE 'errors\.(New|Errorf)\(' internal/ --include='*.go' \
| grep -v _test.go \
| grep -vE ':[0-9]+:\s*(var\s+)?[A-Za-z_]+\s*=\s*errors\.(New|Errorf)' \
| grep -vE ':[0-9]+:\s*[A-Za-z_]+\s+(\w+\s+)?=\s*errors\.(New|Errorf)'
# Any remaining line = violation
```
A `//nolint:err-bare` escape valve with a required justification comment
handles the genuine outliers (a few `init()` patterns, embedded `errors.Mark`
usage, etc.).
### Tier 2 — escape-path linter (~1 day, go/analysis based)
For every gRPC handler method (anything matching the
`internal/{rootcoord,datacoord,querycoordv2}/services.go,root_coord.go,*_handler.go`
pattern, return type `(*proto.XxxResponse, error)` or `(*commonpb.Status,
error)`), trace the err-return data-flow. Any error that:
- transitively originates from an `INTERNAL:`-tagged sentinel, **and**
- reaches a `return Status{Code: merr.Status(err)}` or `return err` without
passing through an `errors.Is(err, internalSentinelX) { ... }` branch,
is a violation. Report file:line of the leak.
This catches the actual invariant violation (the 3 RBAC empty sentinels
would have been flagged), not just naming hygiene. Requires AST analysis;
worth doing if the cost of one more silent `Code=1` to a client is high.
### Tier 3 (no longer necessary if Tier 2 is in place) — wrap-rule linter
Scan `merr\.WrapErr[A-Za-z]+Err\(err, ` (note: first arg `err`, not a
fresh string-only origination) and require the cause `err` to not itself be
a typed merr. This is what
[`feedback_merr_wrap_rule`](../../) already enforces by convention; Tier 2
would catch the symptom (sentinel escape) as a side effect.
## Related rules
- `feedback_merr_wrap_rule` (this repo's collaboration memory): "Add context
to an existing err with `merr.Wrap` / `merr.Wrapf`, never with
`merr.WrapErr*Err` — the latter masks the inner typed code and
retriability (the `errors.Is` chain itself is preserved via `Unwrap()`)."
- `project_errstd_autogen_defects`: three systematic defects in the
auto-generated `errors.Wrap → merr` conversion in this branch series;
defects #2 and #3 are direct consequences of violating the rules in
this document.
@@ -1,5 +1,14 @@
## Appendix D. Error Code
> **⚠️ Deprecated.** This appendix documents the legacy `commonpb.ErrorCode`
> enum (Milvus 2.0-era), which is no longer the source of truth for error codes.
> Current error codes are the `merr` sentinel codes (e.g. `ParameterInvalid`
> = 1100, `ServiceInternal` = 5, `FunctionFailed` = 2400) carried in
> `commonpb.Status.Code`. The canonical list is the sentinel definitions in
> [`pkg/util/merr/errors.go`](../../pkg/util/merr/errors.go); see also
> [error_handling_guide.md](../dev/error_handling_guide.md) and
> [error_sentinel_convention.md](../dev/error_sentinel_convention.md).
**ErrorCode**
```protobuf
+2 -2
View File
@@ -2,9 +2,9 @@ module github.com/milvus-io/milvus/examples/telemetry_e2e_test
go 1.26.4
require (
github.com/milvus-io/milvus-proto/go-api/v2 v2.6.6-0.20260304062516-e7e54d966ef2
github.com/milvus-io/milvus-proto/go-api/v2 v2.6.6-0.20260323081523-53649783989c
github.com/milvus-io/milvus/client/v2 v2.6.4-0.20251104142533-a2ce70d25256
google.golang.org/grpc v1.71.0
google.golang.org/grpc v1.79.3
)
replace (
+2
View File
@@ -1,2 +1,4 @@
github.com/milvus-io/milvus-proto/go-api/v2 v2.6.6-0.20260304062516-e7e54d966ef2/go.mod h1:/6UT4zZl6awVeXLeE7UGDWZvXj3IWkRsh3mqsn0DiAs=
github.com/milvus-io/milvus-proto/go-api/v2 v2.6.6-0.20260323081523-53649783989c/go.mod h1:/6UT4zZl6awVeXLeE7UGDWZvXj3IWkRsh3mqsn0DiAs=
google.golang.org/grpc v1.71.0/go.mod h1:H0GRtasmQOh9LkFoCPDu3ZrwUtD1YGE+b2vYBYd/8Ec=
google.golang.org/grpc v1.79.3/go.mod h1:KmT0Kjez+0dde/v2j9vzwoAScgEPx/Bw1CYChhHLrHQ=
+11 -10
View File
@@ -8,6 +8,7 @@ import (
"github.com/milvus-io/milvus-proto/go-api/v3/schemapb"
"github.com/milvus-io/milvus/pkg/v3/proto/planpb"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
)
const (
@@ -57,7 +58,7 @@ func isSupportedAggregateName(aggregateName string) bool {
func NewAggregate(aggregateName string, aggFieldID int64, originalName string, fieldType schemapb.DataType) ([]AggregateBase, error) {
if !isSupportedAggregateName(aggregateName) {
return nil, fmt.Errorf("invalid Aggregation operator %s", aggregateName)
return nil, merr.WrapErrParameterInvalidMsg("invalid Aggregation operator %s", aggregateName)
}
if err := ValidateAggFieldType(aggregateName, fieldType); err != nil {
@@ -78,7 +79,7 @@ func NewAggregate(aggregateName string, aggFieldID int64, originalName string, f
return []AggregateBase{&MaxAggregate{fieldID: aggFieldID, originalName: originalName}}, nil
default:
// should never happen due to isSupportedAggregateName check
return nil, fmt.Errorf("invalid Aggregation operator %s", aggregateName)
return nil, merr.WrapErrParameterInvalidMsg("invalid Aggregation operator %s", aggregateName)
}
}
@@ -93,13 +94,13 @@ func FromPB(pb *planpb.Aggregate) (AggregateBase, error) {
case planpb.AggregateOp_max:
return &MaxAggregate{fieldID: pb.GetFieldId()}, nil
default:
return nil, fmt.Errorf("invalid Aggregation operator %d", pb.Op)
return nil, merr.WrapErrParameterInvalidMsg("invalid Aggregation operator %d", pb.Op)
}
}
func AccumulateFieldValue(target *FieldValue, new *FieldValue) error {
if target == nil || new == nil {
return fmt.Errorf("target or new field value is nil")
return merr.WrapErrServiceInternalMsg("target or new field value is nil")
}
// Skip null values during accumulation
@@ -121,7 +122,7 @@ func AccumulateFieldValue(target *FieldValue, new *FieldValue) error {
}
// Verify types match
if reflect.TypeOf(target.val) != reflect.TypeOf(new.val) {
return fmt.Errorf("type mismatch: target is %T, new is %T", target.val, new.val)
return merr.WrapErrParameterInvalidMsg("type mismatch: target is %T, new is %T", target.val, new.val)
}
switch target.val.(type) {
@@ -136,7 +137,7 @@ func AccumulateFieldValue(target *FieldValue, new *FieldValue) error {
case float64:
target.val = target.val.(float64) + new.val.(float64)
default:
return fmt.Errorf("unsupported type: %T", target.val)
return merr.WrapErrParameterInvalidMsg("unsupported type: %T", target.val)
}
return nil
}
@@ -255,17 +256,17 @@ func (bucket *Bucket) RowCount() int {
func (bucket *Bucket) Accumulate(row *Row, idx int, keyCount int, aggs []AggregateBase) error {
if idx >= len(bucket.rows) || idx < 0 {
return fmt.Errorf("wrong idx:%d for bucket", idx)
return merr.WrapErrParameterInvalidMsg("wrong idx:%d for bucket", idx)
}
targetRow := bucket.rows[idx]
if targetRow == nil {
return fmt.Errorf("nil row at the target idx:%d, cannot accumulate the row", idx)
return merr.WrapErrServiceInternalMsg("nil row at the target idx:%d, cannot accumulate the row", idx)
}
if row.ColumnCount() != targetRow.ColumnCount() {
return fmt.Errorf("column count:%d in the row must be equal to the target row:%d", row.ColumnCount(), bucket.rows[idx].ColumnCount())
return merr.WrapErrParameterInvalidMsg("column count:%d in the row must be equal to the target row:%d", row.ColumnCount(), bucket.rows[idx].ColumnCount())
}
if row.ColumnCount() != keyCount+len(aggs) {
return fmt.Errorf("column count:%d in the row must be sum of keyCount:%d and the number of aggs:%d", row.ColumnCount(), keyCount, len(aggs))
return merr.WrapErrParameterInvalidMsg("column count:%d in the row must be sum of keyCount:%d and the number of aggs:%d", row.ColumnCount(), keyCount, len(aggs))
}
for col := keyCount; col < row.ColumnCount(); col++ {
targetRow.UpdateFieldValue(row, col, aggs[col-keyCount])
+23 -24
View File
@@ -2,7 +2,6 @@ package agg
import (
"context"
"fmt"
"github.com/milvus-io/milvus-proto/go-api/v3/schemapb"
typeutil2 "github.com/milvus-io/milvus/internal/util/typeutil"
@@ -88,15 +87,15 @@ func (reducer *GroupAggReducer) EmptyAggResult() (*AggregationResult, error) {
} else {
field, err := helper.GetFieldFromID(agg.GetFieldId())
if err != nil {
return nil, fmt.Errorf("failed to get field schema for aggregate fieldID %d: %w", agg.GetFieldId(), err)
return nil, merr.Wrapf(err, "failed to get field schema for aggregate fieldID %d", agg.GetFieldId())
}
resultType, err := getAggregateResultType(agg.GetOp(), field.GetDataType())
if err != nil {
return nil, fmt.Errorf("failed to get result type for aggregate fieldID %d: %w", agg.GetFieldId(), err)
return nil, merr.Wrapf(err, "failed to get result type for aggregate fieldID %d", agg.GetFieldId())
}
emptyFieldData, err := genEmptyFieldDataByType(resultType)
if err != nil {
return nil, fmt.Errorf("failed to generate empty field data for result type %s: %w", resultType.String(), err)
return nil, merr.Wrapf(err, "failed to generate empty field data for result type %s", resultType.String())
}
ret.fieldDatas = append(ret.fieldDatas, emptyFieldData)
}
@@ -160,7 +159,7 @@ func genEmptyFieldDataByType(dataType schemapb.DataType) (*schemapb.FieldData, e
return genEmptyLongFieldData(dataType, []int64{0}), nil
default:
// For other types, try to use the original field's GenEmptyFieldData
return nil, fmt.Errorf("unsupported data type for aggregate result: %s", dataType.String())
return nil, merr.WrapErrParameterInvalidMsg("unsupported data type for aggregate result: %s", dataType.String())
}
}
@@ -187,10 +186,10 @@ func getAggregateResultType(op planpb.AggregateOp, inputType schemapb.DataType)
case schemapb.DataType_Float, schemapb.DataType_Double:
return schemapb.DataType_Double, nil
default:
return schemapb.DataType_None, fmt.Errorf("unsupported input type %s for sum aggregation", inputType.String())
return schemapb.DataType_None, merr.WrapErrParameterInvalidMsg("unsupported input type %s for sum aggregation", inputType.String())
}
default:
return schemapb.DataType_None, fmt.Errorf("unknown aggregate operator: %d", op)
return schemapb.DataType_None, merr.WrapErrParameterInvalidMsg("unknown aggregate operator: %d", op)
}
}
@@ -201,12 +200,12 @@ func getAggregateResultType(op planpb.AggregateOp, inputType schemapb.DataType)
// 3. Each fieldData's Type matches the expected type from schema
func (reducer *GroupAggReducer) validateAggregationResults(results []*AggregationResult) error {
if reducer.schema == nil {
return fmt.Errorf("schema is nil, cannot validate field types")
return merr.WrapErrParameterInvalidMsg("schema is nil, cannot validate field types")
}
helper, err := typeutil.CreateSchemaHelper(reducer.schema)
if err != nil {
return fmt.Errorf("failed to create schema helper: %w", err)
return merr.Wrap(err, "failed to create schema helper")
}
numGroupingKeys := len(reducer.groupByFieldIds)
@@ -220,7 +219,7 @@ func (reducer *GroupAggReducer) validateAggregationResults(results []*Aggregatio
for _, fieldID := range reducer.groupByFieldIds {
field, err := helper.GetFieldFromID(fieldID)
if err != nil {
return fmt.Errorf("failed to get field schema for groupBy fieldID %d: %w", fieldID, err)
return merr.Wrapf(err, "failed to get field schema for groupBy fieldID %d", fieldID)
}
expectedTypes = append(expectedTypes, field.GetDataType())
}
@@ -234,11 +233,11 @@ func (reducer *GroupAggReducer) validateAggregationResults(results []*Aggregatio
} else {
field, err := helper.GetFieldFromID(agg.GetFieldId())
if err != nil {
return fmt.Errorf("failed to get field schema for aggregate fieldID %d: %w", agg.GetFieldId(), err)
return merr.Wrapf(err, "failed to get field schema for aggregate fieldID %d", agg.GetFieldId())
}
expectedType, err = getAggregateResultType(agg.GetOp(), field.GetDataType())
if err != nil {
return fmt.Errorf("failed to get aggregate result type for aggregate fieldID %d: %w", agg.GetFieldId(), err)
return merr.Wrapf(err, "failed to get aggregate result type for aggregate fieldID %d", agg.GetFieldId())
}
}
expectedTypes = append(expectedTypes, expectedType)
@@ -247,27 +246,27 @@ func (reducer *GroupAggReducer) validateAggregationResults(results []*Aggregatio
// Validate each result
for resultIdx, result := range results {
if result == nil {
return fmt.Errorf("result at index %d is nil", resultIdx)
return merr.WrapErrServiceInternalMsg("result at index %d is nil", resultIdx)
}
fieldDatas := result.GetFieldDatas()
// Check 1: fieldDatas length
if len(fieldDatas) != expectedColumnCount {
return fmt.Errorf("result at index %d has fieldDatas length %d, expected %d (numGroupingKeys=%d, numAggs=%d)",
return merr.WrapErrServiceInternalMsg("result at index %d has fieldDatas length %d, expected %d (numGroupingKeys=%d, numAggs=%d)",
resultIdx, len(fieldDatas), expectedColumnCount, numGroupingKeys, numAggs)
}
// Check 2: no nil fieldData and Check 3: type matching
for colIdx, fieldData := range fieldDatas {
if fieldData == nil {
return fmt.Errorf("result at index %d has nil fieldData at column %d", resultIdx, colIdx)
return merr.WrapErrServiceInternalMsg("result at index %d has nil fieldData at column %d", resultIdx, colIdx)
}
expectedType := expectedTypes[colIdx]
actualType := fieldData.GetType()
if actualType != expectedType {
return fmt.Errorf("result at index %d, column %d has type %s, expected %s",
return merr.WrapErrServiceInternalMsg("result at index %d, column %d has type %s, expected %s",
resultIdx, colIdx, schemapb.DataType_name[int32(actualType)], schemapb.DataType_name[int32(expectedType)])
}
}
@@ -352,15 +351,15 @@ func (reducer *GroupAggReducer) Reduce(ctx context.Context, results []*Aggregati
limitReached := false
for _, result := range results {
if result == nil {
return nil, fmt.Errorf("input result from any sources cannot be nil")
return nil, merr.WrapErrServiceInternalMsg("input result from any sources cannot be nil")
}
reducedResult.allRetrieveCount += result.GetAllRetrieveCount()
fieldDatas := result.GetFieldDatas()
if outputColumnCount != len(fieldDatas) {
return nil, fmt.Errorf("retrieved results from different segments have different size of columns")
return nil, merr.WrapErrServiceInternalMsg("retrieved results from different segments have different size of columns")
}
if outputColumnCount == 0 {
return nil, fmt.Errorf("retrieved results have no column data")
return nil, merr.WrapErrServiceInternalMsg("retrieved results have no column data")
}
rowCount := -1
for i := 0; i < outputColumnCount; i++ {
@@ -374,11 +373,11 @@ func (reducer *GroupAggReducer) Reduce(ctx context.Context, results []*Aggregati
rowCount = hashers[i].RowCount()
} else if i < numGroupingKeys {
if rowCount != hashers[i].RowCount() {
return nil, fmt.Errorf("field data:%d for different columns have different row count, %d vs %d, wrong state",
return nil, merr.WrapErrServiceInternalMsg("field data:%d for different columns have different row count, %d vs %d, wrong state",
i, rowCount, hashers[i].RowCount())
}
} else if rowCount != accumulators[i-numGroupingKeys].RowCount() {
return nil, fmt.Errorf("field data:%d for different columns have different row count, %d vs %d, wrong state",
return nil, merr.WrapErrServiceInternalMsg("field data:%d for different columns have different row count, %d vs %d, wrong state",
i, rowCount, accumulators[i-numGroupingKeys].RowCount())
}
}
@@ -443,9 +442,9 @@ func (reducer *GroupAggReducer) Reduce(ctx context.Context, results []*Aggregati
}
}
if totalGroupCount > maxGroupByGroups {
return nil, merr.WrapErrServiceInternal(fmt.Sprintf("GROUP BY produced too many groups (%d). "+
return nil, merr.WrapErrParameterInvalidMsg("GROUP BY produced too many groups (%d). "+
"Add filters or increase common.groupBy.maxGroups (current: %d)",
totalGroupCount, maxGroupByGroups))
totalGroupCount, maxGroupByGroups)
}
// Don't guarantee specific groups to be returned before milvus support order by
}
@@ -478,7 +477,7 @@ func SegcoreResults2AggResult(results []*segcorepb.RetrieveResults) ([]*Aggregat
aggResults := make([]*AggregationResult, len(results))
for i := 0; i < len(results); i++ {
if results[i] == nil {
return nil, fmt.Errorf("input segcore query results from any sources cannot be nil")
return nil, merr.WrapErrServiceInternalMsg("input segcore query results from any sources cannot be nil")
}
fieldsData := results[i].GetFieldsData()
allRetrieveCount := results[i].GetAllRetrieveCount()
+20 -19
View File
@@ -9,6 +9,7 @@ import (
"unsafe"
"github.com/milvus-io/milvus-proto/go-api/v3/schemapb"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
)
func NewFieldAccessor(fieldType schemapb.DataType) (FieldAccessor, error) {
@@ -28,7 +29,7 @@ func NewFieldAccessor(fieldType schemapb.DataType) (FieldAccessor, error) {
case schemapb.DataType_Double:
return newFloat64FieldAccessor(), nil
default:
return nil, fmt.Errorf("unsupported data type for hasher")
return nil, merr.WrapErrParameterInvalidMsg("unsupported data type for hasher")
}
}
@@ -410,7 +411,7 @@ func AssembleSingleValue(fv *FieldValue, fieldData *schemapb.FieldData) error {
case schemapb.DataType_VarChar, schemapb.DataType_String:
fieldData.GetScalars().GetStringData().Data = append(fieldData.GetScalars().GetStringData().GetData(), "")
default:
return fmt.Errorf("unsupported DataType:%d", fieldData.GetType())
return merr.WrapErrParameterInvalidMsg("unsupported DataType:%d", fieldData.GetType())
}
return nil
}
@@ -421,47 +422,47 @@ func AssembleSingleValue(fv *FieldValue, fieldData *schemapb.FieldData) error {
case schemapb.DataType_Bool:
boolVal, ok := val.(bool)
if !ok {
return fmt.Errorf("type assertion failed: expected bool, got %T", val)
return merr.WrapErrServiceInternalMsg("type assertion failed: expected bool, got %T", val)
}
fieldData.GetScalars().GetBoolData().Data = append(fieldData.GetScalars().GetBoolData().GetData(), boolVal)
case schemapb.DataType_Int8, schemapb.DataType_Int16, schemapb.DataType_Int32:
intVal, ok := val.(int32)
if !ok {
return fmt.Errorf("type assertion failed: expected int32, got %T", val)
return merr.WrapErrServiceInternalMsg("type assertion failed: expected int32, got %T", val)
}
fieldData.GetScalars().GetIntData().Data = append(fieldData.GetScalars().GetIntData().GetData(), intVal)
case schemapb.DataType_Int64:
int64Val, ok := val.(int64)
if !ok {
return fmt.Errorf("type assertion failed: expected int64, got %T", val)
return merr.WrapErrServiceInternalMsg("type assertion failed: expected int64, got %T", val)
}
fieldData.GetScalars().GetLongData().Data = append(fieldData.GetScalars().GetLongData().GetData(), int64Val)
case schemapb.DataType_Timestamptz:
timestampVal, ok := val.(int64)
if !ok {
return fmt.Errorf("type assertion failed: expected int64 for Timestamptz, got %T", val)
return merr.WrapErrServiceInternalMsg("type assertion failed: expected int64 for Timestamptz, got %T", val)
}
fieldData.GetScalars().GetTimestamptzData().Data = append(fieldData.GetScalars().GetTimestamptzData().GetData(), timestampVal)
case schemapb.DataType_Float:
floatVal, ok := val.(float32)
if !ok {
return fmt.Errorf("type assertion failed: expected float32, got %T", val)
return merr.WrapErrServiceInternalMsg("type assertion failed: expected float32, got %T", val)
}
fieldData.GetScalars().GetFloatData().Data = append(fieldData.GetScalars().GetFloatData().GetData(), floatVal)
case schemapb.DataType_Double:
doubleVal, ok := val.(float64)
if !ok {
return fmt.Errorf("type assertion failed: expected float64, got %T", val)
return merr.WrapErrServiceInternalMsg("type assertion failed: expected float64, got %T", val)
}
fieldData.GetScalars().GetDoubleData().Data = append(fieldData.GetScalars().GetDoubleData().GetData(), doubleVal)
case schemapb.DataType_VarChar, schemapb.DataType_String:
stringVal, ok := val.(string)
if !ok {
return fmt.Errorf("type assertion failed: expected string, got %T", val)
return merr.WrapErrServiceInternalMsg("type assertion failed: expected string, got %T", val)
}
fieldData.GetScalars().GetStringData().Data = append(fieldData.GetScalars().GetStringData().GetData(), stringVal)
default:
return fmt.Errorf("unsupported DataType:%d", fieldData.GetType())
return merr.WrapErrParameterInvalidMsg("unsupported DataType:%d", fieldData.GetType())
}
return nil
}
@@ -544,13 +545,13 @@ func NewAggregationFieldMap(originalUserOutputFields []string, groupByFields []s
// 2. Global aggregation (no GROUP BY): output_fields can only contain aggregation expressions
// (e.g., "SELECT count(*), int64 FROM t" is invalid SQL — cannot mix aggregates with raw columns)
if numGroupingKeys > 0 {
return nil, fmt.Errorf(
return nil, merr.WrapErrParameterInvalidMsg(
"output field '%s' is not allowed: when using GROUP BY, output_fields can only contain "+
"group_by fields (%v) or aggregation expressions",
outputField, groupByFields,
)
}
return nil, fmt.Errorf(
return nil, merr.WrapErrParameterInvalidMsg(
"output field '%s' is not allowed: when using aggregation functions (e.g., count(*)), "+
"output_fields can only contain aggregation expressions, not regular columns",
outputField,
@@ -566,14 +567,14 @@ func NewAggregationFieldMap(originalUserOutputFields []string, groupByFields []s
// and returns a new Double FieldData containing the average values.
func ComputeAvgFromSumAndCount(sumFieldData *schemapb.FieldData, countFieldData *schemapb.FieldData) (*schemapb.FieldData, error) {
if sumFieldData == nil || countFieldData == nil {
return nil, fmt.Errorf("sumFieldData and countFieldData cannot be nil")
return nil, merr.WrapErrServiceInternalMsg("sumFieldData and countFieldData cannot be nil")
}
sumType := sumFieldData.GetType()
countType := countFieldData.GetType()
if countType != schemapb.DataType_Int64 {
return nil, fmt.Errorf("count field must be Int64 type, got %s", countType.String())
return nil, merr.WrapErrParameterInvalidMsg("count field must be Int64 type, got %s", countType.String())
}
countData := countFieldData.GetScalars().GetLongData().GetData()
@@ -598,27 +599,27 @@ func ComputeAvgFromSumAndCount(sumFieldData *schemapb.FieldData, countFieldData
case schemapb.DataType_Int64:
sumData := sumFieldData.GetScalars().GetLongData().GetData()
if len(sumData) != rowCount {
return nil, fmt.Errorf("sum and count field data must have the same length, got sum:%d, count:%d", len(sumData), rowCount)
return nil, merr.WrapErrParameterInvalidMsg("sum and count field data must have the same length, got sum:%d, count:%d", len(sumData), rowCount)
}
for i := 0; i < rowCount; i++ {
if countData[i] == 0 {
return nil, fmt.Errorf("division by zero: count is 0 at row %d", i)
return nil, merr.WrapErrParameterInvalidMsg("division by zero: count is 0 at row %d", i)
}
resultData = append(resultData, float64(sumData[i])/float64(countData[i]))
}
case schemapb.DataType_Double:
sumData := sumFieldData.GetScalars().GetDoubleData().GetData()
if len(sumData) != rowCount {
return nil, fmt.Errorf("sum and count field data must have the same length, got sum:%d, count:%d", len(sumData), rowCount)
return nil, merr.WrapErrParameterInvalidMsg("sum and count field data must have the same length, got sum:%d, count:%d", len(sumData), rowCount)
}
for i := 0; i < rowCount; i++ {
if countData[i] == 0 {
return nil, fmt.Errorf("division by zero: count is 0 at row %d", i)
return nil, merr.WrapErrParameterInvalidMsg("division by zero: count is 0 at row %d", i)
}
resultData = append(resultData, sumData[i]/float64(countData[i]))
}
default:
return nil, fmt.Errorf("unsupported sum field type for avg computation: %s", sumType.String())
return nil, merr.WrapErrParameterInvalidMsg("unsupported sum field type for avg computation: %s", sumType.String())
}
result.GetScalars().GetDoubleData().Data = resultData
+18 -19
View File
@@ -1,14 +1,13 @@
package agg
import (
"fmt"
"github.com/milvus-io/milvus/pkg/v3/proto/planpb"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
)
func terminateSingleSlot(slots []*FieldValue) (any, error) {
if len(slots) != 1 {
return nil, fmt.Errorf("aggregate expects 1 accumulator slot, got %d", len(slots))
return nil, merr.WrapErrParameterInvalidMsg("aggregate expects 1 accumulator slot, got %d", len(slots))
}
if slots[0] == nil || slots[0].IsNull() {
return nil, nil
@@ -18,7 +17,7 @@ func terminateSingleSlot(slots []*FieldValue) (any, error) {
func terminateAvg(slots []*FieldValue) (any, error) {
if len(slots) != 2 {
return nil, fmt.Errorf("avg expects 2 accumulator slots, got %d", len(slots))
return nil, merr.WrapErrParameterInvalidMsg("avg expects 2 accumulator slots, got %d", len(slots))
}
sum := slots[0]
count := slots[1]
@@ -52,7 +51,7 @@ func fieldValueToFloat64(v any) (float64, error) {
case float64:
return value, nil
default:
return 0, fmt.Errorf("avg expects numeric accumulator, got %T", v)
return 0, merr.WrapErrParameterInvalidMsg("avg expects numeric accumulator, got %T", v)
}
}
@@ -80,7 +79,7 @@ func (sum *SumAggregate) NewState() []*FieldValue {
func (sum *SumAggregate) UpdateState(slots []*FieldValue, new *FieldValue) error {
if len(slots) != 1 {
return fmt.Errorf("aggregate expects 1 accumulator slot, got %d", len(slots))
return merr.WrapErrParameterInvalidMsg("aggregate expects 1 accumulator slot, got %d", len(slots))
}
return sum.Update(slots[0], new)
}
@@ -121,7 +120,7 @@ func (count *CountAggregate) NewState() []*FieldValue {
func (count *CountAggregate) UpdateState(slots []*FieldValue, new *FieldValue) error {
if len(slots) != 1 {
return fmt.Errorf("aggregate expects 1 accumulator slot, got %d", len(slots))
return merr.WrapErrParameterInvalidMsg("aggregate expects 1 accumulator slot, got %d", len(slots))
}
if new == nil || new.IsNull() {
return nil
@@ -166,7 +165,7 @@ func (avg *AvgAggregate) Name() string {
}
func (avg *AvgAggregate) Update(target *FieldValue, new *FieldValue) error {
return fmt.Errorf("avg aggregate updates require aggregate state slots")
return merr.WrapErrServiceInternalMsg("avg aggregate updates require aggregate state slots")
}
func (avg *AvgAggregate) NewState() []*FieldValue {
@@ -175,7 +174,7 @@ func (avg *AvgAggregate) NewState() []*FieldValue {
func (avg *AvgAggregate) UpdateState(slots []*FieldValue, new *FieldValue) error {
if len(slots) != 2 {
return fmt.Errorf("avg expects 2 accumulator slots, got %d", len(slots))
return merr.WrapErrParameterInvalidMsg("avg expects 2 accumulator slots, got %d", len(slots))
}
if err := avg.sum.Update(slots[0], new); err != nil {
return err
@@ -204,7 +203,7 @@ func (avg *AvgAggregate) OriginalName() string {
func updateOrderedState(target *FieldValue, new *FieldValue, op string) error {
if target == nil || new == nil {
return fmt.Errorf("target or new field value is nil")
return merr.WrapErrServiceInternalMsg("target or new field value is nil")
}
if new.IsNull() {
return nil
@@ -234,41 +233,41 @@ func shouldReplaceOrderedValue(target, new any, op string) (bool, error) {
case int:
newVal, ok := new.(int)
if !ok {
return false, fmt.Errorf("type mismatch: target is int, new is %T", new)
return false, merr.WrapErrParameterInvalidMsg("type mismatch: target is int, new is %T", new)
}
return shouldReplaceOrdered(newVal, targetVal, op), nil
case int32:
newVal, ok := new.(int32)
if !ok {
return false, fmt.Errorf("type mismatch: target is int32, new is %T", new)
return false, merr.WrapErrParameterInvalidMsg("type mismatch: target is int32, new is %T", new)
}
return shouldReplaceOrdered(newVal, targetVal, op), nil
case int64:
newVal, ok := new.(int64)
if !ok {
return false, fmt.Errorf("type mismatch: target is int64, new is %T", new)
return false, merr.WrapErrParameterInvalidMsg("type mismatch: target is int64, new is %T", new)
}
return shouldReplaceOrdered(newVal, targetVal, op), nil
case float32:
newVal, ok := new.(float32)
if !ok {
return false, fmt.Errorf("type mismatch: target is float32, new is %T", new)
return false, merr.WrapErrParameterInvalidMsg("type mismatch: target is float32, new is %T", new)
}
return shouldReplaceOrdered(newVal, targetVal, op), nil
case float64:
newVal, ok := new.(float64)
if !ok {
return false, fmt.Errorf("type mismatch: target is float64, new is %T", new)
return false, merr.WrapErrParameterInvalidMsg("type mismatch: target is float64, new is %T", new)
}
return shouldReplaceOrdered(newVal, targetVal, op), nil
case string:
newVal, ok := new.(string)
if !ok {
return false, fmt.Errorf("type mismatch: target is string, new is %T", new)
return false, merr.WrapErrParameterInvalidMsg("type mismatch: target is string, new is %T", new)
}
return shouldReplaceOrdered(newVal, targetVal, op), nil
default:
return false, fmt.Errorf("unsupported type for %s aggregation: %T", op, target)
return false, merr.WrapErrParameterInvalidMsg("unsupported type for %s aggregation: %T", op, target)
}
}
@@ -299,7 +298,7 @@ func (min *MinAggregate) NewState() []*FieldValue {
func (min *MinAggregate) UpdateState(slots []*FieldValue, new *FieldValue) error {
if len(slots) != 1 {
return fmt.Errorf("aggregate expects 1 accumulator slot, got %d", len(slots))
return merr.WrapErrParameterInvalidMsg("aggregate expects 1 accumulator slot, got %d", len(slots))
}
return min.Update(slots[0], new)
}
@@ -339,7 +338,7 @@ func (max *MaxAggregate) NewState() []*FieldValue {
func (max *MaxAggregate) UpdateState(slots []*FieldValue, new *FieldValue) error {
if len(slots) != 1 {
return fmt.Errorf("aggregate expects 1 accumulator slot, got %d", len(slots))
return merr.WrapErrParameterInvalidMsg("aggregate expects 1 accumulator slot, got %d", len(slots))
}
return max.Update(slots[0], new)
}
+2 -3
View File
@@ -1,9 +1,8 @@
package agg
import (
"fmt"
"github.com/milvus-io/milvus-proto/go-api/v3/schemapb"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
)
func isSupportedAggFieldType(aggregateName string, dt schemapb.DataType) bool {
@@ -51,7 +50,7 @@ func isSupportedAggFieldType(aggregateName string, dt schemapb.DataType) bool {
// Note: operator name is expected to be normalized (lowercase) by caller.
func ValidateAggFieldType(aggregateName string, dt schemapb.DataType) error {
if !isSupportedAggFieldType(aggregateName, dt) {
return fmt.Errorf("aggregation operator %s does not support data type %s", aggregateName, dt.String())
return merr.WrapErrParameterInvalidMsg("aggregation operator %s does not support data type %s", aggregateName, dt.String())
}
return nil
}
+4 -7
View File
@@ -18,14 +18,13 @@ package allocator
import (
"context"
"fmt"
"sync"
"time"
"github.com/cockroachdb/errors"
"go.uber.org/zap"
"github.com/milvus-io/milvus/pkg/v3/log"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
)
const (
@@ -249,10 +248,9 @@ func (ta *CachedAllocator) finishSyncRequest() {
func (ta *CachedAllocator) failRemainRequest() {
var err error
if ta.SyncErr != nil {
err = fmt.Errorf("%s failRemainRequest err:%w", ta.Role, ta.SyncErr)
err = merr.Wrapf(ta.SyncErr, "%s failRemainRequest", ta.Role)
} else {
errMsg := fmt.Sprintf("%s failRemainRequest unexpected error", ta.Role)
err = errors.New(errMsg)
err = merr.WrapErrServiceInternalMsg("%s failRemainRequest unexpected error", ta.Role)
}
if len(ta.ToDoReqs) > 0 {
log.Warn("Allocator has some reqs to fail",
@@ -290,8 +288,7 @@ func (ta *CachedAllocator) Close() {
ta.CancelFunc()
ta.wg.Wait()
ta.TChan.Close()
errMsg := fmt.Sprintf("%s is closing", ta.Role)
ta.revokeRequest(errors.New(errMsg))
ta.revokeRequest(merr.WrapErrServiceInternalMsg("%s is closing", ta.Role))
}
// CleanCache is used to force synchronize with global allocator.
+3 -5
View File
@@ -18,14 +18,12 @@ package allocator
import (
"context"
"fmt"
"time"
"github.com/cockroachdb/errors"
"github.com/milvus-io/milvus-proto/go-api/v3/commonpb"
"github.com/milvus-io/milvus/pkg/v3/proto/rootcoordpb"
"github.com/milvus-io/milvus/pkg/v3/util/commonpbutil"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
)
const (
@@ -101,7 +99,7 @@ func (ia *IDAllocator) syncID() (bool, error) {
cancel()
if err != nil {
return false, fmt.Errorf("syncID Failed:%w", err)
return false, merr.Wrapf(err, "syncID failed")
}
ia.idStart = resp.GetID()
ia.idEnd = ia.idStart + int64(resp.GetCount())
@@ -148,7 +146,7 @@ func (ia *IDAllocator) AllocOne() (UniqueID, error) {
// Alloc allocates the id of the count number.
func (ia *IDAllocator) Alloc(count uint32) (UniqueID, UniqueID, error) {
if ia.closed() {
return 0, 0, errors.New("fail to allocate ID, closed allocator")
return 0, 0, merr.WrapErrServiceInternalMsg("fail to allocate ID, closed allocator")
}
req := &IDRequest{BaseRequest: BaseRequest{Done: make(chan error), Valid: false}}
+4 -3
View File
@@ -17,8 +17,9 @@
package allocator
import (
"fmt"
"sync"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
)
// localAllocator implements the Interface.
@@ -40,12 +41,12 @@ func NewLocalAllocator(start, end int64) Interface {
func (a *localAllocator) Alloc(count uint32) (int64, int64, error) {
cnt := int64(count)
if cnt <= 0 {
return 0, 0, fmt.Errorf("non-positive count is not allowed, count=%d", cnt)
return 0, 0, merr.WrapErrParameterInvalidMsg("non-positive count is not allowed, count=%d", cnt)
}
a.mu.Lock()
defer a.mu.Unlock()
if a.idStart+cnt > a.idEnd {
return 0, 0, fmt.Errorf("ID is exhausted, start=%d, end=%d, count=%d", a.idStart, a.idEnd, cnt)
return 0, 0, merr.WrapErrServiceInternalMsg("ID is exhausted, start=%d, end=%d, count=%d", a.idStart, a.idEnd, cnt)
}
start := a.idStart
a.idStart += cnt
+3 -3
View File
@@ -20,7 +20,6 @@ import (
"context"
"crypto/tls"
"github.com/cockroachdb/errors"
"go.uber.org/zap"
"google.golang.org/grpc"
@@ -28,6 +27,7 @@ import (
"github.com/milvus-io/milvus-proto/go-api/v3/milvuspb"
"github.com/milvus-io/milvus/client/v2/milvusclient"
"github.com/milvus-io/milvus/pkg/v3/log"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
)
@@ -52,7 +52,7 @@ func NewMilvusClient(ctx context.Context, cluster *commonpb.MilvusCluster) (Milv
// Build TLS config from per-cluster paramtable config.
tlsConfig, err := buildCDCTLSConfig(cluster.GetClusterId())
if err != nil {
return nil, errors.Wrap(err, "failed to build CDC TLS config")
return nil, merr.Wrap(err, "failed to build CDC TLS config")
}
if tlsConfig != nil {
config.WithTLSConfig(tlsConfig)
@@ -67,7 +67,7 @@ func NewMilvusClient(ctx context.Context, cluster *commonpb.MilvusCluster) (Milv
cli, err := milvusclient.New(ctx, config)
if err != nil {
return nil, errors.Wrap(err, "failed to create milvus client")
return nil, merr.Wrap(err, "failed to create milvus client")
}
return cli, nil
}
+2 -2
View File
@@ -5,11 +5,11 @@ import (
"fmt"
"time"
"github.com/cockroachdb/errors"
clientv3 "go.etcd.io/etcd/client/v3"
"google.golang.org/protobuf/proto"
"github.com/milvus-io/milvus/pkg/v3/proto/streamingpb"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
)
const requestTimeout = 30 * time.Second
@@ -44,7 +44,7 @@ func ListReplicatePChannels(ctx context.Context, etcdCli *clientv3.Client, prefi
meta := &streamingpb.ReplicatePChannelMeta{}
err = proto.Unmarshal([]byte(value), meta)
if err != nil {
return nil, errors.Wrapf(err, "unmarshal replicate pchannel meta %s failed", keys[k])
return nil, merr.Wrapf(err, "unmarshal replicate pchannel meta %s failed", keys[k])
}
channel := &ReplicateChannel{
Key: keys[k],
@@ -35,6 +35,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/streaming/util/message"
"github.com/milvus-io/milvus/pkg/v3/streaming/util/message/adaptor"
"github.com/milvus-io/milvus/pkg/v3/streaming/util/options"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
"github.com/milvus-io/milvus/pkg/v3/util/syncutil"
)
@@ -202,7 +203,7 @@ func (r *channelReplicator) getReplicateCheckpoint() (*utility.ReplicateCheckpoi
}
replicateInfo, err := r.targetClient.GetReplicateInfo(ctx, req)
if err != nil {
return nil, errors.Wrap(err, "failed to get replicate info")
return nil, merr.Wrap(err, "failed to get replicate info")
}
checkpoint := replicateInfo.GetCheckpoint()
@@ -115,7 +115,7 @@ func (q *msgQueue) Enqueue(ctx context.Context, msg message.ImmutableMessage) er
// Optional runtime check: enforce non-decreasing timetick
// if n := len(q.buf); n > 0 {
// if q.buf[n-1].Timetick() > msg.Timetick() {
// return fmt.Errorf("enqueue timetick order violation: last=%d new=%d", q.buf[n-1].Timetick(), msg.Timetick())
// return merr.WrapErrParameterInvalidMsg("enqueue timetick order violation: last=%d new=%d", q.buf[n-1].Timetick(), msg.Timetick())
// }
// }
@@ -165,7 +165,7 @@ func (m *FileResourceObserver) CheckNodeSynced(nodeID int64) bool {
func (m *FileResourceObserver) CheckAllQnReady() error {
// return error if meta is not ready
if m.meta == nil {
return merr.WrapErrParameterInvalidMsg("rootcoord meta is not ready")
return merr.WrapErrServiceUnavailable("rootcoord meta is not ready")
}
resources, version := m.meta.ListFileResource(m.ctx)
@@ -177,7 +177,7 @@ func (m *FileResourceObserver) CheckAllQnReady() error {
var err error
m.distribution.Range(func(nodeID int64, node *NodeInfo) bool {
if node.NodeType == QueryNode && node.Version < version {
err = merr.WrapErrParameterInvalidMsg("node %d file resource not synced", nodeID)
err = merr.WrapErrServiceUnavailableMsg("file resource not synced, node-%d", nodeID)
return false
}
return true
+8 -8
View File
@@ -298,10 +298,10 @@ func (s *mixCoordImpl) getQueryNodes(ctx context.Context) (*querypb.ListQueryNod
Base: commonpbutil.NewMsgBase(),
})
if err != nil {
return nil, fmt.Errorf("failed to list query nodes: %w", err)
return nil, merr.Wrapf(err, "failed to list query nodes")
}
if !merr.Ok(resp.GetStatus()) {
return nil, fmt.Errorf("failed to list query nodes: %s", resp.GetStatus().GetReason())
return nil, merr.Wrapf(merr.Error(resp.GetStatus()), "failed to list query nodes")
}
return resp, nil
}
@@ -514,7 +514,7 @@ func (s *mixCoordImpl) getQueryCoordChannelBalanceActive(ctx context.Context) (b
}
if !merr.Ok(resp.GetStatus()) {
return channelActivate, fmt.Errorf("%s", resp.GetStatus().GetReason())
return channelActivate, merr.Error(resp.GetStatus())
}
return channelActivate && resp.IsActive, nil
@@ -528,7 +528,7 @@ func (s *mixCoordImpl) getQueryCoordBalanceActive(ctx context.Context) (bool, er
return false, err
}
if !merr.Ok(resp.GetStatus()) {
return false, fmt.Errorf("%s", resp.GetStatus().GetReason())
return false, merr.Error(resp.GetStatus())
}
return resp.IsActive, nil
}
@@ -569,7 +569,7 @@ func (s *mixCoordImpl) controlQueryCoordChannelBalanceStatus(ctx context.Context
default:
// If the status is not recognized, return an informative error immediately.
// This avoids proceeding with an invalid state.
err = fmt.Errorf("invalid status value: '%s'. Use 'suspended', 'resumed' or 'active'", status)
err = merr.WrapErrParameterInvalidMsg("invalid status value: '%s'. Use 'suspended', 'resumed' or 'active'", status)
}
// --- Unified Error Handling ---
@@ -578,7 +578,7 @@ func (s *mixCoordImpl) controlQueryCoordChannelBalanceStatus(ctx context.Context
// First, check if there was a low-level error during the method execution (e.g., network issue).
if err != nil {
// Wrap the original error with more context.
return fmt.Errorf("%s: %w", errMsg, err)
return merr.Wrap(err, errMsg)
}
// If no errors were encountered, the operation was successful. Return nil to indicate success.
return nil
@@ -958,11 +958,11 @@ func (s *mixCoordImpl) handleQueryNodeStatusUpdate(ctx context.Context, nodeID i
errMsg = "failed to resume node"
default:
errMsg = "invalid status value. Use 'suspended', 'resumed' or 'active'"
err = fmt.Errorf("%s", errMsg)
err = merr.WrapErrParameterInvalidMsg("%s", errMsg)
}
err = merr.CheckRPCCall(resp, err)
if err != nil {
err = fmt.Errorf("%s: %w", errMsg, err)
err = merr.Wrap(err, errMsg)
}
return err
}
@@ -12,6 +12,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/log"
"github.com/milvus-io/milvus/pkg/v3/streaming/util/types"
"github.com/milvus-io/milvus/pkg/v3/util/funcutil"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/syncutil"
"github.com/milvus-io/milvus/pkg/v3/util/typeutil"
)
@@ -99,7 +100,7 @@ func (s *StreamingNodeManager) GetLatestWALLocated(ctx context.Context, vchannel
}
serverID, ok := balancer.GetLatestWALLocated(ctx, pchannel)
if !ok {
return -1, errors.Errorf("channel: %s not found", vchannel)
return -1, merr.WrapErrChannelNotFound(vchannel)
}
return serverID, nil
}
+4 -4
View File
@@ -19,12 +19,12 @@ package coordinator
import (
"context"
"github.com/cockroachdb/errors"
"go.uber.org/zap"
"github.com/milvus-io/milvus/internal/streamingcoord/server/broadcaster/registry"
"github.com/milvus-io/milvus/pkg/v3/log"
"github.com/milvus-io/milvus/pkg/v3/streaming/util/message"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
)
@@ -66,7 +66,7 @@ func (c *WALCallback) alterWALV2AckCallback(
targetWALName := result.Message.Header().TargetWalName
mqTypeValue := message.WALName(targetWALName).String()
if mqTypeValue == "unknown" {
return errors.Errorf("invalid target WAL name: %v", targetWALName)
return merr.WrapErrServiceInternalMsg("invalid target WAL name: %v", targetWALName)
}
// Get etcd source to update configuration
@@ -74,7 +74,7 @@ func (c *WALCallback) alterWALV2AckCallback(
etcdSource, ok := paramMgr.GetEtcdSource()
if !ok {
logger.Warn("failed to update mq.type config, etcd source not enabled")
return errors.New("etcd source is not enabled, cannot update mq.type configuration")
return merr.WrapErrServiceInternalMsg("etcd source is not enabled, cannot update mq.type configuration")
}
// Update mq.type configuration in etcd
@@ -84,7 +84,7 @@ func (c *WALCallback) alterWALV2AckCallback(
zap.String("configKey", configKey),
zap.String("mqTypeValue", mqTypeValue),
zap.Error(err))
return errors.Wrap(err, "failed to update mq.type configuration in etcd")
return merr.Wrap(err, "failed to update mq.type configuration in etcd")
}
logger.Info("successfully updated mq.type configuration in etcd",
@@ -131,10 +131,13 @@ async fn download_with_retry(
for url in urls {
match client.get(url).send().await {
Ok(resp) if resp.status().is_success() => {
let content = resp
.bytes()
.await
.map_err(|e| TantivyBindingError::InternalError(e.to_string()))?;
let content = match resp.bytes().await {
Ok(c) => c,
Err(e) => {
warn!("Failed to read body from {}: {}", url, e);
continue;
}
};
// Calculate MD5 hash
let mut context = Context::new();
+5 -5
View File
@@ -18,7 +18,6 @@ package datacoord
import (
"context"
"fmt"
"sync"
"go.uber.org/zap"
@@ -28,6 +27,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/log"
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
"github.com/milvus-io/milvus/pkg/v3/proto/workerpb"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/timerecord"
)
@@ -117,7 +117,7 @@ func (m *analyzeMeta) UpdateVersion(taskID int64, nodeID int64) error {
t, ok := m.tasks[taskID]
if !ok {
return fmt.Errorf("there is no task with taskID: %d", taskID)
return merr.WrapErrServiceInternalMsg("there is no task with taskID: %d", taskID)
}
cloneT := proto.Clone(t).(*indexpb.AnalyzeTask)
@@ -134,7 +134,7 @@ func (m *analyzeMeta) BuildingTask(taskID int64) error {
t, ok := m.tasks[taskID]
if !ok {
return fmt.Errorf("there is no task with taskID: %d", taskID)
return merr.WrapErrServiceInternalMsg("there is no task with taskID: %d", taskID)
}
cloneT := proto.Clone(t).(*indexpb.AnalyzeTask)
@@ -150,7 +150,7 @@ func (m *analyzeMeta) UpdateState(taskID int64, state indexpb.JobState, failReas
t, ok := m.tasks[taskID]
if !ok {
return fmt.Errorf("there is no task with taskID: %d", taskID)
return merr.WrapErrServiceInternalMsg("there is no task with taskID: %d", taskID)
}
cloneT := proto.Clone(t).(*indexpb.AnalyzeTask)
@@ -168,7 +168,7 @@ func (m *analyzeMeta) FinishTask(taskID int64, result *workerpb.AnalyzeResult) e
t, ok := m.tasks[taskID]
if !ok {
return fmt.Errorf("there is no task with taskID: %d", taskID)
return merr.WrapErrServiceInternalMsg("there is no task with taskID: %d", taskID)
}
log.Info("finish task meta...", zap.Int64("taskID", taskID), zap.String("state", result.GetState().String()),
+12 -13
View File
@@ -21,10 +21,9 @@ import (
"strconv"
"strings"
"github.com/cockroachdb/errors"
"github.com/milvus-io/milvus/internal/storage"
"github.com/milvus-io/milvus/pkg/v3/proto/datapb"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
)
// BackfillResult is the decoded form of the JSON produced by the Spark
@@ -86,34 +85,34 @@ var knownObjectSchemes = map[string]struct{}{
// 3. Unknown scheme -> return ErrUnsupportedScheme.
func normalizeObjectKey(raw, expectedBucket string) (string, error) {
if raw == "" {
return "", errors.New("empty object path")
return "", merr.WrapErrServiceInternalMsg("empty object path")
}
if !strings.Contains(raw, "://") {
key := strings.TrimPrefix(raw, "/")
if key == "" {
return "", errors.Newf("empty object key in %q", raw)
return "", merr.WrapErrServiceInternalMsg("empty object key in %q", raw)
}
return key, nil
}
idx := strings.Index(raw, "://")
scheme := strings.ToLower(raw[:idx])
if _, ok := knownObjectSchemes[scheme]; !ok {
return "", errors.Newf("unsupported object URI scheme %q in %q", scheme, raw)
return "", merr.WrapErrServiceInternalMsg("unsupported object URI scheme %q in %q", scheme, raw)
}
rest := raw[idx+3:]
slash := strings.Index(rest, "/")
if slash < 0 {
return "", errors.Newf("malformed object URI %q: missing object key", raw)
return "", merr.WrapErrServiceInternalMsg("malformed object URI %q: missing object key", raw)
}
bucket := rest[:slash]
key := rest[slash+1:]
if expectedBucket != "" && bucket != expectedBucket {
return "", errors.Newf("object URI bucket %q differs from datacoord bucket %q (path=%s)", bucket, expectedBucket, raw)
return "", merr.WrapErrServiceInternalMsg("object URI bucket %q differs from datacoord bucket %q (path=%s)", bucket, expectedBucket, raw)
}
// Reject inputs like "s3a://bucket/" that parse to an empty key -- passing
// an empty key to chunkManager.Read has undefined behavior across SDKs.
if key == "" {
return "", errors.Newf("empty object key in %q", raw)
return "", merr.WrapErrServiceInternalMsg("empty object key in %q", raw)
}
return key, nil
}
@@ -147,20 +146,20 @@ func buildV2Groups(bucket string, entry *BackfillSegment) (map[int64]*datapb.Fie
for i := range entry.ColumnGroups {
g := &entry.ColumnGroups[i]
if len(g.FieldIDs) != 1 {
return nil, errors.Newf("backfill invariant violated: column group has %d field_ids (expected 1)", len(g.FieldIDs))
return nil, merr.WrapErrServiceInternalMsg("backfill invariant violated: column group has %d field_ids (expected 1)", len(g.FieldIDs))
}
n := int64(len(g.BinlogFiles))
if n == 0 {
return nil, errors.Newf("column group for field %d has no binlog files", g.FieldIDs[0])
return nil, merr.WrapErrServiceInternalMsg("column group for field %d has no binlog files", g.FieldIDs[0])
}
fid := g.FieldIDs[0]
if _, dup := out[fid]; dup {
return nil, errors.Newf("duplicate column group for field %d", fid)
return nil, merr.WrapErrServiceInternalMsg("duplicate column group for field %d", fid)
}
// row_count flows into EntriesNum; non-positive values are undefined
// (zero collapses presence markers, negatives break accounting).
if g.RowCount <= 0 {
return nil, errors.Newf("column group for field %d has non-positive row_count %d", fid, g.RowCount)
return nil, merr.WrapErrServiceInternalMsg("column group for field %d has non-positive row_count %d", fid, g.RowCount)
}
avg := g.RowCount / n
rem := g.RowCount - avg*n
@@ -176,7 +175,7 @@ func buildV2Groups(bucket string, entry *BackfillSegment) (map[int64]*datapb.Fie
}
logID, ok := parseLogIDFromKey(key)
if !ok {
return nil, errors.Newf("column group for field %d has binlog file %q with non-numeric trailing segment", fid, p)
return nil, merr.WrapErrServiceInternalMsg("column group for field %d has binlog file %q with non-numeric trailing segment", fid, p)
}
// LogPath must be empty at persistence time -- catalog.checkLogID
// rejects any Binlog with LogPath != "" (see
@@ -31,6 +31,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/common"
"github.com/milvus-io/milvus/pkg/v3/log"
"github.com/milvus-io/milvus/pkg/v3/proto/datapb"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
)
@@ -241,7 +242,7 @@ func calculateClusteringCompactionConfig(coll *collectionInfo, view CompactionVi
func estimateRowsBySegmentSize(segments []*SegmentView, expectedSegmentSize int64) (int64, error) {
if expectedSegmentSize <= 0 {
return 0, fmt.Errorf("invalid expected segment size %d", expectedSegmentSize)
return 0, merr.WrapErrServiceInternalMsg("invalid expected segment size %d", expectedSegmentSize)
}
var totalSize float64
@@ -258,17 +259,17 @@ func estimateRowsBySegmentSize(segments []*SegmentView, expectedSegmentSize int6
}
if totalRows == 0 || totalSize == 0 {
return 0, fmt.Errorf("segment view does not contain size info to estimate row count")
return 0, merr.WrapErrServiceInternalMsg("segment view does not contain size info to estimate row count")
}
rowSize := totalSize / float64(totalRows)
if rowSize <= 0 {
return 0, fmt.Errorf("invalid row size calculated from segment view")
return 0, merr.WrapErrServiceInternalMsg("invalid row size calculated from segment view")
}
rows := float64(expectedSegmentSize) / rowSize
if rows <= 0 {
return 0, fmt.Errorf("estimated max row count is not positive")
return 0, merr.WrapErrServiceInternalMsg("estimated max row count is not positive")
}
return int64(rows), nil
@@ -2,7 +2,6 @@ package datacoord
import (
"context"
"fmt"
"math"
"github.com/samber/lo"
@@ -102,7 +101,7 @@ func (policy *forceMergeCompactionPolicy) triggerOneCollection(
configMaxSize := getExpectedSegmentSize(policy.meta, collectionID, collection.Schema)
if targetSizeBytes < configMaxSize {
return nil, 0, merr.WrapErrParameterInvalidMsg(fmt.Sprintf("targetSize %d MB should be greater than or equal to configMaxSize %d MB", targetSize, configMaxSize/(1024*1024)))
return nil, 0, merr.WrapErrParameterInvalidMsg("targetSize %d MB should be greater than or equal to configMaxSize %d MB", targetSize, configMaxSize/(1024*1024))
}
segments := policy.meta.SelectSegments(ctx, WithCollection(collectionID), SegmentFilterFunc(func(segment *SegmentInfo) bool {
@@ -182,7 +181,7 @@ var _ CollectionTopologyQuerier = (*metricsNodeMemoryQuerier)(nil)
func (q *metricsNodeMemoryQuerier) GetCollectionTopology(ctx context.Context, collectionID int64) (*CollectionTopology, error) {
log := log.Ctx(ctx).With(zap.Int64("collectionID", collectionID))
if q.mixCoord == nil {
return nil, fmt.Errorf("mixCoord not available for topology query")
return nil, merr.WrapErrServiceInternalMsg("mixCoord not available for topology query")
}
// 1. Get replica information
+7 -4
View File
@@ -74,9 +74,12 @@ func (pq *PriorityQueue[T]) Update(item *Item[T], value T, priority int) {
heap.Fix(pq, item.index)
}
// errFull / errNoSuchElement are INTERNAL sentinels: caught by errors.Is
// inside the compaction inspector / scheduler loop and never serialized
// across any gRPC boundary. See docs/dev/error_sentinel_convention.md.
var (
ErrFull = errors.New("compaction queue is full")
ErrNoSuchElement = errors.New("compaction queue has no element")
errFull = errors.New("compaction queue is full")
errNoSuchElement = errors.New("compaction queue has no element")
)
type Prioritizer func(t CompactionTask) int
@@ -101,7 +104,7 @@ func (q *CompactionQueue) Enqueue(t CompactionTask) error {
q.lock.Lock()
defer q.lock.Unlock()
if q.capacity > 0 && len(q.pq) >= q.capacity {
return ErrFull
return errFull
}
heap.Push(&q.pq, &Item[CompactionTask]{value: t, priority: q.prioritizer(t)})
@@ -113,7 +116,7 @@ func (q *CompactionQueue) Dequeue() (CompactionTask, error) {
defer q.lock.Unlock()
if len(q.pq) == 0 {
return nil, ErrNoSuchElement
return nil, errNoSuchElement
}
item := heap.Pop(&q.pq).(*Item[CompactionTask])
@@ -23,7 +23,6 @@ import (
"strconv"
"time"
"github.com/cockroachdb/errors"
"github.com/samber/lo"
"go.uber.org/atomic"
"go.uber.org/zap"
@@ -629,7 +628,7 @@ func (t *clusteringCompactionTask) processAnalyzing() error {
}
case indexpb.JobState_JobStateFailed:
log.Warn("analyze task fail", zap.Int64("analyzeID", t.GetTaskProto().GetAnalyzeTaskID()))
return errors.New(analyzeTask.FailReason)
return merr.WrapErrServiceInternalMsg(analyzeTask.FailReason)
default:
}
return nil
+2 -2
View File
@@ -319,7 +319,7 @@ func (t *l0CompactionTask) selectFlushedSegment() ([]*SegmentInfo, []*datapb.Com
for _, info := range flushedSegments {
// Sealed is unexpected, fail fast
if info.GetState() == commonpb.SegmentState_Sealed {
return nil, nil, fmt.Errorf("L0 compaction selected invalid sealed segment %d", info.GetID())
return nil, nil, merr.WrapErrServiceInternalMsg("L0 compaction selected invalid sealed segment %d", info.GetID())
}
sealedSegBinlogs = append(sealedSegBinlogs, &datapb.CompactionSegmentBinlogs{
@@ -455,7 +455,7 @@ func buildL0V3DeltaLogEntries(segmentID int64, deltalogs []*datapb.FieldBinlog)
for _, binlog := range fieldBinlog.GetBinlogs() {
path := binlog.GetLogPath()
if path == "" {
return nil, merr.WrapErrServiceInternal(fmt.Sprintf("L0 V3 compaction result missing deltalog path for segment %d, logID %d", segmentID, binlog.GetLogID()))
return nil, merr.WrapErrServiceInternalMsg("L0 V3 compaction result missing deltalog path for segment %d, logID %d", segmentID, binlog.GetLogID())
}
entries = append(entries, packed.DeltaLogEntry{
Path: path,
+2 -2
View File
@@ -398,7 +398,7 @@ func (t *mixCompactionTask) BuildCompactionRequest() (*datapb.CompactionPlan, er
resources, err := t.meta.GetFileResources(context.Background(), taskSchema.GetFileResourceIds()...)
if err != nil {
log.Warn("get file resources for collection failed", zap.Int64("collectionID", taskProto.GetCollectionID()), zap.Error(err))
return nil, errors.Errorf("get file resources for sort compaction failed: %v", err)
return nil, merr.Wrap(err, "get file resources for sort compaction failed")
}
plan.FileResources = resources
}
@@ -411,7 +411,7 @@ func (t *mixCompactionTask) BuildCompactionRequest() (*datapb.CompactionPlan, er
return nil, merr.WrapErrSegmentNotFound(segID)
}
if taskSchemaVersion < segInfo.GetSchemaVersion() {
return nil, merr.WrapErrIllegalCompactionPlan(fmt.Sprintf("compaction task schema version %d is older than input segment %d schema version %d", taskSchemaVersion, segInfo.GetID(), segInfo.GetSchemaVersion()))
return nil, merr.WrapErrIllegalCompactionPlanMsg("compaction task schema version %d is older than input segment %d schema version %d", taskSchemaVersion, segInfo.GetID(), segInfo.GetSchemaVersion())
}
plan.SegmentBinlogs = append(plan.SegmentBinlogs, &datapb.CompactionSegmentBinlogs{
SegmentID: segID,
+5 -2
View File
@@ -225,7 +225,7 @@ func (t *compactionTrigger) getCollection(collectionID UniqueID) (*collectionInf
defer cancel()
coll, err := t.handler.GetCollection(ctx, collectionID)
if err != nil {
return nil, fmt.Errorf("collection ID %d not found, err: %w", collectionID, err)
return nil, merr.Wrapf(err, "collection ID %d not found", collectionID)
}
return coll, nil
}
@@ -588,7 +588,10 @@ func (t *compactionTrigger) getCandidates(signal *compactionSignal) ([]chanPartS
segments := t.meta.SelectSegments(context.TODO(), filters...)
// some criterion not met or conflicted
if len(signal.segmentIDs) > 0 && len(segments) != len(signal.segmentIDs) {
return nil, merr.WrapErrServiceInternal("not all segment ids provided could be compacted")
// SelectSegments also filters segments that are transiently mid-flush /
// compacting / just dropped, so a count mismatch is usually server-side
// state, not a bad id from the caller.
return nil, merr.WrapErrServiceInternalMsg("not all segment ids provided could be compacted")
}
type category struct {
+1 -1
View File
@@ -583,7 +583,7 @@ func AssembleCopySegmentRequest(task CopySegmentTask, job CopySegmentJob) (*data
if _, exists := newBuildIDs[srcBuildID]; !exists {
newID, err := t.alloc.AllocID(ctx)
if err != nil {
return merr.WrapErrServiceInternal(fmt.Sprintf("failed to allocate new buildID for source buildID %d", srcBuildID), err.Error())
return merr.Wrapf(err, "failed to allocate new buildID for source buildID %d", srcBuildID)
}
newBuildIDs[srcBuildID] = newID
}
+8 -9
View File
@@ -27,6 +27,7 @@ import (
"github.com/milvus-io/milvus/internal/streamingcoord/server/broadcaster/registry"
"github.com/milvus-io/milvus/pkg/v3/log"
"github.com/milvus-io/milvus/pkg/v3/streaming/util/message"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
)
// RegisterDDLCallbacks registers the ddl callbacks.
@@ -89,7 +90,7 @@ func (s *Server) startBroadcastWithCollectionID(ctx context.Context, collectionI
func (s *Server) startBroadcastForRestoreSnapshot(ctx context.Context, collectionID int64, snapshotName string) (broadcaster.BroadcastAPI, error) {
coll, err := s.broker.DescribeCollectionInternal(ctx, collectionID)
if err != nil {
return nil, fmt.Errorf("collection %d does not exist: %w", collectionID, err)
return nil, merr.Wrapf(err, "collection %d does not exist", collectionID)
}
dbName := coll.GetDbName()
collectionName := coll.GetCollectionName()
@@ -161,15 +162,15 @@ func (s *Server) validateRestoreSnapshotResources(ctx context.Context, collectio
sourceCollectionID := snapshotData.SnapshotInfo.GetCollectionId()
snapshot, err := s.meta.snapshotMeta.GetSnapshot(ctx, sourceCollectionID, snapshotData.SnapshotInfo.GetName())
if err != nil {
return fmt.Errorf("snapshot %s does not exist for collection %d: %w",
snapshotData.SnapshotInfo.GetName(), sourceCollectionID, err)
return merr.Wrapf(err, "snapshot %s does not exist for collection %d",
snapshotData.SnapshotInfo.GetName(), sourceCollectionID)
}
log.Info("snapshot validated", zap.String("snapshotName", snapshot.GetName()))
// ========== Validate Collection Exists ==========
coll, err := s.broker.DescribeCollectionInternal(ctx, collectionID)
if err != nil {
return fmt.Errorf("collection %d does not exist: %w", collectionID, err)
return merr.Wrapf(err, "collection %d does not exist", collectionID)
}
dbName := coll.GetDbName()
collectionName := coll.GetCollectionName()
@@ -180,7 +181,7 @@ func (s *Server) validateRestoreSnapshotResources(ctx context.Context, collectio
// ========== Validate Partitions Exist ==========
partitionsResp, err := s.broker.ShowPartitions(ctx, collectionID)
if err != nil {
return fmt.Errorf("failed to get partitions for collection %d: %w", collectionID, err)
return merr.Wrapf(err, "failed to get partitions for collection %d", collectionID)
}
// Build set of existing partition names
@@ -192,8 +193,7 @@ func (s *Server) validateRestoreSnapshotResources(ctx context.Context, collectio
// Check all snapshot partitions exist
for partName := range snapshotData.Collection.GetPartitions() {
if !existingPartitions[partName] {
return fmt.Errorf("partition %s does not exist in collection %d",
partName, collectionID)
return merr.WrapErrPartitionNotFound(partName, fmt.Sprintf("partition does not exist in collection %d", collectionID))
}
}
log.Info("partitions validated", zap.Int("count", len(existingPartitions)))
@@ -211,8 +211,7 @@ func (s *Server) validateRestoreSnapshotResources(ctx context.Context, collectio
}
}
if !indexFound {
return fmt.Errorf("index %s for field %d does not exist in collection %d",
indexInfo.GetIndexName(), indexInfo.GetFieldID(), collectionID)
return merr.WrapErrIndexNotFound(indexInfo.GetIndexName(), fmt.Sprintf("index for field %d does not exist in collection %d", indexInfo.GetFieldID(), collectionID))
}
}
log.Info("indexes validated", zap.Int("count", len(snapshotData.Indexes)))
+4 -4
View File
@@ -134,10 +134,10 @@ func (s *Server) validateImportReplication(ctx context.Context, options []*commo
}
if !paramtable.Get().DataCoordCfg.ImportInReplicatingCluster.GetAsBool() {
return merr.WrapErrImportFailed("import in replicating cluster is not supported yet")
return merr.WrapErrOperationNotSupportedMsg("import in replicating cluster is not supported yet")
}
if importutilv2.IsAutoCommit(options) {
return merr.WrapErrImportFailed("auto_commit=true import in replicating cluster is not supported")
return merr.WrapErrOperationNotSupportedMsg("auto_commit=true import in replicating cluster is not supported")
}
return nil
}
@@ -168,14 +168,14 @@ func (s *Server) broadcastImport(ctx context.Context,
// Validate the request before broadcasting
if err := s.validateImportRequest(ctx, msgFiles, options); err != nil {
return errors.Wrap(err, "failed to validate import request")
return merr.Wrap(err, "failed to validate import request")
}
// Get database name from collection metadata via broker
// This is safer than extracting from schema which may be stale
broadcaster, err := s.startBroadcastWithCollectionID(ctx, collectionID)
if err != nil {
return errors.Wrap(err, "failed to start broadcast with collection id")
return merr.Wrap(err, "failed to start broadcast with collection id")
}
defer broadcaster.Close()
@@ -107,8 +107,9 @@ func (s *ImportCallbacksSuite) TestValidateImportRequest_MaxJobsExceededReturnsE
err := server.validateImportRequest(ctx, files, options)
s.Error(err)
// ValidateMaxImportJobExceed returns WrapErrImportFailed, not ErrServiceQuotaExceeded
s.True(errors.Is(err, merr.ErrImportFailed))
// Job-count backpressure is a server-side condition -> ErrImportSysFailed
// (must not be bucketed as fail_input).
s.True(errors.Is(err, merr.ErrImportSysFailed))
s.Contains(err.Error(), "The number of jobs has reached the limit")
}
@@ -185,7 +186,7 @@ func (s *ImportCallbacksSuite) TestValidateImportRequest_ReplicatingClusterRetur
err := server.validateImportRequest(ctx, files, options)
s.Error(err)
s.True(errors.Is(err, merr.ErrImportFailed))
s.True(errors.Is(err, merr.ErrOperationNotSupported))
s.Contains(err.Error(), "replicating cluster")
}
@@ -230,7 +231,7 @@ func (s *ImportCallbacksSuite) TestValidateImportRequest_ReplicatingClusterEnabl
{Key: "timeout", Value: "300s"},
})
s.Error(err)
s.True(errors.Is(err, merr.ErrImportFailed))
s.True(errors.Is(err, merr.ErrOperationNotSupported))
s.Contains(err.Error(), "auto_commit=true")
err = server.validateImportRequest(ctx, files, []*commonpb.KeyValuePair{
@@ -1159,7 +1160,8 @@ func testBroadcastRequiresVchannels(t *testing.T, broadcastFn func(*Server, cont
err := broadcastFn(server, ctx, job)
assert.Error(t, err)
assert.True(t, errors.Is(err, merr.ErrImportFailed))
// Missing vchannels is internal broadcast state -> ErrImportSysFailed.
assert.True(t, errors.Is(err, merr.ErrImportSysFailed))
assert.Contains(t, err.Error(), "job 7 has no vchannels")
}
@@ -636,7 +636,8 @@ func TestValidateRestoreSnapshotResources_IndexNotFound(t *testing.T) {
err := server.validateRestoreSnapshotResources(ctx, 100, buildBaseSnapshotData())
assert.Error(t, err)
assert.Contains(t, err.Error(), "index vec_idx for field 100 does not exist")
assert.Contains(t, err.Error(), "index for field 100 does not exist")
assert.Contains(t, err.Error(), "vec_idx")
}
func TestValidateRestoreSnapshotResources_IndexFieldMismatch(t *testing.T) {
@@ -656,7 +657,8 @@ func TestValidateRestoreSnapshotResources_IndexFieldMismatch(t *testing.T) {
err := server.validateRestoreSnapshotResources(ctx, 100, buildBaseSnapshotData())
assert.Error(t, err)
assert.Contains(t, err.Error(), "index vec_idx for field 100 does not exist")
assert.Contains(t, err.Error(), "index for field 100 does not exist")
assert.Contains(t, err.Error(), "vec_idx")
}
func TestValidateRestoreSnapshotResources_Success(t *testing.T) {
+4 -1
View File
@@ -20,6 +20,9 @@ import (
"fmt"
"github.com/cockroachdb/errors"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/typeutil"
)
// errors for VerifyResponse
@@ -34,7 +37,7 @@ func msgDataCoordIsUnhealthy(coordID UniqueID) string {
}
func errDataCoordIsUnhealthy(coordID UniqueID) error {
return errors.New(msgDataCoordIsUnhealthy(coordID))
return merr.WrapErrServiceNotReady(typeutil.DataCoordRole, coordID, "not ready")
}
func msgSegmentNotFound(segID UniqueID) string {
@@ -306,7 +306,7 @@ func (m *externalCollectionRefreshManager) cleanupExploreTempForJob(jobID int64)
func (m *externalCollectionRefreshManager) applyFinishedJobSegments(ctx context.Context, job *datapb.ExternalCollectionRefreshJob) error {
tasks := m.refreshMeta.GetTasksByJobID(job.GetJobId())
if len(tasks) == 0 {
return fmt.Errorf("job %d has no tasks to apply", job.GetJobId())
return merr.WrapErrServiceInternalMsg("job %d has no tasks to apply", job.GetJobId())
}
keptSet := make(map[int64]struct{})
@@ -315,11 +315,11 @@ func (m *externalCollectionRefreshManager) applyFinishedJobSegments(ctx context.
updatedSegments := make([]*datapb.SegmentInfo, 0)
for _, task := range tasks {
if task.GetState() != indexpb.JobState_JobStateFinished {
return fmt.Errorf("job %d has non-finished task %d in state %s",
return merr.WrapErrServiceInternalMsg("job %d has non-finished task %d in state %s",
job.GetJobId(), task.GetTaskId(), task.GetState().String())
}
if !task.GetResultReady() {
return fmt.Errorf("job %d has finished task %d without persisted refresh result; please retry refresh",
return merr.WrapErrServiceInternalMsg("job %d has finished task %d without persisted refresh result; please retry refresh",
job.GetJobId(), task.GetTaskId())
}
for _, segmentID := range task.GetKeptSegments() {
@@ -334,7 +334,7 @@ func (m *externalCollectionRefreshManager) applyFinishedJobSegments(ctx context.
continue
}
if _, ok := updatedSet[segment.GetID()]; ok {
return fmt.Errorf("job %d has duplicate updated segment %d from task %d",
return merr.WrapErrServiceInternalMsg("job %d has duplicate updated segment %d from task %d",
job.GetJobId(), segment.GetID(), task.GetTaskId())
}
updatedSet[segment.GetID()] = struct{}{}
@@ -714,7 +714,7 @@ func (m *externalCollectionRefreshManager) createTasksForJob(
if errors.Is(err, packed.ErrLoonTransient) {
return nil, newNonRetriableJobError("explore external files failed: %v", err)
}
return nil, fmt.Errorf("failed to explore external files: %w", err)
return nil, merr.WrapErrServiceInternalErr(err, "failed to explore external files")
}
if len(allFiles) == 0 {
return nil, newNonRetriableJobError("no files found in external source: %s", job.GetExternalSource())
@@ -829,7 +829,7 @@ func normalizeRefreshJobProgress(job *datapb.ExternalCollectionRefreshJob, state
func (m *externalCollectionRefreshManager) GetJobProgress(ctx context.Context, jobID int64) (*datapb.ExternalCollectionRefreshJob, error) {
job := m.refreshMeta.GetJob(jobID)
if job == nil {
return nil, fmt.Errorf("job %d not found", jobID)
return nil, merr.WrapErrParameterInvalidMsg("refresh job %d not found", jobID)
}
// Aggregate state and progress from tasks
@@ -877,17 +877,17 @@ func (m *externalCollectionRefreshManager) exploreExternalFiles(
// when both present.
if job.GetExternalSource() != "" {
if err := externalspec.ValidateSourceAndSpec(job.GetExternalSource(), job.GetExternalSpec()); err != nil {
return nil, "", fmt.Errorf("external source/spec failed revalidation: %w", err)
return nil, "", merr.Wrap(err, "external source/spec failed revalidation")
}
}
spec, err := externalspec.ParseExternalSpec(job.GetExternalSpec())
if err != nil {
return nil, "", fmt.Errorf("failed to parse external spec: %w", err)
return nil, "", merr.Wrap(err, "failed to parse external spec")
}
collInfo := m.mt.GetCollection(job.GetCollectionId())
if collInfo == nil {
return nil, "", fmt.Errorf("collection %d not found", job.GetCollectionId())
return nil, "", merr.WrapErrCollectionNotFound(job.GetCollectionId())
}
columns := packed.GetColumnNamesFromSchema(collInfo.Schema)
@@ -908,7 +908,7 @@ func (m *externalCollectionRefreshManager) exploreExternalFiles(
extfs,
)
if err != nil {
return nil, "", fmt.Errorf("ExploreFilesReturnManifestPath failed: %w", err)
return nil, "", merr.WrapErrServiceInternalErr(err, "failed to explore files returning manifest path")
}
// Convert to proto type
@@ -18,7 +18,6 @@ package datacoord
import (
"context"
"fmt"
"sort"
"time"
@@ -30,6 +29,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/proto/datapb"
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
"github.com/milvus-io/milvus/pkg/v3/util/lock"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/timerecord"
"github.com/milvus-io/milvus/pkg/v3/util/typeutil"
)
@@ -295,7 +295,7 @@ func (m *externalCollectionRefreshMeta) mutateJob(
) (applied bool, err error) {
job, ok := m.jobs.Get(jobID)
if !ok {
return false, fmt.Errorf("job %d not found", jobID)
return false, merr.WrapErrServiceInternalMsg("job %d not found", jobID)
}
m.jobLock.Lock(job.GetCollectionId())
@@ -304,7 +304,7 @@ func (m *externalCollectionRefreshMeta) mutateJob(
// Re-fetch after lock
job, ok = m.jobs.Get(jobID)
if !ok {
return false, fmt.Errorf("job %d not found", jobID)
return false, merr.WrapErrServiceInternalMsg("job %d not found", jobID)
}
cloneJob := proto.Clone(job).(*datapb.ExternalCollectionRefreshJob)
@@ -385,7 +385,7 @@ func (m *externalCollectionRefreshMeta) UpdateJobStateWithPreApply(
) (bool, error) {
job, ok := m.jobs.Get(jobID)
if !ok {
return false, fmt.Errorf("job %d not found", jobID)
return false, merr.WrapErrServiceInternalMsg("job %d not found", jobID)
}
m.jobLock.Lock(job.GetCollectionId())
@@ -395,7 +395,7 @@ func (m *externalCollectionRefreshMeta) UpdateJobStateWithPreApply(
// terminal state owns the one-time side effects.
job, ok = m.jobs.Get(jobID)
if !ok {
return false, fmt.Errorf("job %d not found", jobID)
return false, merr.WrapErrServiceInternalMsg("job %d not found", jobID)
}
if job.GetState() == indexpb.JobState_JobStateFinished ||
job.GetState() == indexpb.JobState_JobStateFailed {
@@ -416,7 +416,7 @@ func (m *externalCollectionRefreshMeta) UpdateJobStateWithPreApply(
log.Warn("update job state after pre-apply failed",
zap.Int64("jobID", jobID),
zap.Error(saveErr))
return false, fmt.Errorf("pre-apply failed: %w; additionally failed to persist Failed job state: %v", err, saveErr)
return false, merr.Wrapf(err, "pre-apply failed; additionally failed to persist Failed job state: %v", saveErr)
}
m.jobs.Insert(jobID, cloneJob)
m.addToCollectionJobs(cloneJob)
@@ -613,7 +613,7 @@ func (m *externalCollectionRefreshMeta) mutateTask(
) (applied bool, cloned *datapb.ExternalCollectionRefreshTask, err error) {
task, ok := m.tasks.Get(taskID)
if !ok {
return false, nil, fmt.Errorf("task %d not found", taskID)
return false, nil, merr.WrapErrServiceInternalMsg("task %d not found", taskID)
}
m.taskLock.Lock(task.GetJobId())
@@ -622,7 +622,7 @@ func (m *externalCollectionRefreshMeta) mutateTask(
// Re-fetch after lock
task, ok = m.tasks.Get(taskID)
if !ok {
return false, nil, fmt.Errorf("task %d not found", taskID)
return false, nil, merr.WrapErrServiceInternalMsg("task %d not found", taskID)
}
cloneTask := proto.Clone(task).(*datapb.ExternalCollectionRefreshTask)
+2 -2
View File
@@ -1067,13 +1067,13 @@ func (gc *garbageCollector) isExpire(dropts Timestamp) bool {
// Returns segmentID or error if path doesn't match.
func parseV3SegmentID(rootPath, filePath string) (int64, error) {
if !strings.HasPrefix(filePath, rootPath) {
return 0, fmt.Errorf("path %q does not contain rootPath %q", filePath, rootPath)
return 0, merr.WrapErrServiceInternalMsg("path %q does not contain rootPath %q", filePath, rootPath)
}
p := strings.TrimPrefix(filePath[len(rootPath):], "/")
parts := strings.Split(p, "/")
// Minimum: insert_log/coll/part/seg/something
if len(parts) < 5 || parts[0] != common.SegmentInsertLogPath {
return 0, fmt.Errorf("not a V3 insert_log path: %s", filePath)
return 0, merr.WrapErrServiceInternalMsg("not a V3 insert_log path: %s", filePath)
}
return strconv.ParseInt(parts[3], 10, 64)
}
+2 -3
View File
@@ -18,7 +18,6 @@ package datacoord
import (
"context"
"fmt"
"path"
"strings"
"sync"
@@ -34,6 +33,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/log"
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
"github.com/milvus-io/milvus/pkg/v3/util/conc"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
"github.com/milvus-io/milvus/pkg/v3/util/typeutil"
)
@@ -271,8 +271,7 @@ func (lobCtx *lobGCContext) collectLOBFilesFromSegment(ctx context.Context, segm
lobFiles, err := lobCtx.cache.Get(ctx, manifestPath, lobCtx.storageConfig)
if err != nil {
return fmt.Errorf("failed to get LOB files from manifest for segment %d (path=%s): %w",
segment.GetID(), manifestPath, err)
return merr.WrapErrServiceInternalErr(err, "failed to get LOB files from manifest for segment %d (path=%s)", segment.GetID(), manifestPath)
}
for _, lobFile := range lobFiles {
+1 -2
View File
@@ -18,7 +18,6 @@ package datacoord
import (
"context"
"fmt"
"time"
"github.com/hashicorp/golang-lru/v2/expirable"
@@ -345,7 +344,7 @@ func (m *importMeta) HandleCommitVchannel(ctx context.Context, jobID int64, vcha
job := m.jobs[jobID]
if job == nil {
return merr.WrapErrImportFailed(fmt.Sprintf("job %d not found", jobID))
return merr.WrapErrImportSysFailedMsg("job %d not found", jobID)
}
// Idempotency: if vchannel already committed, skip.
for _, c := range job.GetCommittedVchannels() {
+10 -10
View File
@@ -104,7 +104,7 @@ func (s *ImportServicesSuite) TestImportV2_AllocatorNilReturnsError() {
s.NoError(err)
s.NotNil(resp)
s.True(errors.Is(merr.Error(resp.GetStatus()), merr.ErrImportFailed))
s.True(errors.Is(merr.Error(resp.GetStatus()), merr.ErrServiceUnavailable))
s.Contains(resp.GetStatus().GetReason(), "allocator not initialized")
}
@@ -114,7 +114,7 @@ func (s *ImportServicesSuite) TestImportV2_AllocatorFailsReturnsError() {
server.stateCode.Store(commonpb.StateCode_Healthy)
mockAllocator := allocator.NewMockAllocator(s.T())
mockAllocator.EXPECT().AllocN(mock.Anything).Return(int64(0), int64(0), errors.New("allocation failed"))
mockAllocator.EXPECT().AllocN(mock.Anything).Return(int64(0), int64(0), merr.WrapErrServiceUnavailable("allocation failed"))
server.allocator = mockAllocator
req := &internalpb.ImportRequestInternal{
@@ -127,7 +127,7 @@ func (s *ImportServicesSuite) TestImportV2_AllocatorFailsReturnsError() {
s.NoError(err)
s.NotNil(resp)
s.True(errors.Is(merr.Error(resp.GetStatus()), merr.ErrImportFailed))
s.True(errors.Is(merr.Error(resp.GetStatus()), merr.ErrServiceUnavailable))
s.Contains(resp.GetStatus().GetReason(), "failed to allocate job ID")
}
@@ -164,7 +164,7 @@ func (s *ImportServicesSuite) TestImportV2_BroadcastFailsReturnsError() {
// Mock StartBroadcastWithResourceKeys to fail
mockBroadcast := mockey.Mock(broadcast.StartBroadcastWithResourceKeys).To(
func(ctx context.Context, keys ...message.ResourceKey) (broadcaster.BroadcastAPI, error) {
return nil, errors.New("broadcast failed")
return nil, merr.WrapErrServiceUnavailable("broadcast failed")
}).Build()
defer mockBroadcast.UnPatch()
@@ -199,8 +199,8 @@ func (s *ImportServicesSuite) TestImportV2_BroadcastFailsReturnsError() {
s.NoError(err)
s.NotNil(resp)
s.True(errors.Is(merr.Error(resp.GetStatus()), merr.ErrImportFailed))
s.Contains(resp.GetStatus().GetReason(), "failed to broadcast import")
s.True(errors.Is(merr.Error(resp.GetStatus()), merr.ErrServiceUnavailable))
s.Contains(resp.GetStatus().GetReason(), "broadcast")
}
func (s *ImportServicesSuite) TestImportV2_SuccessReturnsJobID() {
@@ -395,7 +395,7 @@ func (s *ImportServicesSuite) TestCreateImportJobFromAck_AllocatorFailsReturnsEr
server.stateCode.Store(commonpb.StateCode_Healthy)
mockAllocator := allocator.NewMockAllocator(s.T())
mockAllocator.EXPECT().AllocN(mock.Anything).Return(int64(0), int64(0), errors.New("allocation failed"))
mockAllocator.EXPECT().AllocN(mock.Anything).Return(int64(0), int64(0), merr.WrapErrServiceUnavailable("allocation failed"))
server.allocator = mockAllocator
req := &internalpb.ImportRequestInternal{
@@ -412,7 +412,7 @@ func (s *ImportServicesSuite) TestCreateImportJobFromAck_AllocatorFailsReturnsEr
s.NoError(err)
s.NotNil(resp)
s.True(errors.Is(merr.Error(resp.GetStatus()), merr.ErrImportFailed))
s.True(errors.Is(merr.Error(resp.GetStatus()), merr.ErrServiceUnavailable))
s.Contains(resp.GetStatus().GetReason(), "alloc id failed")
}
@@ -493,7 +493,7 @@ func (s *ImportServicesSuite) TestCreateImportJobFromAck_AddJobFailsReturnsError
catalog.EXPECT().ListImportJobs(mock.Anything).Return(nil, nil)
catalog.EXPECT().ListPreImportTasks(mock.Anything).Return(nil, nil)
catalog.EXPECT().ListImportTasks(mock.Anything).Return(nil, nil)
catalog.EXPECT().SaveImportJob(mock.Anything, mock.Anything).Return(errors.New("save job failed"))
catalog.EXPECT().SaveImportJob(mock.Anything, mock.Anything).Return(merr.WrapErrServiceUnavailable("save job failed"))
importMeta, err := NewImportMeta(context.TODO(), catalog, nil, nil)
s.NoError(err)
@@ -528,7 +528,7 @@ func (s *ImportServicesSuite) TestCreateImportJobFromAck_AddJobFailsReturnsError
s.NoError(err)
s.NotNil(resp)
s.True(errors.Is(merr.Error(resp.GetStatus()), merr.ErrImportFailed))
s.True(errors.Is(merr.Error(resp.GetStatus()), merr.ErrServiceUnavailable))
s.Contains(resp.GetStatus().GetReason(), "add import job failed")
}
+7 -7
View File
@@ -670,8 +670,8 @@ func ListBinlogsAndGroupBySegment(ctx context.Context,
return nil, merr.WrapErrImportFailed("no insert binlogs to import")
}
if len(importFile.GetPaths()) > 2 {
return nil, merr.WrapErrImportFailed(fmt.Sprintf("too many input paths for binlog import. "+
"Valid paths length should be one or two, but got paths:%s", importFile.GetPaths()))
return nil, merr.WrapErrImportFailedMsg("too many input paths for binlog import. "+
"Valid paths length should be one or two, but got paths:%s", importFile.GetPaths())
}
insertPrefix := importFile.GetPaths()[0]
@@ -790,18 +790,18 @@ func ListBinlogImportRequestFiles(ctx context.Context, cm storage.ChunkManager,
}
err := conc.AwaitAll(futures...)
if err != nil {
return nil, merr.WrapErrImportFailed(fmt.Sprintf("list binlogs failed, err=%s", err))
return nil, merr.WrapErrServiceUnavailableMsg("list binlogs failed, err=%s", err)
}
resFiles = lo.Filter(resFiles, func(file *internalpb.ImportFile, _ int) bool {
return len(file.GetPaths()) > 0
})
if len(resFiles) == 0 {
return nil, merr.WrapErrImportFailed(fmt.Sprintf("no binlog to import, input=%s", reqFiles))
return nil, merr.WrapErrImportFailedMsg("no binlog to import, input=%s", reqFiles)
}
if len(resFiles) > paramtable.Get().DataCoordCfg.MaxFilesPerImportReq.GetAsInt() {
return nil, merr.WrapErrImportFailed(fmt.Sprintf("The max number of import files should not exceed %d, but got %d",
paramtable.Get().DataCoordCfg.MaxFilesPerImportReq.GetAsInt(), len(resFiles)))
return nil, merr.WrapErrImportFailedMsg("The max number of import files should not exceed %d, but got %d",
paramtable.Get().DataCoordCfg.MaxFilesPerImportReq.GetAsInt(), len(resFiles))
}
log.Info("list binlogs prefixes for import done", zap.Int("num", len(resFiles)), zap.Any("binlog_prefixes", resFiles))
return resFiles, nil
@@ -812,7 +812,7 @@ func ValidateMaxImportJobExceed(ctx context.Context, importMeta ImportMeta) erro
maxNum := paramtable.Get().DataCoordCfg.MaxImportJobNum.GetAsInt()
executingNum := importMeta.CountJobBy(ctx, WithoutJobStates(internalpb.ImportJobState_Completed, internalpb.ImportJobState_Failed))
if executingNum >= maxNum {
return merr.WrapErrImportFailed(
return merr.WrapErrImportSysFailed(
fmt.Sprintf("The number of jobs has reached the limit, please try again later. " +
"If your request is set to only import a single file, " +
"please consider importing multiple files in one request for better efficiency."))
+9 -5
View File
@@ -468,7 +468,7 @@ func (m *indexMeta) canCreateIndex(req *indexpb.CreateIndexRequest, isJSON bool)
index.IndexName, index.FieldID, index.IndexParams, index.UserIndexParams, index.TypeParams)),
zap.String("current index", fmt.Sprintf("{index_name: %s, field_id: %d, index_params: %v, user_params: %v, type_params: %v}",
req.GetIndexName(), req.GetFieldID(), req.GetIndexParams(), req.GetUserIndexParams(), req.GetTypeParams())))
return 0, fmt.Errorf("CreateIndex failed: %s", errMsg)
return 0, merr.WrapErrParameterInvalidMsg("%s", errMsg)
}
if req.FieldID == index.FieldID {
if isJSON {
@@ -484,7 +484,11 @@ func (m *indexMeta) canCreateIndex(req *indexpb.CreateIndexRequest, isJSON bool)
// creating multiple indexes on same field is not supported
errMsg := "CreateIndex failed: creating multiple indexes on same field is not supported"
log.Warn(errMsg)
return 0, errors.New(errMsg)
// Client-caused conflict, same family as the "one distinct index per
// field" case above: return an input-class error rather than
// ServiceInternal (code 5, "never return out of Milvus") so the
// caller is not blamed with a system error / counted as fail_system.
return 0, merr.WrapErrParameterInvalidMsg("%s", errMsg)
}
}
return 0, nil
@@ -1012,7 +1016,7 @@ func (m *indexMeta) UpdateVersion(buildID, nodeID UniqueID) error {
log.Ctx(m.ctx).Info("IndexCoord metaTable UpdateVersion receive", zap.Int64("buildID", buildID), zap.Int64("nodeID", nodeID))
segIdx, ok := m.segmentBuildInfo.Get(buildID)
if !ok {
return fmt.Errorf("there is no index with buildID: %d", buildID)
return merr.WrapErrServiceInternalMsg("there is no index with buildID: %d", buildID)
}
updateFunc := func(segIdx *model.SegmentIndex) error {
@@ -1031,7 +1035,7 @@ func (m *indexMeta) UpdateIndexState(buildID UniqueID, state commonpb.IndexState
segIdx, ok := m.segmentBuildInfo.Get(buildID)
if !ok {
log.Ctx(m.ctx).Warn("there is no index with buildID", zap.Int64("buildID", buildID))
return fmt.Errorf("there is no index with buildID: %d", buildID)
return merr.WrapErrServiceInternalMsg("there is no index with buildID: %d", buildID)
}
updateFunc := func(segIdx *model.SegmentIndex) error {
@@ -1141,7 +1145,7 @@ func (m *indexMeta) BuildIndex(buildID UniqueID) error {
segIdx, ok := m.segmentBuildInfo.Get(buildID)
if !ok {
return fmt.Errorf("there is no index with buildID: %d", buildID)
return merr.WrapErrServiceInternalMsg("there is no index with buildID: %d", buildID)
}
updateFunc := func(segIdx *model.SegmentIndex) error {
+1 -1
View File
@@ -1052,7 +1052,7 @@ func (s *Server) DropIndex(ctx context.Context, req *indexpb.DropIndexRequest) (
}
if typeutil.IsVectorType(field.GetDataType()) {
log.Warn("vector index cannot be dropped on loaded collection", zap.String("indexName", req.IndexName), zap.Int64("collectionID", req.GetCollectionID()), zap.Int64("fieldID", index.FieldID))
return merr.Status(merr.WrapErrParameterInvalidMsg(fmt.Sprintf("vector index cannot be dropped on loaded collection: %d", req.GetCollectionID()))), nil
return merr.Status(merr.WrapErrParameterInvalidMsg("vector index cannot be dropped on loaded collection: %d", req.GetCollectionID())), nil
}
}
}
+20 -12
View File
@@ -816,7 +816,7 @@ func (m *meta) GetSegmentsChannels(segmentIDs []UniqueID) (map[int64]string, err
for _, segmentID := range segmentIDs {
segment := m.segments.GetSegment(segmentID)
if segment == nil {
return nil, errors.New(fmt.Sprintf("cannot find segment %d", segmentID))
return nil, merr.WrapErrServiceInternalMsg("cannot find segment %d", segmentID)
}
segChannels[segmentID] = segment.GetInsertChannel()
}
@@ -840,7 +840,7 @@ func (m *meta) SetState(ctx context.Context, segmentID UniqueID, targetState com
if targetState == commonpb.SegmentState_Dropped {
return nil
}
return fmt.Errorf("segment is not exist with ID = %d", segmentID)
return merr.WrapErrSegmentNotFound(segmentID)
}
// Persist segment updates first.
clonedSegment := curSegInfo.Clone()
@@ -941,12 +941,12 @@ func (p *updateSegmentPack) Validate() error {
segmentInMeta := p.meta.segments.GetSegment(segment.ID)
if segmentInMeta.State == commonpb.SegmentState_Flushed && segment.State != commonpb.SegmentState_Dropped {
// if the segment is flushed, we should not update the segment meta, ignore the operation directly.
return errors.Wrapf(ErrIgnoredSegmentMetaOperation,
return merr.Wrapf(errIgnoredSegmentMetaOperation,
"segment is flushed, segmentID: %d",
segment.ID)
}
if segment.GetDmlPosition().GetTimestamp() < segmentInMeta.GetDmlPosition().GetTimestamp() {
return errors.Wrapf(ErrIgnoredSegmentMetaOperation,
return merr.Wrapf(errIgnoredSegmentMetaOperation,
"dml time tick is less than the segment meta, segmentID: %d, new incoming time tick: %d, existing time tick: %d",
segment.ID,
segment.GetDmlPosition().GetTimestamp(),
@@ -1223,7 +1223,7 @@ func updateManifestPathIfNewer(segment *SegmentInfo, manifestPath string) error
return err
}
if currentBase != incomingBase {
return merr.WrapErrServiceInternal(fmt.Sprintf("manifest base path mismatch for segment %d: current %s, incoming %s", segment.GetID(), currentBase, incomingBase))
return merr.WrapErrServiceInternalMsg("manifest base path mismatch for segment %d: current %s, incoming %s", segment.GetID(), currentBase, incomingBase)
}
if incomingVersion > currentVersion {
segment.ManifestPath = manifestPath
@@ -1824,6 +1824,14 @@ func (m *meta) UpdateSegmentsInfo(ctx context.Context, operators ...UpdateOperat
// Validate the update pack.
if err := updatePack.Validate(); err != nil {
// A stale save-binlog-paths update (segment already flushed, or an
// outdated time tick) is a benign no-op: skip the meta write and
// report success so the caller does not retry. The signal stays
// inside this package on purpose; see errIgnoredSegmentMetaOperation.
if errors.Is(err, errIgnoredSegmentMetaOperation) {
log.Ctx(ctx).Info("meta update: ignored stale segment meta operation", zap.Error(err))
return nil
}
return err
}
updatePack.prepareSegmentMetricUpdates()
@@ -2100,7 +2108,7 @@ func (m *meta) AddAllocation(segmentID UniqueID, allocation *Allocation) error {
if curSegInfo == nil {
// TODO: Error handling.
log.Ctx(m.ctx).Error("meta update: add allocation failed - segment not found", zap.Int64("segmentID", segmentID))
return errors.New("meta update: add allocation failed - segment not found")
return merr.WrapErrSegmentNotFound(segmentID, "meta update: add allocation failed")
}
// As we use global segment lastExpire to guarantee data correctness after restart
// there is no need to persist allocation to meta store, only update allocation in-memory meta.
@@ -2287,7 +2295,7 @@ func validateCompactionFallbackStartPosition(compactFromSegInfos []*SegmentInfo,
if maxCommitTs == 0 || fallbackStart.GetTimestamp() >= maxCommitTs {
return nil
}
return errors.Errorf(
return merr.WrapErrServiceInternalMsg(
"compaction fallback start position timestamp %d is earlier than max input commit timestamp %d",
fallbackStart.GetTimestamp(),
maxCommitTs)
@@ -2661,7 +2669,7 @@ func (m *meta) HasSegments(segIDs []UniqueID) (bool, error) {
for _, segID := range segIDs {
if _, ok := m.segments.segments[segID]; !ok {
return false, fmt.Errorf("segment is not exist with ID = %d", segID)
return false, merr.WrapErrServiceInternalMsg("segment is not exist with ID = %d", segID)
}
}
return true, nil
@@ -2747,7 +2755,7 @@ func (m *meta) collectionHasTextFields(collectionID int64) bool {
// UpdateChannelCheckpoint updates and saves channel checkpoint.
func (m *meta) UpdateChannelCheckpoint(ctx context.Context, vChannel string, pos *msgpb.MsgPosition) error {
if pos == nil || pos.GetMsgID() == nil {
return fmt.Errorf("channelCP is nil, vChannel=%s", vChannel)
return merr.WrapErrServiceInternalMsg("channelCP is nil, vChannel=%s", vChannel)
}
// Clamp checkpoint to not advance beyond min growing segment checkpoint.
@@ -3331,7 +3339,7 @@ func (m *meta) completeBumpSchemaVersionCompactionMutation(
if currentManifest != resultManifest {
manifestCompare, err := packed.CompareManifestPath(resultManifest, currentManifest)
if err != nil {
return nil, nil, merr.WrapErrIllegalCompactionPlan(fmt.Sprintf("schema bump compaction result manifest is not comparable with current manifest: %v", err))
return nil, nil, merr.WrapErrIllegalCompactionPlanMsg("schema bump compaction result manifest is not comparable with current manifest: %v", err)
}
if manifestCompare <= 0 {
return nil, nil, merr.WrapErrIllegalCompactionPlan("schema bump compaction result manifest is not newer than current manifest")
@@ -3392,7 +3400,7 @@ func (m *meta) completeBumpSchemaVersionReplacementMutation(
) ([]*SegmentInfo, *segMetricMutation, error) {
idRange := t.GetPreAllocatedSegmentIDs()
if idRange == nil || idRange.GetBegin()+1 != idRange.GetEnd() || resultSegment.GetSegmentID() != idRange.GetBegin() {
return nil, nil, merr.WrapErrIllegalCompactionPlan(fmt.Sprintf("schema bump replacement result segment ID %d does not match the pre-allocated segment ID", resultSegment.GetSegmentID()))
return nil, nil, merr.WrapErrIllegalCompactionPlanMsg("schema bump replacement result segment ID %d does not match the pre-allocated segment ID", resultSegment.GetSegmentID())
}
dropped := oldSegment.Clone()
@@ -3555,7 +3563,7 @@ func (m *meta) GetFileResources(ctx context.Context, resourceIDs ...int64) ([]*i
if resource, ok := m.resourceIDMap[id]; ok {
resources = append(resources, resource)
} else {
return nil, errors.Errorf("file resource %d not found", id)
return nil, merr.WrapErrServiceInternalMsg("file resource %d not found", id)
}
}
return resources, nil
+2 -2
View File
@@ -3498,7 +3498,7 @@ func TestUpdateSegmentsInfo(t *testing.T) {
UpdateStartPosition([]*datapb.SegmentStartPosition{{SegmentID: 1, StartPosition: &msgpb.MsgPosition{MsgID: []byte{1, 2, 3}}}}),
UpdateCheckPointOperator(1, []*datapb.CheckPoint{{SegmentID: 1, NumOfRows: 10, Position: &msgpb.MsgPosition{MsgID: []byte{1, 2, 3}, Timestamp: 99}}}, true),
)
assert.True(t, errors.Is(err, ErrIgnoredSegmentMetaOperation))
assert.NoError(t, err) // stale update is swallowed as a benign no-op; segment must stay unchanged below
err = meta.UpdateSegmentsInfo(
context.TODO(),
@@ -3536,7 +3536,7 @@ func TestUpdateSegmentsInfo(t *testing.T) {
[]*datapb.FieldBinlog{}),
UpdateCheckPointOperator(1, []*datapb.CheckPoint{{SegmentID: 1, NumOfRows: 12, Position: &msgpb.MsgPosition{MsgID: []byte{1, 2, 3}, Timestamp: 101}}}, true),
)
assert.True(t, errors.Is(err, ErrIgnoredSegmentMetaOperation))
assert.NoError(t, err) // stale update is swallowed as a benign no-op; segment must stay unchanged below
updated = meta.GetHealthySegment(context.TODO(), 1)
assert.Equal(t, updated.NumOfRows, int64(20))
+10 -1
View File
@@ -22,7 +22,16 @@ import (
"github.com/milvus-io/milvus/pkg/v3/proto/datapb"
)
var ErrIgnoredSegmentMetaOperation = errors.New("ignored segment meta operation")
// errIgnoredSegmentMetaOperation is an internal control-flow signal: the
// requested segment meta update is stale (segment already flushed, or an
// outdated time tick) and is skipped as a benign no-op. It never crosses the
// package boundary — UpdateSegmentsInfo swallows it and returns nil.
//
// It MUST stay a plain errors.New identity sentinel. A merr-typed value would
// make errors.Is match by error code (milvusError.Is compares codes only), so
// any error sharing the code — e.g. a real etcd write failure wrapped as
// ServiceInternal — would be misread as ignorable and acknowledged as success.
var errIgnoredSegmentMetaOperation = errors.New("ignored segment meta operation")
// reviseVChannelInfo will revise the datapb.VchannelInfo for upgrade compatibility from 2.0.2
func reviseVChannelInfo(vChannel *datapb.VchannelInfo) {
+2 -4
View File
@@ -18,10 +18,8 @@ package datacoord
import (
"context"
"fmt"
"sync"
"github.com/cockroachdb/errors"
"github.com/tidwall/gjson"
"go.uber.org/zap"
"golang.org/x/sync/errgroup"
@@ -138,7 +136,7 @@ func (s *Server) getSegmentsJSON(ctx context.Context, req *milvuspb.GetMetricsRe
}
return string(bs), nil
}
return "", fmt.Errorf("invalid param value in=[%s], it should be dc or dn", in)
return "", merr.WrapErrParameterInvalidMsg("invalid param value in=[%s], it should be dc or dn", in)
}
func (s *Server) getDistJSON(ctx context.Context, req *milvuspb.GetMetricsRequest) string {
@@ -308,7 +306,7 @@ func (s *Server) getIndexNodeMetrics(ctx context.Context, req *milvuspb.GetMetri
},
}
if node == nil {
return infos, errors.New("IndexNode is nil")
return infos, merr.WrapErrServiceInternalMsg("index node is nil")
}
metrics, err := node.GetMetrics(ctx, req)
+2 -3
View File
@@ -2,7 +2,6 @@ package datacoord
import (
"context"
"fmt"
"sync"
"go.uber.org/zap"
@@ -183,11 +182,11 @@ func (psm *partitionStatsMeta) innerSaveCurrentPartitionStatsVersion(collectionI
if _, ok := psm.partitionStatsInfos[vChannel]; !ok {
return merr.WrapErrClusteringCompactionMetaError("SaveCurrentPartitionStatsVersion",
fmt.Errorf("update current partition stats version failed, there is no partition info exists with collID: %d, partID: %d, vChannel: %s", collectionID, partitionID, vChannel))
merr.WrapErrServiceInternalMsg("update current partition stats version failed, there is no partition info exists with collID: %d, partID: %d, vChannel: %s", collectionID, partitionID, vChannel))
}
if _, ok := psm.partitionStatsInfos[vChannel][partitionID]; !ok {
return merr.WrapErrClusteringCompactionMetaError("SaveCurrentPartitionStatsVersion",
fmt.Errorf("update current partition stats version failed, there is no partition info exists with collID: %d, partID: %d, vChannel: %s", collectionID, partitionID, vChannel))
merr.WrapErrServiceInternalMsg("update current partition stats version failed, there is no partition info exists with collID: %d, partID: %d, vChannel: %s", collectionID, partitionID, vChannel))
}
if err := psm.catalog.SaveCurrentPartitionStatsVersion(psm.ctx, collectionID, partitionID, vChannel, currentPartitionStatsVersion); err != nil {
@@ -23,12 +23,12 @@ import (
"sort"
"time"
"github.com/cockroachdb/errors"
"github.com/samber/lo"
"github.com/milvus-io/milvus-proto/go-api/v3/commonpb"
"github.com/milvus-io/milvus-proto/go-api/v3/schemapb"
"github.com/milvus-io/milvus/pkg/v3/proto/datapb"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
"github.com/milvus-io/milvus/pkg/v3/util/tsoutil"
"github.com/milvus-io/milvus/pkg/v3/util/typeutil"
@@ -38,7 +38,7 @@ type calUpperLimitPolicy func(schema *schemapb.CollectionSchema) (int, error)
func calBySchemaPolicy(schema *schemapb.CollectionSchema) (int, error) {
if schema == nil {
return -1, errors.New("nil schema")
return -1, merr.WrapErrServiceInternalMsg("nil schema")
}
sizePerRecord, err := typeutil.EstimateSizePerRecord(schema)
if err != nil {
@@ -46,7 +46,7 @@ func calBySchemaPolicy(schema *schemapb.CollectionSchema) (int, error) {
}
// check zero value, preventing panicking
if sizePerRecord == 0 {
return -1, errors.New("zero size record schema found")
return -1, merr.WrapErrServiceInternalMsg("zero size record schema found")
}
threshold := Params.DataCoordCfg.SegmentMaxSize.GetAsFloat() * 1024 * 1024
return int(threshold / float64(sizePerRecord)), nil
+3 -3
View File
@@ -23,7 +23,6 @@ import (
"sync"
"time"
"github.com/cockroachdb/errors"
"github.com/samber/lo"
"go.opentelemetry.io/otel"
"go.uber.org/zap"
@@ -36,6 +35,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/log"
"github.com/milvus-io/milvus/pkg/v3/proto/datapb"
"github.com/milvus-io/milvus/pkg/v3/util/lock"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/metautil"
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
"github.com/milvus-io/milvus/pkg/v3/util/retry"
@@ -299,7 +299,7 @@ func (s *SegmentManager) genLastExpireTsForSegments() (Timestamp, error) {
}, retry.Attempts(Params.DataCoordCfg.AllocLatestExpireAttempt.GetAsUint()), retry.Sleep(200*time.Millisecond))
if allocateErr != nil {
log.Warn("cannot allocate latest lastExpire from rootCoord", zap.Error(allocateErr))
return 0, errors.New("global max expire ts is unavailable for segment manager")
return 0, merr.WrapErrServiceInternalMsg("global max expire ts is unavailable for segment manager")
}
return latestTs, nil
}
@@ -466,7 +466,7 @@ func (s *SegmentManager) estimateMaxNumOfRows(collectionID UniqueID) (int, error
// it's ok to use meta.GetCollection here, since collection meta is set before using segmentManager
collMeta := s.meta.GetCollection(collectionID)
if collMeta == nil {
return -1, fmt.Errorf("failed to get collection %d", collectionID)
return -1, merr.WrapErrServiceInternalMsg("failed to get collection %d", collectionID)
}
return s.estimatePolicy(collMeta.Schema)
}
+4 -5
View File
@@ -26,7 +26,6 @@ import (
"time"
"github.com/blang/semver/v4"
"github.com/cockroachdb/errors"
"github.com/samber/lo"
"github.com/tidwall/gjson"
"github.com/tikv/client-go/v2/txnkv"
@@ -462,7 +461,7 @@ func (s *Server) SetSession(session sessionutil.SessionInterface) error {
s.session = session
s.icSession = session
if s.session == nil {
return errors.New("session is nil, the etcd client connection may have failed")
return merr.WrapErrServiceNotReadyMsg("session is nil, the etcd client connection may have failed")
}
return nil
}
@@ -618,7 +617,7 @@ func (s *Server) initKV() error {
s.kv = etcdkv.NewEtcdKV(s.etcdCli, s.metaRootPath,
etcdkv.WithRequestTimeout(paramtable.Get().EtcdCfg.RequestTimeout.GetAsDuration(time.Millisecond)))
default:
return retry.Unrecoverable(fmt.Errorf("not supported meta store: %s", metaType))
return retry.Unrecoverable(merr.WrapErrServiceInternalMsg("unsupported meta store: %s", metaType))
}
log.Info("data coordinator successfully connected to metadata store", zap.String("metaType", metaType))
return nil
@@ -1027,7 +1026,7 @@ func (s *Server) flushFlushingSegment(ctx context.Context, segmentID UniqueID) e
return ctx.Err()
}
// underlying etcd may return context canceled, so we need to return a error to retry.
return errors.New("flush segment complete failed")
return merr.WrapErrServiceInternalMsg("flush segment complete failed")
}
return nil
}, retry.AttemptAlways())
@@ -1117,7 +1116,7 @@ func (s *Server) CleanMeta() error {
err2 := s.watchClient.RemoveWithPrefix(s.ctx, "")
if err2 != nil {
if err != nil {
err = fmt.Errorf("failed to CleanMeta[metadata cleanup error: %w][watchdata cleanup error: %v]", err, err2)
err = merr.Wrapf(err, "failed to clean meta (watchdata cleanup error: %v)", err2)
} else {
err = err2
}
+21 -22
View File
@@ -137,7 +137,7 @@ func (s *Server) flushCollection(ctx context.Context, collectionID UniqueID, flu
for _, channel := range coll.VChannelNames {
sealedSegmentIDs, err := s.segmentManager.SealAllSegments(ctx, channel, toFlushSegments)
if err != nil {
return nil, errors.Wrapf(err, "failed to flush collection %d", collectionID)
return nil, merr.Wrapf(err, "failed to flush collection %d", collectionID)
}
for _, sealedSegmentID := range sealedSegmentIDs {
sealedSegmentsIDDict[sealedSegmentID] = true
@@ -691,13 +691,12 @@ func (s *Server) SaveBinlogPaths(ctx context.Context, req *datapb.SaveBinlogPath
UpdateAsDroppedIfEmptyWhenFlushing(req.GetSegmentID()),
)
// Update segment info in memory and meta.
// Update segment info in memory and meta. Stale updates (segment already
// flushed / outdated time tick) are swallowed inside UpdateSegmentsInfo as
// benign no-ops, so any error here is a real failure.
if err := s.meta.UpdateSegmentsInfo(ctx, operators...); err != nil {
if !errors.Is(err, ErrIgnoredSegmentMetaOperation) {
log.Error("save binlog and checkpoints failed", zap.Error(err))
return merr.Status(err), nil
}
log.Info("save binlog and checkpoints failed with ignorable error", zap.Error(err))
log.Error("save binlog and checkpoints failed", zap.Error(err))
return merr.Status(err), nil
}
s.meta.SetLastWrittenTime(req.GetSegmentID())
@@ -1292,7 +1291,7 @@ func (s *Server) GetMetrics(ctx context.Context, req *milvuspb.GetMetricsRequest
msg := "failed to get metrics"
log.Warn(msg, zap.Error(err))
return &milvuspb.GetMetricsResponse{
Status: merr.Status(errors.Wrap(err, msg)),
Status: merr.Status(merr.Wrap(err, msg)),
}, nil
}
@@ -1627,7 +1626,7 @@ OUTER:
break OUTER
}
} else {
resp.Status = merr.Status(merr.WrapErrParameterInvalidMsg("FlushAllTss or FlushAllTs is required"))
resp.Status = merr.Status(merr.WrapErrParameterMissingMsg("FlushAllTss or FlushAllTs is required"))
return resp, nil
}
}
@@ -1925,12 +1924,12 @@ func (s *Server) ImportV2(ctx context.Context, in *internalpb.ImportRequestInter
jobID := in.GetJobID()
if jobID == 0 {
if s.allocator == nil {
resp.Status = merr.Status(merr.WrapErrImportFailed("allocator not initialized"))
resp.Status = merr.Status(merr.WrapErrServiceUnavailable("allocator not initialized"))
return resp, nil
}
jobID, _, err = s.allocator.AllocN(1)
if err != nil {
resp.Status = merr.Status(merr.WrapErrImportFailed(fmt.Sprintf("failed to allocate job ID: %v", err)))
resp.Status = merr.Status(merr.Wrap(err, "failed to allocate job ID"))
return resp, nil
}
}
@@ -1950,7 +1949,7 @@ func (s *Server) ImportV2(ctx context.Context, in *internalpb.ImportRequestInter
)
if err != nil {
log.Warn("failed to broadcast import message", zap.Error(err))
resp.Status = merr.Status(merr.WrapErrImportFailed(fmt.Sprintf("failed to broadcast import: %v", err)))
resp.Status = merr.Status(merr.Wrap(err, "failed to broadcast import"))
return resp, nil
}
@@ -2000,7 +1999,7 @@ func (s *Server) createImportJobFromAck(ctx context.Context, in *internalpb.Impo
// Allocate file ids.
idStart, _, err := s.allocator.AllocN(int64(len(files)) + 1)
if err != nil {
resp.Status = merr.Status(merr.WrapErrImportFailed(fmt.Sprintf("alloc id failed: %v", err)))
resp.Status = merr.Status(merr.Wrap(err, "alloc id failed"))
return resp, nil
}
files = lo.Map(files, func(importFile *internalpb.ImportFile, i int) *internalpb.ImportFile {
@@ -2013,7 +2012,7 @@ func (s *Server) createImportJobFromAck(ctx context.Context, in *internalpb.Impo
return resp, nil
}
if err != nil {
resp.Status = merr.Status(merr.WrapErrImportFailed(fmt.Sprintf("get collection failed: %v", err)))
resp.Status = merr.Status(merr.Wrap(err, "get collection failed"))
return resp, nil
}
if importCollectionInfo == nil {
@@ -2048,7 +2047,7 @@ func (s *Server) createImportJobFromAck(ctx context.Context, in *internalpb.Impo
}
err = s.importMeta.AddJob(ctx, job)
if err != nil {
resp.Status = merr.Status(merr.WrapErrImportFailed(fmt.Sprintf("add import job failed: %v", err)))
resp.Status = merr.Status(merr.Wrap(err, "add import job failed"))
return resp, nil
}
@@ -2075,13 +2074,13 @@ func (s *Server) GetImportProgress(ctx context.Context, in *internalpb.GetImport
}
jobID, err := strconv.ParseInt(in.GetJobID(), 10, 64)
if err != nil {
resp.Status = merr.Status(merr.WrapErrImportFailed(fmt.Sprintf("parse job id failed: %v", err)))
resp.Status = merr.Status(merr.WrapErrParameterInvalidMsg("parse job id failed: %v", err))
return resp, nil
}
job := s.importMeta.GetJob(ctx, jobID)
if job == nil {
resp.Status = merr.Status(merr.WrapErrImportFailed(fmt.Sprintf("import job does not exist, jobID=%d", jobID)))
resp.Status = merr.Status(merr.WrapErrImportSysFailedMsg("import job does not exist, jobID=%d", jobID))
return resp, nil
}
progress, state, importedRows, totalRows, reason := GetJobProgress(ctx, jobID, s.importMeta, s.meta)
@@ -2481,14 +2480,14 @@ func (s *Server) RestoreSnapshot(ctx context.Context, req *datapb.RestoreSnapsho
// Validate parameters
if req.GetName() == "" {
err := merr.WrapErrParameterInvalidMsg("snapshot name is required")
err := merr.WrapErrParameterMissingMsg("snapshot name is required")
log.Warn("invalid request", zap.Error(err))
return &datapb.RestoreSnapshotResponse{
Status: merr.Status(err),
}, nil
}
if req.GetTargetCollectionName() == "" {
err := merr.WrapErrParameterInvalidMsg("target collection name is required")
err := merr.WrapErrParameterMissingMsg("target collection name is required")
log.Warn("invalid request", zap.Error(err))
return &datapb.RestoreSnapshotResponse{
Status: merr.Status(err),
@@ -2887,7 +2886,7 @@ func (s *Server) ListRefreshExternalCollectionJobs(ctx context.Context, req *dat
func (s *Server) broadcastCommitImportMessage(ctx context.Context, job ImportJob) error {
vchannels := job.GetVchannels()
if len(vchannels) == 0 {
return merr.WrapErrImportFailed(fmt.Sprintf("job %d has no vchannels", job.GetJobID()))
return merr.WrapErrImportSysFailedMsg("job %d has no vchannels", job.GetJobID())
}
broadcaster, err := s.startBroadcastWithCollectionID(ctx, job.GetCollectionID())
@@ -2914,7 +2913,7 @@ func (s *Server) broadcastCommitImportMessage(ctx context.Context, job ImportJob
func (s *Server) broadcastRollbackImportMessage(ctx context.Context, job ImportJob) error {
vchannels := job.GetVchannels()
if len(vchannels) == 0 {
return merr.WrapErrImportFailed(fmt.Sprintf("job %d has no vchannels", job.GetJobID()))
return merr.WrapErrImportSysFailedMsg("job %d has no vchannels", job.GetJobID())
}
broadcaster, err := s.startBroadcastWithCollectionID(ctx, job.GetCollectionID())
@@ -2949,7 +2948,7 @@ func (s *Server) validateAndExecuteImportAction(
}
job := s.importMeta.GetJob(ctx, jobID)
if job == nil {
return merr.Status(merr.WrapErrImportFailed(fmt.Sprintf("job %d not found", jobID))), nil
return merr.Status(merr.WrapErrImportSysFailedMsg("job %d not found", jobID)), nil
}
if st := validateState(job); st != nil {
return st, nil
@@ -186,7 +186,7 @@ const maxBackfillResultBytes int64 = 64 * 1024 * 1024 // 64MiB
// reject s3a://<other-bucket>/... paths early.
func (s *Server) loadBackfillResult(ctx context.Context, rawPath string) (*BackfillResult, error) {
if rawPath == "" {
return nil, merr.WrapErrParameterInvalidMsg("result_path is required")
return nil, merr.WrapErrParameterMissingMsg("result_path is required")
}
bucket := bucketFromChunkManager(s.meta.chunkManager)
key, err := normalizeObjectKey(rawPath, bucket)
+7 -7
View File
@@ -1645,7 +1645,6 @@ func TestGetRecoveryInfoV2(t *testing.T) {
func TestImportV2(t *testing.T) {
ctx := context.Background()
mockErr := errors.New("mock err")
t.Run("ImportV2", func(t *testing.T) {
// server not healthy
@@ -1682,11 +1681,11 @@ func TestImportV2(t *testing.T) {
s.importMeta, err = NewImportMeta(context.TODO(), catalog, nil, nil)
assert.NoError(t, err)
alloc := allocator.NewMockAllocator(t)
alloc.EXPECT().AllocN(mock.Anything).Return(0, 0, mockErr)
alloc.EXPECT().AllocN(mock.Anything).Return(0, 0, merr.WrapErrServiceUnavailable("mock err"))
s.allocator = alloc
resp, err = s.ImportV2(ctx, &internalpb.ImportRequestInternal{})
assert.NoError(t, err)
assert.True(t, errors.Is(merr.Error(resp.GetStatus()), merr.ErrImportFailed))
assert.True(t, errors.Is(merr.Error(resp.GetStatus()), merr.ErrServiceUnavailable))
})
t.Run("GetImportProgress", func(t *testing.T) {
@@ -1703,7 +1702,7 @@ func TestImportV2(t *testing.T) {
JobID: "@%$%$#%",
})
assert.NoError(t, err)
assert.True(t, errors.Is(merr.Error(resp.GetStatus()), merr.ErrImportFailed))
assert.True(t, errors.Is(merr.Error(resp.GetStatus()), merr.ErrParameterInvalid))
// job does not exist
catalog := mocks.NewDataCoordCatalog(t)
@@ -1723,7 +1722,8 @@ func TestImportV2(t *testing.T) {
JobID: "-1",
})
assert.NoError(t, err)
assert.True(t, errors.Is(merr.Error(resp.GetStatus()), merr.ErrImportFailed))
// job-not-found is a server-side orchestration issue, not malformed user data
assert.True(t, errors.Is(merr.Error(resp.GetStatus()), merr.ErrImportSysFailed))
// normal case
var job ImportJob = &importJob{
@@ -3156,7 +3156,7 @@ func TestServer_RestoreSnapshot(t *testing.T) {
assert.NoError(t, err)
assert.Error(t, merr.Error(resp.GetStatus()))
assert.True(t, errors.Is(merr.Error(resp.GetStatus()), merr.ErrParameterInvalid))
assert.True(t, errors.Is(merr.Error(resp.GetStatus()), merr.ErrParameterMissing))
})
t.Run("missing_collection_name", func(t *testing.T) {
@@ -3173,7 +3173,7 @@ func TestServer_RestoreSnapshot(t *testing.T) {
assert.NoError(t, err)
assert.Error(t, merr.Error(resp.GetStatus()))
assert.True(t, errors.Is(merr.Error(resp.GetStatus()), merr.ErrParameterInvalid))
assert.True(t, errors.Is(merr.Error(resp.GetStatus()), merr.ErrParameterMissing))
})
t.Run("snapshot_not_found", func(t *testing.T) {
+2 -2
View File
@@ -18,12 +18,12 @@ package session
import (
"context"
"fmt"
"github.com/cockroachdb/errors"
"github.com/milvus-io/milvus/internal/types"
"github.com/milvus-io/milvus/pkg/v3/util/lock"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
)
var errDisposed = errors.New("client is disposed")
@@ -84,7 +84,7 @@ func (n *Session) GetOrCreateClient(ctx context.Context) (types.DataNodeClient,
}
if n.clientCreator == nil {
return nil, fmt.Errorf("unable to create client for %s because of a nil client creator", n.info.Address)
return nil, merr.WrapErrServiceInternalMsg("unable to create client for %s because of a nil client creator", n.info.Address)
}
err := n.initClient(ctx)
+30 -29
View File
@@ -33,6 +33,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/log"
"github.com/milvus-io/milvus/pkg/v3/proto/datapb"
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
)
// S3 snapshot storage path constants.
@@ -136,7 +137,7 @@ func getManifestSchemaByVersion(version int) (avro.Schema, error) {
case 3:
return getManifestSchemaV3()
default:
return nil, fmt.Errorf("unsupported manifest schema version: %d", version)
return nil, merr.WrapErrServiceInternalMsg("unsupported manifest schema version: %d", version)
}
}
@@ -508,22 +509,22 @@ func GetSegmentManifestPath(manifestDir string, segmentID int64) string {
func (w *SnapshotWriter) Save(ctx context.Context, snapshot *SnapshotData) (string, error) {
// Validate input parameters
if snapshot == nil {
return "", fmt.Errorf("snapshot cannot be nil")
return "", merr.WrapErrServiceInternalMsg("snapshot cannot be nil")
}
if snapshot.SnapshotInfo == nil {
return "", fmt.Errorf("snapshot info cannot be nil")
return "", merr.WrapErrServiceInternalMsg("snapshot info cannot be nil")
}
collectionID := snapshot.SnapshotInfo.GetCollectionId()
if collectionID <= 0 {
return "", fmt.Errorf("invalid collection ID: %d", collectionID)
return "", merr.WrapErrServiceInternalMsg("invalid collection ID: %d", collectionID)
}
if snapshot.Collection == nil {
return "", fmt.Errorf("collection description cannot be nil")
return "", merr.WrapErrServiceInternalMsg("collection description cannot be nil")
}
snapshotID := snapshot.SnapshotInfo.GetId()
if snapshotID <= 0 {
return "", fmt.Errorf("invalid snapshot ID: %d", snapshotID)
return "", merr.WrapErrServiceInternalMsg("invalid snapshot ID: %d", snapshotID)
}
manifestDir, metadataPath := GetSnapshotPaths(w.chunkManager.RootPath(), collectionID, snapshotID)
@@ -533,7 +534,7 @@ func (w *SnapshotWriter) Save(ctx context.Context, snapshot *SnapshotData) (stri
for _, segment := range snapshot.Segments {
manifestPath := GetSegmentManifestPath(manifestDir, segment.GetSegmentId())
if err := w.writeSegmentManifest(ctx, manifestPath, segment); err != nil {
return "", fmt.Errorf("failed to write manifest for segment %d: %w", segment.GetSegmentId(), err)
return "", merr.Wrapf(err, "failed to write manifest for segment %d", segment.GetSegmentId())
}
manifestPaths = append(manifestPaths, manifestPath)
}
@@ -557,7 +558,7 @@ func (w *SnapshotWriter) Save(ctx context.Context, snapshot *SnapshotData) (stri
// Step 3: Write metadata file with all manifest paths
// This file is the entry point for reading the snapshot
if err := w.writeMetadataFile(ctx, metadataPath, snapshot, manifestPaths, storagev2Manifests); err != nil {
return "", fmt.Errorf("failed to write metadata file: %w", err)
return "", merr.Wrap(err, "failed to write metadata file")
}
log.Info("Successfully wrote metadata file",
@@ -584,12 +585,12 @@ func (w *SnapshotWriter) writeSegmentManifest(ctx context.Context, manifestPath
// Serialize to Avro binary format (single record per file)
avroSchema, err := getManifestSchema()
if err != nil {
return fmt.Errorf("failed to get manifest schema: %w", err)
return merr.WrapErrServiceInternalErr(err, "failed to get manifest schema")
}
binaryData, err := avro.Marshal(avroSchema, entry)
if err != nil {
return fmt.Errorf("failed to serialize entry to avro: %w", err)
return merr.WrapErrServiceInternalErr(err, "failed to serialize entry to avro")
}
// Write to object storage
@@ -634,7 +635,7 @@ func (w *SnapshotWriter) writeMetadataFile(ctx context.Context, metadataPath str
}
jsonData, err := opts.Marshal(metadata)
if err != nil {
return fmt.Errorf("failed to marshal metadata to JSON: %w", err)
return merr.WrapErrServiceInternalErr(err, "failed to marshal metadata to JSON")
}
// Write to object storage
@@ -663,13 +664,13 @@ func (w *SnapshotWriter) writeMetadataFile(ctx context.Context, metadataPath str
func (w *SnapshotWriter) Drop(ctx context.Context, metadataFilePath string) error {
// Validate input parameter
if metadataFilePath == "" {
return fmt.Errorf("metadata file path cannot be empty")
return merr.WrapErrServiceInternalMsg("metadata file path cannot be empty")
}
// Step 1: Read metadata file to discover manifest file paths
metadata, err := w.readMetadataFile(ctx, metadataFilePath)
if err != nil {
return fmt.Errorf("failed to read metadata file: %w", err)
return merr.Wrap(err, "failed to read metadata file")
}
snapshotID := metadata.GetSnapshotInfo().GetId()
@@ -678,7 +679,7 @@ func (w *SnapshotWriter) Drop(ctx context.Context, metadataFilePath string) erro
manifestList := metadata.GetManifestList()
if len(manifestList) > 0 {
if err := w.chunkManager.MultiRemove(ctx, manifestList); err != nil {
return fmt.Errorf("failed to remove manifest files: %w", err)
return merr.Wrap(err, "failed to remove manifest files")
}
log.Info("Successfully removed manifest files",
zap.Int("count", len(manifestList)),
@@ -687,7 +688,7 @@ func (w *SnapshotWriter) Drop(ctx context.Context, metadataFilePath string) erro
// Step 3: Remove the metadata file (entry point)
if err := w.chunkManager.Remove(ctx, metadataFilePath); err != nil {
return fmt.Errorf("failed to remove metadata file: %w", err)
return merr.Wrap(err, "failed to remove metadata file")
}
log.Info("Successfully removed metadata file",
zap.String("metadataFilePath", metadataFilePath))
@@ -703,7 +704,7 @@ func (w *SnapshotWriter) readMetadataFile(ctx context.Context, filePath string)
// Read raw JSON content from object storage
data, err := w.chunkManager.Read(ctx, filePath)
if err != nil {
return nil, fmt.Errorf("failed to read metadata file: %w", err)
return nil, merr.Wrap(err, "failed to read metadata file")
}
// Use protojson for deserialization to correctly handle protobuf oneof fields
@@ -712,7 +713,7 @@ func (w *SnapshotWriter) readMetadataFile(ctx context.Context, filePath string)
DiscardUnknown: true,
}
if err := opts.Unmarshal(data, metadata); err != nil {
return nil, fmt.Errorf("failed to parse metadata JSON: %w", err)
return nil, merr.WrapErrServiceInternalErr(err, "failed to parse metadata JSON")
}
return metadata, nil
@@ -775,13 +776,13 @@ func NewSnapshotReader(cm storage.ChunkManager) *SnapshotReader {
func (r *SnapshotReader) ReadSnapshot(ctx context.Context, metadataFilePath string, includeSegments bool) (*SnapshotData, error) {
// Validate input
if metadataFilePath == "" {
return nil, fmt.Errorf("metadata file path cannot be empty")
return nil, merr.WrapErrServiceInternalMsg("metadata file path cannot be empty")
}
// Step 1: Read metadata file (always required)
metadata, err := r.readMetadataFile(ctx, metadataFilePath)
if err != nil {
return nil, fmt.Errorf("failed to read metadata file: %w", err)
return nil, merr.Wrap(err, "failed to read metadata file")
}
// Step 2: Optionally read segment details from manifest files
@@ -791,7 +792,7 @@ func (r *SnapshotReader) ReadSnapshot(ctx context.Context, metadataFilePath stri
for _, manifestPath := range metadata.GetManifestList() {
segments, err := r.readManifestFile(ctx, manifestPath, int(metadata.GetFormatVersion()))
if err != nil {
return nil, fmt.Errorf("failed to read manifest file %s: %w", manifestPath, err)
return nil, merr.Wrapf(err, "failed to read manifest file %s", manifestPath)
}
allSegments = append(allSegments, segments...)
}
@@ -830,7 +831,7 @@ func (r *SnapshotReader) readMetadataFile(ctx context.Context, filePath string)
// Read raw JSON content from object storage
data, err := r.chunkManager.Read(ctx, filePath)
if err != nil {
return nil, fmt.Errorf("failed to read metadata file: %w", err)
return nil, merr.Wrap(err, "failed to read metadata file")
}
// Use protojson for deserialization to correctly handle protobuf oneof fields
@@ -839,12 +840,12 @@ func (r *SnapshotReader) readMetadataFile(ctx context.Context, filePath string)
DiscardUnknown: true,
}
if err := opts.Unmarshal(data, metadata); err != nil {
return nil, fmt.Errorf("failed to parse metadata JSON: %w", err)
return nil, merr.WrapErrServiceInternalErr(err, "failed to parse metadata JSON")
}
// Validate format version compatibility
if err := validateFormatVersion(int(metadata.GetFormatVersion())); err != nil {
return nil, fmt.Errorf("incompatible snapshot format: %w", err)
return nil, merr.WrapErrServiceInternalErr(err, "incompatible snapshot format")
}
return metadata, nil
@@ -865,7 +866,7 @@ func validateFormatVersion(version int) error {
// Check if version is too new for current code
if version > SnapshotFormatVersion {
return fmt.Errorf("snapshot format version %d is too new, current supported version: %d (please upgrade Milvus)",
return merr.WrapErrServiceInternalMsg("snapshot format version %d is too new, current supported version: %d (please upgrade Milvus)",
version, SnapshotFormatVersion)
}
@@ -888,20 +889,20 @@ func (r *SnapshotReader) readManifestFile(ctx context.Context, filePath string,
// Read raw Avro content from object storage
data, err := r.chunkManager.Read(ctx, filePath)
if err != nil {
return nil, fmt.Errorf("failed to read manifest file: %w", err)
return nil, merr.Wrap(err, "failed to read manifest file")
}
// Get Avro schema for the specified format version
avroSchema, err := getManifestSchemaByVersion(formatVersion)
if err != nil {
return nil, fmt.Errorf("failed to get manifest schema for version %d: %w", formatVersion, err)
return nil, merr.WrapErrServiceInternalErr(err, "failed to get manifest schema for version %d", formatVersion)
}
// Deserialize Avro binary data to single ManifestEntry record
var record ManifestEntry
err = avro.Unmarshal(avroSchema, data, &record)
if err != nil {
return nil, fmt.Errorf("failed to parse avro data: %w", err)
return nil, merr.WrapErrServiceInternalErr(err, "failed to parse avro data")
}
// Convert ManifestEntry record to protobuf SegmentDescription
@@ -974,7 +975,7 @@ func (r *SnapshotReader) readManifestFile(ctx context.Context, filePath string,
func (r *SnapshotReader) ListSnapshots(ctx context.Context, collectionID int64) ([]*datapb.SnapshotInfo, error) {
// Validate input parameters
if collectionID <= 0 {
return nil, fmt.Errorf("invalid collection ID: %d", collectionID)
return nil, merr.WrapErrServiceInternalMsg("invalid collection ID: %d", collectionID)
}
// Construct metadata directory path
@@ -984,7 +985,7 @@ func (r *SnapshotReader) ListSnapshots(ctx context.Context, collectionID int64)
// List all files in the metadata directory
files, _, err := storage.ListAllChunkWithPrefix(ctx, r.chunkManager, metadataDir, false)
if err != nil {
return nil, fmt.Errorf("failed to list metadata files: %w", err)
return nil, merr.Wrap(storage.ToMilvusIoError(metadataDir, err), "failed to list metadata files")
}
// Read each metadata file and extract SnapshotInfo
+22 -22
View File
@@ -562,7 +562,7 @@ func (sm *snapshotManager) validateCMEKCompatibility(
// Get target database properties first (needed for both encrypted and non-encrypted snapshots)
dbResp, err := sm.broker.DescribeDatabase(ctx, targetDbName)
if err != nil {
return fmt.Errorf("failed to describe target database %s: %w", targetDbName, err)
return merr.Wrapf(err, "failed to describe target database %s", targetDbName)
}
targetIsEncrypted := hookutil.IsDBEncrypted(dbResp.GetProperties())
@@ -640,7 +640,7 @@ func (sm *snapshotManager) RestoreSnapshot(
// ========================================================================
phase0Lock, err := startRestoreLock(ctx, sourceCollectionID, snapshotName, targetDbName, targetCollectionName)
if err != nil {
return 0, fmt.Errorf("failed to acquire restore lock: %w", err)
return 0, merr.Wrap(err, "failed to acquire restore lock")
}
// Pin the source snapshot while holding the phase-0 lock. The pin is the
@@ -662,7 +662,7 @@ func (sm *snapshotManager) RestoreSnapshot(
pinID, activePins, err := sm.snapshotMeta.PinSnapshot(ctx, sourceCollectionID, snapshotName, pinTTLSeconds)
if err != nil {
phase0Lock.Close()
return 0, fmt.Errorf("failed to pin source snapshot under restore lock: %w", err)
return 0, merr.Wrap(err, "failed to pin source snapshot under restore lock")
}
setSnapshotActivePinsGauge(sourceCollectionID, snapshotName, activePins)
phase0Lock.Close()
@@ -691,7 +691,7 @@ func (sm *snapshotManager) RestoreSnapshot(
// Phase 1: Read snapshot data (now protected by the refcount guard)
snapshotData, err := sm.ReadSnapshotData(ctx, sourceCollectionID, snapshotName)
if err != nil {
return 0, fmt.Errorf("failed to read snapshot data: %w", err)
return 0, merr.Wrap(err, "failed to read snapshot data")
}
log.Info("snapshot data loaded",
zap.Int("segmentCount", len(snapshotData.Segments)),
@@ -707,7 +707,7 @@ func (sm *snapshotManager) RestoreSnapshot(
// Phase 2: Restore collection and partitions
collectionID, err := sm.RestoreCollection(ctx, snapshotData, targetCollectionName, targetDbName)
if err != nil {
return 0, fmt.Errorf("failed to restore collection: %w", err)
return 0, merr.Wrap(err, "failed to restore collection")
}
log.Info("collection and partitions restored", zap.Int64("collectionID", collectionID))
@@ -718,7 +718,7 @@ func (sm *snapshotManager) RestoreSnapshot(
if rollbackErr := rollback(ctx, targetDbName, targetCollectionName); rollbackErr != nil {
log.Error("rollback failed", zap.Error(rollbackErr))
}
return 0, fmt.Errorf("failed to restore indexes: %w", err)
return 0, merr.Wrap(err, "failed to restore indexes")
}
log.Info("indexes restored", zap.Int("indexCount", len(snapshotData.Indexes)))
@@ -730,7 +730,7 @@ func (sm *snapshotManager) RestoreSnapshot(
if rollbackErr := rollback(ctx, targetDbName, targetCollectionName); rollbackErr != nil {
log.Error("rollback failed", zap.Error(rollbackErr))
}
return 0, fmt.Errorf("failed to allocate job ID: %w", err)
return 0, merr.Wrap(err, "failed to allocate job ID")
}
log.Info("pre-allocated job ID for restore", zap.Int64("jobID", jobID))
@@ -741,7 +741,7 @@ func (sm *snapshotManager) RestoreSnapshot(
if rollbackErr := rollback(ctx, targetDbName, targetCollectionName); rollbackErr != nil {
log.Error("rollback failed", zap.Error(rollbackErr))
}
return 0, fmt.Errorf("failed to start broadcaster for restore message: %w", err)
return 0, merr.Wrap(err, "failed to start broadcaster for restore message")
}
defer func() {
if restoreBroadcaster != nil {
@@ -760,7 +760,7 @@ func (sm *snapshotManager) RestoreSnapshot(
if rollbackErr := rollback(ctx, targetDbName, targetCollectionName); rollbackErr != nil {
log.Error("rollback failed", zap.Error(rollbackErr))
}
err = fmt.Errorf("resource validation failed: %w", valErr)
err = merr.Wrap(valErr, "resource validation failed")
return 0, err
}
@@ -785,7 +785,7 @@ func (sm *snapshotManager) RestoreSnapshot(
if rollbackErr := rollback(ctx, targetDbName, targetCollectionName); rollbackErr != nil {
log.Error("rollback failed", zap.Error(rollbackErr))
}
err = fmt.Errorf("failed to broadcast restore message: %w", bcErr)
err = merr.Wrap(bcErr, "failed to broadcast restore message")
return 0, err
}
@@ -872,14 +872,14 @@ func (sm *snapshotManager) RestoreIndexes(
// Get collection info for dbId
coll, err := sm.broker.DescribeCollectionInternal(ctx, collectionID)
if err != nil {
return fmt.Errorf("failed to describe collection %d: %w", collectionID, err)
return merr.Wrapf(err, "failed to describe collection %d", collectionID)
}
for _, indexInfo := range snapshotData.Indexes {
// Allocate new index ID
indexID, err := sm.allocator.AllocID(ctx)
if err != nil {
return fmt.Errorf("failed to allocate index ID: %w", err)
return merr.Wrap(err, "failed to allocate index ID")
}
// Build index model from snapshot data
@@ -898,19 +898,19 @@ func (sm *snapshotManager) RestoreIndexes(
// Validate the index params (basic validation without JSON path parsing)
if err := ValidateIndexParams(index); err != nil {
return fmt.Errorf("failed to validate index %s: %w", indexInfo.GetIndexName(), err)
return merr.Wrapf(err, "failed to validate index %s", indexInfo.GetIndexName())
}
// Check scalar index engine version for JSON path indexes with new types
if err := sm.checkJSONPathIndexVersion(index); err != nil {
return fmt.Errorf("failed to validate index %s: %w", indexInfo.GetIndexName(), err)
return merr.Wrapf(err, "failed to validate index %s", indexInfo.GetIndexName())
}
// Create a new broadcaster for each index
// (each broadcaster can only be used once due to resource key lock consumption)
b, err := startBroadcaster(ctx, collectionID, snapshotName)
if err != nil {
return fmt.Errorf("failed to start broadcaster for index %s: %w", indexInfo.GetIndexName(), err)
return merr.Wrapf(err, "failed to start broadcaster for index %s", indexInfo.GetIndexName())
}
// Broadcast CreateIndex message directly to DDL WAL
@@ -930,7 +930,7 @@ func (sm *snapshotManager) RestoreIndexes(
)
b.Close()
if err != nil {
return fmt.Errorf("failed to broadcast create index %s: %w", indexInfo.GetIndexName(), err)
return merr.Wrapf(err, "failed to broadcast create index %s", indexInfo.GetIndexName())
}
log.Ctx(ctx).Info("index restored via DDL WAL broadcast",
@@ -977,14 +977,14 @@ func (sm *snapshotManager) RestoreData(
snapshotData, err := sm.ReadSnapshotData(ctx, sourceCollectionID, snapshotName)
if err != nil {
log.Error("failed to read snapshot data", zap.Error(err))
return 0, fmt.Errorf("failed to read snapshot data: %w", err)
return 0, merr.Wrap(err, "failed to read snapshot data")
}
// ========== Phase 2: Build partition mapping ==========
partitionMapping, err := sm.buildPartitionMapping(ctx, snapshotData, collectionID)
if err != nil {
log.Error("failed to build partition mapping", zap.Error(err))
return 0, fmt.Errorf("partition mapping failed: %w", err)
return 0, merr.Wrap(err, "partition mapping failed")
}
log.Info("partition mapping built", zap.Any("partitionMapping", partitionMapping))
@@ -992,14 +992,14 @@ func (sm *snapshotManager) RestoreData(
channelMapping, err := sm.buildChannelMapping(ctx, snapshotData, collectionID)
if err != nil {
log.Error("failed to build channel mapping", zap.Error(err))
return 0, fmt.Errorf("channel mapping failed: %w", err)
return 0, merr.Wrap(err, "channel mapping failed")
}
// ========== Phase 4: Create copy segment job ==========
// Use the pre-allocated jobID from the WAL message
if err := sm.createRestoreJob(ctx, collectionID, channelMapping, partitionMapping, snapshotData, jobID, pinID); err != nil {
log.Error("failed to create restore job", zap.Error(err))
return 0, fmt.Errorf("restore job creation failed: %w", err)
return 0, merr.Wrap(err, "restore job creation failed")
}
log.Info("restore data completed successfully",
@@ -1047,7 +1047,7 @@ func (sm *snapshotManager) restoreUserPartitions(
}
if err := sm.broker.CreatePartition(ctx, req); err != nil {
return fmt.Errorf("failed to create partition %s: %w", partitionName, err)
return merr.Wrapf(err, "failed to create partition %s", partitionName)
}
}
@@ -1330,7 +1330,7 @@ func (sm *snapshotManager) GetRestoreState(ctx context.Context, jobID int64) (*d
// Get job
job := sm.copySegmentMeta.GetJob(ctx, jobID)
if job == nil {
err := merr.WrapErrImportFailed(fmt.Sprintf("restore job not found: jobID=%d", jobID))
err := merr.WrapErrImportSysFailedMsg("restore job not found: jobID=%d", jobID)
log.Warn("restore job not found")
return nil, err
}
+8 -9
View File
@@ -572,7 +572,7 @@ func (sm *snapshotMeta) SaveSnapshot(ctx context.Context, snapshot *SnapshotData
if err := sm.catalog.SaveSnapshot(ctx, snapshot.SnapshotInfo); err != nil {
log.Error("failed to save pending snapshot to catalog", zap.Error(err))
return fmt.Errorf("failed to save pending snapshot to catalog: %w", err)
return merr.Wrap(err, "failed to save pending snapshot to catalog")
}
log.Info("saved pending snapshot to catalog")
@@ -582,7 +582,7 @@ func (sm *snapshotMeta) SaveSnapshot(ctx context.Context, snapshot *SnapshotData
// S3 write failed, leave PENDING record for GC to clean up
log.Error("failed to save snapshot to S3, pending record left for GC cleanup",
zap.Error(err))
return fmt.Errorf("failed to save snapshot to S3: %w", err)
return merr.Wrap(err, "failed to save snapshot to S3")
}
snapshot.SnapshotInfo.S3Location = metadataFilePath
log.Info("saved snapshot data to S3", zap.String("s3Location", metadataFilePath))
@@ -598,7 +598,7 @@ func (sm *snapshotMeta) SaveSnapshot(ctx context.Context, snapshot *SnapshotData
// GC will eventually cleanup via the PENDING marker.
log.Error("failed to update snapshot to committed state, pending record left for GC cleanup",
zap.Error(err))
return fmt.Errorf("failed to update snapshot to committed state: %w", err)
return merr.Wrap(err, "failed to update snapshot to committed state")
}
// Catalog committed — safe to insert into in-memory cache and register protection.
@@ -809,8 +809,7 @@ func (sm *snapshotMeta) DropSnapshotsByCollection(ctx context.Context, collectio
}
if len(errs) > 0 {
return dropped, fmt.Errorf("failed to drop %d/%d snapshots for collection %d: %w",
len(errs), len(snapshotNames), collectionID, merr.Combine(errs...))
return dropped, merr.WrapErrServiceInternalErr(merr.Combine(errs...), "failed to drop %d/%d snapshots for collection %d", len(errs), len(snapshotNames), collectionID)
}
return dropped, nil
@@ -1006,7 +1005,7 @@ func (sm *snapshotMeta) GetPendingSnapshots(ctx context.Context, pendingTimeout
// Get all snapshots from catalog (etcd)
snapshots, err := sm.catalog.ListSnapshots(ctx)
if err != nil {
return nil, fmt.Errorf("failed to list snapshots from catalog: %w", err)
return nil, merr.Wrap(err, "failed to list snapshots from catalog")
}
now := time.Now().UnixMilli()
@@ -1057,7 +1056,7 @@ func (sm *snapshotMeta) CleanupPendingSnapshot(ctx context.Context, snapshot *da
func (sm *snapshotMeta) GetDeletingSnapshots(ctx context.Context) ([]*datapb.SnapshotInfo, error) {
snapshots, err := sm.catalog.ListSnapshots(ctx)
if err != nil {
return nil, fmt.Errorf("failed to list snapshots from catalog: %w", err)
return nil, merr.Wrap(err, "failed to list snapshots from catalog")
}
deletingSnapshots := make([]*datapb.SnapshotInfo, 0)
@@ -1774,7 +1773,7 @@ func (sm *snapshotMeta) generatePinID(snapshot *datapb.SnapshotInfo) (int64, err
for i := 0; i < 3; i++ {
var b [8]byte
if _, err := cryptorand.Read(b[:]); err != nil {
return 0, fmt.Errorf("failed to generate random pin ID: %w", err)
return 0, merr.WrapErrServiceInternalErr(err, "failed to generate random pin ID")
}
id := int64(binary.BigEndian.Uint64(b[:]) >> 1)
if id == 0 {
@@ -1784,5 +1783,5 @@ func (sm *snapshotMeta) generatePinID(snapshot *datapb.SnapshotInfo) (int64, err
return id, nil
}
}
return 0, fmt.Errorf("failed to generate unique pin ID after 3 attempts")
return 0, merr.WrapErrServiceInternalMsg("failed to generate unique pin ID after 3 attempts")
}
+5 -5
View File
@@ -182,7 +182,7 @@ func (stm *statsTaskMeta) UpdateVersion(taskID, nodeID int64) error {
t, ok := stm.tasks.Get(taskID)
if !ok {
return fmt.Errorf("task %d not found", taskID)
return merr.WrapErrServiceInternalMsg("task %d not found", taskID)
}
cloneT := proto.Clone(t).(*indexpb.StatsTask)
@@ -212,7 +212,7 @@ func (stm *statsTaskMeta) UpdateTaskState(taskID int64, state indexpb.JobState,
t, ok := stm.tasks.Get(taskID)
if !ok {
return fmt.Errorf("task %d not found", taskID)
return merr.WrapErrServiceInternalMsg("task %d not found", taskID)
}
cloneT := proto.Clone(t).(*indexpb.StatsTask)
@@ -239,7 +239,7 @@ func (stm *statsTaskMeta) UpdateBuildingTask(taskID int64) error {
t, ok := stm.tasks.Get(taskID)
if !ok {
return fmt.Errorf("task %d not found", taskID)
return merr.WrapErrServiceInternalMsg("task %d not found", taskID)
}
cloneT := proto.Clone(t).(*indexpb.StatsTask)
@@ -267,7 +267,7 @@ func (stm *statsTaskMeta) FinishTask(taskID int64, result *workerpb.StatsResult)
t, ok := stm.tasks.Get(taskID)
if !ok {
return fmt.Errorf("task %d not found", taskID)
return merr.WrapErrServiceInternalMsg("task %d not found", taskID)
}
cloneT := proto.Clone(t).(*indexpb.StatsTask)
@@ -358,7 +358,7 @@ func (stm *statsTaskMeta) MarkTaskCanRecycle(taskID int64) error {
t, ok := stm.tasks.Get(taskID)
if !ok {
return fmt.Errorf("task %d not found", taskID)
return merr.WrapErrServiceInternalMsg("task %d not found", taskID)
}
cloneT := proto.Clone(t).(*indexpb.StatsTask)
+4 -5
View File
@@ -18,7 +18,6 @@ package datacoord
import (
"context"
"fmt"
"path"
"time"
@@ -241,7 +240,7 @@ func (it *indexBuildTask) prepareJobRequest(ctx context.Context, segment *Segmen
var err error
params, err = Params.KnowhereConfig.UpdateIndexParams(GetIndexType(params), paramtable.BuildStage, params)
if err != nil {
return nil, fmt.Errorf("failed to update index build params: %w", err)
return nil, merr.WrapErrServiceInternalErr(err, "failed to update index build params")
}
}
@@ -249,14 +248,14 @@ func (it *indexBuildTask) prepareJobRequest(ctx context.Context, segment *Segmen
var err error
params, err = indexparams.UpdateDiskIndexBuildParams(Params, params)
if err != nil {
return nil, fmt.Errorf("failed to append index build params: %w", err)
return nil, merr.WrapErrServiceInternalErr(err, "failed to append index build params")
}
}
// Get collection info and field
collectionInfo, err := it.handler.GetCollection(ctx, segment.GetCollectionID())
if err != nil {
return nil, fmt.Errorf("failed to get collection info: %w", err)
return nil, merr.Wrap(err, "failed to get collection info")
}
schema := collectionInfo.Schema
@@ -271,7 +270,7 @@ func (it *indexBuildTask) prepareJobRequest(ctx context.Context, segment *Segmen
}
if field == nil {
return nil, fmt.Errorf("field not found with ID %d", fieldID)
return nil, merr.WrapErrFieldNotFound(fieldID)
}
// Extract dim only for vector types to avoid unnecessary warnings
@@ -34,6 +34,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/proto/datapb"
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
"github.com/milvus-io/milvus/pkg/v3/taskcommon"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
)
@@ -115,7 +116,7 @@ func (t *refreshExternalCollectionTask) validateSource() error {
// Validate against job-level snapshot to isolate in-flight tasks from schema changes.
job := t.refreshMeta.GetJob(t.GetJobId())
if job == nil {
return fmt.Errorf("job %d not found", t.GetJobId())
return merr.WrapErrServiceInternalMsg("job %d not found", t.GetJobId())
}
currentSource := job.GetExternalSource()
@@ -125,7 +126,7 @@ func (t *refreshExternalCollectionTask) validateSource() error {
taskSpec := t.GetExternalSpec()
if currentSource != taskSource || currentSpec != taskSpec {
return fmt.Errorf(
return merr.WrapErrServiceInternalMsg(
"task source mismatch: task source=%s/%s, job source=%s/%s (task belongs to a different refresh job)",
taskSource, taskSpec, currentSource, currentSpec,
)
@@ -215,7 +216,7 @@ func applyExternalCollectionSegmentUpdate(
logFields ...zap.Field,
) error {
if mt == nil {
return fmt.Errorf("meta is nil, cannot update segments")
return merr.WrapErrServiceInternalMsg("meta is nil, cannot update segments")
}
fields := append(logFields, zap.Int64("collectionID", collectionID))
log := log.Ctx(ctx).With(fields...)
@@ -228,14 +229,14 @@ func applyExternalCollectionSegmentUpdate(
for _, segID := range keptSegmentIDs {
segment := mt.segments.GetSegment(segID)
if segment == nil {
return fmt.Errorf("kept segment %d not found", segID)
return merr.WrapErrServiceInternalMsg("kept segment %d not found", segID)
}
if segment.GetCollectionID() != collectionID {
return fmt.Errorf("collection mismatch for kept segment %d: existing %d, want %d",
return merr.WrapErrServiceInternalMsg("collection mismatch for kept segment %d: existing %d, want %d",
segID, segment.GetCollectionID(), collectionID)
}
if segment.GetState() == commonpb.SegmentState_Dropped {
return fmt.Errorf("cannot keep dropped segment %d", segID)
return merr.WrapErrServiceInternalMsg("cannot keep dropped segment %d", segID)
}
keptSegmentMap[segID] = true
}
@@ -250,10 +251,10 @@ func applyExternalCollectionSegmentUpdate(
return err
}
if keptSegmentMap[seg.GetID()] {
return fmt.Errorf("segment %d cannot be both kept and updated", seg.GetID())
return merr.WrapErrServiceInternalMsg("segment %d cannot be both kept and updated", seg.GetID())
}
if _, ok := upsertSegmentMap[seg.GetID()]; ok {
return fmt.Errorf("duplicate updated segment %d", seg.GetID())
return merr.WrapErrServiceInternalMsg("duplicate updated segment %d", seg.GetID())
}
upsertSegmentMap[seg.GetID()] = seg
validUpdatedSegments = append(validUpdatedSegments, seg)
@@ -308,7 +309,7 @@ func applyExternalCollectionSegmentUpdate(
zap.Int("activeSegmentCount", activeSegmentCount),
zap.Int("keptSegments", len(keptSegmentMap)),
zap.Int("updatedSegments", len(upsertSegmentMap)))
return fmt.Errorf("safety check failed: refusing to drop all %d segments without replacement (keptSegments=%d, updatedSegments=%d)",
return merr.WrapErrServiceInternalMsg("safety check failed: refusing to drop all %d segments without replacement (keptSegments=%d, updatedSegments=%d)",
activeSegmentCount, len(keptSegmentMap), len(upsertSegmentMap))
}
@@ -330,15 +331,15 @@ func applyExternalCollectionSegmentUpdate(
collInfo := mt.GetCollection(collectionID)
if collInfo == nil {
return fmt.Errorf("collection %d not found in meta", collectionID)
return merr.WrapErrServiceInternalMsg("collection %d not found in meta", collectionID)
}
// External collections are single-shard, single-partition (enforced at creation).
// Assert exactly-one here to catch any invariant violation from data corruption or legacy data.
if len(collInfo.VChannelNames) != 1 {
return fmt.Errorf("external collection %d expected exactly 1 VChannel, got %d", collectionID, len(collInfo.VChannelNames))
return merr.WrapErrServiceInternalMsg("external collection %d expected exactly 1 VChannel, got %d", collectionID, len(collInfo.VChannelNames))
}
if len(collInfo.Partitions) != 1 {
return fmt.Errorf("external collection %d expected exactly 1 partition, got %d", collectionID, len(collInfo.Partitions))
return merr.WrapErrServiceInternalMsg("external collection %d expected exactly 1 partition, got %d", collectionID, len(collInfo.Partitions))
}
insertChannel := collInfo.VChannelNames[0]
partitionID := collInfo.Partitions[0]
@@ -477,14 +478,14 @@ func applyExternalCollectionSegmentUpdate(
func validateExternalRefreshUpdatedSegment(incoming *datapb.SegmentInfo, collectionID int64) error {
if incoming.GetCollectionID() != 0 && incoming.GetCollectionID() != collectionID {
return fmt.Errorf("collection mismatch for segment %d: got %d, want %d",
return merr.WrapErrServiceInternalMsg("collection mismatch for segment %d: got %d, want %d",
incoming.GetID(), incoming.GetCollectionID(), collectionID)
}
if incoming.GetManifestPath() == "" {
return fmt.Errorf("updated segment %d has empty manifest path", incoming.GetID())
return merr.WrapErrServiceInternalMsg("updated segment %d has empty manifest path", incoming.GetID())
}
if len(incoming.GetBinlogs()) == 0 {
return fmt.Errorf("updated segment %d has empty fake binlogs", incoming.GetID())
return merr.WrapErrServiceInternalMsg("updated segment %d has empty fake binlogs", incoming.GetID())
}
return nil
}
@@ -513,36 +514,36 @@ func validateExternalRefreshNewSegment(incoming *datapb.SegmentInfo) error {
func validateExternalRefreshPatch(oldSeg *SegmentInfo, incoming *datapb.SegmentInfo, collectionID int64) error {
if oldSeg == nil {
return fmt.Errorf("existing segment is nil")
return merr.WrapErrServiceInternalMsg("existing segment is nil")
}
if oldSeg.GetCollectionID() != collectionID {
return fmt.Errorf("collection mismatch for segment %d: existing %d, want %d",
return merr.WrapErrServiceInternalMsg("collection mismatch for segment %d: existing %d, want %d",
oldSeg.GetID(), oldSeg.GetCollectionID(), collectionID)
}
if oldSeg.GetState() == commonpb.SegmentState_Dropped {
return fmt.Errorf("cannot patch dropped segment %d", oldSeg.GetID())
return merr.WrapErrServiceInternalMsg("cannot patch dropped segment %d", oldSeg.GetID())
}
if incoming.GetCollectionID() != 0 && incoming.GetCollectionID() != collectionID {
return fmt.Errorf("collection mismatch for segment %d: got %d, want %d",
return merr.WrapErrServiceInternalMsg("collection mismatch for segment %d: got %d, want %d",
incoming.GetID(), incoming.GetCollectionID(), collectionID)
}
if incoming.GetNumOfRows() != oldSeg.GetNumOfRows() {
return fmt.Errorf("row count changed for segment %d: got %d, want %d",
return merr.WrapErrServiceInternalMsg("row count changed for segment %d: got %d, want %d",
incoming.GetID(), incoming.GetNumOfRows(), oldSeg.GetNumOfRows())
}
if incoming.GetStorageVersion() != 0 && incoming.GetStorageVersion() != oldSeg.GetStorageVersion() {
return fmt.Errorf("storage version changed for segment %d: got %d, want %d",
return merr.WrapErrServiceInternalMsg("storage version changed for segment %d: got %d, want %d",
incoming.GetID(), incoming.GetStorageVersion(), oldSeg.GetStorageVersion())
}
if incoming.GetSchemaVersion() < oldSeg.GetSchemaVersion() {
return fmt.Errorf("schema version rollback for segment %d: got %d, want >= %d",
return merr.WrapErrServiceInternalMsg("schema version rollback for segment %d: got %d, want >= %d",
incoming.GetID(), incoming.GetSchemaVersion(), oldSeg.GetSchemaVersion())
}
if incoming.GetManifestPath() == "" {
return fmt.Errorf("patched segment %d has empty manifest path", incoming.GetID())
return merr.WrapErrServiceInternalMsg("patched segment %d has empty manifest path", incoming.GetID())
}
if len(incoming.GetBinlogs()) == 0 {
return fmt.Errorf("patched segment %d has empty fake binlogs", incoming.GetID())
return merr.WrapErrServiceInternalMsg("patched segment %d has empty fake binlogs", incoming.GetID())
}
if err := validateExternalRefreshBinlogRowCount(incoming, oldSeg.GetNumOfRows()); err != nil {
return err
@@ -553,14 +554,14 @@ func validateExternalRefreshPatch(oldSeg *SegmentInfo, incoming *datapb.SegmentI
func validateExternalRefreshBinlogRowCount(segment *datapb.SegmentInfo, expectedRows int64) error {
binlogRows := segmentutil.CalcRowCountFromBinLog(segment)
if binlogRows == -1 {
return fmt.Errorf("invalid binlog row count for segment %d", segment.GetID())
return merr.WrapErrServiceInternalMsg("invalid binlog row count for segment %d", segment.GetID())
}
if expectedRows > 0 && binlogRows != expectedRows {
return fmt.Errorf("binlog row count mismatch for segment %d: got %d, want %d",
return merr.WrapErrServiceInternalMsg("binlog row count mismatch for segment %d: got %d, want %d",
segment.GetID(), binlogRows, expectedRows)
}
if binlogRows > 0 && binlogRows != segment.GetNumOfRows() {
return fmt.Errorf("binlog row count mismatch for segment %d: got %d, segment rows %d",
return merr.WrapErrServiceInternalMsg("binlog row count mismatch for segment %d: got %d, segment rows %d",
segment.GetID(), binlogRows, segment.GetNumOfRows())
}
return nil
@@ -612,7 +613,7 @@ func (t *refreshExternalCollectionTask) CreateTaskOnWorker(nodeID int64, cluster
log.Info("creating refresh task on worker")
if t.mt == nil {
err = fmt.Errorf("meta is nil, cannot create task on worker")
err = merr.WrapErrServiceInternalMsg("meta is nil, cannot create task on worker")
return
}
@@ -625,7 +626,7 @@ func (t *refreshExternalCollectionTask) CreateTaskOnWorker(nodeID int64, cluster
// Re-read task from meta to sync in-memory state (nodeID and version)
updatedTask := t.refreshMeta.GetTask(t.GetTaskId())
if updatedTask == nil {
err = fmt.Errorf("task %d not found after version update", t.GetTaskId())
err = merr.WrapErrServiceInternalMsg("task %d not found after version update", t.GetTaskId())
return
}
t.ExternalCollectionRefreshTask = updatedTask
@@ -668,11 +669,11 @@ func (t *refreshExternalCollectionTask) CreateTaskOnWorker(nodeID int64, cluster
// lock, before they are supported.
collInfo := t.mt.GetCollection(t.GetCollectionId())
if collInfo == nil {
err = fmt.Errorf("collection %d not found in meta", t.GetCollectionId())
err = merr.WrapErrServiceInternalMsg("collection %d not found in meta", t.GetCollectionId())
return
}
if len(collInfo.Partitions) != 1 {
err = fmt.Errorf("external collection %d expected exactly 1 partition, got %d", t.GetCollectionId(), len(collInfo.Partitions))
err = merr.WrapErrServiceInternalMsg("external collection %d expected exactly 1 partition, got %d", t.GetCollectionId(), len(collInfo.Partitions))
return
}
partitionID := collInfo.Partitions[0]
+10 -5
View File
@@ -18,7 +18,6 @@ package datacoord
import (
"context"
"fmt"
"time"
"github.com/cockroachdb/errors"
@@ -299,11 +298,17 @@ func (st *statsTask) handleEmptySegment(ctx context.Context) error {
// Prepare the stats request
func (st *statsTask) prepareJobRequest(ctx context.Context, segment *SegmentInfo) (*workerpb.CreateStatsRequest, error) {
collInfo, err := st.handler.GetCollection(ctx, segment.GetCollectionID())
if err != nil || collInfo == nil {
return nil, fmt.Errorf("failed to get collection info: %w", err)
if err != nil {
return nil, merr.Wrap(err, "failed to get collection info")
}
// GetCollection can return (nil, nil) on a cache miss; merr.Wrap(nil) would
// be nil and silently submit a malformed request, so guard collInfo
// separately with a typed not-found.
if collInfo == nil {
return nil, merr.WrapErrCollectionNotFound(segment.GetCollectionID())
}
if collInfo.Schema == nil || len(collInfo.Schema.GetFields()) == 0 {
return nil, fmt.Errorf("collection schema is nil or has no fields, collectionID: %d", segment.GetCollectionID())
return nil, merr.WrapErrServiceInternalMsg("collection schema is nil or has no fields, collectionID: %d", segment.GetCollectionID())
}
// Calculate binlog allocation
@@ -314,7 +319,7 @@ func (st *statsTask) prepareJobRequest(ctx context.Context, segment *SegmentInfo
// Allocate IDs
start, end, err := st.allocator.AllocN(binlogNum + int64(len(collInfo.Schema.GetFunctions())) + 1)
if err != nil {
return nil, fmt.Errorf("failed to allocate log IDs: %w", err)
return nil, merr.Wrap(err, "failed to allocate log IDs")
}
// Create the request
@@ -26,7 +26,6 @@ import (
"github.com/apache/arrow/go/v17/arrow"
"github.com/apache/arrow/go/v17/arrow/array"
"github.com/cockroachdb/errors"
"go.opentelemetry.io/otel"
"go.uber.org/zap"
@@ -113,14 +112,16 @@ func (t *bumpSchemaVersionCompactionTask) preCompact() error {
return t.ctx.Err()
}
// Check segment binlogs: must have exactly one segment
// Check segment binlogs: must have exactly one segment.
// The plan is produced by datacoord, so a malformed plan is an internal
// protocol violation, not user input.
if len(t.plan.GetSegmentBinlogs()) != 1 {
return errors.Newf("schema bump compaction plan is illegal, must have exactly one segment, but got %d segments, planID = %d", len(t.plan.GetSegmentBinlogs()), t.GetPlanID())
return merr.WrapErrServiceInternalMsg("schema bump compaction plan is illegal, must have exactly one segment, but got %d segments, planID = %d", len(t.plan.GetSegmentBinlogs()), t.GetPlanID())
}
segment := t.plan.GetSegmentBinlogs()[0]
if segment.GetStorageVersion() < storage.StorageV3 || segment.GetManifest() == "" {
return errors.Newf("schema bump compaction requires a StorageV3 segment with manifest, planID = %d, segmentID = %d, storageVersion = %d", t.GetPlanID(), segment.GetSegmentID(), segment.GetStorageVersion())
return merr.WrapErrServiceInternalMsg("schema bump compaction requires a StorageV3 segment with manifest, planID = %d, segmentID = %d, storageVersion = %d", t.GetPlanID(), segment.GetSegmentID(), segment.GetStorageVersion())
}
return nil
@@ -147,7 +148,7 @@ func (t *bumpSchemaVersionCompactionTask) missingFunctionInputSchema(missingFunc
for _, inputFieldID := range functionSchema.GetInputFieldIds() {
inputField := typeutil.GetField(schema, inputFieldID)
if inputField == nil {
return nil, nil, errors.New("input field not found in schema")
return nil, nil, merr.WrapErrParameterInvalidMsg("input field not found in schema")
}
if err := validateMaterializationInputField(functionSchema, inputField); err != nil {
return nil, nil, err
@@ -196,11 +197,11 @@ func bm25AdditionalInputFields(schema *schemapb.CollectionSchema, inputField *sc
return nil, err
}
if multiAnalyzerParams.ByField == "" {
return nil, errors.New("multi_analyzer_params missing required 'by_field' key")
return nil, merr.WrapErrParameterInvalidMsg("multi_analyzer_params missing required 'by_field' key")
}
byField := typeutil.GetFieldByName(schema, multiAnalyzerParams.ByField)
if byField == nil {
return nil, errors.New("input field not found in schema")
return nil, merr.WrapErrParameterInvalidMsg("input field not found in schema")
}
return []*schemapb.FieldSchema{byField}, nil
}
@@ -217,7 +218,7 @@ func (t *bumpSchemaVersionCompactionTask) missingFunctionOutputFields(missingFun
outputFieldID := functionSchema.GetOutputFieldIds()[outputIndex]
outputField := typeutil.GetField(schema, outputFieldID)
if outputField == nil {
return nil, nil, errors.New("output field not found in schema")
return nil, nil, merr.WrapErrParameterInvalidMsg("output field not found in schema")
}
if err := validateMaterializationOutputField(functionSchema, outputField); err != nil {
return nil, nil, err
@@ -227,7 +228,7 @@ func (t *bumpSchemaVersionCompactionTask) missingFunctionOutputFields(missingFun
}
}
if len(fields) == 0 {
return nil, nil, errors.New("no missing function output fields")
return nil, nil, merr.WrapErrParameterInvalidMsg("no missing function output fields")
}
return fields, fieldIDs, nil
}
@@ -236,22 +237,22 @@ func validateSupportedMissingFunctionMaterialization(functionSchema *schemapb.Fu
switch functionSchema.GetType() {
case schemapb.FunctionType_BM25:
if len(functionSchema.GetInputFieldIds()) == 0 {
return errors.New("bm25 function should have input fields")
return merr.WrapErrParameterInvalidMsg("bm25 function should have input fields")
}
if len(functionSchema.GetOutputFieldIds()) == 0 {
return errors.New("bm25 function should have output fields")
return merr.WrapErrParameterInvalidMsg("bm25 function should have output fields")
}
return nil
case schemapb.FunctionType_MinHash:
if len(functionSchema.GetInputFieldIds()) == 0 {
return errors.New("minhash function should have input fields")
return merr.WrapErrParameterInvalidMsg("minhash function should have input fields")
}
if len(functionSchema.GetOutputFieldIds()) == 0 {
return errors.New("minhash function should have output fields")
return merr.WrapErrParameterInvalidMsg("minhash function should have output fields")
}
return nil
default:
return errors.New("unsupported function type")
return merr.WrapErrParameterInvalidMsg("unsupported function type")
}
}
@@ -259,11 +260,11 @@ func validateMaterializationInputField(functionSchema *schemapb.FunctionSchema,
switch functionSchema.GetType() {
case schemapb.FunctionType_BM25:
if field.GetDataType() != schemapb.DataType_VarChar && field.GetDataType() != schemapb.DataType_Text {
return errors.New("input field data type must be varchar or text for bm25 materialization")
return merr.WrapErrParameterInvalidMsg("input field data type must be varchar or text for bm25 materialization")
}
case schemapb.FunctionType_MinHash:
if field.GetDataType() != schemapb.DataType_VarChar && field.GetDataType() != schemapb.DataType_Text {
return errors.New("input field data type must be varchar or text for minhash materialization")
return merr.WrapErrParameterInvalidMsg("input field data type must be varchar or text for minhash materialization")
}
}
return nil
@@ -273,11 +274,11 @@ func validateMaterializationOutputField(functionSchema *schemapb.FunctionSchema,
switch functionSchema.GetType() {
case schemapb.FunctionType_BM25:
if field.GetDataType() != schemapb.DataType_SparseFloatVector {
return errors.New("output field data type must be sparse float vector for bm25 materialization")
return merr.WrapErrParameterInvalidMsg("output field data type must be sparse float vector for bm25 materialization")
}
case schemapb.FunctionType_MinHash:
if field.GetDataType() != schemapb.DataType_BinaryVector {
return errors.New("output field data type must be binary vector for minhash materialization")
return merr.WrapErrParameterInvalidMsg("output field data type must be binary vector for minhash materialization")
}
}
return nil
@@ -546,7 +547,7 @@ func (t *bumpSchemaVersionCompactionTask) runFullSchemaRewrite(existingFields ma
}
manifestPath, err = packed.AddStatsToManifest(manifestPath, t.compactionParams.StorageConfig, statEntries)
if err != nil {
return nil, merr.WrapErrServiceInternal("failed to add writer stats to schema bump full rewrite manifest", err.Error())
return nil, merr.Wrap(err, "failed to add writer stats to schema bump full rewrite manifest")
}
}
sortedInsertLogs := storage.SortFieldBinlogs(insertLogs)
@@ -602,7 +603,7 @@ func (t *bumpSchemaVersionCompactionTask) runFullSchemaRewrite(existingFields ma
func appendBM25StatsFromArrowArray(stats *storage.BM25Stats, arr arrow.Array) (int, error) {
binaryArray, ok := arr.(*array.Binary)
if !ok {
return 0, errors.Newf("bm25 output field must be arrow binary array, got %T", arr)
return 0, merr.WrapErrParameterInvalidMsg("bm25 output field must be arrow binary array, got %T", arr)
}
memorySize := 0
for i := 0; i < binaryArray.Len(); i++ {
@@ -667,7 +668,7 @@ func (t *bumpSchemaVersionCompactionTask) setupWriter(outputFields []*schemapb.F
basePath, existingVersion, err := packed.UnmarshalManifestPath(segment.GetManifest())
if err != nil {
return nil, merr.WrapErrServiceInternal("failed to parse existing manifest for schema bump", err.Error())
return nil, merr.WrapErrDataIntegrityMsg("failed to parse existing manifest for schema bump: %s", err.Error())
}
writerResult, err := t.newV3WriterResult(outputSchema, newColumnGroups, segment, collectionID, basePath, existingVersion)
if err != nil {
@@ -685,7 +686,7 @@ func (t *bumpSchemaVersionCompactionTask) setupWriter(outputFields []*schemapb.F
func (t *bumpSchemaVersionCompactionTask) newV3WriterResult(schema *schemapb.CollectionSchema, columnGroups []storagecommon.ColumnGroup, segment *datapb.CompactionSegmentBinlogs, collectionID int64, basePath string, baseVersion int64) (*bumpSchemaVersionWriterResult, error) {
if segment.GetStorageVersion() < storage.StorageV3 || segment.GetManifest() == "" {
return nil, errors.New("schema bump compaction requires a StorageV3 segment with manifest")
return nil, merr.WrapErrParameterInvalidMsg("schema bump compaction requires a StorageV3 segment with manifest")
}
pluginContext, err := hookutil.GetCPluginContext(t.plan.GetPluginContext(), collectionID)
@@ -732,7 +733,7 @@ func (t *bumpSchemaVersionCompactionTask) addV3Stats(prefix string, fieldID int6
statsRelPath := fmt.Sprintf("_stats/%s.%d/%d", prefix, fieldID, statsID)
absStatsPath := path.Join(writerResult.basePath, statsRelPath)
if err := packed.WriteFile(t.compactionParams.StorageConfig, absStatsPath, bytes); err != nil {
return merr.WrapErrServiceInternal("failed to write V3 stats", err.Error())
return merr.Wrap(err, "failed to write V3 stats")
}
writerResult.v3Stats = append(writerResult.v3Stats, packed.FieldBinlogStatEntry(prefix, fieldID, &datapb.FieldBinlog{
FieldID: fieldID,
@@ -773,7 +774,7 @@ func (t *bumpSchemaVersionCompactionTask) buildMergedLogsV3(segment *datapb.Comp
func (t *bumpSchemaVersionCompactionTask) preserveDeltaLogsV3(segment *datapb.CompactionSegmentBinlogs, manifestPath string) (string, error) {
deltaPaths, err := packed.GetDeltaLogPathsFromManifest(segment.GetManifest(), t.compactionParams.StorageConfig)
if err != nil {
return "", merr.WrapErrServiceInternal("failed to read V3 delta logs from existing manifest", err.Error())
return "", merr.Wrap(err, "failed to read V3 delta logs from existing manifest")
}
if len(deltaPaths) == 0 {
return manifestPath, nil
@@ -784,12 +785,12 @@ func (t *bumpSchemaVersionCompactionTask) preserveDeltaLogsV3(segment *datapb.Co
deltaSummaries = append(deltaSummaries, fieldBinlog.GetBinlogs()...)
}
if len(deltaSummaries) != len(deltaPaths) {
return "", merr.WrapErrServiceInternal(fmt.Sprintf("V3 delta manifest path count %d does not match segment delta summary count %d", len(deltaPaths), len(deltaSummaries)))
return "", merr.WrapErrServiceInternalMsg("V3 delta manifest path count %d does not match segment delta summary count %d", len(deltaPaths), len(deltaSummaries))
}
basePath, _, err := packed.UnmarshalManifestPath(manifestPath)
if err != nil {
return "", merr.WrapErrServiceInternal("failed to parse new V3 manifest for delta preservation", err.Error())
return "", merr.WrapErrDataIntegrityMsg("failed to parse new V3 manifest for delta preservation: %s", err.Error())
}
deltaEntries := make([]packed.DeltaLogEntry, 0, len(deltaPaths))
@@ -797,7 +798,7 @@ func (t *bumpSchemaVersionCompactionTask) preserveDeltaLogsV3(segment *datapb.Co
newDeltaPath := metautil.BuildDeltaLogPathV3(basePath, deltaSummaries[i].GetLogID())
if newDeltaPath != deltaPath {
if err := t.chunkManager.Copy(t.ctx, deltaPath, newDeltaPath); err != nil {
return "", merr.WrapErrServiceInternal("failed to copy V3 delta log for schema bump full rewrite", err.Error())
return "", merr.Wrap(err, "failed to copy V3 delta log for schema bump full rewrite")
}
}
deltaEntries = append(deltaEntries, packed.DeltaLogEntry{
@@ -807,7 +808,7 @@ func (t *bumpSchemaVersionCompactionTask) preserveDeltaLogsV3(segment *datapb.Co
}
newManifest, err := packed.AddDeltaLogsToManifest(manifestPath, t.compactionParams.StorageConfig, deltaEntries)
if err != nil {
return "", merr.WrapErrServiceInternal("failed to preserve V3 delta logs in full rewrite manifest", err.Error())
return "", merr.Wrap(err, "failed to preserve V3 delta logs in full rewrite manifest")
}
return newManifest, nil
}
@@ -923,7 +924,7 @@ func (t *bumpSchemaVersionCompactionTask) runMissingFunctionMaterialization(ctx
if outputCol == nil {
releaseWrappedRecord(wrapped, record)
span.End()
return nil, merr.WrapErrServiceInternal(fmt.Sprintf("output field %d not found in materialized record", outputFieldID))
return nil, merr.WrapErrServiceInternalMsg("output field %d not found in materialized record", outputFieldID)
}
batchMemSize := arrowArrayMemorySize(outputCol)
if stats, ok := statsByField[outputFieldID]; ok {
@@ -997,7 +998,7 @@ func (t *bumpSchemaVersionCompactionTask) runMissingFunctionMaterialization(ctx
},
)
if err != nil {
return nil, merr.WrapErrServiceInternal("failed to commit schema bump V3 manifest", err.Error())
return nil, merr.Wrap(err, "failed to commit schema bump V3 manifest")
}
log.Info("[schema-bump-partial-writer] writer output and bm25 stats committed",
zap.String("manifestPath", manifestPath),
@@ -28,7 +28,6 @@ import (
"sync"
"time"
"github.com/cockroachdb/errors"
"github.com/samber/lo"
"go.opentelemetry.io/otel"
"go.uber.org/atomic"
@@ -663,7 +662,7 @@ func (t *clusteringCompactionTask) mappingSegment(
row, ok := v.Value.(map[typeutil.UniqueID]interface{})
if !ok {
log.Warn("convert interface to map wrong")
return errors.New("unexpected error")
return merr.WrapErrServiceInternalMsg("unexpected error")
}
expireTs := int64(-1)
if hasTTLField {
@@ -976,7 +975,7 @@ func (t *clusteringCompactionTask) iterAndGetScalarAnalyzeResult(pkIter *storage
// rowValue := vIter.GetData().(*iterators.InsertRow).GetValue()
row, ok := (*v).Value.(map[typeutil.UniqueID]interface{})
if !ok {
return nil, 0, errors.New("unexpected error")
return nil, 0, merr.WrapErrServiceInternalMsg("unexpected error")
}
expireTs := int64(-1)
@@ -46,6 +46,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/proto/datapb"
"github.com/milvus-io/milvus/pkg/v3/proto/indexcgopb"
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/metautil"
"github.com/milvus-io/milvus/pkg/v3/util/tsoutil"
"github.com/milvus-io/milvus/pkg/v3/util/typeutil"
@@ -80,7 +81,7 @@ func createTextIndex(ctx context.Context,
}
binlogs, ok := fieldBinlogs[fieldID]
if !ok {
return nil, fmt.Errorf("field binlog not found for field %d", fieldID)
return nil, merr.WrapErrParameterInvalidMsg("field binlog not found for field %d", fieldID)
}
result := make([]string, 0, len(binlogs))
for _, binlog := range binlogs {
@@ -136,7 +137,7 @@ func createTextIndex(ctx context.Context,
if segment.GetManifest() != "" {
basePath, _, err := packed.UnmarshalManifestPath(segment.GetManifest())
if err != nil {
return fmt.Errorf("failed to unmarshal manifest path for text_index basePath: %w", err)
return merr.Wrap(err, "failed to unmarshal manifest path for text_index basePath")
}
statsBasePath = fmt.Sprintf("%s/_stats/text_index.%d", basePath, field.GetFieldID())
}
+6 -3
View File
@@ -18,10 +18,10 @@ package compactor
import (
"context"
"fmt"
"math"
"sync"
"github.com/cockroachdb/errors"
"github.com/samber/lo"
"go.opentelemetry.io/otel"
"go.uber.org/zap"
@@ -40,6 +40,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/util/conc"
"github.com/milvus-io/milvus/pkg/v3/util/funcutil"
"github.com/milvus-io/milvus/pkg/v3/util/hardware"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/metautil"
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
"github.com/milvus-io/milvus/pkg/v3/util/timerecord"
@@ -135,7 +136,9 @@ func (t *LevelZeroCompactionTask) Compact() (*datapb.CompactionPlanResult, error
})
if len(targetSegments) == 0 {
log.Warn("compact wrong, not target sealed segments")
return nil, errors.New("illegal compaction plan with empty target segments")
// The plan is produced by datacoord, so a malformed plan is an internal
// protocol violation, not user input.
return nil, merr.WrapErrServiceInternalMsg("illegal compaction plan with empty target segments")
}
err = binlog.DecompressCompactionBinlogsWithRootPath(t.compactionParams.StorageConfig.GetRootPath(), l0Segments)
if err != nil {
@@ -416,7 +419,7 @@ func (t *LevelZeroCompactionTask) process(ctx context.Context, l0MemSize int64,
ratio := paramtable.Get().DataNodeCfg.L0BatchMemoryRatio.GetAsFloat()
memLimit := float64(hardware.GetFreeMemoryCount()) * ratio
if float64(l0MemSize) > memLimit {
return nil, errors.Newf("L0 compaction failed, not enough memory, request memory size: %v, memory limit: %v", l0MemSize, memLimit)
return nil, merr.Wrap(merr.ErrServiceMemoryLimitExceeded, fmt.Sprintf("L0 compaction failed, not enough memory, request memory size: %v, memory limit: %v", l0MemSize, memLimit))
}
log.Info("L0 compaction process start")
+6 -4
View File
@@ -25,7 +25,6 @@ import (
"time"
"github.com/apache/arrow/go/v17/arrow/array"
"github.com/cockroachdb/errors"
"github.com/samber/lo"
"go.opentelemetry.io/otel"
"go.uber.org/zap"
@@ -42,6 +41,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/proto/datapb"
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
"github.com/milvus-io/milvus/pkg/v3/util/funcutil"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/metautil"
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
"github.com/milvus-io/milvus/pkg/v3/util/timerecord"
@@ -110,11 +110,13 @@ func (t *mixCompactionTask) preCompact() error {
}
if len(t.plan.GetSegmentBinlogs()) < 1 {
return errors.Newf("compaction plan is illegal, there's no segments in compaction plan, planID = %d", t.GetPlanID())
// The plan is produced by datacoord, so a malformed plan is an internal
// protocol violation, not user input.
return merr.WrapErrServiceInternalMsg("compaction plan is illegal, there's no segments in compaction plan, planID = %d", t.GetPlanID())
}
if t.plan.GetMaxSize() == 0 {
return errors.Newf("compaction plan is illegal, empty maxSize, planID = %d", t.GetPlanID())
return merr.WrapErrServiceInternalMsg("compaction plan is illegal, empty maxSize, planID = %d", t.GetPlanID())
}
t.collectionID = t.plan.GetSegmentBinlogs()[0].GetCollectionID()
@@ -440,7 +442,7 @@ func (t *mixCompactionTask) Compact() (*datapb.CompactionPlanResult, error) {
if isEmpty {
log.Warn("compact wrong, all segments' binlogs are empty")
return nil, errors.New("illegal compaction plan")
return nil, merr.WrapErrServiceInternalMsg("illegal compaction plan")
}
sortMergeAppicable := t.compactionParams.UseMergeSort
@@ -17,12 +17,9 @@
package compactor
import (
"fmt"
"github.com/apache/arrow/go/v17/arrow"
"github.com/apache/arrow/go/v17/arrow/array"
"github.com/apache/arrow/go/v17/arrow/memory"
"github.com/cockroachdb/errors"
"github.com/milvus-io/milvus-proto/go-api/v3/schemapb"
"github.com/milvus-io/milvus/internal/storage"
@@ -79,7 +76,7 @@ func NewRecordMaterializer(schema *schemapb.CollectionSchema, functions []*schem
}
if runner == nil {
materializer.Close()
return nil, errors.Newf("failed to set up function runner for %s", functionSchema.GetName())
return nil, merr.WrapErrFunctionFailedMsg("failed to set up function runner for %s", functionSchema.GetName())
}
functionMaterializer, err := newFunctionMaterializer(schema, runner, outputIndexes, true)
if err != nil {
@@ -243,7 +240,7 @@ func (r *selectedRecord) Column(fieldID storage.FieldID) arrow.Array {
defer selected.Release()
col := selected.Column(fieldID)
if col == nil {
r.err = errors.Newf("selected record field %d not found", fieldID)
r.err = merr.WrapErrServiceInternalMsg("selected record field %d not found", fieldID)
return nil
}
col.Retain()
@@ -347,7 +344,7 @@ func newFunctionMaterializer(schema *schemapb.CollectionSchema, runner function.
case schemapb.FunctionType_MinHash:
return newMinHashFunctionMaterializer(schema, runner, missingOutputIndexes, ownRunner)
default:
return nil, errors.Newf("unsupported function type %s", functionSchema.GetType().String())
return nil, merr.WrapErrParameterInvalidMsg("unsupported function type %s", functionSchema.GetType().String())
}
}
@@ -355,35 +352,35 @@ func newMinHashFunctionMaterializer(schema *schemapb.CollectionSchema, runner fu
functionSchema := runner.GetSchema()
inputFields := runner.GetInputFields()
if len(inputFields) == 0 {
return nil, errors.New("minhash function should have input fields")
return nil, merr.WrapErrFunctionFailedMsg("minhash function should have input fields")
}
inputFieldIDs := make([]int64, 0, len(inputFields))
for _, inputField := range inputFields {
if inputField == nil || typeutil.GetField(schema, inputField.GetFieldID()) == nil {
return nil, errors.New("input field not found in schema")
return nil, merr.WrapErrFunctionFailedMsg("input field not found in schema")
}
if inputField.GetDataType() != schemapb.DataType_VarChar && inputField.GetDataType() != schemapb.DataType_Text {
return nil, errors.New("input field data type must be varchar or text for minhash function materialization")
return nil, merr.WrapErrFunctionFailedMsg("input field data type must be varchar or text for minhash function materialization")
}
inputFieldIDs = append(inputFieldIDs, inputField.GetFieldID())
}
outputFieldIDs := functionSchema.GetOutputFieldIds()
if len(outputFieldIDs) == 0 {
return nil, errors.New("minhash function should have output fields")
return nil, merr.WrapErrFunctionFailedMsg("minhash function should have output fields")
}
outputFields := make(map[int64]*schemapb.FieldSchema, len(outputFieldIDs))
for _, outputFieldID := range outputFieldIDs {
outputField := typeutil.GetField(schema, outputFieldID)
if outputField == nil {
return nil, errors.New("output field not found in schema")
return nil, merr.WrapErrFunctionFailedMsg("output field not found in schema")
}
if outputField.GetDataType() != schemapb.DataType_BinaryVector {
return nil, errors.New("output field data type must be binary vector for minhash function materialization")
return nil, merr.WrapErrFunctionFailedMsg("output field data type must be binary vector for minhash function materialization")
}
if outputField.GetNullable() {
return nil, errors.Newf("function output field cannot be nullable: function %s, field %s", functionSchema.GetName(), outputField.GetName())
return nil, merr.WrapErrFunctionFailedMsg("function output field cannot be nullable: function %s, field %s", functionSchema.GetName(), outputField.GetName())
}
outputFields[outputFieldID] = outputField
}
@@ -402,35 +399,35 @@ func newBM25FunctionMaterializer(schema *schemapb.CollectionSchema, runner funct
functionSchema := runner.GetSchema()
inputFields := runner.GetInputFields()
if len(inputFields) == 0 {
return nil, errors.New("bm25 function should have input fields")
return nil, merr.WrapErrParameterInvalidMsg("bm25 function should have input fields")
}
inputFieldIDs := make([]int64, 0, len(inputFields))
for _, inputField := range inputFields {
if inputField == nil || typeutil.GetField(schema, inputField.GetFieldID()) == nil {
return nil, errors.New("input field not found in schema")
return nil, merr.WrapErrParameterInvalidMsg("input field not found in schema")
}
if inputField.GetDataType() != schemapb.DataType_VarChar && inputField.GetDataType() != schemapb.DataType_Text {
return nil, errors.New("input field data type must be varchar or text for bm25 function materialization")
return nil, merr.WrapErrParameterInvalidMsg("input field data type must be varchar or text for bm25 function materialization")
}
inputFieldIDs = append(inputFieldIDs, inputField.GetFieldID())
}
outputFieldIDs := functionSchema.GetOutputFieldIds()
if len(outputFieldIDs) == 0 {
return nil, errors.New("bm25 function should have output fields")
return nil, merr.WrapErrParameterInvalidMsg("bm25 function should have output fields")
}
outputFields := make(map[int64]*schemapb.FieldSchema, len(outputFieldIDs))
for _, outputFieldID := range outputFieldIDs {
outputField := typeutil.GetField(schema, outputFieldID)
if outputField == nil {
return nil, errors.New("output field not found in schema")
return nil, merr.WrapErrParameterInvalidMsg("output field not found in schema")
}
if outputField.GetDataType() != schemapb.DataType_SparseFloatVector {
return nil, errors.New("output field data type must be sparse float vector for bm25 function materialization")
return nil, merr.WrapErrParameterInvalidMsg("output field data type must be sparse float vector for bm25 function materialization")
}
if outputField.GetNullable() {
return nil, errors.Newf("function output field cannot be nullable: function %s, field %s", functionSchema.GetName(), outputField.GetName())
return nil, merr.WrapErrParameterInvalidMsg("function output field cannot be nullable: function %s, field %s", functionSchema.GetName(), outputField.GetName())
}
outputFields[outputFieldID] = outputField
}
@@ -459,7 +456,7 @@ func (m *bm25FunctionMaterializer) Materialize(rec storage.Record) (map[int64]ar
return nil, err
}
if len(outputs) != len(m.outputFieldIDs) {
return nil, errors.Newf("bm25 function materialization expects %d outputs, got %d", len(m.outputFieldIDs), len(outputs))
return nil, merr.WrapErrFunctionFailedMsg("bm25 function materialization expects %d outputs, got %d", len(m.outputFieldIDs), len(outputs))
}
result := make(map[int64]arrow.Array, len(m.missingOutputIndexes))
@@ -468,7 +465,7 @@ func (m *bm25FunctionMaterializer) Materialize(rec storage.Record) (map[int64]ar
outputSparseArray, ok := outputs[outputIndex].(*schemapb.SparseFloatArray)
if !ok {
releaseArrowArrays(result)
return nil, errors.Newf("unexpected output type from BM25 function runner, expected SparseFloatArray, got %T", outputs[outputIndex])
return nil, merr.WrapErrFunctionFailedMsg("unexpected output type from BM25 function runner, expected SparseFloatArray, got %T", outputs[outputIndex])
}
arr, err := buildSparseFloatVectorArrowArray(m.outputFields[outputFieldID], outputSparseArray, rec.Len())
if err != nil {
@@ -500,7 +497,7 @@ func (m *minHashFunctionMaterializer) Materialize(rec storage.Record) (map[int64
return nil, err
}
if len(outputs) != len(m.outputFieldIDs) {
return nil, errors.Newf("minhash function materialization expects %d outputs, got %d", len(m.outputFieldIDs), len(outputs))
return nil, merr.WrapErrFunctionFailedMsg("minhash function materialization expects %d outputs, got %d", len(m.outputFieldIDs), len(outputs))
}
result := make(map[int64]arrow.Array, len(m.missingOutputIndexes))
@@ -509,12 +506,12 @@ func (m *minHashFunctionMaterializer) Materialize(rec storage.Record) (map[int64
outputFieldData, ok := outputs[outputIndex].(*schemapb.FieldData)
if !ok {
releaseArrowArrays(result)
return nil, errors.Newf("unexpected output type from MinHash function runner, expected FieldData, got %T", outputs[outputIndex])
return nil, merr.WrapErrFunctionFailedMsg("unexpected output type from MinHash function runner, expected FieldData, got %T", outputs[outputIndex])
}
vectorField := outputFieldData.GetVectors()
if vectorField == nil || vectorField.GetBinaryVector() == nil {
releaseArrowArrays(result)
return nil, errors.New("unexpected output from MinHash function runner, expected binary vector field data")
return nil, merr.WrapErrFunctionFailedMsg("unexpected output from MinHash function runner, expected binary vector field data")
}
fieldData := &storage.BinaryVectorFieldData{
Data: vectorField.GetBinaryVector(),
@@ -522,7 +519,7 @@ func (m *minHashFunctionMaterializer) Materialize(rec storage.Record) (map[int64
}
if fieldData.RowNum() != rec.Len() {
releaseArrowArrays(result)
return nil, errors.Newf("minhash function output row count mismatch, expected %d, got %d", rec.Len(), fieldData.RowNum())
return nil, merr.WrapErrFunctionFailedMsg("minhash function output row count mismatch, expected %d, got %d", rec.Len(), fieldData.RowNum())
}
arr, err := buildArrowArrayFromFieldData(m.outputFields[outputFieldID], fieldData, rec.Len())
if err != nil {
@@ -577,7 +574,7 @@ func missingNonMaterializedSchemaFields(schema *schemapb.CollectionSchema, exist
func stringInputsFromRecord(rec storage.Record, fieldID int64) ([]string, error) {
col := rec.Column(fieldID)
if col == nil {
return nil, merr.WrapErrServiceInternal(fmt.Sprintf("input field %d not found in record", fieldID))
return nil, merr.WrapErrFunctionFailedMsg("input field %d not found in record", fieldID)
}
inputs := make([]string, rec.Len())
switch values := col.(type) {
@@ -588,16 +585,16 @@ func stringInputsFromRecord(rec storage.Record, fieldID int64) ([]string, error)
}
}
case *array.Binary:
return nil, merr.WrapErrServiceInternal("cannot materialize bm25 from text binary values without lob decoding")
return nil, merr.WrapErrFunctionFailedMsg("cannot materialize bm25 from text binary values without lob decoding")
default:
return nil, merr.WrapErrServiceInternal(fmt.Sprintf("input field %d data type must be varchar or text for bm25 function materialization, got %T", fieldID, col))
return nil, merr.WrapErrFunctionFailedMsg("input field %d data type must be varchar or text for bm25 function materialization, got %T", fieldID, col)
}
return inputs, nil
}
func buildSparseFloatVectorArrowArray(field *schemapb.FieldSchema, outputSparseArray *schemapb.SparseFloatArray, rowCount int) (arrow.Array, error) {
if len(outputSparseArray.GetContents()) != rowCount {
return nil, errors.Newf("bm25 function output row count mismatch, expected %d, got %d", rowCount, len(outputSparseArray.GetContents()))
return nil, merr.WrapErrFunctionFailedMsg("bm25 function output row count mismatch, expected %d, got %d", rowCount, len(outputSparseArray.GetContents()))
}
fieldData := &storage.SparseFloatVectorFieldData{
@@ -612,7 +609,7 @@ func buildSparseFloatVectorArrowArray(field *schemapb.FieldSchema, outputSparseA
func buildArrowArrayFromFieldData(field *schemapb.FieldSchema, fieldData storage.FieldData, rowCount int) (arrow.Array, error) {
if fieldData.RowNum() != rowCount {
return nil, errors.Newf("function output row count mismatch for field %d, expected %d, got %d", field.GetFieldID(), rowCount, fieldData.RowNum())
return nil, merr.WrapErrFunctionFailedMsg("function output row count mismatch for field %d, expected %d, got %d", field.GetFieldID(), rowCount, fieldData.RowNum())
}
outputSchema := &schemapb.CollectionSchema{Fields: []*schemapb.FieldSchema{field}}
@@ -22,7 +22,6 @@ import (
"math"
"github.com/apache/arrow/go/v17/arrow/array"
"github.com/cockroachdb/errors"
"github.com/samber/lo"
"go.uber.org/atomic"
"go.uber.org/zap"
@@ -37,6 +36,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/log"
"github.com/milvus-io/milvus/pkg/v3/proto/datapb"
"github.com/milvus-io/milvus/pkg/v3/proto/etcdpb"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
"github.com/milvus-io/milvus/pkg/v3/util/typeutil"
)
@@ -318,7 +318,7 @@ func (w *SegmentWriter) WriteRecord(r storage.Record) error {
for fieldID, stats := range w.bm25Stats {
field, ok := r.Column(fieldID).(*array.Binary)
if !ok {
return errors.New("bm25 field value not found")
return merr.WrapErrServiceInternalMsg("bm25 field value not found")
}
stats.AppendBytes(field.Value(i))
}
@@ -341,12 +341,12 @@ func (w *SegmentWriter) Write(v *storage.Value) error {
for fieldID, stats := range w.bm25Stats {
data, ok := v.Value.(map[storage.FieldID]interface{})[fieldID]
if !ok {
return errors.New("bm25 field value not found")
return merr.WrapErrServiceInternalMsg("bm25 field value not found")
}
bytes, ok := data.([]byte)
if !ok {
return errors.New("bm25 field value not sparse bytes")
return merr.WrapErrServiceInternalMsg("bm25 field value not sparse bytes")
}
stats.AppendBytes(bytes)
}
@@ -23,7 +23,6 @@ import (
"time"
"github.com/apache/arrow/go/v17/arrow/array"
"github.com/cockroachdb/errors"
"github.com/samber/lo"
"go.opentelemetry.io/otel"
"go.uber.org/zap"
@@ -109,7 +108,9 @@ func (t *sortCompactionTask) preCompact() error {
}
if len(t.plan.GetSegmentBinlogs()) != 1 {
return errors.Newf("sort compaction should handle exactly one segment, but got %d segments, planID = %d",
// The plan is produced by datacoord, so a malformed plan is an internal
// protocol violation, not user input.
return merr.WrapErrServiceInternalMsg("sort compaction should handle exactly one segment, but got %d segments, planID = %d",
len(t.plan.GetSegmentBinlogs()), t.GetPlanID())
}
+3 -3
View File
@@ -27,7 +27,6 @@ import (
"sync"
"time"
"github.com/cockroachdb/errors"
"github.com/tidwall/gjson"
clientv3 "go.etcd.io/etcd/client/v3"
"go.uber.org/zap"
@@ -49,6 +48,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/util/conc"
"github.com/milvus-io/milvus/pkg/v3/util/expr"
"github.com/milvus-io/milvus/pkg/v3/util/lifetime"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/metricsinfo"
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
"github.com/milvus-io/milvus/pkg/v3/util/typeutil"
@@ -166,7 +166,7 @@ func (node *DataNode) Register() error {
func (node *DataNode) initSession() error {
node.session = sessionutil.NewSession(node.ctx)
if node.session == nil {
return errors.New("failed to initialize session")
return merr.WrapErrServiceInternalMsg("failed to initialize session")
}
node.session.Init(typeutil.DataNodeRole, node.address, false)
sessionutil.SaveServerInfo(typeutil.DataNodeRole, node.session.ServerID)
@@ -282,7 +282,7 @@ func (node *DataNode) isHealthy() bool {
// ReadyToFlush tells whether DataNode is ready for flushing
func (node *DataNode) ReadyToFlush() error {
if !node.isHealthy() {
return errors.New("DataNode not in HEALTHY state")
return merr.Wrap(merr.ErrServiceNotReady, "DataNode not in HEALTHY state")
}
return nil
}
+25 -24
View File
@@ -20,6 +20,7 @@ import (
"github.com/milvus-io/milvus/internal/util/function/embedding"
"github.com/milvus-io/milvus/pkg/v3/log"
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/typeutil"
)
@@ -65,11 +66,11 @@ func ExecuteFunctionsForSegment(
inputManifestPath, err := packed.CreateSegmentManifestWithBasePath(
ctx, basePath, format, sourceColumns, fragments, storageConfig)
if err != nil {
return "", fmt.Errorf("create input manifest: %w", err)
return "", merr.Wrap(err, "create input manifest")
}
_, inputVersion, err := packed.UnmarshalManifestPath(inputManifestPath)
if err != nil {
return "", fmt.Errorf("parse input manifest path: %w", err)
return "", merr.Wrap(err, "parse input manifest path")
}
outputFields, outputSchema, err := buildOutputSchema(schema)
@@ -94,7 +95,7 @@ func ExecuteFunctionsForSegment(
colGroups := []storagecommon.ColumnGroup{{Columns: lo.Range(len(outputFields))}}
writer, err := packed.NewFFIPackedWriter(basePath, outputArrow, colGroups, storageConfig, nil)
if err != nil {
return "", fmt.Errorf("open output writer: %w", err)
return "", merr.Wrap(err, "open output writer")
}
writer.AsNewColumnGroups()
@@ -108,7 +109,7 @@ func ExecuteFunctionsForSegment(
output, err := writer.Close()
if err != nil {
return "", fmt.Errorf("close output writer: %w", err)
return "", merr.Wrap(err, "close output writer")
}
if output != nil {
defer output.Destroy()
@@ -120,7 +121,7 @@ func ExecuteFunctionsForSegment(
}
manifestPath, err := packed.CommitManifestUpdates(basePath, inputVersion, storageConfig, updates)
if err != nil {
return "", fmt.Errorf("commit function output manifest: %w", err)
return "", merr.Wrap(err, "commit function output manifest")
}
log.Info("function execution completed",
@@ -139,7 +140,7 @@ func buildOutputSchema(schema *schemapb.CollectionSchema) ([]*schemapb.FieldSche
return nil, nil, err
}
if len(outputFields) == 0 {
return nil, nil, fmt.Errorf("no function output fields; executor should not have been invoked")
return nil, nil, merr.WrapErrServiceInternalMsg("no function output fields; executor should not have been invoked")
}
return outputFields, &schemapb.CollectionSchema{
Name: schema.GetName(),
@@ -177,7 +178,7 @@ func functionOutputFields(schema *schemapb.CollectionSchema) ([]*schemapb.FieldS
continue
}
if _, ok := fieldsByID[fieldID]; !ok {
return nil, fmt.Errorf("function output field id %d not found in schema", fieldID)
return nil, merr.WrapErrParameterInvalidMsg("function output field id %d not found in schema", fieldID)
}
addOutputID(fieldID)
}
@@ -187,7 +188,7 @@ func functionOutputFields(schema *schemapb.CollectionSchema) ([]*schemapb.FieldS
}
field, ok := fieldsByName[fieldName]
if !ok {
return nil, fmt.Errorf("function output field %s not found in schema", fieldName)
return nil, merr.WrapErrParameterInvalidMsg("function output field %s not found in schema", fieldName)
}
addOutputID(field.GetFieldID())
}
@@ -222,7 +223,7 @@ func openInputReader(
}),
)
if err != nil {
return nil, fmt.Errorf("open input manifest: %w", err)
return nil, merr.Wrap(err, "open input manifest")
}
return reader, nil
}
@@ -236,7 +237,7 @@ func buildFunctionExecutionSchema(
schema *schemapb.CollectionSchema,
) (*schemapb.CollectionSchema, *schemapb.CollectionSchema, typeutil.Set[int64], error) {
if schema == nil {
return nil, nil, nil, fmt.Errorf("collection schema is nil")
return nil, nil, nil, merr.WrapErrParameterInvalidMsg("collection schema is nil")
}
inputIDs := make(map[int64]struct{})
for _, fn := range schema.GetFunctions() {
@@ -291,24 +292,24 @@ func buildFunctionExecutionSchema(
inputSchema.Fields = append(inputSchema.Fields, f)
if schema.GetExternalSource() != "" && f.GetExternalField() == "" {
return nil, nil, nil, fmt.Errorf("function input field %s has no external_field", f.GetName())
return nil, nil, nil, merr.WrapErrParameterInvalidMsg("function input field %s has no external_field", f.GetName())
}
}
for inputID := range inputIDs {
if _, ok := seenInputFields[inputID]; !ok {
if _, ok := seenExecutionFields[inputID]; !ok {
return nil, nil, nil, fmt.Errorf("function input field id %d not found in schema", inputID)
return nil, nil, nil, merr.WrapErrParameterInvalidMsg("function input field id %d not found in schema", inputID)
}
}
}
for outputID := range outputIDs {
if _, ok := seenExecutionFields[outputID]; !ok {
return nil, nil, nil, fmt.Errorf("function output field id %d not found in schema", outputID)
return nil, nil, nil, merr.WrapErrParameterInvalidMsg("function output field id %d not found in schema", outputID)
}
}
if len(inputSchema.GetFields()) == 0 {
return nil, nil, nil, fmt.Errorf("no source input columns for function execution")
return nil, nil, nil, merr.WrapErrParameterInvalidMsg("no source input columns for function execution")
}
return inputSchema, executionSchema, requiredInputFields, nil
}
@@ -332,7 +333,7 @@ func streamBatches(
break
}
if err != nil {
return totalRows, fmt.Errorf("read input batch: %w", err)
return totalRows, merr.Wrap(err, "read input batch")
}
if rec == nil {
break
@@ -341,7 +342,7 @@ func streamBatches(
batch, err := storage.RecordToInsertData(rec, executionSchema, requiredInputFields)
rec.Release()
if err != nil {
return totalRows, fmt.Errorf("record to InsertData: %w", err)
return totalRows, merr.Wrap(err, "record to InsertData")
}
if batch.GetRowNum() == 0 {
continue
@@ -351,7 +352,7 @@ func streamBatches(
ClusterID: clusterID,
DBName: schema.GetDbName(),
}); err != nil {
return totalRows, fmt.Errorf("execute functions: %w", err)
return totalRows, merr.Wrap(err, "execute functions")
}
if err := accumulateBM25Stats(batch, bm25Acc); err != nil {
@@ -359,7 +360,7 @@ func streamBatches(
}
if err := writeOutputBatch(batch, outputSchema, outputArrow, writer); err != nil {
return totalRows, fmt.Errorf("write output batch: %w", err)
return totalRows, merr.Wrap(err, "write output batch")
}
totalRows += int64(batch.GetRowNum())
}
@@ -404,13 +405,13 @@ func accumulateBM25Stats(batch *storage.InsertData, acc map[int64]*storage.BM25S
for outID, stats := range acc {
raw, present := batch.Data[outID]
if !present || raw == nil {
return fmt.Errorf(
return merr.WrapErrFunctionFailedMsg(
"BM25 output field %d missing from batch; executeFunctions did not populate it",
outID)
}
fd, ok := raw.(*storage.SparseFloatVectorFieldData)
if !ok {
return fmt.Errorf(
return merr.WrapErrFunctionFailedMsg(
"BM25 output field %d has wrong type %T (want *SparseFloatVectorFieldData)",
outID, raw)
}
@@ -431,18 +432,18 @@ func appendBM25Stats(
}
log := log.Ctx(ctx)
if updates == nil {
return fmt.Errorf("manifest updates is nil")
return merr.WrapErrServiceInternalMsg("manifest updates is nil")
}
entries := make([]packed.StatEntry, 0, len(acc))
for outID, stats := range acc {
blob, err := stats.Serialize()
if err != nil {
return fmt.Errorf("serialize bm25 stats for field %d: %w", outID, err)
return merr.Wrapf(err, "serialize bm25 stats for field %d", outID)
}
fullPath := path.Join(basePath, fmt.Sprintf("_stats/bm25.%d/%d", outID, 0))
if err := packed.WriteFile(storageConfig, fullPath, blob); err != nil {
return fmt.Errorf("write bm25 stats file %s: %w", fullPath, err)
return merr.Wrapf(err, "write bm25 stats file %s", fullPath)
}
entries = append(entries, packed.StatEntry{
Key: fmt.Sprintf("bm25.%d", outID),
@@ -481,7 +482,7 @@ func finalizeBM25Stats(
basePath, version, err := packed.UnmarshalManifestPath(manifestPath)
if err != nil {
return "", fmt.Errorf("parse manifest path: %w", err)
return "", merr.Wrap(err, "parse manifest path")
}
updates := &packed.ManifestUpdates{}
+3 -1
View File
@@ -29,6 +29,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/proto/datapb"
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
"github.com/milvus-io/milvus/pkg/v3/util/conc"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
)
// TaskKey uniquely identifies an external collection task.
@@ -259,7 +260,8 @@ func (m *ExternalCollectionManager) SubmitTask(
zap.ByteString("stack", stack))
reason := fmt.Sprintf("task panicked: %v", r)
m.UpdateResult(clusterID, taskID, indexpb.JobState_JobStateFailed, reason, info.KeptSegments, nil)
retErr = fmt.Errorf("%s", reason)
// A recovered panic is a server-side failure, never caller input.
retErr = merr.WrapErrServiceInternalMsg("%s", reason)
}
}()
log.Info("executing external collection task in pool",
+19 -19
View File
@@ -164,22 +164,22 @@ func (t *RefreshExternalCollectionTask) PreExecute(ctx context.Context) error {
zap.Int64("collectionID", t.req.GetCollectionID()))
if t.req == nil {
return fmt.Errorf("request is nil")
return merr.WrapErrParameterInvalidMsg("request is nil")
}
if t.req.GetSchema() == nil {
return fmt.Errorf("schema is nil in request")
return merr.WrapErrParameterInvalidMsg("schema is nil in request")
}
if t.req.GetStorageConfig() == nil {
return fmt.Errorf("storage config is nil in request")
return merr.WrapErrParameterInvalidMsg("storage config is nil in request")
}
if t.req.GetExternalSource() == "" {
return fmt.Errorf("external source is empty in request")
return merr.WrapErrParameterInvalidMsg("external source is empty in request")
}
// Parse and cache external spec for reuse during Execute
spec, err := externalspec.ParseExternalSpec(t.req.GetExternalSpec())
if err != nil {
return fmt.Errorf("failed to parse external spec: %w", err)
return merr.Wrap(err, "failed to parse external spec")
}
t.parsedSpec = spec
t.columns = packed.GetColumnNamesFromSchema(t.req.GetSchema())
@@ -198,7 +198,7 @@ func (t *RefreshExternalCollectionTask) Execute(ctx context.Context) error {
// Initialize pre-allocated segment IDs from request
if t.req.GetPreAllocatedSegmentIds() == nil {
return fmt.Errorf("pre-allocated segment IDs not provided in request")
return merr.WrapErrParameterInvalidMsg("pre-allocated segment IDs not provided in request")
}
t.preallocatedIDRange = t.req.GetPreAllocatedSegmentIds()
@@ -212,13 +212,13 @@ func (t *RefreshExternalCollectionTask) Execute(ctx context.Context) error {
// Fetch fragments from external source
newFragments, err := t.fetchFragmentsFromExternalSource(ctx)
if err != nil {
return fmt.Errorf("failed to fetch fragments: %w", err)
return merr.Wrap(err, "failed to fetch fragments")
}
// Build current segment -> fragments mapping
currentSegmentFragments, err := t.buildCurrentSegmentFragments()
if err != nil {
return fmt.Errorf("failed to build current segment fragments: %w", err)
return merr.Wrap(err, "failed to build current segment fragments")
}
// Compare and organize segments
@@ -235,7 +235,7 @@ func (t *RefreshExternalCollectionTask) Execute(ctx context.Context) error {
func (t *RefreshExternalCollectionTask) fetchFragmentsFromExternalSource(ctx context.Context) ([]packed.Fragment, error) {
manifestPath := t.req.GetExploreManifestPath()
if manifestPath == "" {
return nil, fmt.Errorf("explore manifest path is required but not provided")
return nil, merr.WrapErrParameterMissingMsg("explore manifest path is required but not provided")
}
log.Ctx(ctx).Info("reading file list from explore manifest",
@@ -329,7 +329,7 @@ func (t *RefreshExternalCollectionTask) organizeSegments(
var err error
outputColumns, err = functionOutputColumnNames(t.req.GetSchema())
if err != nil {
return nil, fmt.Errorf("resolve function output columns: %w", err)
return nil, merr.Wrap(err, "resolve function output columns")
}
}
@@ -455,7 +455,7 @@ func (t *RefreshExternalCollectionTask) segmentHasFunctionOutputColumns(seg *dat
}
hasColumns, err := packed.ManifestHasColumns(seg.GetManifestPath(), t.req.GetStorageConfig(), outputColumns)
if err != nil {
return false, fmt.Errorf("check function output columns for segment %d: %w", seg.GetID(), err)
return false, merr.Wrapf(err, "check function output columns for segment %d", seg.GetID())
}
return hasColumns, nil
}
@@ -557,7 +557,7 @@ func (t *RefreshExternalCollectionTask) patchSegmentForMissingColumns(
t.req.GetStorageConfig(),
)
if err != nil {
return nil, fmt.Errorf("failed to append manifest columns for segment %d: %w", seg.GetID(), err)
return nil, merr.Wrapf(err, "failed to append manifest columns for segment %d", seg.GetID())
}
sampleRows := paramtable.Get().QueryNodeCfg.ExternalCollectionSampleRows.GetAsInt()
@@ -574,7 +574,7 @@ func (t *RefreshExternalCollectionTask) patchSegmentForMissingColumns(
t.req.GetStorageConfig(),
)
if err != nil {
return nil, fmt.Errorf("failed to sample external field sizes for segment %d: %w", seg.GetID(), err)
return nil, merr.Wrapf(err, "failed to sample external field sizes for segment %d", seg.GetID())
}
externalAvgBytes := sumFieldSizes(fieldSizes, schema)
if externalAvgBytes <= 0 {
@@ -690,7 +690,7 @@ func (t *RefreshExternalCollectionTask) balanceFragmentsToSegments(ctx context.C
}
// Each segment needs 2 IDs: one for segment, one for fake binlog logID
if t.nextAllocID+1 >= t.preallocatedIDRange.End {
return nil, fmt.Errorf("insufficient pre-allocated IDs: need 2 more but only have %d IDs in range [%d, %d)",
return nil, merr.WrapErrParameterInvalidMsg("insufficient pre-allocated IDs: need 2 more but only have %d IDs in range [%d, %d)",
t.preallocatedIDRange.End-t.nextAllocID,
t.preallocatedIDRange.Begin,
t.preallocatedIDRange.End)
@@ -739,7 +739,7 @@ func (t *RefreshExternalCollectionTask) balanceFragmentsToSegments(ctx context.C
manifestPath, err = t.createManifestForSegment(ctx, work.segmentID, work.fragments)
}
if err != nil {
return "", fmt.Errorf("failed to create manifest for segment %d: %w", work.segmentID, err)
return "", merr.Wrapf(err, "failed to create manifest for segment %d", work.segmentID)
}
return manifestPath, nil
})
@@ -834,7 +834,7 @@ func (t *RefreshExternalCollectionTask) balanceFragmentsToSegments(ctx context.C
log.Warn("external field size sample produced non-positive total",
zap.String("manifestPath", manifestPath),
zap.Int64("total", total))
recordErr(fmt.Errorf("sampled field sizes sum to %d (schema may have no external_field mappings, or sampled rows are empty)", total))
recordErr(merr.WrapErrParameterInvalidMsg("sampled field sizes sum to %d (schema may have no external_field mappings, or sampled rows are empty)", total))
return 0, false
}
return total, true
@@ -882,9 +882,9 @@ func (t *RefreshExternalCollectionTask) balanceFragmentsToSegments(ctx context.C
if strings.Contains(rootCause, "not found in schema") {
hint = "; check external_field mappings in collection schema against actual parquet columns"
}
return nil, merr.WrapErrParameterInvalidMsg(fmt.Sprintf(
return nil, merr.WrapErrParameterInvalidMsg(
"external field size sampling failed for all %d segment(s): %s%s",
len(manifestPaths), rootCause, hint))
len(manifestPaths), rootCause, hint)
}
// Fill any zero slots (sampling failed mid-loop) with the first
// successful average so every segment gets a non-zero MemorySize.
@@ -1032,7 +1032,7 @@ func estimateFunctionOutputBytesPerRow(schema *schemapb.CollectionSchema) (int64
Fields: []*schemapb.FieldSchema{field},
})
if err != nil {
return 0, fmt.Errorf("estimate function output field %s: %w", field.GetName(), err)
return 0, merr.Wrapf(err, "estimate function output field %s", field.GetName())
}
total += int64(size)
}
@@ -18,7 +18,6 @@ package importv2
import (
"context"
"fmt"
"path"
"strconv"
"strings"
@@ -34,6 +33,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/log"
"github.com/milvus-io/milvus/pkg/v3/proto/datapb"
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/metautil"
)
@@ -114,12 +114,12 @@ func transformManifestPath(
) (string, error) {
basePath, version, err := packed.UnmarshalManifestPath(manifestPath)
if err != nil {
return "", fmt.Errorf("failed to unmarshal manifest path: %w", err)
return "", merr.Wrap(err, "failed to unmarshal manifest path")
}
targetBasePath, err := generateTargetPath(basePath, source, target)
if err != nil {
return "", fmt.Errorf("failed to generate target base path: %w", err)
return "", merr.Wrap(err, "failed to generate target base path")
}
targetManifestPath := packed.MarshalManifestPath(targetBasePath, version)
@@ -227,18 +227,18 @@ func collectSegmentFiles(
// StorageV3+: binlog paths MUST come from manifest
manifestPath := source.GetManifestPath()
if manifestPath == "" {
return nil, fmt.Errorf("storage_version=%d requires manifest_path but it is empty (segmentID=%d)",
return nil, merr.WrapErrParameterInvalidMsg("storage_version=%d requires manifest_path but it is empty (segmentID=%d)",
source.GetStorageVersion(), source.GetSegmentId())
}
basePath, _, err := packed.UnmarshalManifestPath(manifestPath)
if err != nil {
return nil, fmt.Errorf("failed to unmarshal manifest path %q for segment %d: %w", manifestPath, source.GetSegmentId(), err)
return nil, merr.Wrapf(err, "failed to unmarshal manifest path %q for segment %d", manifestPath, source.GetSegmentId())
}
allFiles, listErr := listAllFiles(ctx, cm, basePath)
if listErr != nil {
return nil, fmt.Errorf("failed to list files from manifest base path %q for segment %d: %w", basePath, source.GetSegmentId(), listErr)
return nil, merr.Wrapf(listErr, "failed to list files from manifest base path %q for segment %d", basePath, source.GetSegmentId())
}
// Empty file list is OK for V3 — segment may have only deltas and no insert binlogs
@@ -319,7 +319,7 @@ func generateMappingsFromFiles(
}
if err != nil {
return fmt.Errorf("failed to generate target path for %s file %s: %w", fileType, srcPath, err)
return merr.Wrapf(err, "failed to generate target path for %s file %s", fileType, srcPath)
}
mappings[srcPath] = dstPath
}
@@ -379,13 +379,13 @@ func CopySegmentAndIndexFiles(
// Step 1: Collect all files to copy
files, err := collectSegmentFiles(ctx, cm, source)
if err != nil {
return nil, nil, fmt.Errorf("failed to collect segment files: %w", err)
return nil, nil, merr.Wrap(err, "failed to collect segment files")
}
// Step 2: Generate src->dst mappings for file copying
mappings, err := generateMappingsFromFiles(files, source, target)
if err != nil {
return nil, nil, fmt.Errorf("failed to generate file mappings: %w", err)
return nil, nil, merr.Wrap(err, "failed to generate file mappings")
}
// Step 3: Execute all copy operations
@@ -400,7 +400,7 @@ func CopySegmentAndIndexFiles(
fields = append(fields, logFields...)
fields = append(fields, zap.String("src", src), zap.String("dst", dst), zap.Error(err))
log.Warn("failed to copy file", fields...)
return nil, copiedFiles, fmt.Errorf("failed to copy file from %s to %s: %w", src, dst, err)
return nil, copiedFiles, merr.Wrapf(err, "failed to copy file from %s to %s", src, dst)
}
copiedFiles = append(copiedFiles, dst)
}
@@ -418,7 +418,7 @@ func CopySegmentAndIndexFiles(
if _, exists := mappings[srcPath]; !exists {
dstPath, pathErr := generateTargetPath(srcPath, source, target)
if pathErr != nil {
return nil, copiedFiles, fmt.Errorf("failed to generate target path for pb insert binlog %s: %w", srcPath, pathErr)
return nil, copiedFiles, merr.Wrapf(pathErr, "failed to generate target path for pb insert binlog %s", srcPath)
}
mappings[srcPath] = dstPath
}
@@ -430,20 +430,20 @@ func CopySegmentAndIndexFiles(
// Step 4: Build index metadata from source
indexInfos, textIndexInfos, jsonKeyIndexInfos, err := buildIndexInfoFromSource(source, target, mappings)
if err != nil {
return nil, copiedFiles, fmt.Errorf("failed to build index info: %w", err)
return nil, copiedFiles, merr.Wrap(err, "failed to build index info")
}
// Step 5: Generate segment metadata with path mappings
segmentInfo, err := generateSegmentInfoFromSource(source, target, mappings)
if err != nil {
return nil, copiedFiles, fmt.Errorf("failed to generate segment info: %w", err)
return nil, copiedFiles, merr.Wrap(err, "failed to generate segment info")
}
// Step 6: Compress paths
err = binlog.CompressBinLogs(segmentInfo.GetBinlogs(), segmentInfo.GetStatslogs(),
segmentInfo.GetDeltalogs(), segmentInfo.GetBm25Logs())
if err != nil {
return nil, copiedFiles, fmt.Errorf("failed to compress binlog paths: %w", err)
return nil, copiedFiles, merr.Wrap(err, "failed to compress binlog paths")
}
for _, indexInfo := range indexInfos {
@@ -474,7 +474,7 @@ func CopySegmentAndIndexFiles(
if useManifest {
targetManifestPath, err := transformManifestPath(source.GetManifestPath(), source, target)
if err != nil {
return nil, copiedFiles, fmt.Errorf("failed to transform manifest path: %w", err)
return nil, copiedFiles, merr.Wrap(err, "failed to transform manifest path")
}
result.ManifestPath = targetManifestPath
}
@@ -526,7 +526,7 @@ func transformFieldBinlogs(
}
dstPath, ok := mappings[srcPath]
if !ok {
return nil, 0, fmt.Errorf("no mapping found for source path: %s", srcPath)
return nil, 0, merr.WrapErrServiceInternalMsg("no mapping found for source path: %s", srcPath)
}
dstBinlog.LogPath = dstPath
}
@@ -583,7 +583,7 @@ func generateSegmentInfoFromSource(
// Process insert binlogs (count rows)
binlogs, totalRows, err := transformFieldBinlogs(source.GetInsertBinlogs(), mappings, true, source.GetIsExternalCollection())
if err != nil {
return nil, fmt.Errorf("failed to transform insert binlogs: %w", err)
return nil, merr.Wrap(err, "failed to transform insert binlogs")
}
segmentInfo.Binlogs = binlogs
segmentInfo.ImportedRows = totalRows
@@ -591,21 +591,21 @@ func generateSegmentInfoFromSource(
// Process stats binlogs (no row counting)
statslogs, _, err := transformFieldBinlogs(source.GetStatsBinlogs(), mappings, false, false)
if err != nil {
return nil, fmt.Errorf("failed to transform stats binlogs: %w", err)
return nil, merr.Wrap(err, "failed to transform stats binlogs")
}
segmentInfo.Statslogs = statslogs
// Process delta binlogs (no row counting)
deltalogs, _, err := transformFieldBinlogs(source.GetDeltaBinlogs(), mappings, false, false)
if err != nil {
return nil, fmt.Errorf("failed to transform delta binlogs: %w", err)
return nil, merr.Wrap(err, "failed to transform delta binlogs")
}
segmentInfo.Deltalogs = deltalogs
// Process BM25 binlogs (no row counting)
bm25logs, _, err := transformFieldBinlogs(source.GetBm25Binlogs(), mappings, false, false)
if err != nil {
return nil, fmt.Errorf("failed to transform BM25 binlogs: %w", err)
return nil, merr.Wrap(err, "failed to transform BM25 binlogs")
}
segmentInfo.Bm25Logs = bm25logs
@@ -635,7 +635,7 @@ func generateTargetPath(sourcePath string, source *datapb.CopySegmentSource, tar
}
if logTypeIndex == -1 || logTypeIndex+3 >= len(parts) {
return "", fmt.Errorf("invalid binlog path structure: %s (expected log_type at a valid position)", sourcePath)
return "", merr.WrapErrParameterInvalidMsg("invalid binlog path structure: %s (expected log_type at a valid position)", sourcePath)
}
// Replace IDs in order: collectionID, partitionID, segmentID
@@ -667,7 +667,7 @@ func generateTargetLOBPath(sourcePath string, source *datapb.CopySegmentSource,
// Path: .../{insert_log}/{coll}/{part}/lobs/...
// Need at least logTypeIndex + 2 (coll and part) after insert_log
if logTypeIndex == -1 || logTypeIndex+2 >= len(parts) {
return "", fmt.Errorf("invalid LOB path structure: %s", sourcePath)
return "", merr.WrapErrParameterInvalidMsg("invalid LOB path structure: %s", sourcePath)
}
parts[logTypeIndex+1] = strconv.FormatInt(target.GetCollectionId(), 10)
@@ -709,7 +709,7 @@ func buildIndexInfoFromSource(
for _, srcPath := range srcIndex.GetIndexFilePaths() {
targetPath, ok := mappings[srcPath]
if !ok {
return nil, nil, nil, fmt.Errorf("no mapping found for index file: %s", srcPath)
return nil, nil, nil, merr.WrapErrServiceInternalMsg("no mapping found for index file: %s", srcPath)
}
targetPaths = append(targetPaths, targetPath)
}
@@ -753,7 +753,7 @@ func buildIndexInfoFromSource(
for _, srcFile := range srcText.GetFiles() {
targetFile, ok := mappings[srcFile]
if !ok {
return nil, nil, nil, fmt.Errorf("no mapping found for text index file: %s", srcFile)
return nil, nil, nil, merr.WrapErrServiceInternalMsg("no mapping found for text index file: %s", srcFile)
}
targetFiles = append(targetFiles, targetFile)
}
@@ -785,7 +785,7 @@ func buildIndexInfoFromSource(
for _, srcFile := range srcJSON.GetFiles() {
targetFile, ok := mappings[srcFile]
if !ok {
return nil, nil, nil, fmt.Errorf("no mapping found for JSON index file: %s", srcFile)
return nil, nil, nil, merr.WrapErrServiceInternalMsg("no mapping found for JSON index file: %s", srcFile)
}
targetFiles = append(targetFiles, targetFile)
}
@@ -896,7 +896,7 @@ func generateTargetIndexPath(
}
if keywordIdx == -1 {
return "", fmt.Errorf("keyword '%s' not found in path: %s", keyword, sourcePath)
return "", merr.WrapErrServiceInternalMsg("keyword '%s' not found in path: %s", keyword, sourcePath)
}
// Set offsets based on index type
@@ -928,13 +928,13 @@ func generateTargetIndexPath(
segmentOffset = 6
buildIDOffset = 2
default:
return "", fmt.Errorf("unsupported index type: %s (expected '%s', '%s', '%s', or '%s')",
return "", merr.WrapErrParameterInvalidMsg("unsupported index type: %s (expected '%s', '%s', '%s', or '%s')",
indexType, IndexTypeVectorScalarV0, IndexTypeText, IndexTypeJSONKey, IndexTypeJSONStats)
}
// Validate path structure has enough components
if keywordIdx+segmentOffset >= len(parts) {
return "", fmt.Errorf("invalid %s path structure: %s (expected '%s' with at least %d components after it)",
return "", merr.WrapErrParameterInvalidMsg("invalid %s path structure: %s (expected '%s' with at least %d components after it)",
indexType, sourcePath, indexType, segmentOffset+1)
}
@@ -22,7 +22,6 @@ import (
"sync"
"time"
"github.com/cockroachdb/errors"
"go.uber.org/zap"
"github.com/milvus-io/milvus-proto/go-api/v3/schemapb"
@@ -30,6 +29,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/log"
"github.com/milvus-io/milvus/pkg/v3/proto/datapb"
"github.com/milvus-io/milvus/pkg/v3/util/conc"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/typeutil"
)
@@ -382,7 +382,7 @@ func (t *CopySegmentTask) copySingleSegment(source *datapb.CopySegmentSource, ta
reason := "no insert/delete binlogs for segment"
log.Error(reason, logFields...)
t.manager.Update(t.GetTaskID(), UpdateState(datapb.ImportTaskStateV2_Failed), UpdateReason(reason))
return nil, errors.New(reason)
return nil, merr.WrapErrParameterInvalidMsg(reason)
}
// Step 2: Copy all segment files (binlogs + indexes) together
+3 -2
View File
@@ -35,6 +35,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/proto/datapb"
"github.com/milvus-io/milvus/pkg/v3/proto/internalpb"
"github.com/milvus-io/milvus/pkg/v3/util/conc"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
"github.com/milvus-io/milvus/pkg/v3/util/typeutil"
)
@@ -192,9 +193,9 @@ func (t *PreImportTask) readFileStat(reader importutilv2.Reader, fileIdx int) er
}
maxSize := paramtable.Get().DataNodeCfg.MaxImportFileSizeInGB.GetAsFloat() * 1024 * 1024 * 1024
if fileSize > int64(maxSize) {
return errors.New(fmt.Sprintf(
return merr.WrapErrParameterInvalidMsg(
"The import file size has reached the maximum limit allowed for importing, "+
"fileSize=%d, maxSize=%d", fileSize, int64(maxSize)))
"fileSize=%d, maxSize=%d", fileSize, int64(maxSize))
}
totalRows := 0
+4 -4
View File
@@ -50,7 +50,7 @@ import (
)
func WrapTaskNotFoundError(taskID int64) error {
return merr.WrapErrImportFailed(fmt.Sprintf("cannot find import task with id %d", taskID))
return merr.WrapErrImportSysFailedMsg("cannot find import task with id %d", taskID)
}
func NewSyncTask(ctx context.Context,
@@ -151,7 +151,7 @@ func PickSegment(segments []*datapb.ImportRequestSegment, vchannel string, parti
})
if len(candidates) == 0 {
return 0, fmt.Errorf("no candidate segments found for channel %s and partition %d",
return 0, merr.WrapErrServiceInternalMsg("no candidate segments found for channel %s and partition %d",
vchannel, partitionID)
}
@@ -392,7 +392,7 @@ func AppendNullableDefaultFieldsData(schema *schemapb.CollectionSchema, data *st
}
}
default:
return fmt.Errorf("unexpected data type: %d, cannot be filled with default value", dataType)
return merr.WrapErrServiceInternalMsg("unexpected data type: %d, cannot be filled with default value", dataType)
}
if err != nil {
@@ -409,7 +409,7 @@ func FillDynamicData(schema *schemapb.CollectionSchema, data *storage.InsertData
}
dynamicField := typeutil.GetDynamicField(schema)
if dynamicField == nil {
return merr.WrapErrImportFailed("collection schema is illegal, enable_dynamic_field is true but the dynamic field doesn't exist")
return merr.WrapErrImportSysFailed("collection schema is illegal, enable_dynamic_field is true but the dynamic field doesn't exist")
}
tempData, ok := data.Data[dynamicField.GetFieldID()]
+1 -1
View File
@@ -80,7 +80,7 @@ func (queue *IndexTaskQueue) addUnissuedTask(t Task) error {
defer queue.utLock.Unlock()
if queue.utFull() {
return errors.New("index task queue is full")
return merr.Wrap(merr.ErrServiceResourceInsufficient, "index task queue is full")
}
queue.unissuedTasks.PushBack(t)
select {
+8 -7
View File
@@ -52,6 +52,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
"github.com/milvus-io/milvus/pkg/v3/proto/workerpb"
_ "github.com/milvus-io/milvus/pkg/v3/util/funcutil"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/metautil"
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
"github.com/milvus-io/milvus/pkg/v3/util/timerecord"
@@ -292,7 +293,7 @@ func (st *statsTask) sort(ctx context.Context) ([]*datapb.FieldBinlog, error) {
newManifest, err := packed.AddStatsToManifest(
st.manifestPath, st.req.GetStorageConfig(), statEntries)
if err != nil {
return nil, fmt.Errorf("failed to add stats to manifest: %w", err)
return nil, merr.Wrap(err, "failed to add stats to manifest")
}
st.manifestPath = newManifest
}
@@ -496,7 +497,7 @@ func (st *statsTask) createTextIndex(ctx context.Context,
}
binlogs, ok := fieldBinlogs[fieldID]
if !ok && !enableNull {
return nil, fmt.Errorf("field binlog not found for field %d", fieldID)
return nil, merr.WrapErrServiceInternalMsg("field binlog not found for field %d", fieldID)
}
result := make([]string, 0, len(binlogs))
for _, binlog := range binlogs {
@@ -618,7 +619,7 @@ func (st *statsTask) createTextIndex(ctx context.Context,
newManifest, err := packed.AddStatsToManifest(
st.manifestPath, st.req.GetStorageConfig(), statEntries)
if err != nil {
return fmt.Errorf("failed to add text index stats to manifest: %w", err)
return merr.Wrap(err, "failed to add text index stats to manifest")
}
st.manifestPath = newManifest
// Dual-write: keep textIndexLogs populated so it is stored in segment metadata as a
@@ -685,7 +686,7 @@ func (st *statsTask) createJSONKeyStats(ctx context.Context,
}
binlogs, ok := fieldBinlogs[fieldID]
if !ok && !enableNull {
return nil, fmt.Errorf("field binlog not found for field %d", fieldID)
return nil, merr.WrapErrServiceInternalMsg("field binlog not found for field %d", fieldID)
}
result := make([]string, 0, len(binlogs))
for _, binlog := range binlogs {
@@ -796,7 +797,7 @@ func (st *statsTask) createJSONKeyStats(ctx context.Context,
newManifest, err := packed.AddStatsToManifest(
st.manifestPath, st.req.GetStorageConfig(), statEntries)
if err != nil {
return fmt.Errorf("failed to add JSON key stats to manifest: %w", err)
return merr.Wrap(err, "failed to add JSON key stats to manifest")
}
st.manifestPath = newManifest
// Dual-write: keep jsonKeyIndexStats populated so it is stored in segment metadata as a
@@ -829,7 +830,7 @@ func computeStatsBasePath(req *workerpb.CreateStatsRequest, manifestPath string,
if req.GetStorageVersion() == storage.StorageV3 {
basePath, _, err := packed.UnmarshalManifestPath(manifestPath)
if err != nil {
return "", fmt.Errorf("failed to unmarshal manifest path for %s basePath: %w", statsType, err)
return "", merr.Wrapf(err, "failed to unmarshal manifest path for %s basePath", statsType)
}
return fmt.Sprintf("%s/_stats/%s.%d", basePath, statsType, fieldID), nil
}
@@ -845,7 +846,7 @@ func computeStatsBasePath(req *workerpb.CreateStatsRequest, manifestPath string,
req.GetTaskID(), req.GetTaskVersion(),
req.GetCollectionID(), req.GetPartitionID(), req.GetTargetSegmentID(), fieldID), nil
}
return "", fmt.Errorf("unknown stats type: %s", statsType)
return "", merr.WrapErrParameterInvalidMsg("unknown stats type: %s", statsType)
}
func buildIndexParams(
+9 -8
View File
@@ -21,7 +21,6 @@ import (
"fmt"
"strconv"
"github.com/cockroachdb/errors"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/trace"
@@ -82,7 +81,9 @@ func (node *DataNode) CreateJob(ctx context.Context, req *workerpb.CreateJobRequ
Cancel: taskCancel,
State: commonpb.IndexState_InProgress,
}); oldInfo != nil {
err := merr.WrapErrIndexDuplicate(req.GetIndexName(), "building index task existed")
// Node-internal dedup of a coordinator-dispatched build task: a duplicate
// is a scheduling race, not the user re-creating an index.
err := merr.WrapErrServiceInternalMsg("building index task existed, index=%s, buildID=%d", req.GetIndexName(), req.GetBuildID())
log.Warn("duplicated index build task", zap.Error(err))
metrics.DataNodeBuildIndexTaskCounter.WithLabelValues(paramtable.GetStringNodeID(), metrics.FailLabel).Inc()
return merr.Status(err), nil
@@ -254,7 +255,7 @@ func (node *DataNode) CreateJobV2(ctx context.Context, req *workerpb.CreateJobV2
return node.createStatsTask(ctx, statsRequest)
default:
log.Warn("DataNode receive unknown type job")
return merr.Status(fmt.Errorf("DataNode receive unknown type job with TaskID: %d", req.GetTaskID())), nil
return merr.Status(merr.WrapErrServiceInternalMsg("DataNode receive unknown type job with TaskID: %d", req.GetTaskID())), nil
}
}
@@ -454,7 +455,7 @@ func (node *DataNode) QueryJobsV2(ctx context.Context, req *workerpb.QueryJobsV2
default:
log.Warn("DataNode receive querying unknown type jobs")
return &workerpb.QueryJobsV2Response{
Status: merr.Status(errors.New("DataNode receive querying unknown type jobs")),
Status: merr.Status(merr.WrapErrServiceInternalMsg("DataNode receive querying unknown type jobs")),
}, nil
}
}
@@ -479,7 +480,7 @@ func (node *DataNode) queryIndexTask(ctx context.Context, req *workerpb.QueryJob
log.Debug("query index jobs result success", zap.Any("results", results))
if len(results) == 0 {
return &workerpb.QueryJobsV2Response{
Status: merr.Status(fmt.Errorf("tasks '%v' not found", req.GetTaskIDs())),
Status: merr.Status(merr.WrapErrServiceInternalMsg("tasks '%v' not found", req.GetTaskIDs())),
}, nil
}
return &workerpb.QueryJobsV2Response{
@@ -508,7 +509,7 @@ func (node *DataNode) queryStatsTask(ctx context.Context, req *workerpb.QueryJob
log.Debug("query stats job result success", zap.Any("results", results))
if len(results) == 0 {
return &workerpb.QueryJobsV2Response{
Status: merr.Status(fmt.Errorf("tasks '%v' not found", req.GetTaskIDs())),
Status: merr.Status(merr.WrapErrServiceInternalMsg("tasks '%v' not found", req.GetTaskIDs())),
}, nil
}
return &workerpb.QueryJobsV2Response{
@@ -542,7 +543,7 @@ func (node *DataNode) queryAnalyzeTask(ctx context.Context, req *workerpb.QueryJ
log.Debug("query analyze jobs result success", zap.Any("results", results))
if len(results) == 0 {
return &workerpb.QueryJobsV2Response{
Status: merr.Status(fmt.Errorf("tasks '%v' not found", req.GetTaskIDs())),
Status: merr.Status(merr.WrapErrServiceInternalMsg("tasks '%v' not found", req.GetTaskIDs())),
}, nil
}
return &workerpb.QueryJobsV2Response{
@@ -613,6 +614,6 @@ func (node *DataNode) DropJobsV2(ctx context.Context, req *workerpb.DropJobsV2Re
return merr.Success(), nil
default:
log.Warn("DataNode receive dropping unknown type jobs")
return merr.Status(errors.New("DataNode receive dropping unknown type jobs")), nil
return merr.Status(merr.WrapErrServiceInternalMsg("DataNode receive dropping unknown type jobs")), nil
}
}
+10 -10
View File
@@ -194,11 +194,11 @@ func (node *DataNode) CompactionV2(ctx context.Context, req *datapb.CompactionPl
}
if req.GetBeginLogID() == 0 {
return merr.Status(merr.WrapErrParameterInvalidMsg("invalid beginLogID")), nil
return merr.Status(merr.WrapErrServiceInternalMsg("invalid beginLogID")), nil
}
if req.GetPreAllocatedLogIDs().GetBegin() == 0 || req.GetPreAllocatedLogIDs().GetEnd() == 0 {
return merr.Status(merr.WrapErrParameterInvalidMsg(fmt.Sprintf("invalid beginID %d or invalid endID %d", req.GetPreAllocatedLogIDs().GetBegin(), req.GetPreAllocatedLogIDs().GetEnd()))), nil
return merr.Status(merr.WrapErrServiceInternalMsg(fmt.Sprintf("invalid beginID %d or invalid endID %d", req.GetPreAllocatedLogIDs().GetBegin(), req.GetPreAllocatedLogIDs().GetEnd()))), nil
}
/*
@@ -233,7 +233,7 @@ func (node *DataNode) CompactionV2(ctx context.Context, req *datapb.CompactionPl
)
case datapb.CompactionType_MixCompaction:
if req.GetPreAllocatedSegmentIDs() == nil || req.GetPreAllocatedSegmentIDs().GetBegin() == 0 {
return merr.Status(merr.WrapErrParameterInvalidMsg("invalid pre-allocated segmentID range")), nil
return merr.Status(merr.WrapErrServiceInternalMsg("invalid pre-allocated segmentID range")), nil
}
pk, err := typeutil.GetPrimaryFieldSchema(req.GetSchema())
if err != nil {
@@ -256,7 +256,7 @@ func (node *DataNode) CompactionV2(ctx context.Context, req *datapb.CompactionPl
)
case datapb.CompactionType_ClusteringCompaction:
if req.GetPreAllocatedSegmentIDs() == nil || req.GetPreAllocatedSegmentIDs().GetBegin() == 0 {
return merr.Status(merr.WrapErrParameterInvalidMsg("invalid pre-allocated segmentID range")), nil
return merr.Status(merr.WrapErrServiceInternalMsg("invalid pre-allocated segmentID range")), nil
}
if namespaceEnabled {
var sortFields []int64
@@ -281,7 +281,7 @@ func (node *DataNode) CompactionV2(ctx context.Context, req *datapb.CompactionPl
}
case datapb.CompactionType_SortCompaction:
if req.GetPreAllocatedSegmentIDs() == nil || req.GetPreAllocatedSegmentIDs().GetBegin() == 0 {
return merr.Status(merr.WrapErrParameterInvalidMsg("invalid pre-allocated segmentID range")), nil
return merr.Status(merr.WrapErrServiceInternalMsg("invalid pre-allocated segmentID range")), nil
}
pk, err := typeutil.GetPrimaryFieldSchema(req.GetSchema())
if err != nil {
@@ -306,7 +306,7 @@ func (node *DataNode) CompactionV2(ctx context.Context, req *datapb.CompactionPl
task = compactor.NewBumpSchemaVersionCompactionTask(taskCtx, cm, req, compactionParams)
default:
log.Warn("Unknown compaction type", zap.String("type", req.GetType().String()))
return merr.Status(merr.WrapErrParameterInvalidMsg("Unknown compaction type: %v", req.GetType().String())), nil
return merr.Status(merr.WrapErrServiceInternalMsg("Unknown compaction type: %v", req.GetType().String())), nil
}
succeed, err := node.compactionExecutor.Enqueue(task)
@@ -790,7 +790,7 @@ func (node *DataNode) CreateTask(ctx context.Context, request *workerpb.CreateTa
}
return node.CopySegment(ctx, req)
default:
err := fmt.Errorf("unrecognized task type '%s', properties=%v", taskType, request.GetProperties())
err := merr.WrapErrServiceInternalMsg("unrecognized task type '%s', properties=%v", taskType, request.GetProperties())
log.Ctx(ctx).Warn("CreateTask failed", zap.Error(err))
return merr.Status(err), nil
}
@@ -807,7 +807,7 @@ func wrapQueryTaskResult[Resp proto.Message](resp Resp, properties taskcommon.Pr
}
statusResp, ok := any(resp).(ResponseWithStatus)
if !ok {
return &workerpb.QueryTaskResponse{Status: merr.Status(fmt.Errorf("response does not implement GetStatus"))}, nil
return &workerpb.QueryTaskResponse{Status: merr.Status(merr.WrapErrServiceInternalMsg("response does not implement GetStatus"))}, nil
}
return &workerpb.QueryTaskResponse{
Status: statusResp.GetStatus(),
@@ -937,7 +937,7 @@ func (node *DataNode) QueryTask(ctx context.Context, request *workerpb.QueryTask
resProperties.AppendReason(resp.GetReason())
return wrapQueryTaskResult(resp, resProperties)
default:
err := fmt.Errorf("unrecognized task type '%s', properties=%v", taskType, request.GetProperties())
err := merr.WrapErrServiceInternalMsg("unrecognized task type '%s', properties=%v", taskType, request.GetProperties())
log.Ctx(ctx).Warn("QueryTask failed", zap.Error(err))
return &workerpb.QueryTaskResponse{
Status: merr.Status(err),
@@ -997,7 +997,7 @@ func (node *DataNode) DropTask(ctx context.Context, request *workerpb.DropTaskRe
zap.String("clusterID", clusterID))
return merr.Success(), nil
default:
err := fmt.Errorf("unrecognized task type '%s', properties=%v", taskType, request.GetProperties())
err := merr.WrapErrServiceInternalMsg("unrecognized task type '%s', properties=%v", taskType, request.GetProperties())
log.Ctx(ctx).Warn("DropTask failed", zap.Error(err))
return merr.Status(err), nil
}
+12 -3
View File
@@ -594,7 +594,10 @@ func (s *DataNodeServicesSuite) TestCreateTask() {
}
status, err := s.node.CreateTask(s.ctx, req)
s.NoError(err)
s.Equal(commonpb.ErrorCode_UnexpectedError, status.GetErrorCode())
// taskcommon.GetTaskType classifies an unrecognized task type as
// ServiceInternal: task types are coordinator-assigned, so a mismatch is
// an internal protocol violation, not user input.
s.Equal(merr.Code(merr.ErrServiceInternal), status.GetCode())
})
}
@@ -685,7 +688,10 @@ func (s *DataNodeServicesSuite) TestQueryTask() {
}
resp, err := s.node.QueryTask(s.ctx, req)
s.NoError(err)
s.Equal(commonpb.ErrorCode_UnexpectedError, resp.GetStatus().GetErrorCode())
// taskcommon.GetTaskType classifies an unrecognized task type as
// ServiceInternal: task types are coordinator-assigned, so a mismatch is
// an internal protocol violation, not user input.
s.Equal(merr.Code(merr.ErrServiceInternal), resp.GetStatus().GetCode())
})
}
@@ -771,7 +777,10 @@ func (s *DataNodeServicesSuite) TestDropTask() {
}
status, err := s.node.DropTask(s.ctx, req)
s.NoError(err)
s.Equal(commonpb.ErrorCode_UnexpectedError, status.GetErrorCode())
// taskcommon.GetTaskType classifies an unrecognized task type as
// ServiceInternal: task types are coordinator-assigned, so a mismatch is
// an internal protocol violation, not user input.
s.Equal(merr.Code(merr.ErrServiceInternal), status.GetCode())
})
}
@@ -20,7 +20,6 @@ import (
"context"
"fmt"
"github.com/cockroachdb/errors"
"go.uber.org/zap"
"google.golang.org/grpc"
@@ -36,6 +35,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/proto/workerpb"
"github.com/milvus-io/milvus/pkg/v3/util/commonpbutil"
"github.com/milvus-io/milvus/pkg/v3/util/funcutil"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
"github.com/milvus-io/milvus/pkg/v3/util/typeutil"
)
@@ -57,11 +57,11 @@ type Client struct {
// NewClient creates a client for DataNode.
func NewClient(ctx context.Context, addr string, serverID int64, encryption bool) (types.DataNodeClient, error) {
if addr == "" {
return nil, errors.New("address is empty")
return nil, merr.WrapErrParameterInvalidMsg("address is empty")
}
sess := sessionutil.NewSession(context.Background())
if sess == nil {
err := errors.New("new session error, maybe can not connect to etcd")
err := merr.WrapErrServiceUnavailable("new session error, maybe can not connect to etcd")
log.Ctx(ctx).Debug("DataNodeClient New Etcd Session failed", zap.Error(err))
return nil, err
}
@@ -69,7 +69,7 @@ type Client struct {
func NewClient(ctx context.Context) (types.MixCoordClient, error) {
sess := sessionutil.NewSession(context.Background())
if sess == nil {
err := errors.New("new session error, maybe can not connect to etcd")
err := merr.WrapErrServiceUnavailable("new session error, maybe can not connect to etcd")
log.Ctx(ctx).Debug("New MixCoord Client failed", zap.Error(err))
return nil, err
}
@@ -120,7 +120,7 @@ func (c *Client) getMixCoordAddr() (string, error) {
return c.getCompatibleMixCoordAddr()
} else {
log.Warn("MixCoordClient mess key not exist", zap.Any("key", key))
return "", errors.New("find no available mixcoord, check mixcoord state")
return "", merr.WrapErrNodeNotFound(0, "find no available mixcoord, check mixcoord state")
}
}
log.Debug("MixCoordClient GetSessions success",
@@ -137,12 +137,12 @@ func (c *Client) getCompatibleMixCoordAddr() (string, error) {
msess, _, err := c.sess.GetSessions(c.ctx, typeutil.RootCoordRole)
if err != nil {
log.Debug("mixCoordClient getSessions failed", zap.Any("key", typeutil.RootCoordRole), zap.Error(err))
return "", errors.New("find no available mixcoord, check mixcoord state")
return "", merr.WrapErrNodeNotFound(0, "find no available mixcoord, check mixcoord state")
}
ms, ok := msess[typeutil.RootCoordRole]
if !ok {
log.Warn("MixCoordClient mess key not exist", zap.Any("key", typeutil.RootCoordRole))
return "", errors.New("find no available mixcoord, check mixcoord state")
return "", merr.WrapErrNodeNotFound(0, "find no available mixcoord, check mixcoord state")
}
log.Debug("MixCoordClient GetSessions use rootCoord", zap.Any("key", typeutil.RootCoordRole))
c.grpcClient.SetNodeID(ms.ServerID)
+3 -3
View File
@@ -20,7 +20,6 @@ import (
"context"
"fmt"
"github.com/cockroachdb/errors"
"go.uber.org/zap"
"google.golang.org/grpc"
@@ -35,6 +34,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/proto/proxypb"
"github.com/milvus-io/milvus/pkg/v3/util/commonpbutil"
"github.com/milvus-io/milvus/pkg/v3/util/funcutil"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
"github.com/milvus-io/milvus/pkg/v3/util/typeutil"
)
@@ -50,11 +50,11 @@ type Client struct {
// NewClient creates a new client instance
func NewClient(ctx context.Context, addr string, nodeID int64) (types.ProxyClient, error) {
if addr == "" {
return nil, errors.New("address is empty")
return nil, merr.WrapErrParameterInvalidMsg("address is empty")
}
sess := sessionutil.NewSession(context.Background())
if sess == nil {
err := errors.New("new session error, maybe can not connect to etcd")
err := merr.WrapErrServiceUnavailable("new session error, maybe can not connect to etcd")
log.Ctx(ctx).Debug("Proxy client new session failed", zap.Error(err))
return nil, err
}
@@ -17,13 +17,12 @@
package httpserver
import (
"fmt"
"github.com/gin-gonic/gin"
"google.golang.org/protobuf/proto"
"github.com/milvus-io/milvus-proto/go-api/v3/milvuspb"
"github.com/milvus-io/milvus/internal/types"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
)
// Handlers handles http requests
@@ -107,7 +106,7 @@ func (h *Handlers) handleDummy(c *gin.Context) (interface{}, error) {
// use ShouldBind to supports binding JSON, XML, YAML, and protobuf.
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.Dummy(c, &req)
}
@@ -116,11 +115,11 @@ func (h *Handlers) handleCreateCollection(c *gin.Context) (interface{}, error) {
wrappedReq := WrappedCreateCollectionRequest{}
err := shouldBind(c, &wrappedReq)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
schemaProto, err := proto.Marshal(&wrappedReq.Schema)
if err != nil {
return nil, fmt.Errorf("%w: marshal schema failed: %v", errBadRequest, err)
return nil, badRequestf(err, "marshal schema failed")
}
req := &milvuspb.CreateCollectionRequest{
Base: wrappedReq.Base,
@@ -138,7 +137,7 @@ func (h *Handlers) handleDropCollection(c *gin.Context) (interface{}, error) {
req := milvuspb.DropCollectionRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.DropCollection(c, &req)
}
@@ -147,7 +146,7 @@ func (h *Handlers) handleHasCollection(c *gin.Context) (interface{}, error) {
req := milvuspb.HasCollectionRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.HasCollection(c, &req)
}
@@ -156,7 +155,7 @@ func (h *Handlers) handleDescribeCollection(c *gin.Context) (interface{}, error)
req := milvuspb.DescribeCollectionRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.DescribeCollection(c, &req)
}
@@ -165,7 +164,7 @@ func (h *Handlers) handleLoadCollection(c *gin.Context) (interface{}, error) {
req := milvuspb.LoadCollectionRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.LoadCollection(c, &req)
}
@@ -174,7 +173,7 @@ func (h *Handlers) handleReleaseCollection(c *gin.Context) (interface{}, error)
req := milvuspb.ReleaseCollectionRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.ReleaseCollection(c, &req)
}
@@ -183,7 +182,7 @@ func (h *Handlers) handleGetCollectionStatistics(c *gin.Context) (interface{}, e
req := milvuspb.GetCollectionStatisticsRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.GetCollectionStatistics(c, &req)
}
@@ -192,7 +191,7 @@ func (h *Handlers) handleShowCollections(c *gin.Context) (interface{}, error) {
req := milvuspb.ShowCollectionsRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.ShowCollections(c, &req)
}
@@ -201,7 +200,7 @@ func (h *Handlers) handleCreatePartition(c *gin.Context) (interface{}, error) {
req := milvuspb.CreatePartitionRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.CreatePartition(c, &req)
}
@@ -210,7 +209,7 @@ func (h *Handlers) handleDropPartition(c *gin.Context) (interface{}, error) {
req := milvuspb.DropPartitionRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.DropPartition(c, &req)
}
@@ -219,7 +218,7 @@ func (h *Handlers) handleHasPartition(c *gin.Context) (interface{}, error) {
req := milvuspb.HasPartitionRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.HasPartition(c, &req)
}
@@ -228,7 +227,7 @@ func (h *Handlers) handleLoadPartitions(c *gin.Context) (interface{}, error) {
req := milvuspb.LoadPartitionsRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.LoadPartitions(c, &req)
}
@@ -237,7 +236,7 @@ func (h *Handlers) handleReleasePartitions(c *gin.Context) (interface{}, error)
req := milvuspb.ReleasePartitionsRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.ReleasePartitions(c, &req)
}
@@ -246,7 +245,7 @@ func (h *Handlers) handleGetPartitionStatistics(c *gin.Context) (interface{}, er
req := milvuspb.GetPartitionStatisticsRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.GetPartitionStatistics(c, &req)
}
@@ -255,7 +254,7 @@ func (h *Handlers) handleShowPartitions(c *gin.Context) (interface{}, error) {
req := milvuspb.ShowPartitionsRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.ShowPartitions(c, &req)
}
@@ -264,7 +263,7 @@ func (h *Handlers) handleCreateAlias(c *gin.Context) (interface{}, error) {
req := milvuspb.CreateAliasRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.CreateAlias(c, &req)
}
@@ -273,7 +272,7 @@ func (h *Handlers) handleDropAlias(c *gin.Context) (interface{}, error) {
req := milvuspb.DropAliasRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.DropAlias(c, &req)
}
@@ -282,7 +281,7 @@ func (h *Handlers) handleAlterAlias(c *gin.Context) (interface{}, error) {
req := milvuspb.AlterAliasRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.AlterAlias(c, &req)
}
@@ -291,7 +290,7 @@ func (h *Handlers) handleCreateIndex(c *gin.Context) (interface{}, error) {
req := milvuspb.CreateIndexRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.CreateIndex(c, &req)
}
@@ -300,7 +299,7 @@ func (h *Handlers) handleDescribeIndex(c *gin.Context) (interface{}, error) {
req := milvuspb.DescribeIndexRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.DescribeIndex(c, &req)
}
@@ -309,7 +308,7 @@ func (h *Handlers) handleGetIndexState(c *gin.Context) (interface{}, error) {
req := milvuspb.GetIndexStateRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.GetIndexState(c, &req)
}
@@ -318,7 +317,7 @@ func (h *Handlers) handleGetIndexBuildProgress(c *gin.Context) (interface{}, err
req := milvuspb.GetIndexBuildProgressRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.GetIndexBuildProgress(c, &req)
}
@@ -327,7 +326,7 @@ func (h *Handlers) handleDropIndex(c *gin.Context) (interface{}, error) {
req := milvuspb.DropIndexRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.DropIndex(c, &req)
}
@@ -336,11 +335,11 @@ func (h *Handlers) handleInsert(c *gin.Context) (interface{}, error) {
wrappedReq := WrappedInsertRequest{}
err := shouldBind(c, &wrappedReq)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
req, err := wrappedReq.AsInsertRequest()
if err != nil {
return nil, fmt.Errorf("%w: convert body to pb failed: %v", errBadRequest, err)
return nil, badRequestf(err, "convert body to pb failed")
}
return h.proxy.Insert(c, req)
}
@@ -349,7 +348,7 @@ func (h *Handlers) handleDelete(c *gin.Context) (interface{}, error) {
req := milvuspb.DeleteRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.Delete(c, &req)
}
@@ -358,10 +357,10 @@ func (h *Handlers) handleSearch(c *gin.Context) (interface{}, error) {
wrappedReq := SearchRequest{}
err := shouldBind(c, &wrappedReq)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
if wrappedReq.HasSearchAggregation() {
return nil, fmt.Errorf("%w: searchAggregation is not supported for low-level REST search", errBadRequest)
return nil, badRequestf(merr.WrapErrParameterInvalidMsg("searchAggregation is not supported for low-level REST search"), "invalid request")
}
req := milvuspb.SearchRequest{
Base: wrappedReq.Base,
@@ -392,7 +391,7 @@ func (h *Handlers) handleQuery(c *gin.Context) (interface{}, error) {
req := milvuspb.QueryRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.Query(c, &req)
}
@@ -401,7 +400,7 @@ func (h *Handlers) handleFlush(c *gin.Context) (interface{}, error) {
req := milvuspb.FlushRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.Flush(c, &req)
}
@@ -410,7 +409,7 @@ func (h *Handlers) handleCalcDistance(c *gin.Context) (interface{}, error) {
wrappedReq := WrappedCalcDistanceRequest{}
err := shouldBind(c, &wrappedReq)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
req := milvuspb.CalcDistanceRequest{
@@ -426,7 +425,7 @@ func (h *Handlers) handleGetFlushState(c *gin.Context) (interface{}, error) {
req := milvuspb.GetFlushStateRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.GetFlushState(c, &req)
}
@@ -435,7 +434,7 @@ func (h *Handlers) handleGetPersistentSegmentInfo(c *gin.Context) (interface{},
req := milvuspb.GetPersistentSegmentInfoRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.GetPersistentSegmentInfo(c, &req)
}
@@ -444,7 +443,7 @@ func (h *Handlers) handleGetQuerySegmentInfo(c *gin.Context) (interface{}, error
req := milvuspb.GetQuerySegmentInfoRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.GetQuerySegmentInfo(c, &req)
}
@@ -453,7 +452,7 @@ func (h *Handlers) handleGetReplicas(c *gin.Context) (interface{}, error) {
req := milvuspb.GetReplicasRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.GetReplicas(c, &req)
}
@@ -462,7 +461,7 @@ func (h *Handlers) handleGetMetrics(c *gin.Context) (interface{}, error) {
req := milvuspb.GetMetricsRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.GetMetrics(c, &req)
}
@@ -471,7 +470,7 @@ func (h *Handlers) handleLoadBalance(c *gin.Context) (interface{}, error) {
req := milvuspb.LoadBalanceRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.LoadBalance(c, &req)
}
@@ -480,7 +479,7 @@ func (h *Handlers) handleGetCompactionState(c *gin.Context) (interface{}, error)
req := milvuspb.GetCompactionStateRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.GetCompactionState(c, &req)
}
@@ -489,7 +488,7 @@ func (h *Handlers) handleGetCompactionStateWithPlans(c *gin.Context) (interface{
req := milvuspb.GetCompactionPlansRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.GetCompactionStateWithPlans(c, &req)
}
@@ -498,7 +497,7 @@ func (h *Handlers) handleManualCompaction(c *gin.Context) (interface{}, error) {
req := milvuspb.ManualCompactionRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.ManualCompaction(c, &req)
}
@@ -507,7 +506,7 @@ func (h *Handlers) handleImport(c *gin.Context) (interface{}, error) {
req := milvuspb.ImportRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.Import(c, &req)
}
@@ -516,7 +515,7 @@ func (h *Handlers) handleGetImportState(c *gin.Context) (interface{}, error) {
req := milvuspb.GetImportStateRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.GetImportState(c, &req)
}
@@ -525,7 +524,7 @@ func (h *Handlers) handleListImportTasks(c *gin.Context) (interface{}, error) {
req := milvuspb.ListImportTasksRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.ListImportTasks(c, &req)
}
@@ -534,7 +533,7 @@ func (h *Handlers) handleCreateCredential(c *gin.Context) (interface{}, error) {
req := milvuspb.CreateCredentialRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.CreateCredential(c, &req)
}
@@ -543,7 +542,7 @@ func (h *Handlers) handleUpdateCredential(c *gin.Context) (interface{}, error) {
req := milvuspb.UpdateCredentialRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.UpdateCredential(c, &req)
}
@@ -552,7 +551,7 @@ func (h *Handlers) handleDeleteCredential(c *gin.Context) (interface{}, error) {
req := milvuspb.DeleteCredentialRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.DeleteCredential(c, &req)
}
@@ -561,7 +560,7 @@ func (h *Handlers) handleListCredUsers(c *gin.Context) (interface{}, error) {
req := milvuspb.ListCredUsersRequest{}
err := shouldBind(c, &req)
if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
return nil, badRequestf(err, "parse body failed")
}
return h.proxy.ListCredUsers(c, &req)
}
@@ -1434,7 +1434,7 @@ func TestFp16Bf16VectorsV1(t *testing.T) {
assert.Nil(t, err, "case %d: ", i)
assert.Equal(t, testcase.errCode, returnBody.Code, "case %d: ", i, string(testcase.requestBody))
if testcase.errCode != 0 {
assert.Equal(t, testcase.errMsg, returnBody.Message, "case %d: ", i, string(testcase.requestBody))
assert.Contains(t, returnBody.Message, testcase.errMsg, "case %d: ", i, string(testcase.requestBody))
}
fmt.Println(w.Body.String())
})
@@ -410,6 +410,35 @@ func wrapperPost(newReq newReqFunc, v2 handlerFuncV2) gin.HandlerFunc {
dbName,
collectionName,
).Inc()
// Mirror the fail_input/fail_system metric split into the logs so a
// failed REST request can be filtered by error_type. System failures are
// logged at Warn (actionable); input failures at Info (expected user
// mistakes — keeping them at Warn would spam the logs).
if label == metrics.FailSystemLabel || label == metrics.FailInputLabel {
var status *commonpb.Status
switch r := resp.(type) {
case interface{ GetStatus() *commonpb.Status }:
status = r.GetStatus()
case *commonpb.Status:
status = r
}
errType := merr.SystemError
if label == metrics.FailInputLabel {
errType = merr.InputError
}
logger := log.Ctx(ctx).With(
zap.String("method", methodTag),
zap.String("error_type", errType.String()),
zap.Int32("code", status.GetCode()),
zap.String("reason", status.GetReason()),
)
if errType == merr.InputError {
logger.Info("restful request returned an input error")
} else {
logger.Warn("restful request returned a system error")
}
}
}
}
@@ -543,6 +572,9 @@ func wrapperProxyWithLimit(ctx context.Context, ginCtx *gin.Context, req any, ch
}
if err != nil {
// Expose the exact classification (incl. boundary InputError marks) to
// the REST access log; key must match accesslog/info.ContextErrorType.
ginCtx.Set("error_type", merr.GetErrorType(err).String())
log.Ctx(ctx).Warn("high level restful api, grpc call failed", zap.Error(err))
if !ignoreErr {
HTTPAbortReturn(ginCtx, http.StatusOK, gin.H{HTTPReturnCode: merr.Code(err), HTTPReturnMessage: err.Error()})
@@ -1025,7 +1057,7 @@ func (h *HandlersV2) waitForFlush(ctx context.Context, dbName string, collection
segmentIDs := flushResp.GetCollSegIDs()[collectionName].GetData()
flushTs, ok := flushResp.GetCollFlushTs()[collectionName]
if !ok {
return merr.WrapErrServiceInternal(fmt.Sprintf("failed to get flush timestamp for collection %s", collectionName))
return merr.WrapErrServiceInternalMsg("failed to get flush timestamp for collection %s", collectionName)
}
ticker := time.NewTicker(500 * time.Millisecond)
@@ -1521,7 +1553,7 @@ func generatePlaceholderGroup(ctx context.Context, body string, collSchema *sche
fieldName = field.Name
vectorField = field
} else {
return nil, errors.New("search without annsField, but already found multiple vector fields: [" + fieldName + ", " + field.Name + ",,,]")
return nil, merr.WrapErrParameterInvalidMsg("search without annsField, but already found multiple vector fields: [%s, %s,,,]", fieldName, field.Name)
}
}
}
@@ -1547,7 +1579,7 @@ func generatePlaceholderGroup(ctx context.Context, body string, collSchema *sche
}
}
if vectorField == nil {
return nil, errors.New("cannot find a vector field named: " + fieldName)
return nil, merr.WrapErrFieldNotFound(fieldName, "cannot find a vector field")
}
dim := int64(0)
if !typeutil.IsSparseFloatVectorType(vectorField.DataType) {
@@ -3352,13 +3352,13 @@ func TestSearchV2(t *testing.T) {
queryTestCases = append(queryTestCases, requestBodyTestCase{
path: SearchAction,
requestBody: []byte(`{"collectionName": "book", "data": [[0.1, 0.2]], "annsField": "word_count", "filter": "book_id in [2, 4, 6, 8]", "limit": 4, "outputFields": ["word_count"], "params": {"radius":0.9, "range_filter": 0.1}, "groupingField": "test"}`),
errMsg: "can only accept json format request, error: cannot find a vector field named: word_count",
errMsg: "cannot find a vector field",
errCode: 1801,
})
queryTestCases = append(queryTestCases, requestBodyTestCase{
path: AdvancedSearchAction,
requestBody: []byte(`{"collectionName": "hello_milvus", "search": [{"data": [[0.1, 0.2]], "annsField": "float_vector1", "metricType": "L2", "limit": 3}, {"data": [[0.1, 0.2]], "annsField": "float_vector2", "metricType": "L2", "limit": 3}], "rerank": {"strategy": "rrf", "params": {"k": 1}}}`),
errMsg: "can only accept json format request, error: cannot find a vector field named: float_vector1",
errMsg: "cannot find a vector field",
errCode: 1801,
})
// multiple annsFields
@@ -27,7 +27,6 @@ import (
"sync"
"time"
"github.com/cockroachdb/errors"
"github.com/gin-gonic/gin"
"go.uber.org/zap"
@@ -93,7 +92,7 @@ func (w *timeoutResponseRecorder) Write(data []byte) (int, error) {
w.mu.Lock()
defer w.mu.Unlock()
if w.closed || w.body == nil {
return 0, errors.New("response writer closed")
return 0, merr.WrapErrServiceInternalMsg("response writer closed")
}
if !w.written() {
w.size = 0
@@ -107,7 +106,7 @@ func (w *timeoutResponseRecorder) WriteString(s string) (int, error) {
w.mu.Lock()
defer w.mu.Unlock()
if w.closed || w.body == nil {
return 0, errors.New("response writer closed")
return 0, merr.WrapErrServiceInternalMsg("response writer closed")
}
if !w.written() {
w.size = 0
@@ -161,7 +160,7 @@ func (w *timeoutResponseRecorder) Written() bool {
func (w *timeoutResponseRecorder) Flush() {}
func (w *timeoutResponseRecorder) Hijack() (net.Conn, *bufio.ReadWriter, error) {
return nil, nil, errors.New("response writer does not support hijack")
return nil, nil, merr.WrapErrServiceInternalMsg("response writer does not support hijack")
}
func (w *timeoutResponseRecorder) CloseNotify() <-chan bool {
@@ -176,7 +175,7 @@ func (w *timeoutResponseRecorder) CommitTo(realWriter gin.ResponseWriter) error
w.mu.Lock()
defer w.mu.Unlock()
if w.closed || w.body == nil {
return errors.New("response writer closed")
return merr.WrapErrServiceInternalMsg("response writer closed")
}
dst := realWriter.Header()
+75 -74
View File
@@ -27,7 +27,6 @@ import (
"strings"
"time"
"github.com/cockroachdb/errors"
"github.com/gin-gonic/gin"
"github.com/spf13/cast"
"github.com/tidwall/gjson"
@@ -239,7 +238,7 @@ func convertRange(field *schemapb.FieldSchema, result gjson.Result) (string, err
func checkGetPrimaryKey(coll *schemapb.CollectionSchema, idResult gjson.Result) (string, error) {
primaryField, ok := getPrimaryField(coll)
if !ok {
return "", fmt.Errorf("collection: %s has no primary field", coll.Name)
return "", merr.WrapErrParameterInvalidMsg("collection: %s has no primary field", coll.Name)
}
resultStr, err := convertRange(primaryField, idResult)
if err != nil {
@@ -253,7 +252,7 @@ func checkGetPrimaryKey(coll *schemapb.CollectionSchema, idResult gjson.Result)
// based on the primary key field type
func convertIDsToSchemapbIDs(ids []interface{}, pkField *schemapb.FieldSchema) (*schemapb.IDs, error) {
if len(ids) == 0 {
return nil, errors.New("ids array cannot be empty")
return nil, merr.WrapErrParameterMissingMsg("ids array cannot be empty")
}
switch pkField.DataType {
@@ -270,18 +269,18 @@ func convertIDsToSchemapbIDs(ids []interface{}, pkField *schemapb.FieldSchema) (
// JSON numbers are decoded as float64
// Check if the float has a fractional part
if v != math.Trunc(v) {
return nil, fmt.Errorf("invalid int64 id at index %d: %v has fractional part", i, v)
return nil, merr.WrapErrParameterInvalidMsg("invalid int64 id at index %d: %v has fractional part", i, v)
}
int64ID = int64(v)
case string:
// Try to parse string as int64
parsed, err := strconv.ParseInt(v, 10, 64)
if err != nil {
return nil, fmt.Errorf("invalid int64 id at index %d: %v, error: %v", i, id, err)
return nil, merr.WrapErrParameterInvalidErr(err, "invalid int64 id at index %d: %v", i, id)
}
int64ID = parsed
default:
return nil, fmt.Errorf("invalid id type at index %d: expected int64, got %T", i, id)
return nil, merr.WrapErrParameterInvalidMsg("invalid id type at index %d: expected int64, got %T", i, id)
}
int64IDs = append(int64IDs, int64ID)
}
@@ -304,10 +303,10 @@ func convertIDsToSchemapbIDs(ids []interface{}, pkField *schemapb.FieldSchema) (
// Convert number to string
stringID = fmt.Sprintf("%v", v)
default:
return nil, fmt.Errorf("invalid id type at index %d: expected string, got %T", i, id)
return nil, merr.WrapErrParameterInvalidMsg("invalid id type at index %d: expected string, got %T", i, id)
}
if stringID == "" {
return nil, fmt.Errorf("empty string id at index %d", i)
return nil, merr.WrapErrParameterInvalidMsg("empty string id at index %d", i)
}
stringIDs = append(stringIDs, stringID)
}
@@ -320,7 +319,7 @@ func convertIDsToSchemapbIDs(ids []interface{}, pkField *schemapb.FieldSchema) (
}, nil
default:
return nil, fmt.Errorf("unsupported primary key type: %s", pkField.DataType.String())
return nil, merr.WrapErrParameterInvalidMsg("unsupported primary key type: %s", pkField.DataType.String())
}
}
@@ -784,7 +783,7 @@ func checkAndSetData(body []byte, collSchema *schemapb.CollectionSchema, partial
if !containsString(fieldNames, mapKey) {
if collSchema.EnableDynamicField {
if mapKey == common.MetaFieldName {
return nil, nil, merr.WrapErrParameterInvalidMsg(fmt.Sprintf("use the invalid field name(%s) when enable dynamicField", mapKey))
return nil, nil, merr.WrapErrParameterInvalidMsg("use the invalid field name(%s) when enable dynamicField", mapKey)
}
mapValueStr := mapValue.String()
switch mapValue.Type {
@@ -1106,7 +1105,7 @@ func decodeByteVectorElement(v gjson.Result, dim, bytesPerVec int64, isFloat16 b
return nil, err
}
if int64(len(row)) != dim {
return nil, fmt.Errorf("vector dim mismatch: expect %d, got %d", dim, len(row))
return nil, merr.WrapErrParameterInvalidMsg("vector dim mismatch: expect %d, got %d", dim, len(row))
}
if isFloat16 {
return typeutil.Float32ArrayToFloat16Bytes(row), nil
@@ -1114,14 +1113,14 @@ func decodeByteVectorElement(v gjson.Result, dim, bytesPerVec int64, isFloat16 b
return typeutil.Float32ArrayToBFloat16Bytes(row), nil
}
if v.Type != gjson.String {
return nil, fmt.Errorf("expect float vector array or base64 string")
return nil, merr.WrapErrParameterInvalidMsg("expect float vector array or base64 string")
}
var row []byte
if err := json.Unmarshal([]byte(v.Raw), &row); err != nil {
return nil, err
}
if int64(len(row)) != bytesPerVec {
return nil, fmt.Errorf("byte length mismatch: expect %d, got %d", bytesPerVec, len(row))
return nil, merr.WrapErrParameterInvalidMsg("byte length mismatch: expect %d, got %d", bytesPerVec, len(row))
}
return row, nil
}
@@ -1293,7 +1292,7 @@ func encodeEmbListQuery(vecs []gjson.Result, elemType schemapb.DataType, dim int
func buildStructArrayFieldData(structSchema *schemapb.StructArrayFieldSchema, perRow []structArrayRow) (*schemapb.FieldData, error) {
if len(perRow) == 0 {
return nil, fmt.Errorf("struct array field %s has no rows", structSchema.GetName())
return nil, merr.WrapErrParameterInvalidMsg("struct array field %s has no rows", structSchema.GetName())
}
subs := structSchema.GetFields()
subFieldData := make([]*schemapb.FieldData, 0, len(subs))
@@ -1308,12 +1307,12 @@ func buildStructArrayFieldData(structSchema *schemapb.StructArrayFieldSchema, pe
for rowIdx, row := range perRow {
val, ok := row[short]
if !ok {
return nil, fmt.Errorf("struct %s row %d missing sub-field %s",
return nil, merr.WrapErrParameterInvalidMsg("struct %s row %d missing sub-field %s",
structSchema.GetName(), rowIdx, short)
}
scalar, ok := val.(*schemapb.ScalarField)
if !ok {
return nil, fmt.Errorf("struct %s sub-field %s row %d: unexpected payload type %T",
return nil, merr.WrapErrParameterInvalidMsg("struct %s sub-field %s row %d: unexpected payload type %T",
structSchema.GetName(), short, rowIdx, val)
}
arrayArray.Data = append(arrayArray.Data, scalar)
@@ -1341,12 +1340,12 @@ func buildStructArrayFieldData(structSchema *schemapb.StructArrayFieldSchema, pe
for rowIdx, row := range perRow {
val, ok := row[short]
if !ok {
return nil, fmt.Errorf("struct %s row %d missing sub-field %s",
return nil, merr.WrapErrParameterInvalidMsg("struct %s row %d missing sub-field %s",
structSchema.GetName(), rowIdx, short)
}
vf, ok := val.(*schemapb.VectorField)
if !ok {
return nil, fmt.Errorf("struct %s sub-field %s row %d: unexpected payload type %T",
return nil, merr.WrapErrParameterInvalidMsg("struct %s sub-field %s row %d: unexpected payload type %T",
structSchema.GetName(), short, rowIdx, val)
}
vecArray.Data = append(vecArray.Data, vf)
@@ -1365,7 +1364,7 @@ func buildStructArrayFieldData(structSchema *schemapb.StructArrayFieldSchema, pe
},
})
default:
return nil, fmt.Errorf("unsupported struct sub-field data type: %s", sub.GetDataType())
return nil, merr.WrapErrParameterInvalidMsg("unsupported struct sub-field data type: %s", sub.GetDataType())
}
}
return &schemapb.FieldData{
@@ -1403,11 +1402,11 @@ func extractStructArrayRow(fd *schemapb.FieldData, rowIdx int, schema *schemapb.
case schemapb.DataType_Array:
rowData := sub.GetScalars().GetArrayData().GetData()
if rowIdx >= len(rowData) {
return nil, fmt.Errorf("struct sub-field %s missing row %d", short, rowIdx)
return nil, merr.WrapErrParameterInvalidMsg("struct sub-field %s missing row %d", short, rowIdx)
}
values := scalarArrayToInterfaces(rowData[rowIdx])
if len(values) != elemCount {
return nil, fmt.Errorf("struct sub-field %s element count mismatch: expect %d got %d",
return nil, merr.WrapErrParameterInvalidMsg("struct sub-field %s element count mismatch: expect %d got %d",
short, elemCount, len(values))
}
for i, v := range values {
@@ -1416,28 +1415,28 @@ func extractStructArrayRow(fd *schemapb.FieldData, rowIdx int, schema *schemapb.
case schemapb.DataType_ArrayOfVector:
va := sub.GetVectors().GetVectorArray()
if va == nil {
return nil, fmt.Errorf("struct sub-field %s has no vector array", short)
return nil, merr.WrapErrParameterInvalidMsg("struct sub-field %s has no vector array", short)
}
if rowIdx >= len(va.GetData()) {
return nil, fmt.Errorf("struct sub-field %s missing row %d", short, rowIdx)
return nil, merr.WrapErrParameterInvalidMsg("struct sub-field %s missing row %d", short, rowIdx)
}
dim, ok := subDims[short]
if !ok || dim <= 0 {
return nil, fmt.Errorf("schema missing dim for struct sub-field %s", short)
return nil, merr.WrapErrParameterInvalidMsg("schema missing dim for struct sub-field %s", short)
}
values, err := vectorFieldToInterfaces(va.GetData()[rowIdx], va.GetElementType(), dim)
if err != nil {
return nil, err
}
if len(values) != elemCount {
return nil, fmt.Errorf("struct sub-field %s vector element count mismatch: expect %d got %d",
return nil, merr.WrapErrParameterInvalidMsg("struct sub-field %s vector element count mismatch: expect %d got %d",
short, elemCount, len(values))
}
for i, v := range values {
out[i][short] = v
}
default:
return nil, fmt.Errorf("unsupported struct sub-field type %s", sub.GetType())
return nil, merr.WrapErrParameterInvalidMsg("unsupported struct sub-field type %s", sub.GetType())
}
}
return out, nil
@@ -1455,7 +1454,7 @@ func structArraySubDims(fieldName string, schema *schemapb.CollectionSchema) (ma
}
dim, err := getDim(sub)
if err != nil {
return nil, fmt.Errorf("schema sub-field %s has no dim: %w", sub.GetName(), err)
return nil, merr.WrapErrParameterInvalidErr(err, "schema sub-field %s has no dim", sub.GetName())
}
subDims[subShortName(sub)] = dim
}
@@ -1469,22 +1468,22 @@ func structSubElemCount(sub *schemapb.FieldData, rowIdx int, subDims map[string]
case schemapb.DataType_Array:
rowData := sub.GetScalars().GetArrayData().GetData()
if rowIdx >= len(rowData) {
return 0, fmt.Errorf("struct sub-field %s row %d out of range", sub.GetFieldName(), rowIdx)
return 0, merr.WrapErrParameterInvalidMsg("struct sub-field %s row %d out of range", sub.GetFieldName(), rowIdx)
}
return len(scalarArrayToInterfaces(rowData[rowIdx])), nil
case schemapb.DataType_ArrayOfVector:
va := sub.GetVectors().GetVectorArray()
if va == nil || rowIdx >= len(va.GetData()) {
return 0, fmt.Errorf("struct sub-field %s row %d out of range", sub.GetFieldName(), rowIdx)
return 0, merr.WrapErrParameterInvalidMsg("struct sub-field %s row %d out of range", sub.GetFieldName(), rowIdx)
}
short := structFieldShortName(sub.GetFieldName())
dim, ok := subDims[short]
if !ok || dim <= 0 {
return 0, fmt.Errorf("schema missing dim for struct sub-field %s", short)
return 0, merr.WrapErrParameterInvalidMsg("schema missing dim for struct sub-field %s", short)
}
return vectorFieldElemCount(va.GetData()[rowIdx], va.GetElementType(), dim)
default:
return 0, fmt.Errorf("unsupported struct sub-field type %s", sub.GetType())
return 0, merr.WrapErrParameterInvalidMsg("unsupported struct sub-field type %s", sub.GetType())
}
}
@@ -1539,7 +1538,7 @@ func scalarArrayToInterfaces(sf *schemapb.ScalarField) []interface{} {
func vectorFieldElemCount(vf *schemapb.VectorField, elemType schemapb.DataType, dim int64) (int, error) {
if dim <= 0 {
return 0, fmt.Errorf("invalid dim %d", dim)
return 0, merr.WrapErrParameterInvalidMsg("invalid dim %d", dim)
}
switch elemType {
case schemapb.DataType_FloatVector:
@@ -1553,13 +1552,13 @@ func vectorFieldElemCount(vf *schemapb.VectorField, elemType schemapb.DataType,
case schemapb.DataType_Int8Vector:
return len(vf.GetInt8Vector()) / int(dim), nil
default:
return 0, fmt.Errorf("unsupported vector element type %s", elemType)
return 0, merr.WrapErrParameterInvalidMsg("unsupported vector element type %s", elemType)
}
}
func vectorFieldToInterfaces(vf *schemapb.VectorField, elemType schemapb.DataType, dim int64) ([]interface{}, error) {
if dim <= 0 {
return nil, fmt.Errorf("invalid dim %d", dim)
return nil, merr.WrapErrParameterInvalidMsg("invalid dim %d", dim)
}
switch elemType {
case schemapb.DataType_FloatVector:
@@ -1612,7 +1611,7 @@ func vectorFieldToInterfaces(vf *schemapb.VectorField, elemType schemapb.DataTyp
}
return out, nil
default:
return nil, fmt.Errorf("unsupported vector element type %s", elemType)
return nil, merr.WrapErrParameterInvalidMsg("unsupported vector element type %s", elemType)
}
}
@@ -1620,7 +1619,7 @@ func convertFloatVectorToArray(vector [][]float32, dim int64) ([]float32, error)
floatArray := make([]float32, 0)
for _, arr := range vector {
if int64(len(arr)) != dim {
return nil, fmt.Errorf("[]float32 size %d doesn't equal to vector dimension %d of %s",
return nil, merr.WrapErrParameterInvalidMsg("[]float32 size %d doesn't equal to vector dimension %d of %s",
len(arr), dim, schemapb.DataType_name[int32(schemapb.DataType_FloatVector)])
}
for i := int64(0); i < dim; i++ {
@@ -1643,7 +1642,7 @@ func convertBinaryVectorToArray(vector [][]byte, dim int64, dataType schemapb.Da
binaryArray := make([]byte, 0, len(vector)*int(bytesLen))
for _, arr := range vector {
if int64(len(arr)) != bytesLen {
return nil, fmt.Errorf("[]byte size %d doesn't equal to vector dimension %d of %s",
return nil, merr.WrapErrParameterInvalidMsg("[]byte size %d doesn't equal to vector dimension %d of %s",
len(arr), dim, schemapb.DataType_name[int32(dataType)])
}
for i := int64(0); i < bytesLen; i++ {
@@ -1657,7 +1656,7 @@ func convertInt8VectorToArray(vector [][]int8, dim int64) ([]byte, error) {
byteArray := make([]byte, 0)
for _, arr := range vector {
if int64(len(arr)) != dim {
return nil, fmt.Errorf("[]int8 size %d doesn't equal to vector dimension %d of %s",
return nil, merr.WrapErrParameterInvalidMsg("[]int8 size %d doesn't equal to vector dimension %d of %s",
len(arr), dim, schemapb.DataType_name[int32(schemapb.DataType_Int8Vector)])
}
for i := int64(0); i < dim; i++ {
@@ -1691,7 +1690,7 @@ func reflectValueCandi(v reflect.Value) (map[string]fieldCandi, error) {
}
return result, nil
default:
return nil, fmt.Errorf("unsupport row type: %s", v.Kind().String())
return nil, merr.WrapErrParameterInvalidMsg("unsupport row type: %s", v.Kind().String())
}
}
@@ -1713,7 +1712,7 @@ func convertToIntArray(dataType schemapb.DataType, arr interface{}) []int32 {
func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool, sch *schemapb.CollectionSchema, inInsert bool, partialUpdate bool) ([]*schemapb.FieldData, error) {
rowsLen := len(rows)
if rowsLen == 0 {
return []*schemapb.FieldData{}, errors.New("no row need to be convert to columns")
return []*schemapb.FieldData{}, merr.WrapErrParameterInvalidMsg("no row need to be convert to columns")
}
isDynamic := sch.EnableDynamicField
@@ -1802,7 +1801,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
dim, _ := getDim(field)
nameDims[field.Name] = dim
default:
return nil, fmt.Errorf("the type(%v) of field(%v) is not supported, use other sdk please", field.DataType, field.Name)
return nil, merr.WrapErrParameterInvalidMsg("the type(%v) of field(%v) is not supported, use other sdk please", field.DataType, field.Name)
}
nameColumns[field.Name] = data
fieldData[field.Name] = &schemapb.FieldData{
@@ -1813,7 +1812,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
}
}
if len(nameDims) == 0 && len(sch.Functions) == 0 && !partialUpdate {
return nil, fmt.Errorf("collection: %s has no vector field or functions", sch.Name)
return nil, merr.WrapErrParameterInvalidMsg("collection: %s has no vector field or functions", sch.Name)
}
dynamicCol := make([][]byte, 0, rowsLen)
@@ -1836,7 +1835,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
continue
}
if !allowInsertAutoID {
return nil, merr.WrapErrParameterInvalidMsg(fmt.Sprintf("no need to pass pk field(%s) when autoid==true in insert", field.Name))
return nil, merr.WrapErrParameterInvalidMsg("no need to pass pk field(%s) when autoid==true in insert", field.Name)
}
}
if (field.Nullable || field.DefaultValue != nil) && !ok {
@@ -1851,7 +1850,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
if partialUpdate {
continue
}
return nil, fmt.Errorf("row %d does not has field %s", idx, field.Name)
return nil, merr.WrapErrParameterInvalidMsg("row %d does not has field %s", idx, field.Name)
}
fieldLen[field.Name] += 1
switch field.DataType {
@@ -1893,7 +1892,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
vec := typeutil.Float32ArrayToFloat16Bytes(candi.v.Interface().([]float32))
nameColumns[field.Name] = append(nameColumns[field.Name].([][]byte), vec)
default:
return nil, merr.WrapErrParameterInvalidMsg(fmt.Sprintf("invalid type(%v) of field(%v) ", field.DataType, field.Name))
return nil, merr.WrapErrParameterInvalidMsg("invalid type(%v) of field(%v) ", field.DataType, field.Name)
}
case schemapb.DataType_BFloat16Vector:
switch candi.v.Interface().(type) {
@@ -1903,7 +1902,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
vec := typeutil.Float32ArrayToBFloat16Bytes(candi.v.Interface().([]float32))
nameColumns[field.Name] = append(nameColumns[field.Name].([][]byte), vec)
default:
return nil, merr.WrapErrParameterInvalidMsg(fmt.Sprintf("invalid type(%v) of field(%v) ", field.DataType, field.Name))
return nil, merr.WrapErrParameterInvalidMsg("invalid type(%v) of field(%v) ", field.DataType, field.Name)
}
case schemapb.DataType_SparseFloatVector:
content := candi.v.Interface().([]byte)
@@ -1915,7 +1914,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
case schemapb.DataType_Int8Vector:
nameColumns[field.Name] = append(nameColumns[field.Name].([][]int8), candi.v.Interface().([]int8))
default:
return nil, fmt.Errorf("the type(%v) of field(%v) is not supported, use other sdk please", field.DataType, field.Name)
return nil, merr.WrapErrParameterInvalidMsg("the type(%v) of field(%v) is not supported, use other sdk please", field.DataType, field.Name)
}
delete(set, field.Name)
@@ -1931,7 +1930,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
}
bs, err := json.Marshal(m)
if err != nil {
return nil, fmt.Errorf("failed to marshal dynamic field %w", err)
return nil, merr.WrapErrParameterInvalidErr(err, "failed to marshal dynamic field")
}
dynamicCol = append(dynamicCol, bs)
}
@@ -1949,7 +1948,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
zap.String("fieldName", name),
zap.Int("fieldLen", len(validData)),
zap.Int("rowsLen", rowsLen))
return nil, fmt.Errorf("column %s has length %d, expected %d", name, len(validData), rowsLen)
return nil, merr.WrapErrParameterInvalidMsg("column %s has length %d, expected %d", name, len(validData), rowsLen)
}
} else {
log.Info("skip empty field for partial update",
@@ -1963,7 +1962,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
zap.String("fieldName", name),
zap.Int("fieldLen", fieldLen[name]),
zap.Int("rowsLen", rowsLen))
return nil, fmt.Errorf("column %s has length %d, expected %d", name, fieldLen[name], rowsLen)
return nil, merr.WrapErrParameterInvalidMsg("column %s has length %d, expected %d", name, fieldLen[name], rowsLen)
}
colData := fieldData[name]
@@ -2183,7 +2182,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
},
}
default:
return nil, fmt.Errorf("the type(%v) of field(%v) is not supported, use other sdk please", colData.Type, name)
return nil, merr.WrapErrParameterInvalidMsg("the type(%v) of field(%v) is not supported, use other sdk please", colData.Type, name)
}
colData.ValidData = validDataMap[name]
columns = append(columns, colData)
@@ -2212,11 +2211,11 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
if partialUpdate {
continue
}
return nil, fmt.Errorf("row %d does not has struct field %s", rowIdx, structField.GetName())
return nil, merr.WrapErrParameterInvalidMsg("row %d does not has struct field %s", rowIdx, structField.GetName())
}
sr, ok := val.(structArrayRow)
if !ok {
return nil, fmt.Errorf("row %d struct field %s has unexpected payload type %T",
return nil, merr.WrapErrParameterInvalidMsg("row %d struct field %s has unexpected payload type %T",
rowIdx, structField.GetName(), val)
}
perRow = append(perRow, sr)
@@ -2437,27 +2436,27 @@ func fieldDataValueCount(fieldData *schemapb.FieldData) (int64, error) {
dim := fieldData.GetVectors().GetDim()
bytesPerRow := dim / 8
if bytesPerRow <= 0 {
return 0, fmt.Errorf("invalid binary vector dimension %d for field %s", dim, fieldData.GetFieldName())
return 0, merr.WrapErrParameterInvalidMsg("invalid binary vector dimension %d for field %s", dim, fieldData.GetFieldName())
}
return int64(len(fieldData.GetVectors().GetBinaryVector())) / bytesPerRow, nil
case schemapb.DataType_FloatVector:
dim := fieldData.GetVectors().GetDim()
if dim <= 0 {
return 0, fmt.Errorf("invalid float vector dimension %d for field %s", dim, fieldData.GetFieldName())
return 0, merr.WrapErrParameterInvalidMsg("invalid float vector dimension %d for field %s", dim, fieldData.GetFieldName())
}
return int64(len(fieldData.GetVectors().GetFloatVector().GetData())) / dim, nil
case schemapb.DataType_Float16Vector:
dim := fieldData.GetVectors().GetDim()
bytesPerRow := dim * 2
if bytesPerRow <= 0 {
return 0, fmt.Errorf("invalid float16 vector dimension %d for field %s", dim, fieldData.GetFieldName())
return 0, merr.WrapErrParameterInvalidMsg("invalid float16 vector dimension %d for field %s", dim, fieldData.GetFieldName())
}
return int64(len(fieldData.GetVectors().GetFloat16Vector())) / bytesPerRow, nil
case schemapb.DataType_BFloat16Vector:
dim := fieldData.GetVectors().GetDim()
bytesPerRow := dim * 2
if bytesPerRow <= 0 {
return 0, fmt.Errorf("invalid bfloat16 vector dimension %d for field %s", dim, fieldData.GetFieldName())
return 0, merr.WrapErrParameterInvalidMsg("invalid bfloat16 vector dimension %d for field %s", dim, fieldData.GetFieldName())
}
return int64(len(fieldData.GetVectors().GetBfloat16Vector())) / bytesPerRow, nil
case schemapb.DataType_SparseFloatVector:
@@ -2465,7 +2464,7 @@ func fieldDataValueCount(fieldData *schemapb.FieldData) (int64, error) {
case schemapb.DataType_Int8Vector:
dim := fieldData.GetVectors().GetDim()
if dim <= 0 {
return 0, fmt.Errorf("invalid int8 vector dimension %d for field %s", dim, fieldData.GetFieldName())
return 0, merr.WrapErrParameterInvalidMsg("invalid int8 vector dimension %d for field %s", dim, fieldData.GetFieldName())
}
return int64(len(fieldData.GetVectors().GetInt8Vector())) / dim, nil
case schemapb.DataType_ArrayOfStruct:
@@ -2479,10 +2478,10 @@ func fieldDataValueCount(fieldData *schemapb.FieldData) (int64, error) {
case schemapb.DataType_ArrayOfVector:
return int64(len(subs[0].GetVectors().GetVectorArray().GetData())), nil
default:
return 0, fmt.Errorf("unsupported struct sub-field type %s for field %s", subs[0].GetType(), fieldData.GetFieldName())
return 0, merr.WrapErrParameterInvalidMsg("unsupported struct sub-field type %s for field %s", subs[0].GetType(), fieldData.GetFieldName())
}
default:
return 0, fmt.Errorf("the type(%v) of field(%v) is not supported, use other sdk please", fieldData.GetType(), fieldData.GetFieldName())
return 0, merr.WrapErrParameterInvalidMsg("the type(%v) of field(%v) is not supported, use other sdk please", fieldData.GetType(), fieldData.GetFieldName())
}
}
@@ -2532,7 +2531,7 @@ func newFieldDataRowAccessor(fieldData *schemapb.FieldData) (*fieldDataRowAccess
return accessor, nil
}
if valueCount != validCount {
return nil, fmt.Errorf("field %s has %d valid rows, but data length is %d", fieldData.GetFieldName(), validCount, valueCount)
return nil, merr.WrapErrParameterInvalidMsg("field %s has %d valid rows, but data length is %d", fieldData.GetFieldName(), validCount, valueCount)
}
accessor.compactIndices = compactIndices
return accessor, nil
@@ -2543,7 +2542,7 @@ func (accessor *fieldDataRowAccessor) rowIndex(rowIdx int64) (int64, bool, error
return rowIdx, true, nil
}
if rowIdx >= int64(len(accessor.validData)) {
return 0, false, fmt.Errorf("row index %d out of range for field %s valid data length %d", rowIdx, accessor.fieldData.GetFieldName(), len(accessor.validData))
return 0, false, merr.WrapErrParameterInvalidMsg("row index %d out of range for field %s valid data length %d", rowIdx, accessor.fieldData.GetFieldName(), len(accessor.validData))
}
if !accessor.validData[rowIdx] {
return 0, false, nil
@@ -2575,7 +2574,7 @@ func buildQueryResp(rowsNum int64, needFields []string, fieldDataList []*schemap
stringPks := ids.GetStrId().GetData()
rowsNum = int64(len(stringPks))
default:
return nil, errors.New("the type of primary key(id) is not supported, use other sdk please")
return nil, merr.WrapErrParameterInvalidMsg("the type of primary key(id) is not supported, use other sdk please")
}
}
}
@@ -2716,7 +2715,7 @@ func buildQueryResp(rowsNum int64, needFields []string, fieldDataList []*schemap
stringPks := ids.GetStrId().GetData()
row[pkFieldName] = stringPks[i]
default:
return nil, errors.New("the type of primary key(id) is not supported, use other sdk please")
return nil, merr.WrapErrParameterInvalidMsg("the type of primary key(id) is not supported, use other sdk please")
}
}
if scores != nil && int64(len(scores)) > i {
@@ -2734,29 +2733,31 @@ func hasSearchAggregationResult(results *schemapb.SearchResultData) bool {
func buildSearchAggregationResp(results *schemapb.SearchResultData, enableInt64 bool, collectionSchema *schemapb.CollectionSchema) ([]gin.H, error) {
if results == nil {
return nil, errors.New("search_aggregation result is nil")
// The aggregation payload is produced by the server-side reduce, never
// by the request: a malformed shape is an internal contract violation.
return nil, merr.WrapErrServiceInternalMsg("search_aggregation result is nil")
}
aggTopks := results.GetAggTopks()
pbBuckets := results.GetAggBuckets()
if len(aggTopks) == 0 {
return nil, errors.New("search_aggregation response missing agg_topks")
return nil, merr.WrapErrServiceInternalMsg("search_aggregation response missing agg_topks")
}
if results.GetNumQueries() <= 0 {
return nil, errors.New("search_aggregation response missing nq")
return nil, merr.WrapErrServiceInternalMsg("search_aggregation response missing nq")
}
if len(aggTopks) != int(results.GetNumQueries()) {
return nil, fmt.Errorf("search_aggregation agg_topks length %d does not match nq %d", len(aggTopks), results.GetNumQueries())
return nil, merr.WrapErrServiceInternalMsg("search_aggregation agg_topks length %d does not match nq %d", len(aggTopks), results.GetNumQueries())
}
total := int64(0)
for _, topk := range aggTopks {
if topk < 0 {
return nil, errors.New("search_aggregation agg_topks cannot contain negative values")
return nil, merr.WrapErrServiceInternalMsg("search_aggregation agg_topks cannot contain negative values")
}
total += topk
}
if total != int64(len(pbBuckets)) {
return nil, fmt.Errorf("search_aggregation agg_topks sum %d does not match bucket count %d", total, len(pbBuckets))
return nil, merr.WrapErrServiceInternalMsg("search_aggregation agg_topks sum %d does not match bucket count %d", total, len(pbBuckets))
}
output := make([]gin.H, 0, len(aggTopks))
@@ -2778,7 +2779,7 @@ func buildSearchAggregationResp(results *schemapb.SearchResultData, enableInt64
func buildAggBucketResp(pb *schemapb.AggBucket, enableInt64 bool, collectionSchema *schemapb.CollectionSchema) (gin.H, error) {
if pb == nil {
return nil, errors.New("search_aggregation bucket is nil")
return nil, merr.WrapErrServiceInternalMsg("search_aggregation bucket is nil")
}
bucket := gin.H{
"key": buildAggBucketKeyResp(pb.GetKey(), enableInt64),
@@ -2986,7 +2987,7 @@ func convertConsistencyLevel(reqConsistencyLevel string) (commonpb.ConsistencyLe
if reqConsistencyLevel != "" {
level, ok := commonpb.ConsistencyLevel_value[reqConsistencyLevel]
if !ok {
return 0, false, merr.WrapErrParameterInvalidMsg(fmt.Sprintf("parameter:'%s' is incorrect, please check it", reqConsistencyLevel))
return 0, false, merr.WrapErrParameterInvalidMsg("parameter:'%s' is incorrect, please check it", reqConsistencyLevel)
}
return commonpb.ConsistencyLevel(level), false, nil
}
@@ -3096,7 +3097,7 @@ func convertDefaultValue(value interface{}, dataType schemapb.DataType) (*schema
}
return data, nil
default:
return nil, merr.WrapErrParameterInvalidMsg(fmt.Sprintf("Unexpected default value type: %s", dataType.String()))
return nil, merr.WrapErrParameterInvalidMsg("Unexpected default value type: %s", dataType.String())
}
}
@@ -3491,7 +3492,7 @@ func generateSearchParams(reqSearchParams map[string]interface{}) ([]*commonpb.K
for key, value := range reqSearchParams {
if val, ok := paramsMap[key]; ok {
if !deepEqual(val, value) {
return nil, merr.WrapErrParameterInvalidMsg(fmt.Sprintf("ambiguous parameter: %s, in search_param: %v, in search_param.params: %v", key, value, val))
return nil, merr.WrapErrParameterInvalidMsg("ambiguous parameter: %s, in search_param: %v, in search_param.params: %v", key, value, val)
}
} else if key != Params {
paramsMap[key] = value
@@ -25,6 +25,7 @@ import (
"strings"
"testing"
"github.com/cockroachdb/errors"
"github.com/gin-gonic/gin"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -36,6 +37,7 @@ import (
"github.com/milvus-io/milvus-proto/go-api/v3/schemapb"
"github.com/milvus-io/milvus/internal/json"
"github.com/milvus-io/milvus/pkg/v3/common"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/typeutil"
)
@@ -3005,7 +3007,8 @@ func TestBuildQueryResps(t *testing.T) {
}
_, err := buildQueryResp(int64(0), outputFields, newFieldData([]*schemapb.FieldData{}, 1000), generateIDs(schemapb.DataType_Int64, 3), DefaultScores, true, nil)
assert.Equal(t, "the type(1000) of field(wrong-field-type) is not supported, use other sdk please", err.Error())
assert.Contains(t, err.Error(), "the type(1000) of field(wrong-field-type) is not supported, use other sdk please")
assert.True(t, errors.Is(err, merr.ErrParameterInvalid))
res, err := buildQueryResp(int64(0), outputFields, []*schemapb.FieldData{}, generateIDs(schemapb.DataType_Int64, 3), DefaultScores, true, nil)
assert.Equal(t, 3, len(res))
@@ -17,15 +17,13 @@
package httpserver
import (
"fmt"
"github.com/cockroachdb/errors"
"google.golang.org/protobuf/proto"
"github.com/milvus-io/milvus-proto/go-api/v3/commonpb"
"github.com/milvus-io/milvus-proto/go-api/v3/milvuspb"
"github.com/milvus-io/milvus-proto/go-api/v3/schemapb"
"github.com/milvus-io/milvus/internal/json"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/typeutil"
)
@@ -70,7 +68,7 @@ type WrappedInsertRequest struct {
func (w *WrappedInsertRequest) AsInsertRequest() (*milvuspb.InsertRequest, error) {
fieldData, err := convertFieldDataArray(w.FieldsData)
if err != nil {
return nil, fmt.Errorf("%w: convert field data failed: %v", errBadRequest, err)
return nil, badRequestf(err, "convert field data failed")
}
return &milvuspb.InsertRequest{
Base: w.Base,
@@ -98,12 +96,12 @@ func (f *FieldData) makePbFloat16OrBfloat16Array(raw json.RawMessage, serializeF
return nil, 0, newFieldDataError(f.FieldName, err)
}
if len(wrappedData) < 1 {
return nil, 0, errors.New("at least one row for insert")
return nil, 0, merr.WrapErrParameterInvalidMsg("at least one row for insert")
}
array0 := wrappedData[0]
dim := len(array0)
if dim < 1 {
return nil, 0, errors.New("dim must >= 1")
return nil, 0, merr.WrapErrParameterInvalidMsg("dim must >= 1")
}
data := make([]byte, 0, len(wrappedData)*dim*2)
for _, fp32Array := range wrappedData {
@@ -238,12 +236,12 @@ func (f *FieldData) AsSchemapb() (*schemapb.FieldData, error) {
return nil, newFieldDataError(f.FieldName, err)
}
if len(wrappedData) < 1 {
return nil, errors.New("at least one row for insert")
return nil, merr.WrapErrParameterInvalidMsg("at least one row for insert")
}
array0 := wrappedData[0]
dim := len(array0)
if dim < 1 {
return nil, errors.New("dim must >= 1")
return nil, merr.WrapErrParameterInvalidMsg("dim must >= 1")
}
data := make([]float32, len(wrappedData)*dim)
@@ -299,7 +297,7 @@ func (f *FieldData) AsSchemapb() (*schemapb.FieldData, error) {
return nil, newFieldDataError(f.FieldName, err)
}
if len(wrappedData) < 1 {
return nil, errors.New("at least one row for insert")
return nil, merr.WrapErrParameterInvalidMsg("at least one row for insert")
}
data := make([][]byte, len(wrappedData))
dim := int64(0)
@@ -333,12 +331,12 @@ func (f *FieldData) AsSchemapb() (*schemapb.FieldData, error) {
return nil, newFieldDataError(f.FieldName, err)
}
if len(wrappedData) < 1 {
return nil, errors.New("at least one row for insert")
return nil, merr.WrapErrParameterInvalidMsg("at least one row for insert")
}
array0 := wrappedData[0]
dim := len(array0)
if dim < 1 {
return nil, errors.New("dim must >= 1")
return nil, merr.WrapErrParameterInvalidMsg("dim must >= 1")
}
data := make([]byte, len(wrappedData)*dim)
@@ -358,13 +356,13 @@ func (f *FieldData) AsSchemapb() (*schemapb.FieldData, error) {
},
}
default:
return nil, errors.New("unsupported data type")
return nil, merr.WrapErrParameterInvalidMsg("unsupported data type")
}
return &ret, nil
}
func newFieldDataError(field string, err error) error {
return fmt.Errorf("parse field[%s]: %s", field, err.Error())
return merr.WrapErrParameterInvalidErr(err, "parse field[%s]", field)
}
func convertFieldDataArray(input []*FieldData) ([]*schemapb.FieldData, error) {
@@ -25,10 +25,18 @@ import (
"github.com/gin-gonic/gin/binding"
"github.com/milvus-io/milvus-proto/go-api/v3/commonpb"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
)
var errBadRequest = errors.New("bad request")
// badRequestf wraps err with the package-internal errBadRequest sentinel and
// a contextual message, preserving the inner error chain. wrapHandler maps
// any error that matches errBadRequest to HTTP 400.
func badRequestf(err error, format string, args ...any) error {
return merr.Mark(merr.Wrapf(err, format, args...), errBadRequest)
}
// handlerFunc handles http request with gin context
type handlerFunc func(c *gin.Context) (interface{}, error)

Some files were not shown because too many files have changed in this diff Show More