enhance: standardize error handling on merr + Sys/Input classification (#50221)

issue: #47420

## What this PR does

Project-wide migration of raw `fmt.Errorf` / `errors.New` in function
bodies onto
the `merr` framework, plus the Sys-vs-Input error classification and the
machinery it drives (retriability, fine-grained metrics, segcore
unification),
plus the convention docs and a linter that keeps it from regressing.

Scope: storage, proxy, coordinators (root/data/query), query node, data
node,
`pkg/util` & `internal/util`, expression parser, message queue,
streaming, and
misc packages. Bare raw-error usages went from ~3000 to a ~340 allowlist
(package-level sentinels / build-tag / test sites).

---

## How to review this PR

It is large but the vast majority is mechanical. Changes fall into three
tiers;
spend review budget on Part 2 and Part 3.

### Part 1 — Mechanical standardization (low risk, verify by rule)

Each converted call follows one of a small fixed set of rules. To
review, check
that each site obeys the matching rule rather than reading every line:

| Pattern | Rule |
|---|---|
| `fmt.Errorf("...")` originating a new error | →
`merr.WrapErrXxxMsg("...")` with a code matching the failure's meaning |
| Adding context to an existing typed error | → `merr.Wrap(err, "...")`
/ `merr.Wrapf(...)` — **preserves** the inner code (never `WrapErr*Err`,
which overwrites it) |
| Errors inside the streaming subsystem | → `status.New*` factories
(StreamingError), **not** merr — this is the component-internal dialect
(see `docs/dev/error_handling_guide.md`) |
| Low-level / control-flow signal caught by `errors.Is` | → kept as a
package-level `errors.New` sentinel (lowercase, same-package) |

Conventions are documented in `docs/dev/error_handling_guide.md`
(how-to) and
`docs/dev/error_sentinel_convention.md` (rules + audit). A
`gocritic`/`ruleguard`
rule (`rawmerrerror`, in `rules.go`) enforces "no raw `return
errors.New/fmt.Errorf`"
under `make verifiers`.

### Part 2 — Behavior changes (review these closely)

These are the sites where the wire contract or runtime behavior changes,
not just
the source text. Listed by category; representative locations given,
full set in
the diff.

**A. gRPC wire-code shifts: `UnexpectedError(1)/Code 65535` → typed
code.**
Where a handler previously returned a raw error (collapsed to
`Code=65535` on the
wire), it now returns a typed merr, so the client sees a real code. The
most
common shift is to `IllegalArgument(5)/Code 1100` (ParameterInvalid).
Touch
points include datanode task handlers (CreateTask/Query/Drop), proxy
Upsert,
querynode GetMetrics, datacoord CreateIndex, httpserver query-response
builder,
and typeutil schema validation. One code refinement: an index-param
validation
moved `1100` → `1101` (ParameterMissing). **Client/SDK assertions and
any code
that switched on `Code=65535` for these paths must be re-checked** (the
go_client
e2e assertions were already aligned in this PR).

**B. Prometheus `status` label contract change (externally visible).**
The proxy metric's coarse `fail` / `rejected` values are split into
`fail_input` / `fail_system` and `rejected_user` / `rejected_system` (in
`requestutil.ParseMetricLabel`; auth/privilege rejections count as
`rejected_user`), so dashboards can attribute a failure to caller vs
operator.
**Dashboards/alerts querying `status="fail"` must migrate to
`status=~"fail_.*"`, and `status="rejected"` to
`status=~"rejected_.*"`.** The
in-repo Grafana dashboard is already migrated; external dashboards built
on the
old values silently go empty after upgrade. This is the one change that
requires an ops-side migration.

**C. Retriability semantics.**
- C1: `merr.Status(err)` now forces `Retriable=false` when the error is
an
`InputError` — a malformed request can never succeed on blind retry, so
clients
never get the self-contradictory "your input is wrong but you may
retry".
- C2: `retry.Do` short-circuits an `InputError` (non-retriable) — **but
only when
  the caller did not pass a `RetryErr` predicate**. The check is an
`if c.isRetryErr != nil { ... } else if InputError { ... }` *mutually
exclusive*
branch (`pkg/util/retry/retry.go`): an explicit `RetryErr` takes
precedence and
bypasses the InputError abort. `retry.Handle` deliberately does **not**
apply
the InputError abort (its callers signal abort via `shouldRetry=false`).
Four
flusher startup callsites that must retry through transient "not ready"
errors
  were given explicit `RetryErr` escape hatches.

**D. segcore (C++→Go) error classification.**
A single shared Go-side table (`pkg/util/merr/segcore.go`) maps each
segcore code
to a merr sentinel + InputError/signal category, replacing scattered
hand-written
`if errorCode == ...` switches in the cgo wrappers. **Wire `Code` values
change
for every segcore pass-through error, not just the remapped ones.**
Named
sentinels remap (C++ `2003` → merr `2001`, `2033` → `2002`,
Folly/Knowhere codes
likewise); **all remaining pass-through codes (`2004`–`2043`, previously
surfaced to clients as raw C++ enum values) now serialize as `2000`**
(`ErrSegcore`), with the original C++ code preserved in the `Reason`
text
(`segcoreCode=...`); unknown/future codes collapse to `2000` as well
(pinned by
the `wire_code_projection` test). Transient segcore classes (object
storage /
file IO / OOM / mmap / FieldNotLoaded — 11 codes) now report
`Retriable=true`.
**Any client switching on raw segcore codes in the `2004`–`2043` range
must be
re-checked**; the in-Reason code remains available for diagnostics.
Signal
codes (PretendFinished / FollyCancel) are recognized centrally.
`errors.Is`-based
control flow on these (e.g. scheduler skip/retry) is preserved.

**E. InputError classification (25 sentinels + dynamic marks).**
25 sentinels in `errors.go` carry `WithErrorType(InputError)` (the
Collection /
ResourceGroup / Database families, `ErrIndexDuplicate`,
`ErrParameterInvalid`,
`ErrPrivilegeNotAuthenticated`, `ErrImportFailed`, `ErrQueryPlan`, ...),
plus dynamic
marks for the 8 segcore input codes (ExprInvalid, DimNotMatch,
MetricTypeInvalid, FieldIDInvalid, ...) and
`WrapErrAsInputError`. The widest blast radius is `ErrParameterInvalid`
(1100):
~2335 `WrapErrParameterInvalid*` callsites now classify as input /
non-retriable. Because of C1/C2 this changes retriability for
any path that returns these. **The audit to confirm no transient path
was
mis-marked is the single most important review item** (see Part 3). One
reverse
correction: storage field-stats parsing moved from `ErrParameterInvalid`
(input)
to `ErrDataIntegrity` — a corrupted stored stat is data corruption, not
user
input.

### Part 3 — Known risks & traps (called out proactively)

1. **`merr.Wrap` vs `WrapErr*Err` (code-masking).** `WrapErr*Err` builds
a
`wrappedMilvusError{sentinel: ErrServiceInternal}` whose `code()`
returns the
*outer* sentinel — it overwrites the inner typed code and hides the
`errors.Is`
chain. This is intentional (use it to *deliberately* downgrade), but it
was a
recurring conversion defect; the rule "add context with `merr.Wrap`,
downgrade
with `WrapErr*Err`" is enforced by convention and reviewed across the
diff.
2. **InputError × `retry.Do` blast radius.** Marking a sentinel
`InputError` makes
any `retry.Do(...)` without a `RetryErr` predicate stop retrying it.
Reviewers
should sanity-check that no transient use of the 19 newly-marked
sentinels
(especially `ErrParameterInvalid`) sits inside a retry loop that needed
to keep
   spinning. The known flusher cases were handled (see C2).
3. **The ~340 raw-error allowlist.** What remains as bare `errors.New`
is, by
design: package-level sentinels (caught by `errors.Is`), `//go:build
test`
sites, and out-of-band trees (`cmd/`, `tests/`, codegen, walimpls). The
linter
only bans the *direct-return* form; assignment-then-return escapes and
the full
no-exceptions ban are deferred to an AST-based linter (Tier 2,
documented).
4. **segcore C++ second step deferred.** This PR unifies classification
on the Go
side; splitting the dual-semantic C++ codes at the source is a
follow-up.

---

## Validation

- `make verifiers`: Go side clean (gofmt + static-check across modules,
including
  the new `rawmerrerror` rule with a 0-hit baseline repo-wide).
- `make test-go`: passing; the one real regression introduced (a
datanode
`invalid_task_type` assertion shifting `1` → `5` from a ParameterInvalid
  conversion) was fixed in-tree.
- go_client e2e CreateIndex assertions aligned to the new merr messages.

---------

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
zhenshan.cao
2026-06-12 15:04:51 -07:00
committed by GitHub
co-authored by Claude Opus 4.7
parent 14c4cf65fc
commit e2787d3981
629 changed files with 7465 additions and 4797 deletions
+4 -4
View File
@@ -359,15 +359,15 @@ proxy:
remoteMaxTime: 0 # The time interval allowed for uploading access log files. If the upload time of a log file exceeds this interval, the file will be deleted. Setting the value to 0 disables this feature. remoteMaxTime: 0 # The time interval allowed for uploading access log files. If the upload time of a log file exceeds this interval, the file will be deleted. Setting the value to 0 disables this feature.
formatters: formatters:
base: base:
format: "[$time_now] [ACCESS] <$user_name: $user_addr> $method_name [status: $method_status] [code: $error_code] [sdk: $sdk_version] [msg: $error_msg] [traceID: $trace_id] [timeCost: $time_cost]" format: "[$time_now] [ACCESS] <$user_name: $user_addr> $method_name [status: $method_status] [code: $error_code] [sdk: $sdk_version] [msg: $error_msg] [traceID: $trace_id] [timeCost: $time_cost] [error_type: $error_type]"
query: query:
format: "[$time_now] [ACCESS] <$user_name: $user_addr> $method_name [status: $method_status] [code: $error_code] [sdk: $sdk_version] [msg: $error_msg] [traceID: $trace_id] [timeCost: $time_cost] [database: $database_name] [collection: $collection_name] [partitions: $partition_name] [expr: $method_expr] [params: $query_params]" format: "[$time_now] [ACCESS] <$user_name: $user_addr> $method_name [status: $method_status] [code: $error_code] [sdk: $sdk_version] [msg: $error_msg] [traceID: $trace_id] [timeCost: $time_cost] [database: $database_name] [collection: $collection_name] [partitions: $partition_name] [expr: $method_expr] [params: $query_params] [error_type: $error_type]"
methods: "Query, Delete, /query, /v2/vectordb/entities/query, /v2/vectordb/entities/get, /v2/vectordb/entities/delete" methods: "Query, Delete, /query, /v2/vectordb/entities/query, /v2/vectordb/entities/get, /v2/vectordb/entities/delete"
search: search:
format: "[$time_now] [ACCESS] <$user_name: $user_addr> $method_name [status: $method_status] [code: $error_code] [sdk: $sdk_version] [msg: $error_msg] [traceID: $trace_id] [timeCost: $time_cost] [database: $database_name] [collection: $collection_name] [partitions: $partition_name] [expr: $method_expr] [nq: $nq] [params: $search_params]" format: "[$time_now] [ACCESS] <$user_name: $user_addr> $method_name [status: $method_status] [code: $error_code] [sdk: $sdk_version] [msg: $error_msg] [traceID: $trace_id] [timeCost: $time_cost] [database: $database_name] [collection: $collection_name] [partitions: $partition_name] [expr: $method_expr] [nq: $nq] [params: $search_params] [error_type: $error_type]"
methods: "HybridSearch, Search, /search, /v2/vectordb/entities/search, /v2/vectordb/entities/advanced_search, /v2/vectordb/entities/hybrid_search" methods: "HybridSearch, Search, /search, /v2/vectordb/entities/search, /v2/vectordb/entities/advanced_search, /v2/vectordb/entities/hybrid_search"
upsert: upsert:
format: "[$time_now] [ACCESS] <$user_name: $user_addr> $method_name [status: $method_status] [code: $error_code] [sdk: $sdk_version] [msg: $error_msg] [traceID: $trace_id] [timeCost: $time_cost] [database: $database_name] [collection: $collection_name] [partitions: $partition_name] [partial_update: $partial_update]" format: "[$time_now] [ACCESS] <$user_name: $user_addr> $method_name [status: $method_status] [code: $error_code] [sdk: $sdk_version] [msg: $error_msg] [traceID: $trace_id] [timeCost: $time_cost] [database: $database_name] [collection: $collection_name] [partitions: $partition_name] [partial_update: $partial_update] [error_type: $error_type]"
methods: Upsert methods: Upsert
cacheSize: 0 # Size of log of write cache, in byte. (Close write cache if size was 0) cacheSize: 0 # Size of log of write cache, in byte. (Close write cache if size was 0)
cacheFlushInterval: 3 # time interval of auto flush write cache, in seconds. (Close auto flush if interval was 0) cacheFlushInterval: 3 # time interval of auto flush write cache, in seconds. (Close auto flush if interval was 0)
@@ -4794,7 +4794,7 @@
"uid": "${datasource}" "uid": "${datasource}"
}, },
"exemplar": true, "exemplar": true,
"expr": "sum(increase(milvus_proxy_req_count{app_kubernetes_io_instance=~\"$instance\", app_kubernetes_io_name=\"$app_name\", namespace=\"$namespace\", status=\"fail\"}[2m])/120) by(function_name, pod, node_id)", "expr": "sum(increase(milvus_proxy_req_count{app_kubernetes_io_instance=~\"$instance\", app_kubernetes_io_name=\"$app_name\", namespace=\"$namespace\", status=~\"fail_.*\"}[2m])/120) by(function_name, pod, node_id)",
"interval": "", "interval": "",
"legendFormat": "{{function_name}}-{{pod}}-{{node_id}}", "legendFormat": "{{function_name}}-{{pod}}-{{node_id}}",
"queryType": "randomWalk", "queryType": "randomWalk",
+314
View File
@@ -0,0 +1,314 @@
# Error Handling Guide
How to produce and return errors in Milvus server code — the day-to-day how-to.
For the underlying rules, the sentinel naming convention, and the enforcement
roadmap, see [error_sentinel_convention.md](./error_sentinel_convention.md). For
the canonical numeric code list, see the sentinel definitions in
[`pkg/util/merr/errors.go`](../../pkg/util/merr/errors.go). (The
[appendix_d_error_code.md](../developer_guides/appendix_d_error_code.md)
appendix predates merr and lists the **deprecated** `commonpb.ErrorCode` enum,
not the merr codes.)
## TL;DR
1. **Never** `return errors.New(...)` or `return fmt.Errorf(...)`. A linter
rejects it (see [The one rule](#the-one-rule-never-return-a-bare-error)).
2. An error that leaves your component must be a **typed** error carrying a code.
The lingua franca across Milvus components is **merr**.
3. Pick one of three forms: originate a typed merr, add context with `merr.Wrap`,
or — only if a caller branches on it by identity — a package-level sentinel.
## The mental model: three kinds of error, one boundary rule
Milvus has three legitimate kinds of error, distinguished by **how far they
travel**, not by syntax:
| Kind | Scope | Carrier | Example |
|---|---|---|---|
| **① merr** | across Milvus components & to clients | numeric code on the main gRPC wire | `merr.ErrCollectionNotFound` |
| **② component-internal dialect** | between sub-modules of one big component | that component's own typed error + its own wire | `streamingutil/status.StreamingError` |
| **③ internal sentinel** | within a single Go process | `errors.New` pointer identity, caught by `errors.Is` | `errSessionVersionCheckFailure` |
The single rule that ties them together:
> **The error you must return follows the interface's promise** — it is decided
> by *who is on the other side of the boundary you are crossing*, not by where
> the error was born.
- Crossing into **another Milvus component** (proxy → rootcoord), or returning to
a **client**: the promise is **merr**. Translate to a typed merr at that
boundary.
- Staying **inside one component** (e.g. the streaming sub-modules talking to
each other): the component may speak its own typed dialect (`StreamingError`).
It is **not** required to be merr — but it must still be *typed*, and it gets
translated to merr at the component's outer edge.
- On **no** wire, ever: a **bare** `errors.New` / `fmt.Errorf`. Internal
sentinels (kind ③) are bare, but they never reach a wire — they are caught by
`errors.Is` and translated first.
### Why component-internal dialects are allowed (the StreamingError case)
Streaming is one big component whose sub-modules (streamingnode, streamingcoord,
the streaming client) talk to each other constantly. They use
`streamingutil/status.StreamingError`, which has its **own** error codes and its
**own** gRPC encoding. That is deliberate: it is a *bounded context* with its own
vocabulary. There is intentionally **no** global "StreamingError → merr"
auto-converter — that would erase the dialect. Instead the conversion happens
**once, at the consumer's boundary**, and the consumer decides how, based on
what its own interface promises:
```go
// rootcoord consumes a streaming service inside CreateCollection.
err := s.streamingService.DoSomething(ctx, ...) // may return *StreamingError
if err != nil {
// (a) You care about the code the client sees → translate explicitly:
if se := status.AsStreamingError(err); se != nil && se.IsRateLimitRejected() {
return merr.WrapErrServiceRateLimit("streaming backpressure")
}
// (b) You don't care about a precise code → let it fall back at the
// boundary (see "The safety net"); the client gets a generic
// internal-class error. Prefer being explicit, but this is allowed.
return err
}
```
## The one rule: never return a bare error
```go
return errors.New("segment not loaded") // ❌ linter rejects
return fmt.Errorf("segment %d not loaded", id) // ❌ linter rejects
```
Why it is banned, even though the boundary would "fix it up" anyway: a bare error
that escapes to a gRPC boundary becomes `Code=65535 (Unexpected)` — visually
indistinguishable from "the server hit an unhandled bug". A wall of
`errors.New("reason 1")`, `errors.New("reason 2")` is a sign nobody planned the
error taxonomy: the caller cannot program against it, and it all collapses into
one opaque code on the wire. A typed error costs one extra word and makes the
failure *addressable*. The linter exists to build the habit: **the thing I
return is always typed.**
## Decision tree: what should I return?
```
Am I crossing into another component / returning to a client?
├─ No (staying inside my own component)
│ ├─ My component has its own typed dialect (e.g. streaming)?
│ │ → use that dialect's factory (status.New*), not merr, not errors.New
│ └─ Otherwise → a typed merr (rules below); or, only if a caller in THIS
│ process branches on the outcome by identity, a package-level sentinel (§3.3)
└─ Yes → I must return a typed merr:
├─ Brand-new failure, no underlying error worth carrying?
│ → merr.WrapErrXxxMsg("detail %s", v) (§3.1 originate)
│ and pick Input vs System deliberately (next section)
├─ I hold an underlying error and want to KEEP its code, just add context?
│ → merr.Wrap(err, "while doing X") (§3.2 add context)
└─ I hold an underlying error and want to DOWNGRADE it to a generic class?
→ merr.WrapErrServiceInternalErr(err, "...") (§3.2 — deliberate override)
```
## Input vs System: who is to blame?
Every merr is classified as **InputError** (the request author's fault) or
**SystemError** (Milvus's fault, the default). Choosing a factory chooses the
classification, so when you originate an error, ask one question first:
> **Would a correctly implemented Milvus ever hit this branch, given this
> request?** If the request content itself triggers it → InputError. If
> reaching this branch means a Milvus bug or internal failure → SystemError.
Quick rules for the cases that get misclassified in practice:
- A **plan / task type / request produced by a coordinator** is not user input.
An unrecognized task type or malformed compaction plan is an internal
protocol violation (think mixed-version rolling upgrade) →
`WrapErrServiceInternalMsg`, even though the check looks like validation.
- **Data produced by segcore or another internal component** is not user
input. A violated data-shape contract (ValidData length, truncated vectors)
is a Milvus bug → `WrapErrServiceInternalMsg`.
- A **TOCTOU race** (state was valid at check time, changed by execution time)
is not user input → keep it a system error.
### How classification is attached
Two mechanisms, used in different situations:
1. **Baked-in sentinels.** ~25 sentinels are declared
`WithErrorType(InputError)` in `errors.go` (`ErrParameterInvalid/Missing/
TooLarge`, `ErrPrivilegeNotPermitted`, `ErrDatabaseInvalidName`, ...).
Using their factory *is* the classification — which is exactly why reaching
for `WrapErrParameterInvalidMsg` to express an internal assertion is wrong.
2. **Boundary marking** for dual-use sentinels. `ErrCollectionNotFound` stays
SystemError (internal refresh/retry paths depend on that), and the proxy
boundary stamps it InputError only where the name came from the user:
```go
// proxy meta cache, the central chokepoint for user-supplied names:
return collection, merr.WrapErrAsInputErrorWhen(err,
merr.ErrCollectionNotFound, merr.ErrDatabaseNotFound)
```
`WrapErrAsInputError(err)` marks unconditionally;
`WrapErrAsInputErrorWhen(err, targets...)` marks only if the error's code
matches a target. Both preserve `errors.Is` and the code — they relabel the
classification, nothing else.
### What the classification drives
| Surface | InputError behavior |
|---|---|
| `commonpb.Status` | `ExtraInfo["is_input_error"]="true"`, `Retriable` forced `false` |
| Prometheus | request counted as `fail_input` / `rejected_user` (vs `fail_system` / `rejected_system`) |
| Access log / failure log | `error_type` field set accordingly |
| proxy lb_policy | **no cross-replica failover** — retrying a bad request elsewhere can't help |
| `retry.Do` | aborts immediately instead of retrying |
The last two rows are why misclassification is not cosmetic: marking an
internal failure as InputError disables the retry/failover machinery that
would have healed it, and a dashboard blames users for Milvus bugs.
### Pitfalls (each of these happened)
- **Don't mark a shared sentinel InputError globally** to fix one callsite —
every internal `retry.Do` loop waiting on that error stops retrying. Use
boundary marking instead.
- **Don't classify in a helper** what only the boundary can know. The same
not-found is the user's fault when the name came from a request, and a
system fault when it came from internal state — stamp at the chokepoint
where the origin is known.
- **"Looks like validation" is not the test.** Coordinator-to-node protocol
checks, segcore output checks, and cgo boundary checks all look like
validation; none of them are user input.
## The three correct ways
### 3.1 Originate a typed error — `WrapErrXxxMsg` / `WrapErrXxx`
When the failure starts here and there is no inner error worth preserving:
```go
return merr.WrapErrParameterInvalidMsg("nq (%d) exceeds the limit (%d)", nq, max)
return merr.WrapErrCollectionNotFound(collectionName)
```
Pick the sentinel whose **code** matches the failure's meaning (see the sentinel
definitions in [`pkg/util/merr/errors.go`](../../pkg/util/merr/errors.go)). This is the common
case: most of the time you only need to attach a message to a well-chosen code,
and the framework does the rest.
### 3.2 Add context but keep the code — `merr.Wrap`, never `WrapErr*Err`
When you already hold a typed error and only want to add a breadcrumb:
```go
if err := s.loadSegment(ctx, id); err != nil {
return merr.Wrap(err, "while loading sealed segment") // ✅ keeps the inner code
}
```
`merr.Wrap` / `merr.Wrapf` is a thin wrapper (like `errors.Wrap`): it prepends
context and **preserves** the inner error's code and its `errors.Is` chain.
⚠️ Do **not** reach for `merr.WrapErrServiceInternalErr(err, ...)` (or any
`merr.WrapErrXxxErr`) just to add context. Those build a
`wrappedMilvusError{sentinel: ErrServiceInternal}` whose `code()` returns the
**outer** sentinel's code — they **overwrite** the inner typed code with
ServiceInternal (5) and force it non-retriable (`As()` resolves to the outer
sentinel). The `errors.Is` chain itself survives via `Unwrap()`, but the typed
code and retriability are masked. `WrapErrXxxErr` is *only* for
when you **deliberately** want to relabel the inner error to a new code (e.g.
collapse a noisy internal failure into one ServiceInternal for the client).
This split is intentional, and the framework deliberately does **not** try to be
clever: *keep the code* versus *downgrade the code* is a decision you state, not
one the framework guesses. `merr.Wrap` **always** keeps the inner code;
`WrapErrXxxErr` **always** relabels to the outer one. A helper that "smartly"
preserved the inner code whenever it recognized a typed merr would blur the
intent — a reader could no longer tell from the call site whether the author
meant to preserve or to downgrade. The choice of helper *is* the statement of
intent.
### 3.2.1 The base-package case: pass through, wrap, or relabel?
Low-level packages (`pkg/...`, `internal/util/...`) sit under many callers and
usually receive an error from something even lower — etcd, S3, a third-party
library, another util. For every such error you must consciously pick one of
three. Getting this wrong in a base package is expensive: it is multiplied across
every caller.
| Choice | Use it when | How |
|---|---|---|
| **Pass through** the original `err` | the inner err is already a typed error meaningful to your caller, or your package has no business classifying it — let the caller decide | `return err` |
| **Wrap, keep the code** | you want to add a breadcrumb (which key/path/op failed) without changing what the error *means* | `merr.Wrapf(err, "etcd txn on key %s", k)` |
| **Relabel / downgrade** | the inner err is a leaky implementation / third-party detail your caller should not see; you translate it into the typed merr your package's interface promises | `merr.WrapErrXxxErr(err, "...")` |
Deciding questions, in order:
1. **Is the inner err already typed and meaningful to my caller?** → pass through.
2. **Does any caller `errors.Is` the inner err's identity?** → pass through or
`merr.Wrap` (both preserve the chain); **never relabel** — it hides the chain.
3. **Is the inner err a third-party / implementation detail I promise to hide?**
→ relabel to the typed merr my interface promises.
A base package that relabels too eagerly destroys codes the upper layers needed;
one that passes a raw third-party error straight through leaks an untyped error
toward the boundary. Neither is acceptable — the choice must be deliberate, and it
follows your package's interface promise, not convenience.
### 3.3 Need identity branching or reuse — a package-level sentinel
Define a sentinel when a caller **in the same process** branches on the outcome
by identity, via `errors.Is`:
```go
// internal/util/sessionutil/session_util.go — caught by isNotSessionVersionCheckFailure
// and used as a retry.Do predicate; the identity must survive, so it stays a
// bare sentinel rather than a merr error.
var errSessionVersionCheckFailure = errors.New("session version check failure")
```
Rules for sentinels (full version in
[error_sentinel_convention.md](./error_sentinel_convention.md)):
- **Package-level, never function-local.** A local `x := errors.New(...)` is a
refactor hazard — tomorrow it gets hoisted and silently crosses a boundary.
Lift it to a `var` at package scope.
- **Lowercase / unexported** when it lives in `internal/...`. If a
*cross-package* caller needs the signal, do **not** export the sentinel —
redesign the API to carry the signal in a return value (e.g.
`(ignored bool, err error)`).
- It must be `errors.Is`-caught and translated to a typed merr (or
`merr.Success()`) **before** crossing any gRPC boundary. A sentinel that
reaches the wire is just an opaque `Code=65535`.
When unsure between 3.1 and 3.3: if nobody does `errors.Is` on it, you don't need
a sentinel — just originate a typed merr with a message (3.1).
## The safety net: boundary fallback (and why not panic)
If a non-typed error does reach a gRPC handler, `merr.Status(err)` falls back to
`Code=65535 (Unexpected)`. This is a **backstop, not a feature**: it keeps the
server from leaking internals or crashing, but the client gets an opaque code.
Treat any `Code=65535` in logs as "someone forgot to type their error".
Why the boundary **falls back instead of panicking**: third-party libraries and
deep call chains produce errors on paths that tests cannot fully cover. Panicking
on "not a typed merr" would turn a stray untyped error into an outage. The
contract is therefore: **fall back to a generic code, never panic.** The goal of
the linter and this guide is to make that fallback path *empty in practice* — so
that every error a client sees was deliberately typed at its source.
## Enforcement
- A `gocritic`/`ruleguard` rule (`rawmerrerror` in `rules.go`) rejects
`return errors.New / fmt.Errorf / errors.Errorf` from function bodies. It runs
under `make verifiers` (via `static-check`), so no extra command is needed.
Exempt paths (run outside the request path): `*_test.go`, `cmd/`, `tests/`,
codegen, the walimpls harness, and `/mocks/` (generated mock helpers are
test infrastructure even though some lack a "Code generated" header).
- It catches the **direct-return** form — the one that lets a raw error escape to
a boundary. Assignment-then-return escapes (`e := errors.New(); return e`) and
the full no-exceptions ban require the AST-based linter described as "Tier 2"
in [error_sentinel_convention.md](./error_sentinel_convention.md).
+324
View File
@@ -0,0 +1,324 @@
# Error Sentinel Convention
Milvus error handling has two distinct layers. This document describes the rules
that separate them, the rationale, and an audit of where the codebase currently
violates them.
## The two layers
### 1. Typed merr (wire-protocol errors)
Defined in `pkg/util/merr/errors.go` (`ErrCollectionNotFound`,
`ErrParameterInvalid`, etc.). These carry a numeric error code that is
serialized into `commonpb.Status{ErrorCode, Reason}` and shipped to the
client over gRPC. They are the only thing a client (or another Milvus
component on the receiving end of an RPC) sees.
Creation: `merr.WrapErrXxxMsg(...)` / `merr.WrapErrXxxErr(cause, ...)`
(origination only — never to add context to a typed merr that already exists,
see `merr.Wrap`).
### 2. Internal sentinels (single-process control flow)
Created with `errors.New(...)` at package scope (e.g. `errIgnoredAlterAlias`,
`errReleaseCollectionNotLoaded`, `errNodeNotEnough`). These are signaling
vocabulary inside a single Go process: a callee tells its caller "this is an
idempotent no-op" or "the queue is empty" so the caller can branch / retry /
ignore. They are **not** part of the wire protocol.
The catcher is always `errors.Is(err, sentinelX)` at some boundary in the
calling stack, and the boundary either:
- translates to `merr.Success()` (idempotency: e.g. drop something that
doesn't exist → success), or
- translates to a typed merr `merr.WrapErrXxxMsg(...)` (e.g. user already
exists → `WrapErrParameterInvalid`).
## The hard invariant
> **Any internal sentinel must be `errors.Is`-caught and translated to a
> typed merr (or `Success`) before crossing any gRPC handler boundary —
> client-facing or component-to-component.**
Why "any gRPC boundary, not just client-facing": gRPC serializes errors to
`commonpb.Status{ErrorCode, Reason}`. The Go-level pointer identity that
`errors.New` sentinels rely on does not survive the wire. The peer's
`merr.Error(status)` reconstructs a typed merr from the numeric code; the
sentinel chain is gone forever. So a sentinel that escapes an internal coord
RPC is just as broken as one that escapes a user-facing RPC — only quieter,
because no customer sees the resulting `Code=65535 (unexpected)`.
### What breaks the invariant
The `errors.Is` chain survives `return err`, `errors.Wrap(err, "...")` /
`merr.Wrap(err, "...")` (cockroachdb thin wrap), **and**
`merr.WrapErrServiceInternalErr(err, "...")``milvusError.Unwrap()`
returns the inner error, so `errors.Is(outer, innerSentinel)` stays true through any
of them. It is **destroyed** only by:
- Putting the cause in a format argument instead of the chain:
`merr.WrapErrXxxMsg("...: %s", err)`, or the `%w` mistake (`WrapErr*Msg`
formats with `fmt.Sprintf`, which does **not** honor `%w` and renders
`%!w(...)`). The inner error reaches the message text but is unreachable via
`Unwrap()`, so `errors.Is(outer, innerSentinel)` returns false.
- Any custom wrapper that doesn't implement `Unwrap()`.
Do **not** confuse this with the separate code/retriability rule.
`merr.Wrap(err, ...)` and `merr.WrapErrXxxErr(err, ...)` both keep `errors.Is`
intact, but they differ in what the result reports at the boundary:
- `merr.Wrap(err, ...)` preserves the inner's `Code()` and `IsRetryable` — the
chain still resolves to the inner's `*milvusError`.
- `merr.WrapErrXxxErr(err, ...)` reports the **outer** sentinel's code and
retriability: `As()` resolves to `ErrServiceInternal` (Code 5,
non-retriable), **masking** the inner's classification.
So there are two distinct rules, often conflated:
1. To keep `errors.Is` working: never stuff the cause into a format string;
pass it as the error argument.
2. To add context **without changing the classification** (preserve the inner
code + retriability): use `merr.Wrap`, not `merr.WrapErr*Err`. Reserve
`merr.WrapErrXxxErr` for when you intend to *assert* a new classification
(e.g. this genuinely is a service-internal error). (See also
[feedback rule](#related-rules) on `merr.Wrap` vs `merr.WrapErr*Err`.)
## Naming convention
The convention has two layers, matched to the two error categories:
### Wire-protocol layer (typed merr) — uppercase `Err*` in `pkg/util/merr` only
All errors that may cross any gRPC boundary (client-facing or
component-to-component) must be `*merr.milvusError` defined in
`pkg/util/merr/errors.go`. They have:
- A numeric code passed to `newMilvusError(...)`; uniqueness is enforced by
the init-time code registry (defining a second sentinel on an occupied code
panics at package init, since `milvusError.Is` matches by code alone)
- A `var ErrXxx = newMilvusError(...)` declaration in `pkg/util/merr/errors.go`
- An exported `WrapErrXxxMsg` / `WrapErrXxxErr` helper
If an error needs to be visible to the wire, it lives here. No exceptions.
Every sentinel also carries an Input-vs-System classification (who is to
blame) that drives `Status.Retriable`, the `fail_input`/`fail_system` metric
labels, lb_policy failover and `retry.Do`; see "Input vs System: who is to
blame?" in [error_handling_guide.md](error_handling_guide.md).
### Internal-sentinel layer — lowercase `errXxx`, same-package only
Internal sentinels live in `internal/...` packages, are created with
`errors.New(...)`, and are **lowercase / unexported**. The rule:
> **A `var err* = errors.New(...)` declared in `internal/...` may only be
> referenced inside the same Go package. Cross-package consumers must not
> see it.**
This makes Go visibility do the enforcement: if you need a signal across
package boundaries, you either (a) lift it into the wire layer as a typed
merr, or (b) redesign the API so the signal flows via a return value
(e.g. `(ignored bool, err error)`), not via the error type.
Example (current code, after the 04-coord cleanup):
```go
// errFull / errNoSuchElement are INTERNAL sentinels: caught by errors.Is
// inside the compaction inspector / scheduler loop and never serialized
// across any gRPC boundary.
var (
errFull = errors.New("compaction queue is full")
errNoSuchElement = errors.New("compaction queue has no element")
)
```
### Cross-package idempotency: use a return-value flag, not an exported sentinel
The previous code exported `meta.ErrResourceGroupOperationIgnored` so that
the parent package `querycoordv2` could catch it via `errors.Is` and
translate to `merr.Success()`. This was an exported `Err*` in
`internal/...` — visually indistinguishable from a `merr.ErrXxx` typed
wire error, easy to misuse.
The current code instead encodes the signal in the return value:
```go
// meta/resource_manager.go
func (rm *ResourceManager) CheckIfResourceGroupAddable(...) (ignored bool, err error) {
if proto.Equal(rm.groups[rgName].GetConfig(), cfg) {
return true, nil // idempotent no-op
}
...
}
// querycoordv2/ddl_callbacks_alter_resource_group.go (broadcaster)
func (s *Server) broadcastCreateResourceGroup(...) (ignored bool, err error) {
if ignored, err := s.meta.CheckIfResourceGroupAddable(...); err != nil || ignored {
return ignored, err
}
...
}
// querycoordv2/services.go (RPC handler)
ignored, err := s.broadcastCreateResourceGroup(ctx, req)
if err != nil { return merr.Status(err), nil }
if ignored { return merr.Success(), nil }
```
No sentinel crosses the package boundary; the signal travels via a
structured return value. This is the preferred pattern for any new
cross-package idempotency case.
## Current state (audit done on err-std-04-coord branch, 2026-05-19)
Across `internal/{datacoord,rootcoord,querycoordv2}` there are 28
`errors.New(...)` sentinels. Their fates:
| Kind | Count | Examples | Status |
|---|---|---|---|
| Idempotency: caught → `merr.Success()` | 13 catch sites, ~12 distinct sentinels | `errIgnoredAlterAlias`, `errIgnoredCreateCollection`, `errReleaseCollectionNotLoaded`, `errUserNotFound`, ... | ✅ compliant |
| Caught → translated to typed merr | 3 catch sites (`errUserAlreadyExists`, `errRoleAlreadyExists`, `errRoleNotExists`) | client gets `WrapErrParameterInvalidMsg(...)` or `WrapErrServiceInternalMsg(...)` | ✅ compliant (1100 / 5) |
| Background-only (never enter an RPC handler) | 5 (`errFull`, `errNoSuchElement`, `errNodeNotEnough`, `errDisposed`, `errTypeNotFound`) | compaction queue / resource observer / session lifecycle / checker registry | ✅ compliant |
| Cross-package idempotency via `(ignored bool, err error)` signature | 1 (resource group create/drop) | meta layer returns `ignored=true`; querycoordv2 RPC handler translates to `merr.Success()` — no sentinel crosses package | ✅ compliant (refactored from exported `ErrResourceGroupOperationIgnored` in this branch) |
| Dead code (function with 0 callers) | 3 (`errNilResponse`, `errNilStatusResponse`, `errUnknownResponseType` — all only used by `VerifyResponse` in `datacoord/util.go`) | safe to delete | 🪦 cleanup candidate |
| ~~**Violation**: escapes RPC handler with no catch~~ (resolved) | 3 (`errEmptyUsername`, `errEmptyRoleName`, `errEmptyPrivilegeGroupName`) | origin sites in `meta_table.go` now emit `WrapErrParameterInvalidMsg` directly; the bare sentinels are gone | ✅ resolved |
| ~~**Semantic miscategorization**: caught and wrapped with the wrong typed merr code~~ (resolved) | 1 (`errTypeNotFound` in `ops_services.go:87,101`) | invalid `CheckerID` from the client now wrapped as `WrapErrParameterInvalidMsg` (code 1100), was `WrapErrServiceInternal` (code 5) | ✅ resolved |
### Cleanup status
1. ✅ Done — `errEmptyUsername` / `errEmptyRoleName` / `errEmptyPrivilegeGroupName`
at the origin sites in `internal/rootcoord/meta_table.go` now emit
`merr.WrapErrParameterInvalidMsg("username is empty")` etc. directly; the bare
sentinels no longer exist.
2. ✅ Done — `errTypeNotFound` at the catcher in
`internal/querycoordv2/ops_services.go:87,101` is now wrapped as
`merr.WrapErrParameterInvalidMsg("invalid checker type %d: %v", req.CheckerID, err)`.
3. Not done — delete `VerifyResponse` and its three dead-code sentinels
(`errNilResponse`, `errNilStatusResponse`, `errUnknownResponseType`).
## Future linter ideas
Three candidates, in order of how cheap they are to implement and how
hard the enforcement is. **Tier 1.5's return form is now implemented (see
below); Tier 1 and Tier 2 remain a design queue.**
### Tier 1 — exported-sentinel ban (1 hour to write)
The simplest rule: **`internal/...` packages may not declare exported
`var Err\w+ = errors.New(...)`.** Scan all `internal/...` *.go files,
fail CI if any match. Two paths to fix a violation:
1. Lowercase it (`var errXxx = errors.New(...)`) — only callable inside the
same package. If the lint fails because a cross-package caller needs the
signal, see fix 2.
2. Refactor the API so the signal travels via a return value
(e.g. add `ignored bool` to the return tuple) and delete the sentinel.
This makes Go visibility itself the enforcement mechanism: anything that
needs to look like `merr.ErrXxx` to a reviewer can only exist in
`pkg/util/merr`. Internal sentinels stay quietly lowercase in their owning
package.
Lowercase sentinels (`var errXxx = errors.New(...)`) inside `internal/...`
are still encouraged to carry an `INTERNAL: ...` doc comment for reviewer
context, but it's not enforced — the visibility rule already prevents the
worst-case (`Err*` collision with `merr.ErrXxx`).
### Tier 1.5 — bare-usage ban (1 hour grep, half-day AST for 100% precision)
> **Status — return form implemented.** The **return** form of this ban is now
> enforced by a `gocritic`/`ruleguard` rule (`rawmerrerror` in `rules.go`), run
> under `make verifiers`: it rejects `return errors.New / fmt.Errorf /
> errors.Errorf` from function bodies (package-level sentinels, `cmd/`, `tests/`,
> codegen and the walimpls harness exempt). The day-to-day guide is
> [error_handling_guide.md](./error_handling_guide.md). The **no-exceptions**
> form below (local `:=`, `panic(...)`, function argument) is *not* covered:
> ruleguard's DSL cannot match "a call anywhere in a function body but not in a
> `ValueSpec`", so the full ban still needs the AST-based Tier 2 linter.
**Hardest enforcement, no exceptions.** `internal/...` packages may not
use `errors.New(...)` or `errors.Errorf(...)` **inline inside a function
body**. The only legal site for these calls is a package-level
`var <Name> = errors.New(...)` (sentinel declaration).
Allow/deny matrix:
| Form | Location | Verdict |
|---|---|---|
| `var errInvalid = errors.New("invalid")` | package-level (file top / `var` block) | ✅ allowed |
| `var ( errA = ...; errB = ... )` | package-level `var` block | ✅ allowed |
| `return errors.New(...)` | function body | ❌ banned |
| `x := errors.New(...)` | function body local | ❌ banned |
| `panic(errors.New(...))` | function body | ❌ banned |
| `foo(errors.New(...))` | function body argument | ❌ banned |
**Why no exceptions** (even for "local break signal" / "log-only" /
"panic-bound" cases that look harmless today):
- Today's local var can be hoisted to package-level by tomorrow's refactor
and silently start crossing boundaries.
- A linter with exceptions needs AST-level wire-reachability analysis
(expensive); a no-exception linter is one grep.
- Forces authors to use the right primitive instead of `errors.New` as
a universal escape hatch:
- **break signal from a callback** → define a `type doneSignal struct{}`
that implements `error` and use `errors.As`. Intent is now in the type,
not in a string-keyed sentinel.
- **"unreachable" assertion** → just `panic(...)`. If caller already
does `if err != nil { panic(err) }`, fold it into the callee.
- **input validation / config validation** → `merr.WrapErrParameterInvalidMsg(...)`
or `status.NewInvalidArgument(...)` depending on layer.
Implementation: grep version covers ~95% true violations in ~1 hour.
AST version (go/analysis) covers the edge cases (e.g. `init()` body
assigning to a package var) but needs ~half a day. Start with grep,
upgrade if false-positive rate exceeds 5%.
```bash
# grep skeleton
grep -rnE 'errors\.(New|Errorf)\(' internal/ --include='*.go' \
| grep -v _test.go \
| grep -vE ':[0-9]+:\s*(var\s+)?[A-Za-z_]+\s*=\s*errors\.(New|Errorf)' \
| grep -vE ':[0-9]+:\s*[A-Za-z_]+\s+(\w+\s+)?=\s*errors\.(New|Errorf)'
# Any remaining line = violation
```
A `//nolint:err-bare` escape valve with a required justification comment
handles the genuine outliers (a few `init()` patterns, embedded `errors.Mark`
usage, etc.).
### Tier 2 — escape-path linter (~1 day, go/analysis based)
For every gRPC handler method (anything matching the
`internal/{rootcoord,datacoord,querycoordv2}/services.go,root_coord.go,*_handler.go`
pattern, return type `(*proto.XxxResponse, error)` or `(*commonpb.Status,
error)`), trace the err-return data-flow. Any error that:
- transitively originates from an `INTERNAL:`-tagged sentinel, **and**
- reaches a `return Status{Code: merr.Status(err)}` or `return err` without
passing through an `errors.Is(err, internalSentinelX) { ... }` branch,
is a violation. Report file:line of the leak.
This catches the actual invariant violation (the 3 RBAC empty sentinels
would have been flagged), not just naming hygiene. Requires AST analysis;
worth doing if the cost of one more silent `Code=1` to a client is high.
### Tier 3 (no longer necessary if Tier 2 is in place) — wrap-rule linter
Scan `merr\.WrapErr[A-Za-z]+Err\(err, ` (note: first arg `err`, not a
fresh string-only origination) and require the cause `err` to not itself be
a typed merr. This is what
[`feedback_merr_wrap_rule`](../../) already enforces by convention; Tier 2
would catch the symptom (sentinel escape) as a side effect.
## Related rules
- `feedback_merr_wrap_rule` (this repo's collaboration memory): "Add context
to an existing err with `merr.Wrap` / `merr.Wrapf`, never with
`merr.WrapErr*Err` — the latter masks the inner typed code and
retriability (the `errors.Is` chain itself is preserved via `Unwrap()`)."
- `project_errstd_autogen_defects`: three systematic defects in the
auto-generated `errors.Wrap → merr` conversion in this branch series;
defects #2 and #3 are direct consequences of violating the rules in
this document.
@@ -1,5 +1,14 @@
## Appendix D. Error Code ## Appendix D. Error Code
> **⚠️ Deprecated.** This appendix documents the legacy `commonpb.ErrorCode`
> enum (Milvus 2.0-era), which is no longer the source of truth for error codes.
> Current error codes are the `merr` sentinel codes (e.g. `ParameterInvalid`
> = 1100, `ServiceInternal` = 5, `FunctionFailed` = 2400) carried in
> `commonpb.Status.Code`. The canonical list is the sentinel definitions in
> [`pkg/util/merr/errors.go`](../../pkg/util/merr/errors.go); see also
> [error_handling_guide.md](../dev/error_handling_guide.md) and
> [error_sentinel_convention.md](../dev/error_sentinel_convention.md).
**ErrorCode** **ErrorCode**
```protobuf ```protobuf
+2 -2
View File
@@ -2,9 +2,9 @@ module github.com/milvus-io/milvus/examples/telemetry_e2e_test
go 1.26.4 go 1.26.4
require ( require (
github.com/milvus-io/milvus-proto/go-api/v2 v2.6.6-0.20260304062516-e7e54d966ef2 github.com/milvus-io/milvus-proto/go-api/v2 v2.6.6-0.20260323081523-53649783989c
github.com/milvus-io/milvus/client/v2 v2.6.4-0.20251104142533-a2ce70d25256 github.com/milvus-io/milvus/client/v2 v2.6.4-0.20251104142533-a2ce70d25256
google.golang.org/grpc v1.71.0 google.golang.org/grpc v1.79.3
) )
replace ( replace (
+2
View File
@@ -1,2 +1,4 @@
github.com/milvus-io/milvus-proto/go-api/v2 v2.6.6-0.20260304062516-e7e54d966ef2/go.mod h1:/6UT4zZl6awVeXLeE7UGDWZvXj3IWkRsh3mqsn0DiAs= github.com/milvus-io/milvus-proto/go-api/v2 v2.6.6-0.20260304062516-e7e54d966ef2/go.mod h1:/6UT4zZl6awVeXLeE7UGDWZvXj3IWkRsh3mqsn0DiAs=
github.com/milvus-io/milvus-proto/go-api/v2 v2.6.6-0.20260323081523-53649783989c/go.mod h1:/6UT4zZl6awVeXLeE7UGDWZvXj3IWkRsh3mqsn0DiAs=
google.golang.org/grpc v1.71.0/go.mod h1:H0GRtasmQOh9LkFoCPDu3ZrwUtD1YGE+b2vYBYd/8Ec= google.golang.org/grpc v1.71.0/go.mod h1:H0GRtasmQOh9LkFoCPDu3ZrwUtD1YGE+b2vYBYd/8Ec=
google.golang.org/grpc v1.79.3/go.mod h1:KmT0Kjez+0dde/v2j9vzwoAScgEPx/Bw1CYChhHLrHQ=
+11 -10
View File
@@ -8,6 +8,7 @@ import (
"github.com/milvus-io/milvus-proto/go-api/v3/schemapb" "github.com/milvus-io/milvus-proto/go-api/v3/schemapb"
"github.com/milvus-io/milvus/pkg/v3/proto/planpb" "github.com/milvus-io/milvus/pkg/v3/proto/planpb"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
) )
const ( const (
@@ -57,7 +58,7 @@ func isSupportedAggregateName(aggregateName string) bool {
func NewAggregate(aggregateName string, aggFieldID int64, originalName string, fieldType schemapb.DataType) ([]AggregateBase, error) { func NewAggregate(aggregateName string, aggFieldID int64, originalName string, fieldType schemapb.DataType) ([]AggregateBase, error) {
if !isSupportedAggregateName(aggregateName) { if !isSupportedAggregateName(aggregateName) {
return nil, fmt.Errorf("invalid Aggregation operator %s", aggregateName) return nil, merr.WrapErrParameterInvalidMsg("invalid Aggregation operator %s", aggregateName)
} }
if err := ValidateAggFieldType(aggregateName, fieldType); err != nil { if err := ValidateAggFieldType(aggregateName, fieldType); err != nil {
@@ -78,7 +79,7 @@ func NewAggregate(aggregateName string, aggFieldID int64, originalName string, f
return []AggregateBase{&MaxAggregate{fieldID: aggFieldID, originalName: originalName}}, nil return []AggregateBase{&MaxAggregate{fieldID: aggFieldID, originalName: originalName}}, nil
default: default:
// should never happen due to isSupportedAggregateName check // should never happen due to isSupportedAggregateName check
return nil, fmt.Errorf("invalid Aggregation operator %s", aggregateName) return nil, merr.WrapErrParameterInvalidMsg("invalid Aggregation operator %s", aggregateName)
} }
} }
@@ -93,13 +94,13 @@ func FromPB(pb *planpb.Aggregate) (AggregateBase, error) {
case planpb.AggregateOp_max: case planpb.AggregateOp_max:
return &MaxAggregate{fieldID: pb.GetFieldId()}, nil return &MaxAggregate{fieldID: pb.GetFieldId()}, nil
default: default:
return nil, fmt.Errorf("invalid Aggregation operator %d", pb.Op) return nil, merr.WrapErrParameterInvalidMsg("invalid Aggregation operator %d", pb.Op)
} }
} }
func AccumulateFieldValue(target *FieldValue, new *FieldValue) error { func AccumulateFieldValue(target *FieldValue, new *FieldValue) error {
if target == nil || new == nil { if target == nil || new == nil {
return fmt.Errorf("target or new field value is nil") return merr.WrapErrServiceInternalMsg("target or new field value is nil")
} }
// Skip null values during accumulation // Skip null values during accumulation
@@ -121,7 +122,7 @@ func AccumulateFieldValue(target *FieldValue, new *FieldValue) error {
} }
// Verify types match // Verify types match
if reflect.TypeOf(target.val) != reflect.TypeOf(new.val) { if reflect.TypeOf(target.val) != reflect.TypeOf(new.val) {
return fmt.Errorf("type mismatch: target is %T, new is %T", target.val, new.val) return merr.WrapErrParameterInvalidMsg("type mismatch: target is %T, new is %T", target.val, new.val)
} }
switch target.val.(type) { switch target.val.(type) {
@@ -136,7 +137,7 @@ func AccumulateFieldValue(target *FieldValue, new *FieldValue) error {
case float64: case float64:
target.val = target.val.(float64) + new.val.(float64) target.val = target.val.(float64) + new.val.(float64)
default: default:
return fmt.Errorf("unsupported type: %T", target.val) return merr.WrapErrParameterInvalidMsg("unsupported type: %T", target.val)
} }
return nil return nil
} }
@@ -255,17 +256,17 @@ func (bucket *Bucket) RowCount() int {
func (bucket *Bucket) Accumulate(row *Row, idx int, keyCount int, aggs []AggregateBase) error { func (bucket *Bucket) Accumulate(row *Row, idx int, keyCount int, aggs []AggregateBase) error {
if idx >= len(bucket.rows) || idx < 0 { if idx >= len(bucket.rows) || idx < 0 {
return fmt.Errorf("wrong idx:%d for bucket", idx) return merr.WrapErrParameterInvalidMsg("wrong idx:%d for bucket", idx)
} }
targetRow := bucket.rows[idx] targetRow := bucket.rows[idx]
if targetRow == nil { if targetRow == nil {
return fmt.Errorf("nil row at the target idx:%d, cannot accumulate the row", idx) return merr.WrapErrServiceInternalMsg("nil row at the target idx:%d, cannot accumulate the row", idx)
} }
if row.ColumnCount() != targetRow.ColumnCount() { if row.ColumnCount() != targetRow.ColumnCount() {
return fmt.Errorf("column count:%d in the row must be equal to the target row:%d", row.ColumnCount(), bucket.rows[idx].ColumnCount()) return merr.WrapErrParameterInvalidMsg("column count:%d in the row must be equal to the target row:%d", row.ColumnCount(), bucket.rows[idx].ColumnCount())
} }
if row.ColumnCount() != keyCount+len(aggs) { if row.ColumnCount() != keyCount+len(aggs) {
return fmt.Errorf("column count:%d in the row must be sum of keyCount:%d and the number of aggs:%d", row.ColumnCount(), keyCount, len(aggs)) return merr.WrapErrParameterInvalidMsg("column count:%d in the row must be sum of keyCount:%d and the number of aggs:%d", row.ColumnCount(), keyCount, len(aggs))
} }
for col := keyCount; col < row.ColumnCount(); col++ { for col := keyCount; col < row.ColumnCount(); col++ {
targetRow.UpdateFieldValue(row, col, aggs[col-keyCount]) targetRow.UpdateFieldValue(row, col, aggs[col-keyCount])
+23 -24
View File
@@ -2,7 +2,6 @@ package agg
import ( import (
"context" "context"
"fmt"
"github.com/milvus-io/milvus-proto/go-api/v3/schemapb" "github.com/milvus-io/milvus-proto/go-api/v3/schemapb"
typeutil2 "github.com/milvus-io/milvus/internal/util/typeutil" typeutil2 "github.com/milvus-io/milvus/internal/util/typeutil"
@@ -88,15 +87,15 @@ func (reducer *GroupAggReducer) EmptyAggResult() (*AggregationResult, error) {
} else { } else {
field, err := helper.GetFieldFromID(agg.GetFieldId()) field, err := helper.GetFieldFromID(agg.GetFieldId())
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to get field schema for aggregate fieldID %d: %w", agg.GetFieldId(), err) return nil, merr.Wrapf(err, "failed to get field schema for aggregate fieldID %d", agg.GetFieldId())
} }
resultType, err := getAggregateResultType(agg.GetOp(), field.GetDataType()) resultType, err := getAggregateResultType(agg.GetOp(), field.GetDataType())
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to get result type for aggregate fieldID %d: %w", agg.GetFieldId(), err) return nil, merr.Wrapf(err, "failed to get result type for aggregate fieldID %d", agg.GetFieldId())
} }
emptyFieldData, err := genEmptyFieldDataByType(resultType) emptyFieldData, err := genEmptyFieldDataByType(resultType)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to generate empty field data for result type %s: %w", resultType.String(), err) return nil, merr.Wrapf(err, "failed to generate empty field data for result type %s", resultType.String())
} }
ret.fieldDatas = append(ret.fieldDatas, emptyFieldData) ret.fieldDatas = append(ret.fieldDatas, emptyFieldData)
} }
@@ -160,7 +159,7 @@ func genEmptyFieldDataByType(dataType schemapb.DataType) (*schemapb.FieldData, e
return genEmptyLongFieldData(dataType, []int64{0}), nil return genEmptyLongFieldData(dataType, []int64{0}), nil
default: default:
// For other types, try to use the original field's GenEmptyFieldData // For other types, try to use the original field's GenEmptyFieldData
return nil, fmt.Errorf("unsupported data type for aggregate result: %s", dataType.String()) return nil, merr.WrapErrParameterInvalidMsg("unsupported data type for aggregate result: %s", dataType.String())
} }
} }
@@ -187,10 +186,10 @@ func getAggregateResultType(op planpb.AggregateOp, inputType schemapb.DataType)
case schemapb.DataType_Float, schemapb.DataType_Double: case schemapb.DataType_Float, schemapb.DataType_Double:
return schemapb.DataType_Double, nil return schemapb.DataType_Double, nil
default: default:
return schemapb.DataType_None, fmt.Errorf("unsupported input type %s for sum aggregation", inputType.String()) return schemapb.DataType_None, merr.WrapErrParameterInvalidMsg("unsupported input type %s for sum aggregation", inputType.String())
} }
default: default:
return schemapb.DataType_None, fmt.Errorf("unknown aggregate operator: %d", op) return schemapb.DataType_None, merr.WrapErrParameterInvalidMsg("unknown aggregate operator: %d", op)
} }
} }
@@ -201,12 +200,12 @@ func getAggregateResultType(op planpb.AggregateOp, inputType schemapb.DataType)
// 3. Each fieldData's Type matches the expected type from schema // 3. Each fieldData's Type matches the expected type from schema
func (reducer *GroupAggReducer) validateAggregationResults(results []*AggregationResult) error { func (reducer *GroupAggReducer) validateAggregationResults(results []*AggregationResult) error {
if reducer.schema == nil { if reducer.schema == nil {
return fmt.Errorf("schema is nil, cannot validate field types") return merr.WrapErrParameterInvalidMsg("schema is nil, cannot validate field types")
} }
helper, err := typeutil.CreateSchemaHelper(reducer.schema) helper, err := typeutil.CreateSchemaHelper(reducer.schema)
if err != nil { if err != nil {
return fmt.Errorf("failed to create schema helper: %w", err) return merr.Wrap(err, "failed to create schema helper")
} }
numGroupingKeys := len(reducer.groupByFieldIds) numGroupingKeys := len(reducer.groupByFieldIds)
@@ -220,7 +219,7 @@ func (reducer *GroupAggReducer) validateAggregationResults(results []*Aggregatio
for _, fieldID := range reducer.groupByFieldIds { for _, fieldID := range reducer.groupByFieldIds {
field, err := helper.GetFieldFromID(fieldID) field, err := helper.GetFieldFromID(fieldID)
if err != nil { if err != nil {
return fmt.Errorf("failed to get field schema for groupBy fieldID %d: %w", fieldID, err) return merr.Wrapf(err, "failed to get field schema for groupBy fieldID %d", fieldID)
} }
expectedTypes = append(expectedTypes, field.GetDataType()) expectedTypes = append(expectedTypes, field.GetDataType())
} }
@@ -234,11 +233,11 @@ func (reducer *GroupAggReducer) validateAggregationResults(results []*Aggregatio
} else { } else {
field, err := helper.GetFieldFromID(agg.GetFieldId()) field, err := helper.GetFieldFromID(agg.GetFieldId())
if err != nil { if err != nil {
return fmt.Errorf("failed to get field schema for aggregate fieldID %d: %w", agg.GetFieldId(), err) return merr.Wrapf(err, "failed to get field schema for aggregate fieldID %d", agg.GetFieldId())
} }
expectedType, err = getAggregateResultType(agg.GetOp(), field.GetDataType()) expectedType, err = getAggregateResultType(agg.GetOp(), field.GetDataType())
if err != nil { if err != nil {
return fmt.Errorf("failed to get aggregate result type for aggregate fieldID %d: %w", agg.GetFieldId(), err) return merr.Wrapf(err, "failed to get aggregate result type for aggregate fieldID %d", agg.GetFieldId())
} }
} }
expectedTypes = append(expectedTypes, expectedType) expectedTypes = append(expectedTypes, expectedType)
@@ -247,27 +246,27 @@ func (reducer *GroupAggReducer) validateAggregationResults(results []*Aggregatio
// Validate each result // Validate each result
for resultIdx, result := range results { for resultIdx, result := range results {
if result == nil { if result == nil {
return fmt.Errorf("result at index %d is nil", resultIdx) return merr.WrapErrServiceInternalMsg("result at index %d is nil", resultIdx)
} }
fieldDatas := result.GetFieldDatas() fieldDatas := result.GetFieldDatas()
// Check 1: fieldDatas length // Check 1: fieldDatas length
if len(fieldDatas) != expectedColumnCount { if len(fieldDatas) != expectedColumnCount {
return fmt.Errorf("result at index %d has fieldDatas length %d, expected %d (numGroupingKeys=%d, numAggs=%d)", return merr.WrapErrServiceInternalMsg("result at index %d has fieldDatas length %d, expected %d (numGroupingKeys=%d, numAggs=%d)",
resultIdx, len(fieldDatas), expectedColumnCount, numGroupingKeys, numAggs) resultIdx, len(fieldDatas), expectedColumnCount, numGroupingKeys, numAggs)
} }
// Check 2: no nil fieldData and Check 3: type matching // Check 2: no nil fieldData and Check 3: type matching
for colIdx, fieldData := range fieldDatas { for colIdx, fieldData := range fieldDatas {
if fieldData == nil { if fieldData == nil {
return fmt.Errorf("result at index %d has nil fieldData at column %d", resultIdx, colIdx) return merr.WrapErrServiceInternalMsg("result at index %d has nil fieldData at column %d", resultIdx, colIdx)
} }
expectedType := expectedTypes[colIdx] expectedType := expectedTypes[colIdx]
actualType := fieldData.GetType() actualType := fieldData.GetType()
if actualType != expectedType { if actualType != expectedType {
return fmt.Errorf("result at index %d, column %d has type %s, expected %s", return merr.WrapErrServiceInternalMsg("result at index %d, column %d has type %s, expected %s",
resultIdx, colIdx, schemapb.DataType_name[int32(actualType)], schemapb.DataType_name[int32(expectedType)]) resultIdx, colIdx, schemapb.DataType_name[int32(actualType)], schemapb.DataType_name[int32(expectedType)])
} }
} }
@@ -352,15 +351,15 @@ func (reducer *GroupAggReducer) Reduce(ctx context.Context, results []*Aggregati
limitReached := false limitReached := false
for _, result := range results { for _, result := range results {
if result == nil { if result == nil {
return nil, fmt.Errorf("input result from any sources cannot be nil") return nil, merr.WrapErrServiceInternalMsg("input result from any sources cannot be nil")
} }
reducedResult.allRetrieveCount += result.GetAllRetrieveCount() reducedResult.allRetrieveCount += result.GetAllRetrieveCount()
fieldDatas := result.GetFieldDatas() fieldDatas := result.GetFieldDatas()
if outputColumnCount != len(fieldDatas) { if outputColumnCount != len(fieldDatas) {
return nil, fmt.Errorf("retrieved results from different segments have different size of columns") return nil, merr.WrapErrServiceInternalMsg("retrieved results from different segments have different size of columns")
} }
if outputColumnCount == 0 { if outputColumnCount == 0 {
return nil, fmt.Errorf("retrieved results have no column data") return nil, merr.WrapErrServiceInternalMsg("retrieved results have no column data")
} }
rowCount := -1 rowCount := -1
for i := 0; i < outputColumnCount; i++ { for i := 0; i < outputColumnCount; i++ {
@@ -374,11 +373,11 @@ func (reducer *GroupAggReducer) Reduce(ctx context.Context, results []*Aggregati
rowCount = hashers[i].RowCount() rowCount = hashers[i].RowCount()
} else if i < numGroupingKeys { } else if i < numGroupingKeys {
if rowCount != hashers[i].RowCount() { if rowCount != hashers[i].RowCount() {
return nil, fmt.Errorf("field data:%d for different columns have different row count, %d vs %d, wrong state", return nil, merr.WrapErrServiceInternalMsg("field data:%d for different columns have different row count, %d vs %d, wrong state",
i, rowCount, hashers[i].RowCount()) i, rowCount, hashers[i].RowCount())
} }
} else if rowCount != accumulators[i-numGroupingKeys].RowCount() { } else if rowCount != accumulators[i-numGroupingKeys].RowCount() {
return nil, fmt.Errorf("field data:%d for different columns have different row count, %d vs %d, wrong state", return nil, merr.WrapErrServiceInternalMsg("field data:%d for different columns have different row count, %d vs %d, wrong state",
i, rowCount, accumulators[i-numGroupingKeys].RowCount()) i, rowCount, accumulators[i-numGroupingKeys].RowCount())
} }
} }
@@ -443,9 +442,9 @@ func (reducer *GroupAggReducer) Reduce(ctx context.Context, results []*Aggregati
} }
} }
if totalGroupCount > maxGroupByGroups { if totalGroupCount > maxGroupByGroups {
return nil, merr.WrapErrServiceInternal(fmt.Sprintf("GROUP BY produced too many groups (%d). "+ return nil, merr.WrapErrParameterInvalidMsg("GROUP BY produced too many groups (%d). "+
"Add filters or increase common.groupBy.maxGroups (current: %d)", "Add filters or increase common.groupBy.maxGroups (current: %d)",
totalGroupCount, maxGroupByGroups)) totalGroupCount, maxGroupByGroups)
} }
// Don't guarantee specific groups to be returned before milvus support order by // Don't guarantee specific groups to be returned before milvus support order by
} }
@@ -478,7 +477,7 @@ func SegcoreResults2AggResult(results []*segcorepb.RetrieveResults) ([]*Aggregat
aggResults := make([]*AggregationResult, len(results)) aggResults := make([]*AggregationResult, len(results))
for i := 0; i < len(results); i++ { for i := 0; i < len(results); i++ {
if results[i] == nil { if results[i] == nil {
return nil, fmt.Errorf("input segcore query results from any sources cannot be nil") return nil, merr.WrapErrServiceInternalMsg("input segcore query results from any sources cannot be nil")
} }
fieldsData := results[i].GetFieldsData() fieldsData := results[i].GetFieldsData()
allRetrieveCount := results[i].GetAllRetrieveCount() allRetrieveCount := results[i].GetAllRetrieveCount()
+20 -19
View File
@@ -9,6 +9,7 @@ import (
"unsafe" "unsafe"
"github.com/milvus-io/milvus-proto/go-api/v3/schemapb" "github.com/milvus-io/milvus-proto/go-api/v3/schemapb"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
) )
func NewFieldAccessor(fieldType schemapb.DataType) (FieldAccessor, error) { func NewFieldAccessor(fieldType schemapb.DataType) (FieldAccessor, error) {
@@ -28,7 +29,7 @@ func NewFieldAccessor(fieldType schemapb.DataType) (FieldAccessor, error) {
case schemapb.DataType_Double: case schemapb.DataType_Double:
return newFloat64FieldAccessor(), nil return newFloat64FieldAccessor(), nil
default: default:
return nil, fmt.Errorf("unsupported data type for hasher") return nil, merr.WrapErrParameterInvalidMsg("unsupported data type for hasher")
} }
} }
@@ -410,7 +411,7 @@ func AssembleSingleValue(fv *FieldValue, fieldData *schemapb.FieldData) error {
case schemapb.DataType_VarChar, schemapb.DataType_String: case schemapb.DataType_VarChar, schemapb.DataType_String:
fieldData.GetScalars().GetStringData().Data = append(fieldData.GetScalars().GetStringData().GetData(), "") fieldData.GetScalars().GetStringData().Data = append(fieldData.GetScalars().GetStringData().GetData(), "")
default: default:
return fmt.Errorf("unsupported DataType:%d", fieldData.GetType()) return merr.WrapErrParameterInvalidMsg("unsupported DataType:%d", fieldData.GetType())
} }
return nil return nil
} }
@@ -421,47 +422,47 @@ func AssembleSingleValue(fv *FieldValue, fieldData *schemapb.FieldData) error {
case schemapb.DataType_Bool: case schemapb.DataType_Bool:
boolVal, ok := val.(bool) boolVal, ok := val.(bool)
if !ok { if !ok {
return fmt.Errorf("type assertion failed: expected bool, got %T", val) return merr.WrapErrServiceInternalMsg("type assertion failed: expected bool, got %T", val)
} }
fieldData.GetScalars().GetBoolData().Data = append(fieldData.GetScalars().GetBoolData().GetData(), boolVal) fieldData.GetScalars().GetBoolData().Data = append(fieldData.GetScalars().GetBoolData().GetData(), boolVal)
case schemapb.DataType_Int8, schemapb.DataType_Int16, schemapb.DataType_Int32: case schemapb.DataType_Int8, schemapb.DataType_Int16, schemapb.DataType_Int32:
intVal, ok := val.(int32) intVal, ok := val.(int32)
if !ok { if !ok {
return fmt.Errorf("type assertion failed: expected int32, got %T", val) return merr.WrapErrServiceInternalMsg("type assertion failed: expected int32, got %T", val)
} }
fieldData.GetScalars().GetIntData().Data = append(fieldData.GetScalars().GetIntData().GetData(), intVal) fieldData.GetScalars().GetIntData().Data = append(fieldData.GetScalars().GetIntData().GetData(), intVal)
case schemapb.DataType_Int64: case schemapb.DataType_Int64:
int64Val, ok := val.(int64) int64Val, ok := val.(int64)
if !ok { if !ok {
return fmt.Errorf("type assertion failed: expected int64, got %T", val) return merr.WrapErrServiceInternalMsg("type assertion failed: expected int64, got %T", val)
} }
fieldData.GetScalars().GetLongData().Data = append(fieldData.GetScalars().GetLongData().GetData(), int64Val) fieldData.GetScalars().GetLongData().Data = append(fieldData.GetScalars().GetLongData().GetData(), int64Val)
case schemapb.DataType_Timestamptz: case schemapb.DataType_Timestamptz:
timestampVal, ok := val.(int64) timestampVal, ok := val.(int64)
if !ok { if !ok {
return fmt.Errorf("type assertion failed: expected int64 for Timestamptz, got %T", val) return merr.WrapErrServiceInternalMsg("type assertion failed: expected int64 for Timestamptz, got %T", val)
} }
fieldData.GetScalars().GetTimestamptzData().Data = append(fieldData.GetScalars().GetTimestamptzData().GetData(), timestampVal) fieldData.GetScalars().GetTimestamptzData().Data = append(fieldData.GetScalars().GetTimestamptzData().GetData(), timestampVal)
case schemapb.DataType_Float: case schemapb.DataType_Float:
floatVal, ok := val.(float32) floatVal, ok := val.(float32)
if !ok { if !ok {
return fmt.Errorf("type assertion failed: expected float32, got %T", val) return merr.WrapErrServiceInternalMsg("type assertion failed: expected float32, got %T", val)
} }
fieldData.GetScalars().GetFloatData().Data = append(fieldData.GetScalars().GetFloatData().GetData(), floatVal) fieldData.GetScalars().GetFloatData().Data = append(fieldData.GetScalars().GetFloatData().GetData(), floatVal)
case schemapb.DataType_Double: case schemapb.DataType_Double:
doubleVal, ok := val.(float64) doubleVal, ok := val.(float64)
if !ok { if !ok {
return fmt.Errorf("type assertion failed: expected float64, got %T", val) return merr.WrapErrServiceInternalMsg("type assertion failed: expected float64, got %T", val)
} }
fieldData.GetScalars().GetDoubleData().Data = append(fieldData.GetScalars().GetDoubleData().GetData(), doubleVal) fieldData.GetScalars().GetDoubleData().Data = append(fieldData.GetScalars().GetDoubleData().GetData(), doubleVal)
case schemapb.DataType_VarChar, schemapb.DataType_String: case schemapb.DataType_VarChar, schemapb.DataType_String:
stringVal, ok := val.(string) stringVal, ok := val.(string)
if !ok { if !ok {
return fmt.Errorf("type assertion failed: expected string, got %T", val) return merr.WrapErrServiceInternalMsg("type assertion failed: expected string, got %T", val)
} }
fieldData.GetScalars().GetStringData().Data = append(fieldData.GetScalars().GetStringData().GetData(), stringVal) fieldData.GetScalars().GetStringData().Data = append(fieldData.GetScalars().GetStringData().GetData(), stringVal)
default: default:
return fmt.Errorf("unsupported DataType:%d", fieldData.GetType()) return merr.WrapErrParameterInvalidMsg("unsupported DataType:%d", fieldData.GetType())
} }
return nil return nil
} }
@@ -544,13 +545,13 @@ func NewAggregationFieldMap(originalUserOutputFields []string, groupByFields []s
// 2. Global aggregation (no GROUP BY): output_fields can only contain aggregation expressions // 2. Global aggregation (no GROUP BY): output_fields can only contain aggregation expressions
// (e.g., "SELECT count(*), int64 FROM t" is invalid SQL — cannot mix aggregates with raw columns) // (e.g., "SELECT count(*), int64 FROM t" is invalid SQL — cannot mix aggregates with raw columns)
if numGroupingKeys > 0 { if numGroupingKeys > 0 {
return nil, fmt.Errorf( return nil, merr.WrapErrParameterInvalidMsg(
"output field '%s' is not allowed: when using GROUP BY, output_fields can only contain "+ "output field '%s' is not allowed: when using GROUP BY, output_fields can only contain "+
"group_by fields (%v) or aggregation expressions", "group_by fields (%v) or aggregation expressions",
outputField, groupByFields, outputField, groupByFields,
) )
} }
return nil, fmt.Errorf( return nil, merr.WrapErrParameterInvalidMsg(
"output field '%s' is not allowed: when using aggregation functions (e.g., count(*)), "+ "output field '%s' is not allowed: when using aggregation functions (e.g., count(*)), "+
"output_fields can only contain aggregation expressions, not regular columns", "output_fields can only contain aggregation expressions, not regular columns",
outputField, outputField,
@@ -566,14 +567,14 @@ func NewAggregationFieldMap(originalUserOutputFields []string, groupByFields []s
// and returns a new Double FieldData containing the average values. // and returns a new Double FieldData containing the average values.
func ComputeAvgFromSumAndCount(sumFieldData *schemapb.FieldData, countFieldData *schemapb.FieldData) (*schemapb.FieldData, error) { func ComputeAvgFromSumAndCount(sumFieldData *schemapb.FieldData, countFieldData *schemapb.FieldData) (*schemapb.FieldData, error) {
if sumFieldData == nil || countFieldData == nil { if sumFieldData == nil || countFieldData == nil {
return nil, fmt.Errorf("sumFieldData and countFieldData cannot be nil") return nil, merr.WrapErrServiceInternalMsg("sumFieldData and countFieldData cannot be nil")
} }
sumType := sumFieldData.GetType() sumType := sumFieldData.GetType()
countType := countFieldData.GetType() countType := countFieldData.GetType()
if countType != schemapb.DataType_Int64 { if countType != schemapb.DataType_Int64 {
return nil, fmt.Errorf("count field must be Int64 type, got %s", countType.String()) return nil, merr.WrapErrParameterInvalidMsg("count field must be Int64 type, got %s", countType.String())
} }
countData := countFieldData.GetScalars().GetLongData().GetData() countData := countFieldData.GetScalars().GetLongData().GetData()
@@ -598,27 +599,27 @@ func ComputeAvgFromSumAndCount(sumFieldData *schemapb.FieldData, countFieldData
case schemapb.DataType_Int64: case schemapb.DataType_Int64:
sumData := sumFieldData.GetScalars().GetLongData().GetData() sumData := sumFieldData.GetScalars().GetLongData().GetData()
if len(sumData) != rowCount { if len(sumData) != rowCount {
return nil, fmt.Errorf("sum and count field data must have the same length, got sum:%d, count:%d", len(sumData), rowCount) return nil, merr.WrapErrParameterInvalidMsg("sum and count field data must have the same length, got sum:%d, count:%d", len(sumData), rowCount)
} }
for i := 0; i < rowCount; i++ { for i := 0; i < rowCount; i++ {
if countData[i] == 0 { if countData[i] == 0 {
return nil, fmt.Errorf("division by zero: count is 0 at row %d", i) return nil, merr.WrapErrParameterInvalidMsg("division by zero: count is 0 at row %d", i)
} }
resultData = append(resultData, float64(sumData[i])/float64(countData[i])) resultData = append(resultData, float64(sumData[i])/float64(countData[i]))
} }
case schemapb.DataType_Double: case schemapb.DataType_Double:
sumData := sumFieldData.GetScalars().GetDoubleData().GetData() sumData := sumFieldData.GetScalars().GetDoubleData().GetData()
if len(sumData) != rowCount { if len(sumData) != rowCount {
return nil, fmt.Errorf("sum and count field data must have the same length, got sum:%d, count:%d", len(sumData), rowCount) return nil, merr.WrapErrParameterInvalidMsg("sum and count field data must have the same length, got sum:%d, count:%d", len(sumData), rowCount)
} }
for i := 0; i < rowCount; i++ { for i := 0; i < rowCount; i++ {
if countData[i] == 0 { if countData[i] == 0 {
return nil, fmt.Errorf("division by zero: count is 0 at row %d", i) return nil, merr.WrapErrParameterInvalidMsg("division by zero: count is 0 at row %d", i)
} }
resultData = append(resultData, sumData[i]/float64(countData[i])) resultData = append(resultData, sumData[i]/float64(countData[i]))
} }
default: default:
return nil, fmt.Errorf("unsupported sum field type for avg computation: %s", sumType.String()) return nil, merr.WrapErrParameterInvalidMsg("unsupported sum field type for avg computation: %s", sumType.String())
} }
result.GetScalars().GetDoubleData().Data = resultData result.GetScalars().GetDoubleData().Data = resultData
+18 -19
View File
@@ -1,14 +1,13 @@
package agg package agg
import ( import (
"fmt"
"github.com/milvus-io/milvus/pkg/v3/proto/planpb" "github.com/milvus-io/milvus/pkg/v3/proto/planpb"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
) )
func terminateSingleSlot(slots []*FieldValue) (any, error) { func terminateSingleSlot(slots []*FieldValue) (any, error) {
if len(slots) != 1 { if len(slots) != 1 {
return nil, fmt.Errorf("aggregate expects 1 accumulator slot, got %d", len(slots)) return nil, merr.WrapErrParameterInvalidMsg("aggregate expects 1 accumulator slot, got %d", len(slots))
} }
if slots[0] == nil || slots[0].IsNull() { if slots[0] == nil || slots[0].IsNull() {
return nil, nil return nil, nil
@@ -18,7 +17,7 @@ func terminateSingleSlot(slots []*FieldValue) (any, error) {
func terminateAvg(slots []*FieldValue) (any, error) { func terminateAvg(slots []*FieldValue) (any, error) {
if len(slots) != 2 { if len(slots) != 2 {
return nil, fmt.Errorf("avg expects 2 accumulator slots, got %d", len(slots)) return nil, merr.WrapErrParameterInvalidMsg("avg expects 2 accumulator slots, got %d", len(slots))
} }
sum := slots[0] sum := slots[0]
count := slots[1] count := slots[1]
@@ -52,7 +51,7 @@ func fieldValueToFloat64(v any) (float64, error) {
case float64: case float64:
return value, nil return value, nil
default: default:
return 0, fmt.Errorf("avg expects numeric accumulator, got %T", v) return 0, merr.WrapErrParameterInvalidMsg("avg expects numeric accumulator, got %T", v)
} }
} }
@@ -80,7 +79,7 @@ func (sum *SumAggregate) NewState() []*FieldValue {
func (sum *SumAggregate) UpdateState(slots []*FieldValue, new *FieldValue) error { func (sum *SumAggregate) UpdateState(slots []*FieldValue, new *FieldValue) error {
if len(slots) != 1 { if len(slots) != 1 {
return fmt.Errorf("aggregate expects 1 accumulator slot, got %d", len(slots)) return merr.WrapErrParameterInvalidMsg("aggregate expects 1 accumulator slot, got %d", len(slots))
} }
return sum.Update(slots[0], new) return sum.Update(slots[0], new)
} }
@@ -121,7 +120,7 @@ func (count *CountAggregate) NewState() []*FieldValue {
func (count *CountAggregate) UpdateState(slots []*FieldValue, new *FieldValue) error { func (count *CountAggregate) UpdateState(slots []*FieldValue, new *FieldValue) error {
if len(slots) != 1 { if len(slots) != 1 {
return fmt.Errorf("aggregate expects 1 accumulator slot, got %d", len(slots)) return merr.WrapErrParameterInvalidMsg("aggregate expects 1 accumulator slot, got %d", len(slots))
} }
if new == nil || new.IsNull() { if new == nil || new.IsNull() {
return nil return nil
@@ -166,7 +165,7 @@ func (avg *AvgAggregate) Name() string {
} }
func (avg *AvgAggregate) Update(target *FieldValue, new *FieldValue) error { func (avg *AvgAggregate) Update(target *FieldValue, new *FieldValue) error {
return fmt.Errorf("avg aggregate updates require aggregate state slots") return merr.WrapErrServiceInternalMsg("avg aggregate updates require aggregate state slots")
} }
func (avg *AvgAggregate) NewState() []*FieldValue { func (avg *AvgAggregate) NewState() []*FieldValue {
@@ -175,7 +174,7 @@ func (avg *AvgAggregate) NewState() []*FieldValue {
func (avg *AvgAggregate) UpdateState(slots []*FieldValue, new *FieldValue) error { func (avg *AvgAggregate) UpdateState(slots []*FieldValue, new *FieldValue) error {
if len(slots) != 2 { if len(slots) != 2 {
return fmt.Errorf("avg expects 2 accumulator slots, got %d", len(slots)) return merr.WrapErrParameterInvalidMsg("avg expects 2 accumulator slots, got %d", len(slots))
} }
if err := avg.sum.Update(slots[0], new); err != nil { if err := avg.sum.Update(slots[0], new); err != nil {
return err return err
@@ -204,7 +203,7 @@ func (avg *AvgAggregate) OriginalName() string {
func updateOrderedState(target *FieldValue, new *FieldValue, op string) error { func updateOrderedState(target *FieldValue, new *FieldValue, op string) error {
if target == nil || new == nil { if target == nil || new == nil {
return fmt.Errorf("target or new field value is nil") return merr.WrapErrServiceInternalMsg("target or new field value is nil")
} }
if new.IsNull() { if new.IsNull() {
return nil return nil
@@ -234,41 +233,41 @@ func shouldReplaceOrderedValue(target, new any, op string) (bool, error) {
case int: case int:
newVal, ok := new.(int) newVal, ok := new.(int)
if !ok { if !ok {
return false, fmt.Errorf("type mismatch: target is int, new is %T", new) return false, merr.WrapErrParameterInvalidMsg("type mismatch: target is int, new is %T", new)
} }
return shouldReplaceOrdered(newVal, targetVal, op), nil return shouldReplaceOrdered(newVal, targetVal, op), nil
case int32: case int32:
newVal, ok := new.(int32) newVal, ok := new.(int32)
if !ok { if !ok {
return false, fmt.Errorf("type mismatch: target is int32, new is %T", new) return false, merr.WrapErrParameterInvalidMsg("type mismatch: target is int32, new is %T", new)
} }
return shouldReplaceOrdered(newVal, targetVal, op), nil return shouldReplaceOrdered(newVal, targetVal, op), nil
case int64: case int64:
newVal, ok := new.(int64) newVal, ok := new.(int64)
if !ok { if !ok {
return false, fmt.Errorf("type mismatch: target is int64, new is %T", new) return false, merr.WrapErrParameterInvalidMsg("type mismatch: target is int64, new is %T", new)
} }
return shouldReplaceOrdered(newVal, targetVal, op), nil return shouldReplaceOrdered(newVal, targetVal, op), nil
case float32: case float32:
newVal, ok := new.(float32) newVal, ok := new.(float32)
if !ok { if !ok {
return false, fmt.Errorf("type mismatch: target is float32, new is %T", new) return false, merr.WrapErrParameterInvalidMsg("type mismatch: target is float32, new is %T", new)
} }
return shouldReplaceOrdered(newVal, targetVal, op), nil return shouldReplaceOrdered(newVal, targetVal, op), nil
case float64: case float64:
newVal, ok := new.(float64) newVal, ok := new.(float64)
if !ok { if !ok {
return false, fmt.Errorf("type mismatch: target is float64, new is %T", new) return false, merr.WrapErrParameterInvalidMsg("type mismatch: target is float64, new is %T", new)
} }
return shouldReplaceOrdered(newVal, targetVal, op), nil return shouldReplaceOrdered(newVal, targetVal, op), nil
case string: case string:
newVal, ok := new.(string) newVal, ok := new.(string)
if !ok { if !ok {
return false, fmt.Errorf("type mismatch: target is string, new is %T", new) return false, merr.WrapErrParameterInvalidMsg("type mismatch: target is string, new is %T", new)
} }
return shouldReplaceOrdered(newVal, targetVal, op), nil return shouldReplaceOrdered(newVal, targetVal, op), nil
default: default:
return false, fmt.Errorf("unsupported type for %s aggregation: %T", op, target) return false, merr.WrapErrParameterInvalidMsg("unsupported type for %s aggregation: %T", op, target)
} }
} }
@@ -299,7 +298,7 @@ func (min *MinAggregate) NewState() []*FieldValue {
func (min *MinAggregate) UpdateState(slots []*FieldValue, new *FieldValue) error { func (min *MinAggregate) UpdateState(slots []*FieldValue, new *FieldValue) error {
if len(slots) != 1 { if len(slots) != 1 {
return fmt.Errorf("aggregate expects 1 accumulator slot, got %d", len(slots)) return merr.WrapErrParameterInvalidMsg("aggregate expects 1 accumulator slot, got %d", len(slots))
} }
return min.Update(slots[0], new) return min.Update(slots[0], new)
} }
@@ -339,7 +338,7 @@ func (max *MaxAggregate) NewState() []*FieldValue {
func (max *MaxAggregate) UpdateState(slots []*FieldValue, new *FieldValue) error { func (max *MaxAggregate) UpdateState(slots []*FieldValue, new *FieldValue) error {
if len(slots) != 1 { if len(slots) != 1 {
return fmt.Errorf("aggregate expects 1 accumulator slot, got %d", len(slots)) return merr.WrapErrParameterInvalidMsg("aggregate expects 1 accumulator slot, got %d", len(slots))
} }
return max.Update(slots[0], new) return max.Update(slots[0], new)
} }
+2 -3
View File
@@ -1,9 +1,8 @@
package agg package agg
import ( import (
"fmt"
"github.com/milvus-io/milvus-proto/go-api/v3/schemapb" "github.com/milvus-io/milvus-proto/go-api/v3/schemapb"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
) )
func isSupportedAggFieldType(aggregateName string, dt schemapb.DataType) bool { func isSupportedAggFieldType(aggregateName string, dt schemapb.DataType) bool {
@@ -51,7 +50,7 @@ func isSupportedAggFieldType(aggregateName string, dt schemapb.DataType) bool {
// Note: operator name is expected to be normalized (lowercase) by caller. // Note: operator name is expected to be normalized (lowercase) by caller.
func ValidateAggFieldType(aggregateName string, dt schemapb.DataType) error { func ValidateAggFieldType(aggregateName string, dt schemapb.DataType) error {
if !isSupportedAggFieldType(aggregateName, dt) { if !isSupportedAggFieldType(aggregateName, dt) {
return fmt.Errorf("aggregation operator %s does not support data type %s", aggregateName, dt.String()) return merr.WrapErrParameterInvalidMsg("aggregation operator %s does not support data type %s", aggregateName, dt.String())
} }
return nil return nil
} }
+4 -7
View File
@@ -18,14 +18,13 @@ package allocator
import ( import (
"context" "context"
"fmt"
"sync" "sync"
"time" "time"
"github.com/cockroachdb/errors"
"go.uber.org/zap" "go.uber.org/zap"
"github.com/milvus-io/milvus/pkg/v3/log" "github.com/milvus-io/milvus/pkg/v3/log"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
) )
const ( const (
@@ -249,10 +248,9 @@ func (ta *CachedAllocator) finishSyncRequest() {
func (ta *CachedAllocator) failRemainRequest() { func (ta *CachedAllocator) failRemainRequest() {
var err error var err error
if ta.SyncErr != nil { if ta.SyncErr != nil {
err = fmt.Errorf("%s failRemainRequest err:%w", ta.Role, ta.SyncErr) err = merr.Wrapf(ta.SyncErr, "%s failRemainRequest", ta.Role)
} else { } else {
errMsg := fmt.Sprintf("%s failRemainRequest unexpected error", ta.Role) err = merr.WrapErrServiceInternalMsg("%s failRemainRequest unexpected error", ta.Role)
err = errors.New(errMsg)
} }
if len(ta.ToDoReqs) > 0 { if len(ta.ToDoReqs) > 0 {
log.Warn("Allocator has some reqs to fail", log.Warn("Allocator has some reqs to fail",
@@ -290,8 +288,7 @@ func (ta *CachedAllocator) Close() {
ta.CancelFunc() ta.CancelFunc()
ta.wg.Wait() ta.wg.Wait()
ta.TChan.Close() ta.TChan.Close()
errMsg := fmt.Sprintf("%s is closing", ta.Role) ta.revokeRequest(merr.WrapErrServiceInternalMsg("%s is closing", ta.Role))
ta.revokeRequest(errors.New(errMsg))
} }
// CleanCache is used to force synchronize with global allocator. // CleanCache is used to force synchronize with global allocator.
+3 -5
View File
@@ -18,14 +18,12 @@ package allocator
import ( import (
"context" "context"
"fmt"
"time" "time"
"github.com/cockroachdb/errors"
"github.com/milvus-io/milvus-proto/go-api/v3/commonpb" "github.com/milvus-io/milvus-proto/go-api/v3/commonpb"
"github.com/milvus-io/milvus/pkg/v3/proto/rootcoordpb" "github.com/milvus-io/milvus/pkg/v3/proto/rootcoordpb"
"github.com/milvus-io/milvus/pkg/v3/util/commonpbutil" "github.com/milvus-io/milvus/pkg/v3/util/commonpbutil"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
) )
const ( const (
@@ -101,7 +99,7 @@ func (ia *IDAllocator) syncID() (bool, error) {
cancel() cancel()
if err != nil { if err != nil {
return false, fmt.Errorf("syncID Failed:%w", err) return false, merr.Wrapf(err, "syncID failed")
} }
ia.idStart = resp.GetID() ia.idStart = resp.GetID()
ia.idEnd = ia.idStart + int64(resp.GetCount()) ia.idEnd = ia.idStart + int64(resp.GetCount())
@@ -148,7 +146,7 @@ func (ia *IDAllocator) AllocOne() (UniqueID, error) {
// Alloc allocates the id of the count number. // Alloc allocates the id of the count number.
func (ia *IDAllocator) Alloc(count uint32) (UniqueID, UniqueID, error) { func (ia *IDAllocator) Alloc(count uint32) (UniqueID, UniqueID, error) {
if ia.closed() { if ia.closed() {
return 0, 0, errors.New("fail to allocate ID, closed allocator") return 0, 0, merr.WrapErrServiceInternalMsg("fail to allocate ID, closed allocator")
} }
req := &IDRequest{BaseRequest: BaseRequest{Done: make(chan error), Valid: false}} req := &IDRequest{BaseRequest: BaseRequest{Done: make(chan error), Valid: false}}
+4 -3
View File
@@ -17,8 +17,9 @@
package allocator package allocator
import ( import (
"fmt"
"sync" "sync"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
) )
// localAllocator implements the Interface. // localAllocator implements the Interface.
@@ -40,12 +41,12 @@ func NewLocalAllocator(start, end int64) Interface {
func (a *localAllocator) Alloc(count uint32) (int64, int64, error) { func (a *localAllocator) Alloc(count uint32) (int64, int64, error) {
cnt := int64(count) cnt := int64(count)
if cnt <= 0 { if cnt <= 0 {
return 0, 0, fmt.Errorf("non-positive count is not allowed, count=%d", cnt) return 0, 0, merr.WrapErrParameterInvalidMsg("non-positive count is not allowed, count=%d", cnt)
} }
a.mu.Lock() a.mu.Lock()
defer a.mu.Unlock() defer a.mu.Unlock()
if a.idStart+cnt > a.idEnd { if a.idStart+cnt > a.idEnd {
return 0, 0, fmt.Errorf("ID is exhausted, start=%d, end=%d, count=%d", a.idStart, a.idEnd, cnt) return 0, 0, merr.WrapErrServiceInternalMsg("ID is exhausted, start=%d, end=%d, count=%d", a.idStart, a.idEnd, cnt)
} }
start := a.idStart start := a.idStart
a.idStart += cnt a.idStart += cnt
+3 -3
View File
@@ -20,7 +20,6 @@ import (
"context" "context"
"crypto/tls" "crypto/tls"
"github.com/cockroachdb/errors"
"go.uber.org/zap" "go.uber.org/zap"
"google.golang.org/grpc" "google.golang.org/grpc"
@@ -28,6 +27,7 @@ import (
"github.com/milvus-io/milvus-proto/go-api/v3/milvuspb" "github.com/milvus-io/milvus-proto/go-api/v3/milvuspb"
"github.com/milvus-io/milvus/client/v2/milvusclient" "github.com/milvus-io/milvus/client/v2/milvusclient"
"github.com/milvus-io/milvus/pkg/v3/log" "github.com/milvus-io/milvus/pkg/v3/log"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/paramtable" "github.com/milvus-io/milvus/pkg/v3/util/paramtable"
) )
@@ -52,7 +52,7 @@ func NewMilvusClient(ctx context.Context, cluster *commonpb.MilvusCluster) (Milv
// Build TLS config from per-cluster paramtable config. // Build TLS config from per-cluster paramtable config.
tlsConfig, err := buildCDCTLSConfig(cluster.GetClusterId()) tlsConfig, err := buildCDCTLSConfig(cluster.GetClusterId())
if err != nil { if err != nil {
return nil, errors.Wrap(err, "failed to build CDC TLS config") return nil, merr.Wrap(err, "failed to build CDC TLS config")
} }
if tlsConfig != nil { if tlsConfig != nil {
config.WithTLSConfig(tlsConfig) config.WithTLSConfig(tlsConfig)
@@ -67,7 +67,7 @@ func NewMilvusClient(ctx context.Context, cluster *commonpb.MilvusCluster) (Milv
cli, err := milvusclient.New(ctx, config) cli, err := milvusclient.New(ctx, config)
if err != nil { if err != nil {
return nil, errors.Wrap(err, "failed to create milvus client") return nil, merr.Wrap(err, "failed to create milvus client")
} }
return cli, nil return cli, nil
} }
+2 -2
View File
@@ -5,11 +5,11 @@ import (
"fmt" "fmt"
"time" "time"
"github.com/cockroachdb/errors"
clientv3 "go.etcd.io/etcd/client/v3" clientv3 "go.etcd.io/etcd/client/v3"
"google.golang.org/protobuf/proto" "google.golang.org/protobuf/proto"
"github.com/milvus-io/milvus/pkg/v3/proto/streamingpb" "github.com/milvus-io/milvus/pkg/v3/proto/streamingpb"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
) )
const requestTimeout = 30 * time.Second const requestTimeout = 30 * time.Second
@@ -44,7 +44,7 @@ func ListReplicatePChannels(ctx context.Context, etcdCli *clientv3.Client, prefi
meta := &streamingpb.ReplicatePChannelMeta{} meta := &streamingpb.ReplicatePChannelMeta{}
err = proto.Unmarshal([]byte(value), meta) err = proto.Unmarshal([]byte(value), meta)
if err != nil { if err != nil {
return nil, errors.Wrapf(err, "unmarshal replicate pchannel meta %s failed", keys[k]) return nil, merr.Wrapf(err, "unmarshal replicate pchannel meta %s failed", keys[k])
} }
channel := &ReplicateChannel{ channel := &ReplicateChannel{
Key: keys[k], Key: keys[k],
@@ -35,6 +35,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/streaming/util/message" "github.com/milvus-io/milvus/pkg/v3/streaming/util/message"
"github.com/milvus-io/milvus/pkg/v3/streaming/util/message/adaptor" "github.com/milvus-io/milvus/pkg/v3/streaming/util/message/adaptor"
"github.com/milvus-io/milvus/pkg/v3/streaming/util/options" "github.com/milvus-io/milvus/pkg/v3/streaming/util/options"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/paramtable" "github.com/milvus-io/milvus/pkg/v3/util/paramtable"
"github.com/milvus-io/milvus/pkg/v3/util/syncutil" "github.com/milvus-io/milvus/pkg/v3/util/syncutil"
) )
@@ -202,7 +203,7 @@ func (r *channelReplicator) getReplicateCheckpoint() (*utility.ReplicateCheckpoi
} }
replicateInfo, err := r.targetClient.GetReplicateInfo(ctx, req) replicateInfo, err := r.targetClient.GetReplicateInfo(ctx, req)
if err != nil { if err != nil {
return nil, errors.Wrap(err, "failed to get replicate info") return nil, merr.Wrap(err, "failed to get replicate info")
} }
checkpoint := replicateInfo.GetCheckpoint() checkpoint := replicateInfo.GetCheckpoint()
@@ -115,7 +115,7 @@ func (q *msgQueue) Enqueue(ctx context.Context, msg message.ImmutableMessage) er
// Optional runtime check: enforce non-decreasing timetick // Optional runtime check: enforce non-decreasing timetick
// if n := len(q.buf); n > 0 { // if n := len(q.buf); n > 0 {
// if q.buf[n-1].Timetick() > msg.Timetick() { // if q.buf[n-1].Timetick() > msg.Timetick() {
// return fmt.Errorf("enqueue timetick order violation: last=%d new=%d", q.buf[n-1].Timetick(), msg.Timetick()) // return merr.WrapErrParameterInvalidMsg("enqueue timetick order violation: last=%d new=%d", q.buf[n-1].Timetick(), msg.Timetick())
// } // }
// } // }
@@ -165,7 +165,7 @@ func (m *FileResourceObserver) CheckNodeSynced(nodeID int64) bool {
func (m *FileResourceObserver) CheckAllQnReady() error { func (m *FileResourceObserver) CheckAllQnReady() error {
// return error if meta is not ready // return error if meta is not ready
if m.meta == nil { if m.meta == nil {
return merr.WrapErrParameterInvalidMsg("rootcoord meta is not ready") return merr.WrapErrServiceUnavailable("rootcoord meta is not ready")
} }
resources, version := m.meta.ListFileResource(m.ctx) resources, version := m.meta.ListFileResource(m.ctx)
@@ -177,7 +177,7 @@ func (m *FileResourceObserver) CheckAllQnReady() error {
var err error var err error
m.distribution.Range(func(nodeID int64, node *NodeInfo) bool { m.distribution.Range(func(nodeID int64, node *NodeInfo) bool {
if node.NodeType == QueryNode && node.Version < version { if node.NodeType == QueryNode && node.Version < version {
err = merr.WrapErrParameterInvalidMsg("node %d file resource not synced", nodeID) err = merr.WrapErrServiceUnavailableMsg("file resource not synced, node-%d", nodeID)
return false return false
} }
return true return true
+8 -8
View File
@@ -298,10 +298,10 @@ func (s *mixCoordImpl) getQueryNodes(ctx context.Context) (*querypb.ListQueryNod
Base: commonpbutil.NewMsgBase(), Base: commonpbutil.NewMsgBase(),
}) })
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to list query nodes: %w", err) return nil, merr.Wrapf(err, "failed to list query nodes")
} }
if !merr.Ok(resp.GetStatus()) { if !merr.Ok(resp.GetStatus()) {
return nil, fmt.Errorf("failed to list query nodes: %s", resp.GetStatus().GetReason()) return nil, merr.Wrapf(merr.Error(resp.GetStatus()), "failed to list query nodes")
} }
return resp, nil return resp, nil
} }
@@ -514,7 +514,7 @@ func (s *mixCoordImpl) getQueryCoordChannelBalanceActive(ctx context.Context) (b
} }
if !merr.Ok(resp.GetStatus()) { if !merr.Ok(resp.GetStatus()) {
return channelActivate, fmt.Errorf("%s", resp.GetStatus().GetReason()) return channelActivate, merr.Error(resp.GetStatus())
} }
return channelActivate && resp.IsActive, nil return channelActivate && resp.IsActive, nil
@@ -528,7 +528,7 @@ func (s *mixCoordImpl) getQueryCoordBalanceActive(ctx context.Context) (bool, er
return false, err return false, err
} }
if !merr.Ok(resp.GetStatus()) { if !merr.Ok(resp.GetStatus()) {
return false, fmt.Errorf("%s", resp.GetStatus().GetReason()) return false, merr.Error(resp.GetStatus())
} }
return resp.IsActive, nil return resp.IsActive, nil
} }
@@ -569,7 +569,7 @@ func (s *mixCoordImpl) controlQueryCoordChannelBalanceStatus(ctx context.Context
default: default:
// If the status is not recognized, return an informative error immediately. // If the status is not recognized, return an informative error immediately.
// This avoids proceeding with an invalid state. // This avoids proceeding with an invalid state.
err = fmt.Errorf("invalid status value: '%s'. Use 'suspended', 'resumed' or 'active'", status) err = merr.WrapErrParameterInvalidMsg("invalid status value: '%s'. Use 'suspended', 'resumed' or 'active'", status)
} }
// --- Unified Error Handling --- // --- Unified Error Handling ---
@@ -578,7 +578,7 @@ func (s *mixCoordImpl) controlQueryCoordChannelBalanceStatus(ctx context.Context
// First, check if there was a low-level error during the method execution (e.g., network issue). // First, check if there was a low-level error during the method execution (e.g., network issue).
if err != nil { if err != nil {
// Wrap the original error with more context. // Wrap the original error with more context.
return fmt.Errorf("%s: %w", errMsg, err) return merr.Wrap(err, errMsg)
} }
// If no errors were encountered, the operation was successful. Return nil to indicate success. // If no errors were encountered, the operation was successful. Return nil to indicate success.
return nil return nil
@@ -958,11 +958,11 @@ func (s *mixCoordImpl) handleQueryNodeStatusUpdate(ctx context.Context, nodeID i
errMsg = "failed to resume node" errMsg = "failed to resume node"
default: default:
errMsg = "invalid status value. Use 'suspended', 'resumed' or 'active'" errMsg = "invalid status value. Use 'suspended', 'resumed' or 'active'"
err = fmt.Errorf("%s", errMsg) err = merr.WrapErrParameterInvalidMsg("%s", errMsg)
} }
err = merr.CheckRPCCall(resp, err) err = merr.CheckRPCCall(resp, err)
if err != nil { if err != nil {
err = fmt.Errorf("%s: %w", errMsg, err) err = merr.Wrap(err, errMsg)
} }
return err return err
} }
@@ -12,6 +12,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/log" "github.com/milvus-io/milvus/pkg/v3/log"
"github.com/milvus-io/milvus/pkg/v3/streaming/util/types" "github.com/milvus-io/milvus/pkg/v3/streaming/util/types"
"github.com/milvus-io/milvus/pkg/v3/util/funcutil" "github.com/milvus-io/milvus/pkg/v3/util/funcutil"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/syncutil" "github.com/milvus-io/milvus/pkg/v3/util/syncutil"
"github.com/milvus-io/milvus/pkg/v3/util/typeutil" "github.com/milvus-io/milvus/pkg/v3/util/typeutil"
) )
@@ -99,7 +100,7 @@ func (s *StreamingNodeManager) GetLatestWALLocated(ctx context.Context, vchannel
} }
serverID, ok := balancer.GetLatestWALLocated(ctx, pchannel) serverID, ok := balancer.GetLatestWALLocated(ctx, pchannel)
if !ok { if !ok {
return -1, errors.Errorf("channel: %s not found", vchannel) return -1, merr.WrapErrChannelNotFound(vchannel)
} }
return serverID, nil return serverID, nil
} }
+4 -4
View File
@@ -19,12 +19,12 @@ package coordinator
import ( import (
"context" "context"
"github.com/cockroachdb/errors"
"go.uber.org/zap" "go.uber.org/zap"
"github.com/milvus-io/milvus/internal/streamingcoord/server/broadcaster/registry" "github.com/milvus-io/milvus/internal/streamingcoord/server/broadcaster/registry"
"github.com/milvus-io/milvus/pkg/v3/log" "github.com/milvus-io/milvus/pkg/v3/log"
"github.com/milvus-io/milvus/pkg/v3/streaming/util/message" "github.com/milvus-io/milvus/pkg/v3/streaming/util/message"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/paramtable" "github.com/milvus-io/milvus/pkg/v3/util/paramtable"
) )
@@ -66,7 +66,7 @@ func (c *WALCallback) alterWALV2AckCallback(
targetWALName := result.Message.Header().TargetWalName targetWALName := result.Message.Header().TargetWalName
mqTypeValue := message.WALName(targetWALName).String() mqTypeValue := message.WALName(targetWALName).String()
if mqTypeValue == "unknown" { if mqTypeValue == "unknown" {
return errors.Errorf("invalid target WAL name: %v", targetWALName) return merr.WrapErrServiceInternalMsg("invalid target WAL name: %v", targetWALName)
} }
// Get etcd source to update configuration // Get etcd source to update configuration
@@ -74,7 +74,7 @@ func (c *WALCallback) alterWALV2AckCallback(
etcdSource, ok := paramMgr.GetEtcdSource() etcdSource, ok := paramMgr.GetEtcdSource()
if !ok { if !ok {
logger.Warn("failed to update mq.type config, etcd source not enabled") logger.Warn("failed to update mq.type config, etcd source not enabled")
return errors.New("etcd source is not enabled, cannot update mq.type configuration") return merr.WrapErrServiceInternalMsg("etcd source is not enabled, cannot update mq.type configuration")
} }
// Update mq.type configuration in etcd // Update mq.type configuration in etcd
@@ -84,7 +84,7 @@ func (c *WALCallback) alterWALV2AckCallback(
zap.String("configKey", configKey), zap.String("configKey", configKey),
zap.String("mqTypeValue", mqTypeValue), zap.String("mqTypeValue", mqTypeValue),
zap.Error(err)) zap.Error(err))
return errors.Wrap(err, "failed to update mq.type configuration in etcd") return merr.Wrap(err, "failed to update mq.type configuration in etcd")
} }
logger.Info("successfully updated mq.type configuration in etcd", logger.Info("successfully updated mq.type configuration in etcd",
@@ -131,10 +131,13 @@ async fn download_with_retry(
for url in urls { for url in urls {
match client.get(url).send().await { match client.get(url).send().await {
Ok(resp) if resp.status().is_success() => { Ok(resp) if resp.status().is_success() => {
let content = resp let content = match resp.bytes().await {
.bytes() Ok(c) => c,
.await Err(e) => {
.map_err(|e| TantivyBindingError::InternalError(e.to_string()))?; warn!("Failed to read body from {}: {}", url, e);
continue;
}
};
// Calculate MD5 hash // Calculate MD5 hash
let mut context = Context::new(); let mut context = Context::new();
+5 -5
View File
@@ -18,7 +18,6 @@ package datacoord
import ( import (
"context" "context"
"fmt"
"sync" "sync"
"go.uber.org/zap" "go.uber.org/zap"
@@ -28,6 +27,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/log" "github.com/milvus-io/milvus/pkg/v3/log"
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb" "github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
"github.com/milvus-io/milvus/pkg/v3/proto/workerpb" "github.com/milvus-io/milvus/pkg/v3/proto/workerpb"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/timerecord" "github.com/milvus-io/milvus/pkg/v3/util/timerecord"
) )
@@ -117,7 +117,7 @@ func (m *analyzeMeta) UpdateVersion(taskID int64, nodeID int64) error {
t, ok := m.tasks[taskID] t, ok := m.tasks[taskID]
if !ok { if !ok {
return fmt.Errorf("there is no task with taskID: %d", taskID) return merr.WrapErrServiceInternalMsg("there is no task with taskID: %d", taskID)
} }
cloneT := proto.Clone(t).(*indexpb.AnalyzeTask) cloneT := proto.Clone(t).(*indexpb.AnalyzeTask)
@@ -134,7 +134,7 @@ func (m *analyzeMeta) BuildingTask(taskID int64) error {
t, ok := m.tasks[taskID] t, ok := m.tasks[taskID]
if !ok { if !ok {
return fmt.Errorf("there is no task with taskID: %d", taskID) return merr.WrapErrServiceInternalMsg("there is no task with taskID: %d", taskID)
} }
cloneT := proto.Clone(t).(*indexpb.AnalyzeTask) cloneT := proto.Clone(t).(*indexpb.AnalyzeTask)
@@ -150,7 +150,7 @@ func (m *analyzeMeta) UpdateState(taskID int64, state indexpb.JobState, failReas
t, ok := m.tasks[taskID] t, ok := m.tasks[taskID]
if !ok { if !ok {
return fmt.Errorf("there is no task with taskID: %d", taskID) return merr.WrapErrServiceInternalMsg("there is no task with taskID: %d", taskID)
} }
cloneT := proto.Clone(t).(*indexpb.AnalyzeTask) cloneT := proto.Clone(t).(*indexpb.AnalyzeTask)
@@ -168,7 +168,7 @@ func (m *analyzeMeta) FinishTask(taskID int64, result *workerpb.AnalyzeResult) e
t, ok := m.tasks[taskID] t, ok := m.tasks[taskID]
if !ok { if !ok {
return fmt.Errorf("there is no task with taskID: %d", taskID) return merr.WrapErrServiceInternalMsg("there is no task with taskID: %d", taskID)
} }
log.Info("finish task meta...", zap.Int64("taskID", taskID), zap.String("state", result.GetState().String()), log.Info("finish task meta...", zap.Int64("taskID", taskID), zap.String("state", result.GetState().String()),
+12 -13
View File
@@ -21,10 +21,9 @@ import (
"strconv" "strconv"
"strings" "strings"
"github.com/cockroachdb/errors"
"github.com/milvus-io/milvus/internal/storage" "github.com/milvus-io/milvus/internal/storage"
"github.com/milvus-io/milvus/pkg/v3/proto/datapb" "github.com/milvus-io/milvus/pkg/v3/proto/datapb"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
) )
// BackfillResult is the decoded form of the JSON produced by the Spark // BackfillResult is the decoded form of the JSON produced by the Spark
@@ -86,34 +85,34 @@ var knownObjectSchemes = map[string]struct{}{
// 3. Unknown scheme -> return ErrUnsupportedScheme. // 3. Unknown scheme -> return ErrUnsupportedScheme.
func normalizeObjectKey(raw, expectedBucket string) (string, error) { func normalizeObjectKey(raw, expectedBucket string) (string, error) {
if raw == "" { if raw == "" {
return "", errors.New("empty object path") return "", merr.WrapErrServiceInternalMsg("empty object path")
} }
if !strings.Contains(raw, "://") { if !strings.Contains(raw, "://") {
key := strings.TrimPrefix(raw, "/") key := strings.TrimPrefix(raw, "/")
if key == "" { if key == "" {
return "", errors.Newf("empty object key in %q", raw) return "", merr.WrapErrServiceInternalMsg("empty object key in %q", raw)
} }
return key, nil return key, nil
} }
idx := strings.Index(raw, "://") idx := strings.Index(raw, "://")
scheme := strings.ToLower(raw[:idx]) scheme := strings.ToLower(raw[:idx])
if _, ok := knownObjectSchemes[scheme]; !ok { if _, ok := knownObjectSchemes[scheme]; !ok {
return "", errors.Newf("unsupported object URI scheme %q in %q", scheme, raw) return "", merr.WrapErrServiceInternalMsg("unsupported object URI scheme %q in %q", scheme, raw)
} }
rest := raw[idx+3:] rest := raw[idx+3:]
slash := strings.Index(rest, "/") slash := strings.Index(rest, "/")
if slash < 0 { if slash < 0 {
return "", errors.Newf("malformed object URI %q: missing object key", raw) return "", merr.WrapErrServiceInternalMsg("malformed object URI %q: missing object key", raw)
} }
bucket := rest[:slash] bucket := rest[:slash]
key := rest[slash+1:] key := rest[slash+1:]
if expectedBucket != "" && bucket != expectedBucket { if expectedBucket != "" && bucket != expectedBucket {
return "", errors.Newf("object URI bucket %q differs from datacoord bucket %q (path=%s)", bucket, expectedBucket, raw) return "", merr.WrapErrServiceInternalMsg("object URI bucket %q differs from datacoord bucket %q (path=%s)", bucket, expectedBucket, raw)
} }
// Reject inputs like "s3a://bucket/" that parse to an empty key -- passing // Reject inputs like "s3a://bucket/" that parse to an empty key -- passing
// an empty key to chunkManager.Read has undefined behavior across SDKs. // an empty key to chunkManager.Read has undefined behavior across SDKs.
if key == "" { if key == "" {
return "", errors.Newf("empty object key in %q", raw) return "", merr.WrapErrServiceInternalMsg("empty object key in %q", raw)
} }
return key, nil return key, nil
} }
@@ -147,20 +146,20 @@ func buildV2Groups(bucket string, entry *BackfillSegment) (map[int64]*datapb.Fie
for i := range entry.ColumnGroups { for i := range entry.ColumnGroups {
g := &entry.ColumnGroups[i] g := &entry.ColumnGroups[i]
if len(g.FieldIDs) != 1 { if len(g.FieldIDs) != 1 {
return nil, errors.Newf("backfill invariant violated: column group has %d field_ids (expected 1)", len(g.FieldIDs)) return nil, merr.WrapErrServiceInternalMsg("backfill invariant violated: column group has %d field_ids (expected 1)", len(g.FieldIDs))
} }
n := int64(len(g.BinlogFiles)) n := int64(len(g.BinlogFiles))
if n == 0 { if n == 0 {
return nil, errors.Newf("column group for field %d has no binlog files", g.FieldIDs[0]) return nil, merr.WrapErrServiceInternalMsg("column group for field %d has no binlog files", g.FieldIDs[0])
} }
fid := g.FieldIDs[0] fid := g.FieldIDs[0]
if _, dup := out[fid]; dup { if _, dup := out[fid]; dup {
return nil, errors.Newf("duplicate column group for field %d", fid) return nil, merr.WrapErrServiceInternalMsg("duplicate column group for field %d", fid)
} }
// row_count flows into EntriesNum; non-positive values are undefined // row_count flows into EntriesNum; non-positive values are undefined
// (zero collapses presence markers, negatives break accounting). // (zero collapses presence markers, negatives break accounting).
if g.RowCount <= 0 { if g.RowCount <= 0 {
return nil, errors.Newf("column group for field %d has non-positive row_count %d", fid, g.RowCount) return nil, merr.WrapErrServiceInternalMsg("column group for field %d has non-positive row_count %d", fid, g.RowCount)
} }
avg := g.RowCount / n avg := g.RowCount / n
rem := g.RowCount - avg*n rem := g.RowCount - avg*n
@@ -176,7 +175,7 @@ func buildV2Groups(bucket string, entry *BackfillSegment) (map[int64]*datapb.Fie
} }
logID, ok := parseLogIDFromKey(key) logID, ok := parseLogIDFromKey(key)
if !ok { if !ok {
return nil, errors.Newf("column group for field %d has binlog file %q with non-numeric trailing segment", fid, p) return nil, merr.WrapErrServiceInternalMsg("column group for field %d has binlog file %q with non-numeric trailing segment", fid, p)
} }
// LogPath must be empty at persistence time -- catalog.checkLogID // LogPath must be empty at persistence time -- catalog.checkLogID
// rejects any Binlog with LogPath != "" (see // rejects any Binlog with LogPath != "" (see
@@ -31,6 +31,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/common" "github.com/milvus-io/milvus/pkg/v3/common"
"github.com/milvus-io/milvus/pkg/v3/log" "github.com/milvus-io/milvus/pkg/v3/log"
"github.com/milvus-io/milvus/pkg/v3/proto/datapb" "github.com/milvus-io/milvus/pkg/v3/proto/datapb"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/paramtable" "github.com/milvus-io/milvus/pkg/v3/util/paramtable"
) )
@@ -241,7 +242,7 @@ func calculateClusteringCompactionConfig(coll *collectionInfo, view CompactionVi
func estimateRowsBySegmentSize(segments []*SegmentView, expectedSegmentSize int64) (int64, error) { func estimateRowsBySegmentSize(segments []*SegmentView, expectedSegmentSize int64) (int64, error) {
if expectedSegmentSize <= 0 { if expectedSegmentSize <= 0 {
return 0, fmt.Errorf("invalid expected segment size %d", expectedSegmentSize) return 0, merr.WrapErrServiceInternalMsg("invalid expected segment size %d", expectedSegmentSize)
} }
var totalSize float64 var totalSize float64
@@ -258,17 +259,17 @@ func estimateRowsBySegmentSize(segments []*SegmentView, expectedSegmentSize int6
} }
if totalRows == 0 || totalSize == 0 { if totalRows == 0 || totalSize == 0 {
return 0, fmt.Errorf("segment view does not contain size info to estimate row count") return 0, merr.WrapErrServiceInternalMsg("segment view does not contain size info to estimate row count")
} }
rowSize := totalSize / float64(totalRows) rowSize := totalSize / float64(totalRows)
if rowSize <= 0 { if rowSize <= 0 {
return 0, fmt.Errorf("invalid row size calculated from segment view") return 0, merr.WrapErrServiceInternalMsg("invalid row size calculated from segment view")
} }
rows := float64(expectedSegmentSize) / rowSize rows := float64(expectedSegmentSize) / rowSize
if rows <= 0 { if rows <= 0 {
return 0, fmt.Errorf("estimated max row count is not positive") return 0, merr.WrapErrServiceInternalMsg("estimated max row count is not positive")
} }
return int64(rows), nil return int64(rows), nil
@@ -2,7 +2,6 @@ package datacoord
import ( import (
"context" "context"
"fmt"
"math" "math"
"github.com/samber/lo" "github.com/samber/lo"
@@ -102,7 +101,7 @@ func (policy *forceMergeCompactionPolicy) triggerOneCollection(
configMaxSize := getExpectedSegmentSize(policy.meta, collectionID, collection.Schema) configMaxSize := getExpectedSegmentSize(policy.meta, collectionID, collection.Schema)
if targetSizeBytes < configMaxSize { if targetSizeBytes < configMaxSize {
return nil, 0, merr.WrapErrParameterInvalidMsg(fmt.Sprintf("targetSize %d MB should be greater than or equal to configMaxSize %d MB", targetSize, configMaxSize/(1024*1024))) return nil, 0, merr.WrapErrParameterInvalidMsg("targetSize %d MB should be greater than or equal to configMaxSize %d MB", targetSize, configMaxSize/(1024*1024))
} }
segments := policy.meta.SelectSegments(ctx, WithCollection(collectionID), SegmentFilterFunc(func(segment *SegmentInfo) bool { segments := policy.meta.SelectSegments(ctx, WithCollection(collectionID), SegmentFilterFunc(func(segment *SegmentInfo) bool {
@@ -182,7 +181,7 @@ var _ CollectionTopologyQuerier = (*metricsNodeMemoryQuerier)(nil)
func (q *metricsNodeMemoryQuerier) GetCollectionTopology(ctx context.Context, collectionID int64) (*CollectionTopology, error) { func (q *metricsNodeMemoryQuerier) GetCollectionTopology(ctx context.Context, collectionID int64) (*CollectionTopology, error) {
log := log.Ctx(ctx).With(zap.Int64("collectionID", collectionID)) log := log.Ctx(ctx).With(zap.Int64("collectionID", collectionID))
if q.mixCoord == nil { if q.mixCoord == nil {
return nil, fmt.Errorf("mixCoord not available for topology query") return nil, merr.WrapErrServiceInternalMsg("mixCoord not available for topology query")
} }
// 1. Get replica information // 1. Get replica information
+7 -4
View File
@@ -74,9 +74,12 @@ func (pq *PriorityQueue[T]) Update(item *Item[T], value T, priority int) {
heap.Fix(pq, item.index) heap.Fix(pq, item.index)
} }
// errFull / errNoSuchElement are INTERNAL sentinels: caught by errors.Is
// inside the compaction inspector / scheduler loop and never serialized
// across any gRPC boundary. See docs/dev/error_sentinel_convention.md.
var ( var (
ErrFull = errors.New("compaction queue is full") errFull = errors.New("compaction queue is full")
ErrNoSuchElement = errors.New("compaction queue has no element") errNoSuchElement = errors.New("compaction queue has no element")
) )
type Prioritizer func(t CompactionTask) int type Prioritizer func(t CompactionTask) int
@@ -101,7 +104,7 @@ func (q *CompactionQueue) Enqueue(t CompactionTask) error {
q.lock.Lock() q.lock.Lock()
defer q.lock.Unlock() defer q.lock.Unlock()
if q.capacity > 0 && len(q.pq) >= q.capacity { if q.capacity > 0 && len(q.pq) >= q.capacity {
return ErrFull return errFull
} }
heap.Push(&q.pq, &Item[CompactionTask]{value: t, priority: q.prioritizer(t)}) heap.Push(&q.pq, &Item[CompactionTask]{value: t, priority: q.prioritizer(t)})
@@ -113,7 +116,7 @@ func (q *CompactionQueue) Dequeue() (CompactionTask, error) {
defer q.lock.Unlock() defer q.lock.Unlock()
if len(q.pq) == 0 { if len(q.pq) == 0 {
return nil, ErrNoSuchElement return nil, errNoSuchElement
} }
item := heap.Pop(&q.pq).(*Item[CompactionTask]) item := heap.Pop(&q.pq).(*Item[CompactionTask])
@@ -23,7 +23,6 @@ import (
"strconv" "strconv"
"time" "time"
"github.com/cockroachdb/errors"
"github.com/samber/lo" "github.com/samber/lo"
"go.uber.org/atomic" "go.uber.org/atomic"
"go.uber.org/zap" "go.uber.org/zap"
@@ -629,7 +628,7 @@ func (t *clusteringCompactionTask) processAnalyzing() error {
} }
case indexpb.JobState_JobStateFailed: case indexpb.JobState_JobStateFailed:
log.Warn("analyze task fail", zap.Int64("analyzeID", t.GetTaskProto().GetAnalyzeTaskID())) log.Warn("analyze task fail", zap.Int64("analyzeID", t.GetTaskProto().GetAnalyzeTaskID()))
return errors.New(analyzeTask.FailReason) return merr.WrapErrServiceInternalMsg(analyzeTask.FailReason)
default: default:
} }
return nil return nil
+2 -2
View File
@@ -319,7 +319,7 @@ func (t *l0CompactionTask) selectFlushedSegment() ([]*SegmentInfo, []*datapb.Com
for _, info := range flushedSegments { for _, info := range flushedSegments {
// Sealed is unexpected, fail fast // Sealed is unexpected, fail fast
if info.GetState() == commonpb.SegmentState_Sealed { if info.GetState() == commonpb.SegmentState_Sealed {
return nil, nil, fmt.Errorf("L0 compaction selected invalid sealed segment %d", info.GetID()) return nil, nil, merr.WrapErrServiceInternalMsg("L0 compaction selected invalid sealed segment %d", info.GetID())
} }
sealedSegBinlogs = append(sealedSegBinlogs, &datapb.CompactionSegmentBinlogs{ sealedSegBinlogs = append(sealedSegBinlogs, &datapb.CompactionSegmentBinlogs{
@@ -455,7 +455,7 @@ func buildL0V3DeltaLogEntries(segmentID int64, deltalogs []*datapb.FieldBinlog)
for _, binlog := range fieldBinlog.GetBinlogs() { for _, binlog := range fieldBinlog.GetBinlogs() {
path := binlog.GetLogPath() path := binlog.GetLogPath()
if path == "" { if path == "" {
return nil, merr.WrapErrServiceInternal(fmt.Sprintf("L0 V3 compaction result missing deltalog path for segment %d, logID %d", segmentID, binlog.GetLogID())) return nil, merr.WrapErrServiceInternalMsg("L0 V3 compaction result missing deltalog path for segment %d, logID %d", segmentID, binlog.GetLogID())
} }
entries = append(entries, packed.DeltaLogEntry{ entries = append(entries, packed.DeltaLogEntry{
Path: path, Path: path,
+2 -2
View File
@@ -398,7 +398,7 @@ func (t *mixCompactionTask) BuildCompactionRequest() (*datapb.CompactionPlan, er
resources, err := t.meta.GetFileResources(context.Background(), taskSchema.GetFileResourceIds()...) resources, err := t.meta.GetFileResources(context.Background(), taskSchema.GetFileResourceIds()...)
if err != nil { if err != nil {
log.Warn("get file resources for collection failed", zap.Int64("collectionID", taskProto.GetCollectionID()), zap.Error(err)) log.Warn("get file resources for collection failed", zap.Int64("collectionID", taskProto.GetCollectionID()), zap.Error(err))
return nil, errors.Errorf("get file resources for sort compaction failed: %v", err) return nil, merr.Wrap(err, "get file resources for sort compaction failed")
} }
plan.FileResources = resources plan.FileResources = resources
} }
@@ -411,7 +411,7 @@ func (t *mixCompactionTask) BuildCompactionRequest() (*datapb.CompactionPlan, er
return nil, merr.WrapErrSegmentNotFound(segID) return nil, merr.WrapErrSegmentNotFound(segID)
} }
if taskSchemaVersion < segInfo.GetSchemaVersion() { if taskSchemaVersion < segInfo.GetSchemaVersion() {
return nil, merr.WrapErrIllegalCompactionPlan(fmt.Sprintf("compaction task schema version %d is older than input segment %d schema version %d", taskSchemaVersion, segInfo.GetID(), segInfo.GetSchemaVersion())) return nil, merr.WrapErrIllegalCompactionPlanMsg("compaction task schema version %d is older than input segment %d schema version %d", taskSchemaVersion, segInfo.GetID(), segInfo.GetSchemaVersion())
} }
plan.SegmentBinlogs = append(plan.SegmentBinlogs, &datapb.CompactionSegmentBinlogs{ plan.SegmentBinlogs = append(plan.SegmentBinlogs, &datapb.CompactionSegmentBinlogs{
SegmentID: segID, SegmentID: segID,
+5 -2
View File
@@ -225,7 +225,7 @@ func (t *compactionTrigger) getCollection(collectionID UniqueID) (*collectionInf
defer cancel() defer cancel()
coll, err := t.handler.GetCollection(ctx, collectionID) coll, err := t.handler.GetCollection(ctx, collectionID)
if err != nil { if err != nil {
return nil, fmt.Errorf("collection ID %d not found, err: %w", collectionID, err) return nil, merr.Wrapf(err, "collection ID %d not found", collectionID)
} }
return coll, nil return coll, nil
} }
@@ -588,7 +588,10 @@ func (t *compactionTrigger) getCandidates(signal *compactionSignal) ([]chanPartS
segments := t.meta.SelectSegments(context.TODO(), filters...) segments := t.meta.SelectSegments(context.TODO(), filters...)
// some criterion not met or conflicted // some criterion not met or conflicted
if len(signal.segmentIDs) > 0 && len(segments) != len(signal.segmentIDs) { if len(signal.segmentIDs) > 0 && len(segments) != len(signal.segmentIDs) {
return nil, merr.WrapErrServiceInternal("not all segment ids provided could be compacted") // SelectSegments also filters segments that are transiently mid-flush /
// compacting / just dropped, so a count mismatch is usually server-side
// state, not a bad id from the caller.
return nil, merr.WrapErrServiceInternalMsg("not all segment ids provided could be compacted")
} }
type category struct { type category struct {
+1 -1
View File
@@ -583,7 +583,7 @@ func AssembleCopySegmentRequest(task CopySegmentTask, job CopySegmentJob) (*data
if _, exists := newBuildIDs[srcBuildID]; !exists { if _, exists := newBuildIDs[srcBuildID]; !exists {
newID, err := t.alloc.AllocID(ctx) newID, err := t.alloc.AllocID(ctx)
if err != nil { if err != nil {
return merr.WrapErrServiceInternal(fmt.Sprintf("failed to allocate new buildID for source buildID %d", srcBuildID), err.Error()) return merr.Wrapf(err, "failed to allocate new buildID for source buildID %d", srcBuildID)
} }
newBuildIDs[srcBuildID] = newID newBuildIDs[srcBuildID] = newID
} }
+8 -9
View File
@@ -27,6 +27,7 @@ import (
"github.com/milvus-io/milvus/internal/streamingcoord/server/broadcaster/registry" "github.com/milvus-io/milvus/internal/streamingcoord/server/broadcaster/registry"
"github.com/milvus-io/milvus/pkg/v3/log" "github.com/milvus-io/milvus/pkg/v3/log"
"github.com/milvus-io/milvus/pkg/v3/streaming/util/message" "github.com/milvus-io/milvus/pkg/v3/streaming/util/message"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
) )
// RegisterDDLCallbacks registers the ddl callbacks. // RegisterDDLCallbacks registers the ddl callbacks.
@@ -89,7 +90,7 @@ func (s *Server) startBroadcastWithCollectionID(ctx context.Context, collectionI
func (s *Server) startBroadcastForRestoreSnapshot(ctx context.Context, collectionID int64, snapshotName string) (broadcaster.BroadcastAPI, error) { func (s *Server) startBroadcastForRestoreSnapshot(ctx context.Context, collectionID int64, snapshotName string) (broadcaster.BroadcastAPI, error) {
coll, err := s.broker.DescribeCollectionInternal(ctx, collectionID) coll, err := s.broker.DescribeCollectionInternal(ctx, collectionID)
if err != nil { if err != nil {
return nil, fmt.Errorf("collection %d does not exist: %w", collectionID, err) return nil, merr.Wrapf(err, "collection %d does not exist", collectionID)
} }
dbName := coll.GetDbName() dbName := coll.GetDbName()
collectionName := coll.GetCollectionName() collectionName := coll.GetCollectionName()
@@ -161,15 +162,15 @@ func (s *Server) validateRestoreSnapshotResources(ctx context.Context, collectio
sourceCollectionID := snapshotData.SnapshotInfo.GetCollectionId() sourceCollectionID := snapshotData.SnapshotInfo.GetCollectionId()
snapshot, err := s.meta.snapshotMeta.GetSnapshot(ctx, sourceCollectionID, snapshotData.SnapshotInfo.GetName()) snapshot, err := s.meta.snapshotMeta.GetSnapshot(ctx, sourceCollectionID, snapshotData.SnapshotInfo.GetName())
if err != nil { if err != nil {
return fmt.Errorf("snapshot %s does not exist for collection %d: %w", return merr.Wrapf(err, "snapshot %s does not exist for collection %d",
snapshotData.SnapshotInfo.GetName(), sourceCollectionID, err) snapshotData.SnapshotInfo.GetName(), sourceCollectionID)
} }
log.Info("snapshot validated", zap.String("snapshotName", snapshot.GetName())) log.Info("snapshot validated", zap.String("snapshotName", snapshot.GetName()))
// ========== Validate Collection Exists ========== // ========== Validate Collection Exists ==========
coll, err := s.broker.DescribeCollectionInternal(ctx, collectionID) coll, err := s.broker.DescribeCollectionInternal(ctx, collectionID)
if err != nil { if err != nil {
return fmt.Errorf("collection %d does not exist: %w", collectionID, err) return merr.Wrapf(err, "collection %d does not exist", collectionID)
} }
dbName := coll.GetDbName() dbName := coll.GetDbName()
collectionName := coll.GetCollectionName() collectionName := coll.GetCollectionName()
@@ -180,7 +181,7 @@ func (s *Server) validateRestoreSnapshotResources(ctx context.Context, collectio
// ========== Validate Partitions Exist ========== // ========== Validate Partitions Exist ==========
partitionsResp, err := s.broker.ShowPartitions(ctx, collectionID) partitionsResp, err := s.broker.ShowPartitions(ctx, collectionID)
if err != nil { if err != nil {
return fmt.Errorf("failed to get partitions for collection %d: %w", collectionID, err) return merr.Wrapf(err, "failed to get partitions for collection %d", collectionID)
} }
// Build set of existing partition names // Build set of existing partition names
@@ -192,8 +193,7 @@ func (s *Server) validateRestoreSnapshotResources(ctx context.Context, collectio
// Check all snapshot partitions exist // Check all snapshot partitions exist
for partName := range snapshotData.Collection.GetPartitions() { for partName := range snapshotData.Collection.GetPartitions() {
if !existingPartitions[partName] { if !existingPartitions[partName] {
return fmt.Errorf("partition %s does not exist in collection %d", return merr.WrapErrPartitionNotFound(partName, fmt.Sprintf("partition does not exist in collection %d", collectionID))
partName, collectionID)
} }
} }
log.Info("partitions validated", zap.Int("count", len(existingPartitions))) log.Info("partitions validated", zap.Int("count", len(existingPartitions)))
@@ -211,8 +211,7 @@ func (s *Server) validateRestoreSnapshotResources(ctx context.Context, collectio
} }
} }
if !indexFound { if !indexFound {
return fmt.Errorf("index %s for field %d does not exist in collection %d", return merr.WrapErrIndexNotFound(indexInfo.GetIndexName(), fmt.Sprintf("index for field %d does not exist in collection %d", indexInfo.GetFieldID(), collectionID))
indexInfo.GetIndexName(), indexInfo.GetFieldID(), collectionID)
} }
} }
log.Info("indexes validated", zap.Int("count", len(snapshotData.Indexes))) log.Info("indexes validated", zap.Int("count", len(snapshotData.Indexes)))
+4 -4
View File
@@ -134,10 +134,10 @@ func (s *Server) validateImportReplication(ctx context.Context, options []*commo
} }
if !paramtable.Get().DataCoordCfg.ImportInReplicatingCluster.GetAsBool() { if !paramtable.Get().DataCoordCfg.ImportInReplicatingCluster.GetAsBool() {
return merr.WrapErrImportFailed("import in replicating cluster is not supported yet") return merr.WrapErrOperationNotSupportedMsg("import in replicating cluster is not supported yet")
} }
if importutilv2.IsAutoCommit(options) { if importutilv2.IsAutoCommit(options) {
return merr.WrapErrImportFailed("auto_commit=true import in replicating cluster is not supported") return merr.WrapErrOperationNotSupportedMsg("auto_commit=true import in replicating cluster is not supported")
} }
return nil return nil
} }
@@ -168,14 +168,14 @@ func (s *Server) broadcastImport(ctx context.Context,
// Validate the request before broadcasting // Validate the request before broadcasting
if err := s.validateImportRequest(ctx, msgFiles, options); err != nil { if err := s.validateImportRequest(ctx, msgFiles, options); err != nil {
return errors.Wrap(err, "failed to validate import request") return merr.Wrap(err, "failed to validate import request")
} }
// Get database name from collection metadata via broker // Get database name from collection metadata via broker
// This is safer than extracting from schema which may be stale // This is safer than extracting from schema which may be stale
broadcaster, err := s.startBroadcastWithCollectionID(ctx, collectionID) broadcaster, err := s.startBroadcastWithCollectionID(ctx, collectionID)
if err != nil { if err != nil {
return errors.Wrap(err, "failed to start broadcast with collection id") return merr.Wrap(err, "failed to start broadcast with collection id")
} }
defer broadcaster.Close() defer broadcaster.Close()
@@ -107,8 +107,9 @@ func (s *ImportCallbacksSuite) TestValidateImportRequest_MaxJobsExceededReturnsE
err := server.validateImportRequest(ctx, files, options) err := server.validateImportRequest(ctx, files, options)
s.Error(err) s.Error(err)
// ValidateMaxImportJobExceed returns WrapErrImportFailed, not ErrServiceQuotaExceeded // Job-count backpressure is a server-side condition -> ErrImportSysFailed
s.True(errors.Is(err, merr.ErrImportFailed)) // (must not be bucketed as fail_input).
s.True(errors.Is(err, merr.ErrImportSysFailed))
s.Contains(err.Error(), "The number of jobs has reached the limit") s.Contains(err.Error(), "The number of jobs has reached the limit")
} }
@@ -185,7 +186,7 @@ func (s *ImportCallbacksSuite) TestValidateImportRequest_ReplicatingClusterRetur
err := server.validateImportRequest(ctx, files, options) err := server.validateImportRequest(ctx, files, options)
s.Error(err) s.Error(err)
s.True(errors.Is(err, merr.ErrImportFailed)) s.True(errors.Is(err, merr.ErrOperationNotSupported))
s.Contains(err.Error(), "replicating cluster") s.Contains(err.Error(), "replicating cluster")
} }
@@ -230,7 +231,7 @@ func (s *ImportCallbacksSuite) TestValidateImportRequest_ReplicatingClusterEnabl
{Key: "timeout", Value: "300s"}, {Key: "timeout", Value: "300s"},
}) })
s.Error(err) s.Error(err)
s.True(errors.Is(err, merr.ErrImportFailed)) s.True(errors.Is(err, merr.ErrOperationNotSupported))
s.Contains(err.Error(), "auto_commit=true") s.Contains(err.Error(), "auto_commit=true")
err = server.validateImportRequest(ctx, files, []*commonpb.KeyValuePair{ err = server.validateImportRequest(ctx, files, []*commonpb.KeyValuePair{
@@ -1159,7 +1160,8 @@ func testBroadcastRequiresVchannels(t *testing.T, broadcastFn func(*Server, cont
err := broadcastFn(server, ctx, job) err := broadcastFn(server, ctx, job)
assert.Error(t, err) assert.Error(t, err)
assert.True(t, errors.Is(err, merr.ErrImportFailed)) // Missing vchannels is internal broadcast state -> ErrImportSysFailed.
assert.True(t, errors.Is(err, merr.ErrImportSysFailed))
assert.Contains(t, err.Error(), "job 7 has no vchannels") assert.Contains(t, err.Error(), "job 7 has no vchannels")
} }
@@ -636,7 +636,8 @@ func TestValidateRestoreSnapshotResources_IndexNotFound(t *testing.T) {
err := server.validateRestoreSnapshotResources(ctx, 100, buildBaseSnapshotData()) err := server.validateRestoreSnapshotResources(ctx, 100, buildBaseSnapshotData())
assert.Error(t, err) assert.Error(t, err)
assert.Contains(t, err.Error(), "index vec_idx for field 100 does not exist") assert.Contains(t, err.Error(), "index for field 100 does not exist")
assert.Contains(t, err.Error(), "vec_idx")
} }
func TestValidateRestoreSnapshotResources_IndexFieldMismatch(t *testing.T) { func TestValidateRestoreSnapshotResources_IndexFieldMismatch(t *testing.T) {
@@ -656,7 +657,8 @@ func TestValidateRestoreSnapshotResources_IndexFieldMismatch(t *testing.T) {
err := server.validateRestoreSnapshotResources(ctx, 100, buildBaseSnapshotData()) err := server.validateRestoreSnapshotResources(ctx, 100, buildBaseSnapshotData())
assert.Error(t, err) assert.Error(t, err)
assert.Contains(t, err.Error(), "index vec_idx for field 100 does not exist") assert.Contains(t, err.Error(), "index for field 100 does not exist")
assert.Contains(t, err.Error(), "vec_idx")
} }
func TestValidateRestoreSnapshotResources_Success(t *testing.T) { func TestValidateRestoreSnapshotResources_Success(t *testing.T) {
+4 -1
View File
@@ -20,6 +20,9 @@ import (
"fmt" "fmt"
"github.com/cockroachdb/errors" "github.com/cockroachdb/errors"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/typeutil"
) )
// errors for VerifyResponse // errors for VerifyResponse
@@ -34,7 +37,7 @@ func msgDataCoordIsUnhealthy(coordID UniqueID) string {
} }
func errDataCoordIsUnhealthy(coordID UniqueID) error { func errDataCoordIsUnhealthy(coordID UniqueID) error {
return errors.New(msgDataCoordIsUnhealthy(coordID)) return merr.WrapErrServiceNotReady(typeutil.DataCoordRole, coordID, "not ready")
} }
func msgSegmentNotFound(segID UniqueID) string { func msgSegmentNotFound(segID UniqueID) string {
@@ -306,7 +306,7 @@ func (m *externalCollectionRefreshManager) cleanupExploreTempForJob(jobID int64)
func (m *externalCollectionRefreshManager) applyFinishedJobSegments(ctx context.Context, job *datapb.ExternalCollectionRefreshJob) error { func (m *externalCollectionRefreshManager) applyFinishedJobSegments(ctx context.Context, job *datapb.ExternalCollectionRefreshJob) error {
tasks := m.refreshMeta.GetTasksByJobID(job.GetJobId()) tasks := m.refreshMeta.GetTasksByJobID(job.GetJobId())
if len(tasks) == 0 { if len(tasks) == 0 {
return fmt.Errorf("job %d has no tasks to apply", job.GetJobId()) return merr.WrapErrServiceInternalMsg("job %d has no tasks to apply", job.GetJobId())
} }
keptSet := make(map[int64]struct{}) keptSet := make(map[int64]struct{})
@@ -315,11 +315,11 @@ func (m *externalCollectionRefreshManager) applyFinishedJobSegments(ctx context.
updatedSegments := make([]*datapb.SegmentInfo, 0) updatedSegments := make([]*datapb.SegmentInfo, 0)
for _, task := range tasks { for _, task := range tasks {
if task.GetState() != indexpb.JobState_JobStateFinished { if task.GetState() != indexpb.JobState_JobStateFinished {
return fmt.Errorf("job %d has non-finished task %d in state %s", return merr.WrapErrServiceInternalMsg("job %d has non-finished task %d in state %s",
job.GetJobId(), task.GetTaskId(), task.GetState().String()) job.GetJobId(), task.GetTaskId(), task.GetState().String())
} }
if !task.GetResultReady() { if !task.GetResultReady() {
return fmt.Errorf("job %d has finished task %d without persisted refresh result; please retry refresh", return merr.WrapErrServiceInternalMsg("job %d has finished task %d without persisted refresh result; please retry refresh",
job.GetJobId(), task.GetTaskId()) job.GetJobId(), task.GetTaskId())
} }
for _, segmentID := range task.GetKeptSegments() { for _, segmentID := range task.GetKeptSegments() {
@@ -334,7 +334,7 @@ func (m *externalCollectionRefreshManager) applyFinishedJobSegments(ctx context.
continue continue
} }
if _, ok := updatedSet[segment.GetID()]; ok { if _, ok := updatedSet[segment.GetID()]; ok {
return fmt.Errorf("job %d has duplicate updated segment %d from task %d", return merr.WrapErrServiceInternalMsg("job %d has duplicate updated segment %d from task %d",
job.GetJobId(), segment.GetID(), task.GetTaskId()) job.GetJobId(), segment.GetID(), task.GetTaskId())
} }
updatedSet[segment.GetID()] = struct{}{} updatedSet[segment.GetID()] = struct{}{}
@@ -714,7 +714,7 @@ func (m *externalCollectionRefreshManager) createTasksForJob(
if errors.Is(err, packed.ErrLoonTransient) { if errors.Is(err, packed.ErrLoonTransient) {
return nil, newNonRetriableJobError("explore external files failed: %v", err) return nil, newNonRetriableJobError("explore external files failed: %v", err)
} }
return nil, fmt.Errorf("failed to explore external files: %w", err) return nil, merr.WrapErrServiceInternalErr(err, "failed to explore external files")
} }
if len(allFiles) == 0 { if len(allFiles) == 0 {
return nil, newNonRetriableJobError("no files found in external source: %s", job.GetExternalSource()) return nil, newNonRetriableJobError("no files found in external source: %s", job.GetExternalSource())
@@ -829,7 +829,7 @@ func normalizeRefreshJobProgress(job *datapb.ExternalCollectionRefreshJob, state
func (m *externalCollectionRefreshManager) GetJobProgress(ctx context.Context, jobID int64) (*datapb.ExternalCollectionRefreshJob, error) { func (m *externalCollectionRefreshManager) GetJobProgress(ctx context.Context, jobID int64) (*datapb.ExternalCollectionRefreshJob, error) {
job := m.refreshMeta.GetJob(jobID) job := m.refreshMeta.GetJob(jobID)
if job == nil { if job == nil {
return nil, fmt.Errorf("job %d not found", jobID) return nil, merr.WrapErrParameterInvalidMsg("refresh job %d not found", jobID)
} }
// Aggregate state and progress from tasks // Aggregate state and progress from tasks
@@ -877,17 +877,17 @@ func (m *externalCollectionRefreshManager) exploreExternalFiles(
// when both present. // when both present.
if job.GetExternalSource() != "" { if job.GetExternalSource() != "" {
if err := externalspec.ValidateSourceAndSpec(job.GetExternalSource(), job.GetExternalSpec()); err != nil { if err := externalspec.ValidateSourceAndSpec(job.GetExternalSource(), job.GetExternalSpec()); err != nil {
return nil, "", fmt.Errorf("external source/spec failed revalidation: %w", err) return nil, "", merr.Wrap(err, "external source/spec failed revalidation")
} }
} }
spec, err := externalspec.ParseExternalSpec(job.GetExternalSpec()) spec, err := externalspec.ParseExternalSpec(job.GetExternalSpec())
if err != nil { if err != nil {
return nil, "", fmt.Errorf("failed to parse external spec: %w", err) return nil, "", merr.Wrap(err, "failed to parse external spec")
} }
collInfo := m.mt.GetCollection(job.GetCollectionId()) collInfo := m.mt.GetCollection(job.GetCollectionId())
if collInfo == nil { if collInfo == nil {
return nil, "", fmt.Errorf("collection %d not found", job.GetCollectionId()) return nil, "", merr.WrapErrCollectionNotFound(job.GetCollectionId())
} }
columns := packed.GetColumnNamesFromSchema(collInfo.Schema) columns := packed.GetColumnNamesFromSchema(collInfo.Schema)
@@ -908,7 +908,7 @@ func (m *externalCollectionRefreshManager) exploreExternalFiles(
extfs, extfs,
) )
if err != nil { if err != nil {
return nil, "", fmt.Errorf("ExploreFilesReturnManifestPath failed: %w", err) return nil, "", merr.WrapErrServiceInternalErr(err, "failed to explore files returning manifest path")
} }
// Convert to proto type // Convert to proto type
@@ -18,7 +18,6 @@ package datacoord
import ( import (
"context" "context"
"fmt"
"sort" "sort"
"time" "time"
@@ -30,6 +29,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/proto/datapb" "github.com/milvus-io/milvus/pkg/v3/proto/datapb"
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb" "github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
"github.com/milvus-io/milvus/pkg/v3/util/lock" "github.com/milvus-io/milvus/pkg/v3/util/lock"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/timerecord" "github.com/milvus-io/milvus/pkg/v3/util/timerecord"
"github.com/milvus-io/milvus/pkg/v3/util/typeutil" "github.com/milvus-io/milvus/pkg/v3/util/typeutil"
) )
@@ -295,7 +295,7 @@ func (m *externalCollectionRefreshMeta) mutateJob(
) (applied bool, err error) { ) (applied bool, err error) {
job, ok := m.jobs.Get(jobID) job, ok := m.jobs.Get(jobID)
if !ok { if !ok {
return false, fmt.Errorf("job %d not found", jobID) return false, merr.WrapErrServiceInternalMsg("job %d not found", jobID)
} }
m.jobLock.Lock(job.GetCollectionId()) m.jobLock.Lock(job.GetCollectionId())
@@ -304,7 +304,7 @@ func (m *externalCollectionRefreshMeta) mutateJob(
// Re-fetch after lock // Re-fetch after lock
job, ok = m.jobs.Get(jobID) job, ok = m.jobs.Get(jobID)
if !ok { if !ok {
return false, fmt.Errorf("job %d not found", jobID) return false, merr.WrapErrServiceInternalMsg("job %d not found", jobID)
} }
cloneJob := proto.Clone(job).(*datapb.ExternalCollectionRefreshJob) cloneJob := proto.Clone(job).(*datapb.ExternalCollectionRefreshJob)
@@ -385,7 +385,7 @@ func (m *externalCollectionRefreshMeta) UpdateJobStateWithPreApply(
) (bool, error) { ) (bool, error) {
job, ok := m.jobs.Get(jobID) job, ok := m.jobs.Get(jobID)
if !ok { if !ok {
return false, fmt.Errorf("job %d not found", jobID) return false, merr.WrapErrServiceInternalMsg("job %d not found", jobID)
} }
m.jobLock.Lock(job.GetCollectionId()) m.jobLock.Lock(job.GetCollectionId())
@@ -395,7 +395,7 @@ func (m *externalCollectionRefreshMeta) UpdateJobStateWithPreApply(
// terminal state owns the one-time side effects. // terminal state owns the one-time side effects.
job, ok = m.jobs.Get(jobID) job, ok = m.jobs.Get(jobID)
if !ok { if !ok {
return false, fmt.Errorf("job %d not found", jobID) return false, merr.WrapErrServiceInternalMsg("job %d not found", jobID)
} }
if job.GetState() == indexpb.JobState_JobStateFinished || if job.GetState() == indexpb.JobState_JobStateFinished ||
job.GetState() == indexpb.JobState_JobStateFailed { job.GetState() == indexpb.JobState_JobStateFailed {
@@ -416,7 +416,7 @@ func (m *externalCollectionRefreshMeta) UpdateJobStateWithPreApply(
log.Warn("update job state after pre-apply failed", log.Warn("update job state after pre-apply failed",
zap.Int64("jobID", jobID), zap.Int64("jobID", jobID),
zap.Error(saveErr)) zap.Error(saveErr))
return false, fmt.Errorf("pre-apply failed: %w; additionally failed to persist Failed job state: %v", err, saveErr) return false, merr.Wrapf(err, "pre-apply failed; additionally failed to persist Failed job state: %v", saveErr)
} }
m.jobs.Insert(jobID, cloneJob) m.jobs.Insert(jobID, cloneJob)
m.addToCollectionJobs(cloneJob) m.addToCollectionJobs(cloneJob)
@@ -613,7 +613,7 @@ func (m *externalCollectionRefreshMeta) mutateTask(
) (applied bool, cloned *datapb.ExternalCollectionRefreshTask, err error) { ) (applied bool, cloned *datapb.ExternalCollectionRefreshTask, err error) {
task, ok := m.tasks.Get(taskID) task, ok := m.tasks.Get(taskID)
if !ok { if !ok {
return false, nil, fmt.Errorf("task %d not found", taskID) return false, nil, merr.WrapErrServiceInternalMsg("task %d not found", taskID)
} }
m.taskLock.Lock(task.GetJobId()) m.taskLock.Lock(task.GetJobId())
@@ -622,7 +622,7 @@ func (m *externalCollectionRefreshMeta) mutateTask(
// Re-fetch after lock // Re-fetch after lock
task, ok = m.tasks.Get(taskID) task, ok = m.tasks.Get(taskID)
if !ok { if !ok {
return false, nil, fmt.Errorf("task %d not found", taskID) return false, nil, merr.WrapErrServiceInternalMsg("task %d not found", taskID)
} }
cloneTask := proto.Clone(task).(*datapb.ExternalCollectionRefreshTask) cloneTask := proto.Clone(task).(*datapb.ExternalCollectionRefreshTask)
+2 -2
View File
@@ -1067,13 +1067,13 @@ func (gc *garbageCollector) isExpire(dropts Timestamp) bool {
// Returns segmentID or error if path doesn't match. // Returns segmentID or error if path doesn't match.
func parseV3SegmentID(rootPath, filePath string) (int64, error) { func parseV3SegmentID(rootPath, filePath string) (int64, error) {
if !strings.HasPrefix(filePath, rootPath) { if !strings.HasPrefix(filePath, rootPath) {
return 0, fmt.Errorf("path %q does not contain rootPath %q", filePath, rootPath) return 0, merr.WrapErrServiceInternalMsg("path %q does not contain rootPath %q", filePath, rootPath)
} }
p := strings.TrimPrefix(filePath[len(rootPath):], "/") p := strings.TrimPrefix(filePath[len(rootPath):], "/")
parts := strings.Split(p, "/") parts := strings.Split(p, "/")
// Minimum: insert_log/coll/part/seg/something // Minimum: insert_log/coll/part/seg/something
if len(parts) < 5 || parts[0] != common.SegmentInsertLogPath { if len(parts) < 5 || parts[0] != common.SegmentInsertLogPath {
return 0, fmt.Errorf("not a V3 insert_log path: %s", filePath) return 0, merr.WrapErrServiceInternalMsg("not a V3 insert_log path: %s", filePath)
} }
return strconv.ParseInt(parts[3], 10, 64) return strconv.ParseInt(parts[3], 10, 64)
} }
+2 -3
View File
@@ -18,7 +18,6 @@ package datacoord
import ( import (
"context" "context"
"fmt"
"path" "path"
"strings" "strings"
"sync" "sync"
@@ -34,6 +33,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/log" "github.com/milvus-io/milvus/pkg/v3/log"
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb" "github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
"github.com/milvus-io/milvus/pkg/v3/util/conc" "github.com/milvus-io/milvus/pkg/v3/util/conc"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/paramtable" "github.com/milvus-io/milvus/pkg/v3/util/paramtable"
"github.com/milvus-io/milvus/pkg/v3/util/typeutil" "github.com/milvus-io/milvus/pkg/v3/util/typeutil"
) )
@@ -271,8 +271,7 @@ func (lobCtx *lobGCContext) collectLOBFilesFromSegment(ctx context.Context, segm
lobFiles, err := lobCtx.cache.Get(ctx, manifestPath, lobCtx.storageConfig) lobFiles, err := lobCtx.cache.Get(ctx, manifestPath, lobCtx.storageConfig)
if err != nil { if err != nil {
return fmt.Errorf("failed to get LOB files from manifest for segment %d (path=%s): %w", return merr.WrapErrServiceInternalErr(err, "failed to get LOB files from manifest for segment %d (path=%s)", segment.GetID(), manifestPath)
segment.GetID(), manifestPath, err)
} }
for _, lobFile := range lobFiles { for _, lobFile := range lobFiles {
+1 -2
View File
@@ -18,7 +18,6 @@ package datacoord
import ( import (
"context" "context"
"fmt"
"time" "time"
"github.com/hashicorp/golang-lru/v2/expirable" "github.com/hashicorp/golang-lru/v2/expirable"
@@ -345,7 +344,7 @@ func (m *importMeta) HandleCommitVchannel(ctx context.Context, jobID int64, vcha
job := m.jobs[jobID] job := m.jobs[jobID]
if job == nil { if job == nil {
return merr.WrapErrImportFailed(fmt.Sprintf("job %d not found", jobID)) return merr.WrapErrImportSysFailedMsg("job %d not found", jobID)
} }
// Idempotency: if vchannel already committed, skip. // Idempotency: if vchannel already committed, skip.
for _, c := range job.GetCommittedVchannels() { for _, c := range job.GetCommittedVchannels() {
+10 -10
View File
@@ -104,7 +104,7 @@ func (s *ImportServicesSuite) TestImportV2_AllocatorNilReturnsError() {
s.NoError(err) s.NoError(err)
s.NotNil(resp) s.NotNil(resp)
s.True(errors.Is(merr.Error(resp.GetStatus()), merr.ErrImportFailed)) s.True(errors.Is(merr.Error(resp.GetStatus()), merr.ErrServiceUnavailable))
s.Contains(resp.GetStatus().GetReason(), "allocator not initialized") s.Contains(resp.GetStatus().GetReason(), "allocator not initialized")
} }
@@ -114,7 +114,7 @@ func (s *ImportServicesSuite) TestImportV2_AllocatorFailsReturnsError() {
server.stateCode.Store(commonpb.StateCode_Healthy) server.stateCode.Store(commonpb.StateCode_Healthy)
mockAllocator := allocator.NewMockAllocator(s.T()) mockAllocator := allocator.NewMockAllocator(s.T())
mockAllocator.EXPECT().AllocN(mock.Anything).Return(int64(0), int64(0), errors.New("allocation failed")) mockAllocator.EXPECT().AllocN(mock.Anything).Return(int64(0), int64(0), merr.WrapErrServiceUnavailable("allocation failed"))
server.allocator = mockAllocator server.allocator = mockAllocator
req := &internalpb.ImportRequestInternal{ req := &internalpb.ImportRequestInternal{
@@ -127,7 +127,7 @@ func (s *ImportServicesSuite) TestImportV2_AllocatorFailsReturnsError() {
s.NoError(err) s.NoError(err)
s.NotNil(resp) s.NotNil(resp)
s.True(errors.Is(merr.Error(resp.GetStatus()), merr.ErrImportFailed)) s.True(errors.Is(merr.Error(resp.GetStatus()), merr.ErrServiceUnavailable))
s.Contains(resp.GetStatus().GetReason(), "failed to allocate job ID") s.Contains(resp.GetStatus().GetReason(), "failed to allocate job ID")
} }
@@ -164,7 +164,7 @@ func (s *ImportServicesSuite) TestImportV2_BroadcastFailsReturnsError() {
// Mock StartBroadcastWithResourceKeys to fail // Mock StartBroadcastWithResourceKeys to fail
mockBroadcast := mockey.Mock(broadcast.StartBroadcastWithResourceKeys).To( mockBroadcast := mockey.Mock(broadcast.StartBroadcastWithResourceKeys).To(
func(ctx context.Context, keys ...message.ResourceKey) (broadcaster.BroadcastAPI, error) { func(ctx context.Context, keys ...message.ResourceKey) (broadcaster.BroadcastAPI, error) {
return nil, errors.New("broadcast failed") return nil, merr.WrapErrServiceUnavailable("broadcast failed")
}).Build() }).Build()
defer mockBroadcast.UnPatch() defer mockBroadcast.UnPatch()
@@ -199,8 +199,8 @@ func (s *ImportServicesSuite) TestImportV2_BroadcastFailsReturnsError() {
s.NoError(err) s.NoError(err)
s.NotNil(resp) s.NotNil(resp)
s.True(errors.Is(merr.Error(resp.GetStatus()), merr.ErrImportFailed)) s.True(errors.Is(merr.Error(resp.GetStatus()), merr.ErrServiceUnavailable))
s.Contains(resp.GetStatus().GetReason(), "failed to broadcast import") s.Contains(resp.GetStatus().GetReason(), "broadcast")
} }
func (s *ImportServicesSuite) TestImportV2_SuccessReturnsJobID() { func (s *ImportServicesSuite) TestImportV2_SuccessReturnsJobID() {
@@ -395,7 +395,7 @@ func (s *ImportServicesSuite) TestCreateImportJobFromAck_AllocatorFailsReturnsEr
server.stateCode.Store(commonpb.StateCode_Healthy) server.stateCode.Store(commonpb.StateCode_Healthy)
mockAllocator := allocator.NewMockAllocator(s.T()) mockAllocator := allocator.NewMockAllocator(s.T())
mockAllocator.EXPECT().AllocN(mock.Anything).Return(int64(0), int64(0), errors.New("allocation failed")) mockAllocator.EXPECT().AllocN(mock.Anything).Return(int64(0), int64(0), merr.WrapErrServiceUnavailable("allocation failed"))
server.allocator = mockAllocator server.allocator = mockAllocator
req := &internalpb.ImportRequestInternal{ req := &internalpb.ImportRequestInternal{
@@ -412,7 +412,7 @@ func (s *ImportServicesSuite) TestCreateImportJobFromAck_AllocatorFailsReturnsEr
s.NoError(err) s.NoError(err)
s.NotNil(resp) s.NotNil(resp)
s.True(errors.Is(merr.Error(resp.GetStatus()), merr.ErrImportFailed)) s.True(errors.Is(merr.Error(resp.GetStatus()), merr.ErrServiceUnavailable))
s.Contains(resp.GetStatus().GetReason(), "alloc id failed") s.Contains(resp.GetStatus().GetReason(), "alloc id failed")
} }
@@ -493,7 +493,7 @@ func (s *ImportServicesSuite) TestCreateImportJobFromAck_AddJobFailsReturnsError
catalog.EXPECT().ListImportJobs(mock.Anything).Return(nil, nil) catalog.EXPECT().ListImportJobs(mock.Anything).Return(nil, nil)
catalog.EXPECT().ListPreImportTasks(mock.Anything).Return(nil, nil) catalog.EXPECT().ListPreImportTasks(mock.Anything).Return(nil, nil)
catalog.EXPECT().ListImportTasks(mock.Anything).Return(nil, nil) catalog.EXPECT().ListImportTasks(mock.Anything).Return(nil, nil)
catalog.EXPECT().SaveImportJob(mock.Anything, mock.Anything).Return(errors.New("save job failed")) catalog.EXPECT().SaveImportJob(mock.Anything, mock.Anything).Return(merr.WrapErrServiceUnavailable("save job failed"))
importMeta, err := NewImportMeta(context.TODO(), catalog, nil, nil) importMeta, err := NewImportMeta(context.TODO(), catalog, nil, nil)
s.NoError(err) s.NoError(err)
@@ -528,7 +528,7 @@ func (s *ImportServicesSuite) TestCreateImportJobFromAck_AddJobFailsReturnsError
s.NoError(err) s.NoError(err)
s.NotNil(resp) s.NotNil(resp)
s.True(errors.Is(merr.Error(resp.GetStatus()), merr.ErrImportFailed)) s.True(errors.Is(merr.Error(resp.GetStatus()), merr.ErrServiceUnavailable))
s.Contains(resp.GetStatus().GetReason(), "add import job failed") s.Contains(resp.GetStatus().GetReason(), "add import job failed")
} }
+7 -7
View File
@@ -670,8 +670,8 @@ func ListBinlogsAndGroupBySegment(ctx context.Context,
return nil, merr.WrapErrImportFailed("no insert binlogs to import") return nil, merr.WrapErrImportFailed("no insert binlogs to import")
} }
if len(importFile.GetPaths()) > 2 { if len(importFile.GetPaths()) > 2 {
return nil, merr.WrapErrImportFailed(fmt.Sprintf("too many input paths for binlog import. "+ return nil, merr.WrapErrImportFailedMsg("too many input paths for binlog import. "+
"Valid paths length should be one or two, but got paths:%s", importFile.GetPaths())) "Valid paths length should be one or two, but got paths:%s", importFile.GetPaths())
} }
insertPrefix := importFile.GetPaths()[0] insertPrefix := importFile.GetPaths()[0]
@@ -790,18 +790,18 @@ func ListBinlogImportRequestFiles(ctx context.Context, cm storage.ChunkManager,
} }
err := conc.AwaitAll(futures...) err := conc.AwaitAll(futures...)
if err != nil { if err != nil {
return nil, merr.WrapErrImportFailed(fmt.Sprintf("list binlogs failed, err=%s", err)) return nil, merr.WrapErrServiceUnavailableMsg("list binlogs failed, err=%s", err)
} }
resFiles = lo.Filter(resFiles, func(file *internalpb.ImportFile, _ int) bool { resFiles = lo.Filter(resFiles, func(file *internalpb.ImportFile, _ int) bool {
return len(file.GetPaths()) > 0 return len(file.GetPaths()) > 0
}) })
if len(resFiles) == 0 { if len(resFiles) == 0 {
return nil, merr.WrapErrImportFailed(fmt.Sprintf("no binlog to import, input=%s", reqFiles)) return nil, merr.WrapErrImportFailedMsg("no binlog to import, input=%s", reqFiles)
} }
if len(resFiles) > paramtable.Get().DataCoordCfg.MaxFilesPerImportReq.GetAsInt() { if len(resFiles) > paramtable.Get().DataCoordCfg.MaxFilesPerImportReq.GetAsInt() {
return nil, merr.WrapErrImportFailed(fmt.Sprintf("The max number of import files should not exceed %d, but got %d", return nil, merr.WrapErrImportFailedMsg("The max number of import files should not exceed %d, but got %d",
paramtable.Get().DataCoordCfg.MaxFilesPerImportReq.GetAsInt(), len(resFiles))) paramtable.Get().DataCoordCfg.MaxFilesPerImportReq.GetAsInt(), len(resFiles))
} }
log.Info("list binlogs prefixes for import done", zap.Int("num", len(resFiles)), zap.Any("binlog_prefixes", resFiles)) log.Info("list binlogs prefixes for import done", zap.Int("num", len(resFiles)), zap.Any("binlog_prefixes", resFiles))
return resFiles, nil return resFiles, nil
@@ -812,7 +812,7 @@ func ValidateMaxImportJobExceed(ctx context.Context, importMeta ImportMeta) erro
maxNum := paramtable.Get().DataCoordCfg.MaxImportJobNum.GetAsInt() maxNum := paramtable.Get().DataCoordCfg.MaxImportJobNum.GetAsInt()
executingNum := importMeta.CountJobBy(ctx, WithoutJobStates(internalpb.ImportJobState_Completed, internalpb.ImportJobState_Failed)) executingNum := importMeta.CountJobBy(ctx, WithoutJobStates(internalpb.ImportJobState_Completed, internalpb.ImportJobState_Failed))
if executingNum >= maxNum { if executingNum >= maxNum {
return merr.WrapErrImportFailed( return merr.WrapErrImportSysFailed(
fmt.Sprintf("The number of jobs has reached the limit, please try again later. " + fmt.Sprintf("The number of jobs has reached the limit, please try again later. " +
"If your request is set to only import a single file, " + "If your request is set to only import a single file, " +
"please consider importing multiple files in one request for better efficiency.")) "please consider importing multiple files in one request for better efficiency."))
+9 -5
View File
@@ -468,7 +468,7 @@ func (m *indexMeta) canCreateIndex(req *indexpb.CreateIndexRequest, isJSON bool)
index.IndexName, index.FieldID, index.IndexParams, index.UserIndexParams, index.TypeParams)), index.IndexName, index.FieldID, index.IndexParams, index.UserIndexParams, index.TypeParams)),
zap.String("current index", fmt.Sprintf("{index_name: %s, field_id: %d, index_params: %v, user_params: %v, type_params: %v}", zap.String("current index", fmt.Sprintf("{index_name: %s, field_id: %d, index_params: %v, user_params: %v, type_params: %v}",
req.GetIndexName(), req.GetFieldID(), req.GetIndexParams(), req.GetUserIndexParams(), req.GetTypeParams()))) req.GetIndexName(), req.GetFieldID(), req.GetIndexParams(), req.GetUserIndexParams(), req.GetTypeParams())))
return 0, fmt.Errorf("CreateIndex failed: %s", errMsg) return 0, merr.WrapErrParameterInvalidMsg("%s", errMsg)
} }
if req.FieldID == index.FieldID { if req.FieldID == index.FieldID {
if isJSON { if isJSON {
@@ -484,7 +484,11 @@ func (m *indexMeta) canCreateIndex(req *indexpb.CreateIndexRequest, isJSON bool)
// creating multiple indexes on same field is not supported // creating multiple indexes on same field is not supported
errMsg := "CreateIndex failed: creating multiple indexes on same field is not supported" errMsg := "CreateIndex failed: creating multiple indexes on same field is not supported"
log.Warn(errMsg) log.Warn(errMsg)
return 0, errors.New(errMsg) // Client-caused conflict, same family as the "one distinct index per
// field" case above: return an input-class error rather than
// ServiceInternal (code 5, "never return out of Milvus") so the
// caller is not blamed with a system error / counted as fail_system.
return 0, merr.WrapErrParameterInvalidMsg("%s", errMsg)
} }
} }
return 0, nil return 0, nil
@@ -1012,7 +1016,7 @@ func (m *indexMeta) UpdateVersion(buildID, nodeID UniqueID) error {
log.Ctx(m.ctx).Info("IndexCoord metaTable UpdateVersion receive", zap.Int64("buildID", buildID), zap.Int64("nodeID", nodeID)) log.Ctx(m.ctx).Info("IndexCoord metaTable UpdateVersion receive", zap.Int64("buildID", buildID), zap.Int64("nodeID", nodeID))
segIdx, ok := m.segmentBuildInfo.Get(buildID) segIdx, ok := m.segmentBuildInfo.Get(buildID)
if !ok { if !ok {
return fmt.Errorf("there is no index with buildID: %d", buildID) return merr.WrapErrServiceInternalMsg("there is no index with buildID: %d", buildID)
} }
updateFunc := func(segIdx *model.SegmentIndex) error { updateFunc := func(segIdx *model.SegmentIndex) error {
@@ -1031,7 +1035,7 @@ func (m *indexMeta) UpdateIndexState(buildID UniqueID, state commonpb.IndexState
segIdx, ok := m.segmentBuildInfo.Get(buildID) segIdx, ok := m.segmentBuildInfo.Get(buildID)
if !ok { if !ok {
log.Ctx(m.ctx).Warn("there is no index with buildID", zap.Int64("buildID", buildID)) log.Ctx(m.ctx).Warn("there is no index with buildID", zap.Int64("buildID", buildID))
return fmt.Errorf("there is no index with buildID: %d", buildID) return merr.WrapErrServiceInternalMsg("there is no index with buildID: %d", buildID)
} }
updateFunc := func(segIdx *model.SegmentIndex) error { updateFunc := func(segIdx *model.SegmentIndex) error {
@@ -1141,7 +1145,7 @@ func (m *indexMeta) BuildIndex(buildID UniqueID) error {
segIdx, ok := m.segmentBuildInfo.Get(buildID) segIdx, ok := m.segmentBuildInfo.Get(buildID)
if !ok { if !ok {
return fmt.Errorf("there is no index with buildID: %d", buildID) return merr.WrapErrServiceInternalMsg("there is no index with buildID: %d", buildID)
} }
updateFunc := func(segIdx *model.SegmentIndex) error { updateFunc := func(segIdx *model.SegmentIndex) error {
+1 -1
View File
@@ -1052,7 +1052,7 @@ func (s *Server) DropIndex(ctx context.Context, req *indexpb.DropIndexRequest) (
} }
if typeutil.IsVectorType(field.GetDataType()) { if typeutil.IsVectorType(field.GetDataType()) {
log.Warn("vector index cannot be dropped on loaded collection", zap.String("indexName", req.IndexName), zap.Int64("collectionID", req.GetCollectionID()), zap.Int64("fieldID", index.FieldID)) log.Warn("vector index cannot be dropped on loaded collection", zap.String("indexName", req.IndexName), zap.Int64("collectionID", req.GetCollectionID()), zap.Int64("fieldID", index.FieldID))
return merr.Status(merr.WrapErrParameterInvalidMsg(fmt.Sprintf("vector index cannot be dropped on loaded collection: %d", req.GetCollectionID()))), nil return merr.Status(merr.WrapErrParameterInvalidMsg("vector index cannot be dropped on loaded collection: %d", req.GetCollectionID())), nil
} }
} }
} }
+20 -12
View File
@@ -816,7 +816,7 @@ func (m *meta) GetSegmentsChannels(segmentIDs []UniqueID) (map[int64]string, err
for _, segmentID := range segmentIDs { for _, segmentID := range segmentIDs {
segment := m.segments.GetSegment(segmentID) segment := m.segments.GetSegment(segmentID)
if segment == nil { if segment == nil {
return nil, errors.New(fmt.Sprintf("cannot find segment %d", segmentID)) return nil, merr.WrapErrServiceInternalMsg("cannot find segment %d", segmentID)
} }
segChannels[segmentID] = segment.GetInsertChannel() segChannels[segmentID] = segment.GetInsertChannel()
} }
@@ -840,7 +840,7 @@ func (m *meta) SetState(ctx context.Context, segmentID UniqueID, targetState com
if targetState == commonpb.SegmentState_Dropped { if targetState == commonpb.SegmentState_Dropped {
return nil return nil
} }
return fmt.Errorf("segment is not exist with ID = %d", segmentID) return merr.WrapErrSegmentNotFound(segmentID)
} }
// Persist segment updates first. // Persist segment updates first.
clonedSegment := curSegInfo.Clone() clonedSegment := curSegInfo.Clone()
@@ -941,12 +941,12 @@ func (p *updateSegmentPack) Validate() error {
segmentInMeta := p.meta.segments.GetSegment(segment.ID) segmentInMeta := p.meta.segments.GetSegment(segment.ID)
if segmentInMeta.State == commonpb.SegmentState_Flushed && segment.State != commonpb.SegmentState_Dropped { if segmentInMeta.State == commonpb.SegmentState_Flushed && segment.State != commonpb.SegmentState_Dropped {
// if the segment is flushed, we should not update the segment meta, ignore the operation directly. // if the segment is flushed, we should not update the segment meta, ignore the operation directly.
return errors.Wrapf(ErrIgnoredSegmentMetaOperation, return merr.Wrapf(errIgnoredSegmentMetaOperation,
"segment is flushed, segmentID: %d", "segment is flushed, segmentID: %d",
segment.ID) segment.ID)
} }
if segment.GetDmlPosition().GetTimestamp() < segmentInMeta.GetDmlPosition().GetTimestamp() { if segment.GetDmlPosition().GetTimestamp() < segmentInMeta.GetDmlPosition().GetTimestamp() {
return errors.Wrapf(ErrIgnoredSegmentMetaOperation, return merr.Wrapf(errIgnoredSegmentMetaOperation,
"dml time tick is less than the segment meta, segmentID: %d, new incoming time tick: %d, existing time tick: %d", "dml time tick is less than the segment meta, segmentID: %d, new incoming time tick: %d, existing time tick: %d",
segment.ID, segment.ID,
segment.GetDmlPosition().GetTimestamp(), segment.GetDmlPosition().GetTimestamp(),
@@ -1223,7 +1223,7 @@ func updateManifestPathIfNewer(segment *SegmentInfo, manifestPath string) error
return err return err
} }
if currentBase != incomingBase { if currentBase != incomingBase {
return merr.WrapErrServiceInternal(fmt.Sprintf("manifest base path mismatch for segment %d: current %s, incoming %s", segment.GetID(), currentBase, incomingBase)) return merr.WrapErrServiceInternalMsg("manifest base path mismatch for segment %d: current %s, incoming %s", segment.GetID(), currentBase, incomingBase)
} }
if incomingVersion > currentVersion { if incomingVersion > currentVersion {
segment.ManifestPath = manifestPath segment.ManifestPath = manifestPath
@@ -1824,6 +1824,14 @@ func (m *meta) UpdateSegmentsInfo(ctx context.Context, operators ...UpdateOperat
// Validate the update pack. // Validate the update pack.
if err := updatePack.Validate(); err != nil { if err := updatePack.Validate(); err != nil {
// A stale save-binlog-paths update (segment already flushed, or an
// outdated time tick) is a benign no-op: skip the meta write and
// report success so the caller does not retry. The signal stays
// inside this package on purpose; see errIgnoredSegmentMetaOperation.
if errors.Is(err, errIgnoredSegmentMetaOperation) {
log.Ctx(ctx).Info("meta update: ignored stale segment meta operation", zap.Error(err))
return nil
}
return err return err
} }
updatePack.prepareSegmentMetricUpdates() updatePack.prepareSegmentMetricUpdates()
@@ -2100,7 +2108,7 @@ func (m *meta) AddAllocation(segmentID UniqueID, allocation *Allocation) error {
if curSegInfo == nil { if curSegInfo == nil {
// TODO: Error handling. // TODO: Error handling.
log.Ctx(m.ctx).Error("meta update: add allocation failed - segment not found", zap.Int64("segmentID", segmentID)) log.Ctx(m.ctx).Error("meta update: add allocation failed - segment not found", zap.Int64("segmentID", segmentID))
return errors.New("meta update: add allocation failed - segment not found") return merr.WrapErrSegmentNotFound(segmentID, "meta update: add allocation failed")
} }
// As we use global segment lastExpire to guarantee data correctness after restart // As we use global segment lastExpire to guarantee data correctness after restart
// there is no need to persist allocation to meta store, only update allocation in-memory meta. // there is no need to persist allocation to meta store, only update allocation in-memory meta.
@@ -2287,7 +2295,7 @@ func validateCompactionFallbackStartPosition(compactFromSegInfos []*SegmentInfo,
if maxCommitTs == 0 || fallbackStart.GetTimestamp() >= maxCommitTs { if maxCommitTs == 0 || fallbackStart.GetTimestamp() >= maxCommitTs {
return nil return nil
} }
return errors.Errorf( return merr.WrapErrServiceInternalMsg(
"compaction fallback start position timestamp %d is earlier than max input commit timestamp %d", "compaction fallback start position timestamp %d is earlier than max input commit timestamp %d",
fallbackStart.GetTimestamp(), fallbackStart.GetTimestamp(),
maxCommitTs) maxCommitTs)
@@ -2661,7 +2669,7 @@ func (m *meta) HasSegments(segIDs []UniqueID) (bool, error) {
for _, segID := range segIDs { for _, segID := range segIDs {
if _, ok := m.segments.segments[segID]; !ok { if _, ok := m.segments.segments[segID]; !ok {
return false, fmt.Errorf("segment is not exist with ID = %d", segID) return false, merr.WrapErrServiceInternalMsg("segment is not exist with ID = %d", segID)
} }
} }
return true, nil return true, nil
@@ -2747,7 +2755,7 @@ func (m *meta) collectionHasTextFields(collectionID int64) bool {
// UpdateChannelCheckpoint updates and saves channel checkpoint. // UpdateChannelCheckpoint updates and saves channel checkpoint.
func (m *meta) UpdateChannelCheckpoint(ctx context.Context, vChannel string, pos *msgpb.MsgPosition) error { func (m *meta) UpdateChannelCheckpoint(ctx context.Context, vChannel string, pos *msgpb.MsgPosition) error {
if pos == nil || pos.GetMsgID() == nil { if pos == nil || pos.GetMsgID() == nil {
return fmt.Errorf("channelCP is nil, vChannel=%s", vChannel) return merr.WrapErrServiceInternalMsg("channelCP is nil, vChannel=%s", vChannel)
} }
// Clamp checkpoint to not advance beyond min growing segment checkpoint. // Clamp checkpoint to not advance beyond min growing segment checkpoint.
@@ -3331,7 +3339,7 @@ func (m *meta) completeBumpSchemaVersionCompactionMutation(
if currentManifest != resultManifest { if currentManifest != resultManifest {
manifestCompare, err := packed.CompareManifestPath(resultManifest, currentManifest) manifestCompare, err := packed.CompareManifestPath(resultManifest, currentManifest)
if err != nil { if err != nil {
return nil, nil, merr.WrapErrIllegalCompactionPlan(fmt.Sprintf("schema bump compaction result manifest is not comparable with current manifest: %v", err)) return nil, nil, merr.WrapErrIllegalCompactionPlanMsg("schema bump compaction result manifest is not comparable with current manifest: %v", err)
} }
if manifestCompare <= 0 { if manifestCompare <= 0 {
return nil, nil, merr.WrapErrIllegalCompactionPlan("schema bump compaction result manifest is not newer than current manifest") return nil, nil, merr.WrapErrIllegalCompactionPlan("schema bump compaction result manifest is not newer than current manifest")
@@ -3392,7 +3400,7 @@ func (m *meta) completeBumpSchemaVersionReplacementMutation(
) ([]*SegmentInfo, *segMetricMutation, error) { ) ([]*SegmentInfo, *segMetricMutation, error) {
idRange := t.GetPreAllocatedSegmentIDs() idRange := t.GetPreAllocatedSegmentIDs()
if idRange == nil || idRange.GetBegin()+1 != idRange.GetEnd() || resultSegment.GetSegmentID() != idRange.GetBegin() { if idRange == nil || idRange.GetBegin()+1 != idRange.GetEnd() || resultSegment.GetSegmentID() != idRange.GetBegin() {
return nil, nil, merr.WrapErrIllegalCompactionPlan(fmt.Sprintf("schema bump replacement result segment ID %d does not match the pre-allocated segment ID", resultSegment.GetSegmentID())) return nil, nil, merr.WrapErrIllegalCompactionPlanMsg("schema bump replacement result segment ID %d does not match the pre-allocated segment ID", resultSegment.GetSegmentID())
} }
dropped := oldSegment.Clone() dropped := oldSegment.Clone()
@@ -3555,7 +3563,7 @@ func (m *meta) GetFileResources(ctx context.Context, resourceIDs ...int64) ([]*i
if resource, ok := m.resourceIDMap[id]; ok { if resource, ok := m.resourceIDMap[id]; ok {
resources = append(resources, resource) resources = append(resources, resource)
} else { } else {
return nil, errors.Errorf("file resource %d not found", id) return nil, merr.WrapErrServiceInternalMsg("file resource %d not found", id)
} }
} }
return resources, nil return resources, nil
+2 -2
View File
@@ -3498,7 +3498,7 @@ func TestUpdateSegmentsInfo(t *testing.T) {
UpdateStartPosition([]*datapb.SegmentStartPosition{{SegmentID: 1, StartPosition: &msgpb.MsgPosition{MsgID: []byte{1, 2, 3}}}}), UpdateStartPosition([]*datapb.SegmentStartPosition{{SegmentID: 1, StartPosition: &msgpb.MsgPosition{MsgID: []byte{1, 2, 3}}}}),
UpdateCheckPointOperator(1, []*datapb.CheckPoint{{SegmentID: 1, NumOfRows: 10, Position: &msgpb.MsgPosition{MsgID: []byte{1, 2, 3}, Timestamp: 99}}}, true), UpdateCheckPointOperator(1, []*datapb.CheckPoint{{SegmentID: 1, NumOfRows: 10, Position: &msgpb.MsgPosition{MsgID: []byte{1, 2, 3}, Timestamp: 99}}}, true),
) )
assert.True(t, errors.Is(err, ErrIgnoredSegmentMetaOperation)) assert.NoError(t, err) // stale update is swallowed as a benign no-op; segment must stay unchanged below
err = meta.UpdateSegmentsInfo( err = meta.UpdateSegmentsInfo(
context.TODO(), context.TODO(),
@@ -3536,7 +3536,7 @@ func TestUpdateSegmentsInfo(t *testing.T) {
[]*datapb.FieldBinlog{}), []*datapb.FieldBinlog{}),
UpdateCheckPointOperator(1, []*datapb.CheckPoint{{SegmentID: 1, NumOfRows: 12, Position: &msgpb.MsgPosition{MsgID: []byte{1, 2, 3}, Timestamp: 101}}}, true), UpdateCheckPointOperator(1, []*datapb.CheckPoint{{SegmentID: 1, NumOfRows: 12, Position: &msgpb.MsgPosition{MsgID: []byte{1, 2, 3}, Timestamp: 101}}}, true),
) )
assert.True(t, errors.Is(err, ErrIgnoredSegmentMetaOperation)) assert.NoError(t, err) // stale update is swallowed as a benign no-op; segment must stay unchanged below
updated = meta.GetHealthySegment(context.TODO(), 1) updated = meta.GetHealthySegment(context.TODO(), 1)
assert.Equal(t, updated.NumOfRows, int64(20)) assert.Equal(t, updated.NumOfRows, int64(20))
+10 -1
View File
@@ -22,7 +22,16 @@ import (
"github.com/milvus-io/milvus/pkg/v3/proto/datapb" "github.com/milvus-io/milvus/pkg/v3/proto/datapb"
) )
var ErrIgnoredSegmentMetaOperation = errors.New("ignored segment meta operation") // errIgnoredSegmentMetaOperation is an internal control-flow signal: the
// requested segment meta update is stale (segment already flushed, or an
// outdated time tick) and is skipped as a benign no-op. It never crosses the
// package boundary — UpdateSegmentsInfo swallows it and returns nil.
//
// It MUST stay a plain errors.New identity sentinel. A merr-typed value would
// make errors.Is match by error code (milvusError.Is compares codes only), so
// any error sharing the code — e.g. a real etcd write failure wrapped as
// ServiceInternal — would be misread as ignorable and acknowledged as success.
var errIgnoredSegmentMetaOperation = errors.New("ignored segment meta operation")
// reviseVChannelInfo will revise the datapb.VchannelInfo for upgrade compatibility from 2.0.2 // reviseVChannelInfo will revise the datapb.VchannelInfo for upgrade compatibility from 2.0.2
func reviseVChannelInfo(vChannel *datapb.VchannelInfo) { func reviseVChannelInfo(vChannel *datapb.VchannelInfo) {
+2 -4
View File
@@ -18,10 +18,8 @@ package datacoord
import ( import (
"context" "context"
"fmt"
"sync" "sync"
"github.com/cockroachdb/errors"
"github.com/tidwall/gjson" "github.com/tidwall/gjson"
"go.uber.org/zap" "go.uber.org/zap"
"golang.org/x/sync/errgroup" "golang.org/x/sync/errgroup"
@@ -138,7 +136,7 @@ func (s *Server) getSegmentsJSON(ctx context.Context, req *milvuspb.GetMetricsRe
} }
return string(bs), nil return string(bs), nil
} }
return "", fmt.Errorf("invalid param value in=[%s], it should be dc or dn", in) return "", merr.WrapErrParameterInvalidMsg("invalid param value in=[%s], it should be dc or dn", in)
} }
func (s *Server) getDistJSON(ctx context.Context, req *milvuspb.GetMetricsRequest) string { func (s *Server) getDistJSON(ctx context.Context, req *milvuspb.GetMetricsRequest) string {
@@ -308,7 +306,7 @@ func (s *Server) getIndexNodeMetrics(ctx context.Context, req *milvuspb.GetMetri
}, },
} }
if node == nil { if node == nil {
return infos, errors.New("IndexNode is nil") return infos, merr.WrapErrServiceInternalMsg("index node is nil")
} }
metrics, err := node.GetMetrics(ctx, req) metrics, err := node.GetMetrics(ctx, req)
+2 -3
View File
@@ -2,7 +2,6 @@ package datacoord
import ( import (
"context" "context"
"fmt"
"sync" "sync"
"go.uber.org/zap" "go.uber.org/zap"
@@ -183,11 +182,11 @@ func (psm *partitionStatsMeta) innerSaveCurrentPartitionStatsVersion(collectionI
if _, ok := psm.partitionStatsInfos[vChannel]; !ok { if _, ok := psm.partitionStatsInfos[vChannel]; !ok {
return merr.WrapErrClusteringCompactionMetaError("SaveCurrentPartitionStatsVersion", return merr.WrapErrClusteringCompactionMetaError("SaveCurrentPartitionStatsVersion",
fmt.Errorf("update current partition stats version failed, there is no partition info exists with collID: %d, partID: %d, vChannel: %s", collectionID, partitionID, vChannel)) merr.WrapErrServiceInternalMsg("update current partition stats version failed, there is no partition info exists with collID: %d, partID: %d, vChannel: %s", collectionID, partitionID, vChannel))
} }
if _, ok := psm.partitionStatsInfos[vChannel][partitionID]; !ok { if _, ok := psm.partitionStatsInfos[vChannel][partitionID]; !ok {
return merr.WrapErrClusteringCompactionMetaError("SaveCurrentPartitionStatsVersion", return merr.WrapErrClusteringCompactionMetaError("SaveCurrentPartitionStatsVersion",
fmt.Errorf("update current partition stats version failed, there is no partition info exists with collID: %d, partID: %d, vChannel: %s", collectionID, partitionID, vChannel)) merr.WrapErrServiceInternalMsg("update current partition stats version failed, there is no partition info exists with collID: %d, partID: %d, vChannel: %s", collectionID, partitionID, vChannel))
} }
if err := psm.catalog.SaveCurrentPartitionStatsVersion(psm.ctx, collectionID, partitionID, vChannel, currentPartitionStatsVersion); err != nil { if err := psm.catalog.SaveCurrentPartitionStatsVersion(psm.ctx, collectionID, partitionID, vChannel, currentPartitionStatsVersion); err != nil {
@@ -23,12 +23,12 @@ import (
"sort" "sort"
"time" "time"
"github.com/cockroachdb/errors"
"github.com/samber/lo" "github.com/samber/lo"
"github.com/milvus-io/milvus-proto/go-api/v3/commonpb" "github.com/milvus-io/milvus-proto/go-api/v3/commonpb"
"github.com/milvus-io/milvus-proto/go-api/v3/schemapb" "github.com/milvus-io/milvus-proto/go-api/v3/schemapb"
"github.com/milvus-io/milvus/pkg/v3/proto/datapb" "github.com/milvus-io/milvus/pkg/v3/proto/datapb"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/paramtable" "github.com/milvus-io/milvus/pkg/v3/util/paramtable"
"github.com/milvus-io/milvus/pkg/v3/util/tsoutil" "github.com/milvus-io/milvus/pkg/v3/util/tsoutil"
"github.com/milvus-io/milvus/pkg/v3/util/typeutil" "github.com/milvus-io/milvus/pkg/v3/util/typeutil"
@@ -38,7 +38,7 @@ type calUpperLimitPolicy func(schema *schemapb.CollectionSchema) (int, error)
func calBySchemaPolicy(schema *schemapb.CollectionSchema) (int, error) { func calBySchemaPolicy(schema *schemapb.CollectionSchema) (int, error) {
if schema == nil { if schema == nil {
return -1, errors.New("nil schema") return -1, merr.WrapErrServiceInternalMsg("nil schema")
} }
sizePerRecord, err := typeutil.EstimateSizePerRecord(schema) sizePerRecord, err := typeutil.EstimateSizePerRecord(schema)
if err != nil { if err != nil {
@@ -46,7 +46,7 @@ func calBySchemaPolicy(schema *schemapb.CollectionSchema) (int, error) {
} }
// check zero value, preventing panicking // check zero value, preventing panicking
if sizePerRecord == 0 { if sizePerRecord == 0 {
return -1, errors.New("zero size record schema found") return -1, merr.WrapErrServiceInternalMsg("zero size record schema found")
} }
threshold := Params.DataCoordCfg.SegmentMaxSize.GetAsFloat() * 1024 * 1024 threshold := Params.DataCoordCfg.SegmentMaxSize.GetAsFloat() * 1024 * 1024
return int(threshold / float64(sizePerRecord)), nil return int(threshold / float64(sizePerRecord)), nil
+3 -3
View File
@@ -23,7 +23,6 @@ import (
"sync" "sync"
"time" "time"
"github.com/cockroachdb/errors"
"github.com/samber/lo" "github.com/samber/lo"
"go.opentelemetry.io/otel" "go.opentelemetry.io/otel"
"go.uber.org/zap" "go.uber.org/zap"
@@ -36,6 +35,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/log" "github.com/milvus-io/milvus/pkg/v3/log"
"github.com/milvus-io/milvus/pkg/v3/proto/datapb" "github.com/milvus-io/milvus/pkg/v3/proto/datapb"
"github.com/milvus-io/milvus/pkg/v3/util/lock" "github.com/milvus-io/milvus/pkg/v3/util/lock"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/metautil" "github.com/milvus-io/milvus/pkg/v3/util/metautil"
"github.com/milvus-io/milvus/pkg/v3/util/paramtable" "github.com/milvus-io/milvus/pkg/v3/util/paramtable"
"github.com/milvus-io/milvus/pkg/v3/util/retry" "github.com/milvus-io/milvus/pkg/v3/util/retry"
@@ -299,7 +299,7 @@ func (s *SegmentManager) genLastExpireTsForSegments() (Timestamp, error) {
}, retry.Attempts(Params.DataCoordCfg.AllocLatestExpireAttempt.GetAsUint()), retry.Sleep(200*time.Millisecond)) }, retry.Attempts(Params.DataCoordCfg.AllocLatestExpireAttempt.GetAsUint()), retry.Sleep(200*time.Millisecond))
if allocateErr != nil { if allocateErr != nil {
log.Warn("cannot allocate latest lastExpire from rootCoord", zap.Error(allocateErr)) log.Warn("cannot allocate latest lastExpire from rootCoord", zap.Error(allocateErr))
return 0, errors.New("global max expire ts is unavailable for segment manager") return 0, merr.WrapErrServiceInternalMsg("global max expire ts is unavailable for segment manager")
} }
return latestTs, nil return latestTs, nil
} }
@@ -466,7 +466,7 @@ func (s *SegmentManager) estimateMaxNumOfRows(collectionID UniqueID) (int, error
// it's ok to use meta.GetCollection here, since collection meta is set before using segmentManager // it's ok to use meta.GetCollection here, since collection meta is set before using segmentManager
collMeta := s.meta.GetCollection(collectionID) collMeta := s.meta.GetCollection(collectionID)
if collMeta == nil { if collMeta == nil {
return -1, fmt.Errorf("failed to get collection %d", collectionID) return -1, merr.WrapErrServiceInternalMsg("failed to get collection %d", collectionID)
} }
return s.estimatePolicy(collMeta.Schema) return s.estimatePolicy(collMeta.Schema)
} }
+4 -5
View File
@@ -26,7 +26,6 @@ import (
"time" "time"
"github.com/blang/semver/v4" "github.com/blang/semver/v4"
"github.com/cockroachdb/errors"
"github.com/samber/lo" "github.com/samber/lo"
"github.com/tidwall/gjson" "github.com/tidwall/gjson"
"github.com/tikv/client-go/v2/txnkv" "github.com/tikv/client-go/v2/txnkv"
@@ -462,7 +461,7 @@ func (s *Server) SetSession(session sessionutil.SessionInterface) error {
s.session = session s.session = session
s.icSession = session s.icSession = session
if s.session == nil { if s.session == nil {
return errors.New("session is nil, the etcd client connection may have failed") return merr.WrapErrServiceNotReadyMsg("session is nil, the etcd client connection may have failed")
} }
return nil return nil
} }
@@ -618,7 +617,7 @@ func (s *Server) initKV() error {
s.kv = etcdkv.NewEtcdKV(s.etcdCli, s.metaRootPath, s.kv = etcdkv.NewEtcdKV(s.etcdCli, s.metaRootPath,
etcdkv.WithRequestTimeout(paramtable.Get().EtcdCfg.RequestTimeout.GetAsDuration(time.Millisecond))) etcdkv.WithRequestTimeout(paramtable.Get().EtcdCfg.RequestTimeout.GetAsDuration(time.Millisecond)))
default: default:
return retry.Unrecoverable(fmt.Errorf("not supported meta store: %s", metaType)) return retry.Unrecoverable(merr.WrapErrServiceInternalMsg("unsupported meta store: %s", metaType))
} }
log.Info("data coordinator successfully connected to metadata store", zap.String("metaType", metaType)) log.Info("data coordinator successfully connected to metadata store", zap.String("metaType", metaType))
return nil return nil
@@ -1027,7 +1026,7 @@ func (s *Server) flushFlushingSegment(ctx context.Context, segmentID UniqueID) e
return ctx.Err() return ctx.Err()
} }
// underlying etcd may return context canceled, so we need to return a error to retry. // underlying etcd may return context canceled, so we need to return a error to retry.
return errors.New("flush segment complete failed") return merr.WrapErrServiceInternalMsg("flush segment complete failed")
} }
return nil return nil
}, retry.AttemptAlways()) }, retry.AttemptAlways())
@@ -1117,7 +1116,7 @@ func (s *Server) CleanMeta() error {
err2 := s.watchClient.RemoveWithPrefix(s.ctx, "") err2 := s.watchClient.RemoveWithPrefix(s.ctx, "")
if err2 != nil { if err2 != nil {
if err != nil { if err != nil {
err = fmt.Errorf("failed to CleanMeta[metadata cleanup error: %w][watchdata cleanup error: %v]", err, err2) err = merr.Wrapf(err, "failed to clean meta (watchdata cleanup error: %v)", err2)
} else { } else {
err = err2 err = err2
} }
+21 -22
View File
@@ -137,7 +137,7 @@ func (s *Server) flushCollection(ctx context.Context, collectionID UniqueID, flu
for _, channel := range coll.VChannelNames { for _, channel := range coll.VChannelNames {
sealedSegmentIDs, err := s.segmentManager.SealAllSegments(ctx, channel, toFlushSegments) sealedSegmentIDs, err := s.segmentManager.SealAllSegments(ctx, channel, toFlushSegments)
if err != nil { if err != nil {
return nil, errors.Wrapf(err, "failed to flush collection %d", collectionID) return nil, merr.Wrapf(err, "failed to flush collection %d", collectionID)
} }
for _, sealedSegmentID := range sealedSegmentIDs { for _, sealedSegmentID := range sealedSegmentIDs {
sealedSegmentsIDDict[sealedSegmentID] = true sealedSegmentsIDDict[sealedSegmentID] = true
@@ -691,13 +691,12 @@ func (s *Server) SaveBinlogPaths(ctx context.Context, req *datapb.SaveBinlogPath
UpdateAsDroppedIfEmptyWhenFlushing(req.GetSegmentID()), UpdateAsDroppedIfEmptyWhenFlushing(req.GetSegmentID()),
) )
// Update segment info in memory and meta. // Update segment info in memory and meta. Stale updates (segment already
// flushed / outdated time tick) are swallowed inside UpdateSegmentsInfo as
// benign no-ops, so any error here is a real failure.
if err := s.meta.UpdateSegmentsInfo(ctx, operators...); err != nil { if err := s.meta.UpdateSegmentsInfo(ctx, operators...); err != nil {
if !errors.Is(err, ErrIgnoredSegmentMetaOperation) { log.Error("save binlog and checkpoints failed", zap.Error(err))
log.Error("save binlog and checkpoints failed", zap.Error(err)) return merr.Status(err), nil
return merr.Status(err), nil
}
log.Info("save binlog and checkpoints failed with ignorable error", zap.Error(err))
} }
s.meta.SetLastWrittenTime(req.GetSegmentID()) s.meta.SetLastWrittenTime(req.GetSegmentID())
@@ -1292,7 +1291,7 @@ func (s *Server) GetMetrics(ctx context.Context, req *milvuspb.GetMetricsRequest
msg := "failed to get metrics" msg := "failed to get metrics"
log.Warn(msg, zap.Error(err)) log.Warn(msg, zap.Error(err))
return &milvuspb.GetMetricsResponse{ return &milvuspb.GetMetricsResponse{
Status: merr.Status(errors.Wrap(err, msg)), Status: merr.Status(merr.Wrap(err, msg)),
}, nil }, nil
} }
@@ -1627,7 +1626,7 @@ OUTER:
break OUTER break OUTER
} }
} else { } else {
resp.Status = merr.Status(merr.WrapErrParameterInvalidMsg("FlushAllTss or FlushAllTs is required")) resp.Status = merr.Status(merr.WrapErrParameterMissingMsg("FlushAllTss or FlushAllTs is required"))
return resp, nil return resp, nil
} }
} }
@@ -1925,12 +1924,12 @@ func (s *Server) ImportV2(ctx context.Context, in *internalpb.ImportRequestInter
jobID := in.GetJobID() jobID := in.GetJobID()
if jobID == 0 { if jobID == 0 {
if s.allocator == nil { if s.allocator == nil {
resp.Status = merr.Status(merr.WrapErrImportFailed("allocator not initialized")) resp.Status = merr.Status(merr.WrapErrServiceUnavailable("allocator not initialized"))
return resp, nil return resp, nil
} }
jobID, _, err = s.allocator.AllocN(1) jobID, _, err = s.allocator.AllocN(1)
if err != nil { if err != nil {
resp.Status = merr.Status(merr.WrapErrImportFailed(fmt.Sprintf("failed to allocate job ID: %v", err))) resp.Status = merr.Status(merr.Wrap(err, "failed to allocate job ID"))
return resp, nil return resp, nil
} }
} }
@@ -1950,7 +1949,7 @@ func (s *Server) ImportV2(ctx context.Context, in *internalpb.ImportRequestInter
) )
if err != nil { if err != nil {
log.Warn("failed to broadcast import message", zap.Error(err)) log.Warn("failed to broadcast import message", zap.Error(err))
resp.Status = merr.Status(merr.WrapErrImportFailed(fmt.Sprintf("failed to broadcast import: %v", err))) resp.Status = merr.Status(merr.Wrap(err, "failed to broadcast import"))
return resp, nil return resp, nil
} }
@@ -2000,7 +1999,7 @@ func (s *Server) createImportJobFromAck(ctx context.Context, in *internalpb.Impo
// Allocate file ids. // Allocate file ids.
idStart, _, err := s.allocator.AllocN(int64(len(files)) + 1) idStart, _, err := s.allocator.AllocN(int64(len(files)) + 1)
if err != nil { if err != nil {
resp.Status = merr.Status(merr.WrapErrImportFailed(fmt.Sprintf("alloc id failed: %v", err))) resp.Status = merr.Status(merr.Wrap(err, "alloc id failed"))
return resp, nil return resp, nil
} }
files = lo.Map(files, func(importFile *internalpb.ImportFile, i int) *internalpb.ImportFile { files = lo.Map(files, func(importFile *internalpb.ImportFile, i int) *internalpb.ImportFile {
@@ -2013,7 +2012,7 @@ func (s *Server) createImportJobFromAck(ctx context.Context, in *internalpb.Impo
return resp, nil return resp, nil
} }
if err != nil { if err != nil {
resp.Status = merr.Status(merr.WrapErrImportFailed(fmt.Sprintf("get collection failed: %v", err))) resp.Status = merr.Status(merr.Wrap(err, "get collection failed"))
return resp, nil return resp, nil
} }
if importCollectionInfo == nil { if importCollectionInfo == nil {
@@ -2048,7 +2047,7 @@ func (s *Server) createImportJobFromAck(ctx context.Context, in *internalpb.Impo
} }
err = s.importMeta.AddJob(ctx, job) err = s.importMeta.AddJob(ctx, job)
if err != nil { if err != nil {
resp.Status = merr.Status(merr.WrapErrImportFailed(fmt.Sprintf("add import job failed: %v", err))) resp.Status = merr.Status(merr.Wrap(err, "add import job failed"))
return resp, nil return resp, nil
} }
@@ -2075,13 +2074,13 @@ func (s *Server) GetImportProgress(ctx context.Context, in *internalpb.GetImport
} }
jobID, err := strconv.ParseInt(in.GetJobID(), 10, 64) jobID, err := strconv.ParseInt(in.GetJobID(), 10, 64)
if err != nil { if err != nil {
resp.Status = merr.Status(merr.WrapErrImportFailed(fmt.Sprintf("parse job id failed: %v", err))) resp.Status = merr.Status(merr.WrapErrParameterInvalidMsg("parse job id failed: %v", err))
return resp, nil return resp, nil
} }
job := s.importMeta.GetJob(ctx, jobID) job := s.importMeta.GetJob(ctx, jobID)
if job == nil { if job == nil {
resp.Status = merr.Status(merr.WrapErrImportFailed(fmt.Sprintf("import job does not exist, jobID=%d", jobID))) resp.Status = merr.Status(merr.WrapErrImportSysFailedMsg("import job does not exist, jobID=%d", jobID))
return resp, nil return resp, nil
} }
progress, state, importedRows, totalRows, reason := GetJobProgress(ctx, jobID, s.importMeta, s.meta) progress, state, importedRows, totalRows, reason := GetJobProgress(ctx, jobID, s.importMeta, s.meta)
@@ -2481,14 +2480,14 @@ func (s *Server) RestoreSnapshot(ctx context.Context, req *datapb.RestoreSnapsho
// Validate parameters // Validate parameters
if req.GetName() == "" { if req.GetName() == "" {
err := merr.WrapErrParameterInvalidMsg("snapshot name is required") err := merr.WrapErrParameterMissingMsg("snapshot name is required")
log.Warn("invalid request", zap.Error(err)) log.Warn("invalid request", zap.Error(err))
return &datapb.RestoreSnapshotResponse{ return &datapb.RestoreSnapshotResponse{
Status: merr.Status(err), Status: merr.Status(err),
}, nil }, nil
} }
if req.GetTargetCollectionName() == "" { if req.GetTargetCollectionName() == "" {
err := merr.WrapErrParameterInvalidMsg("target collection name is required") err := merr.WrapErrParameterMissingMsg("target collection name is required")
log.Warn("invalid request", zap.Error(err)) log.Warn("invalid request", zap.Error(err))
return &datapb.RestoreSnapshotResponse{ return &datapb.RestoreSnapshotResponse{
Status: merr.Status(err), Status: merr.Status(err),
@@ -2887,7 +2886,7 @@ func (s *Server) ListRefreshExternalCollectionJobs(ctx context.Context, req *dat
func (s *Server) broadcastCommitImportMessage(ctx context.Context, job ImportJob) error { func (s *Server) broadcastCommitImportMessage(ctx context.Context, job ImportJob) error {
vchannels := job.GetVchannels() vchannels := job.GetVchannels()
if len(vchannels) == 0 { if len(vchannels) == 0 {
return merr.WrapErrImportFailed(fmt.Sprintf("job %d has no vchannels", job.GetJobID())) return merr.WrapErrImportSysFailedMsg("job %d has no vchannels", job.GetJobID())
} }
broadcaster, err := s.startBroadcastWithCollectionID(ctx, job.GetCollectionID()) broadcaster, err := s.startBroadcastWithCollectionID(ctx, job.GetCollectionID())
@@ -2914,7 +2913,7 @@ func (s *Server) broadcastCommitImportMessage(ctx context.Context, job ImportJob
func (s *Server) broadcastRollbackImportMessage(ctx context.Context, job ImportJob) error { func (s *Server) broadcastRollbackImportMessage(ctx context.Context, job ImportJob) error {
vchannels := job.GetVchannels() vchannels := job.GetVchannels()
if len(vchannels) == 0 { if len(vchannels) == 0 {
return merr.WrapErrImportFailed(fmt.Sprintf("job %d has no vchannels", job.GetJobID())) return merr.WrapErrImportSysFailedMsg("job %d has no vchannels", job.GetJobID())
} }
broadcaster, err := s.startBroadcastWithCollectionID(ctx, job.GetCollectionID()) broadcaster, err := s.startBroadcastWithCollectionID(ctx, job.GetCollectionID())
@@ -2949,7 +2948,7 @@ func (s *Server) validateAndExecuteImportAction(
} }
job := s.importMeta.GetJob(ctx, jobID) job := s.importMeta.GetJob(ctx, jobID)
if job == nil { if job == nil {
return merr.Status(merr.WrapErrImportFailed(fmt.Sprintf("job %d not found", jobID))), nil return merr.Status(merr.WrapErrImportSysFailedMsg("job %d not found", jobID)), nil
} }
if st := validateState(job); st != nil { if st := validateState(job); st != nil {
return st, nil return st, nil
@@ -186,7 +186,7 @@ const maxBackfillResultBytes int64 = 64 * 1024 * 1024 // 64MiB
// reject s3a://<other-bucket>/... paths early. // reject s3a://<other-bucket>/... paths early.
func (s *Server) loadBackfillResult(ctx context.Context, rawPath string) (*BackfillResult, error) { func (s *Server) loadBackfillResult(ctx context.Context, rawPath string) (*BackfillResult, error) {
if rawPath == "" { if rawPath == "" {
return nil, merr.WrapErrParameterInvalidMsg("result_path is required") return nil, merr.WrapErrParameterMissingMsg("result_path is required")
} }
bucket := bucketFromChunkManager(s.meta.chunkManager) bucket := bucketFromChunkManager(s.meta.chunkManager)
key, err := normalizeObjectKey(rawPath, bucket) key, err := normalizeObjectKey(rawPath, bucket)
+7 -7
View File
@@ -1645,7 +1645,6 @@ func TestGetRecoveryInfoV2(t *testing.T) {
func TestImportV2(t *testing.T) { func TestImportV2(t *testing.T) {
ctx := context.Background() ctx := context.Background()
mockErr := errors.New("mock err")
t.Run("ImportV2", func(t *testing.T) { t.Run("ImportV2", func(t *testing.T) {
// server not healthy // server not healthy
@@ -1682,11 +1681,11 @@ func TestImportV2(t *testing.T) {
s.importMeta, err = NewImportMeta(context.TODO(), catalog, nil, nil) s.importMeta, err = NewImportMeta(context.TODO(), catalog, nil, nil)
assert.NoError(t, err) assert.NoError(t, err)
alloc := allocator.NewMockAllocator(t) alloc := allocator.NewMockAllocator(t)
alloc.EXPECT().AllocN(mock.Anything).Return(0, 0, mockErr) alloc.EXPECT().AllocN(mock.Anything).Return(0, 0, merr.WrapErrServiceUnavailable("mock err"))
s.allocator = alloc s.allocator = alloc
resp, err = s.ImportV2(ctx, &internalpb.ImportRequestInternal{}) resp, err = s.ImportV2(ctx, &internalpb.ImportRequestInternal{})
assert.NoError(t, err) assert.NoError(t, err)
assert.True(t, errors.Is(merr.Error(resp.GetStatus()), merr.ErrImportFailed)) assert.True(t, errors.Is(merr.Error(resp.GetStatus()), merr.ErrServiceUnavailable))
}) })
t.Run("GetImportProgress", func(t *testing.T) { t.Run("GetImportProgress", func(t *testing.T) {
@@ -1703,7 +1702,7 @@ func TestImportV2(t *testing.T) {
JobID: "@%$%$#%", JobID: "@%$%$#%",
}) })
assert.NoError(t, err) assert.NoError(t, err)
assert.True(t, errors.Is(merr.Error(resp.GetStatus()), merr.ErrImportFailed)) assert.True(t, errors.Is(merr.Error(resp.GetStatus()), merr.ErrParameterInvalid))
// job does not exist // job does not exist
catalog := mocks.NewDataCoordCatalog(t) catalog := mocks.NewDataCoordCatalog(t)
@@ -1723,7 +1722,8 @@ func TestImportV2(t *testing.T) {
JobID: "-1", JobID: "-1",
}) })
assert.NoError(t, err) assert.NoError(t, err)
assert.True(t, errors.Is(merr.Error(resp.GetStatus()), merr.ErrImportFailed)) // job-not-found is a server-side orchestration issue, not malformed user data
assert.True(t, errors.Is(merr.Error(resp.GetStatus()), merr.ErrImportSysFailed))
// normal case // normal case
var job ImportJob = &importJob{ var job ImportJob = &importJob{
@@ -3156,7 +3156,7 @@ func TestServer_RestoreSnapshot(t *testing.T) {
assert.NoError(t, err) assert.NoError(t, err)
assert.Error(t, merr.Error(resp.GetStatus())) assert.Error(t, merr.Error(resp.GetStatus()))
assert.True(t, errors.Is(merr.Error(resp.GetStatus()), merr.ErrParameterInvalid)) assert.True(t, errors.Is(merr.Error(resp.GetStatus()), merr.ErrParameterMissing))
}) })
t.Run("missing_collection_name", func(t *testing.T) { t.Run("missing_collection_name", func(t *testing.T) {
@@ -3173,7 +3173,7 @@ func TestServer_RestoreSnapshot(t *testing.T) {
assert.NoError(t, err) assert.NoError(t, err)
assert.Error(t, merr.Error(resp.GetStatus())) assert.Error(t, merr.Error(resp.GetStatus()))
assert.True(t, errors.Is(merr.Error(resp.GetStatus()), merr.ErrParameterInvalid)) assert.True(t, errors.Is(merr.Error(resp.GetStatus()), merr.ErrParameterMissing))
}) })
t.Run("snapshot_not_found", func(t *testing.T) { t.Run("snapshot_not_found", func(t *testing.T) {
+2 -2
View File
@@ -18,12 +18,12 @@ package session
import ( import (
"context" "context"
"fmt"
"github.com/cockroachdb/errors" "github.com/cockroachdb/errors"
"github.com/milvus-io/milvus/internal/types" "github.com/milvus-io/milvus/internal/types"
"github.com/milvus-io/milvus/pkg/v3/util/lock" "github.com/milvus-io/milvus/pkg/v3/util/lock"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
) )
var errDisposed = errors.New("client is disposed") var errDisposed = errors.New("client is disposed")
@@ -84,7 +84,7 @@ func (n *Session) GetOrCreateClient(ctx context.Context) (types.DataNodeClient,
} }
if n.clientCreator == nil { if n.clientCreator == nil {
return nil, fmt.Errorf("unable to create client for %s because of a nil client creator", n.info.Address) return nil, merr.WrapErrServiceInternalMsg("unable to create client for %s because of a nil client creator", n.info.Address)
} }
err := n.initClient(ctx) err := n.initClient(ctx)
+30 -29
View File
@@ -33,6 +33,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/log" "github.com/milvus-io/milvus/pkg/v3/log"
"github.com/milvus-io/milvus/pkg/v3/proto/datapb" "github.com/milvus-io/milvus/pkg/v3/proto/datapb"
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb" "github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
) )
// S3 snapshot storage path constants. // S3 snapshot storage path constants.
@@ -136,7 +137,7 @@ func getManifestSchemaByVersion(version int) (avro.Schema, error) {
case 3: case 3:
return getManifestSchemaV3() return getManifestSchemaV3()
default: default:
return nil, fmt.Errorf("unsupported manifest schema version: %d", version) return nil, merr.WrapErrServiceInternalMsg("unsupported manifest schema version: %d", version)
} }
} }
@@ -508,22 +509,22 @@ func GetSegmentManifestPath(manifestDir string, segmentID int64) string {
func (w *SnapshotWriter) Save(ctx context.Context, snapshot *SnapshotData) (string, error) { func (w *SnapshotWriter) Save(ctx context.Context, snapshot *SnapshotData) (string, error) {
// Validate input parameters // Validate input parameters
if snapshot == nil { if snapshot == nil {
return "", fmt.Errorf("snapshot cannot be nil") return "", merr.WrapErrServiceInternalMsg("snapshot cannot be nil")
} }
if snapshot.SnapshotInfo == nil { if snapshot.SnapshotInfo == nil {
return "", fmt.Errorf("snapshot info cannot be nil") return "", merr.WrapErrServiceInternalMsg("snapshot info cannot be nil")
} }
collectionID := snapshot.SnapshotInfo.GetCollectionId() collectionID := snapshot.SnapshotInfo.GetCollectionId()
if collectionID <= 0 { if collectionID <= 0 {
return "", fmt.Errorf("invalid collection ID: %d", collectionID) return "", merr.WrapErrServiceInternalMsg("invalid collection ID: %d", collectionID)
} }
if snapshot.Collection == nil { if snapshot.Collection == nil {
return "", fmt.Errorf("collection description cannot be nil") return "", merr.WrapErrServiceInternalMsg("collection description cannot be nil")
} }
snapshotID := snapshot.SnapshotInfo.GetId() snapshotID := snapshot.SnapshotInfo.GetId()
if snapshotID <= 0 { if snapshotID <= 0 {
return "", fmt.Errorf("invalid snapshot ID: %d", snapshotID) return "", merr.WrapErrServiceInternalMsg("invalid snapshot ID: %d", snapshotID)
} }
manifestDir, metadataPath := GetSnapshotPaths(w.chunkManager.RootPath(), collectionID, snapshotID) manifestDir, metadataPath := GetSnapshotPaths(w.chunkManager.RootPath(), collectionID, snapshotID)
@@ -533,7 +534,7 @@ func (w *SnapshotWriter) Save(ctx context.Context, snapshot *SnapshotData) (stri
for _, segment := range snapshot.Segments { for _, segment := range snapshot.Segments {
manifestPath := GetSegmentManifestPath(manifestDir, segment.GetSegmentId()) manifestPath := GetSegmentManifestPath(manifestDir, segment.GetSegmentId())
if err := w.writeSegmentManifest(ctx, manifestPath, segment); err != nil { if err := w.writeSegmentManifest(ctx, manifestPath, segment); err != nil {
return "", fmt.Errorf("failed to write manifest for segment %d: %w", segment.GetSegmentId(), err) return "", merr.Wrapf(err, "failed to write manifest for segment %d", segment.GetSegmentId())
} }
manifestPaths = append(manifestPaths, manifestPath) manifestPaths = append(manifestPaths, manifestPath)
} }
@@ -557,7 +558,7 @@ func (w *SnapshotWriter) Save(ctx context.Context, snapshot *SnapshotData) (stri
// Step 3: Write metadata file with all manifest paths // Step 3: Write metadata file with all manifest paths
// This file is the entry point for reading the snapshot // This file is the entry point for reading the snapshot
if err := w.writeMetadataFile(ctx, metadataPath, snapshot, manifestPaths, storagev2Manifests); err != nil { if err := w.writeMetadataFile(ctx, metadataPath, snapshot, manifestPaths, storagev2Manifests); err != nil {
return "", fmt.Errorf("failed to write metadata file: %w", err) return "", merr.Wrap(err, "failed to write metadata file")
} }
log.Info("Successfully wrote metadata file", log.Info("Successfully wrote metadata file",
@@ -584,12 +585,12 @@ func (w *SnapshotWriter) writeSegmentManifest(ctx context.Context, manifestPath
// Serialize to Avro binary format (single record per file) // Serialize to Avro binary format (single record per file)
avroSchema, err := getManifestSchema() avroSchema, err := getManifestSchema()
if err != nil { if err != nil {
return fmt.Errorf("failed to get manifest schema: %w", err) return merr.WrapErrServiceInternalErr(err, "failed to get manifest schema")
} }
binaryData, err := avro.Marshal(avroSchema, entry) binaryData, err := avro.Marshal(avroSchema, entry)
if err != nil { if err != nil {
return fmt.Errorf("failed to serialize entry to avro: %w", err) return merr.WrapErrServiceInternalErr(err, "failed to serialize entry to avro")
} }
// Write to object storage // Write to object storage
@@ -634,7 +635,7 @@ func (w *SnapshotWriter) writeMetadataFile(ctx context.Context, metadataPath str
} }
jsonData, err := opts.Marshal(metadata) jsonData, err := opts.Marshal(metadata)
if err != nil { if err != nil {
return fmt.Errorf("failed to marshal metadata to JSON: %w", err) return merr.WrapErrServiceInternalErr(err, "failed to marshal metadata to JSON")
} }
// Write to object storage // Write to object storage
@@ -663,13 +664,13 @@ func (w *SnapshotWriter) writeMetadataFile(ctx context.Context, metadataPath str
func (w *SnapshotWriter) Drop(ctx context.Context, metadataFilePath string) error { func (w *SnapshotWriter) Drop(ctx context.Context, metadataFilePath string) error {
// Validate input parameter // Validate input parameter
if metadataFilePath == "" { if metadataFilePath == "" {
return fmt.Errorf("metadata file path cannot be empty") return merr.WrapErrServiceInternalMsg("metadata file path cannot be empty")
} }
// Step 1: Read metadata file to discover manifest file paths // Step 1: Read metadata file to discover manifest file paths
metadata, err := w.readMetadataFile(ctx, metadataFilePath) metadata, err := w.readMetadataFile(ctx, metadataFilePath)
if err != nil { if err != nil {
return fmt.Errorf("failed to read metadata file: %w", err) return merr.Wrap(err, "failed to read metadata file")
} }
snapshotID := metadata.GetSnapshotInfo().GetId() snapshotID := metadata.GetSnapshotInfo().GetId()
@@ -678,7 +679,7 @@ func (w *SnapshotWriter) Drop(ctx context.Context, metadataFilePath string) erro
manifestList := metadata.GetManifestList() manifestList := metadata.GetManifestList()
if len(manifestList) > 0 { if len(manifestList) > 0 {
if err := w.chunkManager.MultiRemove(ctx, manifestList); err != nil { if err := w.chunkManager.MultiRemove(ctx, manifestList); err != nil {
return fmt.Errorf("failed to remove manifest files: %w", err) return merr.Wrap(err, "failed to remove manifest files")
} }
log.Info("Successfully removed manifest files", log.Info("Successfully removed manifest files",
zap.Int("count", len(manifestList)), zap.Int("count", len(manifestList)),
@@ -687,7 +688,7 @@ func (w *SnapshotWriter) Drop(ctx context.Context, metadataFilePath string) erro
// Step 3: Remove the metadata file (entry point) // Step 3: Remove the metadata file (entry point)
if err := w.chunkManager.Remove(ctx, metadataFilePath); err != nil { if err := w.chunkManager.Remove(ctx, metadataFilePath); err != nil {
return fmt.Errorf("failed to remove metadata file: %w", err) return merr.Wrap(err, "failed to remove metadata file")
} }
log.Info("Successfully removed metadata file", log.Info("Successfully removed metadata file",
zap.String("metadataFilePath", metadataFilePath)) zap.String("metadataFilePath", metadataFilePath))
@@ -703,7 +704,7 @@ func (w *SnapshotWriter) readMetadataFile(ctx context.Context, filePath string)
// Read raw JSON content from object storage // Read raw JSON content from object storage
data, err := w.chunkManager.Read(ctx, filePath) data, err := w.chunkManager.Read(ctx, filePath)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to read metadata file: %w", err) return nil, merr.Wrap(err, "failed to read metadata file")
} }
// Use protojson for deserialization to correctly handle protobuf oneof fields // Use protojson for deserialization to correctly handle protobuf oneof fields
@@ -712,7 +713,7 @@ func (w *SnapshotWriter) readMetadataFile(ctx context.Context, filePath string)
DiscardUnknown: true, DiscardUnknown: true,
} }
if err := opts.Unmarshal(data, metadata); err != nil { if err := opts.Unmarshal(data, metadata); err != nil {
return nil, fmt.Errorf("failed to parse metadata JSON: %w", err) return nil, merr.WrapErrServiceInternalErr(err, "failed to parse metadata JSON")
} }
return metadata, nil return metadata, nil
@@ -775,13 +776,13 @@ func NewSnapshotReader(cm storage.ChunkManager) *SnapshotReader {
func (r *SnapshotReader) ReadSnapshot(ctx context.Context, metadataFilePath string, includeSegments bool) (*SnapshotData, error) { func (r *SnapshotReader) ReadSnapshot(ctx context.Context, metadataFilePath string, includeSegments bool) (*SnapshotData, error) {
// Validate input // Validate input
if metadataFilePath == "" { if metadataFilePath == "" {
return nil, fmt.Errorf("metadata file path cannot be empty") return nil, merr.WrapErrServiceInternalMsg("metadata file path cannot be empty")
} }
// Step 1: Read metadata file (always required) // Step 1: Read metadata file (always required)
metadata, err := r.readMetadataFile(ctx, metadataFilePath) metadata, err := r.readMetadataFile(ctx, metadataFilePath)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to read metadata file: %w", err) return nil, merr.Wrap(err, "failed to read metadata file")
} }
// Step 2: Optionally read segment details from manifest files // Step 2: Optionally read segment details from manifest files
@@ -791,7 +792,7 @@ func (r *SnapshotReader) ReadSnapshot(ctx context.Context, metadataFilePath stri
for _, manifestPath := range metadata.GetManifestList() { for _, manifestPath := range metadata.GetManifestList() {
segments, err := r.readManifestFile(ctx, manifestPath, int(metadata.GetFormatVersion())) segments, err := r.readManifestFile(ctx, manifestPath, int(metadata.GetFormatVersion()))
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to read manifest file %s: %w", manifestPath, err) return nil, merr.Wrapf(err, "failed to read manifest file %s", manifestPath)
} }
allSegments = append(allSegments, segments...) allSegments = append(allSegments, segments...)
} }
@@ -830,7 +831,7 @@ func (r *SnapshotReader) readMetadataFile(ctx context.Context, filePath string)
// Read raw JSON content from object storage // Read raw JSON content from object storage
data, err := r.chunkManager.Read(ctx, filePath) data, err := r.chunkManager.Read(ctx, filePath)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to read metadata file: %w", err) return nil, merr.Wrap(err, "failed to read metadata file")
} }
// Use protojson for deserialization to correctly handle protobuf oneof fields // Use protojson for deserialization to correctly handle protobuf oneof fields
@@ -839,12 +840,12 @@ func (r *SnapshotReader) readMetadataFile(ctx context.Context, filePath string)
DiscardUnknown: true, DiscardUnknown: true,
} }
if err := opts.Unmarshal(data, metadata); err != nil { if err := opts.Unmarshal(data, metadata); err != nil {
return nil, fmt.Errorf("failed to parse metadata JSON: %w", err) return nil, merr.WrapErrServiceInternalErr(err, "failed to parse metadata JSON")
} }
// Validate format version compatibility // Validate format version compatibility
if err := validateFormatVersion(int(metadata.GetFormatVersion())); err != nil { if err := validateFormatVersion(int(metadata.GetFormatVersion())); err != nil {
return nil, fmt.Errorf("incompatible snapshot format: %w", err) return nil, merr.WrapErrServiceInternalErr(err, "incompatible snapshot format")
} }
return metadata, nil return metadata, nil
@@ -865,7 +866,7 @@ func validateFormatVersion(version int) error {
// Check if version is too new for current code // Check if version is too new for current code
if version > SnapshotFormatVersion { if version > SnapshotFormatVersion {
return fmt.Errorf("snapshot format version %d is too new, current supported version: %d (please upgrade Milvus)", return merr.WrapErrServiceInternalMsg("snapshot format version %d is too new, current supported version: %d (please upgrade Milvus)",
version, SnapshotFormatVersion) version, SnapshotFormatVersion)
} }
@@ -888,20 +889,20 @@ func (r *SnapshotReader) readManifestFile(ctx context.Context, filePath string,
// Read raw Avro content from object storage // Read raw Avro content from object storage
data, err := r.chunkManager.Read(ctx, filePath) data, err := r.chunkManager.Read(ctx, filePath)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to read manifest file: %w", err) return nil, merr.Wrap(err, "failed to read manifest file")
} }
// Get Avro schema for the specified format version // Get Avro schema for the specified format version
avroSchema, err := getManifestSchemaByVersion(formatVersion) avroSchema, err := getManifestSchemaByVersion(formatVersion)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to get manifest schema for version %d: %w", formatVersion, err) return nil, merr.WrapErrServiceInternalErr(err, "failed to get manifest schema for version %d", formatVersion)
} }
// Deserialize Avro binary data to single ManifestEntry record // Deserialize Avro binary data to single ManifestEntry record
var record ManifestEntry var record ManifestEntry
err = avro.Unmarshal(avroSchema, data, &record) err = avro.Unmarshal(avroSchema, data, &record)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to parse avro data: %w", err) return nil, merr.WrapErrServiceInternalErr(err, "failed to parse avro data")
} }
// Convert ManifestEntry record to protobuf SegmentDescription // Convert ManifestEntry record to protobuf SegmentDescription
@@ -974,7 +975,7 @@ func (r *SnapshotReader) readManifestFile(ctx context.Context, filePath string,
func (r *SnapshotReader) ListSnapshots(ctx context.Context, collectionID int64) ([]*datapb.SnapshotInfo, error) { func (r *SnapshotReader) ListSnapshots(ctx context.Context, collectionID int64) ([]*datapb.SnapshotInfo, error) {
// Validate input parameters // Validate input parameters
if collectionID <= 0 { if collectionID <= 0 {
return nil, fmt.Errorf("invalid collection ID: %d", collectionID) return nil, merr.WrapErrServiceInternalMsg("invalid collection ID: %d", collectionID)
} }
// Construct metadata directory path // Construct metadata directory path
@@ -984,7 +985,7 @@ func (r *SnapshotReader) ListSnapshots(ctx context.Context, collectionID int64)
// List all files in the metadata directory // List all files in the metadata directory
files, _, err := storage.ListAllChunkWithPrefix(ctx, r.chunkManager, metadataDir, false) files, _, err := storage.ListAllChunkWithPrefix(ctx, r.chunkManager, metadataDir, false)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to list metadata files: %w", err) return nil, merr.Wrap(storage.ToMilvusIoError(metadataDir, err), "failed to list metadata files")
} }
// Read each metadata file and extract SnapshotInfo // Read each metadata file and extract SnapshotInfo
+22 -22
View File
@@ -562,7 +562,7 @@ func (sm *snapshotManager) validateCMEKCompatibility(
// Get target database properties first (needed for both encrypted and non-encrypted snapshots) // Get target database properties first (needed for both encrypted and non-encrypted snapshots)
dbResp, err := sm.broker.DescribeDatabase(ctx, targetDbName) dbResp, err := sm.broker.DescribeDatabase(ctx, targetDbName)
if err != nil { if err != nil {
return fmt.Errorf("failed to describe target database %s: %w", targetDbName, err) return merr.Wrapf(err, "failed to describe target database %s", targetDbName)
} }
targetIsEncrypted := hookutil.IsDBEncrypted(dbResp.GetProperties()) targetIsEncrypted := hookutil.IsDBEncrypted(dbResp.GetProperties())
@@ -640,7 +640,7 @@ func (sm *snapshotManager) RestoreSnapshot(
// ======================================================================== // ========================================================================
phase0Lock, err := startRestoreLock(ctx, sourceCollectionID, snapshotName, targetDbName, targetCollectionName) phase0Lock, err := startRestoreLock(ctx, sourceCollectionID, snapshotName, targetDbName, targetCollectionName)
if err != nil { if err != nil {
return 0, fmt.Errorf("failed to acquire restore lock: %w", err) return 0, merr.Wrap(err, "failed to acquire restore lock")
} }
// Pin the source snapshot while holding the phase-0 lock. The pin is the // Pin the source snapshot while holding the phase-0 lock. The pin is the
@@ -662,7 +662,7 @@ func (sm *snapshotManager) RestoreSnapshot(
pinID, activePins, err := sm.snapshotMeta.PinSnapshot(ctx, sourceCollectionID, snapshotName, pinTTLSeconds) pinID, activePins, err := sm.snapshotMeta.PinSnapshot(ctx, sourceCollectionID, snapshotName, pinTTLSeconds)
if err != nil { if err != nil {
phase0Lock.Close() phase0Lock.Close()
return 0, fmt.Errorf("failed to pin source snapshot under restore lock: %w", err) return 0, merr.Wrap(err, "failed to pin source snapshot under restore lock")
} }
setSnapshotActivePinsGauge(sourceCollectionID, snapshotName, activePins) setSnapshotActivePinsGauge(sourceCollectionID, snapshotName, activePins)
phase0Lock.Close() phase0Lock.Close()
@@ -691,7 +691,7 @@ func (sm *snapshotManager) RestoreSnapshot(
// Phase 1: Read snapshot data (now protected by the refcount guard) // Phase 1: Read snapshot data (now protected by the refcount guard)
snapshotData, err := sm.ReadSnapshotData(ctx, sourceCollectionID, snapshotName) snapshotData, err := sm.ReadSnapshotData(ctx, sourceCollectionID, snapshotName)
if err != nil { if err != nil {
return 0, fmt.Errorf("failed to read snapshot data: %w", err) return 0, merr.Wrap(err, "failed to read snapshot data")
} }
log.Info("snapshot data loaded", log.Info("snapshot data loaded",
zap.Int("segmentCount", len(snapshotData.Segments)), zap.Int("segmentCount", len(snapshotData.Segments)),
@@ -707,7 +707,7 @@ func (sm *snapshotManager) RestoreSnapshot(
// Phase 2: Restore collection and partitions // Phase 2: Restore collection and partitions
collectionID, err := sm.RestoreCollection(ctx, snapshotData, targetCollectionName, targetDbName) collectionID, err := sm.RestoreCollection(ctx, snapshotData, targetCollectionName, targetDbName)
if err != nil { if err != nil {
return 0, fmt.Errorf("failed to restore collection: %w", err) return 0, merr.Wrap(err, "failed to restore collection")
} }
log.Info("collection and partitions restored", zap.Int64("collectionID", collectionID)) log.Info("collection and partitions restored", zap.Int64("collectionID", collectionID))
@@ -718,7 +718,7 @@ func (sm *snapshotManager) RestoreSnapshot(
if rollbackErr := rollback(ctx, targetDbName, targetCollectionName); rollbackErr != nil { if rollbackErr := rollback(ctx, targetDbName, targetCollectionName); rollbackErr != nil {
log.Error("rollback failed", zap.Error(rollbackErr)) log.Error("rollback failed", zap.Error(rollbackErr))
} }
return 0, fmt.Errorf("failed to restore indexes: %w", err) return 0, merr.Wrap(err, "failed to restore indexes")
} }
log.Info("indexes restored", zap.Int("indexCount", len(snapshotData.Indexes))) log.Info("indexes restored", zap.Int("indexCount", len(snapshotData.Indexes)))
@@ -730,7 +730,7 @@ func (sm *snapshotManager) RestoreSnapshot(
if rollbackErr := rollback(ctx, targetDbName, targetCollectionName); rollbackErr != nil { if rollbackErr := rollback(ctx, targetDbName, targetCollectionName); rollbackErr != nil {
log.Error("rollback failed", zap.Error(rollbackErr)) log.Error("rollback failed", zap.Error(rollbackErr))
} }
return 0, fmt.Errorf("failed to allocate job ID: %w", err) return 0, merr.Wrap(err, "failed to allocate job ID")
} }
log.Info("pre-allocated job ID for restore", zap.Int64("jobID", jobID)) log.Info("pre-allocated job ID for restore", zap.Int64("jobID", jobID))
@@ -741,7 +741,7 @@ func (sm *snapshotManager) RestoreSnapshot(
if rollbackErr := rollback(ctx, targetDbName, targetCollectionName); rollbackErr != nil { if rollbackErr := rollback(ctx, targetDbName, targetCollectionName); rollbackErr != nil {
log.Error("rollback failed", zap.Error(rollbackErr)) log.Error("rollback failed", zap.Error(rollbackErr))
} }
return 0, fmt.Errorf("failed to start broadcaster for restore message: %w", err) return 0, merr.Wrap(err, "failed to start broadcaster for restore message")
} }
defer func() { defer func() {
if restoreBroadcaster != nil { if restoreBroadcaster != nil {
@@ -760,7 +760,7 @@ func (sm *snapshotManager) RestoreSnapshot(
if rollbackErr := rollback(ctx, targetDbName, targetCollectionName); rollbackErr != nil { if rollbackErr := rollback(ctx, targetDbName, targetCollectionName); rollbackErr != nil {
log.Error("rollback failed", zap.Error(rollbackErr)) log.Error("rollback failed", zap.Error(rollbackErr))
} }
err = fmt.Errorf("resource validation failed: %w", valErr) err = merr.Wrap(valErr, "resource validation failed")
return 0, err return 0, err
} }
@@ -785,7 +785,7 @@ func (sm *snapshotManager) RestoreSnapshot(
if rollbackErr := rollback(ctx, targetDbName, targetCollectionName); rollbackErr != nil { if rollbackErr := rollback(ctx, targetDbName, targetCollectionName); rollbackErr != nil {
log.Error("rollback failed", zap.Error(rollbackErr)) log.Error("rollback failed", zap.Error(rollbackErr))
} }
err = fmt.Errorf("failed to broadcast restore message: %w", bcErr) err = merr.Wrap(bcErr, "failed to broadcast restore message")
return 0, err return 0, err
} }
@@ -872,14 +872,14 @@ func (sm *snapshotManager) RestoreIndexes(
// Get collection info for dbId // Get collection info for dbId
coll, err := sm.broker.DescribeCollectionInternal(ctx, collectionID) coll, err := sm.broker.DescribeCollectionInternal(ctx, collectionID)
if err != nil { if err != nil {
return fmt.Errorf("failed to describe collection %d: %w", collectionID, err) return merr.Wrapf(err, "failed to describe collection %d", collectionID)
} }
for _, indexInfo := range snapshotData.Indexes { for _, indexInfo := range snapshotData.Indexes {
// Allocate new index ID // Allocate new index ID
indexID, err := sm.allocator.AllocID(ctx) indexID, err := sm.allocator.AllocID(ctx)
if err != nil { if err != nil {
return fmt.Errorf("failed to allocate index ID: %w", err) return merr.Wrap(err, "failed to allocate index ID")
} }
// Build index model from snapshot data // Build index model from snapshot data
@@ -898,19 +898,19 @@ func (sm *snapshotManager) RestoreIndexes(
// Validate the index params (basic validation without JSON path parsing) // Validate the index params (basic validation without JSON path parsing)
if err := ValidateIndexParams(index); err != nil { if err := ValidateIndexParams(index); err != nil {
return fmt.Errorf("failed to validate index %s: %w", indexInfo.GetIndexName(), err) return merr.Wrapf(err, "failed to validate index %s", indexInfo.GetIndexName())
} }
// Check scalar index engine version for JSON path indexes with new types // Check scalar index engine version for JSON path indexes with new types
if err := sm.checkJSONPathIndexVersion(index); err != nil { if err := sm.checkJSONPathIndexVersion(index); err != nil {
return fmt.Errorf("failed to validate index %s: %w", indexInfo.GetIndexName(), err) return merr.Wrapf(err, "failed to validate index %s", indexInfo.GetIndexName())
} }
// Create a new broadcaster for each index // Create a new broadcaster for each index
// (each broadcaster can only be used once due to resource key lock consumption) // (each broadcaster can only be used once due to resource key lock consumption)
b, err := startBroadcaster(ctx, collectionID, snapshotName) b, err := startBroadcaster(ctx, collectionID, snapshotName)
if err != nil { if err != nil {
return fmt.Errorf("failed to start broadcaster for index %s: %w", indexInfo.GetIndexName(), err) return merr.Wrapf(err, "failed to start broadcaster for index %s", indexInfo.GetIndexName())
} }
// Broadcast CreateIndex message directly to DDL WAL // Broadcast CreateIndex message directly to DDL WAL
@@ -930,7 +930,7 @@ func (sm *snapshotManager) RestoreIndexes(
) )
b.Close() b.Close()
if err != nil { if err != nil {
return fmt.Errorf("failed to broadcast create index %s: %w", indexInfo.GetIndexName(), err) return merr.Wrapf(err, "failed to broadcast create index %s", indexInfo.GetIndexName())
} }
log.Ctx(ctx).Info("index restored via DDL WAL broadcast", log.Ctx(ctx).Info("index restored via DDL WAL broadcast",
@@ -977,14 +977,14 @@ func (sm *snapshotManager) RestoreData(
snapshotData, err := sm.ReadSnapshotData(ctx, sourceCollectionID, snapshotName) snapshotData, err := sm.ReadSnapshotData(ctx, sourceCollectionID, snapshotName)
if err != nil { if err != nil {
log.Error("failed to read snapshot data", zap.Error(err)) log.Error("failed to read snapshot data", zap.Error(err))
return 0, fmt.Errorf("failed to read snapshot data: %w", err) return 0, merr.Wrap(err, "failed to read snapshot data")
} }
// ========== Phase 2: Build partition mapping ========== // ========== Phase 2: Build partition mapping ==========
partitionMapping, err := sm.buildPartitionMapping(ctx, snapshotData, collectionID) partitionMapping, err := sm.buildPartitionMapping(ctx, snapshotData, collectionID)
if err != nil { if err != nil {
log.Error("failed to build partition mapping", zap.Error(err)) log.Error("failed to build partition mapping", zap.Error(err))
return 0, fmt.Errorf("partition mapping failed: %w", err) return 0, merr.Wrap(err, "partition mapping failed")
} }
log.Info("partition mapping built", zap.Any("partitionMapping", partitionMapping)) log.Info("partition mapping built", zap.Any("partitionMapping", partitionMapping))
@@ -992,14 +992,14 @@ func (sm *snapshotManager) RestoreData(
channelMapping, err := sm.buildChannelMapping(ctx, snapshotData, collectionID) channelMapping, err := sm.buildChannelMapping(ctx, snapshotData, collectionID)
if err != nil { if err != nil {
log.Error("failed to build channel mapping", zap.Error(err)) log.Error("failed to build channel mapping", zap.Error(err))
return 0, fmt.Errorf("channel mapping failed: %w", err) return 0, merr.Wrap(err, "channel mapping failed")
} }
// ========== Phase 4: Create copy segment job ========== // ========== Phase 4: Create copy segment job ==========
// Use the pre-allocated jobID from the WAL message // Use the pre-allocated jobID from the WAL message
if err := sm.createRestoreJob(ctx, collectionID, channelMapping, partitionMapping, snapshotData, jobID, pinID); err != nil { if err := sm.createRestoreJob(ctx, collectionID, channelMapping, partitionMapping, snapshotData, jobID, pinID); err != nil {
log.Error("failed to create restore job", zap.Error(err)) log.Error("failed to create restore job", zap.Error(err))
return 0, fmt.Errorf("restore job creation failed: %w", err) return 0, merr.Wrap(err, "restore job creation failed")
} }
log.Info("restore data completed successfully", log.Info("restore data completed successfully",
@@ -1047,7 +1047,7 @@ func (sm *snapshotManager) restoreUserPartitions(
} }
if err := sm.broker.CreatePartition(ctx, req); err != nil { if err := sm.broker.CreatePartition(ctx, req); err != nil {
return fmt.Errorf("failed to create partition %s: %w", partitionName, err) return merr.Wrapf(err, "failed to create partition %s", partitionName)
} }
} }
@@ -1330,7 +1330,7 @@ func (sm *snapshotManager) GetRestoreState(ctx context.Context, jobID int64) (*d
// Get job // Get job
job := sm.copySegmentMeta.GetJob(ctx, jobID) job := sm.copySegmentMeta.GetJob(ctx, jobID)
if job == nil { if job == nil {
err := merr.WrapErrImportFailed(fmt.Sprintf("restore job not found: jobID=%d", jobID)) err := merr.WrapErrImportSysFailedMsg("restore job not found: jobID=%d", jobID)
log.Warn("restore job not found") log.Warn("restore job not found")
return nil, err return nil, err
} }
+8 -9
View File
@@ -572,7 +572,7 @@ func (sm *snapshotMeta) SaveSnapshot(ctx context.Context, snapshot *SnapshotData
if err := sm.catalog.SaveSnapshot(ctx, snapshot.SnapshotInfo); err != nil { if err := sm.catalog.SaveSnapshot(ctx, snapshot.SnapshotInfo); err != nil {
log.Error("failed to save pending snapshot to catalog", zap.Error(err)) log.Error("failed to save pending snapshot to catalog", zap.Error(err))
return fmt.Errorf("failed to save pending snapshot to catalog: %w", err) return merr.Wrap(err, "failed to save pending snapshot to catalog")
} }
log.Info("saved pending snapshot to catalog") log.Info("saved pending snapshot to catalog")
@@ -582,7 +582,7 @@ func (sm *snapshotMeta) SaveSnapshot(ctx context.Context, snapshot *SnapshotData
// S3 write failed, leave PENDING record for GC to clean up // S3 write failed, leave PENDING record for GC to clean up
log.Error("failed to save snapshot to S3, pending record left for GC cleanup", log.Error("failed to save snapshot to S3, pending record left for GC cleanup",
zap.Error(err)) zap.Error(err))
return fmt.Errorf("failed to save snapshot to S3: %w", err) return merr.Wrap(err, "failed to save snapshot to S3")
} }
snapshot.SnapshotInfo.S3Location = metadataFilePath snapshot.SnapshotInfo.S3Location = metadataFilePath
log.Info("saved snapshot data to S3", zap.String("s3Location", metadataFilePath)) log.Info("saved snapshot data to S3", zap.String("s3Location", metadataFilePath))
@@ -598,7 +598,7 @@ func (sm *snapshotMeta) SaveSnapshot(ctx context.Context, snapshot *SnapshotData
// GC will eventually cleanup via the PENDING marker. // GC will eventually cleanup via the PENDING marker.
log.Error("failed to update snapshot to committed state, pending record left for GC cleanup", log.Error("failed to update snapshot to committed state, pending record left for GC cleanup",
zap.Error(err)) zap.Error(err))
return fmt.Errorf("failed to update snapshot to committed state: %w", err) return merr.Wrap(err, "failed to update snapshot to committed state")
} }
// Catalog committed — safe to insert into in-memory cache and register protection. // Catalog committed — safe to insert into in-memory cache and register protection.
@@ -809,8 +809,7 @@ func (sm *snapshotMeta) DropSnapshotsByCollection(ctx context.Context, collectio
} }
if len(errs) > 0 { if len(errs) > 0 {
return dropped, fmt.Errorf("failed to drop %d/%d snapshots for collection %d: %w", return dropped, merr.WrapErrServiceInternalErr(merr.Combine(errs...), "failed to drop %d/%d snapshots for collection %d", len(errs), len(snapshotNames), collectionID)
len(errs), len(snapshotNames), collectionID, merr.Combine(errs...))
} }
return dropped, nil return dropped, nil
@@ -1006,7 +1005,7 @@ func (sm *snapshotMeta) GetPendingSnapshots(ctx context.Context, pendingTimeout
// Get all snapshots from catalog (etcd) // Get all snapshots from catalog (etcd)
snapshots, err := sm.catalog.ListSnapshots(ctx) snapshots, err := sm.catalog.ListSnapshots(ctx)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to list snapshots from catalog: %w", err) return nil, merr.Wrap(err, "failed to list snapshots from catalog")
} }
now := time.Now().UnixMilli() now := time.Now().UnixMilli()
@@ -1057,7 +1056,7 @@ func (sm *snapshotMeta) CleanupPendingSnapshot(ctx context.Context, snapshot *da
func (sm *snapshotMeta) GetDeletingSnapshots(ctx context.Context) ([]*datapb.SnapshotInfo, error) { func (sm *snapshotMeta) GetDeletingSnapshots(ctx context.Context) ([]*datapb.SnapshotInfo, error) {
snapshots, err := sm.catalog.ListSnapshots(ctx) snapshots, err := sm.catalog.ListSnapshots(ctx)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to list snapshots from catalog: %w", err) return nil, merr.Wrap(err, "failed to list snapshots from catalog")
} }
deletingSnapshots := make([]*datapb.SnapshotInfo, 0) deletingSnapshots := make([]*datapb.SnapshotInfo, 0)
@@ -1774,7 +1773,7 @@ func (sm *snapshotMeta) generatePinID(snapshot *datapb.SnapshotInfo) (int64, err
for i := 0; i < 3; i++ { for i := 0; i < 3; i++ {
var b [8]byte var b [8]byte
if _, err := cryptorand.Read(b[:]); err != nil { if _, err := cryptorand.Read(b[:]); err != nil {
return 0, fmt.Errorf("failed to generate random pin ID: %w", err) return 0, merr.WrapErrServiceInternalErr(err, "failed to generate random pin ID")
} }
id := int64(binary.BigEndian.Uint64(b[:]) >> 1) id := int64(binary.BigEndian.Uint64(b[:]) >> 1)
if id == 0 { if id == 0 {
@@ -1784,5 +1783,5 @@ func (sm *snapshotMeta) generatePinID(snapshot *datapb.SnapshotInfo) (int64, err
return id, nil return id, nil
} }
} }
return 0, fmt.Errorf("failed to generate unique pin ID after 3 attempts") return 0, merr.WrapErrServiceInternalMsg("failed to generate unique pin ID after 3 attempts")
} }
+5 -5
View File
@@ -182,7 +182,7 @@ func (stm *statsTaskMeta) UpdateVersion(taskID, nodeID int64) error {
t, ok := stm.tasks.Get(taskID) t, ok := stm.tasks.Get(taskID)
if !ok { if !ok {
return fmt.Errorf("task %d not found", taskID) return merr.WrapErrServiceInternalMsg("task %d not found", taskID)
} }
cloneT := proto.Clone(t).(*indexpb.StatsTask) cloneT := proto.Clone(t).(*indexpb.StatsTask)
@@ -212,7 +212,7 @@ func (stm *statsTaskMeta) UpdateTaskState(taskID int64, state indexpb.JobState,
t, ok := stm.tasks.Get(taskID) t, ok := stm.tasks.Get(taskID)
if !ok { if !ok {
return fmt.Errorf("task %d not found", taskID) return merr.WrapErrServiceInternalMsg("task %d not found", taskID)
} }
cloneT := proto.Clone(t).(*indexpb.StatsTask) cloneT := proto.Clone(t).(*indexpb.StatsTask)
@@ -239,7 +239,7 @@ func (stm *statsTaskMeta) UpdateBuildingTask(taskID int64) error {
t, ok := stm.tasks.Get(taskID) t, ok := stm.tasks.Get(taskID)
if !ok { if !ok {
return fmt.Errorf("task %d not found", taskID) return merr.WrapErrServiceInternalMsg("task %d not found", taskID)
} }
cloneT := proto.Clone(t).(*indexpb.StatsTask) cloneT := proto.Clone(t).(*indexpb.StatsTask)
@@ -267,7 +267,7 @@ func (stm *statsTaskMeta) FinishTask(taskID int64, result *workerpb.StatsResult)
t, ok := stm.tasks.Get(taskID) t, ok := stm.tasks.Get(taskID)
if !ok { if !ok {
return fmt.Errorf("task %d not found", taskID) return merr.WrapErrServiceInternalMsg("task %d not found", taskID)
} }
cloneT := proto.Clone(t).(*indexpb.StatsTask) cloneT := proto.Clone(t).(*indexpb.StatsTask)
@@ -358,7 +358,7 @@ func (stm *statsTaskMeta) MarkTaskCanRecycle(taskID int64) error {
t, ok := stm.tasks.Get(taskID) t, ok := stm.tasks.Get(taskID)
if !ok { if !ok {
return fmt.Errorf("task %d not found", taskID) return merr.WrapErrServiceInternalMsg("task %d not found", taskID)
} }
cloneT := proto.Clone(t).(*indexpb.StatsTask) cloneT := proto.Clone(t).(*indexpb.StatsTask)
+4 -5
View File
@@ -18,7 +18,6 @@ package datacoord
import ( import (
"context" "context"
"fmt"
"path" "path"
"time" "time"
@@ -241,7 +240,7 @@ func (it *indexBuildTask) prepareJobRequest(ctx context.Context, segment *Segmen
var err error var err error
params, err = Params.KnowhereConfig.UpdateIndexParams(GetIndexType(params), paramtable.BuildStage, params) params, err = Params.KnowhereConfig.UpdateIndexParams(GetIndexType(params), paramtable.BuildStage, params)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to update index build params: %w", err) return nil, merr.WrapErrServiceInternalErr(err, "failed to update index build params")
} }
} }
@@ -249,14 +248,14 @@ func (it *indexBuildTask) prepareJobRequest(ctx context.Context, segment *Segmen
var err error var err error
params, err = indexparams.UpdateDiskIndexBuildParams(Params, params) params, err = indexparams.UpdateDiskIndexBuildParams(Params, params)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to append index build params: %w", err) return nil, merr.WrapErrServiceInternalErr(err, "failed to append index build params")
} }
} }
// Get collection info and field // Get collection info and field
collectionInfo, err := it.handler.GetCollection(ctx, segment.GetCollectionID()) collectionInfo, err := it.handler.GetCollection(ctx, segment.GetCollectionID())
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to get collection info: %w", err) return nil, merr.Wrap(err, "failed to get collection info")
} }
schema := collectionInfo.Schema schema := collectionInfo.Schema
@@ -271,7 +270,7 @@ func (it *indexBuildTask) prepareJobRequest(ctx context.Context, segment *Segmen
} }
if field == nil { if field == nil {
return nil, fmt.Errorf("field not found with ID %d", fieldID) return nil, merr.WrapErrFieldNotFound(fieldID)
} }
// Extract dim only for vector types to avoid unnecessary warnings // Extract dim only for vector types to avoid unnecessary warnings
@@ -34,6 +34,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/proto/datapb" "github.com/milvus-io/milvus/pkg/v3/proto/datapb"
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb" "github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
"github.com/milvus-io/milvus/pkg/v3/taskcommon" "github.com/milvus-io/milvus/pkg/v3/taskcommon"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/paramtable" "github.com/milvus-io/milvus/pkg/v3/util/paramtable"
) )
@@ -115,7 +116,7 @@ func (t *refreshExternalCollectionTask) validateSource() error {
// Validate against job-level snapshot to isolate in-flight tasks from schema changes. // Validate against job-level snapshot to isolate in-flight tasks from schema changes.
job := t.refreshMeta.GetJob(t.GetJobId()) job := t.refreshMeta.GetJob(t.GetJobId())
if job == nil { if job == nil {
return fmt.Errorf("job %d not found", t.GetJobId()) return merr.WrapErrServiceInternalMsg("job %d not found", t.GetJobId())
} }
currentSource := job.GetExternalSource() currentSource := job.GetExternalSource()
@@ -125,7 +126,7 @@ func (t *refreshExternalCollectionTask) validateSource() error {
taskSpec := t.GetExternalSpec() taskSpec := t.GetExternalSpec()
if currentSource != taskSource || currentSpec != taskSpec { if currentSource != taskSource || currentSpec != taskSpec {
return fmt.Errorf( return merr.WrapErrServiceInternalMsg(
"task source mismatch: task source=%s/%s, job source=%s/%s (task belongs to a different refresh job)", "task source mismatch: task source=%s/%s, job source=%s/%s (task belongs to a different refresh job)",
taskSource, taskSpec, currentSource, currentSpec, taskSource, taskSpec, currentSource, currentSpec,
) )
@@ -215,7 +216,7 @@ func applyExternalCollectionSegmentUpdate(
logFields ...zap.Field, logFields ...zap.Field,
) error { ) error {
if mt == nil { if mt == nil {
return fmt.Errorf("meta is nil, cannot update segments") return merr.WrapErrServiceInternalMsg("meta is nil, cannot update segments")
} }
fields := append(logFields, zap.Int64("collectionID", collectionID)) fields := append(logFields, zap.Int64("collectionID", collectionID))
log := log.Ctx(ctx).With(fields...) log := log.Ctx(ctx).With(fields...)
@@ -228,14 +229,14 @@ func applyExternalCollectionSegmentUpdate(
for _, segID := range keptSegmentIDs { for _, segID := range keptSegmentIDs {
segment := mt.segments.GetSegment(segID) segment := mt.segments.GetSegment(segID)
if segment == nil { if segment == nil {
return fmt.Errorf("kept segment %d not found", segID) return merr.WrapErrServiceInternalMsg("kept segment %d not found", segID)
} }
if segment.GetCollectionID() != collectionID { if segment.GetCollectionID() != collectionID {
return fmt.Errorf("collection mismatch for kept segment %d: existing %d, want %d", return merr.WrapErrServiceInternalMsg("collection mismatch for kept segment %d: existing %d, want %d",
segID, segment.GetCollectionID(), collectionID) segID, segment.GetCollectionID(), collectionID)
} }
if segment.GetState() == commonpb.SegmentState_Dropped { if segment.GetState() == commonpb.SegmentState_Dropped {
return fmt.Errorf("cannot keep dropped segment %d", segID) return merr.WrapErrServiceInternalMsg("cannot keep dropped segment %d", segID)
} }
keptSegmentMap[segID] = true keptSegmentMap[segID] = true
} }
@@ -250,10 +251,10 @@ func applyExternalCollectionSegmentUpdate(
return err return err
} }
if keptSegmentMap[seg.GetID()] { if keptSegmentMap[seg.GetID()] {
return fmt.Errorf("segment %d cannot be both kept and updated", seg.GetID()) return merr.WrapErrServiceInternalMsg("segment %d cannot be both kept and updated", seg.GetID())
} }
if _, ok := upsertSegmentMap[seg.GetID()]; ok { if _, ok := upsertSegmentMap[seg.GetID()]; ok {
return fmt.Errorf("duplicate updated segment %d", seg.GetID()) return merr.WrapErrServiceInternalMsg("duplicate updated segment %d", seg.GetID())
} }
upsertSegmentMap[seg.GetID()] = seg upsertSegmentMap[seg.GetID()] = seg
validUpdatedSegments = append(validUpdatedSegments, seg) validUpdatedSegments = append(validUpdatedSegments, seg)
@@ -308,7 +309,7 @@ func applyExternalCollectionSegmentUpdate(
zap.Int("activeSegmentCount", activeSegmentCount), zap.Int("activeSegmentCount", activeSegmentCount),
zap.Int("keptSegments", len(keptSegmentMap)), zap.Int("keptSegments", len(keptSegmentMap)),
zap.Int("updatedSegments", len(upsertSegmentMap))) zap.Int("updatedSegments", len(upsertSegmentMap)))
return fmt.Errorf("safety check failed: refusing to drop all %d segments without replacement (keptSegments=%d, updatedSegments=%d)", return merr.WrapErrServiceInternalMsg("safety check failed: refusing to drop all %d segments without replacement (keptSegments=%d, updatedSegments=%d)",
activeSegmentCount, len(keptSegmentMap), len(upsertSegmentMap)) activeSegmentCount, len(keptSegmentMap), len(upsertSegmentMap))
} }
@@ -330,15 +331,15 @@ func applyExternalCollectionSegmentUpdate(
collInfo := mt.GetCollection(collectionID) collInfo := mt.GetCollection(collectionID)
if collInfo == nil { if collInfo == nil {
return fmt.Errorf("collection %d not found in meta", collectionID) return merr.WrapErrServiceInternalMsg("collection %d not found in meta", collectionID)
} }
// External collections are single-shard, single-partition (enforced at creation). // External collections are single-shard, single-partition (enforced at creation).
// Assert exactly-one here to catch any invariant violation from data corruption or legacy data. // Assert exactly-one here to catch any invariant violation from data corruption or legacy data.
if len(collInfo.VChannelNames) != 1 { if len(collInfo.VChannelNames) != 1 {
return fmt.Errorf("external collection %d expected exactly 1 VChannel, got %d", collectionID, len(collInfo.VChannelNames)) return merr.WrapErrServiceInternalMsg("external collection %d expected exactly 1 VChannel, got %d", collectionID, len(collInfo.VChannelNames))
} }
if len(collInfo.Partitions) != 1 { if len(collInfo.Partitions) != 1 {
return fmt.Errorf("external collection %d expected exactly 1 partition, got %d", collectionID, len(collInfo.Partitions)) return merr.WrapErrServiceInternalMsg("external collection %d expected exactly 1 partition, got %d", collectionID, len(collInfo.Partitions))
} }
insertChannel := collInfo.VChannelNames[0] insertChannel := collInfo.VChannelNames[0]
partitionID := collInfo.Partitions[0] partitionID := collInfo.Partitions[0]
@@ -477,14 +478,14 @@ func applyExternalCollectionSegmentUpdate(
func validateExternalRefreshUpdatedSegment(incoming *datapb.SegmentInfo, collectionID int64) error { func validateExternalRefreshUpdatedSegment(incoming *datapb.SegmentInfo, collectionID int64) error {
if incoming.GetCollectionID() != 0 && incoming.GetCollectionID() != collectionID { if incoming.GetCollectionID() != 0 && incoming.GetCollectionID() != collectionID {
return fmt.Errorf("collection mismatch for segment %d: got %d, want %d", return merr.WrapErrServiceInternalMsg("collection mismatch for segment %d: got %d, want %d",
incoming.GetID(), incoming.GetCollectionID(), collectionID) incoming.GetID(), incoming.GetCollectionID(), collectionID)
} }
if incoming.GetManifestPath() == "" { if incoming.GetManifestPath() == "" {
return fmt.Errorf("updated segment %d has empty manifest path", incoming.GetID()) return merr.WrapErrServiceInternalMsg("updated segment %d has empty manifest path", incoming.GetID())
} }
if len(incoming.GetBinlogs()) == 0 { if len(incoming.GetBinlogs()) == 0 {
return fmt.Errorf("updated segment %d has empty fake binlogs", incoming.GetID()) return merr.WrapErrServiceInternalMsg("updated segment %d has empty fake binlogs", incoming.GetID())
} }
return nil return nil
} }
@@ -513,36 +514,36 @@ func validateExternalRefreshNewSegment(incoming *datapb.SegmentInfo) error {
func validateExternalRefreshPatch(oldSeg *SegmentInfo, incoming *datapb.SegmentInfo, collectionID int64) error { func validateExternalRefreshPatch(oldSeg *SegmentInfo, incoming *datapb.SegmentInfo, collectionID int64) error {
if oldSeg == nil { if oldSeg == nil {
return fmt.Errorf("existing segment is nil") return merr.WrapErrServiceInternalMsg("existing segment is nil")
} }
if oldSeg.GetCollectionID() != collectionID { if oldSeg.GetCollectionID() != collectionID {
return fmt.Errorf("collection mismatch for segment %d: existing %d, want %d", return merr.WrapErrServiceInternalMsg("collection mismatch for segment %d: existing %d, want %d",
oldSeg.GetID(), oldSeg.GetCollectionID(), collectionID) oldSeg.GetID(), oldSeg.GetCollectionID(), collectionID)
} }
if oldSeg.GetState() == commonpb.SegmentState_Dropped { if oldSeg.GetState() == commonpb.SegmentState_Dropped {
return fmt.Errorf("cannot patch dropped segment %d", oldSeg.GetID()) return merr.WrapErrServiceInternalMsg("cannot patch dropped segment %d", oldSeg.GetID())
} }
if incoming.GetCollectionID() != 0 && incoming.GetCollectionID() != collectionID { if incoming.GetCollectionID() != 0 && incoming.GetCollectionID() != collectionID {
return fmt.Errorf("collection mismatch for segment %d: got %d, want %d", return merr.WrapErrServiceInternalMsg("collection mismatch for segment %d: got %d, want %d",
incoming.GetID(), incoming.GetCollectionID(), collectionID) incoming.GetID(), incoming.GetCollectionID(), collectionID)
} }
if incoming.GetNumOfRows() != oldSeg.GetNumOfRows() { if incoming.GetNumOfRows() != oldSeg.GetNumOfRows() {
return fmt.Errorf("row count changed for segment %d: got %d, want %d", return merr.WrapErrServiceInternalMsg("row count changed for segment %d: got %d, want %d",
incoming.GetID(), incoming.GetNumOfRows(), oldSeg.GetNumOfRows()) incoming.GetID(), incoming.GetNumOfRows(), oldSeg.GetNumOfRows())
} }
if incoming.GetStorageVersion() != 0 && incoming.GetStorageVersion() != oldSeg.GetStorageVersion() { if incoming.GetStorageVersion() != 0 && incoming.GetStorageVersion() != oldSeg.GetStorageVersion() {
return fmt.Errorf("storage version changed for segment %d: got %d, want %d", return merr.WrapErrServiceInternalMsg("storage version changed for segment %d: got %d, want %d",
incoming.GetID(), incoming.GetStorageVersion(), oldSeg.GetStorageVersion()) incoming.GetID(), incoming.GetStorageVersion(), oldSeg.GetStorageVersion())
} }
if incoming.GetSchemaVersion() < oldSeg.GetSchemaVersion() { if incoming.GetSchemaVersion() < oldSeg.GetSchemaVersion() {
return fmt.Errorf("schema version rollback for segment %d: got %d, want >= %d", return merr.WrapErrServiceInternalMsg("schema version rollback for segment %d: got %d, want >= %d",
incoming.GetID(), incoming.GetSchemaVersion(), oldSeg.GetSchemaVersion()) incoming.GetID(), incoming.GetSchemaVersion(), oldSeg.GetSchemaVersion())
} }
if incoming.GetManifestPath() == "" { if incoming.GetManifestPath() == "" {
return fmt.Errorf("patched segment %d has empty manifest path", incoming.GetID()) return merr.WrapErrServiceInternalMsg("patched segment %d has empty manifest path", incoming.GetID())
} }
if len(incoming.GetBinlogs()) == 0 { if len(incoming.GetBinlogs()) == 0 {
return fmt.Errorf("patched segment %d has empty fake binlogs", incoming.GetID()) return merr.WrapErrServiceInternalMsg("patched segment %d has empty fake binlogs", incoming.GetID())
} }
if err := validateExternalRefreshBinlogRowCount(incoming, oldSeg.GetNumOfRows()); err != nil { if err := validateExternalRefreshBinlogRowCount(incoming, oldSeg.GetNumOfRows()); err != nil {
return err return err
@@ -553,14 +554,14 @@ func validateExternalRefreshPatch(oldSeg *SegmentInfo, incoming *datapb.SegmentI
func validateExternalRefreshBinlogRowCount(segment *datapb.SegmentInfo, expectedRows int64) error { func validateExternalRefreshBinlogRowCount(segment *datapb.SegmentInfo, expectedRows int64) error {
binlogRows := segmentutil.CalcRowCountFromBinLog(segment) binlogRows := segmentutil.CalcRowCountFromBinLog(segment)
if binlogRows == -1 { if binlogRows == -1 {
return fmt.Errorf("invalid binlog row count for segment %d", segment.GetID()) return merr.WrapErrServiceInternalMsg("invalid binlog row count for segment %d", segment.GetID())
} }
if expectedRows > 0 && binlogRows != expectedRows { if expectedRows > 0 && binlogRows != expectedRows {
return fmt.Errorf("binlog row count mismatch for segment %d: got %d, want %d", return merr.WrapErrServiceInternalMsg("binlog row count mismatch for segment %d: got %d, want %d",
segment.GetID(), binlogRows, expectedRows) segment.GetID(), binlogRows, expectedRows)
} }
if binlogRows > 0 && binlogRows != segment.GetNumOfRows() { if binlogRows > 0 && binlogRows != segment.GetNumOfRows() {
return fmt.Errorf("binlog row count mismatch for segment %d: got %d, segment rows %d", return merr.WrapErrServiceInternalMsg("binlog row count mismatch for segment %d: got %d, segment rows %d",
segment.GetID(), binlogRows, segment.GetNumOfRows()) segment.GetID(), binlogRows, segment.GetNumOfRows())
} }
return nil return nil
@@ -612,7 +613,7 @@ func (t *refreshExternalCollectionTask) CreateTaskOnWorker(nodeID int64, cluster
log.Info("creating refresh task on worker") log.Info("creating refresh task on worker")
if t.mt == nil { if t.mt == nil {
err = fmt.Errorf("meta is nil, cannot create task on worker") err = merr.WrapErrServiceInternalMsg("meta is nil, cannot create task on worker")
return return
} }
@@ -625,7 +626,7 @@ func (t *refreshExternalCollectionTask) CreateTaskOnWorker(nodeID int64, cluster
// Re-read task from meta to sync in-memory state (nodeID and version) // Re-read task from meta to sync in-memory state (nodeID and version)
updatedTask := t.refreshMeta.GetTask(t.GetTaskId()) updatedTask := t.refreshMeta.GetTask(t.GetTaskId())
if updatedTask == nil { if updatedTask == nil {
err = fmt.Errorf("task %d not found after version update", t.GetTaskId()) err = merr.WrapErrServiceInternalMsg("task %d not found after version update", t.GetTaskId())
return return
} }
t.ExternalCollectionRefreshTask = updatedTask t.ExternalCollectionRefreshTask = updatedTask
@@ -668,11 +669,11 @@ func (t *refreshExternalCollectionTask) CreateTaskOnWorker(nodeID int64, cluster
// lock, before they are supported. // lock, before they are supported.
collInfo := t.mt.GetCollection(t.GetCollectionId()) collInfo := t.mt.GetCollection(t.GetCollectionId())
if collInfo == nil { if collInfo == nil {
err = fmt.Errorf("collection %d not found in meta", t.GetCollectionId()) err = merr.WrapErrServiceInternalMsg("collection %d not found in meta", t.GetCollectionId())
return return
} }
if len(collInfo.Partitions) != 1 { if len(collInfo.Partitions) != 1 {
err = fmt.Errorf("external collection %d expected exactly 1 partition, got %d", t.GetCollectionId(), len(collInfo.Partitions)) err = merr.WrapErrServiceInternalMsg("external collection %d expected exactly 1 partition, got %d", t.GetCollectionId(), len(collInfo.Partitions))
return return
} }
partitionID := collInfo.Partitions[0] partitionID := collInfo.Partitions[0]
+10 -5
View File
@@ -18,7 +18,6 @@ package datacoord
import ( import (
"context" "context"
"fmt"
"time" "time"
"github.com/cockroachdb/errors" "github.com/cockroachdb/errors"
@@ -299,11 +298,17 @@ func (st *statsTask) handleEmptySegment(ctx context.Context) error {
// Prepare the stats request // Prepare the stats request
func (st *statsTask) prepareJobRequest(ctx context.Context, segment *SegmentInfo) (*workerpb.CreateStatsRequest, error) { func (st *statsTask) prepareJobRequest(ctx context.Context, segment *SegmentInfo) (*workerpb.CreateStatsRequest, error) {
collInfo, err := st.handler.GetCollection(ctx, segment.GetCollectionID()) collInfo, err := st.handler.GetCollection(ctx, segment.GetCollectionID())
if err != nil || collInfo == nil { if err != nil {
return nil, fmt.Errorf("failed to get collection info: %w", err) return nil, merr.Wrap(err, "failed to get collection info")
}
// GetCollection can return (nil, nil) on a cache miss; merr.Wrap(nil) would
// be nil and silently submit a malformed request, so guard collInfo
// separately with a typed not-found.
if collInfo == nil {
return nil, merr.WrapErrCollectionNotFound(segment.GetCollectionID())
} }
if collInfo.Schema == nil || len(collInfo.Schema.GetFields()) == 0 { if collInfo.Schema == nil || len(collInfo.Schema.GetFields()) == 0 {
return nil, fmt.Errorf("collection schema is nil or has no fields, collectionID: %d", segment.GetCollectionID()) return nil, merr.WrapErrServiceInternalMsg("collection schema is nil or has no fields, collectionID: %d", segment.GetCollectionID())
} }
// Calculate binlog allocation // Calculate binlog allocation
@@ -314,7 +319,7 @@ func (st *statsTask) prepareJobRequest(ctx context.Context, segment *SegmentInfo
// Allocate IDs // Allocate IDs
start, end, err := st.allocator.AllocN(binlogNum + int64(len(collInfo.Schema.GetFunctions())) + 1) start, end, err := st.allocator.AllocN(binlogNum + int64(len(collInfo.Schema.GetFunctions())) + 1)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to allocate log IDs: %w", err) return nil, merr.Wrap(err, "failed to allocate log IDs")
} }
// Create the request // Create the request
@@ -26,7 +26,6 @@ import (
"github.com/apache/arrow/go/v17/arrow" "github.com/apache/arrow/go/v17/arrow"
"github.com/apache/arrow/go/v17/arrow/array" "github.com/apache/arrow/go/v17/arrow/array"
"github.com/cockroachdb/errors"
"go.opentelemetry.io/otel" "go.opentelemetry.io/otel"
"go.uber.org/zap" "go.uber.org/zap"
@@ -113,14 +112,16 @@ func (t *bumpSchemaVersionCompactionTask) preCompact() error {
return t.ctx.Err() return t.ctx.Err()
} }
// Check segment binlogs: must have exactly one segment // Check segment binlogs: must have exactly one segment.
// The plan is produced by datacoord, so a malformed plan is an internal
// protocol violation, not user input.
if len(t.plan.GetSegmentBinlogs()) != 1 { if len(t.plan.GetSegmentBinlogs()) != 1 {
return errors.Newf("schema bump compaction plan is illegal, must have exactly one segment, but got %d segments, planID = %d", len(t.plan.GetSegmentBinlogs()), t.GetPlanID()) return merr.WrapErrServiceInternalMsg("schema bump compaction plan is illegal, must have exactly one segment, but got %d segments, planID = %d", len(t.plan.GetSegmentBinlogs()), t.GetPlanID())
} }
segment := t.plan.GetSegmentBinlogs()[0] segment := t.plan.GetSegmentBinlogs()[0]
if segment.GetStorageVersion() < storage.StorageV3 || segment.GetManifest() == "" { if segment.GetStorageVersion() < storage.StorageV3 || segment.GetManifest() == "" {
return errors.Newf("schema bump compaction requires a StorageV3 segment with manifest, planID = %d, segmentID = %d, storageVersion = %d", t.GetPlanID(), segment.GetSegmentID(), segment.GetStorageVersion()) return merr.WrapErrServiceInternalMsg("schema bump compaction requires a StorageV3 segment with manifest, planID = %d, segmentID = %d, storageVersion = %d", t.GetPlanID(), segment.GetSegmentID(), segment.GetStorageVersion())
} }
return nil return nil
@@ -147,7 +148,7 @@ func (t *bumpSchemaVersionCompactionTask) missingFunctionInputSchema(missingFunc
for _, inputFieldID := range functionSchema.GetInputFieldIds() { for _, inputFieldID := range functionSchema.GetInputFieldIds() {
inputField := typeutil.GetField(schema, inputFieldID) inputField := typeutil.GetField(schema, inputFieldID)
if inputField == nil { if inputField == nil {
return nil, nil, errors.New("input field not found in schema") return nil, nil, merr.WrapErrParameterInvalidMsg("input field not found in schema")
} }
if err := validateMaterializationInputField(functionSchema, inputField); err != nil { if err := validateMaterializationInputField(functionSchema, inputField); err != nil {
return nil, nil, err return nil, nil, err
@@ -196,11 +197,11 @@ func bm25AdditionalInputFields(schema *schemapb.CollectionSchema, inputField *sc
return nil, err return nil, err
} }
if multiAnalyzerParams.ByField == "" { if multiAnalyzerParams.ByField == "" {
return nil, errors.New("multi_analyzer_params missing required 'by_field' key") return nil, merr.WrapErrParameterInvalidMsg("multi_analyzer_params missing required 'by_field' key")
} }
byField := typeutil.GetFieldByName(schema, multiAnalyzerParams.ByField) byField := typeutil.GetFieldByName(schema, multiAnalyzerParams.ByField)
if byField == nil { if byField == nil {
return nil, errors.New("input field not found in schema") return nil, merr.WrapErrParameterInvalidMsg("input field not found in schema")
} }
return []*schemapb.FieldSchema{byField}, nil return []*schemapb.FieldSchema{byField}, nil
} }
@@ -217,7 +218,7 @@ func (t *bumpSchemaVersionCompactionTask) missingFunctionOutputFields(missingFun
outputFieldID := functionSchema.GetOutputFieldIds()[outputIndex] outputFieldID := functionSchema.GetOutputFieldIds()[outputIndex]
outputField := typeutil.GetField(schema, outputFieldID) outputField := typeutil.GetField(schema, outputFieldID)
if outputField == nil { if outputField == nil {
return nil, nil, errors.New("output field not found in schema") return nil, nil, merr.WrapErrParameterInvalidMsg("output field not found in schema")
} }
if err := validateMaterializationOutputField(functionSchema, outputField); err != nil { if err := validateMaterializationOutputField(functionSchema, outputField); err != nil {
return nil, nil, err return nil, nil, err
@@ -227,7 +228,7 @@ func (t *bumpSchemaVersionCompactionTask) missingFunctionOutputFields(missingFun
} }
} }
if len(fields) == 0 { if len(fields) == 0 {
return nil, nil, errors.New("no missing function output fields") return nil, nil, merr.WrapErrParameterInvalidMsg("no missing function output fields")
} }
return fields, fieldIDs, nil return fields, fieldIDs, nil
} }
@@ -236,22 +237,22 @@ func validateSupportedMissingFunctionMaterialization(functionSchema *schemapb.Fu
switch functionSchema.GetType() { switch functionSchema.GetType() {
case schemapb.FunctionType_BM25: case schemapb.FunctionType_BM25:
if len(functionSchema.GetInputFieldIds()) == 0 { if len(functionSchema.GetInputFieldIds()) == 0 {
return errors.New("bm25 function should have input fields") return merr.WrapErrParameterInvalidMsg("bm25 function should have input fields")
} }
if len(functionSchema.GetOutputFieldIds()) == 0 { if len(functionSchema.GetOutputFieldIds()) == 0 {
return errors.New("bm25 function should have output fields") return merr.WrapErrParameterInvalidMsg("bm25 function should have output fields")
} }
return nil return nil
case schemapb.FunctionType_MinHash: case schemapb.FunctionType_MinHash:
if len(functionSchema.GetInputFieldIds()) == 0 { if len(functionSchema.GetInputFieldIds()) == 0 {
return errors.New("minhash function should have input fields") return merr.WrapErrParameterInvalidMsg("minhash function should have input fields")
} }
if len(functionSchema.GetOutputFieldIds()) == 0 { if len(functionSchema.GetOutputFieldIds()) == 0 {
return errors.New("minhash function should have output fields") return merr.WrapErrParameterInvalidMsg("minhash function should have output fields")
} }
return nil return nil
default: default:
return errors.New("unsupported function type") return merr.WrapErrParameterInvalidMsg("unsupported function type")
} }
} }
@@ -259,11 +260,11 @@ func validateMaterializationInputField(functionSchema *schemapb.FunctionSchema,
switch functionSchema.GetType() { switch functionSchema.GetType() {
case schemapb.FunctionType_BM25: case schemapb.FunctionType_BM25:
if field.GetDataType() != schemapb.DataType_VarChar && field.GetDataType() != schemapb.DataType_Text { if field.GetDataType() != schemapb.DataType_VarChar && field.GetDataType() != schemapb.DataType_Text {
return errors.New("input field data type must be varchar or text for bm25 materialization") return merr.WrapErrParameterInvalidMsg("input field data type must be varchar or text for bm25 materialization")
} }
case schemapb.FunctionType_MinHash: case schemapb.FunctionType_MinHash:
if field.GetDataType() != schemapb.DataType_VarChar && field.GetDataType() != schemapb.DataType_Text { if field.GetDataType() != schemapb.DataType_VarChar && field.GetDataType() != schemapb.DataType_Text {
return errors.New("input field data type must be varchar or text for minhash materialization") return merr.WrapErrParameterInvalidMsg("input field data type must be varchar or text for minhash materialization")
} }
} }
return nil return nil
@@ -273,11 +274,11 @@ func validateMaterializationOutputField(functionSchema *schemapb.FunctionSchema,
switch functionSchema.GetType() { switch functionSchema.GetType() {
case schemapb.FunctionType_BM25: case schemapb.FunctionType_BM25:
if field.GetDataType() != schemapb.DataType_SparseFloatVector { if field.GetDataType() != schemapb.DataType_SparseFloatVector {
return errors.New("output field data type must be sparse float vector for bm25 materialization") return merr.WrapErrParameterInvalidMsg("output field data type must be sparse float vector for bm25 materialization")
} }
case schemapb.FunctionType_MinHash: case schemapb.FunctionType_MinHash:
if field.GetDataType() != schemapb.DataType_BinaryVector { if field.GetDataType() != schemapb.DataType_BinaryVector {
return errors.New("output field data type must be binary vector for minhash materialization") return merr.WrapErrParameterInvalidMsg("output field data type must be binary vector for minhash materialization")
} }
} }
return nil return nil
@@ -546,7 +547,7 @@ func (t *bumpSchemaVersionCompactionTask) runFullSchemaRewrite(existingFields ma
} }
manifestPath, err = packed.AddStatsToManifest(manifestPath, t.compactionParams.StorageConfig, statEntries) manifestPath, err = packed.AddStatsToManifest(manifestPath, t.compactionParams.StorageConfig, statEntries)
if err != nil { if err != nil {
return nil, merr.WrapErrServiceInternal("failed to add writer stats to schema bump full rewrite manifest", err.Error()) return nil, merr.Wrap(err, "failed to add writer stats to schema bump full rewrite manifest")
} }
} }
sortedInsertLogs := storage.SortFieldBinlogs(insertLogs) sortedInsertLogs := storage.SortFieldBinlogs(insertLogs)
@@ -602,7 +603,7 @@ func (t *bumpSchemaVersionCompactionTask) runFullSchemaRewrite(existingFields ma
func appendBM25StatsFromArrowArray(stats *storage.BM25Stats, arr arrow.Array) (int, error) { func appendBM25StatsFromArrowArray(stats *storage.BM25Stats, arr arrow.Array) (int, error) {
binaryArray, ok := arr.(*array.Binary) binaryArray, ok := arr.(*array.Binary)
if !ok { if !ok {
return 0, errors.Newf("bm25 output field must be arrow binary array, got %T", arr) return 0, merr.WrapErrParameterInvalidMsg("bm25 output field must be arrow binary array, got %T", arr)
} }
memorySize := 0 memorySize := 0
for i := 0; i < binaryArray.Len(); i++ { for i := 0; i < binaryArray.Len(); i++ {
@@ -667,7 +668,7 @@ func (t *bumpSchemaVersionCompactionTask) setupWriter(outputFields []*schemapb.F
basePath, existingVersion, err := packed.UnmarshalManifestPath(segment.GetManifest()) basePath, existingVersion, err := packed.UnmarshalManifestPath(segment.GetManifest())
if err != nil { if err != nil {
return nil, merr.WrapErrServiceInternal("failed to parse existing manifest for schema bump", err.Error()) return nil, merr.WrapErrDataIntegrityMsg("failed to parse existing manifest for schema bump: %s", err.Error())
} }
writerResult, err := t.newV3WriterResult(outputSchema, newColumnGroups, segment, collectionID, basePath, existingVersion) writerResult, err := t.newV3WriterResult(outputSchema, newColumnGroups, segment, collectionID, basePath, existingVersion)
if err != nil { if err != nil {
@@ -685,7 +686,7 @@ func (t *bumpSchemaVersionCompactionTask) setupWriter(outputFields []*schemapb.F
func (t *bumpSchemaVersionCompactionTask) newV3WriterResult(schema *schemapb.CollectionSchema, columnGroups []storagecommon.ColumnGroup, segment *datapb.CompactionSegmentBinlogs, collectionID int64, basePath string, baseVersion int64) (*bumpSchemaVersionWriterResult, error) { func (t *bumpSchemaVersionCompactionTask) newV3WriterResult(schema *schemapb.CollectionSchema, columnGroups []storagecommon.ColumnGroup, segment *datapb.CompactionSegmentBinlogs, collectionID int64, basePath string, baseVersion int64) (*bumpSchemaVersionWriterResult, error) {
if segment.GetStorageVersion() < storage.StorageV3 || segment.GetManifest() == "" { if segment.GetStorageVersion() < storage.StorageV3 || segment.GetManifest() == "" {
return nil, errors.New("schema bump compaction requires a StorageV3 segment with manifest") return nil, merr.WrapErrParameterInvalidMsg("schema bump compaction requires a StorageV3 segment with manifest")
} }
pluginContext, err := hookutil.GetCPluginContext(t.plan.GetPluginContext(), collectionID) pluginContext, err := hookutil.GetCPluginContext(t.plan.GetPluginContext(), collectionID)
@@ -732,7 +733,7 @@ func (t *bumpSchemaVersionCompactionTask) addV3Stats(prefix string, fieldID int6
statsRelPath := fmt.Sprintf("_stats/%s.%d/%d", prefix, fieldID, statsID) statsRelPath := fmt.Sprintf("_stats/%s.%d/%d", prefix, fieldID, statsID)
absStatsPath := path.Join(writerResult.basePath, statsRelPath) absStatsPath := path.Join(writerResult.basePath, statsRelPath)
if err := packed.WriteFile(t.compactionParams.StorageConfig, absStatsPath, bytes); err != nil { if err := packed.WriteFile(t.compactionParams.StorageConfig, absStatsPath, bytes); err != nil {
return merr.WrapErrServiceInternal("failed to write V3 stats", err.Error()) return merr.Wrap(err, "failed to write V3 stats")
} }
writerResult.v3Stats = append(writerResult.v3Stats, packed.FieldBinlogStatEntry(prefix, fieldID, &datapb.FieldBinlog{ writerResult.v3Stats = append(writerResult.v3Stats, packed.FieldBinlogStatEntry(prefix, fieldID, &datapb.FieldBinlog{
FieldID: fieldID, FieldID: fieldID,
@@ -773,7 +774,7 @@ func (t *bumpSchemaVersionCompactionTask) buildMergedLogsV3(segment *datapb.Comp
func (t *bumpSchemaVersionCompactionTask) preserveDeltaLogsV3(segment *datapb.CompactionSegmentBinlogs, manifestPath string) (string, error) { func (t *bumpSchemaVersionCompactionTask) preserveDeltaLogsV3(segment *datapb.CompactionSegmentBinlogs, manifestPath string) (string, error) {
deltaPaths, err := packed.GetDeltaLogPathsFromManifest(segment.GetManifest(), t.compactionParams.StorageConfig) deltaPaths, err := packed.GetDeltaLogPathsFromManifest(segment.GetManifest(), t.compactionParams.StorageConfig)
if err != nil { if err != nil {
return "", merr.WrapErrServiceInternal("failed to read V3 delta logs from existing manifest", err.Error()) return "", merr.Wrap(err, "failed to read V3 delta logs from existing manifest")
} }
if len(deltaPaths) == 0 { if len(deltaPaths) == 0 {
return manifestPath, nil return manifestPath, nil
@@ -784,12 +785,12 @@ func (t *bumpSchemaVersionCompactionTask) preserveDeltaLogsV3(segment *datapb.Co
deltaSummaries = append(deltaSummaries, fieldBinlog.GetBinlogs()...) deltaSummaries = append(deltaSummaries, fieldBinlog.GetBinlogs()...)
} }
if len(deltaSummaries) != len(deltaPaths) { if len(deltaSummaries) != len(deltaPaths) {
return "", merr.WrapErrServiceInternal(fmt.Sprintf("V3 delta manifest path count %d does not match segment delta summary count %d", len(deltaPaths), len(deltaSummaries))) return "", merr.WrapErrServiceInternalMsg("V3 delta manifest path count %d does not match segment delta summary count %d", len(deltaPaths), len(deltaSummaries))
} }
basePath, _, err := packed.UnmarshalManifestPath(manifestPath) basePath, _, err := packed.UnmarshalManifestPath(manifestPath)
if err != nil { if err != nil {
return "", merr.WrapErrServiceInternal("failed to parse new V3 manifest for delta preservation", err.Error()) return "", merr.WrapErrDataIntegrityMsg("failed to parse new V3 manifest for delta preservation: %s", err.Error())
} }
deltaEntries := make([]packed.DeltaLogEntry, 0, len(deltaPaths)) deltaEntries := make([]packed.DeltaLogEntry, 0, len(deltaPaths))
@@ -797,7 +798,7 @@ func (t *bumpSchemaVersionCompactionTask) preserveDeltaLogsV3(segment *datapb.Co
newDeltaPath := metautil.BuildDeltaLogPathV3(basePath, deltaSummaries[i].GetLogID()) newDeltaPath := metautil.BuildDeltaLogPathV3(basePath, deltaSummaries[i].GetLogID())
if newDeltaPath != deltaPath { if newDeltaPath != deltaPath {
if err := t.chunkManager.Copy(t.ctx, deltaPath, newDeltaPath); err != nil { if err := t.chunkManager.Copy(t.ctx, deltaPath, newDeltaPath); err != nil {
return "", merr.WrapErrServiceInternal("failed to copy V3 delta log for schema bump full rewrite", err.Error()) return "", merr.Wrap(err, "failed to copy V3 delta log for schema bump full rewrite")
} }
} }
deltaEntries = append(deltaEntries, packed.DeltaLogEntry{ deltaEntries = append(deltaEntries, packed.DeltaLogEntry{
@@ -807,7 +808,7 @@ func (t *bumpSchemaVersionCompactionTask) preserveDeltaLogsV3(segment *datapb.Co
} }
newManifest, err := packed.AddDeltaLogsToManifest(manifestPath, t.compactionParams.StorageConfig, deltaEntries) newManifest, err := packed.AddDeltaLogsToManifest(manifestPath, t.compactionParams.StorageConfig, deltaEntries)
if err != nil { if err != nil {
return "", merr.WrapErrServiceInternal("failed to preserve V3 delta logs in full rewrite manifest", err.Error()) return "", merr.Wrap(err, "failed to preserve V3 delta logs in full rewrite manifest")
} }
return newManifest, nil return newManifest, nil
} }
@@ -923,7 +924,7 @@ func (t *bumpSchemaVersionCompactionTask) runMissingFunctionMaterialization(ctx
if outputCol == nil { if outputCol == nil {
releaseWrappedRecord(wrapped, record) releaseWrappedRecord(wrapped, record)
span.End() span.End()
return nil, merr.WrapErrServiceInternal(fmt.Sprintf("output field %d not found in materialized record", outputFieldID)) return nil, merr.WrapErrServiceInternalMsg("output field %d not found in materialized record", outputFieldID)
} }
batchMemSize := arrowArrayMemorySize(outputCol) batchMemSize := arrowArrayMemorySize(outputCol)
if stats, ok := statsByField[outputFieldID]; ok { if stats, ok := statsByField[outputFieldID]; ok {
@@ -997,7 +998,7 @@ func (t *bumpSchemaVersionCompactionTask) runMissingFunctionMaterialization(ctx
}, },
) )
if err != nil { if err != nil {
return nil, merr.WrapErrServiceInternal("failed to commit schema bump V3 manifest", err.Error()) return nil, merr.Wrap(err, "failed to commit schema bump V3 manifest")
} }
log.Info("[schema-bump-partial-writer] writer output and bm25 stats committed", log.Info("[schema-bump-partial-writer] writer output and bm25 stats committed",
zap.String("manifestPath", manifestPath), zap.String("manifestPath", manifestPath),
@@ -28,7 +28,6 @@ import (
"sync" "sync"
"time" "time"
"github.com/cockroachdb/errors"
"github.com/samber/lo" "github.com/samber/lo"
"go.opentelemetry.io/otel" "go.opentelemetry.io/otel"
"go.uber.org/atomic" "go.uber.org/atomic"
@@ -663,7 +662,7 @@ func (t *clusteringCompactionTask) mappingSegment(
row, ok := v.Value.(map[typeutil.UniqueID]interface{}) row, ok := v.Value.(map[typeutil.UniqueID]interface{})
if !ok { if !ok {
log.Warn("convert interface to map wrong") log.Warn("convert interface to map wrong")
return errors.New("unexpected error") return merr.WrapErrServiceInternalMsg("unexpected error")
} }
expireTs := int64(-1) expireTs := int64(-1)
if hasTTLField { if hasTTLField {
@@ -976,7 +975,7 @@ func (t *clusteringCompactionTask) iterAndGetScalarAnalyzeResult(pkIter *storage
// rowValue := vIter.GetData().(*iterators.InsertRow).GetValue() // rowValue := vIter.GetData().(*iterators.InsertRow).GetValue()
row, ok := (*v).Value.(map[typeutil.UniqueID]interface{}) row, ok := (*v).Value.(map[typeutil.UniqueID]interface{})
if !ok { if !ok {
return nil, 0, errors.New("unexpected error") return nil, 0, merr.WrapErrServiceInternalMsg("unexpected error")
} }
expireTs := int64(-1) expireTs := int64(-1)
@@ -46,6 +46,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/proto/datapb" "github.com/milvus-io/milvus/pkg/v3/proto/datapb"
"github.com/milvus-io/milvus/pkg/v3/proto/indexcgopb" "github.com/milvus-io/milvus/pkg/v3/proto/indexcgopb"
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb" "github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/metautil" "github.com/milvus-io/milvus/pkg/v3/util/metautil"
"github.com/milvus-io/milvus/pkg/v3/util/tsoutil" "github.com/milvus-io/milvus/pkg/v3/util/tsoutil"
"github.com/milvus-io/milvus/pkg/v3/util/typeutil" "github.com/milvus-io/milvus/pkg/v3/util/typeutil"
@@ -80,7 +81,7 @@ func createTextIndex(ctx context.Context,
} }
binlogs, ok := fieldBinlogs[fieldID] binlogs, ok := fieldBinlogs[fieldID]
if !ok { if !ok {
return nil, fmt.Errorf("field binlog not found for field %d", fieldID) return nil, merr.WrapErrParameterInvalidMsg("field binlog not found for field %d", fieldID)
} }
result := make([]string, 0, len(binlogs)) result := make([]string, 0, len(binlogs))
for _, binlog := range binlogs { for _, binlog := range binlogs {
@@ -136,7 +137,7 @@ func createTextIndex(ctx context.Context,
if segment.GetManifest() != "" { if segment.GetManifest() != "" {
basePath, _, err := packed.UnmarshalManifestPath(segment.GetManifest()) basePath, _, err := packed.UnmarshalManifestPath(segment.GetManifest())
if err != nil { if err != nil {
return fmt.Errorf("failed to unmarshal manifest path for text_index basePath: %w", err) return merr.Wrap(err, "failed to unmarshal manifest path for text_index basePath")
} }
statsBasePath = fmt.Sprintf("%s/_stats/text_index.%d", basePath, field.GetFieldID()) statsBasePath = fmt.Sprintf("%s/_stats/text_index.%d", basePath, field.GetFieldID())
} }
+6 -3
View File
@@ -18,10 +18,10 @@ package compactor
import ( import (
"context" "context"
"fmt"
"math" "math"
"sync" "sync"
"github.com/cockroachdb/errors"
"github.com/samber/lo" "github.com/samber/lo"
"go.opentelemetry.io/otel" "go.opentelemetry.io/otel"
"go.uber.org/zap" "go.uber.org/zap"
@@ -40,6 +40,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/util/conc" "github.com/milvus-io/milvus/pkg/v3/util/conc"
"github.com/milvus-io/milvus/pkg/v3/util/funcutil" "github.com/milvus-io/milvus/pkg/v3/util/funcutil"
"github.com/milvus-io/milvus/pkg/v3/util/hardware" "github.com/milvus-io/milvus/pkg/v3/util/hardware"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/metautil" "github.com/milvus-io/milvus/pkg/v3/util/metautil"
"github.com/milvus-io/milvus/pkg/v3/util/paramtable" "github.com/milvus-io/milvus/pkg/v3/util/paramtable"
"github.com/milvus-io/milvus/pkg/v3/util/timerecord" "github.com/milvus-io/milvus/pkg/v3/util/timerecord"
@@ -135,7 +136,9 @@ func (t *LevelZeroCompactionTask) Compact() (*datapb.CompactionPlanResult, error
}) })
if len(targetSegments) == 0 { if len(targetSegments) == 0 {
log.Warn("compact wrong, not target sealed segments") log.Warn("compact wrong, not target sealed segments")
return nil, errors.New("illegal compaction plan with empty target segments") // The plan is produced by datacoord, so a malformed plan is an internal
// protocol violation, not user input.
return nil, merr.WrapErrServiceInternalMsg("illegal compaction plan with empty target segments")
} }
err = binlog.DecompressCompactionBinlogsWithRootPath(t.compactionParams.StorageConfig.GetRootPath(), l0Segments) err = binlog.DecompressCompactionBinlogsWithRootPath(t.compactionParams.StorageConfig.GetRootPath(), l0Segments)
if err != nil { if err != nil {
@@ -416,7 +419,7 @@ func (t *LevelZeroCompactionTask) process(ctx context.Context, l0MemSize int64,
ratio := paramtable.Get().DataNodeCfg.L0BatchMemoryRatio.GetAsFloat() ratio := paramtable.Get().DataNodeCfg.L0BatchMemoryRatio.GetAsFloat()
memLimit := float64(hardware.GetFreeMemoryCount()) * ratio memLimit := float64(hardware.GetFreeMemoryCount()) * ratio
if float64(l0MemSize) > memLimit { if float64(l0MemSize) > memLimit {
return nil, errors.Newf("L0 compaction failed, not enough memory, request memory size: %v, memory limit: %v", l0MemSize, memLimit) return nil, merr.Wrap(merr.ErrServiceMemoryLimitExceeded, fmt.Sprintf("L0 compaction failed, not enough memory, request memory size: %v, memory limit: %v", l0MemSize, memLimit))
} }
log.Info("L0 compaction process start") log.Info("L0 compaction process start")
+6 -4
View File
@@ -25,7 +25,6 @@ import (
"time" "time"
"github.com/apache/arrow/go/v17/arrow/array" "github.com/apache/arrow/go/v17/arrow/array"
"github.com/cockroachdb/errors"
"github.com/samber/lo" "github.com/samber/lo"
"go.opentelemetry.io/otel" "go.opentelemetry.io/otel"
"go.uber.org/zap" "go.uber.org/zap"
@@ -42,6 +41,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/proto/datapb" "github.com/milvus-io/milvus/pkg/v3/proto/datapb"
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb" "github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
"github.com/milvus-io/milvus/pkg/v3/util/funcutil" "github.com/milvus-io/milvus/pkg/v3/util/funcutil"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/metautil" "github.com/milvus-io/milvus/pkg/v3/util/metautil"
"github.com/milvus-io/milvus/pkg/v3/util/paramtable" "github.com/milvus-io/milvus/pkg/v3/util/paramtable"
"github.com/milvus-io/milvus/pkg/v3/util/timerecord" "github.com/milvus-io/milvus/pkg/v3/util/timerecord"
@@ -110,11 +110,13 @@ func (t *mixCompactionTask) preCompact() error {
} }
if len(t.plan.GetSegmentBinlogs()) < 1 { if len(t.plan.GetSegmentBinlogs()) < 1 {
return errors.Newf("compaction plan is illegal, there's no segments in compaction plan, planID = %d", t.GetPlanID()) // The plan is produced by datacoord, so a malformed plan is an internal
// protocol violation, not user input.
return merr.WrapErrServiceInternalMsg("compaction plan is illegal, there's no segments in compaction plan, planID = %d", t.GetPlanID())
} }
if t.plan.GetMaxSize() == 0 { if t.plan.GetMaxSize() == 0 {
return errors.Newf("compaction plan is illegal, empty maxSize, planID = %d", t.GetPlanID()) return merr.WrapErrServiceInternalMsg("compaction plan is illegal, empty maxSize, planID = %d", t.GetPlanID())
} }
t.collectionID = t.plan.GetSegmentBinlogs()[0].GetCollectionID() t.collectionID = t.plan.GetSegmentBinlogs()[0].GetCollectionID()
@@ -440,7 +442,7 @@ func (t *mixCompactionTask) Compact() (*datapb.CompactionPlanResult, error) {
if isEmpty { if isEmpty {
log.Warn("compact wrong, all segments' binlogs are empty") log.Warn("compact wrong, all segments' binlogs are empty")
return nil, errors.New("illegal compaction plan") return nil, merr.WrapErrServiceInternalMsg("illegal compaction plan")
} }
sortMergeAppicable := t.compactionParams.UseMergeSort sortMergeAppicable := t.compactionParams.UseMergeSort
@@ -17,12 +17,9 @@
package compactor package compactor
import ( import (
"fmt"
"github.com/apache/arrow/go/v17/arrow" "github.com/apache/arrow/go/v17/arrow"
"github.com/apache/arrow/go/v17/arrow/array" "github.com/apache/arrow/go/v17/arrow/array"
"github.com/apache/arrow/go/v17/arrow/memory" "github.com/apache/arrow/go/v17/arrow/memory"
"github.com/cockroachdb/errors"
"github.com/milvus-io/milvus-proto/go-api/v3/schemapb" "github.com/milvus-io/milvus-proto/go-api/v3/schemapb"
"github.com/milvus-io/milvus/internal/storage" "github.com/milvus-io/milvus/internal/storage"
@@ -79,7 +76,7 @@ func NewRecordMaterializer(schema *schemapb.CollectionSchema, functions []*schem
} }
if runner == nil { if runner == nil {
materializer.Close() materializer.Close()
return nil, errors.Newf("failed to set up function runner for %s", functionSchema.GetName()) return nil, merr.WrapErrFunctionFailedMsg("failed to set up function runner for %s", functionSchema.GetName())
} }
functionMaterializer, err := newFunctionMaterializer(schema, runner, outputIndexes, true) functionMaterializer, err := newFunctionMaterializer(schema, runner, outputIndexes, true)
if err != nil { if err != nil {
@@ -243,7 +240,7 @@ func (r *selectedRecord) Column(fieldID storage.FieldID) arrow.Array {
defer selected.Release() defer selected.Release()
col := selected.Column(fieldID) col := selected.Column(fieldID)
if col == nil { if col == nil {
r.err = errors.Newf("selected record field %d not found", fieldID) r.err = merr.WrapErrServiceInternalMsg("selected record field %d not found", fieldID)
return nil return nil
} }
col.Retain() col.Retain()
@@ -347,7 +344,7 @@ func newFunctionMaterializer(schema *schemapb.CollectionSchema, runner function.
case schemapb.FunctionType_MinHash: case schemapb.FunctionType_MinHash:
return newMinHashFunctionMaterializer(schema, runner, missingOutputIndexes, ownRunner) return newMinHashFunctionMaterializer(schema, runner, missingOutputIndexes, ownRunner)
default: default:
return nil, errors.Newf("unsupported function type %s", functionSchema.GetType().String()) return nil, merr.WrapErrParameterInvalidMsg("unsupported function type %s", functionSchema.GetType().String())
} }
} }
@@ -355,35 +352,35 @@ func newMinHashFunctionMaterializer(schema *schemapb.CollectionSchema, runner fu
functionSchema := runner.GetSchema() functionSchema := runner.GetSchema()
inputFields := runner.GetInputFields() inputFields := runner.GetInputFields()
if len(inputFields) == 0 { if len(inputFields) == 0 {
return nil, errors.New("minhash function should have input fields") return nil, merr.WrapErrFunctionFailedMsg("minhash function should have input fields")
} }
inputFieldIDs := make([]int64, 0, len(inputFields)) inputFieldIDs := make([]int64, 0, len(inputFields))
for _, inputField := range inputFields { for _, inputField := range inputFields {
if inputField == nil || typeutil.GetField(schema, inputField.GetFieldID()) == nil { if inputField == nil || typeutil.GetField(schema, inputField.GetFieldID()) == nil {
return nil, errors.New("input field not found in schema") return nil, merr.WrapErrFunctionFailedMsg("input field not found in schema")
} }
if inputField.GetDataType() != schemapb.DataType_VarChar && inputField.GetDataType() != schemapb.DataType_Text { if inputField.GetDataType() != schemapb.DataType_VarChar && inputField.GetDataType() != schemapb.DataType_Text {
return nil, errors.New("input field data type must be varchar or text for minhash function materialization") return nil, merr.WrapErrFunctionFailedMsg("input field data type must be varchar or text for minhash function materialization")
} }
inputFieldIDs = append(inputFieldIDs, inputField.GetFieldID()) inputFieldIDs = append(inputFieldIDs, inputField.GetFieldID())
} }
outputFieldIDs := functionSchema.GetOutputFieldIds() outputFieldIDs := functionSchema.GetOutputFieldIds()
if len(outputFieldIDs) == 0 { if len(outputFieldIDs) == 0 {
return nil, errors.New("minhash function should have output fields") return nil, merr.WrapErrFunctionFailedMsg("minhash function should have output fields")
} }
outputFields := make(map[int64]*schemapb.FieldSchema, len(outputFieldIDs)) outputFields := make(map[int64]*schemapb.FieldSchema, len(outputFieldIDs))
for _, outputFieldID := range outputFieldIDs { for _, outputFieldID := range outputFieldIDs {
outputField := typeutil.GetField(schema, outputFieldID) outputField := typeutil.GetField(schema, outputFieldID)
if outputField == nil { if outputField == nil {
return nil, errors.New("output field not found in schema") return nil, merr.WrapErrFunctionFailedMsg("output field not found in schema")
} }
if outputField.GetDataType() != schemapb.DataType_BinaryVector { if outputField.GetDataType() != schemapb.DataType_BinaryVector {
return nil, errors.New("output field data type must be binary vector for minhash function materialization") return nil, merr.WrapErrFunctionFailedMsg("output field data type must be binary vector for minhash function materialization")
} }
if outputField.GetNullable() { if outputField.GetNullable() {
return nil, errors.Newf("function output field cannot be nullable: function %s, field %s", functionSchema.GetName(), outputField.GetName()) return nil, merr.WrapErrFunctionFailedMsg("function output field cannot be nullable: function %s, field %s", functionSchema.GetName(), outputField.GetName())
} }
outputFields[outputFieldID] = outputField outputFields[outputFieldID] = outputField
} }
@@ -402,35 +399,35 @@ func newBM25FunctionMaterializer(schema *schemapb.CollectionSchema, runner funct
functionSchema := runner.GetSchema() functionSchema := runner.GetSchema()
inputFields := runner.GetInputFields() inputFields := runner.GetInputFields()
if len(inputFields) == 0 { if len(inputFields) == 0 {
return nil, errors.New("bm25 function should have input fields") return nil, merr.WrapErrParameterInvalidMsg("bm25 function should have input fields")
} }
inputFieldIDs := make([]int64, 0, len(inputFields)) inputFieldIDs := make([]int64, 0, len(inputFields))
for _, inputField := range inputFields { for _, inputField := range inputFields {
if inputField == nil || typeutil.GetField(schema, inputField.GetFieldID()) == nil { if inputField == nil || typeutil.GetField(schema, inputField.GetFieldID()) == nil {
return nil, errors.New("input field not found in schema") return nil, merr.WrapErrParameterInvalidMsg("input field not found in schema")
} }
if inputField.GetDataType() != schemapb.DataType_VarChar && inputField.GetDataType() != schemapb.DataType_Text { if inputField.GetDataType() != schemapb.DataType_VarChar && inputField.GetDataType() != schemapb.DataType_Text {
return nil, errors.New("input field data type must be varchar or text for bm25 function materialization") return nil, merr.WrapErrParameterInvalidMsg("input field data type must be varchar or text for bm25 function materialization")
} }
inputFieldIDs = append(inputFieldIDs, inputField.GetFieldID()) inputFieldIDs = append(inputFieldIDs, inputField.GetFieldID())
} }
outputFieldIDs := functionSchema.GetOutputFieldIds() outputFieldIDs := functionSchema.GetOutputFieldIds()
if len(outputFieldIDs) == 0 { if len(outputFieldIDs) == 0 {
return nil, errors.New("bm25 function should have output fields") return nil, merr.WrapErrParameterInvalidMsg("bm25 function should have output fields")
} }
outputFields := make(map[int64]*schemapb.FieldSchema, len(outputFieldIDs)) outputFields := make(map[int64]*schemapb.FieldSchema, len(outputFieldIDs))
for _, outputFieldID := range outputFieldIDs { for _, outputFieldID := range outputFieldIDs {
outputField := typeutil.GetField(schema, outputFieldID) outputField := typeutil.GetField(schema, outputFieldID)
if outputField == nil { if outputField == nil {
return nil, errors.New("output field not found in schema") return nil, merr.WrapErrParameterInvalidMsg("output field not found in schema")
} }
if outputField.GetDataType() != schemapb.DataType_SparseFloatVector { if outputField.GetDataType() != schemapb.DataType_SparseFloatVector {
return nil, errors.New("output field data type must be sparse float vector for bm25 function materialization") return nil, merr.WrapErrParameterInvalidMsg("output field data type must be sparse float vector for bm25 function materialization")
} }
if outputField.GetNullable() { if outputField.GetNullable() {
return nil, errors.Newf("function output field cannot be nullable: function %s, field %s", functionSchema.GetName(), outputField.GetName()) return nil, merr.WrapErrParameterInvalidMsg("function output field cannot be nullable: function %s, field %s", functionSchema.GetName(), outputField.GetName())
} }
outputFields[outputFieldID] = outputField outputFields[outputFieldID] = outputField
} }
@@ -459,7 +456,7 @@ func (m *bm25FunctionMaterializer) Materialize(rec storage.Record) (map[int64]ar
return nil, err return nil, err
} }
if len(outputs) != len(m.outputFieldIDs) { if len(outputs) != len(m.outputFieldIDs) {
return nil, errors.Newf("bm25 function materialization expects %d outputs, got %d", len(m.outputFieldIDs), len(outputs)) return nil, merr.WrapErrFunctionFailedMsg("bm25 function materialization expects %d outputs, got %d", len(m.outputFieldIDs), len(outputs))
} }
result := make(map[int64]arrow.Array, len(m.missingOutputIndexes)) result := make(map[int64]arrow.Array, len(m.missingOutputIndexes))
@@ -468,7 +465,7 @@ func (m *bm25FunctionMaterializer) Materialize(rec storage.Record) (map[int64]ar
outputSparseArray, ok := outputs[outputIndex].(*schemapb.SparseFloatArray) outputSparseArray, ok := outputs[outputIndex].(*schemapb.SparseFloatArray)
if !ok { if !ok {
releaseArrowArrays(result) releaseArrowArrays(result)
return nil, errors.Newf("unexpected output type from BM25 function runner, expected SparseFloatArray, got %T", outputs[outputIndex]) return nil, merr.WrapErrFunctionFailedMsg("unexpected output type from BM25 function runner, expected SparseFloatArray, got %T", outputs[outputIndex])
} }
arr, err := buildSparseFloatVectorArrowArray(m.outputFields[outputFieldID], outputSparseArray, rec.Len()) arr, err := buildSparseFloatVectorArrowArray(m.outputFields[outputFieldID], outputSparseArray, rec.Len())
if err != nil { if err != nil {
@@ -500,7 +497,7 @@ func (m *minHashFunctionMaterializer) Materialize(rec storage.Record) (map[int64
return nil, err return nil, err
} }
if len(outputs) != len(m.outputFieldIDs) { if len(outputs) != len(m.outputFieldIDs) {
return nil, errors.Newf("minhash function materialization expects %d outputs, got %d", len(m.outputFieldIDs), len(outputs)) return nil, merr.WrapErrFunctionFailedMsg("minhash function materialization expects %d outputs, got %d", len(m.outputFieldIDs), len(outputs))
} }
result := make(map[int64]arrow.Array, len(m.missingOutputIndexes)) result := make(map[int64]arrow.Array, len(m.missingOutputIndexes))
@@ -509,12 +506,12 @@ func (m *minHashFunctionMaterializer) Materialize(rec storage.Record) (map[int64
outputFieldData, ok := outputs[outputIndex].(*schemapb.FieldData) outputFieldData, ok := outputs[outputIndex].(*schemapb.FieldData)
if !ok { if !ok {
releaseArrowArrays(result) releaseArrowArrays(result)
return nil, errors.Newf("unexpected output type from MinHash function runner, expected FieldData, got %T", outputs[outputIndex]) return nil, merr.WrapErrFunctionFailedMsg("unexpected output type from MinHash function runner, expected FieldData, got %T", outputs[outputIndex])
} }
vectorField := outputFieldData.GetVectors() vectorField := outputFieldData.GetVectors()
if vectorField == nil || vectorField.GetBinaryVector() == nil { if vectorField == nil || vectorField.GetBinaryVector() == nil {
releaseArrowArrays(result) releaseArrowArrays(result)
return nil, errors.New("unexpected output from MinHash function runner, expected binary vector field data") return nil, merr.WrapErrFunctionFailedMsg("unexpected output from MinHash function runner, expected binary vector field data")
} }
fieldData := &storage.BinaryVectorFieldData{ fieldData := &storage.BinaryVectorFieldData{
Data: vectorField.GetBinaryVector(), Data: vectorField.GetBinaryVector(),
@@ -522,7 +519,7 @@ func (m *minHashFunctionMaterializer) Materialize(rec storage.Record) (map[int64
} }
if fieldData.RowNum() != rec.Len() { if fieldData.RowNum() != rec.Len() {
releaseArrowArrays(result) releaseArrowArrays(result)
return nil, errors.Newf("minhash function output row count mismatch, expected %d, got %d", rec.Len(), fieldData.RowNum()) return nil, merr.WrapErrFunctionFailedMsg("minhash function output row count mismatch, expected %d, got %d", rec.Len(), fieldData.RowNum())
} }
arr, err := buildArrowArrayFromFieldData(m.outputFields[outputFieldID], fieldData, rec.Len()) arr, err := buildArrowArrayFromFieldData(m.outputFields[outputFieldID], fieldData, rec.Len())
if err != nil { if err != nil {
@@ -577,7 +574,7 @@ func missingNonMaterializedSchemaFields(schema *schemapb.CollectionSchema, exist
func stringInputsFromRecord(rec storage.Record, fieldID int64) ([]string, error) { func stringInputsFromRecord(rec storage.Record, fieldID int64) ([]string, error) {
col := rec.Column(fieldID) col := rec.Column(fieldID)
if col == nil { if col == nil {
return nil, merr.WrapErrServiceInternal(fmt.Sprintf("input field %d not found in record", fieldID)) return nil, merr.WrapErrFunctionFailedMsg("input field %d not found in record", fieldID)
} }
inputs := make([]string, rec.Len()) inputs := make([]string, rec.Len())
switch values := col.(type) { switch values := col.(type) {
@@ -588,16 +585,16 @@ func stringInputsFromRecord(rec storage.Record, fieldID int64) ([]string, error)
} }
} }
case *array.Binary: case *array.Binary:
return nil, merr.WrapErrServiceInternal("cannot materialize bm25 from text binary values without lob decoding") return nil, merr.WrapErrFunctionFailedMsg("cannot materialize bm25 from text binary values without lob decoding")
default: default:
return nil, merr.WrapErrServiceInternal(fmt.Sprintf("input field %d data type must be varchar or text for bm25 function materialization, got %T", fieldID, col)) return nil, merr.WrapErrFunctionFailedMsg("input field %d data type must be varchar or text for bm25 function materialization, got %T", fieldID, col)
} }
return inputs, nil return inputs, nil
} }
func buildSparseFloatVectorArrowArray(field *schemapb.FieldSchema, outputSparseArray *schemapb.SparseFloatArray, rowCount int) (arrow.Array, error) { func buildSparseFloatVectorArrowArray(field *schemapb.FieldSchema, outputSparseArray *schemapb.SparseFloatArray, rowCount int) (arrow.Array, error) {
if len(outputSparseArray.GetContents()) != rowCount { if len(outputSparseArray.GetContents()) != rowCount {
return nil, errors.Newf("bm25 function output row count mismatch, expected %d, got %d", rowCount, len(outputSparseArray.GetContents())) return nil, merr.WrapErrFunctionFailedMsg("bm25 function output row count mismatch, expected %d, got %d", rowCount, len(outputSparseArray.GetContents()))
} }
fieldData := &storage.SparseFloatVectorFieldData{ fieldData := &storage.SparseFloatVectorFieldData{
@@ -612,7 +609,7 @@ func buildSparseFloatVectorArrowArray(field *schemapb.FieldSchema, outputSparseA
func buildArrowArrayFromFieldData(field *schemapb.FieldSchema, fieldData storage.FieldData, rowCount int) (arrow.Array, error) { func buildArrowArrayFromFieldData(field *schemapb.FieldSchema, fieldData storage.FieldData, rowCount int) (arrow.Array, error) {
if fieldData.RowNum() != rowCount { if fieldData.RowNum() != rowCount {
return nil, errors.Newf("function output row count mismatch for field %d, expected %d, got %d", field.GetFieldID(), rowCount, fieldData.RowNum()) return nil, merr.WrapErrFunctionFailedMsg("function output row count mismatch for field %d, expected %d, got %d", field.GetFieldID(), rowCount, fieldData.RowNum())
} }
outputSchema := &schemapb.CollectionSchema{Fields: []*schemapb.FieldSchema{field}} outputSchema := &schemapb.CollectionSchema{Fields: []*schemapb.FieldSchema{field}}
@@ -22,7 +22,6 @@ import (
"math" "math"
"github.com/apache/arrow/go/v17/arrow/array" "github.com/apache/arrow/go/v17/arrow/array"
"github.com/cockroachdb/errors"
"github.com/samber/lo" "github.com/samber/lo"
"go.uber.org/atomic" "go.uber.org/atomic"
"go.uber.org/zap" "go.uber.org/zap"
@@ -37,6 +36,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/log" "github.com/milvus-io/milvus/pkg/v3/log"
"github.com/milvus-io/milvus/pkg/v3/proto/datapb" "github.com/milvus-io/milvus/pkg/v3/proto/datapb"
"github.com/milvus-io/milvus/pkg/v3/proto/etcdpb" "github.com/milvus-io/milvus/pkg/v3/proto/etcdpb"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/paramtable" "github.com/milvus-io/milvus/pkg/v3/util/paramtable"
"github.com/milvus-io/milvus/pkg/v3/util/typeutil" "github.com/milvus-io/milvus/pkg/v3/util/typeutil"
) )
@@ -318,7 +318,7 @@ func (w *SegmentWriter) WriteRecord(r storage.Record) error {
for fieldID, stats := range w.bm25Stats { for fieldID, stats := range w.bm25Stats {
field, ok := r.Column(fieldID).(*array.Binary) field, ok := r.Column(fieldID).(*array.Binary)
if !ok { if !ok {
return errors.New("bm25 field value not found") return merr.WrapErrServiceInternalMsg("bm25 field value not found")
} }
stats.AppendBytes(field.Value(i)) stats.AppendBytes(field.Value(i))
} }
@@ -341,12 +341,12 @@ func (w *SegmentWriter) Write(v *storage.Value) error {
for fieldID, stats := range w.bm25Stats { for fieldID, stats := range w.bm25Stats {
data, ok := v.Value.(map[storage.FieldID]interface{})[fieldID] data, ok := v.Value.(map[storage.FieldID]interface{})[fieldID]
if !ok { if !ok {
return errors.New("bm25 field value not found") return merr.WrapErrServiceInternalMsg("bm25 field value not found")
} }
bytes, ok := data.([]byte) bytes, ok := data.([]byte)
if !ok { if !ok {
return errors.New("bm25 field value not sparse bytes") return merr.WrapErrServiceInternalMsg("bm25 field value not sparse bytes")
} }
stats.AppendBytes(bytes) stats.AppendBytes(bytes)
} }
@@ -23,7 +23,6 @@ import (
"time" "time"
"github.com/apache/arrow/go/v17/arrow/array" "github.com/apache/arrow/go/v17/arrow/array"
"github.com/cockroachdb/errors"
"github.com/samber/lo" "github.com/samber/lo"
"go.opentelemetry.io/otel" "go.opentelemetry.io/otel"
"go.uber.org/zap" "go.uber.org/zap"
@@ -109,7 +108,9 @@ func (t *sortCompactionTask) preCompact() error {
} }
if len(t.plan.GetSegmentBinlogs()) != 1 { if len(t.plan.GetSegmentBinlogs()) != 1 {
return errors.Newf("sort compaction should handle exactly one segment, but got %d segments, planID = %d", // The plan is produced by datacoord, so a malformed plan is an internal
// protocol violation, not user input.
return merr.WrapErrServiceInternalMsg("sort compaction should handle exactly one segment, but got %d segments, planID = %d",
len(t.plan.GetSegmentBinlogs()), t.GetPlanID()) len(t.plan.GetSegmentBinlogs()), t.GetPlanID())
} }
+3 -3
View File
@@ -27,7 +27,6 @@ import (
"sync" "sync"
"time" "time"
"github.com/cockroachdb/errors"
"github.com/tidwall/gjson" "github.com/tidwall/gjson"
clientv3 "go.etcd.io/etcd/client/v3" clientv3 "go.etcd.io/etcd/client/v3"
"go.uber.org/zap" "go.uber.org/zap"
@@ -49,6 +48,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/util/conc" "github.com/milvus-io/milvus/pkg/v3/util/conc"
"github.com/milvus-io/milvus/pkg/v3/util/expr" "github.com/milvus-io/milvus/pkg/v3/util/expr"
"github.com/milvus-io/milvus/pkg/v3/util/lifetime" "github.com/milvus-io/milvus/pkg/v3/util/lifetime"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/metricsinfo" "github.com/milvus-io/milvus/pkg/v3/util/metricsinfo"
"github.com/milvus-io/milvus/pkg/v3/util/paramtable" "github.com/milvus-io/milvus/pkg/v3/util/paramtable"
"github.com/milvus-io/milvus/pkg/v3/util/typeutil" "github.com/milvus-io/milvus/pkg/v3/util/typeutil"
@@ -166,7 +166,7 @@ func (node *DataNode) Register() error {
func (node *DataNode) initSession() error { func (node *DataNode) initSession() error {
node.session = sessionutil.NewSession(node.ctx) node.session = sessionutil.NewSession(node.ctx)
if node.session == nil { if node.session == nil {
return errors.New("failed to initialize session") return merr.WrapErrServiceInternalMsg("failed to initialize session")
} }
node.session.Init(typeutil.DataNodeRole, node.address, false) node.session.Init(typeutil.DataNodeRole, node.address, false)
sessionutil.SaveServerInfo(typeutil.DataNodeRole, node.session.ServerID) sessionutil.SaveServerInfo(typeutil.DataNodeRole, node.session.ServerID)
@@ -282,7 +282,7 @@ func (node *DataNode) isHealthy() bool {
// ReadyToFlush tells whether DataNode is ready for flushing // ReadyToFlush tells whether DataNode is ready for flushing
func (node *DataNode) ReadyToFlush() error { func (node *DataNode) ReadyToFlush() error {
if !node.isHealthy() { if !node.isHealthy() {
return errors.New("DataNode not in HEALTHY state") return merr.Wrap(merr.ErrServiceNotReady, "DataNode not in HEALTHY state")
} }
return nil return nil
} }
+25 -24
View File
@@ -20,6 +20,7 @@ import (
"github.com/milvus-io/milvus/internal/util/function/embedding" "github.com/milvus-io/milvus/internal/util/function/embedding"
"github.com/milvus-io/milvus/pkg/v3/log" "github.com/milvus-io/milvus/pkg/v3/log"
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb" "github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/typeutil" "github.com/milvus-io/milvus/pkg/v3/util/typeutil"
) )
@@ -65,11 +66,11 @@ func ExecuteFunctionsForSegment(
inputManifestPath, err := packed.CreateSegmentManifestWithBasePath( inputManifestPath, err := packed.CreateSegmentManifestWithBasePath(
ctx, basePath, format, sourceColumns, fragments, storageConfig) ctx, basePath, format, sourceColumns, fragments, storageConfig)
if err != nil { if err != nil {
return "", fmt.Errorf("create input manifest: %w", err) return "", merr.Wrap(err, "create input manifest")
} }
_, inputVersion, err := packed.UnmarshalManifestPath(inputManifestPath) _, inputVersion, err := packed.UnmarshalManifestPath(inputManifestPath)
if err != nil { if err != nil {
return "", fmt.Errorf("parse input manifest path: %w", err) return "", merr.Wrap(err, "parse input manifest path")
} }
outputFields, outputSchema, err := buildOutputSchema(schema) outputFields, outputSchema, err := buildOutputSchema(schema)
@@ -94,7 +95,7 @@ func ExecuteFunctionsForSegment(
colGroups := []storagecommon.ColumnGroup{{Columns: lo.Range(len(outputFields))}} colGroups := []storagecommon.ColumnGroup{{Columns: lo.Range(len(outputFields))}}
writer, err := packed.NewFFIPackedWriter(basePath, outputArrow, colGroups, storageConfig, nil) writer, err := packed.NewFFIPackedWriter(basePath, outputArrow, colGroups, storageConfig, nil)
if err != nil { if err != nil {
return "", fmt.Errorf("open output writer: %w", err) return "", merr.Wrap(err, "open output writer")
} }
writer.AsNewColumnGroups() writer.AsNewColumnGroups()
@@ -108,7 +109,7 @@ func ExecuteFunctionsForSegment(
output, err := writer.Close() output, err := writer.Close()
if err != nil { if err != nil {
return "", fmt.Errorf("close output writer: %w", err) return "", merr.Wrap(err, "close output writer")
} }
if output != nil { if output != nil {
defer output.Destroy() defer output.Destroy()
@@ -120,7 +121,7 @@ func ExecuteFunctionsForSegment(
} }
manifestPath, err := packed.CommitManifestUpdates(basePath, inputVersion, storageConfig, updates) manifestPath, err := packed.CommitManifestUpdates(basePath, inputVersion, storageConfig, updates)
if err != nil { if err != nil {
return "", fmt.Errorf("commit function output manifest: %w", err) return "", merr.Wrap(err, "commit function output manifest")
} }
log.Info("function execution completed", log.Info("function execution completed",
@@ -139,7 +140,7 @@ func buildOutputSchema(schema *schemapb.CollectionSchema) ([]*schemapb.FieldSche
return nil, nil, err return nil, nil, err
} }
if len(outputFields) == 0 { if len(outputFields) == 0 {
return nil, nil, fmt.Errorf("no function output fields; executor should not have been invoked") return nil, nil, merr.WrapErrServiceInternalMsg("no function output fields; executor should not have been invoked")
} }
return outputFields, &schemapb.CollectionSchema{ return outputFields, &schemapb.CollectionSchema{
Name: schema.GetName(), Name: schema.GetName(),
@@ -177,7 +178,7 @@ func functionOutputFields(schema *schemapb.CollectionSchema) ([]*schemapb.FieldS
continue continue
} }
if _, ok := fieldsByID[fieldID]; !ok { if _, ok := fieldsByID[fieldID]; !ok {
return nil, fmt.Errorf("function output field id %d not found in schema", fieldID) return nil, merr.WrapErrParameterInvalidMsg("function output field id %d not found in schema", fieldID)
} }
addOutputID(fieldID) addOutputID(fieldID)
} }
@@ -187,7 +188,7 @@ func functionOutputFields(schema *schemapb.CollectionSchema) ([]*schemapb.FieldS
} }
field, ok := fieldsByName[fieldName] field, ok := fieldsByName[fieldName]
if !ok { if !ok {
return nil, fmt.Errorf("function output field %s not found in schema", fieldName) return nil, merr.WrapErrParameterInvalidMsg("function output field %s not found in schema", fieldName)
} }
addOutputID(field.GetFieldID()) addOutputID(field.GetFieldID())
} }
@@ -222,7 +223,7 @@ func openInputReader(
}), }),
) )
if err != nil { if err != nil {
return nil, fmt.Errorf("open input manifest: %w", err) return nil, merr.Wrap(err, "open input manifest")
} }
return reader, nil return reader, nil
} }
@@ -236,7 +237,7 @@ func buildFunctionExecutionSchema(
schema *schemapb.CollectionSchema, schema *schemapb.CollectionSchema,
) (*schemapb.CollectionSchema, *schemapb.CollectionSchema, typeutil.Set[int64], error) { ) (*schemapb.CollectionSchema, *schemapb.CollectionSchema, typeutil.Set[int64], error) {
if schema == nil { if schema == nil {
return nil, nil, nil, fmt.Errorf("collection schema is nil") return nil, nil, nil, merr.WrapErrParameterInvalidMsg("collection schema is nil")
} }
inputIDs := make(map[int64]struct{}) inputIDs := make(map[int64]struct{})
for _, fn := range schema.GetFunctions() { for _, fn := range schema.GetFunctions() {
@@ -291,24 +292,24 @@ func buildFunctionExecutionSchema(
inputSchema.Fields = append(inputSchema.Fields, f) inputSchema.Fields = append(inputSchema.Fields, f)
if schema.GetExternalSource() != "" && f.GetExternalField() == "" { if schema.GetExternalSource() != "" && f.GetExternalField() == "" {
return nil, nil, nil, fmt.Errorf("function input field %s has no external_field", f.GetName()) return nil, nil, nil, merr.WrapErrParameterInvalidMsg("function input field %s has no external_field", f.GetName())
} }
} }
for inputID := range inputIDs { for inputID := range inputIDs {
if _, ok := seenInputFields[inputID]; !ok { if _, ok := seenInputFields[inputID]; !ok {
if _, ok := seenExecutionFields[inputID]; !ok { if _, ok := seenExecutionFields[inputID]; !ok {
return nil, nil, nil, fmt.Errorf("function input field id %d not found in schema", inputID) return nil, nil, nil, merr.WrapErrParameterInvalidMsg("function input field id %d not found in schema", inputID)
} }
} }
} }
for outputID := range outputIDs { for outputID := range outputIDs {
if _, ok := seenExecutionFields[outputID]; !ok { if _, ok := seenExecutionFields[outputID]; !ok {
return nil, nil, nil, fmt.Errorf("function output field id %d not found in schema", outputID) return nil, nil, nil, merr.WrapErrParameterInvalidMsg("function output field id %d not found in schema", outputID)
} }
} }
if len(inputSchema.GetFields()) == 0 { if len(inputSchema.GetFields()) == 0 {
return nil, nil, nil, fmt.Errorf("no source input columns for function execution") return nil, nil, nil, merr.WrapErrParameterInvalidMsg("no source input columns for function execution")
} }
return inputSchema, executionSchema, requiredInputFields, nil return inputSchema, executionSchema, requiredInputFields, nil
} }
@@ -332,7 +333,7 @@ func streamBatches(
break break
} }
if err != nil { if err != nil {
return totalRows, fmt.Errorf("read input batch: %w", err) return totalRows, merr.Wrap(err, "read input batch")
} }
if rec == nil { if rec == nil {
break break
@@ -341,7 +342,7 @@ func streamBatches(
batch, err := storage.RecordToInsertData(rec, executionSchema, requiredInputFields) batch, err := storage.RecordToInsertData(rec, executionSchema, requiredInputFields)
rec.Release() rec.Release()
if err != nil { if err != nil {
return totalRows, fmt.Errorf("record to InsertData: %w", err) return totalRows, merr.Wrap(err, "record to InsertData")
} }
if batch.GetRowNum() == 0 { if batch.GetRowNum() == 0 {
continue continue
@@ -351,7 +352,7 @@ func streamBatches(
ClusterID: clusterID, ClusterID: clusterID,
DBName: schema.GetDbName(), DBName: schema.GetDbName(),
}); err != nil { }); err != nil {
return totalRows, fmt.Errorf("execute functions: %w", err) return totalRows, merr.Wrap(err, "execute functions")
} }
if err := accumulateBM25Stats(batch, bm25Acc); err != nil { if err := accumulateBM25Stats(batch, bm25Acc); err != nil {
@@ -359,7 +360,7 @@ func streamBatches(
} }
if err := writeOutputBatch(batch, outputSchema, outputArrow, writer); err != nil { if err := writeOutputBatch(batch, outputSchema, outputArrow, writer); err != nil {
return totalRows, fmt.Errorf("write output batch: %w", err) return totalRows, merr.Wrap(err, "write output batch")
} }
totalRows += int64(batch.GetRowNum()) totalRows += int64(batch.GetRowNum())
} }
@@ -404,13 +405,13 @@ func accumulateBM25Stats(batch *storage.InsertData, acc map[int64]*storage.BM25S
for outID, stats := range acc { for outID, stats := range acc {
raw, present := batch.Data[outID] raw, present := batch.Data[outID]
if !present || raw == nil { if !present || raw == nil {
return fmt.Errorf( return merr.WrapErrFunctionFailedMsg(
"BM25 output field %d missing from batch; executeFunctions did not populate it", "BM25 output field %d missing from batch; executeFunctions did not populate it",
outID) outID)
} }
fd, ok := raw.(*storage.SparseFloatVectorFieldData) fd, ok := raw.(*storage.SparseFloatVectorFieldData)
if !ok { if !ok {
return fmt.Errorf( return merr.WrapErrFunctionFailedMsg(
"BM25 output field %d has wrong type %T (want *SparseFloatVectorFieldData)", "BM25 output field %d has wrong type %T (want *SparseFloatVectorFieldData)",
outID, raw) outID, raw)
} }
@@ -431,18 +432,18 @@ func appendBM25Stats(
} }
log := log.Ctx(ctx) log := log.Ctx(ctx)
if updates == nil { if updates == nil {
return fmt.Errorf("manifest updates is nil") return merr.WrapErrServiceInternalMsg("manifest updates is nil")
} }
entries := make([]packed.StatEntry, 0, len(acc)) entries := make([]packed.StatEntry, 0, len(acc))
for outID, stats := range acc { for outID, stats := range acc {
blob, err := stats.Serialize() blob, err := stats.Serialize()
if err != nil { if err != nil {
return fmt.Errorf("serialize bm25 stats for field %d: %w", outID, err) return merr.Wrapf(err, "serialize bm25 stats for field %d", outID)
} }
fullPath := path.Join(basePath, fmt.Sprintf("_stats/bm25.%d/%d", outID, 0)) fullPath := path.Join(basePath, fmt.Sprintf("_stats/bm25.%d/%d", outID, 0))
if err := packed.WriteFile(storageConfig, fullPath, blob); err != nil { if err := packed.WriteFile(storageConfig, fullPath, blob); err != nil {
return fmt.Errorf("write bm25 stats file %s: %w", fullPath, err) return merr.Wrapf(err, "write bm25 stats file %s", fullPath)
} }
entries = append(entries, packed.StatEntry{ entries = append(entries, packed.StatEntry{
Key: fmt.Sprintf("bm25.%d", outID), Key: fmt.Sprintf("bm25.%d", outID),
@@ -481,7 +482,7 @@ func finalizeBM25Stats(
basePath, version, err := packed.UnmarshalManifestPath(manifestPath) basePath, version, err := packed.UnmarshalManifestPath(manifestPath)
if err != nil { if err != nil {
return "", fmt.Errorf("parse manifest path: %w", err) return "", merr.Wrap(err, "parse manifest path")
} }
updates := &packed.ManifestUpdates{} updates := &packed.ManifestUpdates{}
+3 -1
View File
@@ -29,6 +29,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/proto/datapb" "github.com/milvus-io/milvus/pkg/v3/proto/datapb"
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb" "github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
"github.com/milvus-io/milvus/pkg/v3/util/conc" "github.com/milvus-io/milvus/pkg/v3/util/conc"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
) )
// TaskKey uniquely identifies an external collection task. // TaskKey uniquely identifies an external collection task.
@@ -259,7 +260,8 @@ func (m *ExternalCollectionManager) SubmitTask(
zap.ByteString("stack", stack)) zap.ByteString("stack", stack))
reason := fmt.Sprintf("task panicked: %v", r) reason := fmt.Sprintf("task panicked: %v", r)
m.UpdateResult(clusterID, taskID, indexpb.JobState_JobStateFailed, reason, info.KeptSegments, nil) m.UpdateResult(clusterID, taskID, indexpb.JobState_JobStateFailed, reason, info.KeptSegments, nil)
retErr = fmt.Errorf("%s", reason) // A recovered panic is a server-side failure, never caller input.
retErr = merr.WrapErrServiceInternalMsg("%s", reason)
} }
}() }()
log.Info("executing external collection task in pool", log.Info("executing external collection task in pool",
+19 -19
View File
@@ -164,22 +164,22 @@ func (t *RefreshExternalCollectionTask) PreExecute(ctx context.Context) error {
zap.Int64("collectionID", t.req.GetCollectionID())) zap.Int64("collectionID", t.req.GetCollectionID()))
if t.req == nil { if t.req == nil {
return fmt.Errorf("request is nil") return merr.WrapErrParameterInvalidMsg("request is nil")
} }
if t.req.GetSchema() == nil { if t.req.GetSchema() == nil {
return fmt.Errorf("schema is nil in request") return merr.WrapErrParameterInvalidMsg("schema is nil in request")
} }
if t.req.GetStorageConfig() == nil { if t.req.GetStorageConfig() == nil {
return fmt.Errorf("storage config is nil in request") return merr.WrapErrParameterInvalidMsg("storage config is nil in request")
} }
if t.req.GetExternalSource() == "" { if t.req.GetExternalSource() == "" {
return fmt.Errorf("external source is empty in request") return merr.WrapErrParameterInvalidMsg("external source is empty in request")
} }
// Parse and cache external spec for reuse during Execute // Parse and cache external spec for reuse during Execute
spec, err := externalspec.ParseExternalSpec(t.req.GetExternalSpec()) spec, err := externalspec.ParseExternalSpec(t.req.GetExternalSpec())
if err != nil { if err != nil {
return fmt.Errorf("failed to parse external spec: %w", err) return merr.Wrap(err, "failed to parse external spec")
} }
t.parsedSpec = spec t.parsedSpec = spec
t.columns = packed.GetColumnNamesFromSchema(t.req.GetSchema()) t.columns = packed.GetColumnNamesFromSchema(t.req.GetSchema())
@@ -198,7 +198,7 @@ func (t *RefreshExternalCollectionTask) Execute(ctx context.Context) error {
// Initialize pre-allocated segment IDs from request // Initialize pre-allocated segment IDs from request
if t.req.GetPreAllocatedSegmentIds() == nil { if t.req.GetPreAllocatedSegmentIds() == nil {
return fmt.Errorf("pre-allocated segment IDs not provided in request") return merr.WrapErrParameterInvalidMsg("pre-allocated segment IDs not provided in request")
} }
t.preallocatedIDRange = t.req.GetPreAllocatedSegmentIds() t.preallocatedIDRange = t.req.GetPreAllocatedSegmentIds()
@@ -212,13 +212,13 @@ func (t *RefreshExternalCollectionTask) Execute(ctx context.Context) error {
// Fetch fragments from external source // Fetch fragments from external source
newFragments, err := t.fetchFragmentsFromExternalSource(ctx) newFragments, err := t.fetchFragmentsFromExternalSource(ctx)
if err != nil { if err != nil {
return fmt.Errorf("failed to fetch fragments: %w", err) return merr.Wrap(err, "failed to fetch fragments")
} }
// Build current segment -> fragments mapping // Build current segment -> fragments mapping
currentSegmentFragments, err := t.buildCurrentSegmentFragments() currentSegmentFragments, err := t.buildCurrentSegmentFragments()
if err != nil { if err != nil {
return fmt.Errorf("failed to build current segment fragments: %w", err) return merr.Wrap(err, "failed to build current segment fragments")
} }
// Compare and organize segments // Compare and organize segments
@@ -235,7 +235,7 @@ func (t *RefreshExternalCollectionTask) Execute(ctx context.Context) error {
func (t *RefreshExternalCollectionTask) fetchFragmentsFromExternalSource(ctx context.Context) ([]packed.Fragment, error) { func (t *RefreshExternalCollectionTask) fetchFragmentsFromExternalSource(ctx context.Context) ([]packed.Fragment, error) {
manifestPath := t.req.GetExploreManifestPath() manifestPath := t.req.GetExploreManifestPath()
if manifestPath == "" { if manifestPath == "" {
return nil, fmt.Errorf("explore manifest path is required but not provided") return nil, merr.WrapErrParameterMissingMsg("explore manifest path is required but not provided")
} }
log.Ctx(ctx).Info("reading file list from explore manifest", log.Ctx(ctx).Info("reading file list from explore manifest",
@@ -329,7 +329,7 @@ func (t *RefreshExternalCollectionTask) organizeSegments(
var err error var err error
outputColumns, err = functionOutputColumnNames(t.req.GetSchema()) outputColumns, err = functionOutputColumnNames(t.req.GetSchema())
if err != nil { if err != nil {
return nil, fmt.Errorf("resolve function output columns: %w", err) return nil, merr.Wrap(err, "resolve function output columns")
} }
} }
@@ -455,7 +455,7 @@ func (t *RefreshExternalCollectionTask) segmentHasFunctionOutputColumns(seg *dat
} }
hasColumns, err := packed.ManifestHasColumns(seg.GetManifestPath(), t.req.GetStorageConfig(), outputColumns) hasColumns, err := packed.ManifestHasColumns(seg.GetManifestPath(), t.req.GetStorageConfig(), outputColumns)
if err != nil { if err != nil {
return false, fmt.Errorf("check function output columns for segment %d: %w", seg.GetID(), err) return false, merr.Wrapf(err, "check function output columns for segment %d", seg.GetID())
} }
return hasColumns, nil return hasColumns, nil
} }
@@ -557,7 +557,7 @@ func (t *RefreshExternalCollectionTask) patchSegmentForMissingColumns(
t.req.GetStorageConfig(), t.req.GetStorageConfig(),
) )
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to append manifest columns for segment %d: %w", seg.GetID(), err) return nil, merr.Wrapf(err, "failed to append manifest columns for segment %d", seg.GetID())
} }
sampleRows := paramtable.Get().QueryNodeCfg.ExternalCollectionSampleRows.GetAsInt() sampleRows := paramtable.Get().QueryNodeCfg.ExternalCollectionSampleRows.GetAsInt()
@@ -574,7 +574,7 @@ func (t *RefreshExternalCollectionTask) patchSegmentForMissingColumns(
t.req.GetStorageConfig(), t.req.GetStorageConfig(),
) )
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to sample external field sizes for segment %d: %w", seg.GetID(), err) return nil, merr.Wrapf(err, "failed to sample external field sizes for segment %d", seg.GetID())
} }
externalAvgBytes := sumFieldSizes(fieldSizes, schema) externalAvgBytes := sumFieldSizes(fieldSizes, schema)
if externalAvgBytes <= 0 { if externalAvgBytes <= 0 {
@@ -690,7 +690,7 @@ func (t *RefreshExternalCollectionTask) balanceFragmentsToSegments(ctx context.C
} }
// Each segment needs 2 IDs: one for segment, one for fake binlog logID // Each segment needs 2 IDs: one for segment, one for fake binlog logID
if t.nextAllocID+1 >= t.preallocatedIDRange.End { if t.nextAllocID+1 >= t.preallocatedIDRange.End {
return nil, fmt.Errorf("insufficient pre-allocated IDs: need 2 more but only have %d IDs in range [%d, %d)", return nil, merr.WrapErrParameterInvalidMsg("insufficient pre-allocated IDs: need 2 more but only have %d IDs in range [%d, %d)",
t.preallocatedIDRange.End-t.nextAllocID, t.preallocatedIDRange.End-t.nextAllocID,
t.preallocatedIDRange.Begin, t.preallocatedIDRange.Begin,
t.preallocatedIDRange.End) t.preallocatedIDRange.End)
@@ -739,7 +739,7 @@ func (t *RefreshExternalCollectionTask) balanceFragmentsToSegments(ctx context.C
manifestPath, err = t.createManifestForSegment(ctx, work.segmentID, work.fragments) manifestPath, err = t.createManifestForSegment(ctx, work.segmentID, work.fragments)
} }
if err != nil { if err != nil {
return "", fmt.Errorf("failed to create manifest for segment %d: %w", work.segmentID, err) return "", merr.Wrapf(err, "failed to create manifest for segment %d", work.segmentID)
} }
return manifestPath, nil return manifestPath, nil
}) })
@@ -834,7 +834,7 @@ func (t *RefreshExternalCollectionTask) balanceFragmentsToSegments(ctx context.C
log.Warn("external field size sample produced non-positive total", log.Warn("external field size sample produced non-positive total",
zap.String("manifestPath", manifestPath), zap.String("manifestPath", manifestPath),
zap.Int64("total", total)) zap.Int64("total", total))
recordErr(fmt.Errorf("sampled field sizes sum to %d (schema may have no external_field mappings, or sampled rows are empty)", total)) recordErr(merr.WrapErrParameterInvalidMsg("sampled field sizes sum to %d (schema may have no external_field mappings, or sampled rows are empty)", total))
return 0, false return 0, false
} }
return total, true return total, true
@@ -882,9 +882,9 @@ func (t *RefreshExternalCollectionTask) balanceFragmentsToSegments(ctx context.C
if strings.Contains(rootCause, "not found in schema") { if strings.Contains(rootCause, "not found in schema") {
hint = "; check external_field mappings in collection schema against actual parquet columns" hint = "; check external_field mappings in collection schema against actual parquet columns"
} }
return nil, merr.WrapErrParameterInvalidMsg(fmt.Sprintf( return nil, merr.WrapErrParameterInvalidMsg(
"external field size sampling failed for all %d segment(s): %s%s", "external field size sampling failed for all %d segment(s): %s%s",
len(manifestPaths), rootCause, hint)) len(manifestPaths), rootCause, hint)
} }
// Fill any zero slots (sampling failed mid-loop) with the first // Fill any zero slots (sampling failed mid-loop) with the first
// successful average so every segment gets a non-zero MemorySize. // successful average so every segment gets a non-zero MemorySize.
@@ -1032,7 +1032,7 @@ func estimateFunctionOutputBytesPerRow(schema *schemapb.CollectionSchema) (int64
Fields: []*schemapb.FieldSchema{field}, Fields: []*schemapb.FieldSchema{field},
}) })
if err != nil { if err != nil {
return 0, fmt.Errorf("estimate function output field %s: %w", field.GetName(), err) return 0, merr.Wrapf(err, "estimate function output field %s", field.GetName())
} }
total += int64(size) total += int64(size)
} }
@@ -18,7 +18,6 @@ package importv2
import ( import (
"context" "context"
"fmt"
"path" "path"
"strconv" "strconv"
"strings" "strings"
@@ -34,6 +33,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/log" "github.com/milvus-io/milvus/pkg/v3/log"
"github.com/milvus-io/milvus/pkg/v3/proto/datapb" "github.com/milvus-io/milvus/pkg/v3/proto/datapb"
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb" "github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/metautil" "github.com/milvus-io/milvus/pkg/v3/util/metautil"
) )
@@ -114,12 +114,12 @@ func transformManifestPath(
) (string, error) { ) (string, error) {
basePath, version, err := packed.UnmarshalManifestPath(manifestPath) basePath, version, err := packed.UnmarshalManifestPath(manifestPath)
if err != nil { if err != nil {
return "", fmt.Errorf("failed to unmarshal manifest path: %w", err) return "", merr.Wrap(err, "failed to unmarshal manifest path")
} }
targetBasePath, err := generateTargetPath(basePath, source, target) targetBasePath, err := generateTargetPath(basePath, source, target)
if err != nil { if err != nil {
return "", fmt.Errorf("failed to generate target base path: %w", err) return "", merr.Wrap(err, "failed to generate target base path")
} }
targetManifestPath := packed.MarshalManifestPath(targetBasePath, version) targetManifestPath := packed.MarshalManifestPath(targetBasePath, version)
@@ -227,18 +227,18 @@ func collectSegmentFiles(
// StorageV3+: binlog paths MUST come from manifest // StorageV3+: binlog paths MUST come from manifest
manifestPath := source.GetManifestPath() manifestPath := source.GetManifestPath()
if manifestPath == "" { if manifestPath == "" {
return nil, fmt.Errorf("storage_version=%d requires manifest_path but it is empty (segmentID=%d)", return nil, merr.WrapErrParameterInvalidMsg("storage_version=%d requires manifest_path but it is empty (segmentID=%d)",
source.GetStorageVersion(), source.GetSegmentId()) source.GetStorageVersion(), source.GetSegmentId())
} }
basePath, _, err := packed.UnmarshalManifestPath(manifestPath) basePath, _, err := packed.UnmarshalManifestPath(manifestPath)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to unmarshal manifest path %q for segment %d: %w", manifestPath, source.GetSegmentId(), err) return nil, merr.Wrapf(err, "failed to unmarshal manifest path %q for segment %d", manifestPath, source.GetSegmentId())
} }
allFiles, listErr := listAllFiles(ctx, cm, basePath) allFiles, listErr := listAllFiles(ctx, cm, basePath)
if listErr != nil { if listErr != nil {
return nil, fmt.Errorf("failed to list files from manifest base path %q for segment %d: %w", basePath, source.GetSegmentId(), listErr) return nil, merr.Wrapf(listErr, "failed to list files from manifest base path %q for segment %d", basePath, source.GetSegmentId())
} }
// Empty file list is OK for V3 — segment may have only deltas and no insert binlogs // Empty file list is OK for V3 — segment may have only deltas and no insert binlogs
@@ -319,7 +319,7 @@ func generateMappingsFromFiles(
} }
if err != nil { if err != nil {
return fmt.Errorf("failed to generate target path for %s file %s: %w", fileType, srcPath, err) return merr.Wrapf(err, "failed to generate target path for %s file %s", fileType, srcPath)
} }
mappings[srcPath] = dstPath mappings[srcPath] = dstPath
} }
@@ -379,13 +379,13 @@ func CopySegmentAndIndexFiles(
// Step 1: Collect all files to copy // Step 1: Collect all files to copy
files, err := collectSegmentFiles(ctx, cm, source) files, err := collectSegmentFiles(ctx, cm, source)
if err != nil { if err != nil {
return nil, nil, fmt.Errorf("failed to collect segment files: %w", err) return nil, nil, merr.Wrap(err, "failed to collect segment files")
} }
// Step 2: Generate src->dst mappings for file copying // Step 2: Generate src->dst mappings for file copying
mappings, err := generateMappingsFromFiles(files, source, target) mappings, err := generateMappingsFromFiles(files, source, target)
if err != nil { if err != nil {
return nil, nil, fmt.Errorf("failed to generate file mappings: %w", err) return nil, nil, merr.Wrap(err, "failed to generate file mappings")
} }
// Step 3: Execute all copy operations // Step 3: Execute all copy operations
@@ -400,7 +400,7 @@ func CopySegmentAndIndexFiles(
fields = append(fields, logFields...) fields = append(fields, logFields...)
fields = append(fields, zap.String("src", src), zap.String("dst", dst), zap.Error(err)) fields = append(fields, zap.String("src", src), zap.String("dst", dst), zap.Error(err))
log.Warn("failed to copy file", fields...) log.Warn("failed to copy file", fields...)
return nil, copiedFiles, fmt.Errorf("failed to copy file from %s to %s: %w", src, dst, err) return nil, copiedFiles, merr.Wrapf(err, "failed to copy file from %s to %s", src, dst)
} }
copiedFiles = append(copiedFiles, dst) copiedFiles = append(copiedFiles, dst)
} }
@@ -418,7 +418,7 @@ func CopySegmentAndIndexFiles(
if _, exists := mappings[srcPath]; !exists { if _, exists := mappings[srcPath]; !exists {
dstPath, pathErr := generateTargetPath(srcPath, source, target) dstPath, pathErr := generateTargetPath(srcPath, source, target)
if pathErr != nil { if pathErr != nil {
return nil, copiedFiles, fmt.Errorf("failed to generate target path for pb insert binlog %s: %w", srcPath, pathErr) return nil, copiedFiles, merr.Wrapf(pathErr, "failed to generate target path for pb insert binlog %s", srcPath)
} }
mappings[srcPath] = dstPath mappings[srcPath] = dstPath
} }
@@ -430,20 +430,20 @@ func CopySegmentAndIndexFiles(
// Step 4: Build index metadata from source // Step 4: Build index metadata from source
indexInfos, textIndexInfos, jsonKeyIndexInfos, err := buildIndexInfoFromSource(source, target, mappings) indexInfos, textIndexInfos, jsonKeyIndexInfos, err := buildIndexInfoFromSource(source, target, mappings)
if err != nil { if err != nil {
return nil, copiedFiles, fmt.Errorf("failed to build index info: %w", err) return nil, copiedFiles, merr.Wrap(err, "failed to build index info")
} }
// Step 5: Generate segment metadata with path mappings // Step 5: Generate segment metadata with path mappings
segmentInfo, err := generateSegmentInfoFromSource(source, target, mappings) segmentInfo, err := generateSegmentInfoFromSource(source, target, mappings)
if err != nil { if err != nil {
return nil, copiedFiles, fmt.Errorf("failed to generate segment info: %w", err) return nil, copiedFiles, merr.Wrap(err, "failed to generate segment info")
} }
// Step 6: Compress paths // Step 6: Compress paths
err = binlog.CompressBinLogs(segmentInfo.GetBinlogs(), segmentInfo.GetStatslogs(), err = binlog.CompressBinLogs(segmentInfo.GetBinlogs(), segmentInfo.GetStatslogs(),
segmentInfo.GetDeltalogs(), segmentInfo.GetBm25Logs()) segmentInfo.GetDeltalogs(), segmentInfo.GetBm25Logs())
if err != nil { if err != nil {
return nil, copiedFiles, fmt.Errorf("failed to compress binlog paths: %w", err) return nil, copiedFiles, merr.Wrap(err, "failed to compress binlog paths")
} }
for _, indexInfo := range indexInfos { for _, indexInfo := range indexInfos {
@@ -474,7 +474,7 @@ func CopySegmentAndIndexFiles(
if useManifest { if useManifest {
targetManifestPath, err := transformManifestPath(source.GetManifestPath(), source, target) targetManifestPath, err := transformManifestPath(source.GetManifestPath(), source, target)
if err != nil { if err != nil {
return nil, copiedFiles, fmt.Errorf("failed to transform manifest path: %w", err) return nil, copiedFiles, merr.Wrap(err, "failed to transform manifest path")
} }
result.ManifestPath = targetManifestPath result.ManifestPath = targetManifestPath
} }
@@ -526,7 +526,7 @@ func transformFieldBinlogs(
} }
dstPath, ok := mappings[srcPath] dstPath, ok := mappings[srcPath]
if !ok { if !ok {
return nil, 0, fmt.Errorf("no mapping found for source path: %s", srcPath) return nil, 0, merr.WrapErrServiceInternalMsg("no mapping found for source path: %s", srcPath)
} }
dstBinlog.LogPath = dstPath dstBinlog.LogPath = dstPath
} }
@@ -583,7 +583,7 @@ func generateSegmentInfoFromSource(
// Process insert binlogs (count rows) // Process insert binlogs (count rows)
binlogs, totalRows, err := transformFieldBinlogs(source.GetInsertBinlogs(), mappings, true, source.GetIsExternalCollection()) binlogs, totalRows, err := transformFieldBinlogs(source.GetInsertBinlogs(), mappings, true, source.GetIsExternalCollection())
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to transform insert binlogs: %w", err) return nil, merr.Wrap(err, "failed to transform insert binlogs")
} }
segmentInfo.Binlogs = binlogs segmentInfo.Binlogs = binlogs
segmentInfo.ImportedRows = totalRows segmentInfo.ImportedRows = totalRows
@@ -591,21 +591,21 @@ func generateSegmentInfoFromSource(
// Process stats binlogs (no row counting) // Process stats binlogs (no row counting)
statslogs, _, err := transformFieldBinlogs(source.GetStatsBinlogs(), mappings, false, false) statslogs, _, err := transformFieldBinlogs(source.GetStatsBinlogs(), mappings, false, false)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to transform stats binlogs: %w", err) return nil, merr.Wrap(err, "failed to transform stats binlogs")
} }
segmentInfo.Statslogs = statslogs segmentInfo.Statslogs = statslogs
// Process delta binlogs (no row counting) // Process delta binlogs (no row counting)
deltalogs, _, err := transformFieldBinlogs(source.GetDeltaBinlogs(), mappings, false, false) deltalogs, _, err := transformFieldBinlogs(source.GetDeltaBinlogs(), mappings, false, false)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to transform delta binlogs: %w", err) return nil, merr.Wrap(err, "failed to transform delta binlogs")
} }
segmentInfo.Deltalogs = deltalogs segmentInfo.Deltalogs = deltalogs
// Process BM25 binlogs (no row counting) // Process BM25 binlogs (no row counting)
bm25logs, _, err := transformFieldBinlogs(source.GetBm25Binlogs(), mappings, false, false) bm25logs, _, err := transformFieldBinlogs(source.GetBm25Binlogs(), mappings, false, false)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to transform BM25 binlogs: %w", err) return nil, merr.Wrap(err, "failed to transform BM25 binlogs")
} }
segmentInfo.Bm25Logs = bm25logs segmentInfo.Bm25Logs = bm25logs
@@ -635,7 +635,7 @@ func generateTargetPath(sourcePath string, source *datapb.CopySegmentSource, tar
} }
if logTypeIndex == -1 || logTypeIndex+3 >= len(parts) { if logTypeIndex == -1 || logTypeIndex+3 >= len(parts) {
return "", fmt.Errorf("invalid binlog path structure: %s (expected log_type at a valid position)", sourcePath) return "", merr.WrapErrParameterInvalidMsg("invalid binlog path structure: %s (expected log_type at a valid position)", sourcePath)
} }
// Replace IDs in order: collectionID, partitionID, segmentID // Replace IDs in order: collectionID, partitionID, segmentID
@@ -667,7 +667,7 @@ func generateTargetLOBPath(sourcePath string, source *datapb.CopySegmentSource,
// Path: .../{insert_log}/{coll}/{part}/lobs/... // Path: .../{insert_log}/{coll}/{part}/lobs/...
// Need at least logTypeIndex + 2 (coll and part) after insert_log // Need at least logTypeIndex + 2 (coll and part) after insert_log
if logTypeIndex == -1 || logTypeIndex+2 >= len(parts) { if logTypeIndex == -1 || logTypeIndex+2 >= len(parts) {
return "", fmt.Errorf("invalid LOB path structure: %s", sourcePath) return "", merr.WrapErrParameterInvalidMsg("invalid LOB path structure: %s", sourcePath)
} }
parts[logTypeIndex+1] = strconv.FormatInt(target.GetCollectionId(), 10) parts[logTypeIndex+1] = strconv.FormatInt(target.GetCollectionId(), 10)
@@ -709,7 +709,7 @@ func buildIndexInfoFromSource(
for _, srcPath := range srcIndex.GetIndexFilePaths() { for _, srcPath := range srcIndex.GetIndexFilePaths() {
targetPath, ok := mappings[srcPath] targetPath, ok := mappings[srcPath]
if !ok { if !ok {
return nil, nil, nil, fmt.Errorf("no mapping found for index file: %s", srcPath) return nil, nil, nil, merr.WrapErrServiceInternalMsg("no mapping found for index file: %s", srcPath)
} }
targetPaths = append(targetPaths, targetPath) targetPaths = append(targetPaths, targetPath)
} }
@@ -753,7 +753,7 @@ func buildIndexInfoFromSource(
for _, srcFile := range srcText.GetFiles() { for _, srcFile := range srcText.GetFiles() {
targetFile, ok := mappings[srcFile] targetFile, ok := mappings[srcFile]
if !ok { if !ok {
return nil, nil, nil, fmt.Errorf("no mapping found for text index file: %s", srcFile) return nil, nil, nil, merr.WrapErrServiceInternalMsg("no mapping found for text index file: %s", srcFile)
} }
targetFiles = append(targetFiles, targetFile) targetFiles = append(targetFiles, targetFile)
} }
@@ -785,7 +785,7 @@ func buildIndexInfoFromSource(
for _, srcFile := range srcJSON.GetFiles() { for _, srcFile := range srcJSON.GetFiles() {
targetFile, ok := mappings[srcFile] targetFile, ok := mappings[srcFile]
if !ok { if !ok {
return nil, nil, nil, fmt.Errorf("no mapping found for JSON index file: %s", srcFile) return nil, nil, nil, merr.WrapErrServiceInternalMsg("no mapping found for JSON index file: %s", srcFile)
} }
targetFiles = append(targetFiles, targetFile) targetFiles = append(targetFiles, targetFile)
} }
@@ -896,7 +896,7 @@ func generateTargetIndexPath(
} }
if keywordIdx == -1 { if keywordIdx == -1 {
return "", fmt.Errorf("keyword '%s' not found in path: %s", keyword, sourcePath) return "", merr.WrapErrServiceInternalMsg("keyword '%s' not found in path: %s", keyword, sourcePath)
} }
// Set offsets based on index type // Set offsets based on index type
@@ -928,13 +928,13 @@ func generateTargetIndexPath(
segmentOffset = 6 segmentOffset = 6
buildIDOffset = 2 buildIDOffset = 2
default: default:
return "", fmt.Errorf("unsupported index type: %s (expected '%s', '%s', '%s', or '%s')", return "", merr.WrapErrParameterInvalidMsg("unsupported index type: %s (expected '%s', '%s', '%s', or '%s')",
indexType, IndexTypeVectorScalarV0, IndexTypeText, IndexTypeJSONKey, IndexTypeJSONStats) indexType, IndexTypeVectorScalarV0, IndexTypeText, IndexTypeJSONKey, IndexTypeJSONStats)
} }
// Validate path structure has enough components // Validate path structure has enough components
if keywordIdx+segmentOffset >= len(parts) { if keywordIdx+segmentOffset >= len(parts) {
return "", fmt.Errorf("invalid %s path structure: %s (expected '%s' with at least %d components after it)", return "", merr.WrapErrParameterInvalidMsg("invalid %s path structure: %s (expected '%s' with at least %d components after it)",
indexType, sourcePath, indexType, segmentOffset+1) indexType, sourcePath, indexType, segmentOffset+1)
} }
@@ -22,7 +22,6 @@ import (
"sync" "sync"
"time" "time"
"github.com/cockroachdb/errors"
"go.uber.org/zap" "go.uber.org/zap"
"github.com/milvus-io/milvus-proto/go-api/v3/schemapb" "github.com/milvus-io/milvus-proto/go-api/v3/schemapb"
@@ -30,6 +29,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/log" "github.com/milvus-io/milvus/pkg/v3/log"
"github.com/milvus-io/milvus/pkg/v3/proto/datapb" "github.com/milvus-io/milvus/pkg/v3/proto/datapb"
"github.com/milvus-io/milvus/pkg/v3/util/conc" "github.com/milvus-io/milvus/pkg/v3/util/conc"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/typeutil" "github.com/milvus-io/milvus/pkg/v3/util/typeutil"
) )
@@ -382,7 +382,7 @@ func (t *CopySegmentTask) copySingleSegment(source *datapb.CopySegmentSource, ta
reason := "no insert/delete binlogs for segment" reason := "no insert/delete binlogs for segment"
log.Error(reason, logFields...) log.Error(reason, logFields...)
t.manager.Update(t.GetTaskID(), UpdateState(datapb.ImportTaskStateV2_Failed), UpdateReason(reason)) t.manager.Update(t.GetTaskID(), UpdateState(datapb.ImportTaskStateV2_Failed), UpdateReason(reason))
return nil, errors.New(reason) return nil, merr.WrapErrParameterInvalidMsg(reason)
} }
// Step 2: Copy all segment files (binlogs + indexes) together // Step 2: Copy all segment files (binlogs + indexes) together
+3 -2
View File
@@ -35,6 +35,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/proto/datapb" "github.com/milvus-io/milvus/pkg/v3/proto/datapb"
"github.com/milvus-io/milvus/pkg/v3/proto/internalpb" "github.com/milvus-io/milvus/pkg/v3/proto/internalpb"
"github.com/milvus-io/milvus/pkg/v3/util/conc" "github.com/milvus-io/milvus/pkg/v3/util/conc"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/paramtable" "github.com/milvus-io/milvus/pkg/v3/util/paramtable"
"github.com/milvus-io/milvus/pkg/v3/util/typeutil" "github.com/milvus-io/milvus/pkg/v3/util/typeutil"
) )
@@ -192,9 +193,9 @@ func (t *PreImportTask) readFileStat(reader importutilv2.Reader, fileIdx int) er
} }
maxSize := paramtable.Get().DataNodeCfg.MaxImportFileSizeInGB.GetAsFloat() * 1024 * 1024 * 1024 maxSize := paramtable.Get().DataNodeCfg.MaxImportFileSizeInGB.GetAsFloat() * 1024 * 1024 * 1024
if fileSize > int64(maxSize) { if fileSize > int64(maxSize) {
return errors.New(fmt.Sprintf( return merr.WrapErrParameterInvalidMsg(
"The import file size has reached the maximum limit allowed for importing, "+ "The import file size has reached the maximum limit allowed for importing, "+
"fileSize=%d, maxSize=%d", fileSize, int64(maxSize))) "fileSize=%d, maxSize=%d", fileSize, int64(maxSize))
} }
totalRows := 0 totalRows := 0
+4 -4
View File
@@ -50,7 +50,7 @@ import (
) )
func WrapTaskNotFoundError(taskID int64) error { func WrapTaskNotFoundError(taskID int64) error {
return merr.WrapErrImportFailed(fmt.Sprintf("cannot find import task with id %d", taskID)) return merr.WrapErrImportSysFailedMsg("cannot find import task with id %d", taskID)
} }
func NewSyncTask(ctx context.Context, func NewSyncTask(ctx context.Context,
@@ -151,7 +151,7 @@ func PickSegment(segments []*datapb.ImportRequestSegment, vchannel string, parti
}) })
if len(candidates) == 0 { if len(candidates) == 0 {
return 0, fmt.Errorf("no candidate segments found for channel %s and partition %d", return 0, merr.WrapErrServiceInternalMsg("no candidate segments found for channel %s and partition %d",
vchannel, partitionID) vchannel, partitionID)
} }
@@ -392,7 +392,7 @@ func AppendNullableDefaultFieldsData(schema *schemapb.CollectionSchema, data *st
} }
} }
default: default:
return fmt.Errorf("unexpected data type: %d, cannot be filled with default value", dataType) return merr.WrapErrServiceInternalMsg("unexpected data type: %d, cannot be filled with default value", dataType)
} }
if err != nil { if err != nil {
@@ -409,7 +409,7 @@ func FillDynamicData(schema *schemapb.CollectionSchema, data *storage.InsertData
} }
dynamicField := typeutil.GetDynamicField(schema) dynamicField := typeutil.GetDynamicField(schema)
if dynamicField == nil { if dynamicField == nil {
return merr.WrapErrImportFailed("collection schema is illegal, enable_dynamic_field is true but the dynamic field doesn't exist") return merr.WrapErrImportSysFailed("collection schema is illegal, enable_dynamic_field is true but the dynamic field doesn't exist")
} }
tempData, ok := data.Data[dynamicField.GetFieldID()] tempData, ok := data.Data[dynamicField.GetFieldID()]
+1 -1
View File
@@ -80,7 +80,7 @@ func (queue *IndexTaskQueue) addUnissuedTask(t Task) error {
defer queue.utLock.Unlock() defer queue.utLock.Unlock()
if queue.utFull() { if queue.utFull() {
return errors.New("index task queue is full") return merr.Wrap(merr.ErrServiceResourceInsufficient, "index task queue is full")
} }
queue.unissuedTasks.PushBack(t) queue.unissuedTasks.PushBack(t)
select { select {
+8 -7
View File
@@ -52,6 +52,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb" "github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
"github.com/milvus-io/milvus/pkg/v3/proto/workerpb" "github.com/milvus-io/milvus/pkg/v3/proto/workerpb"
_ "github.com/milvus-io/milvus/pkg/v3/util/funcutil" _ "github.com/milvus-io/milvus/pkg/v3/util/funcutil"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/metautil" "github.com/milvus-io/milvus/pkg/v3/util/metautil"
"github.com/milvus-io/milvus/pkg/v3/util/paramtable" "github.com/milvus-io/milvus/pkg/v3/util/paramtable"
"github.com/milvus-io/milvus/pkg/v3/util/timerecord" "github.com/milvus-io/milvus/pkg/v3/util/timerecord"
@@ -292,7 +293,7 @@ func (st *statsTask) sort(ctx context.Context) ([]*datapb.FieldBinlog, error) {
newManifest, err := packed.AddStatsToManifest( newManifest, err := packed.AddStatsToManifest(
st.manifestPath, st.req.GetStorageConfig(), statEntries) st.manifestPath, st.req.GetStorageConfig(), statEntries)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to add stats to manifest: %w", err) return nil, merr.Wrap(err, "failed to add stats to manifest")
} }
st.manifestPath = newManifest st.manifestPath = newManifest
} }
@@ -496,7 +497,7 @@ func (st *statsTask) createTextIndex(ctx context.Context,
} }
binlogs, ok := fieldBinlogs[fieldID] binlogs, ok := fieldBinlogs[fieldID]
if !ok && !enableNull { if !ok && !enableNull {
return nil, fmt.Errorf("field binlog not found for field %d", fieldID) return nil, merr.WrapErrServiceInternalMsg("field binlog not found for field %d", fieldID)
} }
result := make([]string, 0, len(binlogs)) result := make([]string, 0, len(binlogs))
for _, binlog := range binlogs { for _, binlog := range binlogs {
@@ -618,7 +619,7 @@ func (st *statsTask) createTextIndex(ctx context.Context,
newManifest, err := packed.AddStatsToManifest( newManifest, err := packed.AddStatsToManifest(
st.manifestPath, st.req.GetStorageConfig(), statEntries) st.manifestPath, st.req.GetStorageConfig(), statEntries)
if err != nil { if err != nil {
return fmt.Errorf("failed to add text index stats to manifest: %w", err) return merr.Wrap(err, "failed to add text index stats to manifest")
} }
st.manifestPath = newManifest st.manifestPath = newManifest
// Dual-write: keep textIndexLogs populated so it is stored in segment metadata as a // Dual-write: keep textIndexLogs populated so it is stored in segment metadata as a
@@ -685,7 +686,7 @@ func (st *statsTask) createJSONKeyStats(ctx context.Context,
} }
binlogs, ok := fieldBinlogs[fieldID] binlogs, ok := fieldBinlogs[fieldID]
if !ok && !enableNull { if !ok && !enableNull {
return nil, fmt.Errorf("field binlog not found for field %d", fieldID) return nil, merr.WrapErrServiceInternalMsg("field binlog not found for field %d", fieldID)
} }
result := make([]string, 0, len(binlogs)) result := make([]string, 0, len(binlogs))
for _, binlog := range binlogs { for _, binlog := range binlogs {
@@ -796,7 +797,7 @@ func (st *statsTask) createJSONKeyStats(ctx context.Context,
newManifest, err := packed.AddStatsToManifest( newManifest, err := packed.AddStatsToManifest(
st.manifestPath, st.req.GetStorageConfig(), statEntries) st.manifestPath, st.req.GetStorageConfig(), statEntries)
if err != nil { if err != nil {
return fmt.Errorf("failed to add JSON key stats to manifest: %w", err) return merr.Wrap(err, "failed to add JSON key stats to manifest")
} }
st.manifestPath = newManifest st.manifestPath = newManifest
// Dual-write: keep jsonKeyIndexStats populated so it is stored in segment metadata as a // Dual-write: keep jsonKeyIndexStats populated so it is stored in segment metadata as a
@@ -829,7 +830,7 @@ func computeStatsBasePath(req *workerpb.CreateStatsRequest, manifestPath string,
if req.GetStorageVersion() == storage.StorageV3 { if req.GetStorageVersion() == storage.StorageV3 {
basePath, _, err := packed.UnmarshalManifestPath(manifestPath) basePath, _, err := packed.UnmarshalManifestPath(manifestPath)
if err != nil { if err != nil {
return "", fmt.Errorf("failed to unmarshal manifest path for %s basePath: %w", statsType, err) return "", merr.Wrapf(err, "failed to unmarshal manifest path for %s basePath", statsType)
} }
return fmt.Sprintf("%s/_stats/%s.%d", basePath, statsType, fieldID), nil return fmt.Sprintf("%s/_stats/%s.%d", basePath, statsType, fieldID), nil
} }
@@ -845,7 +846,7 @@ func computeStatsBasePath(req *workerpb.CreateStatsRequest, manifestPath string,
req.GetTaskID(), req.GetTaskVersion(), req.GetTaskID(), req.GetTaskVersion(),
req.GetCollectionID(), req.GetPartitionID(), req.GetTargetSegmentID(), fieldID), nil req.GetCollectionID(), req.GetPartitionID(), req.GetTargetSegmentID(), fieldID), nil
} }
return "", fmt.Errorf("unknown stats type: %s", statsType) return "", merr.WrapErrParameterInvalidMsg("unknown stats type: %s", statsType)
} }
func buildIndexParams( func buildIndexParams(
+9 -8
View File
@@ -21,7 +21,6 @@ import (
"fmt" "fmt"
"strconv" "strconv"
"github.com/cockroachdb/errors"
"go.opentelemetry.io/otel" "go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/trace" "go.opentelemetry.io/otel/trace"
@@ -82,7 +81,9 @@ func (node *DataNode) CreateJob(ctx context.Context, req *workerpb.CreateJobRequ
Cancel: taskCancel, Cancel: taskCancel,
State: commonpb.IndexState_InProgress, State: commonpb.IndexState_InProgress,
}); oldInfo != nil { }); oldInfo != nil {
err := merr.WrapErrIndexDuplicate(req.GetIndexName(), "building index task existed") // Node-internal dedup of a coordinator-dispatched build task: a duplicate
// is a scheduling race, not the user re-creating an index.
err := merr.WrapErrServiceInternalMsg("building index task existed, index=%s, buildID=%d", req.GetIndexName(), req.GetBuildID())
log.Warn("duplicated index build task", zap.Error(err)) log.Warn("duplicated index build task", zap.Error(err))
metrics.DataNodeBuildIndexTaskCounter.WithLabelValues(paramtable.GetStringNodeID(), metrics.FailLabel).Inc() metrics.DataNodeBuildIndexTaskCounter.WithLabelValues(paramtable.GetStringNodeID(), metrics.FailLabel).Inc()
return merr.Status(err), nil return merr.Status(err), nil
@@ -254,7 +255,7 @@ func (node *DataNode) CreateJobV2(ctx context.Context, req *workerpb.CreateJobV2
return node.createStatsTask(ctx, statsRequest) return node.createStatsTask(ctx, statsRequest)
default: default:
log.Warn("DataNode receive unknown type job") log.Warn("DataNode receive unknown type job")
return merr.Status(fmt.Errorf("DataNode receive unknown type job with TaskID: %d", req.GetTaskID())), nil return merr.Status(merr.WrapErrServiceInternalMsg("DataNode receive unknown type job with TaskID: %d", req.GetTaskID())), nil
} }
} }
@@ -454,7 +455,7 @@ func (node *DataNode) QueryJobsV2(ctx context.Context, req *workerpb.QueryJobsV2
default: default:
log.Warn("DataNode receive querying unknown type jobs") log.Warn("DataNode receive querying unknown type jobs")
return &workerpb.QueryJobsV2Response{ return &workerpb.QueryJobsV2Response{
Status: merr.Status(errors.New("DataNode receive querying unknown type jobs")), Status: merr.Status(merr.WrapErrServiceInternalMsg("DataNode receive querying unknown type jobs")),
}, nil }, nil
} }
} }
@@ -479,7 +480,7 @@ func (node *DataNode) queryIndexTask(ctx context.Context, req *workerpb.QueryJob
log.Debug("query index jobs result success", zap.Any("results", results)) log.Debug("query index jobs result success", zap.Any("results", results))
if len(results) == 0 { if len(results) == 0 {
return &workerpb.QueryJobsV2Response{ return &workerpb.QueryJobsV2Response{
Status: merr.Status(fmt.Errorf("tasks '%v' not found", req.GetTaskIDs())), Status: merr.Status(merr.WrapErrServiceInternalMsg("tasks '%v' not found", req.GetTaskIDs())),
}, nil }, nil
} }
return &workerpb.QueryJobsV2Response{ return &workerpb.QueryJobsV2Response{
@@ -508,7 +509,7 @@ func (node *DataNode) queryStatsTask(ctx context.Context, req *workerpb.QueryJob
log.Debug("query stats job result success", zap.Any("results", results)) log.Debug("query stats job result success", zap.Any("results", results))
if len(results) == 0 { if len(results) == 0 {
return &workerpb.QueryJobsV2Response{ return &workerpb.QueryJobsV2Response{
Status: merr.Status(fmt.Errorf("tasks '%v' not found", req.GetTaskIDs())), Status: merr.Status(merr.WrapErrServiceInternalMsg("tasks '%v' not found", req.GetTaskIDs())),
}, nil }, nil
} }
return &workerpb.QueryJobsV2Response{ return &workerpb.QueryJobsV2Response{
@@ -542,7 +543,7 @@ func (node *DataNode) queryAnalyzeTask(ctx context.Context, req *workerpb.QueryJ
log.Debug("query analyze jobs result success", zap.Any("results", results)) log.Debug("query analyze jobs result success", zap.Any("results", results))
if len(results) == 0 { if len(results) == 0 {
return &workerpb.QueryJobsV2Response{ return &workerpb.QueryJobsV2Response{
Status: merr.Status(fmt.Errorf("tasks '%v' not found", req.GetTaskIDs())), Status: merr.Status(merr.WrapErrServiceInternalMsg("tasks '%v' not found", req.GetTaskIDs())),
}, nil }, nil
} }
return &workerpb.QueryJobsV2Response{ return &workerpb.QueryJobsV2Response{
@@ -613,6 +614,6 @@ func (node *DataNode) DropJobsV2(ctx context.Context, req *workerpb.DropJobsV2Re
return merr.Success(), nil return merr.Success(), nil
default: default:
log.Warn("DataNode receive dropping unknown type jobs") log.Warn("DataNode receive dropping unknown type jobs")
return merr.Status(errors.New("DataNode receive dropping unknown type jobs")), nil return merr.Status(merr.WrapErrServiceInternalMsg("DataNode receive dropping unknown type jobs")), nil
} }
} }
+10 -10
View File
@@ -194,11 +194,11 @@ func (node *DataNode) CompactionV2(ctx context.Context, req *datapb.CompactionPl
} }
if req.GetBeginLogID() == 0 { if req.GetBeginLogID() == 0 {
return merr.Status(merr.WrapErrParameterInvalidMsg("invalid beginLogID")), nil return merr.Status(merr.WrapErrServiceInternalMsg("invalid beginLogID")), nil
} }
if req.GetPreAllocatedLogIDs().GetBegin() == 0 || req.GetPreAllocatedLogIDs().GetEnd() == 0 { if req.GetPreAllocatedLogIDs().GetBegin() == 0 || req.GetPreAllocatedLogIDs().GetEnd() == 0 {
return merr.Status(merr.WrapErrParameterInvalidMsg(fmt.Sprintf("invalid beginID %d or invalid endID %d", req.GetPreAllocatedLogIDs().GetBegin(), req.GetPreAllocatedLogIDs().GetEnd()))), nil return merr.Status(merr.WrapErrServiceInternalMsg(fmt.Sprintf("invalid beginID %d or invalid endID %d", req.GetPreAllocatedLogIDs().GetBegin(), req.GetPreAllocatedLogIDs().GetEnd()))), nil
} }
/* /*
@@ -233,7 +233,7 @@ func (node *DataNode) CompactionV2(ctx context.Context, req *datapb.CompactionPl
) )
case datapb.CompactionType_MixCompaction: case datapb.CompactionType_MixCompaction:
if req.GetPreAllocatedSegmentIDs() == nil || req.GetPreAllocatedSegmentIDs().GetBegin() == 0 { if req.GetPreAllocatedSegmentIDs() == nil || req.GetPreAllocatedSegmentIDs().GetBegin() == 0 {
return merr.Status(merr.WrapErrParameterInvalidMsg("invalid pre-allocated segmentID range")), nil return merr.Status(merr.WrapErrServiceInternalMsg("invalid pre-allocated segmentID range")), nil
} }
pk, err := typeutil.GetPrimaryFieldSchema(req.GetSchema()) pk, err := typeutil.GetPrimaryFieldSchema(req.GetSchema())
if err != nil { if err != nil {
@@ -256,7 +256,7 @@ func (node *DataNode) CompactionV2(ctx context.Context, req *datapb.CompactionPl
) )
case datapb.CompactionType_ClusteringCompaction: case datapb.CompactionType_ClusteringCompaction:
if req.GetPreAllocatedSegmentIDs() == nil || req.GetPreAllocatedSegmentIDs().GetBegin() == 0 { if req.GetPreAllocatedSegmentIDs() == nil || req.GetPreAllocatedSegmentIDs().GetBegin() == 0 {
return merr.Status(merr.WrapErrParameterInvalidMsg("invalid pre-allocated segmentID range")), nil return merr.Status(merr.WrapErrServiceInternalMsg("invalid pre-allocated segmentID range")), nil
} }
if namespaceEnabled { if namespaceEnabled {
var sortFields []int64 var sortFields []int64
@@ -281,7 +281,7 @@ func (node *DataNode) CompactionV2(ctx context.Context, req *datapb.CompactionPl
} }
case datapb.CompactionType_SortCompaction: case datapb.CompactionType_SortCompaction:
if req.GetPreAllocatedSegmentIDs() == nil || req.GetPreAllocatedSegmentIDs().GetBegin() == 0 { if req.GetPreAllocatedSegmentIDs() == nil || req.GetPreAllocatedSegmentIDs().GetBegin() == 0 {
return merr.Status(merr.WrapErrParameterInvalidMsg("invalid pre-allocated segmentID range")), nil return merr.Status(merr.WrapErrServiceInternalMsg("invalid pre-allocated segmentID range")), nil
} }
pk, err := typeutil.GetPrimaryFieldSchema(req.GetSchema()) pk, err := typeutil.GetPrimaryFieldSchema(req.GetSchema())
if err != nil { if err != nil {
@@ -306,7 +306,7 @@ func (node *DataNode) CompactionV2(ctx context.Context, req *datapb.CompactionPl
task = compactor.NewBumpSchemaVersionCompactionTask(taskCtx, cm, req, compactionParams) task = compactor.NewBumpSchemaVersionCompactionTask(taskCtx, cm, req, compactionParams)
default: default:
log.Warn("Unknown compaction type", zap.String("type", req.GetType().String())) log.Warn("Unknown compaction type", zap.String("type", req.GetType().String()))
return merr.Status(merr.WrapErrParameterInvalidMsg("Unknown compaction type: %v", req.GetType().String())), nil return merr.Status(merr.WrapErrServiceInternalMsg("Unknown compaction type: %v", req.GetType().String())), nil
} }
succeed, err := node.compactionExecutor.Enqueue(task) succeed, err := node.compactionExecutor.Enqueue(task)
@@ -790,7 +790,7 @@ func (node *DataNode) CreateTask(ctx context.Context, request *workerpb.CreateTa
} }
return node.CopySegment(ctx, req) return node.CopySegment(ctx, req)
default: default:
err := fmt.Errorf("unrecognized task type '%s', properties=%v", taskType, request.GetProperties()) err := merr.WrapErrServiceInternalMsg("unrecognized task type '%s', properties=%v", taskType, request.GetProperties())
log.Ctx(ctx).Warn("CreateTask failed", zap.Error(err)) log.Ctx(ctx).Warn("CreateTask failed", zap.Error(err))
return merr.Status(err), nil return merr.Status(err), nil
} }
@@ -807,7 +807,7 @@ func wrapQueryTaskResult[Resp proto.Message](resp Resp, properties taskcommon.Pr
} }
statusResp, ok := any(resp).(ResponseWithStatus) statusResp, ok := any(resp).(ResponseWithStatus)
if !ok { if !ok {
return &workerpb.QueryTaskResponse{Status: merr.Status(fmt.Errorf("response does not implement GetStatus"))}, nil return &workerpb.QueryTaskResponse{Status: merr.Status(merr.WrapErrServiceInternalMsg("response does not implement GetStatus"))}, nil
} }
return &workerpb.QueryTaskResponse{ return &workerpb.QueryTaskResponse{
Status: statusResp.GetStatus(), Status: statusResp.GetStatus(),
@@ -937,7 +937,7 @@ func (node *DataNode) QueryTask(ctx context.Context, request *workerpb.QueryTask
resProperties.AppendReason(resp.GetReason()) resProperties.AppendReason(resp.GetReason())
return wrapQueryTaskResult(resp, resProperties) return wrapQueryTaskResult(resp, resProperties)
default: default:
err := fmt.Errorf("unrecognized task type '%s', properties=%v", taskType, request.GetProperties()) err := merr.WrapErrServiceInternalMsg("unrecognized task type '%s', properties=%v", taskType, request.GetProperties())
log.Ctx(ctx).Warn("QueryTask failed", zap.Error(err)) log.Ctx(ctx).Warn("QueryTask failed", zap.Error(err))
return &workerpb.QueryTaskResponse{ return &workerpb.QueryTaskResponse{
Status: merr.Status(err), Status: merr.Status(err),
@@ -997,7 +997,7 @@ func (node *DataNode) DropTask(ctx context.Context, request *workerpb.DropTaskRe
zap.String("clusterID", clusterID)) zap.String("clusterID", clusterID))
return merr.Success(), nil return merr.Success(), nil
default: default:
err := fmt.Errorf("unrecognized task type '%s', properties=%v", taskType, request.GetProperties()) err := merr.WrapErrServiceInternalMsg("unrecognized task type '%s', properties=%v", taskType, request.GetProperties())
log.Ctx(ctx).Warn("DropTask failed", zap.Error(err)) log.Ctx(ctx).Warn("DropTask failed", zap.Error(err))
return merr.Status(err), nil return merr.Status(err), nil
} }
+12 -3
View File
@@ -594,7 +594,10 @@ func (s *DataNodeServicesSuite) TestCreateTask() {
} }
status, err := s.node.CreateTask(s.ctx, req) status, err := s.node.CreateTask(s.ctx, req)
s.NoError(err) s.NoError(err)
s.Equal(commonpb.ErrorCode_UnexpectedError, status.GetErrorCode()) // taskcommon.GetTaskType classifies an unrecognized task type as
// ServiceInternal: task types are coordinator-assigned, so a mismatch is
// an internal protocol violation, not user input.
s.Equal(merr.Code(merr.ErrServiceInternal), status.GetCode())
}) })
} }
@@ -685,7 +688,10 @@ func (s *DataNodeServicesSuite) TestQueryTask() {
} }
resp, err := s.node.QueryTask(s.ctx, req) resp, err := s.node.QueryTask(s.ctx, req)
s.NoError(err) s.NoError(err)
s.Equal(commonpb.ErrorCode_UnexpectedError, resp.GetStatus().GetErrorCode()) // taskcommon.GetTaskType classifies an unrecognized task type as
// ServiceInternal: task types are coordinator-assigned, so a mismatch is
// an internal protocol violation, not user input.
s.Equal(merr.Code(merr.ErrServiceInternal), resp.GetStatus().GetCode())
}) })
} }
@@ -771,7 +777,10 @@ func (s *DataNodeServicesSuite) TestDropTask() {
} }
status, err := s.node.DropTask(s.ctx, req) status, err := s.node.DropTask(s.ctx, req)
s.NoError(err) s.NoError(err)
s.Equal(commonpb.ErrorCode_UnexpectedError, status.GetErrorCode()) // taskcommon.GetTaskType classifies an unrecognized task type as
// ServiceInternal: task types are coordinator-assigned, so a mismatch is
// an internal protocol violation, not user input.
s.Equal(merr.Code(merr.ErrServiceInternal), status.GetCode())
}) })
} }
@@ -20,7 +20,6 @@ import (
"context" "context"
"fmt" "fmt"
"github.com/cockroachdb/errors"
"go.uber.org/zap" "go.uber.org/zap"
"google.golang.org/grpc" "google.golang.org/grpc"
@@ -36,6 +35,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/proto/workerpb" "github.com/milvus-io/milvus/pkg/v3/proto/workerpb"
"github.com/milvus-io/milvus/pkg/v3/util/commonpbutil" "github.com/milvus-io/milvus/pkg/v3/util/commonpbutil"
"github.com/milvus-io/milvus/pkg/v3/util/funcutil" "github.com/milvus-io/milvus/pkg/v3/util/funcutil"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/paramtable" "github.com/milvus-io/milvus/pkg/v3/util/paramtable"
"github.com/milvus-io/milvus/pkg/v3/util/typeutil" "github.com/milvus-io/milvus/pkg/v3/util/typeutil"
) )
@@ -57,11 +57,11 @@ type Client struct {
// NewClient creates a client for DataNode. // NewClient creates a client for DataNode.
func NewClient(ctx context.Context, addr string, serverID int64, encryption bool) (types.DataNodeClient, error) { func NewClient(ctx context.Context, addr string, serverID int64, encryption bool) (types.DataNodeClient, error) {
if addr == "" { if addr == "" {
return nil, errors.New("address is empty") return nil, merr.WrapErrParameterInvalidMsg("address is empty")
} }
sess := sessionutil.NewSession(context.Background()) sess := sessionutil.NewSession(context.Background())
if sess == nil { if sess == nil {
err := errors.New("new session error, maybe can not connect to etcd") err := merr.WrapErrServiceUnavailable("new session error, maybe can not connect to etcd")
log.Ctx(ctx).Debug("DataNodeClient New Etcd Session failed", zap.Error(err)) log.Ctx(ctx).Debug("DataNodeClient New Etcd Session failed", zap.Error(err))
return nil, err return nil, err
} }
@@ -69,7 +69,7 @@ type Client struct {
func NewClient(ctx context.Context) (types.MixCoordClient, error) { func NewClient(ctx context.Context) (types.MixCoordClient, error) {
sess := sessionutil.NewSession(context.Background()) sess := sessionutil.NewSession(context.Background())
if sess == nil { if sess == nil {
err := errors.New("new session error, maybe can not connect to etcd") err := merr.WrapErrServiceUnavailable("new session error, maybe can not connect to etcd")
log.Ctx(ctx).Debug("New MixCoord Client failed", zap.Error(err)) log.Ctx(ctx).Debug("New MixCoord Client failed", zap.Error(err))
return nil, err return nil, err
} }
@@ -120,7 +120,7 @@ func (c *Client) getMixCoordAddr() (string, error) {
return c.getCompatibleMixCoordAddr() return c.getCompatibleMixCoordAddr()
} else { } else {
log.Warn("MixCoordClient mess key not exist", zap.Any("key", key)) log.Warn("MixCoordClient mess key not exist", zap.Any("key", key))
return "", errors.New("find no available mixcoord, check mixcoord state") return "", merr.WrapErrNodeNotFound(0, "find no available mixcoord, check mixcoord state")
} }
} }
log.Debug("MixCoordClient GetSessions success", log.Debug("MixCoordClient GetSessions success",
@@ -137,12 +137,12 @@ func (c *Client) getCompatibleMixCoordAddr() (string, error) {
msess, _, err := c.sess.GetSessions(c.ctx, typeutil.RootCoordRole) msess, _, err := c.sess.GetSessions(c.ctx, typeutil.RootCoordRole)
if err != nil { if err != nil {
log.Debug("mixCoordClient getSessions failed", zap.Any("key", typeutil.RootCoordRole), zap.Error(err)) log.Debug("mixCoordClient getSessions failed", zap.Any("key", typeutil.RootCoordRole), zap.Error(err))
return "", errors.New("find no available mixcoord, check mixcoord state") return "", merr.WrapErrNodeNotFound(0, "find no available mixcoord, check mixcoord state")
} }
ms, ok := msess[typeutil.RootCoordRole] ms, ok := msess[typeutil.RootCoordRole]
if !ok { if !ok {
log.Warn("MixCoordClient mess key not exist", zap.Any("key", typeutil.RootCoordRole)) log.Warn("MixCoordClient mess key not exist", zap.Any("key", typeutil.RootCoordRole))
return "", errors.New("find no available mixcoord, check mixcoord state") return "", merr.WrapErrNodeNotFound(0, "find no available mixcoord, check mixcoord state")
} }
log.Debug("MixCoordClient GetSessions use rootCoord", zap.Any("key", typeutil.RootCoordRole)) log.Debug("MixCoordClient GetSessions use rootCoord", zap.Any("key", typeutil.RootCoordRole))
c.grpcClient.SetNodeID(ms.ServerID) c.grpcClient.SetNodeID(ms.ServerID)
+3 -3
View File
@@ -20,7 +20,6 @@ import (
"context" "context"
"fmt" "fmt"
"github.com/cockroachdb/errors"
"go.uber.org/zap" "go.uber.org/zap"
"google.golang.org/grpc" "google.golang.org/grpc"
@@ -35,6 +34,7 @@ import (
"github.com/milvus-io/milvus/pkg/v3/proto/proxypb" "github.com/milvus-io/milvus/pkg/v3/proto/proxypb"
"github.com/milvus-io/milvus/pkg/v3/util/commonpbutil" "github.com/milvus-io/milvus/pkg/v3/util/commonpbutil"
"github.com/milvus-io/milvus/pkg/v3/util/funcutil" "github.com/milvus-io/milvus/pkg/v3/util/funcutil"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/paramtable" "github.com/milvus-io/milvus/pkg/v3/util/paramtable"
"github.com/milvus-io/milvus/pkg/v3/util/typeutil" "github.com/milvus-io/milvus/pkg/v3/util/typeutil"
) )
@@ -50,11 +50,11 @@ type Client struct {
// NewClient creates a new client instance // NewClient creates a new client instance
func NewClient(ctx context.Context, addr string, nodeID int64) (types.ProxyClient, error) { func NewClient(ctx context.Context, addr string, nodeID int64) (types.ProxyClient, error) {
if addr == "" { if addr == "" {
return nil, errors.New("address is empty") return nil, merr.WrapErrParameterInvalidMsg("address is empty")
} }
sess := sessionutil.NewSession(context.Background()) sess := sessionutil.NewSession(context.Background())
if sess == nil { if sess == nil {
err := errors.New("new session error, maybe can not connect to etcd") err := merr.WrapErrServiceUnavailable("new session error, maybe can not connect to etcd")
log.Ctx(ctx).Debug("Proxy client new session failed", zap.Error(err)) log.Ctx(ctx).Debug("Proxy client new session failed", zap.Error(err))
return nil, err return nil, err
} }
@@ -17,13 +17,12 @@
package httpserver package httpserver
import ( import (
"fmt"
"github.com/gin-gonic/gin" "github.com/gin-gonic/gin"
"google.golang.org/protobuf/proto" "google.golang.org/protobuf/proto"
"github.com/milvus-io/milvus-proto/go-api/v3/milvuspb" "github.com/milvus-io/milvus-proto/go-api/v3/milvuspb"
"github.com/milvus-io/milvus/internal/types" "github.com/milvus-io/milvus/internal/types"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
) )
// Handlers handles http requests // Handlers handles http requests
@@ -107,7 +106,7 @@ func (h *Handlers) handleDummy(c *gin.Context) (interface{}, error) {
// use ShouldBind to supports binding JSON, XML, YAML, and protobuf. // use ShouldBind to supports binding JSON, XML, YAML, and protobuf.
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.Dummy(c, &req) return h.proxy.Dummy(c, &req)
} }
@@ -116,11 +115,11 @@ func (h *Handlers) handleCreateCollection(c *gin.Context) (interface{}, error) {
wrappedReq := WrappedCreateCollectionRequest{} wrappedReq := WrappedCreateCollectionRequest{}
err := shouldBind(c, &wrappedReq) err := shouldBind(c, &wrappedReq)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
schemaProto, err := proto.Marshal(&wrappedReq.Schema) schemaProto, err := proto.Marshal(&wrappedReq.Schema)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: marshal schema failed: %v", errBadRequest, err) return nil, badRequestf(err, "marshal schema failed")
} }
req := &milvuspb.CreateCollectionRequest{ req := &milvuspb.CreateCollectionRequest{
Base: wrappedReq.Base, Base: wrappedReq.Base,
@@ -138,7 +137,7 @@ func (h *Handlers) handleDropCollection(c *gin.Context) (interface{}, error) {
req := milvuspb.DropCollectionRequest{} req := milvuspb.DropCollectionRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.DropCollection(c, &req) return h.proxy.DropCollection(c, &req)
} }
@@ -147,7 +146,7 @@ func (h *Handlers) handleHasCollection(c *gin.Context) (interface{}, error) {
req := milvuspb.HasCollectionRequest{} req := milvuspb.HasCollectionRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.HasCollection(c, &req) return h.proxy.HasCollection(c, &req)
} }
@@ -156,7 +155,7 @@ func (h *Handlers) handleDescribeCollection(c *gin.Context) (interface{}, error)
req := milvuspb.DescribeCollectionRequest{} req := milvuspb.DescribeCollectionRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.DescribeCollection(c, &req) return h.proxy.DescribeCollection(c, &req)
} }
@@ -165,7 +164,7 @@ func (h *Handlers) handleLoadCollection(c *gin.Context) (interface{}, error) {
req := milvuspb.LoadCollectionRequest{} req := milvuspb.LoadCollectionRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.LoadCollection(c, &req) return h.proxy.LoadCollection(c, &req)
} }
@@ -174,7 +173,7 @@ func (h *Handlers) handleReleaseCollection(c *gin.Context) (interface{}, error)
req := milvuspb.ReleaseCollectionRequest{} req := milvuspb.ReleaseCollectionRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.ReleaseCollection(c, &req) return h.proxy.ReleaseCollection(c, &req)
} }
@@ -183,7 +182,7 @@ func (h *Handlers) handleGetCollectionStatistics(c *gin.Context) (interface{}, e
req := milvuspb.GetCollectionStatisticsRequest{} req := milvuspb.GetCollectionStatisticsRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.GetCollectionStatistics(c, &req) return h.proxy.GetCollectionStatistics(c, &req)
} }
@@ -192,7 +191,7 @@ func (h *Handlers) handleShowCollections(c *gin.Context) (interface{}, error) {
req := milvuspb.ShowCollectionsRequest{} req := milvuspb.ShowCollectionsRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.ShowCollections(c, &req) return h.proxy.ShowCollections(c, &req)
} }
@@ -201,7 +200,7 @@ func (h *Handlers) handleCreatePartition(c *gin.Context) (interface{}, error) {
req := milvuspb.CreatePartitionRequest{} req := milvuspb.CreatePartitionRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.CreatePartition(c, &req) return h.proxy.CreatePartition(c, &req)
} }
@@ -210,7 +209,7 @@ func (h *Handlers) handleDropPartition(c *gin.Context) (interface{}, error) {
req := milvuspb.DropPartitionRequest{} req := milvuspb.DropPartitionRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.DropPartition(c, &req) return h.proxy.DropPartition(c, &req)
} }
@@ -219,7 +218,7 @@ func (h *Handlers) handleHasPartition(c *gin.Context) (interface{}, error) {
req := milvuspb.HasPartitionRequest{} req := milvuspb.HasPartitionRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.HasPartition(c, &req) return h.proxy.HasPartition(c, &req)
} }
@@ -228,7 +227,7 @@ func (h *Handlers) handleLoadPartitions(c *gin.Context) (interface{}, error) {
req := milvuspb.LoadPartitionsRequest{} req := milvuspb.LoadPartitionsRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.LoadPartitions(c, &req) return h.proxy.LoadPartitions(c, &req)
} }
@@ -237,7 +236,7 @@ func (h *Handlers) handleReleasePartitions(c *gin.Context) (interface{}, error)
req := milvuspb.ReleasePartitionsRequest{} req := milvuspb.ReleasePartitionsRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.ReleasePartitions(c, &req) return h.proxy.ReleasePartitions(c, &req)
} }
@@ -246,7 +245,7 @@ func (h *Handlers) handleGetPartitionStatistics(c *gin.Context) (interface{}, er
req := milvuspb.GetPartitionStatisticsRequest{} req := milvuspb.GetPartitionStatisticsRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.GetPartitionStatistics(c, &req) return h.proxy.GetPartitionStatistics(c, &req)
} }
@@ -255,7 +254,7 @@ func (h *Handlers) handleShowPartitions(c *gin.Context) (interface{}, error) {
req := milvuspb.ShowPartitionsRequest{} req := milvuspb.ShowPartitionsRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.ShowPartitions(c, &req) return h.proxy.ShowPartitions(c, &req)
} }
@@ -264,7 +263,7 @@ func (h *Handlers) handleCreateAlias(c *gin.Context) (interface{}, error) {
req := milvuspb.CreateAliasRequest{} req := milvuspb.CreateAliasRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.CreateAlias(c, &req) return h.proxy.CreateAlias(c, &req)
} }
@@ -273,7 +272,7 @@ func (h *Handlers) handleDropAlias(c *gin.Context) (interface{}, error) {
req := milvuspb.DropAliasRequest{} req := milvuspb.DropAliasRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.DropAlias(c, &req) return h.proxy.DropAlias(c, &req)
} }
@@ -282,7 +281,7 @@ func (h *Handlers) handleAlterAlias(c *gin.Context) (interface{}, error) {
req := milvuspb.AlterAliasRequest{} req := milvuspb.AlterAliasRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.AlterAlias(c, &req) return h.proxy.AlterAlias(c, &req)
} }
@@ -291,7 +290,7 @@ func (h *Handlers) handleCreateIndex(c *gin.Context) (interface{}, error) {
req := milvuspb.CreateIndexRequest{} req := milvuspb.CreateIndexRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.CreateIndex(c, &req) return h.proxy.CreateIndex(c, &req)
} }
@@ -300,7 +299,7 @@ func (h *Handlers) handleDescribeIndex(c *gin.Context) (interface{}, error) {
req := milvuspb.DescribeIndexRequest{} req := milvuspb.DescribeIndexRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.DescribeIndex(c, &req) return h.proxy.DescribeIndex(c, &req)
} }
@@ -309,7 +308,7 @@ func (h *Handlers) handleGetIndexState(c *gin.Context) (interface{}, error) {
req := milvuspb.GetIndexStateRequest{} req := milvuspb.GetIndexStateRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.GetIndexState(c, &req) return h.proxy.GetIndexState(c, &req)
} }
@@ -318,7 +317,7 @@ func (h *Handlers) handleGetIndexBuildProgress(c *gin.Context) (interface{}, err
req := milvuspb.GetIndexBuildProgressRequest{} req := milvuspb.GetIndexBuildProgressRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.GetIndexBuildProgress(c, &req) return h.proxy.GetIndexBuildProgress(c, &req)
} }
@@ -327,7 +326,7 @@ func (h *Handlers) handleDropIndex(c *gin.Context) (interface{}, error) {
req := milvuspb.DropIndexRequest{} req := milvuspb.DropIndexRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.DropIndex(c, &req) return h.proxy.DropIndex(c, &req)
} }
@@ -336,11 +335,11 @@ func (h *Handlers) handleInsert(c *gin.Context) (interface{}, error) {
wrappedReq := WrappedInsertRequest{} wrappedReq := WrappedInsertRequest{}
err := shouldBind(c, &wrappedReq) err := shouldBind(c, &wrappedReq)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
req, err := wrappedReq.AsInsertRequest() req, err := wrappedReq.AsInsertRequest()
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: convert body to pb failed: %v", errBadRequest, err) return nil, badRequestf(err, "convert body to pb failed")
} }
return h.proxy.Insert(c, req) return h.proxy.Insert(c, req)
} }
@@ -349,7 +348,7 @@ func (h *Handlers) handleDelete(c *gin.Context) (interface{}, error) {
req := milvuspb.DeleteRequest{} req := milvuspb.DeleteRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.Delete(c, &req) return h.proxy.Delete(c, &req)
} }
@@ -358,10 +357,10 @@ func (h *Handlers) handleSearch(c *gin.Context) (interface{}, error) {
wrappedReq := SearchRequest{} wrappedReq := SearchRequest{}
err := shouldBind(c, &wrappedReq) err := shouldBind(c, &wrappedReq)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
if wrappedReq.HasSearchAggregation() { if wrappedReq.HasSearchAggregation() {
return nil, fmt.Errorf("%w: searchAggregation is not supported for low-level REST search", errBadRequest) return nil, badRequestf(merr.WrapErrParameterInvalidMsg("searchAggregation is not supported for low-level REST search"), "invalid request")
} }
req := milvuspb.SearchRequest{ req := milvuspb.SearchRequest{
Base: wrappedReq.Base, Base: wrappedReq.Base,
@@ -392,7 +391,7 @@ func (h *Handlers) handleQuery(c *gin.Context) (interface{}, error) {
req := milvuspb.QueryRequest{} req := milvuspb.QueryRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.Query(c, &req) return h.proxy.Query(c, &req)
} }
@@ -401,7 +400,7 @@ func (h *Handlers) handleFlush(c *gin.Context) (interface{}, error) {
req := milvuspb.FlushRequest{} req := milvuspb.FlushRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.Flush(c, &req) return h.proxy.Flush(c, &req)
} }
@@ -410,7 +409,7 @@ func (h *Handlers) handleCalcDistance(c *gin.Context) (interface{}, error) {
wrappedReq := WrappedCalcDistanceRequest{} wrappedReq := WrappedCalcDistanceRequest{}
err := shouldBind(c, &wrappedReq) err := shouldBind(c, &wrappedReq)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
req := milvuspb.CalcDistanceRequest{ req := milvuspb.CalcDistanceRequest{
@@ -426,7 +425,7 @@ func (h *Handlers) handleGetFlushState(c *gin.Context) (interface{}, error) {
req := milvuspb.GetFlushStateRequest{} req := milvuspb.GetFlushStateRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.GetFlushState(c, &req) return h.proxy.GetFlushState(c, &req)
} }
@@ -435,7 +434,7 @@ func (h *Handlers) handleGetPersistentSegmentInfo(c *gin.Context) (interface{},
req := milvuspb.GetPersistentSegmentInfoRequest{} req := milvuspb.GetPersistentSegmentInfoRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.GetPersistentSegmentInfo(c, &req) return h.proxy.GetPersistentSegmentInfo(c, &req)
} }
@@ -444,7 +443,7 @@ func (h *Handlers) handleGetQuerySegmentInfo(c *gin.Context) (interface{}, error
req := milvuspb.GetQuerySegmentInfoRequest{} req := milvuspb.GetQuerySegmentInfoRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.GetQuerySegmentInfo(c, &req) return h.proxy.GetQuerySegmentInfo(c, &req)
} }
@@ -453,7 +452,7 @@ func (h *Handlers) handleGetReplicas(c *gin.Context) (interface{}, error) {
req := milvuspb.GetReplicasRequest{} req := milvuspb.GetReplicasRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.GetReplicas(c, &req) return h.proxy.GetReplicas(c, &req)
} }
@@ -462,7 +461,7 @@ func (h *Handlers) handleGetMetrics(c *gin.Context) (interface{}, error) {
req := milvuspb.GetMetricsRequest{} req := milvuspb.GetMetricsRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.GetMetrics(c, &req) return h.proxy.GetMetrics(c, &req)
} }
@@ -471,7 +470,7 @@ func (h *Handlers) handleLoadBalance(c *gin.Context) (interface{}, error) {
req := milvuspb.LoadBalanceRequest{} req := milvuspb.LoadBalanceRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.LoadBalance(c, &req) return h.proxy.LoadBalance(c, &req)
} }
@@ -480,7 +479,7 @@ func (h *Handlers) handleGetCompactionState(c *gin.Context) (interface{}, error)
req := milvuspb.GetCompactionStateRequest{} req := milvuspb.GetCompactionStateRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.GetCompactionState(c, &req) return h.proxy.GetCompactionState(c, &req)
} }
@@ -489,7 +488,7 @@ func (h *Handlers) handleGetCompactionStateWithPlans(c *gin.Context) (interface{
req := milvuspb.GetCompactionPlansRequest{} req := milvuspb.GetCompactionPlansRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.GetCompactionStateWithPlans(c, &req) return h.proxy.GetCompactionStateWithPlans(c, &req)
} }
@@ -498,7 +497,7 @@ func (h *Handlers) handleManualCompaction(c *gin.Context) (interface{}, error) {
req := milvuspb.ManualCompactionRequest{} req := milvuspb.ManualCompactionRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.ManualCompaction(c, &req) return h.proxy.ManualCompaction(c, &req)
} }
@@ -507,7 +506,7 @@ func (h *Handlers) handleImport(c *gin.Context) (interface{}, error) {
req := milvuspb.ImportRequest{} req := milvuspb.ImportRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.Import(c, &req) return h.proxy.Import(c, &req)
} }
@@ -516,7 +515,7 @@ func (h *Handlers) handleGetImportState(c *gin.Context) (interface{}, error) {
req := milvuspb.GetImportStateRequest{} req := milvuspb.GetImportStateRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.GetImportState(c, &req) return h.proxy.GetImportState(c, &req)
} }
@@ -525,7 +524,7 @@ func (h *Handlers) handleListImportTasks(c *gin.Context) (interface{}, error) {
req := milvuspb.ListImportTasksRequest{} req := milvuspb.ListImportTasksRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.ListImportTasks(c, &req) return h.proxy.ListImportTasks(c, &req)
} }
@@ -534,7 +533,7 @@ func (h *Handlers) handleCreateCredential(c *gin.Context) (interface{}, error) {
req := milvuspb.CreateCredentialRequest{} req := milvuspb.CreateCredentialRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.CreateCredential(c, &req) return h.proxy.CreateCredential(c, &req)
} }
@@ -543,7 +542,7 @@ func (h *Handlers) handleUpdateCredential(c *gin.Context) (interface{}, error) {
req := milvuspb.UpdateCredentialRequest{} req := milvuspb.UpdateCredentialRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.UpdateCredential(c, &req) return h.proxy.UpdateCredential(c, &req)
} }
@@ -552,7 +551,7 @@ func (h *Handlers) handleDeleteCredential(c *gin.Context) (interface{}, error) {
req := milvuspb.DeleteCredentialRequest{} req := milvuspb.DeleteCredentialRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.DeleteCredential(c, &req) return h.proxy.DeleteCredential(c, &req)
} }
@@ -561,7 +560,7 @@ func (h *Handlers) handleListCredUsers(c *gin.Context) (interface{}, error) {
req := milvuspb.ListCredUsersRequest{} req := milvuspb.ListCredUsersRequest{}
err := shouldBind(c, &req) err := shouldBind(c, &req)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err) return nil, badRequestf(err, "parse body failed")
} }
return h.proxy.ListCredUsers(c, &req) return h.proxy.ListCredUsers(c, &req)
} }
@@ -1434,7 +1434,7 @@ func TestFp16Bf16VectorsV1(t *testing.T) {
assert.Nil(t, err, "case %d: ", i) assert.Nil(t, err, "case %d: ", i)
assert.Equal(t, testcase.errCode, returnBody.Code, "case %d: ", i, string(testcase.requestBody)) assert.Equal(t, testcase.errCode, returnBody.Code, "case %d: ", i, string(testcase.requestBody))
if testcase.errCode != 0 { if testcase.errCode != 0 {
assert.Equal(t, testcase.errMsg, returnBody.Message, "case %d: ", i, string(testcase.requestBody)) assert.Contains(t, returnBody.Message, testcase.errMsg, "case %d: ", i, string(testcase.requestBody))
} }
fmt.Println(w.Body.String()) fmt.Println(w.Body.String())
}) })
@@ -410,6 +410,35 @@ func wrapperPost(newReq newReqFunc, v2 handlerFuncV2) gin.HandlerFunc {
dbName, dbName,
collectionName, collectionName,
).Inc() ).Inc()
// Mirror the fail_input/fail_system metric split into the logs so a
// failed REST request can be filtered by error_type. System failures are
// logged at Warn (actionable); input failures at Info (expected user
// mistakes — keeping them at Warn would spam the logs).
if label == metrics.FailSystemLabel || label == metrics.FailInputLabel {
var status *commonpb.Status
switch r := resp.(type) {
case interface{ GetStatus() *commonpb.Status }:
status = r.GetStatus()
case *commonpb.Status:
status = r
}
errType := merr.SystemError
if label == metrics.FailInputLabel {
errType = merr.InputError
}
logger := log.Ctx(ctx).With(
zap.String("method", methodTag),
zap.String("error_type", errType.String()),
zap.Int32("code", status.GetCode()),
zap.String("reason", status.GetReason()),
)
if errType == merr.InputError {
logger.Info("restful request returned an input error")
} else {
logger.Warn("restful request returned a system error")
}
}
} }
} }
@@ -543,6 +572,9 @@ func wrapperProxyWithLimit(ctx context.Context, ginCtx *gin.Context, req any, ch
} }
if err != nil { if err != nil {
// Expose the exact classification (incl. boundary InputError marks) to
// the REST access log; key must match accesslog/info.ContextErrorType.
ginCtx.Set("error_type", merr.GetErrorType(err).String())
log.Ctx(ctx).Warn("high level restful api, grpc call failed", zap.Error(err)) log.Ctx(ctx).Warn("high level restful api, grpc call failed", zap.Error(err))
if !ignoreErr { if !ignoreErr {
HTTPAbortReturn(ginCtx, http.StatusOK, gin.H{HTTPReturnCode: merr.Code(err), HTTPReturnMessage: err.Error()}) HTTPAbortReturn(ginCtx, http.StatusOK, gin.H{HTTPReturnCode: merr.Code(err), HTTPReturnMessage: err.Error()})
@@ -1025,7 +1057,7 @@ func (h *HandlersV2) waitForFlush(ctx context.Context, dbName string, collection
segmentIDs := flushResp.GetCollSegIDs()[collectionName].GetData() segmentIDs := flushResp.GetCollSegIDs()[collectionName].GetData()
flushTs, ok := flushResp.GetCollFlushTs()[collectionName] flushTs, ok := flushResp.GetCollFlushTs()[collectionName]
if !ok { if !ok {
return merr.WrapErrServiceInternal(fmt.Sprintf("failed to get flush timestamp for collection %s", collectionName)) return merr.WrapErrServiceInternalMsg("failed to get flush timestamp for collection %s", collectionName)
} }
ticker := time.NewTicker(500 * time.Millisecond) ticker := time.NewTicker(500 * time.Millisecond)
@@ -1521,7 +1553,7 @@ func generatePlaceholderGroup(ctx context.Context, body string, collSchema *sche
fieldName = field.Name fieldName = field.Name
vectorField = field vectorField = field
} else { } else {
return nil, errors.New("search without annsField, but already found multiple vector fields: [" + fieldName + ", " + field.Name + ",,,]") return nil, merr.WrapErrParameterInvalidMsg("search without annsField, but already found multiple vector fields: [%s, %s,,,]", fieldName, field.Name)
} }
} }
} }
@@ -1547,7 +1579,7 @@ func generatePlaceholderGroup(ctx context.Context, body string, collSchema *sche
} }
} }
if vectorField == nil { if vectorField == nil {
return nil, errors.New("cannot find a vector field named: " + fieldName) return nil, merr.WrapErrFieldNotFound(fieldName, "cannot find a vector field")
} }
dim := int64(0) dim := int64(0)
if !typeutil.IsSparseFloatVectorType(vectorField.DataType) { if !typeutil.IsSparseFloatVectorType(vectorField.DataType) {
@@ -3352,13 +3352,13 @@ func TestSearchV2(t *testing.T) {
queryTestCases = append(queryTestCases, requestBodyTestCase{ queryTestCases = append(queryTestCases, requestBodyTestCase{
path: SearchAction, path: SearchAction,
requestBody: []byte(`{"collectionName": "book", "data": [[0.1, 0.2]], "annsField": "word_count", "filter": "book_id in [2, 4, 6, 8]", "limit": 4, "outputFields": ["word_count"], "params": {"radius":0.9, "range_filter": 0.1}, "groupingField": "test"}`), requestBody: []byte(`{"collectionName": "book", "data": [[0.1, 0.2]], "annsField": "word_count", "filter": "book_id in [2, 4, 6, 8]", "limit": 4, "outputFields": ["word_count"], "params": {"radius":0.9, "range_filter": 0.1}, "groupingField": "test"}`),
errMsg: "can only accept json format request, error: cannot find a vector field named: word_count", errMsg: "cannot find a vector field",
errCode: 1801, errCode: 1801,
}) })
queryTestCases = append(queryTestCases, requestBodyTestCase{ queryTestCases = append(queryTestCases, requestBodyTestCase{
path: AdvancedSearchAction, path: AdvancedSearchAction,
requestBody: []byte(`{"collectionName": "hello_milvus", "search": [{"data": [[0.1, 0.2]], "annsField": "float_vector1", "metricType": "L2", "limit": 3}, {"data": [[0.1, 0.2]], "annsField": "float_vector2", "metricType": "L2", "limit": 3}], "rerank": {"strategy": "rrf", "params": {"k": 1}}}`), requestBody: []byte(`{"collectionName": "hello_milvus", "search": [{"data": [[0.1, 0.2]], "annsField": "float_vector1", "metricType": "L2", "limit": 3}, {"data": [[0.1, 0.2]], "annsField": "float_vector2", "metricType": "L2", "limit": 3}], "rerank": {"strategy": "rrf", "params": {"k": 1}}}`),
errMsg: "can only accept json format request, error: cannot find a vector field named: float_vector1", errMsg: "cannot find a vector field",
errCode: 1801, errCode: 1801,
}) })
// multiple annsFields // multiple annsFields
@@ -27,7 +27,6 @@ import (
"sync" "sync"
"time" "time"
"github.com/cockroachdb/errors"
"github.com/gin-gonic/gin" "github.com/gin-gonic/gin"
"go.uber.org/zap" "go.uber.org/zap"
@@ -93,7 +92,7 @@ func (w *timeoutResponseRecorder) Write(data []byte) (int, error) {
w.mu.Lock() w.mu.Lock()
defer w.mu.Unlock() defer w.mu.Unlock()
if w.closed || w.body == nil { if w.closed || w.body == nil {
return 0, errors.New("response writer closed") return 0, merr.WrapErrServiceInternalMsg("response writer closed")
} }
if !w.written() { if !w.written() {
w.size = 0 w.size = 0
@@ -107,7 +106,7 @@ func (w *timeoutResponseRecorder) WriteString(s string) (int, error) {
w.mu.Lock() w.mu.Lock()
defer w.mu.Unlock() defer w.mu.Unlock()
if w.closed || w.body == nil { if w.closed || w.body == nil {
return 0, errors.New("response writer closed") return 0, merr.WrapErrServiceInternalMsg("response writer closed")
} }
if !w.written() { if !w.written() {
w.size = 0 w.size = 0
@@ -161,7 +160,7 @@ func (w *timeoutResponseRecorder) Written() bool {
func (w *timeoutResponseRecorder) Flush() {} func (w *timeoutResponseRecorder) Flush() {}
func (w *timeoutResponseRecorder) Hijack() (net.Conn, *bufio.ReadWriter, error) { func (w *timeoutResponseRecorder) Hijack() (net.Conn, *bufio.ReadWriter, error) {
return nil, nil, errors.New("response writer does not support hijack") return nil, nil, merr.WrapErrServiceInternalMsg("response writer does not support hijack")
} }
func (w *timeoutResponseRecorder) CloseNotify() <-chan bool { func (w *timeoutResponseRecorder) CloseNotify() <-chan bool {
@@ -176,7 +175,7 @@ func (w *timeoutResponseRecorder) CommitTo(realWriter gin.ResponseWriter) error
w.mu.Lock() w.mu.Lock()
defer w.mu.Unlock() defer w.mu.Unlock()
if w.closed || w.body == nil { if w.closed || w.body == nil {
return errors.New("response writer closed") return merr.WrapErrServiceInternalMsg("response writer closed")
} }
dst := realWriter.Header() dst := realWriter.Header()
+75 -74
View File
@@ -27,7 +27,6 @@ import (
"strings" "strings"
"time" "time"
"github.com/cockroachdb/errors"
"github.com/gin-gonic/gin" "github.com/gin-gonic/gin"
"github.com/spf13/cast" "github.com/spf13/cast"
"github.com/tidwall/gjson" "github.com/tidwall/gjson"
@@ -239,7 +238,7 @@ func convertRange(field *schemapb.FieldSchema, result gjson.Result) (string, err
func checkGetPrimaryKey(coll *schemapb.CollectionSchema, idResult gjson.Result) (string, error) { func checkGetPrimaryKey(coll *schemapb.CollectionSchema, idResult gjson.Result) (string, error) {
primaryField, ok := getPrimaryField(coll) primaryField, ok := getPrimaryField(coll)
if !ok { if !ok {
return "", fmt.Errorf("collection: %s has no primary field", coll.Name) return "", merr.WrapErrParameterInvalidMsg("collection: %s has no primary field", coll.Name)
} }
resultStr, err := convertRange(primaryField, idResult) resultStr, err := convertRange(primaryField, idResult)
if err != nil { if err != nil {
@@ -253,7 +252,7 @@ func checkGetPrimaryKey(coll *schemapb.CollectionSchema, idResult gjson.Result)
// based on the primary key field type // based on the primary key field type
func convertIDsToSchemapbIDs(ids []interface{}, pkField *schemapb.FieldSchema) (*schemapb.IDs, error) { func convertIDsToSchemapbIDs(ids []interface{}, pkField *schemapb.FieldSchema) (*schemapb.IDs, error) {
if len(ids) == 0 { if len(ids) == 0 {
return nil, errors.New("ids array cannot be empty") return nil, merr.WrapErrParameterMissingMsg("ids array cannot be empty")
} }
switch pkField.DataType { switch pkField.DataType {
@@ -270,18 +269,18 @@ func convertIDsToSchemapbIDs(ids []interface{}, pkField *schemapb.FieldSchema) (
// JSON numbers are decoded as float64 // JSON numbers are decoded as float64
// Check if the float has a fractional part // Check if the float has a fractional part
if v != math.Trunc(v) { if v != math.Trunc(v) {
return nil, fmt.Errorf("invalid int64 id at index %d: %v has fractional part", i, v) return nil, merr.WrapErrParameterInvalidMsg("invalid int64 id at index %d: %v has fractional part", i, v)
} }
int64ID = int64(v) int64ID = int64(v)
case string: case string:
// Try to parse string as int64 // Try to parse string as int64
parsed, err := strconv.ParseInt(v, 10, 64) parsed, err := strconv.ParseInt(v, 10, 64)
if err != nil { if err != nil {
return nil, fmt.Errorf("invalid int64 id at index %d: %v, error: %v", i, id, err) return nil, merr.WrapErrParameterInvalidErr(err, "invalid int64 id at index %d: %v", i, id)
} }
int64ID = parsed int64ID = parsed
default: default:
return nil, fmt.Errorf("invalid id type at index %d: expected int64, got %T", i, id) return nil, merr.WrapErrParameterInvalidMsg("invalid id type at index %d: expected int64, got %T", i, id)
} }
int64IDs = append(int64IDs, int64ID) int64IDs = append(int64IDs, int64ID)
} }
@@ -304,10 +303,10 @@ func convertIDsToSchemapbIDs(ids []interface{}, pkField *schemapb.FieldSchema) (
// Convert number to string // Convert number to string
stringID = fmt.Sprintf("%v", v) stringID = fmt.Sprintf("%v", v)
default: default:
return nil, fmt.Errorf("invalid id type at index %d: expected string, got %T", i, id) return nil, merr.WrapErrParameterInvalidMsg("invalid id type at index %d: expected string, got %T", i, id)
} }
if stringID == "" { if stringID == "" {
return nil, fmt.Errorf("empty string id at index %d", i) return nil, merr.WrapErrParameterInvalidMsg("empty string id at index %d", i)
} }
stringIDs = append(stringIDs, stringID) stringIDs = append(stringIDs, stringID)
} }
@@ -320,7 +319,7 @@ func convertIDsToSchemapbIDs(ids []interface{}, pkField *schemapb.FieldSchema) (
}, nil }, nil
default: default:
return nil, fmt.Errorf("unsupported primary key type: %s", pkField.DataType.String()) return nil, merr.WrapErrParameterInvalidMsg("unsupported primary key type: %s", pkField.DataType.String())
} }
} }
@@ -784,7 +783,7 @@ func checkAndSetData(body []byte, collSchema *schemapb.CollectionSchema, partial
if !containsString(fieldNames, mapKey) { if !containsString(fieldNames, mapKey) {
if collSchema.EnableDynamicField { if collSchema.EnableDynamicField {
if mapKey == common.MetaFieldName { if mapKey == common.MetaFieldName {
return nil, nil, merr.WrapErrParameterInvalidMsg(fmt.Sprintf("use the invalid field name(%s) when enable dynamicField", mapKey)) return nil, nil, merr.WrapErrParameterInvalidMsg("use the invalid field name(%s) when enable dynamicField", mapKey)
} }
mapValueStr := mapValue.String() mapValueStr := mapValue.String()
switch mapValue.Type { switch mapValue.Type {
@@ -1106,7 +1105,7 @@ func decodeByteVectorElement(v gjson.Result, dim, bytesPerVec int64, isFloat16 b
return nil, err return nil, err
} }
if int64(len(row)) != dim { if int64(len(row)) != dim {
return nil, fmt.Errorf("vector dim mismatch: expect %d, got %d", dim, len(row)) return nil, merr.WrapErrParameterInvalidMsg("vector dim mismatch: expect %d, got %d", dim, len(row))
} }
if isFloat16 { if isFloat16 {
return typeutil.Float32ArrayToFloat16Bytes(row), nil return typeutil.Float32ArrayToFloat16Bytes(row), nil
@@ -1114,14 +1113,14 @@ func decodeByteVectorElement(v gjson.Result, dim, bytesPerVec int64, isFloat16 b
return typeutil.Float32ArrayToBFloat16Bytes(row), nil return typeutil.Float32ArrayToBFloat16Bytes(row), nil
} }
if v.Type != gjson.String { if v.Type != gjson.String {
return nil, fmt.Errorf("expect float vector array or base64 string") return nil, merr.WrapErrParameterInvalidMsg("expect float vector array or base64 string")
} }
var row []byte var row []byte
if err := json.Unmarshal([]byte(v.Raw), &row); err != nil { if err := json.Unmarshal([]byte(v.Raw), &row); err != nil {
return nil, err return nil, err
} }
if int64(len(row)) != bytesPerVec { if int64(len(row)) != bytesPerVec {
return nil, fmt.Errorf("byte length mismatch: expect %d, got %d", bytesPerVec, len(row)) return nil, merr.WrapErrParameterInvalidMsg("byte length mismatch: expect %d, got %d", bytesPerVec, len(row))
} }
return row, nil return row, nil
} }
@@ -1293,7 +1292,7 @@ func encodeEmbListQuery(vecs []gjson.Result, elemType schemapb.DataType, dim int
func buildStructArrayFieldData(structSchema *schemapb.StructArrayFieldSchema, perRow []structArrayRow) (*schemapb.FieldData, error) { func buildStructArrayFieldData(structSchema *schemapb.StructArrayFieldSchema, perRow []structArrayRow) (*schemapb.FieldData, error) {
if len(perRow) == 0 { if len(perRow) == 0 {
return nil, fmt.Errorf("struct array field %s has no rows", structSchema.GetName()) return nil, merr.WrapErrParameterInvalidMsg("struct array field %s has no rows", structSchema.GetName())
} }
subs := structSchema.GetFields() subs := structSchema.GetFields()
subFieldData := make([]*schemapb.FieldData, 0, len(subs)) subFieldData := make([]*schemapb.FieldData, 0, len(subs))
@@ -1308,12 +1307,12 @@ func buildStructArrayFieldData(structSchema *schemapb.StructArrayFieldSchema, pe
for rowIdx, row := range perRow { for rowIdx, row := range perRow {
val, ok := row[short] val, ok := row[short]
if !ok { if !ok {
return nil, fmt.Errorf("struct %s row %d missing sub-field %s", return nil, merr.WrapErrParameterInvalidMsg("struct %s row %d missing sub-field %s",
structSchema.GetName(), rowIdx, short) structSchema.GetName(), rowIdx, short)
} }
scalar, ok := val.(*schemapb.ScalarField) scalar, ok := val.(*schemapb.ScalarField)
if !ok { if !ok {
return nil, fmt.Errorf("struct %s sub-field %s row %d: unexpected payload type %T", return nil, merr.WrapErrParameterInvalidMsg("struct %s sub-field %s row %d: unexpected payload type %T",
structSchema.GetName(), short, rowIdx, val) structSchema.GetName(), short, rowIdx, val)
} }
arrayArray.Data = append(arrayArray.Data, scalar) arrayArray.Data = append(arrayArray.Data, scalar)
@@ -1341,12 +1340,12 @@ func buildStructArrayFieldData(structSchema *schemapb.StructArrayFieldSchema, pe
for rowIdx, row := range perRow { for rowIdx, row := range perRow {
val, ok := row[short] val, ok := row[short]
if !ok { if !ok {
return nil, fmt.Errorf("struct %s row %d missing sub-field %s", return nil, merr.WrapErrParameterInvalidMsg("struct %s row %d missing sub-field %s",
structSchema.GetName(), rowIdx, short) structSchema.GetName(), rowIdx, short)
} }
vf, ok := val.(*schemapb.VectorField) vf, ok := val.(*schemapb.VectorField)
if !ok { if !ok {
return nil, fmt.Errorf("struct %s sub-field %s row %d: unexpected payload type %T", return nil, merr.WrapErrParameterInvalidMsg("struct %s sub-field %s row %d: unexpected payload type %T",
structSchema.GetName(), short, rowIdx, val) structSchema.GetName(), short, rowIdx, val)
} }
vecArray.Data = append(vecArray.Data, vf) vecArray.Data = append(vecArray.Data, vf)
@@ -1365,7 +1364,7 @@ func buildStructArrayFieldData(structSchema *schemapb.StructArrayFieldSchema, pe
}, },
}) })
default: default:
return nil, fmt.Errorf("unsupported struct sub-field data type: %s", sub.GetDataType()) return nil, merr.WrapErrParameterInvalidMsg("unsupported struct sub-field data type: %s", sub.GetDataType())
} }
} }
return &schemapb.FieldData{ return &schemapb.FieldData{
@@ -1403,11 +1402,11 @@ func extractStructArrayRow(fd *schemapb.FieldData, rowIdx int, schema *schemapb.
case schemapb.DataType_Array: case schemapb.DataType_Array:
rowData := sub.GetScalars().GetArrayData().GetData() rowData := sub.GetScalars().GetArrayData().GetData()
if rowIdx >= len(rowData) { if rowIdx >= len(rowData) {
return nil, fmt.Errorf("struct sub-field %s missing row %d", short, rowIdx) return nil, merr.WrapErrParameterInvalidMsg("struct sub-field %s missing row %d", short, rowIdx)
} }
values := scalarArrayToInterfaces(rowData[rowIdx]) values := scalarArrayToInterfaces(rowData[rowIdx])
if len(values) != elemCount { if len(values) != elemCount {
return nil, fmt.Errorf("struct sub-field %s element count mismatch: expect %d got %d", return nil, merr.WrapErrParameterInvalidMsg("struct sub-field %s element count mismatch: expect %d got %d",
short, elemCount, len(values)) short, elemCount, len(values))
} }
for i, v := range values { for i, v := range values {
@@ -1416,28 +1415,28 @@ func extractStructArrayRow(fd *schemapb.FieldData, rowIdx int, schema *schemapb.
case schemapb.DataType_ArrayOfVector: case schemapb.DataType_ArrayOfVector:
va := sub.GetVectors().GetVectorArray() va := sub.GetVectors().GetVectorArray()
if va == nil { if va == nil {
return nil, fmt.Errorf("struct sub-field %s has no vector array", short) return nil, merr.WrapErrParameterInvalidMsg("struct sub-field %s has no vector array", short)
} }
if rowIdx >= len(va.GetData()) { if rowIdx >= len(va.GetData()) {
return nil, fmt.Errorf("struct sub-field %s missing row %d", short, rowIdx) return nil, merr.WrapErrParameterInvalidMsg("struct sub-field %s missing row %d", short, rowIdx)
} }
dim, ok := subDims[short] dim, ok := subDims[short]
if !ok || dim <= 0 { if !ok || dim <= 0 {
return nil, fmt.Errorf("schema missing dim for struct sub-field %s", short) return nil, merr.WrapErrParameterInvalidMsg("schema missing dim for struct sub-field %s", short)
} }
values, err := vectorFieldToInterfaces(va.GetData()[rowIdx], va.GetElementType(), dim) values, err := vectorFieldToInterfaces(va.GetData()[rowIdx], va.GetElementType(), dim)
if err != nil { if err != nil {
return nil, err return nil, err
} }
if len(values) != elemCount { if len(values) != elemCount {
return nil, fmt.Errorf("struct sub-field %s vector element count mismatch: expect %d got %d", return nil, merr.WrapErrParameterInvalidMsg("struct sub-field %s vector element count mismatch: expect %d got %d",
short, elemCount, len(values)) short, elemCount, len(values))
} }
for i, v := range values { for i, v := range values {
out[i][short] = v out[i][short] = v
} }
default: default:
return nil, fmt.Errorf("unsupported struct sub-field type %s", sub.GetType()) return nil, merr.WrapErrParameterInvalidMsg("unsupported struct sub-field type %s", sub.GetType())
} }
} }
return out, nil return out, nil
@@ -1455,7 +1454,7 @@ func structArraySubDims(fieldName string, schema *schemapb.CollectionSchema) (ma
} }
dim, err := getDim(sub) dim, err := getDim(sub)
if err != nil { if err != nil {
return nil, fmt.Errorf("schema sub-field %s has no dim: %w", sub.GetName(), err) return nil, merr.WrapErrParameterInvalidErr(err, "schema sub-field %s has no dim", sub.GetName())
} }
subDims[subShortName(sub)] = dim subDims[subShortName(sub)] = dim
} }
@@ -1469,22 +1468,22 @@ func structSubElemCount(sub *schemapb.FieldData, rowIdx int, subDims map[string]
case schemapb.DataType_Array: case schemapb.DataType_Array:
rowData := sub.GetScalars().GetArrayData().GetData() rowData := sub.GetScalars().GetArrayData().GetData()
if rowIdx >= len(rowData) { if rowIdx >= len(rowData) {
return 0, fmt.Errorf("struct sub-field %s row %d out of range", sub.GetFieldName(), rowIdx) return 0, merr.WrapErrParameterInvalidMsg("struct sub-field %s row %d out of range", sub.GetFieldName(), rowIdx)
} }
return len(scalarArrayToInterfaces(rowData[rowIdx])), nil return len(scalarArrayToInterfaces(rowData[rowIdx])), nil
case schemapb.DataType_ArrayOfVector: case schemapb.DataType_ArrayOfVector:
va := sub.GetVectors().GetVectorArray() va := sub.GetVectors().GetVectorArray()
if va == nil || rowIdx >= len(va.GetData()) { if va == nil || rowIdx >= len(va.GetData()) {
return 0, fmt.Errorf("struct sub-field %s row %d out of range", sub.GetFieldName(), rowIdx) return 0, merr.WrapErrParameterInvalidMsg("struct sub-field %s row %d out of range", sub.GetFieldName(), rowIdx)
} }
short := structFieldShortName(sub.GetFieldName()) short := structFieldShortName(sub.GetFieldName())
dim, ok := subDims[short] dim, ok := subDims[short]
if !ok || dim <= 0 { if !ok || dim <= 0 {
return 0, fmt.Errorf("schema missing dim for struct sub-field %s", short) return 0, merr.WrapErrParameterInvalidMsg("schema missing dim for struct sub-field %s", short)
} }
return vectorFieldElemCount(va.GetData()[rowIdx], va.GetElementType(), dim) return vectorFieldElemCount(va.GetData()[rowIdx], va.GetElementType(), dim)
default: default:
return 0, fmt.Errorf("unsupported struct sub-field type %s", sub.GetType()) return 0, merr.WrapErrParameterInvalidMsg("unsupported struct sub-field type %s", sub.GetType())
} }
} }
@@ -1539,7 +1538,7 @@ func scalarArrayToInterfaces(sf *schemapb.ScalarField) []interface{} {
func vectorFieldElemCount(vf *schemapb.VectorField, elemType schemapb.DataType, dim int64) (int, error) { func vectorFieldElemCount(vf *schemapb.VectorField, elemType schemapb.DataType, dim int64) (int, error) {
if dim <= 0 { if dim <= 0 {
return 0, fmt.Errorf("invalid dim %d", dim) return 0, merr.WrapErrParameterInvalidMsg("invalid dim %d", dim)
} }
switch elemType { switch elemType {
case schemapb.DataType_FloatVector: case schemapb.DataType_FloatVector:
@@ -1553,13 +1552,13 @@ func vectorFieldElemCount(vf *schemapb.VectorField, elemType schemapb.DataType,
case schemapb.DataType_Int8Vector: case schemapb.DataType_Int8Vector:
return len(vf.GetInt8Vector()) / int(dim), nil return len(vf.GetInt8Vector()) / int(dim), nil
default: default:
return 0, fmt.Errorf("unsupported vector element type %s", elemType) return 0, merr.WrapErrParameterInvalidMsg("unsupported vector element type %s", elemType)
} }
} }
func vectorFieldToInterfaces(vf *schemapb.VectorField, elemType schemapb.DataType, dim int64) ([]interface{}, error) { func vectorFieldToInterfaces(vf *schemapb.VectorField, elemType schemapb.DataType, dim int64) ([]interface{}, error) {
if dim <= 0 { if dim <= 0 {
return nil, fmt.Errorf("invalid dim %d", dim) return nil, merr.WrapErrParameterInvalidMsg("invalid dim %d", dim)
} }
switch elemType { switch elemType {
case schemapb.DataType_FloatVector: case schemapb.DataType_FloatVector:
@@ -1612,7 +1611,7 @@ func vectorFieldToInterfaces(vf *schemapb.VectorField, elemType schemapb.DataTyp
} }
return out, nil return out, nil
default: default:
return nil, fmt.Errorf("unsupported vector element type %s", elemType) return nil, merr.WrapErrParameterInvalidMsg("unsupported vector element type %s", elemType)
} }
} }
@@ -1620,7 +1619,7 @@ func convertFloatVectorToArray(vector [][]float32, dim int64) ([]float32, error)
floatArray := make([]float32, 0) floatArray := make([]float32, 0)
for _, arr := range vector { for _, arr := range vector {
if int64(len(arr)) != dim { if int64(len(arr)) != dim {
return nil, fmt.Errorf("[]float32 size %d doesn't equal to vector dimension %d of %s", return nil, merr.WrapErrParameterInvalidMsg("[]float32 size %d doesn't equal to vector dimension %d of %s",
len(arr), dim, schemapb.DataType_name[int32(schemapb.DataType_FloatVector)]) len(arr), dim, schemapb.DataType_name[int32(schemapb.DataType_FloatVector)])
} }
for i := int64(0); i < dim; i++ { for i := int64(0); i < dim; i++ {
@@ -1643,7 +1642,7 @@ func convertBinaryVectorToArray(vector [][]byte, dim int64, dataType schemapb.Da
binaryArray := make([]byte, 0, len(vector)*int(bytesLen)) binaryArray := make([]byte, 0, len(vector)*int(bytesLen))
for _, arr := range vector { for _, arr := range vector {
if int64(len(arr)) != bytesLen { if int64(len(arr)) != bytesLen {
return nil, fmt.Errorf("[]byte size %d doesn't equal to vector dimension %d of %s", return nil, merr.WrapErrParameterInvalidMsg("[]byte size %d doesn't equal to vector dimension %d of %s",
len(arr), dim, schemapb.DataType_name[int32(dataType)]) len(arr), dim, schemapb.DataType_name[int32(dataType)])
} }
for i := int64(0); i < bytesLen; i++ { for i := int64(0); i < bytesLen; i++ {
@@ -1657,7 +1656,7 @@ func convertInt8VectorToArray(vector [][]int8, dim int64) ([]byte, error) {
byteArray := make([]byte, 0) byteArray := make([]byte, 0)
for _, arr := range vector { for _, arr := range vector {
if int64(len(arr)) != dim { if int64(len(arr)) != dim {
return nil, fmt.Errorf("[]int8 size %d doesn't equal to vector dimension %d of %s", return nil, merr.WrapErrParameterInvalidMsg("[]int8 size %d doesn't equal to vector dimension %d of %s",
len(arr), dim, schemapb.DataType_name[int32(schemapb.DataType_Int8Vector)]) len(arr), dim, schemapb.DataType_name[int32(schemapb.DataType_Int8Vector)])
} }
for i := int64(0); i < dim; i++ { for i := int64(0); i < dim; i++ {
@@ -1691,7 +1690,7 @@ func reflectValueCandi(v reflect.Value) (map[string]fieldCandi, error) {
} }
return result, nil return result, nil
default: default:
return nil, fmt.Errorf("unsupport row type: %s", v.Kind().String()) return nil, merr.WrapErrParameterInvalidMsg("unsupport row type: %s", v.Kind().String())
} }
} }
@@ -1713,7 +1712,7 @@ func convertToIntArray(dataType schemapb.DataType, arr interface{}) []int32 {
func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool, sch *schemapb.CollectionSchema, inInsert bool, partialUpdate bool) ([]*schemapb.FieldData, error) { func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool, sch *schemapb.CollectionSchema, inInsert bool, partialUpdate bool) ([]*schemapb.FieldData, error) {
rowsLen := len(rows) rowsLen := len(rows)
if rowsLen == 0 { if rowsLen == 0 {
return []*schemapb.FieldData{}, errors.New("no row need to be convert to columns") return []*schemapb.FieldData{}, merr.WrapErrParameterInvalidMsg("no row need to be convert to columns")
} }
isDynamic := sch.EnableDynamicField isDynamic := sch.EnableDynamicField
@@ -1802,7 +1801,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
dim, _ := getDim(field) dim, _ := getDim(field)
nameDims[field.Name] = dim nameDims[field.Name] = dim
default: default:
return nil, fmt.Errorf("the type(%v) of field(%v) is not supported, use other sdk please", field.DataType, field.Name) return nil, merr.WrapErrParameterInvalidMsg("the type(%v) of field(%v) is not supported, use other sdk please", field.DataType, field.Name)
} }
nameColumns[field.Name] = data nameColumns[field.Name] = data
fieldData[field.Name] = &schemapb.FieldData{ fieldData[field.Name] = &schemapb.FieldData{
@@ -1813,7 +1812,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
} }
} }
if len(nameDims) == 0 && len(sch.Functions) == 0 && !partialUpdate { if len(nameDims) == 0 && len(sch.Functions) == 0 && !partialUpdate {
return nil, fmt.Errorf("collection: %s has no vector field or functions", sch.Name) return nil, merr.WrapErrParameterInvalidMsg("collection: %s has no vector field or functions", sch.Name)
} }
dynamicCol := make([][]byte, 0, rowsLen) dynamicCol := make([][]byte, 0, rowsLen)
@@ -1836,7 +1835,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
continue continue
} }
if !allowInsertAutoID { if !allowInsertAutoID {
return nil, merr.WrapErrParameterInvalidMsg(fmt.Sprintf("no need to pass pk field(%s) when autoid==true in insert", field.Name)) return nil, merr.WrapErrParameterInvalidMsg("no need to pass pk field(%s) when autoid==true in insert", field.Name)
} }
} }
if (field.Nullable || field.DefaultValue != nil) && !ok { if (field.Nullable || field.DefaultValue != nil) && !ok {
@@ -1851,7 +1850,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
if partialUpdate { if partialUpdate {
continue continue
} }
return nil, fmt.Errorf("row %d does not has field %s", idx, field.Name) return nil, merr.WrapErrParameterInvalidMsg("row %d does not has field %s", idx, field.Name)
} }
fieldLen[field.Name] += 1 fieldLen[field.Name] += 1
switch field.DataType { switch field.DataType {
@@ -1893,7 +1892,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
vec := typeutil.Float32ArrayToFloat16Bytes(candi.v.Interface().([]float32)) vec := typeutil.Float32ArrayToFloat16Bytes(candi.v.Interface().([]float32))
nameColumns[field.Name] = append(nameColumns[field.Name].([][]byte), vec) nameColumns[field.Name] = append(nameColumns[field.Name].([][]byte), vec)
default: default:
return nil, merr.WrapErrParameterInvalidMsg(fmt.Sprintf("invalid type(%v) of field(%v) ", field.DataType, field.Name)) return nil, merr.WrapErrParameterInvalidMsg("invalid type(%v) of field(%v) ", field.DataType, field.Name)
} }
case schemapb.DataType_BFloat16Vector: case schemapb.DataType_BFloat16Vector:
switch candi.v.Interface().(type) { switch candi.v.Interface().(type) {
@@ -1903,7 +1902,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
vec := typeutil.Float32ArrayToBFloat16Bytes(candi.v.Interface().([]float32)) vec := typeutil.Float32ArrayToBFloat16Bytes(candi.v.Interface().([]float32))
nameColumns[field.Name] = append(nameColumns[field.Name].([][]byte), vec) nameColumns[field.Name] = append(nameColumns[field.Name].([][]byte), vec)
default: default:
return nil, merr.WrapErrParameterInvalidMsg(fmt.Sprintf("invalid type(%v) of field(%v) ", field.DataType, field.Name)) return nil, merr.WrapErrParameterInvalidMsg("invalid type(%v) of field(%v) ", field.DataType, field.Name)
} }
case schemapb.DataType_SparseFloatVector: case schemapb.DataType_SparseFloatVector:
content := candi.v.Interface().([]byte) content := candi.v.Interface().([]byte)
@@ -1915,7 +1914,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
case schemapb.DataType_Int8Vector: case schemapb.DataType_Int8Vector:
nameColumns[field.Name] = append(nameColumns[field.Name].([][]int8), candi.v.Interface().([]int8)) nameColumns[field.Name] = append(nameColumns[field.Name].([][]int8), candi.v.Interface().([]int8))
default: default:
return nil, fmt.Errorf("the type(%v) of field(%v) is not supported, use other sdk please", field.DataType, field.Name) return nil, merr.WrapErrParameterInvalidMsg("the type(%v) of field(%v) is not supported, use other sdk please", field.DataType, field.Name)
} }
delete(set, field.Name) delete(set, field.Name)
@@ -1931,7 +1930,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
} }
bs, err := json.Marshal(m) bs, err := json.Marshal(m)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to marshal dynamic field %w", err) return nil, merr.WrapErrParameterInvalidErr(err, "failed to marshal dynamic field")
} }
dynamicCol = append(dynamicCol, bs) dynamicCol = append(dynamicCol, bs)
} }
@@ -1949,7 +1948,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
zap.String("fieldName", name), zap.String("fieldName", name),
zap.Int("fieldLen", len(validData)), zap.Int("fieldLen", len(validData)),
zap.Int("rowsLen", rowsLen)) zap.Int("rowsLen", rowsLen))
return nil, fmt.Errorf("column %s has length %d, expected %d", name, len(validData), rowsLen) return nil, merr.WrapErrParameterInvalidMsg("column %s has length %d, expected %d", name, len(validData), rowsLen)
} }
} else { } else {
log.Info("skip empty field for partial update", log.Info("skip empty field for partial update",
@@ -1963,7 +1962,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
zap.String("fieldName", name), zap.String("fieldName", name),
zap.Int("fieldLen", fieldLen[name]), zap.Int("fieldLen", fieldLen[name]),
zap.Int("rowsLen", rowsLen)) zap.Int("rowsLen", rowsLen))
return nil, fmt.Errorf("column %s has length %d, expected %d", name, fieldLen[name], rowsLen) return nil, merr.WrapErrParameterInvalidMsg("column %s has length %d, expected %d", name, fieldLen[name], rowsLen)
} }
colData := fieldData[name] colData := fieldData[name]
@@ -2183,7 +2182,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
}, },
} }
default: default:
return nil, fmt.Errorf("the type(%v) of field(%v) is not supported, use other sdk please", colData.Type, name) return nil, merr.WrapErrParameterInvalidMsg("the type(%v) of field(%v) is not supported, use other sdk please", colData.Type, name)
} }
colData.ValidData = validDataMap[name] colData.ValidData = validDataMap[name]
columns = append(columns, colData) columns = append(columns, colData)
@@ -2212,11 +2211,11 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
if partialUpdate { if partialUpdate {
continue continue
} }
return nil, fmt.Errorf("row %d does not has struct field %s", rowIdx, structField.GetName()) return nil, merr.WrapErrParameterInvalidMsg("row %d does not has struct field %s", rowIdx, structField.GetName())
} }
sr, ok := val.(structArrayRow) sr, ok := val.(structArrayRow)
if !ok { if !ok {
return nil, fmt.Errorf("row %d struct field %s has unexpected payload type %T", return nil, merr.WrapErrParameterInvalidMsg("row %d struct field %s has unexpected payload type %T",
rowIdx, structField.GetName(), val) rowIdx, structField.GetName(), val)
} }
perRow = append(perRow, sr) perRow = append(perRow, sr)
@@ -2437,27 +2436,27 @@ func fieldDataValueCount(fieldData *schemapb.FieldData) (int64, error) {
dim := fieldData.GetVectors().GetDim() dim := fieldData.GetVectors().GetDim()
bytesPerRow := dim / 8 bytesPerRow := dim / 8
if bytesPerRow <= 0 { if bytesPerRow <= 0 {
return 0, fmt.Errorf("invalid binary vector dimension %d for field %s", dim, fieldData.GetFieldName()) return 0, merr.WrapErrParameterInvalidMsg("invalid binary vector dimension %d for field %s", dim, fieldData.GetFieldName())
} }
return int64(len(fieldData.GetVectors().GetBinaryVector())) / bytesPerRow, nil return int64(len(fieldData.GetVectors().GetBinaryVector())) / bytesPerRow, nil
case schemapb.DataType_FloatVector: case schemapb.DataType_FloatVector:
dim := fieldData.GetVectors().GetDim() dim := fieldData.GetVectors().GetDim()
if dim <= 0 { if dim <= 0 {
return 0, fmt.Errorf("invalid float vector dimension %d for field %s", dim, fieldData.GetFieldName()) return 0, merr.WrapErrParameterInvalidMsg("invalid float vector dimension %d for field %s", dim, fieldData.GetFieldName())
} }
return int64(len(fieldData.GetVectors().GetFloatVector().GetData())) / dim, nil return int64(len(fieldData.GetVectors().GetFloatVector().GetData())) / dim, nil
case schemapb.DataType_Float16Vector: case schemapb.DataType_Float16Vector:
dim := fieldData.GetVectors().GetDim() dim := fieldData.GetVectors().GetDim()
bytesPerRow := dim * 2 bytesPerRow := dim * 2
if bytesPerRow <= 0 { if bytesPerRow <= 0 {
return 0, fmt.Errorf("invalid float16 vector dimension %d for field %s", dim, fieldData.GetFieldName()) return 0, merr.WrapErrParameterInvalidMsg("invalid float16 vector dimension %d for field %s", dim, fieldData.GetFieldName())
} }
return int64(len(fieldData.GetVectors().GetFloat16Vector())) / bytesPerRow, nil return int64(len(fieldData.GetVectors().GetFloat16Vector())) / bytesPerRow, nil
case schemapb.DataType_BFloat16Vector: case schemapb.DataType_BFloat16Vector:
dim := fieldData.GetVectors().GetDim() dim := fieldData.GetVectors().GetDim()
bytesPerRow := dim * 2 bytesPerRow := dim * 2
if bytesPerRow <= 0 { if bytesPerRow <= 0 {
return 0, fmt.Errorf("invalid bfloat16 vector dimension %d for field %s", dim, fieldData.GetFieldName()) return 0, merr.WrapErrParameterInvalidMsg("invalid bfloat16 vector dimension %d for field %s", dim, fieldData.GetFieldName())
} }
return int64(len(fieldData.GetVectors().GetBfloat16Vector())) / bytesPerRow, nil return int64(len(fieldData.GetVectors().GetBfloat16Vector())) / bytesPerRow, nil
case schemapb.DataType_SparseFloatVector: case schemapb.DataType_SparseFloatVector:
@@ -2465,7 +2464,7 @@ func fieldDataValueCount(fieldData *schemapb.FieldData) (int64, error) {
case schemapb.DataType_Int8Vector: case schemapb.DataType_Int8Vector:
dim := fieldData.GetVectors().GetDim() dim := fieldData.GetVectors().GetDim()
if dim <= 0 { if dim <= 0 {
return 0, fmt.Errorf("invalid int8 vector dimension %d for field %s", dim, fieldData.GetFieldName()) return 0, merr.WrapErrParameterInvalidMsg("invalid int8 vector dimension %d for field %s", dim, fieldData.GetFieldName())
} }
return int64(len(fieldData.GetVectors().GetInt8Vector())) / dim, nil return int64(len(fieldData.GetVectors().GetInt8Vector())) / dim, nil
case schemapb.DataType_ArrayOfStruct: case schemapb.DataType_ArrayOfStruct:
@@ -2479,10 +2478,10 @@ func fieldDataValueCount(fieldData *schemapb.FieldData) (int64, error) {
case schemapb.DataType_ArrayOfVector: case schemapb.DataType_ArrayOfVector:
return int64(len(subs[0].GetVectors().GetVectorArray().GetData())), nil return int64(len(subs[0].GetVectors().GetVectorArray().GetData())), nil
default: default:
return 0, fmt.Errorf("unsupported struct sub-field type %s for field %s", subs[0].GetType(), fieldData.GetFieldName()) return 0, merr.WrapErrParameterInvalidMsg("unsupported struct sub-field type %s for field %s", subs[0].GetType(), fieldData.GetFieldName())
} }
default: default:
return 0, fmt.Errorf("the type(%v) of field(%v) is not supported, use other sdk please", fieldData.GetType(), fieldData.GetFieldName()) return 0, merr.WrapErrParameterInvalidMsg("the type(%v) of field(%v) is not supported, use other sdk please", fieldData.GetType(), fieldData.GetFieldName())
} }
} }
@@ -2532,7 +2531,7 @@ func newFieldDataRowAccessor(fieldData *schemapb.FieldData) (*fieldDataRowAccess
return accessor, nil return accessor, nil
} }
if valueCount != validCount { if valueCount != validCount {
return nil, fmt.Errorf("field %s has %d valid rows, but data length is %d", fieldData.GetFieldName(), validCount, valueCount) return nil, merr.WrapErrParameterInvalidMsg("field %s has %d valid rows, but data length is %d", fieldData.GetFieldName(), validCount, valueCount)
} }
accessor.compactIndices = compactIndices accessor.compactIndices = compactIndices
return accessor, nil return accessor, nil
@@ -2543,7 +2542,7 @@ func (accessor *fieldDataRowAccessor) rowIndex(rowIdx int64) (int64, bool, error
return rowIdx, true, nil return rowIdx, true, nil
} }
if rowIdx >= int64(len(accessor.validData)) { if rowIdx >= int64(len(accessor.validData)) {
return 0, false, fmt.Errorf("row index %d out of range for field %s valid data length %d", rowIdx, accessor.fieldData.GetFieldName(), len(accessor.validData)) return 0, false, merr.WrapErrParameterInvalidMsg("row index %d out of range for field %s valid data length %d", rowIdx, accessor.fieldData.GetFieldName(), len(accessor.validData))
} }
if !accessor.validData[rowIdx] { if !accessor.validData[rowIdx] {
return 0, false, nil return 0, false, nil
@@ -2575,7 +2574,7 @@ func buildQueryResp(rowsNum int64, needFields []string, fieldDataList []*schemap
stringPks := ids.GetStrId().GetData() stringPks := ids.GetStrId().GetData()
rowsNum = int64(len(stringPks)) rowsNum = int64(len(stringPks))
default: default:
return nil, errors.New("the type of primary key(id) is not supported, use other sdk please") return nil, merr.WrapErrParameterInvalidMsg("the type of primary key(id) is not supported, use other sdk please")
} }
} }
} }
@@ -2716,7 +2715,7 @@ func buildQueryResp(rowsNum int64, needFields []string, fieldDataList []*schemap
stringPks := ids.GetStrId().GetData() stringPks := ids.GetStrId().GetData()
row[pkFieldName] = stringPks[i] row[pkFieldName] = stringPks[i]
default: default:
return nil, errors.New("the type of primary key(id) is not supported, use other sdk please") return nil, merr.WrapErrParameterInvalidMsg("the type of primary key(id) is not supported, use other sdk please")
} }
} }
if scores != nil && int64(len(scores)) > i { if scores != nil && int64(len(scores)) > i {
@@ -2734,29 +2733,31 @@ func hasSearchAggregationResult(results *schemapb.SearchResultData) bool {
func buildSearchAggregationResp(results *schemapb.SearchResultData, enableInt64 bool, collectionSchema *schemapb.CollectionSchema) ([]gin.H, error) { func buildSearchAggregationResp(results *schemapb.SearchResultData, enableInt64 bool, collectionSchema *schemapb.CollectionSchema) ([]gin.H, error) {
if results == nil { if results == nil {
return nil, errors.New("search_aggregation result is nil") // The aggregation payload is produced by the server-side reduce, never
// by the request: a malformed shape is an internal contract violation.
return nil, merr.WrapErrServiceInternalMsg("search_aggregation result is nil")
} }
aggTopks := results.GetAggTopks() aggTopks := results.GetAggTopks()
pbBuckets := results.GetAggBuckets() pbBuckets := results.GetAggBuckets()
if len(aggTopks) == 0 { if len(aggTopks) == 0 {
return nil, errors.New("search_aggregation response missing agg_topks") return nil, merr.WrapErrServiceInternalMsg("search_aggregation response missing agg_topks")
} }
if results.GetNumQueries() <= 0 { if results.GetNumQueries() <= 0 {
return nil, errors.New("search_aggregation response missing nq") return nil, merr.WrapErrServiceInternalMsg("search_aggregation response missing nq")
} }
if len(aggTopks) != int(results.GetNumQueries()) { if len(aggTopks) != int(results.GetNumQueries()) {
return nil, fmt.Errorf("search_aggregation agg_topks length %d does not match nq %d", len(aggTopks), results.GetNumQueries()) return nil, merr.WrapErrServiceInternalMsg("search_aggregation agg_topks length %d does not match nq %d", len(aggTopks), results.GetNumQueries())
} }
total := int64(0) total := int64(0)
for _, topk := range aggTopks { for _, topk := range aggTopks {
if topk < 0 { if topk < 0 {
return nil, errors.New("search_aggregation agg_topks cannot contain negative values") return nil, merr.WrapErrServiceInternalMsg("search_aggregation agg_topks cannot contain negative values")
} }
total += topk total += topk
} }
if total != int64(len(pbBuckets)) { if total != int64(len(pbBuckets)) {
return nil, fmt.Errorf("search_aggregation agg_topks sum %d does not match bucket count %d", total, len(pbBuckets)) return nil, merr.WrapErrServiceInternalMsg("search_aggregation agg_topks sum %d does not match bucket count %d", total, len(pbBuckets))
} }
output := make([]gin.H, 0, len(aggTopks)) output := make([]gin.H, 0, len(aggTopks))
@@ -2778,7 +2779,7 @@ func buildSearchAggregationResp(results *schemapb.SearchResultData, enableInt64
func buildAggBucketResp(pb *schemapb.AggBucket, enableInt64 bool, collectionSchema *schemapb.CollectionSchema) (gin.H, error) { func buildAggBucketResp(pb *schemapb.AggBucket, enableInt64 bool, collectionSchema *schemapb.CollectionSchema) (gin.H, error) {
if pb == nil { if pb == nil {
return nil, errors.New("search_aggregation bucket is nil") return nil, merr.WrapErrServiceInternalMsg("search_aggregation bucket is nil")
} }
bucket := gin.H{ bucket := gin.H{
"key": buildAggBucketKeyResp(pb.GetKey(), enableInt64), "key": buildAggBucketKeyResp(pb.GetKey(), enableInt64),
@@ -2986,7 +2987,7 @@ func convertConsistencyLevel(reqConsistencyLevel string) (commonpb.ConsistencyLe
if reqConsistencyLevel != "" { if reqConsistencyLevel != "" {
level, ok := commonpb.ConsistencyLevel_value[reqConsistencyLevel] level, ok := commonpb.ConsistencyLevel_value[reqConsistencyLevel]
if !ok { if !ok {
return 0, false, merr.WrapErrParameterInvalidMsg(fmt.Sprintf("parameter:'%s' is incorrect, please check it", reqConsistencyLevel)) return 0, false, merr.WrapErrParameterInvalidMsg("parameter:'%s' is incorrect, please check it", reqConsistencyLevel)
} }
return commonpb.ConsistencyLevel(level), false, nil return commonpb.ConsistencyLevel(level), false, nil
} }
@@ -3096,7 +3097,7 @@ func convertDefaultValue(value interface{}, dataType schemapb.DataType) (*schema
} }
return data, nil return data, nil
default: default:
return nil, merr.WrapErrParameterInvalidMsg(fmt.Sprintf("Unexpected default value type: %s", dataType.String())) return nil, merr.WrapErrParameterInvalidMsg("Unexpected default value type: %s", dataType.String())
} }
} }
@@ -3491,7 +3492,7 @@ func generateSearchParams(reqSearchParams map[string]interface{}) ([]*commonpb.K
for key, value := range reqSearchParams { for key, value := range reqSearchParams {
if val, ok := paramsMap[key]; ok { if val, ok := paramsMap[key]; ok {
if !deepEqual(val, value) { if !deepEqual(val, value) {
return nil, merr.WrapErrParameterInvalidMsg(fmt.Sprintf("ambiguous parameter: %s, in search_param: %v, in search_param.params: %v", key, value, val)) return nil, merr.WrapErrParameterInvalidMsg("ambiguous parameter: %s, in search_param: %v, in search_param.params: %v", key, value, val)
} }
} else if key != Params { } else if key != Params {
paramsMap[key] = value paramsMap[key] = value
@@ -25,6 +25,7 @@ import (
"strings" "strings"
"testing" "testing"
"github.com/cockroachdb/errors"
"github.com/gin-gonic/gin" "github.com/gin-gonic/gin"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
@@ -36,6 +37,7 @@ import (
"github.com/milvus-io/milvus-proto/go-api/v3/schemapb" "github.com/milvus-io/milvus-proto/go-api/v3/schemapb"
"github.com/milvus-io/milvus/internal/json" "github.com/milvus-io/milvus/internal/json"
"github.com/milvus-io/milvus/pkg/v3/common" "github.com/milvus-io/milvus/pkg/v3/common"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/typeutil" "github.com/milvus-io/milvus/pkg/v3/util/typeutil"
) )
@@ -3005,7 +3007,8 @@ func TestBuildQueryResps(t *testing.T) {
} }
_, err := buildQueryResp(int64(0), outputFields, newFieldData([]*schemapb.FieldData{}, 1000), generateIDs(schemapb.DataType_Int64, 3), DefaultScores, true, nil) _, err := buildQueryResp(int64(0), outputFields, newFieldData([]*schemapb.FieldData{}, 1000), generateIDs(schemapb.DataType_Int64, 3), DefaultScores, true, nil)
assert.Equal(t, "the type(1000) of field(wrong-field-type) is not supported, use other sdk please", err.Error()) assert.Contains(t, err.Error(), "the type(1000) of field(wrong-field-type) is not supported, use other sdk please")
assert.True(t, errors.Is(err, merr.ErrParameterInvalid))
res, err := buildQueryResp(int64(0), outputFields, []*schemapb.FieldData{}, generateIDs(schemapb.DataType_Int64, 3), DefaultScores, true, nil) res, err := buildQueryResp(int64(0), outputFields, []*schemapb.FieldData{}, generateIDs(schemapb.DataType_Int64, 3), DefaultScores, true, nil)
assert.Equal(t, 3, len(res)) assert.Equal(t, 3, len(res))
@@ -17,15 +17,13 @@
package httpserver package httpserver
import ( import (
"fmt"
"github.com/cockroachdb/errors"
"google.golang.org/protobuf/proto" "google.golang.org/protobuf/proto"
"github.com/milvus-io/milvus-proto/go-api/v3/commonpb" "github.com/milvus-io/milvus-proto/go-api/v3/commonpb"
"github.com/milvus-io/milvus-proto/go-api/v3/milvuspb" "github.com/milvus-io/milvus-proto/go-api/v3/milvuspb"
"github.com/milvus-io/milvus-proto/go-api/v3/schemapb" "github.com/milvus-io/milvus-proto/go-api/v3/schemapb"
"github.com/milvus-io/milvus/internal/json" "github.com/milvus-io/milvus/internal/json"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
"github.com/milvus-io/milvus/pkg/v3/util/typeutil" "github.com/milvus-io/milvus/pkg/v3/util/typeutil"
) )
@@ -70,7 +68,7 @@ type WrappedInsertRequest struct {
func (w *WrappedInsertRequest) AsInsertRequest() (*milvuspb.InsertRequest, error) { func (w *WrappedInsertRequest) AsInsertRequest() (*milvuspb.InsertRequest, error) {
fieldData, err := convertFieldDataArray(w.FieldsData) fieldData, err := convertFieldDataArray(w.FieldsData)
if err != nil { if err != nil {
return nil, fmt.Errorf("%w: convert field data failed: %v", errBadRequest, err) return nil, badRequestf(err, "convert field data failed")
} }
return &milvuspb.InsertRequest{ return &milvuspb.InsertRequest{
Base: w.Base, Base: w.Base,
@@ -98,12 +96,12 @@ func (f *FieldData) makePbFloat16OrBfloat16Array(raw json.RawMessage, serializeF
return nil, 0, newFieldDataError(f.FieldName, err) return nil, 0, newFieldDataError(f.FieldName, err)
} }
if len(wrappedData) < 1 { if len(wrappedData) < 1 {
return nil, 0, errors.New("at least one row for insert") return nil, 0, merr.WrapErrParameterInvalidMsg("at least one row for insert")
} }
array0 := wrappedData[0] array0 := wrappedData[0]
dim := len(array0) dim := len(array0)
if dim < 1 { if dim < 1 {
return nil, 0, errors.New("dim must >= 1") return nil, 0, merr.WrapErrParameterInvalidMsg("dim must >= 1")
} }
data := make([]byte, 0, len(wrappedData)*dim*2) data := make([]byte, 0, len(wrappedData)*dim*2)
for _, fp32Array := range wrappedData { for _, fp32Array := range wrappedData {
@@ -238,12 +236,12 @@ func (f *FieldData) AsSchemapb() (*schemapb.FieldData, error) {
return nil, newFieldDataError(f.FieldName, err) return nil, newFieldDataError(f.FieldName, err)
} }
if len(wrappedData) < 1 { if len(wrappedData) < 1 {
return nil, errors.New("at least one row for insert") return nil, merr.WrapErrParameterInvalidMsg("at least one row for insert")
} }
array0 := wrappedData[0] array0 := wrappedData[0]
dim := len(array0) dim := len(array0)
if dim < 1 { if dim < 1 {
return nil, errors.New("dim must >= 1") return nil, merr.WrapErrParameterInvalidMsg("dim must >= 1")
} }
data := make([]float32, len(wrappedData)*dim) data := make([]float32, len(wrappedData)*dim)
@@ -299,7 +297,7 @@ func (f *FieldData) AsSchemapb() (*schemapb.FieldData, error) {
return nil, newFieldDataError(f.FieldName, err) return nil, newFieldDataError(f.FieldName, err)
} }
if len(wrappedData) < 1 { if len(wrappedData) < 1 {
return nil, errors.New("at least one row for insert") return nil, merr.WrapErrParameterInvalidMsg("at least one row for insert")
} }
data := make([][]byte, len(wrappedData)) data := make([][]byte, len(wrappedData))
dim := int64(0) dim := int64(0)
@@ -333,12 +331,12 @@ func (f *FieldData) AsSchemapb() (*schemapb.FieldData, error) {
return nil, newFieldDataError(f.FieldName, err) return nil, newFieldDataError(f.FieldName, err)
} }
if len(wrappedData) < 1 { if len(wrappedData) < 1 {
return nil, errors.New("at least one row for insert") return nil, merr.WrapErrParameterInvalidMsg("at least one row for insert")
} }
array0 := wrappedData[0] array0 := wrappedData[0]
dim := len(array0) dim := len(array0)
if dim < 1 { if dim < 1 {
return nil, errors.New("dim must >= 1") return nil, merr.WrapErrParameterInvalidMsg("dim must >= 1")
} }
data := make([]byte, len(wrappedData)*dim) data := make([]byte, len(wrappedData)*dim)
@@ -358,13 +356,13 @@ func (f *FieldData) AsSchemapb() (*schemapb.FieldData, error) {
}, },
} }
default: default:
return nil, errors.New("unsupported data type") return nil, merr.WrapErrParameterInvalidMsg("unsupported data type")
} }
return &ret, nil return &ret, nil
} }
func newFieldDataError(field string, err error) error { func newFieldDataError(field string, err error) error {
return fmt.Errorf("parse field[%s]: %s", field, err.Error()) return merr.WrapErrParameterInvalidErr(err, "parse field[%s]", field)
} }
func convertFieldDataArray(input []*FieldData) ([]*schemapb.FieldData, error) { func convertFieldDataArray(input []*FieldData) ([]*schemapb.FieldData, error) {
@@ -25,10 +25,18 @@ import (
"github.com/gin-gonic/gin/binding" "github.com/gin-gonic/gin/binding"
"github.com/milvus-io/milvus-proto/go-api/v3/commonpb" "github.com/milvus-io/milvus-proto/go-api/v3/commonpb"
"github.com/milvus-io/milvus/pkg/v3/util/merr"
) )
var errBadRequest = errors.New("bad request") var errBadRequest = errors.New("bad request")
// badRequestf wraps err with the package-internal errBadRequest sentinel and
// a contextual message, preserving the inner error chain. wrapHandler maps
// any error that matches errBadRequest to HTTP 400.
func badRequestf(err error, format string, args ...any) error {
return merr.Mark(merr.Wrapf(err, format, args...), errBadRequest)
}
// handlerFunc handles http request with gin context // handlerFunc handles http request with gin context
type handlerFunc func(c *gin.Context) (interface{}, error) type handlerFunc func(c *gin.Context) (interface{}, error)

Some files were not shown because too many files have changed in this diff Show More