mirror of
https://github.com/milvus-io/milvus.git
synced 2026-07-20 09:45:43 +00:00
enhance: standardize error handling on merr + Sys/Input classification (#50221)
issue: #47420 ## What this PR does Project-wide migration of raw `fmt.Errorf` / `errors.New` in function bodies onto the `merr` framework, plus the Sys-vs-Input error classification and the machinery it drives (retriability, fine-grained metrics, segcore unification), plus the convention docs and a linter that keeps it from regressing. Scope: storage, proxy, coordinators (root/data/query), query node, data node, `pkg/util` & `internal/util`, expression parser, message queue, streaming, and misc packages. Bare raw-error usages went from ~3000 to a ~340 allowlist (package-level sentinels / build-tag / test sites). --- ## How to review this PR It is large but the vast majority is mechanical. Changes fall into three tiers; spend review budget on Part 2 and Part 3. ### Part 1 — Mechanical standardization (low risk, verify by rule) Each converted call follows one of a small fixed set of rules. To review, check that each site obeys the matching rule rather than reading every line: | Pattern | Rule | |---|---| | `fmt.Errorf("...")` originating a new error | → `merr.WrapErrXxxMsg("...")` with a code matching the failure's meaning | | Adding context to an existing typed error | → `merr.Wrap(err, "...")` / `merr.Wrapf(...)` — **preserves** the inner code (never `WrapErr*Err`, which overwrites it) | | Errors inside the streaming subsystem | → `status.New*` factories (StreamingError), **not** merr — this is the component-internal dialect (see `docs/dev/error_handling_guide.md`) | | Low-level / control-flow signal caught by `errors.Is` | → kept as a package-level `errors.New` sentinel (lowercase, same-package) | Conventions are documented in `docs/dev/error_handling_guide.md` (how-to) and `docs/dev/error_sentinel_convention.md` (rules + audit). A `gocritic`/`ruleguard` rule (`rawmerrerror`, in `rules.go`) enforces "no raw `return errors.New/fmt.Errorf`" under `make verifiers`. ### Part 2 — Behavior changes (review these closely) These are the sites where the wire contract or runtime behavior changes, not just the source text. Listed by category; representative locations given, full set in the diff. **A. gRPC wire-code shifts: `UnexpectedError(1)/Code 65535` → typed code.** Where a handler previously returned a raw error (collapsed to `Code=65535` on the wire), it now returns a typed merr, so the client sees a real code. The most common shift is to `IllegalArgument(5)/Code 1100` (ParameterInvalid). Touch points include datanode task handlers (CreateTask/Query/Drop), proxy Upsert, querynode GetMetrics, datacoord CreateIndex, httpserver query-response builder, and typeutil schema validation. One code refinement: an index-param validation moved `1100` → `1101` (ParameterMissing). **Client/SDK assertions and any code that switched on `Code=65535` for these paths must be re-checked** (the go_client e2e assertions were already aligned in this PR). **B. Prometheus `status` label contract change (externally visible).** The proxy metric's coarse `fail` / `rejected` values are split into `fail_input` / `fail_system` and `rejected_user` / `rejected_system` (in `requestutil.ParseMetricLabel`; auth/privilege rejections count as `rejected_user`), so dashboards can attribute a failure to caller vs operator. **Dashboards/alerts querying `status="fail"` must migrate to `status=~"fail_.*"`, and `status="rejected"` to `status=~"rejected_.*"`.** The in-repo Grafana dashboard is already migrated; external dashboards built on the old values silently go empty after upgrade. This is the one change that requires an ops-side migration. **C. Retriability semantics.** - C1: `merr.Status(err)` now forces `Retriable=false` when the error is an `InputError` — a malformed request can never succeed on blind retry, so clients never get the self-contradictory "your input is wrong but you may retry". - C2: `retry.Do` short-circuits an `InputError` (non-retriable) — **but only when the caller did not pass a `RetryErr` predicate**. The check is an `if c.isRetryErr != nil { ... } else if InputError { ... }` *mutually exclusive* branch (`pkg/util/retry/retry.go`): an explicit `RetryErr` takes precedence and bypasses the InputError abort. `retry.Handle` deliberately does **not** apply the InputError abort (its callers signal abort via `shouldRetry=false`). Four flusher startup callsites that must retry through transient "not ready" errors were given explicit `RetryErr` escape hatches. **D. segcore (C++→Go) error classification.** A single shared Go-side table (`pkg/util/merr/segcore.go`) maps each segcore code to a merr sentinel + InputError/signal category, replacing scattered hand-written `if errorCode == ...` switches in the cgo wrappers. **Wire `Code` values change for every segcore pass-through error, not just the remapped ones.** Named sentinels remap (C++ `2003` → merr `2001`, `2033` → `2002`, Folly/Knowhere codes likewise); **all remaining pass-through codes (`2004`–`2043`, previously surfaced to clients as raw C++ enum values) now serialize as `2000`** (`ErrSegcore`), with the original C++ code preserved in the `Reason` text (`segcoreCode=...`); unknown/future codes collapse to `2000` as well (pinned by the `wire_code_projection` test). Transient segcore classes (object storage / file IO / OOM / mmap / FieldNotLoaded — 11 codes) now report `Retriable=true`. **Any client switching on raw segcore codes in the `2004`–`2043` range must be re-checked**; the in-Reason code remains available for diagnostics. Signal codes (PretendFinished / FollyCancel) are recognized centrally. `errors.Is`-based control flow on these (e.g. scheduler skip/retry) is preserved. **E. InputError classification (25 sentinels + dynamic marks).** 25 sentinels in `errors.go` carry `WithErrorType(InputError)` (the Collection / ResourceGroup / Database families, `ErrIndexDuplicate`, `ErrParameterInvalid`, `ErrPrivilegeNotAuthenticated`, `ErrImportFailed`, `ErrQueryPlan`, ...), plus dynamic marks for the 8 segcore input codes (ExprInvalid, DimNotMatch, MetricTypeInvalid, FieldIDInvalid, ...) and `WrapErrAsInputError`. The widest blast radius is `ErrParameterInvalid` (1100): ~2335 `WrapErrParameterInvalid*` callsites now classify as input / non-retriable. Because of C1/C2 this changes retriability for any path that returns these. **The audit to confirm no transient path was mis-marked is the single most important review item** (see Part 3). One reverse correction: storage field-stats parsing moved from `ErrParameterInvalid` (input) to `ErrDataIntegrity` — a corrupted stored stat is data corruption, not user input. ### Part 3 — Known risks & traps (called out proactively) 1. **`merr.Wrap` vs `WrapErr*Err` (code-masking).** `WrapErr*Err` builds a `wrappedMilvusError{sentinel: ErrServiceInternal}` whose `code()` returns the *outer* sentinel — it overwrites the inner typed code and hides the `errors.Is` chain. This is intentional (use it to *deliberately* downgrade), but it was a recurring conversion defect; the rule "add context with `merr.Wrap`, downgrade with `WrapErr*Err`" is enforced by convention and reviewed across the diff. 2. **InputError × `retry.Do` blast radius.** Marking a sentinel `InputError` makes any `retry.Do(...)` without a `RetryErr` predicate stop retrying it. Reviewers should sanity-check that no transient use of the 19 newly-marked sentinels (especially `ErrParameterInvalid`) sits inside a retry loop that needed to keep spinning. The known flusher cases were handled (see C2). 3. **The ~340 raw-error allowlist.** What remains as bare `errors.New` is, by design: package-level sentinels (caught by `errors.Is`), `//go:build test` sites, and out-of-band trees (`cmd/`, `tests/`, codegen, walimpls). The linter only bans the *direct-return* form; assignment-then-return escapes and the full no-exceptions ban are deferred to an AST-based linter (Tier 2, documented). 4. **segcore C++ second step deferred.** This PR unifies classification on the Go side; splitting the dual-semantic C++ codes at the source is a follow-up. --- ## Validation - `make verifiers`: Go side clean (gofmt + static-check across modules, including the new `rawmerrerror` rule with a 0-hit baseline repo-wide). - `make test-go`: passing; the one real regression introduced (a datanode `invalid_task_type` assertion shifting `1` → `5` from a ParameterInvalid conversion) was fixed in-tree. - go_client e2e CreateIndex assertions aligned to the new merr messages. --------- Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.7
parent
14c4cf65fc
commit
e2787d3981
+4
-4
@@ -359,15 +359,15 @@ proxy:
|
||||
remoteMaxTime: 0 # The time interval allowed for uploading access log files. If the upload time of a log file exceeds this interval, the file will be deleted. Setting the value to 0 disables this feature.
|
||||
formatters:
|
||||
base:
|
||||
format: "[$time_now] [ACCESS] <$user_name: $user_addr> $method_name [status: $method_status] [code: $error_code] [sdk: $sdk_version] [msg: $error_msg] [traceID: $trace_id] [timeCost: $time_cost]"
|
||||
format: "[$time_now] [ACCESS] <$user_name: $user_addr> $method_name [status: $method_status] [code: $error_code] [sdk: $sdk_version] [msg: $error_msg] [traceID: $trace_id] [timeCost: $time_cost] [error_type: $error_type]"
|
||||
query:
|
||||
format: "[$time_now] [ACCESS] <$user_name: $user_addr> $method_name [status: $method_status] [code: $error_code] [sdk: $sdk_version] [msg: $error_msg] [traceID: $trace_id] [timeCost: $time_cost] [database: $database_name] [collection: $collection_name] [partitions: $partition_name] [expr: $method_expr] [params: $query_params]"
|
||||
format: "[$time_now] [ACCESS] <$user_name: $user_addr> $method_name [status: $method_status] [code: $error_code] [sdk: $sdk_version] [msg: $error_msg] [traceID: $trace_id] [timeCost: $time_cost] [database: $database_name] [collection: $collection_name] [partitions: $partition_name] [expr: $method_expr] [params: $query_params] [error_type: $error_type]"
|
||||
methods: "Query, Delete, /query, /v2/vectordb/entities/query, /v2/vectordb/entities/get, /v2/vectordb/entities/delete"
|
||||
search:
|
||||
format: "[$time_now] [ACCESS] <$user_name: $user_addr> $method_name [status: $method_status] [code: $error_code] [sdk: $sdk_version] [msg: $error_msg] [traceID: $trace_id] [timeCost: $time_cost] [database: $database_name] [collection: $collection_name] [partitions: $partition_name] [expr: $method_expr] [nq: $nq] [params: $search_params]"
|
||||
format: "[$time_now] [ACCESS] <$user_name: $user_addr> $method_name [status: $method_status] [code: $error_code] [sdk: $sdk_version] [msg: $error_msg] [traceID: $trace_id] [timeCost: $time_cost] [database: $database_name] [collection: $collection_name] [partitions: $partition_name] [expr: $method_expr] [nq: $nq] [params: $search_params] [error_type: $error_type]"
|
||||
methods: "HybridSearch, Search, /search, /v2/vectordb/entities/search, /v2/vectordb/entities/advanced_search, /v2/vectordb/entities/hybrid_search"
|
||||
upsert:
|
||||
format: "[$time_now] [ACCESS] <$user_name: $user_addr> $method_name [status: $method_status] [code: $error_code] [sdk: $sdk_version] [msg: $error_msg] [traceID: $trace_id] [timeCost: $time_cost] [database: $database_name] [collection: $collection_name] [partitions: $partition_name] [partial_update: $partial_update]"
|
||||
format: "[$time_now] [ACCESS] <$user_name: $user_addr> $method_name [status: $method_status] [code: $error_code] [sdk: $sdk_version] [msg: $error_msg] [traceID: $trace_id] [timeCost: $time_cost] [database: $database_name] [collection: $collection_name] [partitions: $partition_name] [partial_update: $partial_update] [error_type: $error_type]"
|
||||
methods: Upsert
|
||||
cacheSize: 0 # Size of log of write cache, in byte. (Close write cache if size was 0)
|
||||
cacheFlushInterval: 3 # time interval of auto flush write cache, in seconds. (Close auto flush if interval was 0)
|
||||
|
||||
@@ -4794,7 +4794,7 @@
|
||||
"uid": "${datasource}"
|
||||
},
|
||||
"exemplar": true,
|
||||
"expr": "sum(increase(milvus_proxy_req_count{app_kubernetes_io_instance=~\"$instance\", app_kubernetes_io_name=\"$app_name\", namespace=\"$namespace\", status=\"fail\"}[2m])/120) by(function_name, pod, node_id)",
|
||||
"expr": "sum(increase(milvus_proxy_req_count{app_kubernetes_io_instance=~\"$instance\", app_kubernetes_io_name=\"$app_name\", namespace=\"$namespace\", status=~\"fail_.*\"}[2m])/120) by(function_name, pod, node_id)",
|
||||
"interval": "",
|
||||
"legendFormat": "{{function_name}}-{{pod}}-{{node_id}}",
|
||||
"queryType": "randomWalk",
|
||||
|
||||
@@ -0,0 +1,314 @@
|
||||
# Error Handling Guide
|
||||
|
||||
How to produce and return errors in Milvus server code — the day-to-day how-to.
|
||||
For the underlying rules, the sentinel naming convention, and the enforcement
|
||||
roadmap, see [error_sentinel_convention.md](./error_sentinel_convention.md). For
|
||||
the canonical numeric code list, see the sentinel definitions in
|
||||
[`pkg/util/merr/errors.go`](../../pkg/util/merr/errors.go). (The
|
||||
[appendix_d_error_code.md](../developer_guides/appendix_d_error_code.md)
|
||||
appendix predates merr and lists the **deprecated** `commonpb.ErrorCode` enum,
|
||||
not the merr codes.)
|
||||
|
||||
## TL;DR
|
||||
|
||||
1. **Never** `return errors.New(...)` or `return fmt.Errorf(...)`. A linter
|
||||
rejects it (see [The one rule](#the-one-rule-never-return-a-bare-error)).
|
||||
2. An error that leaves your component must be a **typed** error carrying a code.
|
||||
The lingua franca across Milvus components is **merr**.
|
||||
3. Pick one of three forms: originate a typed merr, add context with `merr.Wrap`,
|
||||
or — only if a caller branches on it by identity — a package-level sentinel.
|
||||
|
||||
## The mental model: three kinds of error, one boundary rule
|
||||
|
||||
Milvus has three legitimate kinds of error, distinguished by **how far they
|
||||
travel**, not by syntax:
|
||||
|
||||
| Kind | Scope | Carrier | Example |
|
||||
|---|---|---|---|
|
||||
| **① merr** | across Milvus components & to clients | numeric code on the main gRPC wire | `merr.ErrCollectionNotFound` |
|
||||
| **② component-internal dialect** | between sub-modules of one big component | that component's own typed error + its own wire | `streamingutil/status.StreamingError` |
|
||||
| **③ internal sentinel** | within a single Go process | `errors.New` pointer identity, caught by `errors.Is` | `errSessionVersionCheckFailure` |
|
||||
|
||||
The single rule that ties them together:
|
||||
|
||||
> **The error you must return follows the interface's promise** — it is decided
|
||||
> by *who is on the other side of the boundary you are crossing*, not by where
|
||||
> the error was born.
|
||||
|
||||
- Crossing into **another Milvus component** (proxy → rootcoord), or returning to
|
||||
a **client**: the promise is **merr**. Translate to a typed merr at that
|
||||
boundary.
|
||||
- Staying **inside one component** (e.g. the streaming sub-modules talking to
|
||||
each other): the component may speak its own typed dialect (`StreamingError`).
|
||||
It is **not** required to be merr — but it must still be *typed*, and it gets
|
||||
translated to merr at the component's outer edge.
|
||||
- On **no** wire, ever: a **bare** `errors.New` / `fmt.Errorf`. Internal
|
||||
sentinels (kind ③) are bare, but they never reach a wire — they are caught by
|
||||
`errors.Is` and translated first.
|
||||
|
||||
### Why component-internal dialects are allowed (the StreamingError case)
|
||||
|
||||
Streaming is one big component whose sub-modules (streamingnode, streamingcoord,
|
||||
the streaming client) talk to each other constantly. They use
|
||||
`streamingutil/status.StreamingError`, which has its **own** error codes and its
|
||||
**own** gRPC encoding. That is deliberate: it is a *bounded context* with its own
|
||||
vocabulary. There is intentionally **no** global "StreamingError → merr"
|
||||
auto-converter — that would erase the dialect. Instead the conversion happens
|
||||
**once, at the consumer's boundary**, and the consumer decides how, based on
|
||||
what its own interface promises:
|
||||
|
||||
```go
|
||||
// rootcoord consumes a streaming service inside CreateCollection.
|
||||
err := s.streamingService.DoSomething(ctx, ...) // may return *StreamingError
|
||||
if err != nil {
|
||||
// (a) You care about the code the client sees → translate explicitly:
|
||||
if se := status.AsStreamingError(err); se != nil && se.IsRateLimitRejected() {
|
||||
return merr.WrapErrServiceRateLimit("streaming backpressure")
|
||||
}
|
||||
// (b) You don't care about a precise code → let it fall back at the
|
||||
// boundary (see "The safety net"); the client gets a generic
|
||||
// internal-class error. Prefer being explicit, but this is allowed.
|
||||
return err
|
||||
}
|
||||
```
|
||||
|
||||
## The one rule: never return a bare error
|
||||
|
||||
```go
|
||||
return errors.New("segment not loaded") // ❌ linter rejects
|
||||
return fmt.Errorf("segment %d not loaded", id) // ❌ linter rejects
|
||||
```
|
||||
|
||||
Why it is banned, even though the boundary would "fix it up" anyway: a bare error
|
||||
that escapes to a gRPC boundary becomes `Code=65535 (Unexpected)` — visually
|
||||
indistinguishable from "the server hit an unhandled bug". A wall of
|
||||
`errors.New("reason 1")`, `errors.New("reason 2")` is a sign nobody planned the
|
||||
error taxonomy: the caller cannot program against it, and it all collapses into
|
||||
one opaque code on the wire. A typed error costs one extra word and makes the
|
||||
failure *addressable*. The linter exists to build the habit: **the thing I
|
||||
return is always typed.**
|
||||
|
||||
## Decision tree: what should I return?
|
||||
|
||||
```
|
||||
Am I crossing into another component / returning to a client?
|
||||
│
|
||||
├─ No (staying inside my own component)
|
||||
│ ├─ My component has its own typed dialect (e.g. streaming)?
|
||||
│ │ → use that dialect's factory (status.New*), not merr, not errors.New
|
||||
│ └─ Otherwise → a typed merr (rules below); or, only if a caller in THIS
|
||||
│ process branches on the outcome by identity, a package-level sentinel (§3.3)
|
||||
│
|
||||
└─ Yes → I must return a typed merr:
|
||||
├─ Brand-new failure, no underlying error worth carrying?
|
||||
│ → merr.WrapErrXxxMsg("detail %s", v) (§3.1 originate)
|
||||
│ and pick Input vs System deliberately (next section)
|
||||
├─ I hold an underlying error and want to KEEP its code, just add context?
|
||||
│ → merr.Wrap(err, "while doing X") (§3.2 add context)
|
||||
└─ I hold an underlying error and want to DOWNGRADE it to a generic class?
|
||||
→ merr.WrapErrServiceInternalErr(err, "...") (§3.2 — deliberate override)
|
||||
```
|
||||
|
||||
## Input vs System: who is to blame?
|
||||
|
||||
Every merr is classified as **InputError** (the request author's fault) or
|
||||
**SystemError** (Milvus's fault, the default). Choosing a factory chooses the
|
||||
classification, so when you originate an error, ask one question first:
|
||||
|
||||
> **Would a correctly implemented Milvus ever hit this branch, given this
|
||||
> request?** If the request content itself triggers it → InputError. If
|
||||
> reaching this branch means a Milvus bug or internal failure → SystemError.
|
||||
|
||||
Quick rules for the cases that get misclassified in practice:
|
||||
|
||||
- A **plan / task type / request produced by a coordinator** is not user input.
|
||||
An unrecognized task type or malformed compaction plan is an internal
|
||||
protocol violation (think mixed-version rolling upgrade) →
|
||||
`WrapErrServiceInternalMsg`, even though the check looks like validation.
|
||||
- **Data produced by segcore or another internal component** is not user
|
||||
input. A violated data-shape contract (ValidData length, truncated vectors)
|
||||
is a Milvus bug → `WrapErrServiceInternalMsg`.
|
||||
- A **TOCTOU race** (state was valid at check time, changed by execution time)
|
||||
is not user input → keep it a system error.
|
||||
|
||||
### How classification is attached
|
||||
|
||||
Two mechanisms, used in different situations:
|
||||
|
||||
1. **Baked-in sentinels.** ~25 sentinels are declared
|
||||
`WithErrorType(InputError)` in `errors.go` (`ErrParameterInvalid/Missing/
|
||||
TooLarge`, `ErrPrivilegeNotPermitted`, `ErrDatabaseInvalidName`, ...).
|
||||
Using their factory *is* the classification — which is exactly why reaching
|
||||
for `WrapErrParameterInvalidMsg` to express an internal assertion is wrong.
|
||||
2. **Boundary marking** for dual-use sentinels. `ErrCollectionNotFound` stays
|
||||
SystemError (internal refresh/retry paths depend on that), and the proxy
|
||||
boundary stamps it InputError only where the name came from the user:
|
||||
|
||||
```go
|
||||
// proxy meta cache, the central chokepoint for user-supplied names:
|
||||
return collection, merr.WrapErrAsInputErrorWhen(err,
|
||||
merr.ErrCollectionNotFound, merr.ErrDatabaseNotFound)
|
||||
```
|
||||
|
||||
`WrapErrAsInputError(err)` marks unconditionally;
|
||||
`WrapErrAsInputErrorWhen(err, targets...)` marks only if the error's code
|
||||
matches a target. Both preserve `errors.Is` and the code — they relabel the
|
||||
classification, nothing else.
|
||||
|
||||
### What the classification drives
|
||||
|
||||
| Surface | InputError behavior |
|
||||
|---|---|
|
||||
| `commonpb.Status` | `ExtraInfo["is_input_error"]="true"`, `Retriable` forced `false` |
|
||||
| Prometheus | request counted as `fail_input` / `rejected_user` (vs `fail_system` / `rejected_system`) |
|
||||
| Access log / failure log | `error_type` field set accordingly |
|
||||
| proxy lb_policy | **no cross-replica failover** — retrying a bad request elsewhere can't help |
|
||||
| `retry.Do` | aborts immediately instead of retrying |
|
||||
|
||||
The last two rows are why misclassification is not cosmetic: marking an
|
||||
internal failure as InputError disables the retry/failover machinery that
|
||||
would have healed it, and a dashboard blames users for Milvus bugs.
|
||||
|
||||
### Pitfalls (each of these happened)
|
||||
|
||||
- **Don't mark a shared sentinel InputError globally** to fix one callsite —
|
||||
every internal `retry.Do` loop waiting on that error stops retrying. Use
|
||||
boundary marking instead.
|
||||
- **Don't classify in a helper** what only the boundary can know. The same
|
||||
not-found is the user's fault when the name came from a request, and a
|
||||
system fault when it came from internal state — stamp at the chokepoint
|
||||
where the origin is known.
|
||||
- **"Looks like validation" is not the test.** Coordinator-to-node protocol
|
||||
checks, segcore output checks, and cgo boundary checks all look like
|
||||
validation; none of them are user input.
|
||||
|
||||
## The three correct ways
|
||||
|
||||
### 3.1 Originate a typed error — `WrapErrXxxMsg` / `WrapErrXxx`
|
||||
|
||||
When the failure starts here and there is no inner error worth preserving:
|
||||
|
||||
```go
|
||||
return merr.WrapErrParameterInvalidMsg("nq (%d) exceeds the limit (%d)", nq, max)
|
||||
return merr.WrapErrCollectionNotFound(collectionName)
|
||||
```
|
||||
|
||||
Pick the sentinel whose **code** matches the failure's meaning (see the sentinel
|
||||
definitions in [`pkg/util/merr/errors.go`](../../pkg/util/merr/errors.go)). This is the common
|
||||
case: most of the time you only need to attach a message to a well-chosen code,
|
||||
and the framework does the rest.
|
||||
|
||||
### 3.2 Add context but keep the code — `merr.Wrap`, never `WrapErr*Err`
|
||||
|
||||
When you already hold a typed error and only want to add a breadcrumb:
|
||||
|
||||
```go
|
||||
if err := s.loadSegment(ctx, id); err != nil {
|
||||
return merr.Wrap(err, "while loading sealed segment") // ✅ keeps the inner code
|
||||
}
|
||||
```
|
||||
|
||||
`merr.Wrap` / `merr.Wrapf` is a thin wrapper (like `errors.Wrap`): it prepends
|
||||
context and **preserves** the inner error's code and its `errors.Is` chain.
|
||||
|
||||
⚠️ Do **not** reach for `merr.WrapErrServiceInternalErr(err, ...)` (or any
|
||||
`merr.WrapErrXxxErr`) just to add context. Those build a
|
||||
`wrappedMilvusError{sentinel: ErrServiceInternal}` whose `code()` returns the
|
||||
**outer** sentinel's code — they **overwrite** the inner typed code with
|
||||
ServiceInternal (5) and force it non-retriable (`As()` resolves to the outer
|
||||
sentinel). The `errors.Is` chain itself survives via `Unwrap()`, but the typed
|
||||
code and retriability are masked. `WrapErrXxxErr` is *only* for
|
||||
when you **deliberately** want to relabel the inner error to a new code (e.g.
|
||||
collapse a noisy internal failure into one ServiceInternal for the client).
|
||||
|
||||
This split is intentional, and the framework deliberately does **not** try to be
|
||||
clever: *keep the code* versus *downgrade the code* is a decision you state, not
|
||||
one the framework guesses. `merr.Wrap` **always** keeps the inner code;
|
||||
`WrapErrXxxErr` **always** relabels to the outer one. A helper that "smartly"
|
||||
preserved the inner code whenever it recognized a typed merr would blur the
|
||||
intent — a reader could no longer tell from the call site whether the author
|
||||
meant to preserve or to downgrade. The choice of helper *is* the statement of
|
||||
intent.
|
||||
|
||||
### 3.2.1 The base-package case: pass through, wrap, or relabel?
|
||||
|
||||
Low-level packages (`pkg/...`, `internal/util/...`) sit under many callers and
|
||||
usually receive an error from something even lower — etcd, S3, a third-party
|
||||
library, another util. For every such error you must consciously pick one of
|
||||
three. Getting this wrong in a base package is expensive: it is multiplied across
|
||||
every caller.
|
||||
|
||||
| Choice | Use it when | How |
|
||||
|---|---|---|
|
||||
| **Pass through** the original `err` | the inner err is already a typed error meaningful to your caller, or your package has no business classifying it — let the caller decide | `return err` |
|
||||
| **Wrap, keep the code** | you want to add a breadcrumb (which key/path/op failed) without changing what the error *means* | `merr.Wrapf(err, "etcd txn on key %s", k)` |
|
||||
| **Relabel / downgrade** | the inner err is a leaky implementation / third-party detail your caller should not see; you translate it into the typed merr your package's interface promises | `merr.WrapErrXxxErr(err, "...")` |
|
||||
|
||||
Deciding questions, in order:
|
||||
|
||||
1. **Is the inner err already typed and meaningful to my caller?** → pass through.
|
||||
2. **Does any caller `errors.Is` the inner err's identity?** → pass through or
|
||||
`merr.Wrap` (both preserve the chain); **never relabel** — it hides the chain.
|
||||
3. **Is the inner err a third-party / implementation detail I promise to hide?**
|
||||
→ relabel to the typed merr my interface promises.
|
||||
|
||||
A base package that relabels too eagerly destroys codes the upper layers needed;
|
||||
one that passes a raw third-party error straight through leaks an untyped error
|
||||
toward the boundary. Neither is acceptable — the choice must be deliberate, and it
|
||||
follows your package's interface promise, not convenience.
|
||||
|
||||
### 3.3 Need identity branching or reuse — a package-level sentinel
|
||||
|
||||
Define a sentinel when a caller **in the same process** branches on the outcome
|
||||
by identity, via `errors.Is`:
|
||||
|
||||
```go
|
||||
// internal/util/sessionutil/session_util.go — caught by isNotSessionVersionCheckFailure
|
||||
// and used as a retry.Do predicate; the identity must survive, so it stays a
|
||||
// bare sentinel rather than a merr error.
|
||||
var errSessionVersionCheckFailure = errors.New("session version check failure")
|
||||
```
|
||||
|
||||
Rules for sentinels (full version in
|
||||
[error_sentinel_convention.md](./error_sentinel_convention.md)):
|
||||
|
||||
- **Package-level, never function-local.** A local `x := errors.New(...)` is a
|
||||
refactor hazard — tomorrow it gets hoisted and silently crosses a boundary.
|
||||
Lift it to a `var` at package scope.
|
||||
- **Lowercase / unexported** when it lives in `internal/...`. If a
|
||||
*cross-package* caller needs the signal, do **not** export the sentinel —
|
||||
redesign the API to carry the signal in a return value (e.g.
|
||||
`(ignored bool, err error)`).
|
||||
- It must be `errors.Is`-caught and translated to a typed merr (or
|
||||
`merr.Success()`) **before** crossing any gRPC boundary. A sentinel that
|
||||
reaches the wire is just an opaque `Code=65535`.
|
||||
|
||||
When unsure between 3.1 and 3.3: if nobody does `errors.Is` on it, you don't need
|
||||
a sentinel — just originate a typed merr with a message (3.1).
|
||||
|
||||
## The safety net: boundary fallback (and why not panic)
|
||||
|
||||
If a non-typed error does reach a gRPC handler, `merr.Status(err)` falls back to
|
||||
`Code=65535 (Unexpected)`. This is a **backstop, not a feature**: it keeps the
|
||||
server from leaking internals or crashing, but the client gets an opaque code.
|
||||
Treat any `Code=65535` in logs as "someone forgot to type their error".
|
||||
|
||||
Why the boundary **falls back instead of panicking**: third-party libraries and
|
||||
deep call chains produce errors on paths that tests cannot fully cover. Panicking
|
||||
on "not a typed merr" would turn a stray untyped error into an outage. The
|
||||
contract is therefore: **fall back to a generic code, never panic.** The goal of
|
||||
the linter and this guide is to make that fallback path *empty in practice* — so
|
||||
that every error a client sees was deliberately typed at its source.
|
||||
|
||||
## Enforcement
|
||||
|
||||
- A `gocritic`/`ruleguard` rule (`rawmerrerror` in `rules.go`) rejects
|
||||
`return errors.New / fmt.Errorf / errors.Errorf` from function bodies. It runs
|
||||
under `make verifiers` (via `static-check`), so no extra command is needed.
|
||||
Exempt paths (run outside the request path): `*_test.go`, `cmd/`, `tests/`,
|
||||
codegen, the walimpls harness, and `/mocks/` (generated mock helpers are
|
||||
test infrastructure even though some lack a "Code generated" header).
|
||||
- It catches the **direct-return** form — the one that lets a raw error escape to
|
||||
a boundary. Assignment-then-return escapes (`e := errors.New(); return e`) and
|
||||
the full no-exceptions ban require the AST-based linter described as "Tier 2"
|
||||
in [error_sentinel_convention.md](./error_sentinel_convention.md).
|
||||
@@ -0,0 +1,324 @@
|
||||
# Error Sentinel Convention
|
||||
|
||||
Milvus error handling has two distinct layers. This document describes the rules
|
||||
that separate them, the rationale, and an audit of where the codebase currently
|
||||
violates them.
|
||||
|
||||
## The two layers
|
||||
|
||||
### 1. Typed merr (wire-protocol errors)
|
||||
|
||||
Defined in `pkg/util/merr/errors.go` (`ErrCollectionNotFound`,
|
||||
`ErrParameterInvalid`, etc.). These carry a numeric error code that is
|
||||
serialized into `commonpb.Status{ErrorCode, Reason}` and shipped to the
|
||||
client over gRPC. They are the only thing a client (or another Milvus
|
||||
component on the receiving end of an RPC) sees.
|
||||
|
||||
Creation: `merr.WrapErrXxxMsg(...)` / `merr.WrapErrXxxErr(cause, ...)`
|
||||
(origination only — never to add context to a typed merr that already exists,
|
||||
see `merr.Wrap`).
|
||||
|
||||
### 2. Internal sentinels (single-process control flow)
|
||||
|
||||
Created with `errors.New(...)` at package scope (e.g. `errIgnoredAlterAlias`,
|
||||
`errReleaseCollectionNotLoaded`, `errNodeNotEnough`). These are signaling
|
||||
vocabulary inside a single Go process: a callee tells its caller "this is an
|
||||
idempotent no-op" or "the queue is empty" so the caller can branch / retry /
|
||||
ignore. They are **not** part of the wire protocol.
|
||||
|
||||
The catcher is always `errors.Is(err, sentinelX)` at some boundary in the
|
||||
calling stack, and the boundary either:
|
||||
|
||||
- translates to `merr.Success()` (idempotency: e.g. drop something that
|
||||
doesn't exist → success), or
|
||||
- translates to a typed merr `merr.WrapErrXxxMsg(...)` (e.g. user already
|
||||
exists → `WrapErrParameterInvalid`).
|
||||
|
||||
## The hard invariant
|
||||
|
||||
> **Any internal sentinel must be `errors.Is`-caught and translated to a
|
||||
> typed merr (or `Success`) before crossing any gRPC handler boundary —
|
||||
> client-facing or component-to-component.**
|
||||
|
||||
Why "any gRPC boundary, not just client-facing": gRPC serializes errors to
|
||||
`commonpb.Status{ErrorCode, Reason}`. The Go-level pointer identity that
|
||||
`errors.New` sentinels rely on does not survive the wire. The peer's
|
||||
`merr.Error(status)` reconstructs a typed merr from the numeric code; the
|
||||
sentinel chain is gone forever. So a sentinel that escapes an internal coord
|
||||
RPC is just as broken as one that escapes a user-facing RPC — only quieter,
|
||||
because no customer sees the resulting `Code=65535 (unexpected)`.
|
||||
|
||||
### What breaks the invariant
|
||||
|
||||
The `errors.Is` chain survives `return err`, `errors.Wrap(err, "...")` /
|
||||
`merr.Wrap(err, "...")` (cockroachdb thin wrap), **and**
|
||||
`merr.WrapErrServiceInternalErr(err, "...")` — `milvusError.Unwrap()`
|
||||
returns the inner error, so `errors.Is(outer, innerSentinel)` stays true through any
|
||||
of them. It is **destroyed** only by:
|
||||
|
||||
- Putting the cause in a format argument instead of the chain:
|
||||
`merr.WrapErrXxxMsg("...: %s", err)`, or the `%w` mistake (`WrapErr*Msg`
|
||||
formats with `fmt.Sprintf`, which does **not** honor `%w` and renders
|
||||
`%!w(...)`). The inner error reaches the message text but is unreachable via
|
||||
`Unwrap()`, so `errors.Is(outer, innerSentinel)` returns false.
|
||||
- Any custom wrapper that doesn't implement `Unwrap()`.
|
||||
|
||||
Do **not** confuse this with the separate code/retriability rule.
|
||||
`merr.Wrap(err, ...)` and `merr.WrapErrXxxErr(err, ...)` both keep `errors.Is`
|
||||
intact, but they differ in what the result reports at the boundary:
|
||||
|
||||
- `merr.Wrap(err, ...)` preserves the inner's `Code()` and `IsRetryable` — the
|
||||
chain still resolves to the inner's `*milvusError`.
|
||||
- `merr.WrapErrXxxErr(err, ...)` reports the **outer** sentinel's code and
|
||||
retriability: `As()` resolves to `ErrServiceInternal` (Code 5,
|
||||
non-retriable), **masking** the inner's classification.
|
||||
|
||||
So there are two distinct rules, often conflated:
|
||||
|
||||
1. To keep `errors.Is` working: never stuff the cause into a format string;
|
||||
pass it as the error argument.
|
||||
2. To add context **without changing the classification** (preserve the inner
|
||||
code + retriability): use `merr.Wrap`, not `merr.WrapErr*Err`. Reserve
|
||||
`merr.WrapErrXxxErr` for when you intend to *assert* a new classification
|
||||
(e.g. this genuinely is a service-internal error). (See also
|
||||
[feedback rule](#related-rules) on `merr.Wrap` vs `merr.WrapErr*Err`.)
|
||||
|
||||
## Naming convention
|
||||
|
||||
The convention has two layers, matched to the two error categories:
|
||||
|
||||
### Wire-protocol layer (typed merr) — uppercase `Err*` in `pkg/util/merr` only
|
||||
|
||||
All errors that may cross any gRPC boundary (client-facing or
|
||||
component-to-component) must be `*merr.milvusError` defined in
|
||||
`pkg/util/merr/errors.go`. They have:
|
||||
|
||||
- A numeric code passed to `newMilvusError(...)`; uniqueness is enforced by
|
||||
the init-time code registry (defining a second sentinel on an occupied code
|
||||
panics at package init, since `milvusError.Is` matches by code alone)
|
||||
- A `var ErrXxx = newMilvusError(...)` declaration in `pkg/util/merr/errors.go`
|
||||
- An exported `WrapErrXxxMsg` / `WrapErrXxxErr` helper
|
||||
|
||||
If an error needs to be visible to the wire, it lives here. No exceptions.
|
||||
|
||||
Every sentinel also carries an Input-vs-System classification (who is to
|
||||
blame) that drives `Status.Retriable`, the `fail_input`/`fail_system` metric
|
||||
labels, lb_policy failover and `retry.Do`; see "Input vs System: who is to
|
||||
blame?" in [error_handling_guide.md](error_handling_guide.md).
|
||||
|
||||
### Internal-sentinel layer — lowercase `errXxx`, same-package only
|
||||
|
||||
Internal sentinels live in `internal/...` packages, are created with
|
||||
`errors.New(...)`, and are **lowercase / unexported**. The rule:
|
||||
|
||||
> **A `var err* = errors.New(...)` declared in `internal/...` may only be
|
||||
> referenced inside the same Go package. Cross-package consumers must not
|
||||
> see it.**
|
||||
|
||||
This makes Go visibility do the enforcement: if you need a signal across
|
||||
package boundaries, you either (a) lift it into the wire layer as a typed
|
||||
merr, or (b) redesign the API so the signal flows via a return value
|
||||
(e.g. `(ignored bool, err error)`), not via the error type.
|
||||
|
||||
Example (current code, after the 04-coord cleanup):
|
||||
|
||||
```go
|
||||
// errFull / errNoSuchElement are INTERNAL sentinels: caught by errors.Is
|
||||
// inside the compaction inspector / scheduler loop and never serialized
|
||||
// across any gRPC boundary.
|
||||
var (
|
||||
errFull = errors.New("compaction queue is full")
|
||||
errNoSuchElement = errors.New("compaction queue has no element")
|
||||
)
|
||||
```
|
||||
|
||||
### Cross-package idempotency: use a return-value flag, not an exported sentinel
|
||||
|
||||
The previous code exported `meta.ErrResourceGroupOperationIgnored` so that
|
||||
the parent package `querycoordv2` could catch it via `errors.Is` and
|
||||
translate to `merr.Success()`. This was an exported `Err*` in
|
||||
`internal/...` — visually indistinguishable from a `merr.ErrXxx` typed
|
||||
wire error, easy to misuse.
|
||||
|
||||
The current code instead encodes the signal in the return value:
|
||||
|
||||
```go
|
||||
// meta/resource_manager.go
|
||||
func (rm *ResourceManager) CheckIfResourceGroupAddable(...) (ignored bool, err error) {
|
||||
if proto.Equal(rm.groups[rgName].GetConfig(), cfg) {
|
||||
return true, nil // idempotent no-op
|
||||
}
|
||||
...
|
||||
}
|
||||
|
||||
// querycoordv2/ddl_callbacks_alter_resource_group.go (broadcaster)
|
||||
func (s *Server) broadcastCreateResourceGroup(...) (ignored bool, err error) {
|
||||
if ignored, err := s.meta.CheckIfResourceGroupAddable(...); err != nil || ignored {
|
||||
return ignored, err
|
||||
}
|
||||
...
|
||||
}
|
||||
|
||||
// querycoordv2/services.go (RPC handler)
|
||||
ignored, err := s.broadcastCreateResourceGroup(ctx, req)
|
||||
if err != nil { return merr.Status(err), nil }
|
||||
if ignored { return merr.Success(), nil }
|
||||
```
|
||||
|
||||
No sentinel crosses the package boundary; the signal travels via a
|
||||
structured return value. This is the preferred pattern for any new
|
||||
cross-package idempotency case.
|
||||
|
||||
## Current state (audit done on err-std-04-coord branch, 2026-05-19)
|
||||
|
||||
Across `internal/{datacoord,rootcoord,querycoordv2}` there are 28
|
||||
`errors.New(...)` sentinels. Their fates:
|
||||
|
||||
| Kind | Count | Examples | Status |
|
||||
|---|---|---|---|
|
||||
| Idempotency: caught → `merr.Success()` | 13 catch sites, ~12 distinct sentinels | `errIgnoredAlterAlias`, `errIgnoredCreateCollection`, `errReleaseCollectionNotLoaded`, `errUserNotFound`, ... | ✅ compliant |
|
||||
| Caught → translated to typed merr | 3 catch sites (`errUserAlreadyExists`, `errRoleAlreadyExists`, `errRoleNotExists`) | client gets `WrapErrParameterInvalidMsg(...)` or `WrapErrServiceInternalMsg(...)` | ✅ compliant (1100 / 5) |
|
||||
| Background-only (never enter an RPC handler) | 5 (`errFull`, `errNoSuchElement`, `errNodeNotEnough`, `errDisposed`, `errTypeNotFound`) | compaction queue / resource observer / session lifecycle / checker registry | ✅ compliant |
|
||||
| Cross-package idempotency via `(ignored bool, err error)` signature | 1 (resource group create/drop) | meta layer returns `ignored=true`; querycoordv2 RPC handler translates to `merr.Success()` — no sentinel crosses package | ✅ compliant (refactored from exported `ErrResourceGroupOperationIgnored` in this branch) |
|
||||
| Dead code (function with 0 callers) | 3 (`errNilResponse`, `errNilStatusResponse`, `errUnknownResponseType` — all only used by `VerifyResponse` in `datacoord/util.go`) | safe to delete | 🪦 cleanup candidate |
|
||||
| ~~**Violation**: escapes RPC handler with no catch~~ (resolved) | 3 (`errEmptyUsername`, `errEmptyRoleName`, `errEmptyPrivilegeGroupName`) | origin sites in `meta_table.go` now emit `WrapErrParameterInvalidMsg` directly; the bare sentinels are gone | ✅ resolved |
|
||||
| ~~**Semantic miscategorization**: caught and wrapped with the wrong typed merr code~~ (resolved) | 1 (`errTypeNotFound` in `ops_services.go:87,101`) | invalid `CheckerID` from the client now wrapped as `WrapErrParameterInvalidMsg` (code 1100), was `WrapErrServiceInternal` (code 5) | ✅ resolved |
|
||||
|
||||
### Cleanup status
|
||||
|
||||
1. ✅ Done — `errEmptyUsername` / `errEmptyRoleName` / `errEmptyPrivilegeGroupName`
|
||||
at the origin sites in `internal/rootcoord/meta_table.go` now emit
|
||||
`merr.WrapErrParameterInvalidMsg("username is empty")` etc. directly; the bare
|
||||
sentinels no longer exist.
|
||||
2. ✅ Done — `errTypeNotFound` at the catcher in
|
||||
`internal/querycoordv2/ops_services.go:87,101` is now wrapped as
|
||||
`merr.WrapErrParameterInvalidMsg("invalid checker type %d: %v", req.CheckerID, err)`.
|
||||
3. Not done — delete `VerifyResponse` and its three dead-code sentinels
|
||||
(`errNilResponse`, `errNilStatusResponse`, `errUnknownResponseType`).
|
||||
|
||||
## Future linter ideas
|
||||
|
||||
Three candidates, in order of how cheap they are to implement and how
|
||||
hard the enforcement is. **Tier 1.5's return form is now implemented (see
|
||||
below); Tier 1 and Tier 2 remain a design queue.**
|
||||
|
||||
### Tier 1 — exported-sentinel ban (1 hour to write)
|
||||
|
||||
The simplest rule: **`internal/...` packages may not declare exported
|
||||
`var Err\w+ = errors.New(...)`.** Scan all `internal/...` *.go files,
|
||||
fail CI if any match. Two paths to fix a violation:
|
||||
|
||||
1. Lowercase it (`var errXxx = errors.New(...)`) — only callable inside the
|
||||
same package. If the lint fails because a cross-package caller needs the
|
||||
signal, see fix 2.
|
||||
2. Refactor the API so the signal travels via a return value
|
||||
(e.g. add `ignored bool` to the return tuple) and delete the sentinel.
|
||||
|
||||
This makes Go visibility itself the enforcement mechanism: anything that
|
||||
needs to look like `merr.ErrXxx` to a reviewer can only exist in
|
||||
`pkg/util/merr`. Internal sentinels stay quietly lowercase in their owning
|
||||
package.
|
||||
|
||||
Lowercase sentinels (`var errXxx = errors.New(...)`) inside `internal/...`
|
||||
are still encouraged to carry an `INTERNAL: ...` doc comment for reviewer
|
||||
context, but it's not enforced — the visibility rule already prevents the
|
||||
worst-case (`Err*` collision with `merr.ErrXxx`).
|
||||
|
||||
### Tier 1.5 — bare-usage ban (1 hour grep, half-day AST for 100% precision)
|
||||
|
||||
> **Status — return form implemented.** The **return** form of this ban is now
|
||||
> enforced by a `gocritic`/`ruleguard` rule (`rawmerrerror` in `rules.go`), run
|
||||
> under `make verifiers`: it rejects `return errors.New / fmt.Errorf /
|
||||
> errors.Errorf` from function bodies (package-level sentinels, `cmd/`, `tests/`,
|
||||
> codegen and the walimpls harness exempt). The day-to-day guide is
|
||||
> [error_handling_guide.md](./error_handling_guide.md). The **no-exceptions**
|
||||
> form below (local `:=`, `panic(...)`, function argument) is *not* covered:
|
||||
> ruleguard's DSL cannot match "a call anywhere in a function body but not in a
|
||||
> `ValueSpec`", so the full ban still needs the AST-based Tier 2 linter.
|
||||
|
||||
**Hardest enforcement, no exceptions.** `internal/...` packages may not
|
||||
use `errors.New(...)` or `errors.Errorf(...)` **inline inside a function
|
||||
body**. The only legal site for these calls is a package-level
|
||||
`var <Name> = errors.New(...)` (sentinel declaration).
|
||||
|
||||
Allow/deny matrix:
|
||||
|
||||
| Form | Location | Verdict |
|
||||
|---|---|---|
|
||||
| `var errInvalid = errors.New("invalid")` | package-level (file top / `var` block) | ✅ allowed |
|
||||
| `var ( errA = ...; errB = ... )` | package-level `var` block | ✅ allowed |
|
||||
| `return errors.New(...)` | function body | ❌ banned |
|
||||
| `x := errors.New(...)` | function body local | ❌ banned |
|
||||
| `panic(errors.New(...))` | function body | ❌ banned |
|
||||
| `foo(errors.New(...))` | function body argument | ❌ banned |
|
||||
|
||||
**Why no exceptions** (even for "local break signal" / "log-only" /
|
||||
"panic-bound" cases that look harmless today):
|
||||
|
||||
- Today's local var can be hoisted to package-level by tomorrow's refactor
|
||||
and silently start crossing boundaries.
|
||||
- A linter with exceptions needs AST-level wire-reachability analysis
|
||||
(expensive); a no-exception linter is one grep.
|
||||
- Forces authors to use the right primitive instead of `errors.New` as
|
||||
a universal escape hatch:
|
||||
- **break signal from a callback** → define a `type doneSignal struct{}`
|
||||
that implements `error` and use `errors.As`. Intent is now in the type,
|
||||
not in a string-keyed sentinel.
|
||||
- **"unreachable" assertion** → just `panic(...)`. If caller already
|
||||
does `if err != nil { panic(err) }`, fold it into the callee.
|
||||
- **input validation / config validation** → `merr.WrapErrParameterInvalidMsg(...)`
|
||||
or `status.NewInvalidArgument(...)` depending on layer.
|
||||
|
||||
Implementation: grep version covers ~95% true violations in ~1 hour.
|
||||
AST version (go/analysis) covers the edge cases (e.g. `init()` body
|
||||
assigning to a package var) but needs ~half a day. Start with grep,
|
||||
upgrade if false-positive rate exceeds 5%.
|
||||
|
||||
```bash
|
||||
# grep skeleton
|
||||
grep -rnE 'errors\.(New|Errorf)\(' internal/ --include='*.go' \
|
||||
| grep -v _test.go \
|
||||
| grep -vE ':[0-9]+:\s*(var\s+)?[A-Za-z_]+\s*=\s*errors\.(New|Errorf)' \
|
||||
| grep -vE ':[0-9]+:\s*[A-Za-z_]+\s+(\w+\s+)?=\s*errors\.(New|Errorf)'
|
||||
# Any remaining line = violation
|
||||
```
|
||||
|
||||
A `//nolint:err-bare` escape valve with a required justification comment
|
||||
handles the genuine outliers (a few `init()` patterns, embedded `errors.Mark`
|
||||
usage, etc.).
|
||||
|
||||
### Tier 2 — escape-path linter (~1 day, go/analysis based)
|
||||
|
||||
For every gRPC handler method (anything matching the
|
||||
`internal/{rootcoord,datacoord,querycoordv2}/services.go,root_coord.go,*_handler.go`
|
||||
pattern, return type `(*proto.XxxResponse, error)` or `(*commonpb.Status,
|
||||
error)`), trace the err-return data-flow. Any error that:
|
||||
|
||||
- transitively originates from an `INTERNAL:`-tagged sentinel, **and**
|
||||
- reaches a `return Status{Code: merr.Status(err)}` or `return err` without
|
||||
passing through an `errors.Is(err, internalSentinelX) { ... }` branch,
|
||||
|
||||
is a violation. Report file:line of the leak.
|
||||
|
||||
This catches the actual invariant violation (the 3 RBAC empty sentinels
|
||||
would have been flagged), not just naming hygiene. Requires AST analysis;
|
||||
worth doing if the cost of one more silent `Code=1` to a client is high.
|
||||
|
||||
### Tier 3 (no longer necessary if Tier 2 is in place) — wrap-rule linter
|
||||
|
||||
Scan `merr\.WrapErr[A-Za-z]+Err\(err, ` (note: first arg `err`, not a
|
||||
fresh string-only origination) and require the cause `err` to not itself be
|
||||
a typed merr. This is what
|
||||
[`feedback_merr_wrap_rule`](../../) already enforces by convention; Tier 2
|
||||
would catch the symptom (sentinel escape) as a side effect.
|
||||
|
||||
## Related rules
|
||||
|
||||
- `feedback_merr_wrap_rule` (this repo's collaboration memory): "Add context
|
||||
to an existing err with `merr.Wrap` / `merr.Wrapf`, never with
|
||||
`merr.WrapErr*Err` — the latter masks the inner typed code and
|
||||
retriability (the `errors.Is` chain itself is preserved via `Unwrap()`)."
|
||||
- `project_errstd_autogen_defects`: three systematic defects in the
|
||||
auto-generated `errors.Wrap → merr` conversion in this branch series;
|
||||
defects #2 and #3 are direct consequences of violating the rules in
|
||||
this document.
|
||||
@@ -1,5 +1,14 @@
|
||||
## Appendix D. Error Code
|
||||
|
||||
> **⚠️ Deprecated.** This appendix documents the legacy `commonpb.ErrorCode`
|
||||
> enum (Milvus 2.0-era), which is no longer the source of truth for error codes.
|
||||
> Current error codes are the `merr` sentinel codes (e.g. `ParameterInvalid`
|
||||
> = 1100, `ServiceInternal` = 5, `FunctionFailed` = 2400) carried in
|
||||
> `commonpb.Status.Code`. The canonical list is the sentinel definitions in
|
||||
> [`pkg/util/merr/errors.go`](../../pkg/util/merr/errors.go); see also
|
||||
> [error_handling_guide.md](../dev/error_handling_guide.md) and
|
||||
> [error_sentinel_convention.md](../dev/error_sentinel_convention.md).
|
||||
|
||||
**ErrorCode**
|
||||
|
||||
```protobuf
|
||||
|
||||
@@ -2,9 +2,9 @@ module github.com/milvus-io/milvus/examples/telemetry_e2e_test
|
||||
go 1.26.4
|
||||
|
||||
require (
|
||||
github.com/milvus-io/milvus-proto/go-api/v2 v2.6.6-0.20260304062516-e7e54d966ef2
|
||||
github.com/milvus-io/milvus-proto/go-api/v2 v2.6.6-0.20260323081523-53649783989c
|
||||
github.com/milvus-io/milvus/client/v2 v2.6.4-0.20251104142533-a2ce70d25256
|
||||
google.golang.org/grpc v1.71.0
|
||||
google.golang.org/grpc v1.79.3
|
||||
)
|
||||
|
||||
replace (
|
||||
|
||||
@@ -1,2 +1,4 @@
|
||||
github.com/milvus-io/milvus-proto/go-api/v2 v2.6.6-0.20260304062516-e7e54d966ef2/go.mod h1:/6UT4zZl6awVeXLeE7UGDWZvXj3IWkRsh3mqsn0DiAs=
|
||||
github.com/milvus-io/milvus-proto/go-api/v2 v2.6.6-0.20260323081523-53649783989c/go.mod h1:/6UT4zZl6awVeXLeE7UGDWZvXj3IWkRsh3mqsn0DiAs=
|
||||
google.golang.org/grpc v1.71.0/go.mod h1:H0GRtasmQOh9LkFoCPDu3ZrwUtD1YGE+b2vYBYd/8Ec=
|
||||
google.golang.org/grpc v1.79.3/go.mod h1:KmT0Kjez+0dde/v2j9vzwoAScgEPx/Bw1CYChhHLrHQ=
|
||||
|
||||
+11
-10
@@ -8,6 +8,7 @@ import (
|
||||
|
||||
"github.com/milvus-io/milvus-proto/go-api/v3/schemapb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/planpb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
)
|
||||
|
||||
const (
|
||||
@@ -57,7 +58,7 @@ func isSupportedAggregateName(aggregateName string) bool {
|
||||
|
||||
func NewAggregate(aggregateName string, aggFieldID int64, originalName string, fieldType schemapb.DataType) ([]AggregateBase, error) {
|
||||
if !isSupportedAggregateName(aggregateName) {
|
||||
return nil, fmt.Errorf("invalid Aggregation operator %s", aggregateName)
|
||||
return nil, merr.WrapErrParameterInvalidMsg("invalid Aggregation operator %s", aggregateName)
|
||||
}
|
||||
|
||||
if err := ValidateAggFieldType(aggregateName, fieldType); err != nil {
|
||||
@@ -78,7 +79,7 @@ func NewAggregate(aggregateName string, aggFieldID int64, originalName string, f
|
||||
return []AggregateBase{&MaxAggregate{fieldID: aggFieldID, originalName: originalName}}, nil
|
||||
default:
|
||||
// should never happen due to isSupportedAggregateName check
|
||||
return nil, fmt.Errorf("invalid Aggregation operator %s", aggregateName)
|
||||
return nil, merr.WrapErrParameterInvalidMsg("invalid Aggregation operator %s", aggregateName)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -93,13 +94,13 @@ func FromPB(pb *planpb.Aggregate) (AggregateBase, error) {
|
||||
case planpb.AggregateOp_max:
|
||||
return &MaxAggregate{fieldID: pb.GetFieldId()}, nil
|
||||
default:
|
||||
return nil, fmt.Errorf("invalid Aggregation operator %d", pb.Op)
|
||||
return nil, merr.WrapErrParameterInvalidMsg("invalid Aggregation operator %d", pb.Op)
|
||||
}
|
||||
}
|
||||
|
||||
func AccumulateFieldValue(target *FieldValue, new *FieldValue) error {
|
||||
if target == nil || new == nil {
|
||||
return fmt.Errorf("target or new field value is nil")
|
||||
return merr.WrapErrServiceInternalMsg("target or new field value is nil")
|
||||
}
|
||||
|
||||
// Skip null values during accumulation
|
||||
@@ -121,7 +122,7 @@ func AccumulateFieldValue(target *FieldValue, new *FieldValue) error {
|
||||
}
|
||||
// Verify types match
|
||||
if reflect.TypeOf(target.val) != reflect.TypeOf(new.val) {
|
||||
return fmt.Errorf("type mismatch: target is %T, new is %T", target.val, new.val)
|
||||
return merr.WrapErrParameterInvalidMsg("type mismatch: target is %T, new is %T", target.val, new.val)
|
||||
}
|
||||
|
||||
switch target.val.(type) {
|
||||
@@ -136,7 +137,7 @@ func AccumulateFieldValue(target *FieldValue, new *FieldValue) error {
|
||||
case float64:
|
||||
target.val = target.val.(float64) + new.val.(float64)
|
||||
default:
|
||||
return fmt.Errorf("unsupported type: %T", target.val)
|
||||
return merr.WrapErrParameterInvalidMsg("unsupported type: %T", target.val)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -255,17 +256,17 @@ func (bucket *Bucket) RowCount() int {
|
||||
|
||||
func (bucket *Bucket) Accumulate(row *Row, idx int, keyCount int, aggs []AggregateBase) error {
|
||||
if idx >= len(bucket.rows) || idx < 0 {
|
||||
return fmt.Errorf("wrong idx:%d for bucket", idx)
|
||||
return merr.WrapErrParameterInvalidMsg("wrong idx:%d for bucket", idx)
|
||||
}
|
||||
targetRow := bucket.rows[idx]
|
||||
if targetRow == nil {
|
||||
return fmt.Errorf("nil row at the target idx:%d, cannot accumulate the row", idx)
|
||||
return merr.WrapErrServiceInternalMsg("nil row at the target idx:%d, cannot accumulate the row", idx)
|
||||
}
|
||||
if row.ColumnCount() != targetRow.ColumnCount() {
|
||||
return fmt.Errorf("column count:%d in the row must be equal to the target row:%d", row.ColumnCount(), bucket.rows[idx].ColumnCount())
|
||||
return merr.WrapErrParameterInvalidMsg("column count:%d in the row must be equal to the target row:%d", row.ColumnCount(), bucket.rows[idx].ColumnCount())
|
||||
}
|
||||
if row.ColumnCount() != keyCount+len(aggs) {
|
||||
return fmt.Errorf("column count:%d in the row must be sum of keyCount:%d and the number of aggs:%d", row.ColumnCount(), keyCount, len(aggs))
|
||||
return merr.WrapErrParameterInvalidMsg("column count:%d in the row must be sum of keyCount:%d and the number of aggs:%d", row.ColumnCount(), keyCount, len(aggs))
|
||||
}
|
||||
for col := keyCount; col < row.ColumnCount(); col++ {
|
||||
targetRow.UpdateFieldValue(row, col, aggs[col-keyCount])
|
||||
|
||||
@@ -2,7 +2,6 @@ package agg
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"github.com/milvus-io/milvus-proto/go-api/v3/schemapb"
|
||||
typeutil2 "github.com/milvus-io/milvus/internal/util/typeutil"
|
||||
@@ -88,15 +87,15 @@ func (reducer *GroupAggReducer) EmptyAggResult() (*AggregationResult, error) {
|
||||
} else {
|
||||
field, err := helper.GetFieldFromID(agg.GetFieldId())
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to get field schema for aggregate fieldID %d: %w", agg.GetFieldId(), err)
|
||||
return nil, merr.Wrapf(err, "failed to get field schema for aggregate fieldID %d", agg.GetFieldId())
|
||||
}
|
||||
resultType, err := getAggregateResultType(agg.GetOp(), field.GetDataType())
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to get result type for aggregate fieldID %d: %w", agg.GetFieldId(), err)
|
||||
return nil, merr.Wrapf(err, "failed to get result type for aggregate fieldID %d", agg.GetFieldId())
|
||||
}
|
||||
emptyFieldData, err := genEmptyFieldDataByType(resultType)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to generate empty field data for result type %s: %w", resultType.String(), err)
|
||||
return nil, merr.Wrapf(err, "failed to generate empty field data for result type %s", resultType.String())
|
||||
}
|
||||
ret.fieldDatas = append(ret.fieldDatas, emptyFieldData)
|
||||
}
|
||||
@@ -160,7 +159,7 @@ func genEmptyFieldDataByType(dataType schemapb.DataType) (*schemapb.FieldData, e
|
||||
return genEmptyLongFieldData(dataType, []int64{0}), nil
|
||||
default:
|
||||
// For other types, try to use the original field's GenEmptyFieldData
|
||||
return nil, fmt.Errorf("unsupported data type for aggregate result: %s", dataType.String())
|
||||
return nil, merr.WrapErrParameterInvalidMsg("unsupported data type for aggregate result: %s", dataType.String())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -187,10 +186,10 @@ func getAggregateResultType(op planpb.AggregateOp, inputType schemapb.DataType)
|
||||
case schemapb.DataType_Float, schemapb.DataType_Double:
|
||||
return schemapb.DataType_Double, nil
|
||||
default:
|
||||
return schemapb.DataType_None, fmt.Errorf("unsupported input type %s for sum aggregation", inputType.String())
|
||||
return schemapb.DataType_None, merr.WrapErrParameterInvalidMsg("unsupported input type %s for sum aggregation", inputType.String())
|
||||
}
|
||||
default:
|
||||
return schemapb.DataType_None, fmt.Errorf("unknown aggregate operator: %d", op)
|
||||
return schemapb.DataType_None, merr.WrapErrParameterInvalidMsg("unknown aggregate operator: %d", op)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -201,12 +200,12 @@ func getAggregateResultType(op planpb.AggregateOp, inputType schemapb.DataType)
|
||||
// 3. Each fieldData's Type matches the expected type from schema
|
||||
func (reducer *GroupAggReducer) validateAggregationResults(results []*AggregationResult) error {
|
||||
if reducer.schema == nil {
|
||||
return fmt.Errorf("schema is nil, cannot validate field types")
|
||||
return merr.WrapErrParameterInvalidMsg("schema is nil, cannot validate field types")
|
||||
}
|
||||
|
||||
helper, err := typeutil.CreateSchemaHelper(reducer.schema)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to create schema helper: %w", err)
|
||||
return merr.Wrap(err, "failed to create schema helper")
|
||||
}
|
||||
|
||||
numGroupingKeys := len(reducer.groupByFieldIds)
|
||||
@@ -220,7 +219,7 @@ func (reducer *GroupAggReducer) validateAggregationResults(results []*Aggregatio
|
||||
for _, fieldID := range reducer.groupByFieldIds {
|
||||
field, err := helper.GetFieldFromID(fieldID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to get field schema for groupBy fieldID %d: %w", fieldID, err)
|
||||
return merr.Wrapf(err, "failed to get field schema for groupBy fieldID %d", fieldID)
|
||||
}
|
||||
expectedTypes = append(expectedTypes, field.GetDataType())
|
||||
}
|
||||
@@ -234,11 +233,11 @@ func (reducer *GroupAggReducer) validateAggregationResults(results []*Aggregatio
|
||||
} else {
|
||||
field, err := helper.GetFieldFromID(agg.GetFieldId())
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to get field schema for aggregate fieldID %d: %w", agg.GetFieldId(), err)
|
||||
return merr.Wrapf(err, "failed to get field schema for aggregate fieldID %d", agg.GetFieldId())
|
||||
}
|
||||
expectedType, err = getAggregateResultType(agg.GetOp(), field.GetDataType())
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to get aggregate result type for aggregate fieldID %d: %w", agg.GetFieldId(), err)
|
||||
return merr.Wrapf(err, "failed to get aggregate result type for aggregate fieldID %d", agg.GetFieldId())
|
||||
}
|
||||
}
|
||||
expectedTypes = append(expectedTypes, expectedType)
|
||||
@@ -247,27 +246,27 @@ func (reducer *GroupAggReducer) validateAggregationResults(results []*Aggregatio
|
||||
// Validate each result
|
||||
for resultIdx, result := range results {
|
||||
if result == nil {
|
||||
return fmt.Errorf("result at index %d is nil", resultIdx)
|
||||
return merr.WrapErrServiceInternalMsg("result at index %d is nil", resultIdx)
|
||||
}
|
||||
|
||||
fieldDatas := result.GetFieldDatas()
|
||||
|
||||
// Check 1: fieldDatas length
|
||||
if len(fieldDatas) != expectedColumnCount {
|
||||
return fmt.Errorf("result at index %d has fieldDatas length %d, expected %d (numGroupingKeys=%d, numAggs=%d)",
|
||||
return merr.WrapErrServiceInternalMsg("result at index %d has fieldDatas length %d, expected %d (numGroupingKeys=%d, numAggs=%d)",
|
||||
resultIdx, len(fieldDatas), expectedColumnCount, numGroupingKeys, numAggs)
|
||||
}
|
||||
|
||||
// Check 2: no nil fieldData and Check 3: type matching
|
||||
for colIdx, fieldData := range fieldDatas {
|
||||
if fieldData == nil {
|
||||
return fmt.Errorf("result at index %d has nil fieldData at column %d", resultIdx, colIdx)
|
||||
return merr.WrapErrServiceInternalMsg("result at index %d has nil fieldData at column %d", resultIdx, colIdx)
|
||||
}
|
||||
|
||||
expectedType := expectedTypes[colIdx]
|
||||
actualType := fieldData.GetType()
|
||||
if actualType != expectedType {
|
||||
return fmt.Errorf("result at index %d, column %d has type %s, expected %s",
|
||||
return merr.WrapErrServiceInternalMsg("result at index %d, column %d has type %s, expected %s",
|
||||
resultIdx, colIdx, schemapb.DataType_name[int32(actualType)], schemapb.DataType_name[int32(expectedType)])
|
||||
}
|
||||
}
|
||||
@@ -352,15 +351,15 @@ func (reducer *GroupAggReducer) Reduce(ctx context.Context, results []*Aggregati
|
||||
limitReached := false
|
||||
for _, result := range results {
|
||||
if result == nil {
|
||||
return nil, fmt.Errorf("input result from any sources cannot be nil")
|
||||
return nil, merr.WrapErrServiceInternalMsg("input result from any sources cannot be nil")
|
||||
}
|
||||
reducedResult.allRetrieveCount += result.GetAllRetrieveCount()
|
||||
fieldDatas := result.GetFieldDatas()
|
||||
if outputColumnCount != len(fieldDatas) {
|
||||
return nil, fmt.Errorf("retrieved results from different segments have different size of columns")
|
||||
return nil, merr.WrapErrServiceInternalMsg("retrieved results from different segments have different size of columns")
|
||||
}
|
||||
if outputColumnCount == 0 {
|
||||
return nil, fmt.Errorf("retrieved results have no column data")
|
||||
return nil, merr.WrapErrServiceInternalMsg("retrieved results have no column data")
|
||||
}
|
||||
rowCount := -1
|
||||
for i := 0; i < outputColumnCount; i++ {
|
||||
@@ -374,11 +373,11 @@ func (reducer *GroupAggReducer) Reduce(ctx context.Context, results []*Aggregati
|
||||
rowCount = hashers[i].RowCount()
|
||||
} else if i < numGroupingKeys {
|
||||
if rowCount != hashers[i].RowCount() {
|
||||
return nil, fmt.Errorf("field data:%d for different columns have different row count, %d vs %d, wrong state",
|
||||
return nil, merr.WrapErrServiceInternalMsg("field data:%d for different columns have different row count, %d vs %d, wrong state",
|
||||
i, rowCount, hashers[i].RowCount())
|
||||
}
|
||||
} else if rowCount != accumulators[i-numGroupingKeys].RowCount() {
|
||||
return nil, fmt.Errorf("field data:%d for different columns have different row count, %d vs %d, wrong state",
|
||||
return nil, merr.WrapErrServiceInternalMsg("field data:%d for different columns have different row count, %d vs %d, wrong state",
|
||||
i, rowCount, accumulators[i-numGroupingKeys].RowCount())
|
||||
}
|
||||
}
|
||||
@@ -443,9 +442,9 @@ func (reducer *GroupAggReducer) Reduce(ctx context.Context, results []*Aggregati
|
||||
}
|
||||
}
|
||||
if totalGroupCount > maxGroupByGroups {
|
||||
return nil, merr.WrapErrServiceInternal(fmt.Sprintf("GROUP BY produced too many groups (%d). "+
|
||||
return nil, merr.WrapErrParameterInvalidMsg("GROUP BY produced too many groups (%d). "+
|
||||
"Add filters or increase common.groupBy.maxGroups (current: %d)",
|
||||
totalGroupCount, maxGroupByGroups))
|
||||
totalGroupCount, maxGroupByGroups)
|
||||
}
|
||||
// Don't guarantee specific groups to be returned before milvus support order by
|
||||
}
|
||||
@@ -478,7 +477,7 @@ func SegcoreResults2AggResult(results []*segcorepb.RetrieveResults) ([]*Aggregat
|
||||
aggResults := make([]*AggregationResult, len(results))
|
||||
for i := 0; i < len(results); i++ {
|
||||
if results[i] == nil {
|
||||
return nil, fmt.Errorf("input segcore query results from any sources cannot be nil")
|
||||
return nil, merr.WrapErrServiceInternalMsg("input segcore query results from any sources cannot be nil")
|
||||
}
|
||||
fieldsData := results[i].GetFieldsData()
|
||||
allRetrieveCount := results[i].GetAllRetrieveCount()
|
||||
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
"unsafe"
|
||||
|
||||
"github.com/milvus-io/milvus-proto/go-api/v3/schemapb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
)
|
||||
|
||||
func NewFieldAccessor(fieldType schemapb.DataType) (FieldAccessor, error) {
|
||||
@@ -28,7 +29,7 @@ func NewFieldAccessor(fieldType schemapb.DataType) (FieldAccessor, error) {
|
||||
case schemapb.DataType_Double:
|
||||
return newFloat64FieldAccessor(), nil
|
||||
default:
|
||||
return nil, fmt.Errorf("unsupported data type for hasher")
|
||||
return nil, merr.WrapErrParameterInvalidMsg("unsupported data type for hasher")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -410,7 +411,7 @@ func AssembleSingleValue(fv *FieldValue, fieldData *schemapb.FieldData) error {
|
||||
case schemapb.DataType_VarChar, schemapb.DataType_String:
|
||||
fieldData.GetScalars().GetStringData().Data = append(fieldData.GetScalars().GetStringData().GetData(), "")
|
||||
default:
|
||||
return fmt.Errorf("unsupported DataType:%d", fieldData.GetType())
|
||||
return merr.WrapErrParameterInvalidMsg("unsupported DataType:%d", fieldData.GetType())
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -421,47 +422,47 @@ func AssembleSingleValue(fv *FieldValue, fieldData *schemapb.FieldData) error {
|
||||
case schemapb.DataType_Bool:
|
||||
boolVal, ok := val.(bool)
|
||||
if !ok {
|
||||
return fmt.Errorf("type assertion failed: expected bool, got %T", val)
|
||||
return merr.WrapErrServiceInternalMsg("type assertion failed: expected bool, got %T", val)
|
||||
}
|
||||
fieldData.GetScalars().GetBoolData().Data = append(fieldData.GetScalars().GetBoolData().GetData(), boolVal)
|
||||
case schemapb.DataType_Int8, schemapb.DataType_Int16, schemapb.DataType_Int32:
|
||||
intVal, ok := val.(int32)
|
||||
if !ok {
|
||||
return fmt.Errorf("type assertion failed: expected int32, got %T", val)
|
||||
return merr.WrapErrServiceInternalMsg("type assertion failed: expected int32, got %T", val)
|
||||
}
|
||||
fieldData.GetScalars().GetIntData().Data = append(fieldData.GetScalars().GetIntData().GetData(), intVal)
|
||||
case schemapb.DataType_Int64:
|
||||
int64Val, ok := val.(int64)
|
||||
if !ok {
|
||||
return fmt.Errorf("type assertion failed: expected int64, got %T", val)
|
||||
return merr.WrapErrServiceInternalMsg("type assertion failed: expected int64, got %T", val)
|
||||
}
|
||||
fieldData.GetScalars().GetLongData().Data = append(fieldData.GetScalars().GetLongData().GetData(), int64Val)
|
||||
case schemapb.DataType_Timestamptz:
|
||||
timestampVal, ok := val.(int64)
|
||||
if !ok {
|
||||
return fmt.Errorf("type assertion failed: expected int64 for Timestamptz, got %T", val)
|
||||
return merr.WrapErrServiceInternalMsg("type assertion failed: expected int64 for Timestamptz, got %T", val)
|
||||
}
|
||||
fieldData.GetScalars().GetTimestamptzData().Data = append(fieldData.GetScalars().GetTimestamptzData().GetData(), timestampVal)
|
||||
case schemapb.DataType_Float:
|
||||
floatVal, ok := val.(float32)
|
||||
if !ok {
|
||||
return fmt.Errorf("type assertion failed: expected float32, got %T", val)
|
||||
return merr.WrapErrServiceInternalMsg("type assertion failed: expected float32, got %T", val)
|
||||
}
|
||||
fieldData.GetScalars().GetFloatData().Data = append(fieldData.GetScalars().GetFloatData().GetData(), floatVal)
|
||||
case schemapb.DataType_Double:
|
||||
doubleVal, ok := val.(float64)
|
||||
if !ok {
|
||||
return fmt.Errorf("type assertion failed: expected float64, got %T", val)
|
||||
return merr.WrapErrServiceInternalMsg("type assertion failed: expected float64, got %T", val)
|
||||
}
|
||||
fieldData.GetScalars().GetDoubleData().Data = append(fieldData.GetScalars().GetDoubleData().GetData(), doubleVal)
|
||||
case schemapb.DataType_VarChar, schemapb.DataType_String:
|
||||
stringVal, ok := val.(string)
|
||||
if !ok {
|
||||
return fmt.Errorf("type assertion failed: expected string, got %T", val)
|
||||
return merr.WrapErrServiceInternalMsg("type assertion failed: expected string, got %T", val)
|
||||
}
|
||||
fieldData.GetScalars().GetStringData().Data = append(fieldData.GetScalars().GetStringData().GetData(), stringVal)
|
||||
default:
|
||||
return fmt.Errorf("unsupported DataType:%d", fieldData.GetType())
|
||||
return merr.WrapErrParameterInvalidMsg("unsupported DataType:%d", fieldData.GetType())
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -544,13 +545,13 @@ func NewAggregationFieldMap(originalUserOutputFields []string, groupByFields []s
|
||||
// 2. Global aggregation (no GROUP BY): output_fields can only contain aggregation expressions
|
||||
// (e.g., "SELECT count(*), int64 FROM t" is invalid SQL — cannot mix aggregates with raw columns)
|
||||
if numGroupingKeys > 0 {
|
||||
return nil, fmt.Errorf(
|
||||
return nil, merr.WrapErrParameterInvalidMsg(
|
||||
"output field '%s' is not allowed: when using GROUP BY, output_fields can only contain "+
|
||||
"group_by fields (%v) or aggregation expressions",
|
||||
outputField, groupByFields,
|
||||
)
|
||||
}
|
||||
return nil, fmt.Errorf(
|
||||
return nil, merr.WrapErrParameterInvalidMsg(
|
||||
"output field '%s' is not allowed: when using aggregation functions (e.g., count(*)), "+
|
||||
"output_fields can only contain aggregation expressions, not regular columns",
|
||||
outputField,
|
||||
@@ -566,14 +567,14 @@ func NewAggregationFieldMap(originalUserOutputFields []string, groupByFields []s
|
||||
// and returns a new Double FieldData containing the average values.
|
||||
func ComputeAvgFromSumAndCount(sumFieldData *schemapb.FieldData, countFieldData *schemapb.FieldData) (*schemapb.FieldData, error) {
|
||||
if sumFieldData == nil || countFieldData == nil {
|
||||
return nil, fmt.Errorf("sumFieldData and countFieldData cannot be nil")
|
||||
return nil, merr.WrapErrServiceInternalMsg("sumFieldData and countFieldData cannot be nil")
|
||||
}
|
||||
|
||||
sumType := sumFieldData.GetType()
|
||||
countType := countFieldData.GetType()
|
||||
|
||||
if countType != schemapb.DataType_Int64 {
|
||||
return nil, fmt.Errorf("count field must be Int64 type, got %s", countType.String())
|
||||
return nil, merr.WrapErrParameterInvalidMsg("count field must be Int64 type, got %s", countType.String())
|
||||
}
|
||||
|
||||
countData := countFieldData.GetScalars().GetLongData().GetData()
|
||||
@@ -598,27 +599,27 @@ func ComputeAvgFromSumAndCount(sumFieldData *schemapb.FieldData, countFieldData
|
||||
case schemapb.DataType_Int64:
|
||||
sumData := sumFieldData.GetScalars().GetLongData().GetData()
|
||||
if len(sumData) != rowCount {
|
||||
return nil, fmt.Errorf("sum and count field data must have the same length, got sum:%d, count:%d", len(sumData), rowCount)
|
||||
return nil, merr.WrapErrParameterInvalidMsg("sum and count field data must have the same length, got sum:%d, count:%d", len(sumData), rowCount)
|
||||
}
|
||||
for i := 0; i < rowCount; i++ {
|
||||
if countData[i] == 0 {
|
||||
return nil, fmt.Errorf("division by zero: count is 0 at row %d", i)
|
||||
return nil, merr.WrapErrParameterInvalidMsg("division by zero: count is 0 at row %d", i)
|
||||
}
|
||||
resultData = append(resultData, float64(sumData[i])/float64(countData[i]))
|
||||
}
|
||||
case schemapb.DataType_Double:
|
||||
sumData := sumFieldData.GetScalars().GetDoubleData().GetData()
|
||||
if len(sumData) != rowCount {
|
||||
return nil, fmt.Errorf("sum and count field data must have the same length, got sum:%d, count:%d", len(sumData), rowCount)
|
||||
return nil, merr.WrapErrParameterInvalidMsg("sum and count field data must have the same length, got sum:%d, count:%d", len(sumData), rowCount)
|
||||
}
|
||||
for i := 0; i < rowCount; i++ {
|
||||
if countData[i] == 0 {
|
||||
return nil, fmt.Errorf("division by zero: count is 0 at row %d", i)
|
||||
return nil, merr.WrapErrParameterInvalidMsg("division by zero: count is 0 at row %d", i)
|
||||
}
|
||||
resultData = append(resultData, sumData[i]/float64(countData[i]))
|
||||
}
|
||||
default:
|
||||
return nil, fmt.Errorf("unsupported sum field type for avg computation: %s", sumType.String())
|
||||
return nil, merr.WrapErrParameterInvalidMsg("unsupported sum field type for avg computation: %s", sumType.String())
|
||||
}
|
||||
|
||||
result.GetScalars().GetDoubleData().Data = resultData
|
||||
|
||||
+18
-19
@@ -1,14 +1,13 @@
|
||||
package agg
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/planpb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
)
|
||||
|
||||
func terminateSingleSlot(slots []*FieldValue) (any, error) {
|
||||
if len(slots) != 1 {
|
||||
return nil, fmt.Errorf("aggregate expects 1 accumulator slot, got %d", len(slots))
|
||||
return nil, merr.WrapErrParameterInvalidMsg("aggregate expects 1 accumulator slot, got %d", len(slots))
|
||||
}
|
||||
if slots[0] == nil || slots[0].IsNull() {
|
||||
return nil, nil
|
||||
@@ -18,7 +17,7 @@ func terminateSingleSlot(slots []*FieldValue) (any, error) {
|
||||
|
||||
func terminateAvg(slots []*FieldValue) (any, error) {
|
||||
if len(slots) != 2 {
|
||||
return nil, fmt.Errorf("avg expects 2 accumulator slots, got %d", len(slots))
|
||||
return nil, merr.WrapErrParameterInvalidMsg("avg expects 2 accumulator slots, got %d", len(slots))
|
||||
}
|
||||
sum := slots[0]
|
||||
count := slots[1]
|
||||
@@ -52,7 +51,7 @@ func fieldValueToFloat64(v any) (float64, error) {
|
||||
case float64:
|
||||
return value, nil
|
||||
default:
|
||||
return 0, fmt.Errorf("avg expects numeric accumulator, got %T", v)
|
||||
return 0, merr.WrapErrParameterInvalidMsg("avg expects numeric accumulator, got %T", v)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -80,7 +79,7 @@ func (sum *SumAggregate) NewState() []*FieldValue {
|
||||
|
||||
func (sum *SumAggregate) UpdateState(slots []*FieldValue, new *FieldValue) error {
|
||||
if len(slots) != 1 {
|
||||
return fmt.Errorf("aggregate expects 1 accumulator slot, got %d", len(slots))
|
||||
return merr.WrapErrParameterInvalidMsg("aggregate expects 1 accumulator slot, got %d", len(slots))
|
||||
}
|
||||
return sum.Update(slots[0], new)
|
||||
}
|
||||
@@ -121,7 +120,7 @@ func (count *CountAggregate) NewState() []*FieldValue {
|
||||
|
||||
func (count *CountAggregate) UpdateState(slots []*FieldValue, new *FieldValue) error {
|
||||
if len(slots) != 1 {
|
||||
return fmt.Errorf("aggregate expects 1 accumulator slot, got %d", len(slots))
|
||||
return merr.WrapErrParameterInvalidMsg("aggregate expects 1 accumulator slot, got %d", len(slots))
|
||||
}
|
||||
if new == nil || new.IsNull() {
|
||||
return nil
|
||||
@@ -166,7 +165,7 @@ func (avg *AvgAggregate) Name() string {
|
||||
}
|
||||
|
||||
func (avg *AvgAggregate) Update(target *FieldValue, new *FieldValue) error {
|
||||
return fmt.Errorf("avg aggregate updates require aggregate state slots")
|
||||
return merr.WrapErrServiceInternalMsg("avg aggregate updates require aggregate state slots")
|
||||
}
|
||||
|
||||
func (avg *AvgAggregate) NewState() []*FieldValue {
|
||||
@@ -175,7 +174,7 @@ func (avg *AvgAggregate) NewState() []*FieldValue {
|
||||
|
||||
func (avg *AvgAggregate) UpdateState(slots []*FieldValue, new *FieldValue) error {
|
||||
if len(slots) != 2 {
|
||||
return fmt.Errorf("avg expects 2 accumulator slots, got %d", len(slots))
|
||||
return merr.WrapErrParameterInvalidMsg("avg expects 2 accumulator slots, got %d", len(slots))
|
||||
}
|
||||
if err := avg.sum.Update(slots[0], new); err != nil {
|
||||
return err
|
||||
@@ -204,7 +203,7 @@ func (avg *AvgAggregate) OriginalName() string {
|
||||
|
||||
func updateOrderedState(target *FieldValue, new *FieldValue, op string) error {
|
||||
if target == nil || new == nil {
|
||||
return fmt.Errorf("target or new field value is nil")
|
||||
return merr.WrapErrServiceInternalMsg("target or new field value is nil")
|
||||
}
|
||||
if new.IsNull() {
|
||||
return nil
|
||||
@@ -234,41 +233,41 @@ func shouldReplaceOrderedValue(target, new any, op string) (bool, error) {
|
||||
case int:
|
||||
newVal, ok := new.(int)
|
||||
if !ok {
|
||||
return false, fmt.Errorf("type mismatch: target is int, new is %T", new)
|
||||
return false, merr.WrapErrParameterInvalidMsg("type mismatch: target is int, new is %T", new)
|
||||
}
|
||||
return shouldReplaceOrdered(newVal, targetVal, op), nil
|
||||
case int32:
|
||||
newVal, ok := new.(int32)
|
||||
if !ok {
|
||||
return false, fmt.Errorf("type mismatch: target is int32, new is %T", new)
|
||||
return false, merr.WrapErrParameterInvalidMsg("type mismatch: target is int32, new is %T", new)
|
||||
}
|
||||
return shouldReplaceOrdered(newVal, targetVal, op), nil
|
||||
case int64:
|
||||
newVal, ok := new.(int64)
|
||||
if !ok {
|
||||
return false, fmt.Errorf("type mismatch: target is int64, new is %T", new)
|
||||
return false, merr.WrapErrParameterInvalidMsg("type mismatch: target is int64, new is %T", new)
|
||||
}
|
||||
return shouldReplaceOrdered(newVal, targetVal, op), nil
|
||||
case float32:
|
||||
newVal, ok := new.(float32)
|
||||
if !ok {
|
||||
return false, fmt.Errorf("type mismatch: target is float32, new is %T", new)
|
||||
return false, merr.WrapErrParameterInvalidMsg("type mismatch: target is float32, new is %T", new)
|
||||
}
|
||||
return shouldReplaceOrdered(newVal, targetVal, op), nil
|
||||
case float64:
|
||||
newVal, ok := new.(float64)
|
||||
if !ok {
|
||||
return false, fmt.Errorf("type mismatch: target is float64, new is %T", new)
|
||||
return false, merr.WrapErrParameterInvalidMsg("type mismatch: target is float64, new is %T", new)
|
||||
}
|
||||
return shouldReplaceOrdered(newVal, targetVal, op), nil
|
||||
case string:
|
||||
newVal, ok := new.(string)
|
||||
if !ok {
|
||||
return false, fmt.Errorf("type mismatch: target is string, new is %T", new)
|
||||
return false, merr.WrapErrParameterInvalidMsg("type mismatch: target is string, new is %T", new)
|
||||
}
|
||||
return shouldReplaceOrdered(newVal, targetVal, op), nil
|
||||
default:
|
||||
return false, fmt.Errorf("unsupported type for %s aggregation: %T", op, target)
|
||||
return false, merr.WrapErrParameterInvalidMsg("unsupported type for %s aggregation: %T", op, target)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -299,7 +298,7 @@ func (min *MinAggregate) NewState() []*FieldValue {
|
||||
|
||||
func (min *MinAggregate) UpdateState(slots []*FieldValue, new *FieldValue) error {
|
||||
if len(slots) != 1 {
|
||||
return fmt.Errorf("aggregate expects 1 accumulator slot, got %d", len(slots))
|
||||
return merr.WrapErrParameterInvalidMsg("aggregate expects 1 accumulator slot, got %d", len(slots))
|
||||
}
|
||||
return min.Update(slots[0], new)
|
||||
}
|
||||
@@ -339,7 +338,7 @@ func (max *MaxAggregate) NewState() []*FieldValue {
|
||||
|
||||
func (max *MaxAggregate) UpdateState(slots []*FieldValue, new *FieldValue) error {
|
||||
if len(slots) != 1 {
|
||||
return fmt.Errorf("aggregate expects 1 accumulator slot, got %d", len(slots))
|
||||
return merr.WrapErrParameterInvalidMsg("aggregate expects 1 accumulator slot, got %d", len(slots))
|
||||
}
|
||||
return max.Update(slots[0], new)
|
||||
}
|
||||
|
||||
@@ -1,9 +1,8 @@
|
||||
package agg
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/milvus-io/milvus-proto/go-api/v3/schemapb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
)
|
||||
|
||||
func isSupportedAggFieldType(aggregateName string, dt schemapb.DataType) bool {
|
||||
@@ -51,7 +50,7 @@ func isSupportedAggFieldType(aggregateName string, dt schemapb.DataType) bool {
|
||||
// Note: operator name is expected to be normalized (lowercase) by caller.
|
||||
func ValidateAggFieldType(aggregateName string, dt schemapb.DataType) error {
|
||||
if !isSupportedAggFieldType(aggregateName, dt) {
|
||||
return fmt.Errorf("aggregation operator %s does not support data type %s", aggregateName, dt.String())
|
||||
return merr.WrapErrParameterInvalidMsg("aggregation operator %s does not support data type %s", aggregateName, dt.String())
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -18,14 +18,13 @@ package allocator
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/cockroachdb/errors"
|
||||
"go.uber.org/zap"
|
||||
|
||||
"github.com/milvus-io/milvus/pkg/v3/log"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
)
|
||||
|
||||
const (
|
||||
@@ -249,10 +248,9 @@ func (ta *CachedAllocator) finishSyncRequest() {
|
||||
func (ta *CachedAllocator) failRemainRequest() {
|
||||
var err error
|
||||
if ta.SyncErr != nil {
|
||||
err = fmt.Errorf("%s failRemainRequest err:%w", ta.Role, ta.SyncErr)
|
||||
err = merr.Wrapf(ta.SyncErr, "%s failRemainRequest", ta.Role)
|
||||
} else {
|
||||
errMsg := fmt.Sprintf("%s failRemainRequest unexpected error", ta.Role)
|
||||
err = errors.New(errMsg)
|
||||
err = merr.WrapErrServiceInternalMsg("%s failRemainRequest unexpected error", ta.Role)
|
||||
}
|
||||
if len(ta.ToDoReqs) > 0 {
|
||||
log.Warn("Allocator has some reqs to fail",
|
||||
@@ -290,8 +288,7 @@ func (ta *CachedAllocator) Close() {
|
||||
ta.CancelFunc()
|
||||
ta.wg.Wait()
|
||||
ta.TChan.Close()
|
||||
errMsg := fmt.Sprintf("%s is closing", ta.Role)
|
||||
ta.revokeRequest(errors.New(errMsg))
|
||||
ta.revokeRequest(merr.WrapErrServiceInternalMsg("%s is closing", ta.Role))
|
||||
}
|
||||
|
||||
// CleanCache is used to force synchronize with global allocator.
|
||||
|
||||
@@ -18,14 +18,12 @@ package allocator
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/cockroachdb/errors"
|
||||
|
||||
"github.com/milvus-io/milvus-proto/go-api/v3/commonpb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/rootcoordpb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/commonpbutil"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
)
|
||||
|
||||
const (
|
||||
@@ -101,7 +99,7 @@ func (ia *IDAllocator) syncID() (bool, error) {
|
||||
|
||||
cancel()
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("syncID Failed:%w", err)
|
||||
return false, merr.Wrapf(err, "syncID failed")
|
||||
}
|
||||
ia.idStart = resp.GetID()
|
||||
ia.idEnd = ia.idStart + int64(resp.GetCount())
|
||||
@@ -148,7 +146,7 @@ func (ia *IDAllocator) AllocOne() (UniqueID, error) {
|
||||
// Alloc allocates the id of the count number.
|
||||
func (ia *IDAllocator) Alloc(count uint32) (UniqueID, UniqueID, error) {
|
||||
if ia.closed() {
|
||||
return 0, 0, errors.New("fail to allocate ID, closed allocator")
|
||||
return 0, 0, merr.WrapErrServiceInternalMsg("fail to allocate ID, closed allocator")
|
||||
}
|
||||
req := &IDRequest{BaseRequest: BaseRequest{Done: make(chan error), Valid: false}}
|
||||
|
||||
|
||||
@@ -17,8 +17,9 @@
|
||||
package allocator
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sync"
|
||||
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
)
|
||||
|
||||
// localAllocator implements the Interface.
|
||||
@@ -40,12 +41,12 @@ func NewLocalAllocator(start, end int64) Interface {
|
||||
func (a *localAllocator) Alloc(count uint32) (int64, int64, error) {
|
||||
cnt := int64(count)
|
||||
if cnt <= 0 {
|
||||
return 0, 0, fmt.Errorf("non-positive count is not allowed, count=%d", cnt)
|
||||
return 0, 0, merr.WrapErrParameterInvalidMsg("non-positive count is not allowed, count=%d", cnt)
|
||||
}
|
||||
a.mu.Lock()
|
||||
defer a.mu.Unlock()
|
||||
if a.idStart+cnt > a.idEnd {
|
||||
return 0, 0, fmt.Errorf("ID is exhausted, start=%d, end=%d, count=%d", a.idStart, a.idEnd, cnt)
|
||||
return 0, 0, merr.WrapErrServiceInternalMsg("ID is exhausted, start=%d, end=%d, count=%d", a.idStart, a.idEnd, cnt)
|
||||
}
|
||||
start := a.idStart
|
||||
a.idStart += cnt
|
||||
|
||||
@@ -20,7 +20,6 @@ import (
|
||||
"context"
|
||||
"crypto/tls"
|
||||
|
||||
"github.com/cockroachdb/errors"
|
||||
"go.uber.org/zap"
|
||||
"google.golang.org/grpc"
|
||||
|
||||
@@ -28,6 +27,7 @@ import (
|
||||
"github.com/milvus-io/milvus-proto/go-api/v3/milvuspb"
|
||||
"github.com/milvus-io/milvus/client/v2/milvusclient"
|
||||
"github.com/milvus-io/milvus/pkg/v3/log"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
|
||||
)
|
||||
|
||||
@@ -52,7 +52,7 @@ func NewMilvusClient(ctx context.Context, cluster *commonpb.MilvusCluster) (Milv
|
||||
// Build TLS config from per-cluster paramtable config.
|
||||
tlsConfig, err := buildCDCTLSConfig(cluster.GetClusterId())
|
||||
if err != nil {
|
||||
return nil, errors.Wrap(err, "failed to build CDC TLS config")
|
||||
return nil, merr.Wrap(err, "failed to build CDC TLS config")
|
||||
}
|
||||
if tlsConfig != nil {
|
||||
config.WithTLSConfig(tlsConfig)
|
||||
@@ -67,7 +67,7 @@ func NewMilvusClient(ctx context.Context, cluster *commonpb.MilvusCluster) (Milv
|
||||
|
||||
cli, err := milvusclient.New(ctx, config)
|
||||
if err != nil {
|
||||
return nil, errors.Wrap(err, "failed to create milvus client")
|
||||
return nil, merr.Wrap(err, "failed to create milvus client")
|
||||
}
|
||||
return cli, nil
|
||||
}
|
||||
|
||||
@@ -5,11 +5,11 @@ import (
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/cockroachdb/errors"
|
||||
clientv3 "go.etcd.io/etcd/client/v3"
|
||||
"google.golang.org/protobuf/proto"
|
||||
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/streamingpb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
)
|
||||
|
||||
const requestTimeout = 30 * time.Second
|
||||
@@ -44,7 +44,7 @@ func ListReplicatePChannels(ctx context.Context, etcdCli *clientv3.Client, prefi
|
||||
meta := &streamingpb.ReplicatePChannelMeta{}
|
||||
err = proto.Unmarshal([]byte(value), meta)
|
||||
if err != nil {
|
||||
return nil, errors.Wrapf(err, "unmarshal replicate pchannel meta %s failed", keys[k])
|
||||
return nil, merr.Wrapf(err, "unmarshal replicate pchannel meta %s failed", keys[k])
|
||||
}
|
||||
channel := &ReplicateChannel{
|
||||
Key: keys[k],
|
||||
|
||||
@@ -35,6 +35,7 @@ import (
|
||||
"github.com/milvus-io/milvus/pkg/v3/streaming/util/message"
|
||||
"github.com/milvus-io/milvus/pkg/v3/streaming/util/message/adaptor"
|
||||
"github.com/milvus-io/milvus/pkg/v3/streaming/util/options"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/syncutil"
|
||||
)
|
||||
@@ -202,7 +203,7 @@ func (r *channelReplicator) getReplicateCheckpoint() (*utility.ReplicateCheckpoi
|
||||
}
|
||||
replicateInfo, err := r.targetClient.GetReplicateInfo(ctx, req)
|
||||
if err != nil {
|
||||
return nil, errors.Wrap(err, "failed to get replicate info")
|
||||
return nil, merr.Wrap(err, "failed to get replicate info")
|
||||
}
|
||||
|
||||
checkpoint := replicateInfo.GetCheckpoint()
|
||||
|
||||
@@ -115,7 +115,7 @@ func (q *msgQueue) Enqueue(ctx context.Context, msg message.ImmutableMessage) er
|
||||
// Optional runtime check: enforce non-decreasing timetick
|
||||
// if n := len(q.buf); n > 0 {
|
||||
// if q.buf[n-1].Timetick() > msg.Timetick() {
|
||||
// return fmt.Errorf("enqueue timetick order violation: last=%d new=%d", q.buf[n-1].Timetick(), msg.Timetick())
|
||||
// return merr.WrapErrParameterInvalidMsg("enqueue timetick order violation: last=%d new=%d", q.buf[n-1].Timetick(), msg.Timetick())
|
||||
// }
|
||||
// }
|
||||
|
||||
|
||||
@@ -165,7 +165,7 @@ func (m *FileResourceObserver) CheckNodeSynced(nodeID int64) bool {
|
||||
func (m *FileResourceObserver) CheckAllQnReady() error {
|
||||
// return error if meta is not ready
|
||||
if m.meta == nil {
|
||||
return merr.WrapErrParameterInvalidMsg("rootcoord meta is not ready")
|
||||
return merr.WrapErrServiceUnavailable("rootcoord meta is not ready")
|
||||
}
|
||||
|
||||
resources, version := m.meta.ListFileResource(m.ctx)
|
||||
@@ -177,7 +177,7 @@ func (m *FileResourceObserver) CheckAllQnReady() error {
|
||||
var err error
|
||||
m.distribution.Range(func(nodeID int64, node *NodeInfo) bool {
|
||||
if node.NodeType == QueryNode && node.Version < version {
|
||||
err = merr.WrapErrParameterInvalidMsg("node %d file resource not synced", nodeID)
|
||||
err = merr.WrapErrServiceUnavailableMsg("file resource not synced, node-%d", nodeID)
|
||||
return false
|
||||
}
|
||||
return true
|
||||
|
||||
@@ -298,10 +298,10 @@ func (s *mixCoordImpl) getQueryNodes(ctx context.Context) (*querypb.ListQueryNod
|
||||
Base: commonpbutil.NewMsgBase(),
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to list query nodes: %w", err)
|
||||
return nil, merr.Wrapf(err, "failed to list query nodes")
|
||||
}
|
||||
if !merr.Ok(resp.GetStatus()) {
|
||||
return nil, fmt.Errorf("failed to list query nodes: %s", resp.GetStatus().GetReason())
|
||||
return nil, merr.Wrapf(merr.Error(resp.GetStatus()), "failed to list query nodes")
|
||||
}
|
||||
return resp, nil
|
||||
}
|
||||
@@ -514,7 +514,7 @@ func (s *mixCoordImpl) getQueryCoordChannelBalanceActive(ctx context.Context) (b
|
||||
}
|
||||
|
||||
if !merr.Ok(resp.GetStatus()) {
|
||||
return channelActivate, fmt.Errorf("%s", resp.GetStatus().GetReason())
|
||||
return channelActivate, merr.Error(resp.GetStatus())
|
||||
}
|
||||
|
||||
return channelActivate && resp.IsActive, nil
|
||||
@@ -528,7 +528,7 @@ func (s *mixCoordImpl) getQueryCoordBalanceActive(ctx context.Context) (bool, er
|
||||
return false, err
|
||||
}
|
||||
if !merr.Ok(resp.GetStatus()) {
|
||||
return false, fmt.Errorf("%s", resp.GetStatus().GetReason())
|
||||
return false, merr.Error(resp.GetStatus())
|
||||
}
|
||||
return resp.IsActive, nil
|
||||
}
|
||||
@@ -569,7 +569,7 @@ func (s *mixCoordImpl) controlQueryCoordChannelBalanceStatus(ctx context.Context
|
||||
default:
|
||||
// If the status is not recognized, return an informative error immediately.
|
||||
// This avoids proceeding with an invalid state.
|
||||
err = fmt.Errorf("invalid status value: '%s'. Use 'suspended', 'resumed' or 'active'", status)
|
||||
err = merr.WrapErrParameterInvalidMsg("invalid status value: '%s'. Use 'suspended', 'resumed' or 'active'", status)
|
||||
}
|
||||
|
||||
// --- Unified Error Handling ---
|
||||
@@ -578,7 +578,7 @@ func (s *mixCoordImpl) controlQueryCoordChannelBalanceStatus(ctx context.Context
|
||||
// First, check if there was a low-level error during the method execution (e.g., network issue).
|
||||
if err != nil {
|
||||
// Wrap the original error with more context.
|
||||
return fmt.Errorf("%s: %w", errMsg, err)
|
||||
return merr.Wrap(err, errMsg)
|
||||
}
|
||||
// If no errors were encountered, the operation was successful. Return nil to indicate success.
|
||||
return nil
|
||||
@@ -958,11 +958,11 @@ func (s *mixCoordImpl) handleQueryNodeStatusUpdate(ctx context.Context, nodeID i
|
||||
errMsg = "failed to resume node"
|
||||
default:
|
||||
errMsg = "invalid status value. Use 'suspended', 'resumed' or 'active'"
|
||||
err = fmt.Errorf("%s", errMsg)
|
||||
err = merr.WrapErrParameterInvalidMsg("%s", errMsg)
|
||||
}
|
||||
err = merr.CheckRPCCall(resp, err)
|
||||
if err != nil {
|
||||
err = fmt.Errorf("%s: %w", errMsg, err)
|
||||
err = merr.Wrap(err, errMsg)
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -12,6 +12,7 @@ import (
|
||||
"github.com/milvus-io/milvus/pkg/v3/log"
|
||||
"github.com/milvus-io/milvus/pkg/v3/streaming/util/types"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/funcutil"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/syncutil"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/typeutil"
|
||||
)
|
||||
@@ -99,7 +100,7 @@ func (s *StreamingNodeManager) GetLatestWALLocated(ctx context.Context, vchannel
|
||||
}
|
||||
serverID, ok := balancer.GetLatestWALLocated(ctx, pchannel)
|
||||
if !ok {
|
||||
return -1, errors.Errorf("channel: %s not found", vchannel)
|
||||
return -1, merr.WrapErrChannelNotFound(vchannel)
|
||||
}
|
||||
return serverID, nil
|
||||
}
|
||||
|
||||
@@ -19,12 +19,12 @@ package coordinator
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/cockroachdb/errors"
|
||||
"go.uber.org/zap"
|
||||
|
||||
"github.com/milvus-io/milvus/internal/streamingcoord/server/broadcaster/registry"
|
||||
"github.com/milvus-io/milvus/pkg/v3/log"
|
||||
"github.com/milvus-io/milvus/pkg/v3/streaming/util/message"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
|
||||
)
|
||||
|
||||
@@ -66,7 +66,7 @@ func (c *WALCallback) alterWALV2AckCallback(
|
||||
targetWALName := result.Message.Header().TargetWalName
|
||||
mqTypeValue := message.WALName(targetWALName).String()
|
||||
if mqTypeValue == "unknown" {
|
||||
return errors.Errorf("invalid target WAL name: %v", targetWALName)
|
||||
return merr.WrapErrServiceInternalMsg("invalid target WAL name: %v", targetWALName)
|
||||
}
|
||||
|
||||
// Get etcd source to update configuration
|
||||
@@ -74,7 +74,7 @@ func (c *WALCallback) alterWALV2AckCallback(
|
||||
etcdSource, ok := paramMgr.GetEtcdSource()
|
||||
if !ok {
|
||||
logger.Warn("failed to update mq.type config, etcd source not enabled")
|
||||
return errors.New("etcd source is not enabled, cannot update mq.type configuration")
|
||||
return merr.WrapErrServiceInternalMsg("etcd source is not enabled, cannot update mq.type configuration")
|
||||
}
|
||||
|
||||
// Update mq.type configuration in etcd
|
||||
@@ -84,7 +84,7 @@ func (c *WALCallback) alterWALV2AckCallback(
|
||||
zap.String("configKey", configKey),
|
||||
zap.String("mqTypeValue", mqTypeValue),
|
||||
zap.Error(err))
|
||||
return errors.Wrap(err, "failed to update mq.type configuration in etcd")
|
||||
return merr.Wrap(err, "failed to update mq.type configuration in etcd")
|
||||
}
|
||||
|
||||
logger.Info("successfully updated mq.type configuration in etcd",
|
||||
|
||||
+7
-4
@@ -131,10 +131,13 @@ async fn download_with_retry(
|
||||
for url in urls {
|
||||
match client.get(url).send().await {
|
||||
Ok(resp) if resp.status().is_success() => {
|
||||
let content = resp
|
||||
.bytes()
|
||||
.await
|
||||
.map_err(|e| TantivyBindingError::InternalError(e.to_string()))?;
|
||||
let content = match resp.bytes().await {
|
||||
Ok(c) => c,
|
||||
Err(e) => {
|
||||
warn!("Failed to read body from {}: {}", url, e);
|
||||
continue;
|
||||
}
|
||||
};
|
||||
|
||||
// Calculate MD5 hash
|
||||
let mut context = Context::new();
|
||||
|
||||
@@ -18,7 +18,6 @@ package datacoord
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"sync"
|
||||
|
||||
"go.uber.org/zap"
|
||||
@@ -28,6 +27,7 @@ import (
|
||||
"github.com/milvus-io/milvus/pkg/v3/log"
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/workerpb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/timerecord"
|
||||
)
|
||||
|
||||
@@ -117,7 +117,7 @@ func (m *analyzeMeta) UpdateVersion(taskID int64, nodeID int64) error {
|
||||
|
||||
t, ok := m.tasks[taskID]
|
||||
if !ok {
|
||||
return fmt.Errorf("there is no task with taskID: %d", taskID)
|
||||
return merr.WrapErrServiceInternalMsg("there is no task with taskID: %d", taskID)
|
||||
}
|
||||
|
||||
cloneT := proto.Clone(t).(*indexpb.AnalyzeTask)
|
||||
@@ -134,7 +134,7 @@ func (m *analyzeMeta) BuildingTask(taskID int64) error {
|
||||
|
||||
t, ok := m.tasks[taskID]
|
||||
if !ok {
|
||||
return fmt.Errorf("there is no task with taskID: %d", taskID)
|
||||
return merr.WrapErrServiceInternalMsg("there is no task with taskID: %d", taskID)
|
||||
}
|
||||
|
||||
cloneT := proto.Clone(t).(*indexpb.AnalyzeTask)
|
||||
@@ -150,7 +150,7 @@ func (m *analyzeMeta) UpdateState(taskID int64, state indexpb.JobState, failReas
|
||||
|
||||
t, ok := m.tasks[taskID]
|
||||
if !ok {
|
||||
return fmt.Errorf("there is no task with taskID: %d", taskID)
|
||||
return merr.WrapErrServiceInternalMsg("there is no task with taskID: %d", taskID)
|
||||
}
|
||||
|
||||
cloneT := proto.Clone(t).(*indexpb.AnalyzeTask)
|
||||
@@ -168,7 +168,7 @@ func (m *analyzeMeta) FinishTask(taskID int64, result *workerpb.AnalyzeResult) e
|
||||
|
||||
t, ok := m.tasks[taskID]
|
||||
if !ok {
|
||||
return fmt.Errorf("there is no task with taskID: %d", taskID)
|
||||
return merr.WrapErrServiceInternalMsg("there is no task with taskID: %d", taskID)
|
||||
}
|
||||
|
||||
log.Info("finish task meta...", zap.Int64("taskID", taskID), zap.String("state", result.GetState().String()),
|
||||
|
||||
@@ -21,10 +21,9 @@ import (
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/cockroachdb/errors"
|
||||
|
||||
"github.com/milvus-io/milvus/internal/storage"
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/datapb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
)
|
||||
|
||||
// BackfillResult is the decoded form of the JSON produced by the Spark
|
||||
@@ -86,34 +85,34 @@ var knownObjectSchemes = map[string]struct{}{
|
||||
// 3. Unknown scheme -> return ErrUnsupportedScheme.
|
||||
func normalizeObjectKey(raw, expectedBucket string) (string, error) {
|
||||
if raw == "" {
|
||||
return "", errors.New("empty object path")
|
||||
return "", merr.WrapErrServiceInternalMsg("empty object path")
|
||||
}
|
||||
if !strings.Contains(raw, "://") {
|
||||
key := strings.TrimPrefix(raw, "/")
|
||||
if key == "" {
|
||||
return "", errors.Newf("empty object key in %q", raw)
|
||||
return "", merr.WrapErrServiceInternalMsg("empty object key in %q", raw)
|
||||
}
|
||||
return key, nil
|
||||
}
|
||||
idx := strings.Index(raw, "://")
|
||||
scheme := strings.ToLower(raw[:idx])
|
||||
if _, ok := knownObjectSchemes[scheme]; !ok {
|
||||
return "", errors.Newf("unsupported object URI scheme %q in %q", scheme, raw)
|
||||
return "", merr.WrapErrServiceInternalMsg("unsupported object URI scheme %q in %q", scheme, raw)
|
||||
}
|
||||
rest := raw[idx+3:]
|
||||
slash := strings.Index(rest, "/")
|
||||
if slash < 0 {
|
||||
return "", errors.Newf("malformed object URI %q: missing object key", raw)
|
||||
return "", merr.WrapErrServiceInternalMsg("malformed object URI %q: missing object key", raw)
|
||||
}
|
||||
bucket := rest[:slash]
|
||||
key := rest[slash+1:]
|
||||
if expectedBucket != "" && bucket != expectedBucket {
|
||||
return "", errors.Newf("object URI bucket %q differs from datacoord bucket %q (path=%s)", bucket, expectedBucket, raw)
|
||||
return "", merr.WrapErrServiceInternalMsg("object URI bucket %q differs from datacoord bucket %q (path=%s)", bucket, expectedBucket, raw)
|
||||
}
|
||||
// Reject inputs like "s3a://bucket/" that parse to an empty key -- passing
|
||||
// an empty key to chunkManager.Read has undefined behavior across SDKs.
|
||||
if key == "" {
|
||||
return "", errors.Newf("empty object key in %q", raw)
|
||||
return "", merr.WrapErrServiceInternalMsg("empty object key in %q", raw)
|
||||
}
|
||||
return key, nil
|
||||
}
|
||||
@@ -147,20 +146,20 @@ func buildV2Groups(bucket string, entry *BackfillSegment) (map[int64]*datapb.Fie
|
||||
for i := range entry.ColumnGroups {
|
||||
g := &entry.ColumnGroups[i]
|
||||
if len(g.FieldIDs) != 1 {
|
||||
return nil, errors.Newf("backfill invariant violated: column group has %d field_ids (expected 1)", len(g.FieldIDs))
|
||||
return nil, merr.WrapErrServiceInternalMsg("backfill invariant violated: column group has %d field_ids (expected 1)", len(g.FieldIDs))
|
||||
}
|
||||
n := int64(len(g.BinlogFiles))
|
||||
if n == 0 {
|
||||
return nil, errors.Newf("column group for field %d has no binlog files", g.FieldIDs[0])
|
||||
return nil, merr.WrapErrServiceInternalMsg("column group for field %d has no binlog files", g.FieldIDs[0])
|
||||
}
|
||||
fid := g.FieldIDs[0]
|
||||
if _, dup := out[fid]; dup {
|
||||
return nil, errors.Newf("duplicate column group for field %d", fid)
|
||||
return nil, merr.WrapErrServiceInternalMsg("duplicate column group for field %d", fid)
|
||||
}
|
||||
// row_count flows into EntriesNum; non-positive values are undefined
|
||||
// (zero collapses presence markers, negatives break accounting).
|
||||
if g.RowCount <= 0 {
|
||||
return nil, errors.Newf("column group for field %d has non-positive row_count %d", fid, g.RowCount)
|
||||
return nil, merr.WrapErrServiceInternalMsg("column group for field %d has non-positive row_count %d", fid, g.RowCount)
|
||||
}
|
||||
avg := g.RowCount / n
|
||||
rem := g.RowCount - avg*n
|
||||
@@ -176,7 +175,7 @@ func buildV2Groups(bucket string, entry *BackfillSegment) (map[int64]*datapb.Fie
|
||||
}
|
||||
logID, ok := parseLogIDFromKey(key)
|
||||
if !ok {
|
||||
return nil, errors.Newf("column group for field %d has binlog file %q with non-numeric trailing segment", fid, p)
|
||||
return nil, merr.WrapErrServiceInternalMsg("column group for field %d has binlog file %q with non-numeric trailing segment", fid, p)
|
||||
}
|
||||
// LogPath must be empty at persistence time -- catalog.checkLogID
|
||||
// rejects any Binlog with LogPath != "" (see
|
||||
|
||||
@@ -31,6 +31,7 @@ import (
|
||||
"github.com/milvus-io/milvus/pkg/v3/common"
|
||||
"github.com/milvus-io/milvus/pkg/v3/log"
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/datapb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
|
||||
)
|
||||
|
||||
@@ -241,7 +242,7 @@ func calculateClusteringCompactionConfig(coll *collectionInfo, view CompactionVi
|
||||
|
||||
func estimateRowsBySegmentSize(segments []*SegmentView, expectedSegmentSize int64) (int64, error) {
|
||||
if expectedSegmentSize <= 0 {
|
||||
return 0, fmt.Errorf("invalid expected segment size %d", expectedSegmentSize)
|
||||
return 0, merr.WrapErrServiceInternalMsg("invalid expected segment size %d", expectedSegmentSize)
|
||||
}
|
||||
|
||||
var totalSize float64
|
||||
@@ -258,17 +259,17 @@ func estimateRowsBySegmentSize(segments []*SegmentView, expectedSegmentSize int6
|
||||
}
|
||||
|
||||
if totalRows == 0 || totalSize == 0 {
|
||||
return 0, fmt.Errorf("segment view does not contain size info to estimate row count")
|
||||
return 0, merr.WrapErrServiceInternalMsg("segment view does not contain size info to estimate row count")
|
||||
}
|
||||
|
||||
rowSize := totalSize / float64(totalRows)
|
||||
if rowSize <= 0 {
|
||||
return 0, fmt.Errorf("invalid row size calculated from segment view")
|
||||
return 0, merr.WrapErrServiceInternalMsg("invalid row size calculated from segment view")
|
||||
}
|
||||
|
||||
rows := float64(expectedSegmentSize) / rowSize
|
||||
if rows <= 0 {
|
||||
return 0, fmt.Errorf("estimated max row count is not positive")
|
||||
return 0, merr.WrapErrServiceInternalMsg("estimated max row count is not positive")
|
||||
}
|
||||
|
||||
return int64(rows), nil
|
||||
|
||||
@@ -2,7 +2,6 @@ package datacoord
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"math"
|
||||
|
||||
"github.com/samber/lo"
|
||||
@@ -102,7 +101,7 @@ func (policy *forceMergeCompactionPolicy) triggerOneCollection(
|
||||
|
||||
configMaxSize := getExpectedSegmentSize(policy.meta, collectionID, collection.Schema)
|
||||
if targetSizeBytes < configMaxSize {
|
||||
return nil, 0, merr.WrapErrParameterInvalidMsg(fmt.Sprintf("targetSize %d MB should be greater than or equal to configMaxSize %d MB", targetSize, configMaxSize/(1024*1024)))
|
||||
return nil, 0, merr.WrapErrParameterInvalidMsg("targetSize %d MB should be greater than or equal to configMaxSize %d MB", targetSize, configMaxSize/(1024*1024))
|
||||
}
|
||||
|
||||
segments := policy.meta.SelectSegments(ctx, WithCollection(collectionID), SegmentFilterFunc(func(segment *SegmentInfo) bool {
|
||||
@@ -182,7 +181,7 @@ var _ CollectionTopologyQuerier = (*metricsNodeMemoryQuerier)(nil)
|
||||
func (q *metricsNodeMemoryQuerier) GetCollectionTopology(ctx context.Context, collectionID int64) (*CollectionTopology, error) {
|
||||
log := log.Ctx(ctx).With(zap.Int64("collectionID", collectionID))
|
||||
if q.mixCoord == nil {
|
||||
return nil, fmt.Errorf("mixCoord not available for topology query")
|
||||
return nil, merr.WrapErrServiceInternalMsg("mixCoord not available for topology query")
|
||||
}
|
||||
|
||||
// 1. Get replica information
|
||||
|
||||
@@ -74,9 +74,12 @@ func (pq *PriorityQueue[T]) Update(item *Item[T], value T, priority int) {
|
||||
heap.Fix(pq, item.index)
|
||||
}
|
||||
|
||||
// errFull / errNoSuchElement are INTERNAL sentinels: caught by errors.Is
|
||||
// inside the compaction inspector / scheduler loop and never serialized
|
||||
// across any gRPC boundary. See docs/dev/error_sentinel_convention.md.
|
||||
var (
|
||||
ErrFull = errors.New("compaction queue is full")
|
||||
ErrNoSuchElement = errors.New("compaction queue has no element")
|
||||
errFull = errors.New("compaction queue is full")
|
||||
errNoSuchElement = errors.New("compaction queue has no element")
|
||||
)
|
||||
|
||||
type Prioritizer func(t CompactionTask) int
|
||||
@@ -101,7 +104,7 @@ func (q *CompactionQueue) Enqueue(t CompactionTask) error {
|
||||
q.lock.Lock()
|
||||
defer q.lock.Unlock()
|
||||
if q.capacity > 0 && len(q.pq) >= q.capacity {
|
||||
return ErrFull
|
||||
return errFull
|
||||
}
|
||||
|
||||
heap.Push(&q.pq, &Item[CompactionTask]{value: t, priority: q.prioritizer(t)})
|
||||
@@ -113,7 +116,7 @@ func (q *CompactionQueue) Dequeue() (CompactionTask, error) {
|
||||
defer q.lock.Unlock()
|
||||
|
||||
if len(q.pq) == 0 {
|
||||
return nil, ErrNoSuchElement
|
||||
return nil, errNoSuchElement
|
||||
}
|
||||
|
||||
item := heap.Pop(&q.pq).(*Item[CompactionTask])
|
||||
|
||||
@@ -23,7 +23,6 @@ import (
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"github.com/cockroachdb/errors"
|
||||
"github.com/samber/lo"
|
||||
"go.uber.org/atomic"
|
||||
"go.uber.org/zap"
|
||||
@@ -629,7 +628,7 @@ func (t *clusteringCompactionTask) processAnalyzing() error {
|
||||
}
|
||||
case indexpb.JobState_JobStateFailed:
|
||||
log.Warn("analyze task fail", zap.Int64("analyzeID", t.GetTaskProto().GetAnalyzeTaskID()))
|
||||
return errors.New(analyzeTask.FailReason)
|
||||
return merr.WrapErrServiceInternalMsg(analyzeTask.FailReason)
|
||||
default:
|
||||
}
|
||||
return nil
|
||||
|
||||
@@ -319,7 +319,7 @@ func (t *l0CompactionTask) selectFlushedSegment() ([]*SegmentInfo, []*datapb.Com
|
||||
for _, info := range flushedSegments {
|
||||
// Sealed is unexpected, fail fast
|
||||
if info.GetState() == commonpb.SegmentState_Sealed {
|
||||
return nil, nil, fmt.Errorf("L0 compaction selected invalid sealed segment %d", info.GetID())
|
||||
return nil, nil, merr.WrapErrServiceInternalMsg("L0 compaction selected invalid sealed segment %d", info.GetID())
|
||||
}
|
||||
|
||||
sealedSegBinlogs = append(sealedSegBinlogs, &datapb.CompactionSegmentBinlogs{
|
||||
@@ -455,7 +455,7 @@ func buildL0V3DeltaLogEntries(segmentID int64, deltalogs []*datapb.FieldBinlog)
|
||||
for _, binlog := range fieldBinlog.GetBinlogs() {
|
||||
path := binlog.GetLogPath()
|
||||
if path == "" {
|
||||
return nil, merr.WrapErrServiceInternal(fmt.Sprintf("L0 V3 compaction result missing deltalog path for segment %d, logID %d", segmentID, binlog.GetLogID()))
|
||||
return nil, merr.WrapErrServiceInternalMsg("L0 V3 compaction result missing deltalog path for segment %d, logID %d", segmentID, binlog.GetLogID())
|
||||
}
|
||||
entries = append(entries, packed.DeltaLogEntry{
|
||||
Path: path,
|
||||
|
||||
@@ -398,7 +398,7 @@ func (t *mixCompactionTask) BuildCompactionRequest() (*datapb.CompactionPlan, er
|
||||
resources, err := t.meta.GetFileResources(context.Background(), taskSchema.GetFileResourceIds()...)
|
||||
if err != nil {
|
||||
log.Warn("get file resources for collection failed", zap.Int64("collectionID", taskProto.GetCollectionID()), zap.Error(err))
|
||||
return nil, errors.Errorf("get file resources for sort compaction failed: %v", err)
|
||||
return nil, merr.Wrap(err, "get file resources for sort compaction failed")
|
||||
}
|
||||
plan.FileResources = resources
|
||||
}
|
||||
@@ -411,7 +411,7 @@ func (t *mixCompactionTask) BuildCompactionRequest() (*datapb.CompactionPlan, er
|
||||
return nil, merr.WrapErrSegmentNotFound(segID)
|
||||
}
|
||||
if taskSchemaVersion < segInfo.GetSchemaVersion() {
|
||||
return nil, merr.WrapErrIllegalCompactionPlan(fmt.Sprintf("compaction task schema version %d is older than input segment %d schema version %d", taskSchemaVersion, segInfo.GetID(), segInfo.GetSchemaVersion()))
|
||||
return nil, merr.WrapErrIllegalCompactionPlanMsg("compaction task schema version %d is older than input segment %d schema version %d", taskSchemaVersion, segInfo.GetID(), segInfo.GetSchemaVersion())
|
||||
}
|
||||
plan.SegmentBinlogs = append(plan.SegmentBinlogs, &datapb.CompactionSegmentBinlogs{
|
||||
SegmentID: segID,
|
||||
|
||||
@@ -225,7 +225,7 @@ func (t *compactionTrigger) getCollection(collectionID UniqueID) (*collectionInf
|
||||
defer cancel()
|
||||
coll, err := t.handler.GetCollection(ctx, collectionID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("collection ID %d not found, err: %w", collectionID, err)
|
||||
return nil, merr.Wrapf(err, "collection ID %d not found", collectionID)
|
||||
}
|
||||
return coll, nil
|
||||
}
|
||||
@@ -588,7 +588,10 @@ func (t *compactionTrigger) getCandidates(signal *compactionSignal) ([]chanPartS
|
||||
segments := t.meta.SelectSegments(context.TODO(), filters...)
|
||||
// some criterion not met or conflicted
|
||||
if len(signal.segmentIDs) > 0 && len(segments) != len(signal.segmentIDs) {
|
||||
return nil, merr.WrapErrServiceInternal("not all segment ids provided could be compacted")
|
||||
// SelectSegments also filters segments that are transiently mid-flush /
|
||||
// compacting / just dropped, so a count mismatch is usually server-side
|
||||
// state, not a bad id from the caller.
|
||||
return nil, merr.WrapErrServiceInternalMsg("not all segment ids provided could be compacted")
|
||||
}
|
||||
|
||||
type category struct {
|
||||
|
||||
@@ -583,7 +583,7 @@ func AssembleCopySegmentRequest(task CopySegmentTask, job CopySegmentJob) (*data
|
||||
if _, exists := newBuildIDs[srcBuildID]; !exists {
|
||||
newID, err := t.alloc.AllocID(ctx)
|
||||
if err != nil {
|
||||
return merr.WrapErrServiceInternal(fmt.Sprintf("failed to allocate new buildID for source buildID %d", srcBuildID), err.Error())
|
||||
return merr.Wrapf(err, "failed to allocate new buildID for source buildID %d", srcBuildID)
|
||||
}
|
||||
newBuildIDs[srcBuildID] = newID
|
||||
}
|
||||
|
||||
@@ -27,6 +27,7 @@ import (
|
||||
"github.com/milvus-io/milvus/internal/streamingcoord/server/broadcaster/registry"
|
||||
"github.com/milvus-io/milvus/pkg/v3/log"
|
||||
"github.com/milvus-io/milvus/pkg/v3/streaming/util/message"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
)
|
||||
|
||||
// RegisterDDLCallbacks registers the ddl callbacks.
|
||||
@@ -89,7 +90,7 @@ func (s *Server) startBroadcastWithCollectionID(ctx context.Context, collectionI
|
||||
func (s *Server) startBroadcastForRestoreSnapshot(ctx context.Context, collectionID int64, snapshotName string) (broadcaster.BroadcastAPI, error) {
|
||||
coll, err := s.broker.DescribeCollectionInternal(ctx, collectionID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("collection %d does not exist: %w", collectionID, err)
|
||||
return nil, merr.Wrapf(err, "collection %d does not exist", collectionID)
|
||||
}
|
||||
dbName := coll.GetDbName()
|
||||
collectionName := coll.GetCollectionName()
|
||||
@@ -161,15 +162,15 @@ func (s *Server) validateRestoreSnapshotResources(ctx context.Context, collectio
|
||||
sourceCollectionID := snapshotData.SnapshotInfo.GetCollectionId()
|
||||
snapshot, err := s.meta.snapshotMeta.GetSnapshot(ctx, sourceCollectionID, snapshotData.SnapshotInfo.GetName())
|
||||
if err != nil {
|
||||
return fmt.Errorf("snapshot %s does not exist for collection %d: %w",
|
||||
snapshotData.SnapshotInfo.GetName(), sourceCollectionID, err)
|
||||
return merr.Wrapf(err, "snapshot %s does not exist for collection %d",
|
||||
snapshotData.SnapshotInfo.GetName(), sourceCollectionID)
|
||||
}
|
||||
log.Info("snapshot validated", zap.String("snapshotName", snapshot.GetName()))
|
||||
|
||||
// ========== Validate Collection Exists ==========
|
||||
coll, err := s.broker.DescribeCollectionInternal(ctx, collectionID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("collection %d does not exist: %w", collectionID, err)
|
||||
return merr.Wrapf(err, "collection %d does not exist", collectionID)
|
||||
}
|
||||
dbName := coll.GetDbName()
|
||||
collectionName := coll.GetCollectionName()
|
||||
@@ -180,7 +181,7 @@ func (s *Server) validateRestoreSnapshotResources(ctx context.Context, collectio
|
||||
// ========== Validate Partitions Exist ==========
|
||||
partitionsResp, err := s.broker.ShowPartitions(ctx, collectionID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to get partitions for collection %d: %w", collectionID, err)
|
||||
return merr.Wrapf(err, "failed to get partitions for collection %d", collectionID)
|
||||
}
|
||||
|
||||
// Build set of existing partition names
|
||||
@@ -192,8 +193,7 @@ func (s *Server) validateRestoreSnapshotResources(ctx context.Context, collectio
|
||||
// Check all snapshot partitions exist
|
||||
for partName := range snapshotData.Collection.GetPartitions() {
|
||||
if !existingPartitions[partName] {
|
||||
return fmt.Errorf("partition %s does not exist in collection %d",
|
||||
partName, collectionID)
|
||||
return merr.WrapErrPartitionNotFound(partName, fmt.Sprintf("partition does not exist in collection %d", collectionID))
|
||||
}
|
||||
}
|
||||
log.Info("partitions validated", zap.Int("count", len(existingPartitions)))
|
||||
@@ -211,8 +211,7 @@ func (s *Server) validateRestoreSnapshotResources(ctx context.Context, collectio
|
||||
}
|
||||
}
|
||||
if !indexFound {
|
||||
return fmt.Errorf("index %s for field %d does not exist in collection %d",
|
||||
indexInfo.GetIndexName(), indexInfo.GetFieldID(), collectionID)
|
||||
return merr.WrapErrIndexNotFound(indexInfo.GetIndexName(), fmt.Sprintf("index for field %d does not exist in collection %d", indexInfo.GetFieldID(), collectionID))
|
||||
}
|
||||
}
|
||||
log.Info("indexes validated", zap.Int("count", len(snapshotData.Indexes)))
|
||||
|
||||
@@ -134,10 +134,10 @@ func (s *Server) validateImportReplication(ctx context.Context, options []*commo
|
||||
}
|
||||
|
||||
if !paramtable.Get().DataCoordCfg.ImportInReplicatingCluster.GetAsBool() {
|
||||
return merr.WrapErrImportFailed("import in replicating cluster is not supported yet")
|
||||
return merr.WrapErrOperationNotSupportedMsg("import in replicating cluster is not supported yet")
|
||||
}
|
||||
if importutilv2.IsAutoCommit(options) {
|
||||
return merr.WrapErrImportFailed("auto_commit=true import in replicating cluster is not supported")
|
||||
return merr.WrapErrOperationNotSupportedMsg("auto_commit=true import in replicating cluster is not supported")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -168,14 +168,14 @@ func (s *Server) broadcastImport(ctx context.Context,
|
||||
|
||||
// Validate the request before broadcasting
|
||||
if err := s.validateImportRequest(ctx, msgFiles, options); err != nil {
|
||||
return errors.Wrap(err, "failed to validate import request")
|
||||
return merr.Wrap(err, "failed to validate import request")
|
||||
}
|
||||
|
||||
// Get database name from collection metadata via broker
|
||||
// This is safer than extracting from schema which may be stale
|
||||
broadcaster, err := s.startBroadcastWithCollectionID(ctx, collectionID)
|
||||
if err != nil {
|
||||
return errors.Wrap(err, "failed to start broadcast with collection id")
|
||||
return merr.Wrap(err, "failed to start broadcast with collection id")
|
||||
}
|
||||
defer broadcaster.Close()
|
||||
|
||||
|
||||
@@ -107,8 +107,9 @@ func (s *ImportCallbacksSuite) TestValidateImportRequest_MaxJobsExceededReturnsE
|
||||
err := server.validateImportRequest(ctx, files, options)
|
||||
|
||||
s.Error(err)
|
||||
// ValidateMaxImportJobExceed returns WrapErrImportFailed, not ErrServiceQuotaExceeded
|
||||
s.True(errors.Is(err, merr.ErrImportFailed))
|
||||
// Job-count backpressure is a server-side condition -> ErrImportSysFailed
|
||||
// (must not be bucketed as fail_input).
|
||||
s.True(errors.Is(err, merr.ErrImportSysFailed))
|
||||
s.Contains(err.Error(), "The number of jobs has reached the limit")
|
||||
}
|
||||
|
||||
@@ -185,7 +186,7 @@ func (s *ImportCallbacksSuite) TestValidateImportRequest_ReplicatingClusterRetur
|
||||
err := server.validateImportRequest(ctx, files, options)
|
||||
|
||||
s.Error(err)
|
||||
s.True(errors.Is(err, merr.ErrImportFailed))
|
||||
s.True(errors.Is(err, merr.ErrOperationNotSupported))
|
||||
s.Contains(err.Error(), "replicating cluster")
|
||||
}
|
||||
|
||||
@@ -230,7 +231,7 @@ func (s *ImportCallbacksSuite) TestValidateImportRequest_ReplicatingClusterEnabl
|
||||
{Key: "timeout", Value: "300s"},
|
||||
})
|
||||
s.Error(err)
|
||||
s.True(errors.Is(err, merr.ErrImportFailed))
|
||||
s.True(errors.Is(err, merr.ErrOperationNotSupported))
|
||||
s.Contains(err.Error(), "auto_commit=true")
|
||||
|
||||
err = server.validateImportRequest(ctx, files, []*commonpb.KeyValuePair{
|
||||
@@ -1159,7 +1160,8 @@ func testBroadcastRequiresVchannels(t *testing.T, broadcastFn func(*Server, cont
|
||||
|
||||
err := broadcastFn(server, ctx, job)
|
||||
assert.Error(t, err)
|
||||
assert.True(t, errors.Is(err, merr.ErrImportFailed))
|
||||
// Missing vchannels is internal broadcast state -> ErrImportSysFailed.
|
||||
assert.True(t, errors.Is(err, merr.ErrImportSysFailed))
|
||||
assert.Contains(t, err.Error(), "job 7 has no vchannels")
|
||||
}
|
||||
|
||||
|
||||
@@ -636,7 +636,8 @@ func TestValidateRestoreSnapshotResources_IndexNotFound(t *testing.T) {
|
||||
err := server.validateRestoreSnapshotResources(ctx, 100, buildBaseSnapshotData())
|
||||
|
||||
assert.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "index vec_idx for field 100 does not exist")
|
||||
assert.Contains(t, err.Error(), "index for field 100 does not exist")
|
||||
assert.Contains(t, err.Error(), "vec_idx")
|
||||
}
|
||||
|
||||
func TestValidateRestoreSnapshotResources_IndexFieldMismatch(t *testing.T) {
|
||||
@@ -656,7 +657,8 @@ func TestValidateRestoreSnapshotResources_IndexFieldMismatch(t *testing.T) {
|
||||
err := server.validateRestoreSnapshotResources(ctx, 100, buildBaseSnapshotData())
|
||||
|
||||
assert.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "index vec_idx for field 100 does not exist")
|
||||
assert.Contains(t, err.Error(), "index for field 100 does not exist")
|
||||
assert.Contains(t, err.Error(), "vec_idx")
|
||||
}
|
||||
|
||||
func TestValidateRestoreSnapshotResources_Success(t *testing.T) {
|
||||
|
||||
@@ -20,6 +20,9 @@ import (
|
||||
"fmt"
|
||||
|
||||
"github.com/cockroachdb/errors"
|
||||
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/typeutil"
|
||||
)
|
||||
|
||||
// errors for VerifyResponse
|
||||
@@ -34,7 +37,7 @@ func msgDataCoordIsUnhealthy(coordID UniqueID) string {
|
||||
}
|
||||
|
||||
func errDataCoordIsUnhealthy(coordID UniqueID) error {
|
||||
return errors.New(msgDataCoordIsUnhealthy(coordID))
|
||||
return merr.WrapErrServiceNotReady(typeutil.DataCoordRole, coordID, "not ready")
|
||||
}
|
||||
|
||||
func msgSegmentNotFound(segID UniqueID) string {
|
||||
|
||||
@@ -306,7 +306,7 @@ func (m *externalCollectionRefreshManager) cleanupExploreTempForJob(jobID int64)
|
||||
func (m *externalCollectionRefreshManager) applyFinishedJobSegments(ctx context.Context, job *datapb.ExternalCollectionRefreshJob) error {
|
||||
tasks := m.refreshMeta.GetTasksByJobID(job.GetJobId())
|
||||
if len(tasks) == 0 {
|
||||
return fmt.Errorf("job %d has no tasks to apply", job.GetJobId())
|
||||
return merr.WrapErrServiceInternalMsg("job %d has no tasks to apply", job.GetJobId())
|
||||
}
|
||||
|
||||
keptSet := make(map[int64]struct{})
|
||||
@@ -315,11 +315,11 @@ func (m *externalCollectionRefreshManager) applyFinishedJobSegments(ctx context.
|
||||
updatedSegments := make([]*datapb.SegmentInfo, 0)
|
||||
for _, task := range tasks {
|
||||
if task.GetState() != indexpb.JobState_JobStateFinished {
|
||||
return fmt.Errorf("job %d has non-finished task %d in state %s",
|
||||
return merr.WrapErrServiceInternalMsg("job %d has non-finished task %d in state %s",
|
||||
job.GetJobId(), task.GetTaskId(), task.GetState().String())
|
||||
}
|
||||
if !task.GetResultReady() {
|
||||
return fmt.Errorf("job %d has finished task %d without persisted refresh result; please retry refresh",
|
||||
return merr.WrapErrServiceInternalMsg("job %d has finished task %d without persisted refresh result; please retry refresh",
|
||||
job.GetJobId(), task.GetTaskId())
|
||||
}
|
||||
for _, segmentID := range task.GetKeptSegments() {
|
||||
@@ -334,7 +334,7 @@ func (m *externalCollectionRefreshManager) applyFinishedJobSegments(ctx context.
|
||||
continue
|
||||
}
|
||||
if _, ok := updatedSet[segment.GetID()]; ok {
|
||||
return fmt.Errorf("job %d has duplicate updated segment %d from task %d",
|
||||
return merr.WrapErrServiceInternalMsg("job %d has duplicate updated segment %d from task %d",
|
||||
job.GetJobId(), segment.GetID(), task.GetTaskId())
|
||||
}
|
||||
updatedSet[segment.GetID()] = struct{}{}
|
||||
@@ -714,7 +714,7 @@ func (m *externalCollectionRefreshManager) createTasksForJob(
|
||||
if errors.Is(err, packed.ErrLoonTransient) {
|
||||
return nil, newNonRetriableJobError("explore external files failed: %v", err)
|
||||
}
|
||||
return nil, fmt.Errorf("failed to explore external files: %w", err)
|
||||
return nil, merr.WrapErrServiceInternalErr(err, "failed to explore external files")
|
||||
}
|
||||
if len(allFiles) == 0 {
|
||||
return nil, newNonRetriableJobError("no files found in external source: %s", job.GetExternalSource())
|
||||
@@ -829,7 +829,7 @@ func normalizeRefreshJobProgress(job *datapb.ExternalCollectionRefreshJob, state
|
||||
func (m *externalCollectionRefreshManager) GetJobProgress(ctx context.Context, jobID int64) (*datapb.ExternalCollectionRefreshJob, error) {
|
||||
job := m.refreshMeta.GetJob(jobID)
|
||||
if job == nil {
|
||||
return nil, fmt.Errorf("job %d not found", jobID)
|
||||
return nil, merr.WrapErrParameterInvalidMsg("refresh job %d not found", jobID)
|
||||
}
|
||||
|
||||
// Aggregate state and progress from tasks
|
||||
@@ -877,17 +877,17 @@ func (m *externalCollectionRefreshManager) exploreExternalFiles(
|
||||
// when both present.
|
||||
if job.GetExternalSource() != "" {
|
||||
if err := externalspec.ValidateSourceAndSpec(job.GetExternalSource(), job.GetExternalSpec()); err != nil {
|
||||
return nil, "", fmt.Errorf("external source/spec failed revalidation: %w", err)
|
||||
return nil, "", merr.Wrap(err, "external source/spec failed revalidation")
|
||||
}
|
||||
}
|
||||
spec, err := externalspec.ParseExternalSpec(job.GetExternalSpec())
|
||||
if err != nil {
|
||||
return nil, "", fmt.Errorf("failed to parse external spec: %w", err)
|
||||
return nil, "", merr.Wrap(err, "failed to parse external spec")
|
||||
}
|
||||
|
||||
collInfo := m.mt.GetCollection(job.GetCollectionId())
|
||||
if collInfo == nil {
|
||||
return nil, "", fmt.Errorf("collection %d not found", job.GetCollectionId())
|
||||
return nil, "", merr.WrapErrCollectionNotFound(job.GetCollectionId())
|
||||
}
|
||||
|
||||
columns := packed.GetColumnNamesFromSchema(collInfo.Schema)
|
||||
@@ -908,7 +908,7 @@ func (m *externalCollectionRefreshManager) exploreExternalFiles(
|
||||
extfs,
|
||||
)
|
||||
if err != nil {
|
||||
return nil, "", fmt.Errorf("ExploreFilesReturnManifestPath failed: %w", err)
|
||||
return nil, "", merr.WrapErrServiceInternalErr(err, "failed to explore files returning manifest path")
|
||||
}
|
||||
|
||||
// Convert to proto type
|
||||
|
||||
@@ -18,7 +18,6 @@ package datacoord
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
@@ -30,6 +29,7 @@ import (
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/datapb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/lock"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/timerecord"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/typeutil"
|
||||
)
|
||||
@@ -295,7 +295,7 @@ func (m *externalCollectionRefreshMeta) mutateJob(
|
||||
) (applied bool, err error) {
|
||||
job, ok := m.jobs.Get(jobID)
|
||||
if !ok {
|
||||
return false, fmt.Errorf("job %d not found", jobID)
|
||||
return false, merr.WrapErrServiceInternalMsg("job %d not found", jobID)
|
||||
}
|
||||
|
||||
m.jobLock.Lock(job.GetCollectionId())
|
||||
@@ -304,7 +304,7 @@ func (m *externalCollectionRefreshMeta) mutateJob(
|
||||
// Re-fetch after lock
|
||||
job, ok = m.jobs.Get(jobID)
|
||||
if !ok {
|
||||
return false, fmt.Errorf("job %d not found", jobID)
|
||||
return false, merr.WrapErrServiceInternalMsg("job %d not found", jobID)
|
||||
}
|
||||
|
||||
cloneJob := proto.Clone(job).(*datapb.ExternalCollectionRefreshJob)
|
||||
@@ -385,7 +385,7 @@ func (m *externalCollectionRefreshMeta) UpdateJobStateWithPreApply(
|
||||
) (bool, error) {
|
||||
job, ok := m.jobs.Get(jobID)
|
||||
if !ok {
|
||||
return false, fmt.Errorf("job %d not found", jobID)
|
||||
return false, merr.WrapErrServiceInternalMsg("job %d not found", jobID)
|
||||
}
|
||||
|
||||
m.jobLock.Lock(job.GetCollectionId())
|
||||
@@ -395,7 +395,7 @@ func (m *externalCollectionRefreshMeta) UpdateJobStateWithPreApply(
|
||||
// terminal state owns the one-time side effects.
|
||||
job, ok = m.jobs.Get(jobID)
|
||||
if !ok {
|
||||
return false, fmt.Errorf("job %d not found", jobID)
|
||||
return false, merr.WrapErrServiceInternalMsg("job %d not found", jobID)
|
||||
}
|
||||
if job.GetState() == indexpb.JobState_JobStateFinished ||
|
||||
job.GetState() == indexpb.JobState_JobStateFailed {
|
||||
@@ -416,7 +416,7 @@ func (m *externalCollectionRefreshMeta) UpdateJobStateWithPreApply(
|
||||
log.Warn("update job state after pre-apply failed",
|
||||
zap.Int64("jobID", jobID),
|
||||
zap.Error(saveErr))
|
||||
return false, fmt.Errorf("pre-apply failed: %w; additionally failed to persist Failed job state: %v", err, saveErr)
|
||||
return false, merr.Wrapf(err, "pre-apply failed; additionally failed to persist Failed job state: %v", saveErr)
|
||||
}
|
||||
m.jobs.Insert(jobID, cloneJob)
|
||||
m.addToCollectionJobs(cloneJob)
|
||||
@@ -613,7 +613,7 @@ func (m *externalCollectionRefreshMeta) mutateTask(
|
||||
) (applied bool, cloned *datapb.ExternalCollectionRefreshTask, err error) {
|
||||
task, ok := m.tasks.Get(taskID)
|
||||
if !ok {
|
||||
return false, nil, fmt.Errorf("task %d not found", taskID)
|
||||
return false, nil, merr.WrapErrServiceInternalMsg("task %d not found", taskID)
|
||||
}
|
||||
|
||||
m.taskLock.Lock(task.GetJobId())
|
||||
@@ -622,7 +622,7 @@ func (m *externalCollectionRefreshMeta) mutateTask(
|
||||
// Re-fetch after lock
|
||||
task, ok = m.tasks.Get(taskID)
|
||||
if !ok {
|
||||
return false, nil, fmt.Errorf("task %d not found", taskID)
|
||||
return false, nil, merr.WrapErrServiceInternalMsg("task %d not found", taskID)
|
||||
}
|
||||
|
||||
cloneTask := proto.Clone(task).(*datapb.ExternalCollectionRefreshTask)
|
||||
|
||||
@@ -1067,13 +1067,13 @@ func (gc *garbageCollector) isExpire(dropts Timestamp) bool {
|
||||
// Returns segmentID or error if path doesn't match.
|
||||
func parseV3SegmentID(rootPath, filePath string) (int64, error) {
|
||||
if !strings.HasPrefix(filePath, rootPath) {
|
||||
return 0, fmt.Errorf("path %q does not contain rootPath %q", filePath, rootPath)
|
||||
return 0, merr.WrapErrServiceInternalMsg("path %q does not contain rootPath %q", filePath, rootPath)
|
||||
}
|
||||
p := strings.TrimPrefix(filePath[len(rootPath):], "/")
|
||||
parts := strings.Split(p, "/")
|
||||
// Minimum: insert_log/coll/part/seg/something
|
||||
if len(parts) < 5 || parts[0] != common.SegmentInsertLogPath {
|
||||
return 0, fmt.Errorf("not a V3 insert_log path: %s", filePath)
|
||||
return 0, merr.WrapErrServiceInternalMsg("not a V3 insert_log path: %s", filePath)
|
||||
}
|
||||
return strconv.ParseInt(parts[3], 10, 64)
|
||||
}
|
||||
|
||||
@@ -18,7 +18,6 @@ package datacoord
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"path"
|
||||
"strings"
|
||||
"sync"
|
||||
@@ -34,6 +33,7 @@ import (
|
||||
"github.com/milvus-io/milvus/pkg/v3/log"
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/conc"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/typeutil"
|
||||
)
|
||||
@@ -271,8 +271,7 @@ func (lobCtx *lobGCContext) collectLOBFilesFromSegment(ctx context.Context, segm
|
||||
|
||||
lobFiles, err := lobCtx.cache.Get(ctx, manifestPath, lobCtx.storageConfig)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to get LOB files from manifest for segment %d (path=%s): %w",
|
||||
segment.GetID(), manifestPath, err)
|
||||
return merr.WrapErrServiceInternalErr(err, "failed to get LOB files from manifest for segment %d (path=%s)", segment.GetID(), manifestPath)
|
||||
}
|
||||
|
||||
for _, lobFile := range lobFiles {
|
||||
|
||||
@@ -18,7 +18,6 @@ package datacoord
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/hashicorp/golang-lru/v2/expirable"
|
||||
@@ -345,7 +344,7 @@ func (m *importMeta) HandleCommitVchannel(ctx context.Context, jobID int64, vcha
|
||||
|
||||
job := m.jobs[jobID]
|
||||
if job == nil {
|
||||
return merr.WrapErrImportFailed(fmt.Sprintf("job %d not found", jobID))
|
||||
return merr.WrapErrImportSysFailedMsg("job %d not found", jobID)
|
||||
}
|
||||
// Idempotency: if vchannel already committed, skip.
|
||||
for _, c := range job.GetCommittedVchannels() {
|
||||
|
||||
@@ -104,7 +104,7 @@ func (s *ImportServicesSuite) TestImportV2_AllocatorNilReturnsError() {
|
||||
|
||||
s.NoError(err)
|
||||
s.NotNil(resp)
|
||||
s.True(errors.Is(merr.Error(resp.GetStatus()), merr.ErrImportFailed))
|
||||
s.True(errors.Is(merr.Error(resp.GetStatus()), merr.ErrServiceUnavailable))
|
||||
s.Contains(resp.GetStatus().GetReason(), "allocator not initialized")
|
||||
}
|
||||
|
||||
@@ -114,7 +114,7 @@ func (s *ImportServicesSuite) TestImportV2_AllocatorFailsReturnsError() {
|
||||
server.stateCode.Store(commonpb.StateCode_Healthy)
|
||||
|
||||
mockAllocator := allocator.NewMockAllocator(s.T())
|
||||
mockAllocator.EXPECT().AllocN(mock.Anything).Return(int64(0), int64(0), errors.New("allocation failed"))
|
||||
mockAllocator.EXPECT().AllocN(mock.Anything).Return(int64(0), int64(0), merr.WrapErrServiceUnavailable("allocation failed"))
|
||||
server.allocator = mockAllocator
|
||||
|
||||
req := &internalpb.ImportRequestInternal{
|
||||
@@ -127,7 +127,7 @@ func (s *ImportServicesSuite) TestImportV2_AllocatorFailsReturnsError() {
|
||||
|
||||
s.NoError(err)
|
||||
s.NotNil(resp)
|
||||
s.True(errors.Is(merr.Error(resp.GetStatus()), merr.ErrImportFailed))
|
||||
s.True(errors.Is(merr.Error(resp.GetStatus()), merr.ErrServiceUnavailable))
|
||||
s.Contains(resp.GetStatus().GetReason(), "failed to allocate job ID")
|
||||
}
|
||||
|
||||
@@ -164,7 +164,7 @@ func (s *ImportServicesSuite) TestImportV2_BroadcastFailsReturnsError() {
|
||||
// Mock StartBroadcastWithResourceKeys to fail
|
||||
mockBroadcast := mockey.Mock(broadcast.StartBroadcastWithResourceKeys).To(
|
||||
func(ctx context.Context, keys ...message.ResourceKey) (broadcaster.BroadcastAPI, error) {
|
||||
return nil, errors.New("broadcast failed")
|
||||
return nil, merr.WrapErrServiceUnavailable("broadcast failed")
|
||||
}).Build()
|
||||
defer mockBroadcast.UnPatch()
|
||||
|
||||
@@ -199,8 +199,8 @@ func (s *ImportServicesSuite) TestImportV2_BroadcastFailsReturnsError() {
|
||||
|
||||
s.NoError(err)
|
||||
s.NotNil(resp)
|
||||
s.True(errors.Is(merr.Error(resp.GetStatus()), merr.ErrImportFailed))
|
||||
s.Contains(resp.GetStatus().GetReason(), "failed to broadcast import")
|
||||
s.True(errors.Is(merr.Error(resp.GetStatus()), merr.ErrServiceUnavailable))
|
||||
s.Contains(resp.GetStatus().GetReason(), "broadcast")
|
||||
}
|
||||
|
||||
func (s *ImportServicesSuite) TestImportV2_SuccessReturnsJobID() {
|
||||
@@ -395,7 +395,7 @@ func (s *ImportServicesSuite) TestCreateImportJobFromAck_AllocatorFailsReturnsEr
|
||||
server.stateCode.Store(commonpb.StateCode_Healthy)
|
||||
|
||||
mockAllocator := allocator.NewMockAllocator(s.T())
|
||||
mockAllocator.EXPECT().AllocN(mock.Anything).Return(int64(0), int64(0), errors.New("allocation failed"))
|
||||
mockAllocator.EXPECT().AllocN(mock.Anything).Return(int64(0), int64(0), merr.WrapErrServiceUnavailable("allocation failed"))
|
||||
server.allocator = mockAllocator
|
||||
|
||||
req := &internalpb.ImportRequestInternal{
|
||||
@@ -412,7 +412,7 @@ func (s *ImportServicesSuite) TestCreateImportJobFromAck_AllocatorFailsReturnsEr
|
||||
|
||||
s.NoError(err)
|
||||
s.NotNil(resp)
|
||||
s.True(errors.Is(merr.Error(resp.GetStatus()), merr.ErrImportFailed))
|
||||
s.True(errors.Is(merr.Error(resp.GetStatus()), merr.ErrServiceUnavailable))
|
||||
s.Contains(resp.GetStatus().GetReason(), "alloc id failed")
|
||||
}
|
||||
|
||||
@@ -493,7 +493,7 @@ func (s *ImportServicesSuite) TestCreateImportJobFromAck_AddJobFailsReturnsError
|
||||
catalog.EXPECT().ListImportJobs(mock.Anything).Return(nil, nil)
|
||||
catalog.EXPECT().ListPreImportTasks(mock.Anything).Return(nil, nil)
|
||||
catalog.EXPECT().ListImportTasks(mock.Anything).Return(nil, nil)
|
||||
catalog.EXPECT().SaveImportJob(mock.Anything, mock.Anything).Return(errors.New("save job failed"))
|
||||
catalog.EXPECT().SaveImportJob(mock.Anything, mock.Anything).Return(merr.WrapErrServiceUnavailable("save job failed"))
|
||||
|
||||
importMeta, err := NewImportMeta(context.TODO(), catalog, nil, nil)
|
||||
s.NoError(err)
|
||||
@@ -528,7 +528,7 @@ func (s *ImportServicesSuite) TestCreateImportJobFromAck_AddJobFailsReturnsError
|
||||
|
||||
s.NoError(err)
|
||||
s.NotNil(resp)
|
||||
s.True(errors.Is(merr.Error(resp.GetStatus()), merr.ErrImportFailed))
|
||||
s.True(errors.Is(merr.Error(resp.GetStatus()), merr.ErrServiceUnavailable))
|
||||
s.Contains(resp.GetStatus().GetReason(), "add import job failed")
|
||||
}
|
||||
|
||||
|
||||
@@ -670,8 +670,8 @@ func ListBinlogsAndGroupBySegment(ctx context.Context,
|
||||
return nil, merr.WrapErrImportFailed("no insert binlogs to import")
|
||||
}
|
||||
if len(importFile.GetPaths()) > 2 {
|
||||
return nil, merr.WrapErrImportFailed(fmt.Sprintf("too many input paths for binlog import. "+
|
||||
"Valid paths length should be one or two, but got paths:%s", importFile.GetPaths()))
|
||||
return nil, merr.WrapErrImportFailedMsg("too many input paths for binlog import. "+
|
||||
"Valid paths length should be one or two, but got paths:%s", importFile.GetPaths())
|
||||
}
|
||||
|
||||
insertPrefix := importFile.GetPaths()[0]
|
||||
@@ -790,18 +790,18 @@ func ListBinlogImportRequestFiles(ctx context.Context, cm storage.ChunkManager,
|
||||
}
|
||||
err := conc.AwaitAll(futures...)
|
||||
if err != nil {
|
||||
return nil, merr.WrapErrImportFailed(fmt.Sprintf("list binlogs failed, err=%s", err))
|
||||
return nil, merr.WrapErrServiceUnavailableMsg("list binlogs failed, err=%s", err)
|
||||
}
|
||||
|
||||
resFiles = lo.Filter(resFiles, func(file *internalpb.ImportFile, _ int) bool {
|
||||
return len(file.GetPaths()) > 0
|
||||
})
|
||||
if len(resFiles) == 0 {
|
||||
return nil, merr.WrapErrImportFailed(fmt.Sprintf("no binlog to import, input=%s", reqFiles))
|
||||
return nil, merr.WrapErrImportFailedMsg("no binlog to import, input=%s", reqFiles)
|
||||
}
|
||||
if len(resFiles) > paramtable.Get().DataCoordCfg.MaxFilesPerImportReq.GetAsInt() {
|
||||
return nil, merr.WrapErrImportFailed(fmt.Sprintf("The max number of import files should not exceed %d, but got %d",
|
||||
paramtable.Get().DataCoordCfg.MaxFilesPerImportReq.GetAsInt(), len(resFiles)))
|
||||
return nil, merr.WrapErrImportFailedMsg("The max number of import files should not exceed %d, but got %d",
|
||||
paramtable.Get().DataCoordCfg.MaxFilesPerImportReq.GetAsInt(), len(resFiles))
|
||||
}
|
||||
log.Info("list binlogs prefixes for import done", zap.Int("num", len(resFiles)), zap.Any("binlog_prefixes", resFiles))
|
||||
return resFiles, nil
|
||||
@@ -812,7 +812,7 @@ func ValidateMaxImportJobExceed(ctx context.Context, importMeta ImportMeta) erro
|
||||
maxNum := paramtable.Get().DataCoordCfg.MaxImportJobNum.GetAsInt()
|
||||
executingNum := importMeta.CountJobBy(ctx, WithoutJobStates(internalpb.ImportJobState_Completed, internalpb.ImportJobState_Failed))
|
||||
if executingNum >= maxNum {
|
||||
return merr.WrapErrImportFailed(
|
||||
return merr.WrapErrImportSysFailed(
|
||||
fmt.Sprintf("The number of jobs has reached the limit, please try again later. " +
|
||||
"If your request is set to only import a single file, " +
|
||||
"please consider importing multiple files in one request for better efficiency."))
|
||||
|
||||
@@ -468,7 +468,7 @@ func (m *indexMeta) canCreateIndex(req *indexpb.CreateIndexRequest, isJSON bool)
|
||||
index.IndexName, index.FieldID, index.IndexParams, index.UserIndexParams, index.TypeParams)),
|
||||
zap.String("current index", fmt.Sprintf("{index_name: %s, field_id: %d, index_params: %v, user_params: %v, type_params: %v}",
|
||||
req.GetIndexName(), req.GetFieldID(), req.GetIndexParams(), req.GetUserIndexParams(), req.GetTypeParams())))
|
||||
return 0, fmt.Errorf("CreateIndex failed: %s", errMsg)
|
||||
return 0, merr.WrapErrParameterInvalidMsg("%s", errMsg)
|
||||
}
|
||||
if req.FieldID == index.FieldID {
|
||||
if isJSON {
|
||||
@@ -484,7 +484,11 @@ func (m *indexMeta) canCreateIndex(req *indexpb.CreateIndexRequest, isJSON bool)
|
||||
// creating multiple indexes on same field is not supported
|
||||
errMsg := "CreateIndex failed: creating multiple indexes on same field is not supported"
|
||||
log.Warn(errMsg)
|
||||
return 0, errors.New(errMsg)
|
||||
// Client-caused conflict, same family as the "one distinct index per
|
||||
// field" case above: return an input-class error rather than
|
||||
// ServiceInternal (code 5, "never return out of Milvus") so the
|
||||
// caller is not blamed with a system error / counted as fail_system.
|
||||
return 0, merr.WrapErrParameterInvalidMsg("%s", errMsg)
|
||||
}
|
||||
}
|
||||
return 0, nil
|
||||
@@ -1012,7 +1016,7 @@ func (m *indexMeta) UpdateVersion(buildID, nodeID UniqueID) error {
|
||||
log.Ctx(m.ctx).Info("IndexCoord metaTable UpdateVersion receive", zap.Int64("buildID", buildID), zap.Int64("nodeID", nodeID))
|
||||
segIdx, ok := m.segmentBuildInfo.Get(buildID)
|
||||
if !ok {
|
||||
return fmt.Errorf("there is no index with buildID: %d", buildID)
|
||||
return merr.WrapErrServiceInternalMsg("there is no index with buildID: %d", buildID)
|
||||
}
|
||||
|
||||
updateFunc := func(segIdx *model.SegmentIndex) error {
|
||||
@@ -1031,7 +1035,7 @@ func (m *indexMeta) UpdateIndexState(buildID UniqueID, state commonpb.IndexState
|
||||
segIdx, ok := m.segmentBuildInfo.Get(buildID)
|
||||
if !ok {
|
||||
log.Ctx(m.ctx).Warn("there is no index with buildID", zap.Int64("buildID", buildID))
|
||||
return fmt.Errorf("there is no index with buildID: %d", buildID)
|
||||
return merr.WrapErrServiceInternalMsg("there is no index with buildID: %d", buildID)
|
||||
}
|
||||
|
||||
updateFunc := func(segIdx *model.SegmentIndex) error {
|
||||
@@ -1141,7 +1145,7 @@ func (m *indexMeta) BuildIndex(buildID UniqueID) error {
|
||||
|
||||
segIdx, ok := m.segmentBuildInfo.Get(buildID)
|
||||
if !ok {
|
||||
return fmt.Errorf("there is no index with buildID: %d", buildID)
|
||||
return merr.WrapErrServiceInternalMsg("there is no index with buildID: %d", buildID)
|
||||
}
|
||||
|
||||
updateFunc := func(segIdx *model.SegmentIndex) error {
|
||||
|
||||
@@ -1052,7 +1052,7 @@ func (s *Server) DropIndex(ctx context.Context, req *indexpb.DropIndexRequest) (
|
||||
}
|
||||
if typeutil.IsVectorType(field.GetDataType()) {
|
||||
log.Warn("vector index cannot be dropped on loaded collection", zap.String("indexName", req.IndexName), zap.Int64("collectionID", req.GetCollectionID()), zap.Int64("fieldID", index.FieldID))
|
||||
return merr.Status(merr.WrapErrParameterInvalidMsg(fmt.Sprintf("vector index cannot be dropped on loaded collection: %d", req.GetCollectionID()))), nil
|
||||
return merr.Status(merr.WrapErrParameterInvalidMsg("vector index cannot be dropped on loaded collection: %d", req.GetCollectionID())), nil
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+20
-12
@@ -816,7 +816,7 @@ func (m *meta) GetSegmentsChannels(segmentIDs []UniqueID) (map[int64]string, err
|
||||
for _, segmentID := range segmentIDs {
|
||||
segment := m.segments.GetSegment(segmentID)
|
||||
if segment == nil {
|
||||
return nil, errors.New(fmt.Sprintf("cannot find segment %d", segmentID))
|
||||
return nil, merr.WrapErrServiceInternalMsg("cannot find segment %d", segmentID)
|
||||
}
|
||||
segChannels[segmentID] = segment.GetInsertChannel()
|
||||
}
|
||||
@@ -840,7 +840,7 @@ func (m *meta) SetState(ctx context.Context, segmentID UniqueID, targetState com
|
||||
if targetState == commonpb.SegmentState_Dropped {
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("segment is not exist with ID = %d", segmentID)
|
||||
return merr.WrapErrSegmentNotFound(segmentID)
|
||||
}
|
||||
// Persist segment updates first.
|
||||
clonedSegment := curSegInfo.Clone()
|
||||
@@ -941,12 +941,12 @@ func (p *updateSegmentPack) Validate() error {
|
||||
segmentInMeta := p.meta.segments.GetSegment(segment.ID)
|
||||
if segmentInMeta.State == commonpb.SegmentState_Flushed && segment.State != commonpb.SegmentState_Dropped {
|
||||
// if the segment is flushed, we should not update the segment meta, ignore the operation directly.
|
||||
return errors.Wrapf(ErrIgnoredSegmentMetaOperation,
|
||||
return merr.Wrapf(errIgnoredSegmentMetaOperation,
|
||||
"segment is flushed, segmentID: %d",
|
||||
segment.ID)
|
||||
}
|
||||
if segment.GetDmlPosition().GetTimestamp() < segmentInMeta.GetDmlPosition().GetTimestamp() {
|
||||
return errors.Wrapf(ErrIgnoredSegmentMetaOperation,
|
||||
return merr.Wrapf(errIgnoredSegmentMetaOperation,
|
||||
"dml time tick is less than the segment meta, segmentID: %d, new incoming time tick: %d, existing time tick: %d",
|
||||
segment.ID,
|
||||
segment.GetDmlPosition().GetTimestamp(),
|
||||
@@ -1223,7 +1223,7 @@ func updateManifestPathIfNewer(segment *SegmentInfo, manifestPath string) error
|
||||
return err
|
||||
}
|
||||
if currentBase != incomingBase {
|
||||
return merr.WrapErrServiceInternal(fmt.Sprintf("manifest base path mismatch for segment %d: current %s, incoming %s", segment.GetID(), currentBase, incomingBase))
|
||||
return merr.WrapErrServiceInternalMsg("manifest base path mismatch for segment %d: current %s, incoming %s", segment.GetID(), currentBase, incomingBase)
|
||||
}
|
||||
if incomingVersion > currentVersion {
|
||||
segment.ManifestPath = manifestPath
|
||||
@@ -1824,6 +1824,14 @@ func (m *meta) UpdateSegmentsInfo(ctx context.Context, operators ...UpdateOperat
|
||||
|
||||
// Validate the update pack.
|
||||
if err := updatePack.Validate(); err != nil {
|
||||
// A stale save-binlog-paths update (segment already flushed, or an
|
||||
// outdated time tick) is a benign no-op: skip the meta write and
|
||||
// report success so the caller does not retry. The signal stays
|
||||
// inside this package on purpose; see errIgnoredSegmentMetaOperation.
|
||||
if errors.Is(err, errIgnoredSegmentMetaOperation) {
|
||||
log.Ctx(ctx).Info("meta update: ignored stale segment meta operation", zap.Error(err))
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
}
|
||||
updatePack.prepareSegmentMetricUpdates()
|
||||
@@ -2100,7 +2108,7 @@ func (m *meta) AddAllocation(segmentID UniqueID, allocation *Allocation) error {
|
||||
if curSegInfo == nil {
|
||||
// TODO: Error handling.
|
||||
log.Ctx(m.ctx).Error("meta update: add allocation failed - segment not found", zap.Int64("segmentID", segmentID))
|
||||
return errors.New("meta update: add allocation failed - segment not found")
|
||||
return merr.WrapErrSegmentNotFound(segmentID, "meta update: add allocation failed")
|
||||
}
|
||||
// As we use global segment lastExpire to guarantee data correctness after restart
|
||||
// there is no need to persist allocation to meta store, only update allocation in-memory meta.
|
||||
@@ -2287,7 +2295,7 @@ func validateCompactionFallbackStartPosition(compactFromSegInfos []*SegmentInfo,
|
||||
if maxCommitTs == 0 || fallbackStart.GetTimestamp() >= maxCommitTs {
|
||||
return nil
|
||||
}
|
||||
return errors.Errorf(
|
||||
return merr.WrapErrServiceInternalMsg(
|
||||
"compaction fallback start position timestamp %d is earlier than max input commit timestamp %d",
|
||||
fallbackStart.GetTimestamp(),
|
||||
maxCommitTs)
|
||||
@@ -2661,7 +2669,7 @@ func (m *meta) HasSegments(segIDs []UniqueID) (bool, error) {
|
||||
|
||||
for _, segID := range segIDs {
|
||||
if _, ok := m.segments.segments[segID]; !ok {
|
||||
return false, fmt.Errorf("segment is not exist with ID = %d", segID)
|
||||
return false, merr.WrapErrServiceInternalMsg("segment is not exist with ID = %d", segID)
|
||||
}
|
||||
}
|
||||
return true, nil
|
||||
@@ -2747,7 +2755,7 @@ func (m *meta) collectionHasTextFields(collectionID int64) bool {
|
||||
// UpdateChannelCheckpoint updates and saves channel checkpoint.
|
||||
func (m *meta) UpdateChannelCheckpoint(ctx context.Context, vChannel string, pos *msgpb.MsgPosition) error {
|
||||
if pos == nil || pos.GetMsgID() == nil {
|
||||
return fmt.Errorf("channelCP is nil, vChannel=%s", vChannel)
|
||||
return merr.WrapErrServiceInternalMsg("channelCP is nil, vChannel=%s", vChannel)
|
||||
}
|
||||
|
||||
// Clamp checkpoint to not advance beyond min growing segment checkpoint.
|
||||
@@ -3331,7 +3339,7 @@ func (m *meta) completeBumpSchemaVersionCompactionMutation(
|
||||
if currentManifest != resultManifest {
|
||||
manifestCompare, err := packed.CompareManifestPath(resultManifest, currentManifest)
|
||||
if err != nil {
|
||||
return nil, nil, merr.WrapErrIllegalCompactionPlan(fmt.Sprintf("schema bump compaction result manifest is not comparable with current manifest: %v", err))
|
||||
return nil, nil, merr.WrapErrIllegalCompactionPlanMsg("schema bump compaction result manifest is not comparable with current manifest: %v", err)
|
||||
}
|
||||
if manifestCompare <= 0 {
|
||||
return nil, nil, merr.WrapErrIllegalCompactionPlan("schema bump compaction result manifest is not newer than current manifest")
|
||||
@@ -3392,7 +3400,7 @@ func (m *meta) completeBumpSchemaVersionReplacementMutation(
|
||||
) ([]*SegmentInfo, *segMetricMutation, error) {
|
||||
idRange := t.GetPreAllocatedSegmentIDs()
|
||||
if idRange == nil || idRange.GetBegin()+1 != idRange.GetEnd() || resultSegment.GetSegmentID() != idRange.GetBegin() {
|
||||
return nil, nil, merr.WrapErrIllegalCompactionPlan(fmt.Sprintf("schema bump replacement result segment ID %d does not match the pre-allocated segment ID", resultSegment.GetSegmentID()))
|
||||
return nil, nil, merr.WrapErrIllegalCompactionPlanMsg("schema bump replacement result segment ID %d does not match the pre-allocated segment ID", resultSegment.GetSegmentID())
|
||||
}
|
||||
|
||||
dropped := oldSegment.Clone()
|
||||
@@ -3555,7 +3563,7 @@ func (m *meta) GetFileResources(ctx context.Context, resourceIDs ...int64) ([]*i
|
||||
if resource, ok := m.resourceIDMap[id]; ok {
|
||||
resources = append(resources, resource)
|
||||
} else {
|
||||
return nil, errors.Errorf("file resource %d not found", id)
|
||||
return nil, merr.WrapErrServiceInternalMsg("file resource %d not found", id)
|
||||
}
|
||||
}
|
||||
return resources, nil
|
||||
|
||||
@@ -3498,7 +3498,7 @@ func TestUpdateSegmentsInfo(t *testing.T) {
|
||||
UpdateStartPosition([]*datapb.SegmentStartPosition{{SegmentID: 1, StartPosition: &msgpb.MsgPosition{MsgID: []byte{1, 2, 3}}}}),
|
||||
UpdateCheckPointOperator(1, []*datapb.CheckPoint{{SegmentID: 1, NumOfRows: 10, Position: &msgpb.MsgPosition{MsgID: []byte{1, 2, 3}, Timestamp: 99}}}, true),
|
||||
)
|
||||
assert.True(t, errors.Is(err, ErrIgnoredSegmentMetaOperation))
|
||||
assert.NoError(t, err) // stale update is swallowed as a benign no-op; segment must stay unchanged below
|
||||
|
||||
err = meta.UpdateSegmentsInfo(
|
||||
context.TODO(),
|
||||
@@ -3536,7 +3536,7 @@ func TestUpdateSegmentsInfo(t *testing.T) {
|
||||
[]*datapb.FieldBinlog{}),
|
||||
UpdateCheckPointOperator(1, []*datapb.CheckPoint{{SegmentID: 1, NumOfRows: 12, Position: &msgpb.MsgPosition{MsgID: []byte{1, 2, 3}, Timestamp: 101}}}, true),
|
||||
)
|
||||
assert.True(t, errors.Is(err, ErrIgnoredSegmentMetaOperation))
|
||||
assert.NoError(t, err) // stale update is swallowed as a benign no-op; segment must stay unchanged below
|
||||
|
||||
updated = meta.GetHealthySegment(context.TODO(), 1)
|
||||
assert.Equal(t, updated.NumOfRows, int64(20))
|
||||
|
||||
@@ -22,7 +22,16 @@ import (
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/datapb"
|
||||
)
|
||||
|
||||
var ErrIgnoredSegmentMetaOperation = errors.New("ignored segment meta operation")
|
||||
// errIgnoredSegmentMetaOperation is an internal control-flow signal: the
|
||||
// requested segment meta update is stale (segment already flushed, or an
|
||||
// outdated time tick) and is skipped as a benign no-op. It never crosses the
|
||||
// package boundary — UpdateSegmentsInfo swallows it and returns nil.
|
||||
//
|
||||
// It MUST stay a plain errors.New identity sentinel. A merr-typed value would
|
||||
// make errors.Is match by error code (milvusError.Is compares codes only), so
|
||||
// any error sharing the code — e.g. a real etcd write failure wrapped as
|
||||
// ServiceInternal — would be misread as ignorable and acknowledged as success.
|
||||
var errIgnoredSegmentMetaOperation = errors.New("ignored segment meta operation")
|
||||
|
||||
// reviseVChannelInfo will revise the datapb.VchannelInfo for upgrade compatibility from 2.0.2
|
||||
func reviseVChannelInfo(vChannel *datapb.VchannelInfo) {
|
||||
|
||||
@@ -18,10 +18,8 @@ package datacoord
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"sync"
|
||||
|
||||
"github.com/cockroachdb/errors"
|
||||
"github.com/tidwall/gjson"
|
||||
"go.uber.org/zap"
|
||||
"golang.org/x/sync/errgroup"
|
||||
@@ -138,7 +136,7 @@ func (s *Server) getSegmentsJSON(ctx context.Context, req *milvuspb.GetMetricsRe
|
||||
}
|
||||
return string(bs), nil
|
||||
}
|
||||
return "", fmt.Errorf("invalid param value in=[%s], it should be dc or dn", in)
|
||||
return "", merr.WrapErrParameterInvalidMsg("invalid param value in=[%s], it should be dc or dn", in)
|
||||
}
|
||||
|
||||
func (s *Server) getDistJSON(ctx context.Context, req *milvuspb.GetMetricsRequest) string {
|
||||
@@ -308,7 +306,7 @@ func (s *Server) getIndexNodeMetrics(ctx context.Context, req *milvuspb.GetMetri
|
||||
},
|
||||
}
|
||||
if node == nil {
|
||||
return infos, errors.New("IndexNode is nil")
|
||||
return infos, merr.WrapErrServiceInternalMsg("index node is nil")
|
||||
}
|
||||
|
||||
metrics, err := node.GetMetrics(ctx, req)
|
||||
|
||||
@@ -2,7 +2,6 @@ package datacoord
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"sync"
|
||||
|
||||
"go.uber.org/zap"
|
||||
@@ -183,11 +182,11 @@ func (psm *partitionStatsMeta) innerSaveCurrentPartitionStatsVersion(collectionI
|
||||
|
||||
if _, ok := psm.partitionStatsInfos[vChannel]; !ok {
|
||||
return merr.WrapErrClusteringCompactionMetaError("SaveCurrentPartitionStatsVersion",
|
||||
fmt.Errorf("update current partition stats version failed, there is no partition info exists with collID: %d, partID: %d, vChannel: %s", collectionID, partitionID, vChannel))
|
||||
merr.WrapErrServiceInternalMsg("update current partition stats version failed, there is no partition info exists with collID: %d, partID: %d, vChannel: %s", collectionID, partitionID, vChannel))
|
||||
}
|
||||
if _, ok := psm.partitionStatsInfos[vChannel][partitionID]; !ok {
|
||||
return merr.WrapErrClusteringCompactionMetaError("SaveCurrentPartitionStatsVersion",
|
||||
fmt.Errorf("update current partition stats version failed, there is no partition info exists with collID: %d, partID: %d, vChannel: %s", collectionID, partitionID, vChannel))
|
||||
merr.WrapErrServiceInternalMsg("update current partition stats version failed, there is no partition info exists with collID: %d, partID: %d, vChannel: %s", collectionID, partitionID, vChannel))
|
||||
}
|
||||
|
||||
if err := psm.catalog.SaveCurrentPartitionStatsVersion(psm.ctx, collectionID, partitionID, vChannel, currentPartitionStatsVersion); err != nil {
|
||||
|
||||
@@ -23,12 +23,12 @@ import (
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"github.com/cockroachdb/errors"
|
||||
"github.com/samber/lo"
|
||||
|
||||
"github.com/milvus-io/milvus-proto/go-api/v3/commonpb"
|
||||
"github.com/milvus-io/milvus-proto/go-api/v3/schemapb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/datapb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/tsoutil"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/typeutil"
|
||||
@@ -38,7 +38,7 @@ type calUpperLimitPolicy func(schema *schemapb.CollectionSchema) (int, error)
|
||||
|
||||
func calBySchemaPolicy(schema *schemapb.CollectionSchema) (int, error) {
|
||||
if schema == nil {
|
||||
return -1, errors.New("nil schema")
|
||||
return -1, merr.WrapErrServiceInternalMsg("nil schema")
|
||||
}
|
||||
sizePerRecord, err := typeutil.EstimateSizePerRecord(schema)
|
||||
if err != nil {
|
||||
@@ -46,7 +46,7 @@ func calBySchemaPolicy(schema *schemapb.CollectionSchema) (int, error) {
|
||||
}
|
||||
// check zero value, preventing panicking
|
||||
if sizePerRecord == 0 {
|
||||
return -1, errors.New("zero size record schema found")
|
||||
return -1, merr.WrapErrServiceInternalMsg("zero size record schema found")
|
||||
}
|
||||
threshold := Params.DataCoordCfg.SegmentMaxSize.GetAsFloat() * 1024 * 1024
|
||||
return int(threshold / float64(sizePerRecord)), nil
|
||||
|
||||
@@ -23,7 +23,6 @@ import (
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/cockroachdb/errors"
|
||||
"github.com/samber/lo"
|
||||
"go.opentelemetry.io/otel"
|
||||
"go.uber.org/zap"
|
||||
@@ -36,6 +35,7 @@ import (
|
||||
"github.com/milvus-io/milvus/pkg/v3/log"
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/datapb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/lock"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/metautil"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/retry"
|
||||
@@ -299,7 +299,7 @@ func (s *SegmentManager) genLastExpireTsForSegments() (Timestamp, error) {
|
||||
}, retry.Attempts(Params.DataCoordCfg.AllocLatestExpireAttempt.GetAsUint()), retry.Sleep(200*time.Millisecond))
|
||||
if allocateErr != nil {
|
||||
log.Warn("cannot allocate latest lastExpire from rootCoord", zap.Error(allocateErr))
|
||||
return 0, errors.New("global max expire ts is unavailable for segment manager")
|
||||
return 0, merr.WrapErrServiceInternalMsg("global max expire ts is unavailable for segment manager")
|
||||
}
|
||||
return latestTs, nil
|
||||
}
|
||||
@@ -466,7 +466,7 @@ func (s *SegmentManager) estimateMaxNumOfRows(collectionID UniqueID) (int, error
|
||||
// it's ok to use meta.GetCollection here, since collection meta is set before using segmentManager
|
||||
collMeta := s.meta.GetCollection(collectionID)
|
||||
if collMeta == nil {
|
||||
return -1, fmt.Errorf("failed to get collection %d", collectionID)
|
||||
return -1, merr.WrapErrServiceInternalMsg("failed to get collection %d", collectionID)
|
||||
}
|
||||
return s.estimatePolicy(collMeta.Schema)
|
||||
}
|
||||
|
||||
@@ -26,7 +26,6 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/blang/semver/v4"
|
||||
"github.com/cockroachdb/errors"
|
||||
"github.com/samber/lo"
|
||||
"github.com/tidwall/gjson"
|
||||
"github.com/tikv/client-go/v2/txnkv"
|
||||
@@ -462,7 +461,7 @@ func (s *Server) SetSession(session sessionutil.SessionInterface) error {
|
||||
s.session = session
|
||||
s.icSession = session
|
||||
if s.session == nil {
|
||||
return errors.New("session is nil, the etcd client connection may have failed")
|
||||
return merr.WrapErrServiceNotReadyMsg("session is nil, the etcd client connection may have failed")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -618,7 +617,7 @@ func (s *Server) initKV() error {
|
||||
s.kv = etcdkv.NewEtcdKV(s.etcdCli, s.metaRootPath,
|
||||
etcdkv.WithRequestTimeout(paramtable.Get().EtcdCfg.RequestTimeout.GetAsDuration(time.Millisecond)))
|
||||
default:
|
||||
return retry.Unrecoverable(fmt.Errorf("not supported meta store: %s", metaType))
|
||||
return retry.Unrecoverable(merr.WrapErrServiceInternalMsg("unsupported meta store: %s", metaType))
|
||||
}
|
||||
log.Info("data coordinator successfully connected to metadata store", zap.String("metaType", metaType))
|
||||
return nil
|
||||
@@ -1027,7 +1026,7 @@ func (s *Server) flushFlushingSegment(ctx context.Context, segmentID UniqueID) e
|
||||
return ctx.Err()
|
||||
}
|
||||
// underlying etcd may return context canceled, so we need to return a error to retry.
|
||||
return errors.New("flush segment complete failed")
|
||||
return merr.WrapErrServiceInternalMsg("flush segment complete failed")
|
||||
}
|
||||
return nil
|
||||
}, retry.AttemptAlways())
|
||||
@@ -1117,7 +1116,7 @@ func (s *Server) CleanMeta() error {
|
||||
err2 := s.watchClient.RemoveWithPrefix(s.ctx, "")
|
||||
if err2 != nil {
|
||||
if err != nil {
|
||||
err = fmt.Errorf("failed to CleanMeta[metadata cleanup error: %w][watchdata cleanup error: %v]", err, err2)
|
||||
err = merr.Wrapf(err, "failed to clean meta (watchdata cleanup error: %v)", err2)
|
||||
} else {
|
||||
err = err2
|
||||
}
|
||||
|
||||
@@ -137,7 +137,7 @@ func (s *Server) flushCollection(ctx context.Context, collectionID UniqueID, flu
|
||||
for _, channel := range coll.VChannelNames {
|
||||
sealedSegmentIDs, err := s.segmentManager.SealAllSegments(ctx, channel, toFlushSegments)
|
||||
if err != nil {
|
||||
return nil, errors.Wrapf(err, "failed to flush collection %d", collectionID)
|
||||
return nil, merr.Wrapf(err, "failed to flush collection %d", collectionID)
|
||||
}
|
||||
for _, sealedSegmentID := range sealedSegmentIDs {
|
||||
sealedSegmentsIDDict[sealedSegmentID] = true
|
||||
@@ -691,13 +691,12 @@ func (s *Server) SaveBinlogPaths(ctx context.Context, req *datapb.SaveBinlogPath
|
||||
UpdateAsDroppedIfEmptyWhenFlushing(req.GetSegmentID()),
|
||||
)
|
||||
|
||||
// Update segment info in memory and meta.
|
||||
// Update segment info in memory and meta. Stale updates (segment already
|
||||
// flushed / outdated time tick) are swallowed inside UpdateSegmentsInfo as
|
||||
// benign no-ops, so any error here is a real failure.
|
||||
if err := s.meta.UpdateSegmentsInfo(ctx, operators...); err != nil {
|
||||
if !errors.Is(err, ErrIgnoredSegmentMetaOperation) {
|
||||
log.Error("save binlog and checkpoints failed", zap.Error(err))
|
||||
return merr.Status(err), nil
|
||||
}
|
||||
log.Info("save binlog and checkpoints failed with ignorable error", zap.Error(err))
|
||||
log.Error("save binlog and checkpoints failed", zap.Error(err))
|
||||
return merr.Status(err), nil
|
||||
}
|
||||
|
||||
s.meta.SetLastWrittenTime(req.GetSegmentID())
|
||||
@@ -1292,7 +1291,7 @@ func (s *Server) GetMetrics(ctx context.Context, req *milvuspb.GetMetricsRequest
|
||||
msg := "failed to get metrics"
|
||||
log.Warn(msg, zap.Error(err))
|
||||
return &milvuspb.GetMetricsResponse{
|
||||
Status: merr.Status(errors.Wrap(err, msg)),
|
||||
Status: merr.Status(merr.Wrap(err, msg)),
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -1627,7 +1626,7 @@ OUTER:
|
||||
break OUTER
|
||||
}
|
||||
} else {
|
||||
resp.Status = merr.Status(merr.WrapErrParameterInvalidMsg("FlushAllTss or FlushAllTs is required"))
|
||||
resp.Status = merr.Status(merr.WrapErrParameterMissingMsg("FlushAllTss or FlushAllTs is required"))
|
||||
return resp, nil
|
||||
}
|
||||
}
|
||||
@@ -1925,12 +1924,12 @@ func (s *Server) ImportV2(ctx context.Context, in *internalpb.ImportRequestInter
|
||||
jobID := in.GetJobID()
|
||||
if jobID == 0 {
|
||||
if s.allocator == nil {
|
||||
resp.Status = merr.Status(merr.WrapErrImportFailed("allocator not initialized"))
|
||||
resp.Status = merr.Status(merr.WrapErrServiceUnavailable("allocator not initialized"))
|
||||
return resp, nil
|
||||
}
|
||||
jobID, _, err = s.allocator.AllocN(1)
|
||||
if err != nil {
|
||||
resp.Status = merr.Status(merr.WrapErrImportFailed(fmt.Sprintf("failed to allocate job ID: %v", err)))
|
||||
resp.Status = merr.Status(merr.Wrap(err, "failed to allocate job ID"))
|
||||
return resp, nil
|
||||
}
|
||||
}
|
||||
@@ -1950,7 +1949,7 @@ func (s *Server) ImportV2(ctx context.Context, in *internalpb.ImportRequestInter
|
||||
)
|
||||
if err != nil {
|
||||
log.Warn("failed to broadcast import message", zap.Error(err))
|
||||
resp.Status = merr.Status(merr.WrapErrImportFailed(fmt.Sprintf("failed to broadcast import: %v", err)))
|
||||
resp.Status = merr.Status(merr.Wrap(err, "failed to broadcast import"))
|
||||
return resp, nil
|
||||
}
|
||||
|
||||
@@ -2000,7 +1999,7 @@ func (s *Server) createImportJobFromAck(ctx context.Context, in *internalpb.Impo
|
||||
// Allocate file ids.
|
||||
idStart, _, err := s.allocator.AllocN(int64(len(files)) + 1)
|
||||
if err != nil {
|
||||
resp.Status = merr.Status(merr.WrapErrImportFailed(fmt.Sprintf("alloc id failed: %v", err)))
|
||||
resp.Status = merr.Status(merr.Wrap(err, "alloc id failed"))
|
||||
return resp, nil
|
||||
}
|
||||
files = lo.Map(files, func(importFile *internalpb.ImportFile, i int) *internalpb.ImportFile {
|
||||
@@ -2013,7 +2012,7 @@ func (s *Server) createImportJobFromAck(ctx context.Context, in *internalpb.Impo
|
||||
return resp, nil
|
||||
}
|
||||
if err != nil {
|
||||
resp.Status = merr.Status(merr.WrapErrImportFailed(fmt.Sprintf("get collection failed: %v", err)))
|
||||
resp.Status = merr.Status(merr.Wrap(err, "get collection failed"))
|
||||
return resp, nil
|
||||
}
|
||||
if importCollectionInfo == nil {
|
||||
@@ -2048,7 +2047,7 @@ func (s *Server) createImportJobFromAck(ctx context.Context, in *internalpb.Impo
|
||||
}
|
||||
err = s.importMeta.AddJob(ctx, job)
|
||||
if err != nil {
|
||||
resp.Status = merr.Status(merr.WrapErrImportFailed(fmt.Sprintf("add import job failed: %v", err)))
|
||||
resp.Status = merr.Status(merr.Wrap(err, "add import job failed"))
|
||||
return resp, nil
|
||||
}
|
||||
|
||||
@@ -2075,13 +2074,13 @@ func (s *Server) GetImportProgress(ctx context.Context, in *internalpb.GetImport
|
||||
}
|
||||
jobID, err := strconv.ParseInt(in.GetJobID(), 10, 64)
|
||||
if err != nil {
|
||||
resp.Status = merr.Status(merr.WrapErrImportFailed(fmt.Sprintf("parse job id failed: %v", err)))
|
||||
resp.Status = merr.Status(merr.WrapErrParameterInvalidMsg("parse job id failed: %v", err))
|
||||
return resp, nil
|
||||
}
|
||||
|
||||
job := s.importMeta.GetJob(ctx, jobID)
|
||||
if job == nil {
|
||||
resp.Status = merr.Status(merr.WrapErrImportFailed(fmt.Sprintf("import job does not exist, jobID=%d", jobID)))
|
||||
resp.Status = merr.Status(merr.WrapErrImportSysFailedMsg("import job does not exist, jobID=%d", jobID))
|
||||
return resp, nil
|
||||
}
|
||||
progress, state, importedRows, totalRows, reason := GetJobProgress(ctx, jobID, s.importMeta, s.meta)
|
||||
@@ -2481,14 +2480,14 @@ func (s *Server) RestoreSnapshot(ctx context.Context, req *datapb.RestoreSnapsho
|
||||
|
||||
// Validate parameters
|
||||
if req.GetName() == "" {
|
||||
err := merr.WrapErrParameterInvalidMsg("snapshot name is required")
|
||||
err := merr.WrapErrParameterMissingMsg("snapshot name is required")
|
||||
log.Warn("invalid request", zap.Error(err))
|
||||
return &datapb.RestoreSnapshotResponse{
|
||||
Status: merr.Status(err),
|
||||
}, nil
|
||||
}
|
||||
if req.GetTargetCollectionName() == "" {
|
||||
err := merr.WrapErrParameterInvalidMsg("target collection name is required")
|
||||
err := merr.WrapErrParameterMissingMsg("target collection name is required")
|
||||
log.Warn("invalid request", zap.Error(err))
|
||||
return &datapb.RestoreSnapshotResponse{
|
||||
Status: merr.Status(err),
|
||||
@@ -2887,7 +2886,7 @@ func (s *Server) ListRefreshExternalCollectionJobs(ctx context.Context, req *dat
|
||||
func (s *Server) broadcastCommitImportMessage(ctx context.Context, job ImportJob) error {
|
||||
vchannels := job.GetVchannels()
|
||||
if len(vchannels) == 0 {
|
||||
return merr.WrapErrImportFailed(fmt.Sprintf("job %d has no vchannels", job.GetJobID()))
|
||||
return merr.WrapErrImportSysFailedMsg("job %d has no vchannels", job.GetJobID())
|
||||
}
|
||||
|
||||
broadcaster, err := s.startBroadcastWithCollectionID(ctx, job.GetCollectionID())
|
||||
@@ -2914,7 +2913,7 @@ func (s *Server) broadcastCommitImportMessage(ctx context.Context, job ImportJob
|
||||
func (s *Server) broadcastRollbackImportMessage(ctx context.Context, job ImportJob) error {
|
||||
vchannels := job.GetVchannels()
|
||||
if len(vchannels) == 0 {
|
||||
return merr.WrapErrImportFailed(fmt.Sprintf("job %d has no vchannels", job.GetJobID()))
|
||||
return merr.WrapErrImportSysFailedMsg("job %d has no vchannels", job.GetJobID())
|
||||
}
|
||||
|
||||
broadcaster, err := s.startBroadcastWithCollectionID(ctx, job.GetCollectionID())
|
||||
@@ -2949,7 +2948,7 @@ func (s *Server) validateAndExecuteImportAction(
|
||||
}
|
||||
job := s.importMeta.GetJob(ctx, jobID)
|
||||
if job == nil {
|
||||
return merr.Status(merr.WrapErrImportFailed(fmt.Sprintf("job %d not found", jobID))), nil
|
||||
return merr.Status(merr.WrapErrImportSysFailedMsg("job %d not found", jobID)), nil
|
||||
}
|
||||
if st := validateState(job); st != nil {
|
||||
return st, nil
|
||||
|
||||
@@ -186,7 +186,7 @@ const maxBackfillResultBytes int64 = 64 * 1024 * 1024 // 64MiB
|
||||
// reject s3a://<other-bucket>/... paths early.
|
||||
func (s *Server) loadBackfillResult(ctx context.Context, rawPath string) (*BackfillResult, error) {
|
||||
if rawPath == "" {
|
||||
return nil, merr.WrapErrParameterInvalidMsg("result_path is required")
|
||||
return nil, merr.WrapErrParameterMissingMsg("result_path is required")
|
||||
}
|
||||
bucket := bucketFromChunkManager(s.meta.chunkManager)
|
||||
key, err := normalizeObjectKey(rawPath, bucket)
|
||||
|
||||
@@ -1645,7 +1645,6 @@ func TestGetRecoveryInfoV2(t *testing.T) {
|
||||
|
||||
func TestImportV2(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
mockErr := errors.New("mock err")
|
||||
|
||||
t.Run("ImportV2", func(t *testing.T) {
|
||||
// server not healthy
|
||||
@@ -1682,11 +1681,11 @@ func TestImportV2(t *testing.T) {
|
||||
s.importMeta, err = NewImportMeta(context.TODO(), catalog, nil, nil)
|
||||
assert.NoError(t, err)
|
||||
alloc := allocator.NewMockAllocator(t)
|
||||
alloc.EXPECT().AllocN(mock.Anything).Return(0, 0, mockErr)
|
||||
alloc.EXPECT().AllocN(mock.Anything).Return(0, 0, merr.WrapErrServiceUnavailable("mock err"))
|
||||
s.allocator = alloc
|
||||
resp, err = s.ImportV2(ctx, &internalpb.ImportRequestInternal{})
|
||||
assert.NoError(t, err)
|
||||
assert.True(t, errors.Is(merr.Error(resp.GetStatus()), merr.ErrImportFailed))
|
||||
assert.True(t, errors.Is(merr.Error(resp.GetStatus()), merr.ErrServiceUnavailable))
|
||||
})
|
||||
|
||||
t.Run("GetImportProgress", func(t *testing.T) {
|
||||
@@ -1703,7 +1702,7 @@ func TestImportV2(t *testing.T) {
|
||||
JobID: "@%$%$#%",
|
||||
})
|
||||
assert.NoError(t, err)
|
||||
assert.True(t, errors.Is(merr.Error(resp.GetStatus()), merr.ErrImportFailed))
|
||||
assert.True(t, errors.Is(merr.Error(resp.GetStatus()), merr.ErrParameterInvalid))
|
||||
|
||||
// job does not exist
|
||||
catalog := mocks.NewDataCoordCatalog(t)
|
||||
@@ -1723,7 +1722,8 @@ func TestImportV2(t *testing.T) {
|
||||
JobID: "-1",
|
||||
})
|
||||
assert.NoError(t, err)
|
||||
assert.True(t, errors.Is(merr.Error(resp.GetStatus()), merr.ErrImportFailed))
|
||||
// job-not-found is a server-side orchestration issue, not malformed user data
|
||||
assert.True(t, errors.Is(merr.Error(resp.GetStatus()), merr.ErrImportSysFailed))
|
||||
|
||||
// normal case
|
||||
var job ImportJob = &importJob{
|
||||
@@ -3156,7 +3156,7 @@ func TestServer_RestoreSnapshot(t *testing.T) {
|
||||
|
||||
assert.NoError(t, err)
|
||||
assert.Error(t, merr.Error(resp.GetStatus()))
|
||||
assert.True(t, errors.Is(merr.Error(resp.GetStatus()), merr.ErrParameterInvalid))
|
||||
assert.True(t, errors.Is(merr.Error(resp.GetStatus()), merr.ErrParameterMissing))
|
||||
})
|
||||
|
||||
t.Run("missing_collection_name", func(t *testing.T) {
|
||||
@@ -3173,7 +3173,7 @@ func TestServer_RestoreSnapshot(t *testing.T) {
|
||||
|
||||
assert.NoError(t, err)
|
||||
assert.Error(t, merr.Error(resp.GetStatus()))
|
||||
assert.True(t, errors.Is(merr.Error(resp.GetStatus()), merr.ErrParameterInvalid))
|
||||
assert.True(t, errors.Is(merr.Error(resp.GetStatus()), merr.ErrParameterMissing))
|
||||
})
|
||||
|
||||
t.Run("snapshot_not_found", func(t *testing.T) {
|
||||
|
||||
@@ -18,12 +18,12 @@ package session
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"github.com/cockroachdb/errors"
|
||||
|
||||
"github.com/milvus-io/milvus/internal/types"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/lock"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
)
|
||||
|
||||
var errDisposed = errors.New("client is disposed")
|
||||
@@ -84,7 +84,7 @@ func (n *Session) GetOrCreateClient(ctx context.Context) (types.DataNodeClient,
|
||||
}
|
||||
|
||||
if n.clientCreator == nil {
|
||||
return nil, fmt.Errorf("unable to create client for %s because of a nil client creator", n.info.Address)
|
||||
return nil, merr.WrapErrServiceInternalMsg("unable to create client for %s because of a nil client creator", n.info.Address)
|
||||
}
|
||||
|
||||
err := n.initClient(ctx)
|
||||
|
||||
@@ -33,6 +33,7 @@ import (
|
||||
"github.com/milvus-io/milvus/pkg/v3/log"
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/datapb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
)
|
||||
|
||||
// S3 snapshot storage path constants.
|
||||
@@ -136,7 +137,7 @@ func getManifestSchemaByVersion(version int) (avro.Schema, error) {
|
||||
case 3:
|
||||
return getManifestSchemaV3()
|
||||
default:
|
||||
return nil, fmt.Errorf("unsupported manifest schema version: %d", version)
|
||||
return nil, merr.WrapErrServiceInternalMsg("unsupported manifest schema version: %d", version)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -508,22 +509,22 @@ func GetSegmentManifestPath(manifestDir string, segmentID int64) string {
|
||||
func (w *SnapshotWriter) Save(ctx context.Context, snapshot *SnapshotData) (string, error) {
|
||||
// Validate input parameters
|
||||
if snapshot == nil {
|
||||
return "", fmt.Errorf("snapshot cannot be nil")
|
||||
return "", merr.WrapErrServiceInternalMsg("snapshot cannot be nil")
|
||||
}
|
||||
if snapshot.SnapshotInfo == nil {
|
||||
return "", fmt.Errorf("snapshot info cannot be nil")
|
||||
return "", merr.WrapErrServiceInternalMsg("snapshot info cannot be nil")
|
||||
}
|
||||
collectionID := snapshot.SnapshotInfo.GetCollectionId()
|
||||
if collectionID <= 0 {
|
||||
return "", fmt.Errorf("invalid collection ID: %d", collectionID)
|
||||
return "", merr.WrapErrServiceInternalMsg("invalid collection ID: %d", collectionID)
|
||||
}
|
||||
if snapshot.Collection == nil {
|
||||
return "", fmt.Errorf("collection description cannot be nil")
|
||||
return "", merr.WrapErrServiceInternalMsg("collection description cannot be nil")
|
||||
}
|
||||
|
||||
snapshotID := snapshot.SnapshotInfo.GetId()
|
||||
if snapshotID <= 0 {
|
||||
return "", fmt.Errorf("invalid snapshot ID: %d", snapshotID)
|
||||
return "", merr.WrapErrServiceInternalMsg("invalid snapshot ID: %d", snapshotID)
|
||||
}
|
||||
manifestDir, metadataPath := GetSnapshotPaths(w.chunkManager.RootPath(), collectionID, snapshotID)
|
||||
|
||||
@@ -533,7 +534,7 @@ func (w *SnapshotWriter) Save(ctx context.Context, snapshot *SnapshotData) (stri
|
||||
for _, segment := range snapshot.Segments {
|
||||
manifestPath := GetSegmentManifestPath(manifestDir, segment.GetSegmentId())
|
||||
if err := w.writeSegmentManifest(ctx, manifestPath, segment); err != nil {
|
||||
return "", fmt.Errorf("failed to write manifest for segment %d: %w", segment.GetSegmentId(), err)
|
||||
return "", merr.Wrapf(err, "failed to write manifest for segment %d", segment.GetSegmentId())
|
||||
}
|
||||
manifestPaths = append(manifestPaths, manifestPath)
|
||||
}
|
||||
@@ -557,7 +558,7 @@ func (w *SnapshotWriter) Save(ctx context.Context, snapshot *SnapshotData) (stri
|
||||
// Step 3: Write metadata file with all manifest paths
|
||||
// This file is the entry point for reading the snapshot
|
||||
if err := w.writeMetadataFile(ctx, metadataPath, snapshot, manifestPaths, storagev2Manifests); err != nil {
|
||||
return "", fmt.Errorf("failed to write metadata file: %w", err)
|
||||
return "", merr.Wrap(err, "failed to write metadata file")
|
||||
}
|
||||
|
||||
log.Info("Successfully wrote metadata file",
|
||||
@@ -584,12 +585,12 @@ func (w *SnapshotWriter) writeSegmentManifest(ctx context.Context, manifestPath
|
||||
// Serialize to Avro binary format (single record per file)
|
||||
avroSchema, err := getManifestSchema()
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to get manifest schema: %w", err)
|
||||
return merr.WrapErrServiceInternalErr(err, "failed to get manifest schema")
|
||||
}
|
||||
|
||||
binaryData, err := avro.Marshal(avroSchema, entry)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to serialize entry to avro: %w", err)
|
||||
return merr.WrapErrServiceInternalErr(err, "failed to serialize entry to avro")
|
||||
}
|
||||
|
||||
// Write to object storage
|
||||
@@ -634,7 +635,7 @@ func (w *SnapshotWriter) writeMetadataFile(ctx context.Context, metadataPath str
|
||||
}
|
||||
jsonData, err := opts.Marshal(metadata)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to marshal metadata to JSON: %w", err)
|
||||
return merr.WrapErrServiceInternalErr(err, "failed to marshal metadata to JSON")
|
||||
}
|
||||
|
||||
// Write to object storage
|
||||
@@ -663,13 +664,13 @@ func (w *SnapshotWriter) writeMetadataFile(ctx context.Context, metadataPath str
|
||||
func (w *SnapshotWriter) Drop(ctx context.Context, metadataFilePath string) error {
|
||||
// Validate input parameter
|
||||
if metadataFilePath == "" {
|
||||
return fmt.Errorf("metadata file path cannot be empty")
|
||||
return merr.WrapErrServiceInternalMsg("metadata file path cannot be empty")
|
||||
}
|
||||
|
||||
// Step 1: Read metadata file to discover manifest file paths
|
||||
metadata, err := w.readMetadataFile(ctx, metadataFilePath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to read metadata file: %w", err)
|
||||
return merr.Wrap(err, "failed to read metadata file")
|
||||
}
|
||||
|
||||
snapshotID := metadata.GetSnapshotInfo().GetId()
|
||||
@@ -678,7 +679,7 @@ func (w *SnapshotWriter) Drop(ctx context.Context, metadataFilePath string) erro
|
||||
manifestList := metadata.GetManifestList()
|
||||
if len(manifestList) > 0 {
|
||||
if err := w.chunkManager.MultiRemove(ctx, manifestList); err != nil {
|
||||
return fmt.Errorf("failed to remove manifest files: %w", err)
|
||||
return merr.Wrap(err, "failed to remove manifest files")
|
||||
}
|
||||
log.Info("Successfully removed manifest files",
|
||||
zap.Int("count", len(manifestList)),
|
||||
@@ -687,7 +688,7 @@ func (w *SnapshotWriter) Drop(ctx context.Context, metadataFilePath string) erro
|
||||
|
||||
// Step 3: Remove the metadata file (entry point)
|
||||
if err := w.chunkManager.Remove(ctx, metadataFilePath); err != nil {
|
||||
return fmt.Errorf("failed to remove metadata file: %w", err)
|
||||
return merr.Wrap(err, "failed to remove metadata file")
|
||||
}
|
||||
log.Info("Successfully removed metadata file",
|
||||
zap.String("metadataFilePath", metadataFilePath))
|
||||
@@ -703,7 +704,7 @@ func (w *SnapshotWriter) readMetadataFile(ctx context.Context, filePath string)
|
||||
// Read raw JSON content from object storage
|
||||
data, err := w.chunkManager.Read(ctx, filePath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to read metadata file: %w", err)
|
||||
return nil, merr.Wrap(err, "failed to read metadata file")
|
||||
}
|
||||
|
||||
// Use protojson for deserialization to correctly handle protobuf oneof fields
|
||||
@@ -712,7 +713,7 @@ func (w *SnapshotWriter) readMetadataFile(ctx context.Context, filePath string)
|
||||
DiscardUnknown: true,
|
||||
}
|
||||
if err := opts.Unmarshal(data, metadata); err != nil {
|
||||
return nil, fmt.Errorf("failed to parse metadata JSON: %w", err)
|
||||
return nil, merr.WrapErrServiceInternalErr(err, "failed to parse metadata JSON")
|
||||
}
|
||||
|
||||
return metadata, nil
|
||||
@@ -775,13 +776,13 @@ func NewSnapshotReader(cm storage.ChunkManager) *SnapshotReader {
|
||||
func (r *SnapshotReader) ReadSnapshot(ctx context.Context, metadataFilePath string, includeSegments bool) (*SnapshotData, error) {
|
||||
// Validate input
|
||||
if metadataFilePath == "" {
|
||||
return nil, fmt.Errorf("metadata file path cannot be empty")
|
||||
return nil, merr.WrapErrServiceInternalMsg("metadata file path cannot be empty")
|
||||
}
|
||||
|
||||
// Step 1: Read metadata file (always required)
|
||||
metadata, err := r.readMetadataFile(ctx, metadataFilePath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to read metadata file: %w", err)
|
||||
return nil, merr.Wrap(err, "failed to read metadata file")
|
||||
}
|
||||
|
||||
// Step 2: Optionally read segment details from manifest files
|
||||
@@ -791,7 +792,7 @@ func (r *SnapshotReader) ReadSnapshot(ctx context.Context, metadataFilePath stri
|
||||
for _, manifestPath := range metadata.GetManifestList() {
|
||||
segments, err := r.readManifestFile(ctx, manifestPath, int(metadata.GetFormatVersion()))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to read manifest file %s: %w", manifestPath, err)
|
||||
return nil, merr.Wrapf(err, "failed to read manifest file %s", manifestPath)
|
||||
}
|
||||
allSegments = append(allSegments, segments...)
|
||||
}
|
||||
@@ -830,7 +831,7 @@ func (r *SnapshotReader) readMetadataFile(ctx context.Context, filePath string)
|
||||
// Read raw JSON content from object storage
|
||||
data, err := r.chunkManager.Read(ctx, filePath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to read metadata file: %w", err)
|
||||
return nil, merr.Wrap(err, "failed to read metadata file")
|
||||
}
|
||||
|
||||
// Use protojson for deserialization to correctly handle protobuf oneof fields
|
||||
@@ -839,12 +840,12 @@ func (r *SnapshotReader) readMetadataFile(ctx context.Context, filePath string)
|
||||
DiscardUnknown: true,
|
||||
}
|
||||
if err := opts.Unmarshal(data, metadata); err != nil {
|
||||
return nil, fmt.Errorf("failed to parse metadata JSON: %w", err)
|
||||
return nil, merr.WrapErrServiceInternalErr(err, "failed to parse metadata JSON")
|
||||
}
|
||||
|
||||
// Validate format version compatibility
|
||||
if err := validateFormatVersion(int(metadata.GetFormatVersion())); err != nil {
|
||||
return nil, fmt.Errorf("incompatible snapshot format: %w", err)
|
||||
return nil, merr.WrapErrServiceInternalErr(err, "incompatible snapshot format")
|
||||
}
|
||||
|
||||
return metadata, nil
|
||||
@@ -865,7 +866,7 @@ func validateFormatVersion(version int) error {
|
||||
|
||||
// Check if version is too new for current code
|
||||
if version > SnapshotFormatVersion {
|
||||
return fmt.Errorf("snapshot format version %d is too new, current supported version: %d (please upgrade Milvus)",
|
||||
return merr.WrapErrServiceInternalMsg("snapshot format version %d is too new, current supported version: %d (please upgrade Milvus)",
|
||||
version, SnapshotFormatVersion)
|
||||
}
|
||||
|
||||
@@ -888,20 +889,20 @@ func (r *SnapshotReader) readManifestFile(ctx context.Context, filePath string,
|
||||
// Read raw Avro content from object storage
|
||||
data, err := r.chunkManager.Read(ctx, filePath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to read manifest file: %w", err)
|
||||
return nil, merr.Wrap(err, "failed to read manifest file")
|
||||
}
|
||||
|
||||
// Get Avro schema for the specified format version
|
||||
avroSchema, err := getManifestSchemaByVersion(formatVersion)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to get manifest schema for version %d: %w", formatVersion, err)
|
||||
return nil, merr.WrapErrServiceInternalErr(err, "failed to get manifest schema for version %d", formatVersion)
|
||||
}
|
||||
|
||||
// Deserialize Avro binary data to single ManifestEntry record
|
||||
var record ManifestEntry
|
||||
err = avro.Unmarshal(avroSchema, data, &record)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to parse avro data: %w", err)
|
||||
return nil, merr.WrapErrServiceInternalErr(err, "failed to parse avro data")
|
||||
}
|
||||
|
||||
// Convert ManifestEntry record to protobuf SegmentDescription
|
||||
@@ -974,7 +975,7 @@ func (r *SnapshotReader) readManifestFile(ctx context.Context, filePath string,
|
||||
func (r *SnapshotReader) ListSnapshots(ctx context.Context, collectionID int64) ([]*datapb.SnapshotInfo, error) {
|
||||
// Validate input parameters
|
||||
if collectionID <= 0 {
|
||||
return nil, fmt.Errorf("invalid collection ID: %d", collectionID)
|
||||
return nil, merr.WrapErrServiceInternalMsg("invalid collection ID: %d", collectionID)
|
||||
}
|
||||
|
||||
// Construct metadata directory path
|
||||
@@ -984,7 +985,7 @@ func (r *SnapshotReader) ListSnapshots(ctx context.Context, collectionID int64)
|
||||
// List all files in the metadata directory
|
||||
files, _, err := storage.ListAllChunkWithPrefix(ctx, r.chunkManager, metadataDir, false)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to list metadata files: %w", err)
|
||||
return nil, merr.Wrap(storage.ToMilvusIoError(metadataDir, err), "failed to list metadata files")
|
||||
}
|
||||
|
||||
// Read each metadata file and extract SnapshotInfo
|
||||
|
||||
@@ -562,7 +562,7 @@ func (sm *snapshotManager) validateCMEKCompatibility(
|
||||
// Get target database properties first (needed for both encrypted and non-encrypted snapshots)
|
||||
dbResp, err := sm.broker.DescribeDatabase(ctx, targetDbName)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to describe target database %s: %w", targetDbName, err)
|
||||
return merr.Wrapf(err, "failed to describe target database %s", targetDbName)
|
||||
}
|
||||
targetIsEncrypted := hookutil.IsDBEncrypted(dbResp.GetProperties())
|
||||
|
||||
@@ -640,7 +640,7 @@ func (sm *snapshotManager) RestoreSnapshot(
|
||||
// ========================================================================
|
||||
phase0Lock, err := startRestoreLock(ctx, sourceCollectionID, snapshotName, targetDbName, targetCollectionName)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("failed to acquire restore lock: %w", err)
|
||||
return 0, merr.Wrap(err, "failed to acquire restore lock")
|
||||
}
|
||||
|
||||
// Pin the source snapshot while holding the phase-0 lock. The pin is the
|
||||
@@ -662,7 +662,7 @@ func (sm *snapshotManager) RestoreSnapshot(
|
||||
pinID, activePins, err := sm.snapshotMeta.PinSnapshot(ctx, sourceCollectionID, snapshotName, pinTTLSeconds)
|
||||
if err != nil {
|
||||
phase0Lock.Close()
|
||||
return 0, fmt.Errorf("failed to pin source snapshot under restore lock: %w", err)
|
||||
return 0, merr.Wrap(err, "failed to pin source snapshot under restore lock")
|
||||
}
|
||||
setSnapshotActivePinsGauge(sourceCollectionID, snapshotName, activePins)
|
||||
phase0Lock.Close()
|
||||
@@ -691,7 +691,7 @@ func (sm *snapshotManager) RestoreSnapshot(
|
||||
// Phase 1: Read snapshot data (now protected by the refcount guard)
|
||||
snapshotData, err := sm.ReadSnapshotData(ctx, sourceCollectionID, snapshotName)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("failed to read snapshot data: %w", err)
|
||||
return 0, merr.Wrap(err, "failed to read snapshot data")
|
||||
}
|
||||
log.Info("snapshot data loaded",
|
||||
zap.Int("segmentCount", len(snapshotData.Segments)),
|
||||
@@ -707,7 +707,7 @@ func (sm *snapshotManager) RestoreSnapshot(
|
||||
// Phase 2: Restore collection and partitions
|
||||
collectionID, err := sm.RestoreCollection(ctx, snapshotData, targetCollectionName, targetDbName)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("failed to restore collection: %w", err)
|
||||
return 0, merr.Wrap(err, "failed to restore collection")
|
||||
}
|
||||
log.Info("collection and partitions restored", zap.Int64("collectionID", collectionID))
|
||||
|
||||
@@ -718,7 +718,7 @@ func (sm *snapshotManager) RestoreSnapshot(
|
||||
if rollbackErr := rollback(ctx, targetDbName, targetCollectionName); rollbackErr != nil {
|
||||
log.Error("rollback failed", zap.Error(rollbackErr))
|
||||
}
|
||||
return 0, fmt.Errorf("failed to restore indexes: %w", err)
|
||||
return 0, merr.Wrap(err, "failed to restore indexes")
|
||||
}
|
||||
log.Info("indexes restored", zap.Int("indexCount", len(snapshotData.Indexes)))
|
||||
|
||||
@@ -730,7 +730,7 @@ func (sm *snapshotManager) RestoreSnapshot(
|
||||
if rollbackErr := rollback(ctx, targetDbName, targetCollectionName); rollbackErr != nil {
|
||||
log.Error("rollback failed", zap.Error(rollbackErr))
|
||||
}
|
||||
return 0, fmt.Errorf("failed to allocate job ID: %w", err)
|
||||
return 0, merr.Wrap(err, "failed to allocate job ID")
|
||||
}
|
||||
log.Info("pre-allocated job ID for restore", zap.Int64("jobID", jobID))
|
||||
|
||||
@@ -741,7 +741,7 @@ func (sm *snapshotManager) RestoreSnapshot(
|
||||
if rollbackErr := rollback(ctx, targetDbName, targetCollectionName); rollbackErr != nil {
|
||||
log.Error("rollback failed", zap.Error(rollbackErr))
|
||||
}
|
||||
return 0, fmt.Errorf("failed to start broadcaster for restore message: %w", err)
|
||||
return 0, merr.Wrap(err, "failed to start broadcaster for restore message")
|
||||
}
|
||||
defer func() {
|
||||
if restoreBroadcaster != nil {
|
||||
@@ -760,7 +760,7 @@ func (sm *snapshotManager) RestoreSnapshot(
|
||||
if rollbackErr := rollback(ctx, targetDbName, targetCollectionName); rollbackErr != nil {
|
||||
log.Error("rollback failed", zap.Error(rollbackErr))
|
||||
}
|
||||
err = fmt.Errorf("resource validation failed: %w", valErr)
|
||||
err = merr.Wrap(valErr, "resource validation failed")
|
||||
return 0, err
|
||||
}
|
||||
|
||||
@@ -785,7 +785,7 @@ func (sm *snapshotManager) RestoreSnapshot(
|
||||
if rollbackErr := rollback(ctx, targetDbName, targetCollectionName); rollbackErr != nil {
|
||||
log.Error("rollback failed", zap.Error(rollbackErr))
|
||||
}
|
||||
err = fmt.Errorf("failed to broadcast restore message: %w", bcErr)
|
||||
err = merr.Wrap(bcErr, "failed to broadcast restore message")
|
||||
return 0, err
|
||||
}
|
||||
|
||||
@@ -872,14 +872,14 @@ func (sm *snapshotManager) RestoreIndexes(
|
||||
// Get collection info for dbId
|
||||
coll, err := sm.broker.DescribeCollectionInternal(ctx, collectionID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to describe collection %d: %w", collectionID, err)
|
||||
return merr.Wrapf(err, "failed to describe collection %d", collectionID)
|
||||
}
|
||||
|
||||
for _, indexInfo := range snapshotData.Indexes {
|
||||
// Allocate new index ID
|
||||
indexID, err := sm.allocator.AllocID(ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to allocate index ID: %w", err)
|
||||
return merr.Wrap(err, "failed to allocate index ID")
|
||||
}
|
||||
|
||||
// Build index model from snapshot data
|
||||
@@ -898,19 +898,19 @@ func (sm *snapshotManager) RestoreIndexes(
|
||||
|
||||
// Validate the index params (basic validation without JSON path parsing)
|
||||
if err := ValidateIndexParams(index); err != nil {
|
||||
return fmt.Errorf("failed to validate index %s: %w", indexInfo.GetIndexName(), err)
|
||||
return merr.Wrapf(err, "failed to validate index %s", indexInfo.GetIndexName())
|
||||
}
|
||||
|
||||
// Check scalar index engine version for JSON path indexes with new types
|
||||
if err := sm.checkJSONPathIndexVersion(index); err != nil {
|
||||
return fmt.Errorf("failed to validate index %s: %w", indexInfo.GetIndexName(), err)
|
||||
return merr.Wrapf(err, "failed to validate index %s", indexInfo.GetIndexName())
|
||||
}
|
||||
|
||||
// Create a new broadcaster for each index
|
||||
// (each broadcaster can only be used once due to resource key lock consumption)
|
||||
b, err := startBroadcaster(ctx, collectionID, snapshotName)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to start broadcaster for index %s: %w", indexInfo.GetIndexName(), err)
|
||||
return merr.Wrapf(err, "failed to start broadcaster for index %s", indexInfo.GetIndexName())
|
||||
}
|
||||
|
||||
// Broadcast CreateIndex message directly to DDL WAL
|
||||
@@ -930,7 +930,7 @@ func (sm *snapshotManager) RestoreIndexes(
|
||||
)
|
||||
b.Close()
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to broadcast create index %s: %w", indexInfo.GetIndexName(), err)
|
||||
return merr.Wrapf(err, "failed to broadcast create index %s", indexInfo.GetIndexName())
|
||||
}
|
||||
|
||||
log.Ctx(ctx).Info("index restored via DDL WAL broadcast",
|
||||
@@ -977,14 +977,14 @@ func (sm *snapshotManager) RestoreData(
|
||||
snapshotData, err := sm.ReadSnapshotData(ctx, sourceCollectionID, snapshotName)
|
||||
if err != nil {
|
||||
log.Error("failed to read snapshot data", zap.Error(err))
|
||||
return 0, fmt.Errorf("failed to read snapshot data: %w", err)
|
||||
return 0, merr.Wrap(err, "failed to read snapshot data")
|
||||
}
|
||||
|
||||
// ========== Phase 2: Build partition mapping ==========
|
||||
partitionMapping, err := sm.buildPartitionMapping(ctx, snapshotData, collectionID)
|
||||
if err != nil {
|
||||
log.Error("failed to build partition mapping", zap.Error(err))
|
||||
return 0, fmt.Errorf("partition mapping failed: %w", err)
|
||||
return 0, merr.Wrap(err, "partition mapping failed")
|
||||
}
|
||||
log.Info("partition mapping built", zap.Any("partitionMapping", partitionMapping))
|
||||
|
||||
@@ -992,14 +992,14 @@ func (sm *snapshotManager) RestoreData(
|
||||
channelMapping, err := sm.buildChannelMapping(ctx, snapshotData, collectionID)
|
||||
if err != nil {
|
||||
log.Error("failed to build channel mapping", zap.Error(err))
|
||||
return 0, fmt.Errorf("channel mapping failed: %w", err)
|
||||
return 0, merr.Wrap(err, "channel mapping failed")
|
||||
}
|
||||
|
||||
// ========== Phase 4: Create copy segment job ==========
|
||||
// Use the pre-allocated jobID from the WAL message
|
||||
if err := sm.createRestoreJob(ctx, collectionID, channelMapping, partitionMapping, snapshotData, jobID, pinID); err != nil {
|
||||
log.Error("failed to create restore job", zap.Error(err))
|
||||
return 0, fmt.Errorf("restore job creation failed: %w", err)
|
||||
return 0, merr.Wrap(err, "restore job creation failed")
|
||||
}
|
||||
|
||||
log.Info("restore data completed successfully",
|
||||
@@ -1047,7 +1047,7 @@ func (sm *snapshotManager) restoreUserPartitions(
|
||||
}
|
||||
|
||||
if err := sm.broker.CreatePartition(ctx, req); err != nil {
|
||||
return fmt.Errorf("failed to create partition %s: %w", partitionName, err)
|
||||
return merr.Wrapf(err, "failed to create partition %s", partitionName)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1330,7 +1330,7 @@ func (sm *snapshotManager) GetRestoreState(ctx context.Context, jobID int64) (*d
|
||||
// Get job
|
||||
job := sm.copySegmentMeta.GetJob(ctx, jobID)
|
||||
if job == nil {
|
||||
err := merr.WrapErrImportFailed(fmt.Sprintf("restore job not found: jobID=%d", jobID))
|
||||
err := merr.WrapErrImportSysFailedMsg("restore job not found: jobID=%d", jobID)
|
||||
log.Warn("restore job not found")
|
||||
return nil, err
|
||||
}
|
||||
|
||||
@@ -572,7 +572,7 @@ func (sm *snapshotMeta) SaveSnapshot(ctx context.Context, snapshot *SnapshotData
|
||||
|
||||
if err := sm.catalog.SaveSnapshot(ctx, snapshot.SnapshotInfo); err != nil {
|
||||
log.Error("failed to save pending snapshot to catalog", zap.Error(err))
|
||||
return fmt.Errorf("failed to save pending snapshot to catalog: %w", err)
|
||||
return merr.Wrap(err, "failed to save pending snapshot to catalog")
|
||||
}
|
||||
log.Info("saved pending snapshot to catalog")
|
||||
|
||||
@@ -582,7 +582,7 @@ func (sm *snapshotMeta) SaveSnapshot(ctx context.Context, snapshot *SnapshotData
|
||||
// S3 write failed, leave PENDING record for GC to clean up
|
||||
log.Error("failed to save snapshot to S3, pending record left for GC cleanup",
|
||||
zap.Error(err))
|
||||
return fmt.Errorf("failed to save snapshot to S3: %w", err)
|
||||
return merr.Wrap(err, "failed to save snapshot to S3")
|
||||
}
|
||||
snapshot.SnapshotInfo.S3Location = metadataFilePath
|
||||
log.Info("saved snapshot data to S3", zap.String("s3Location", metadataFilePath))
|
||||
@@ -598,7 +598,7 @@ func (sm *snapshotMeta) SaveSnapshot(ctx context.Context, snapshot *SnapshotData
|
||||
// GC will eventually cleanup via the PENDING marker.
|
||||
log.Error("failed to update snapshot to committed state, pending record left for GC cleanup",
|
||||
zap.Error(err))
|
||||
return fmt.Errorf("failed to update snapshot to committed state: %w", err)
|
||||
return merr.Wrap(err, "failed to update snapshot to committed state")
|
||||
}
|
||||
|
||||
// Catalog committed — safe to insert into in-memory cache and register protection.
|
||||
@@ -809,8 +809,7 @@ func (sm *snapshotMeta) DropSnapshotsByCollection(ctx context.Context, collectio
|
||||
}
|
||||
|
||||
if len(errs) > 0 {
|
||||
return dropped, fmt.Errorf("failed to drop %d/%d snapshots for collection %d: %w",
|
||||
len(errs), len(snapshotNames), collectionID, merr.Combine(errs...))
|
||||
return dropped, merr.WrapErrServiceInternalErr(merr.Combine(errs...), "failed to drop %d/%d snapshots for collection %d", len(errs), len(snapshotNames), collectionID)
|
||||
}
|
||||
|
||||
return dropped, nil
|
||||
@@ -1006,7 +1005,7 @@ func (sm *snapshotMeta) GetPendingSnapshots(ctx context.Context, pendingTimeout
|
||||
// Get all snapshots from catalog (etcd)
|
||||
snapshots, err := sm.catalog.ListSnapshots(ctx)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to list snapshots from catalog: %w", err)
|
||||
return nil, merr.Wrap(err, "failed to list snapshots from catalog")
|
||||
}
|
||||
|
||||
now := time.Now().UnixMilli()
|
||||
@@ -1057,7 +1056,7 @@ func (sm *snapshotMeta) CleanupPendingSnapshot(ctx context.Context, snapshot *da
|
||||
func (sm *snapshotMeta) GetDeletingSnapshots(ctx context.Context) ([]*datapb.SnapshotInfo, error) {
|
||||
snapshots, err := sm.catalog.ListSnapshots(ctx)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to list snapshots from catalog: %w", err)
|
||||
return nil, merr.Wrap(err, "failed to list snapshots from catalog")
|
||||
}
|
||||
|
||||
deletingSnapshots := make([]*datapb.SnapshotInfo, 0)
|
||||
@@ -1774,7 +1773,7 @@ func (sm *snapshotMeta) generatePinID(snapshot *datapb.SnapshotInfo) (int64, err
|
||||
for i := 0; i < 3; i++ {
|
||||
var b [8]byte
|
||||
if _, err := cryptorand.Read(b[:]); err != nil {
|
||||
return 0, fmt.Errorf("failed to generate random pin ID: %w", err)
|
||||
return 0, merr.WrapErrServiceInternalErr(err, "failed to generate random pin ID")
|
||||
}
|
||||
id := int64(binary.BigEndian.Uint64(b[:]) >> 1)
|
||||
if id == 0 {
|
||||
@@ -1784,5 +1783,5 @@ func (sm *snapshotMeta) generatePinID(snapshot *datapb.SnapshotInfo) (int64, err
|
||||
return id, nil
|
||||
}
|
||||
}
|
||||
return 0, fmt.Errorf("failed to generate unique pin ID after 3 attempts")
|
||||
return 0, merr.WrapErrServiceInternalMsg("failed to generate unique pin ID after 3 attempts")
|
||||
}
|
||||
|
||||
@@ -182,7 +182,7 @@ func (stm *statsTaskMeta) UpdateVersion(taskID, nodeID int64) error {
|
||||
|
||||
t, ok := stm.tasks.Get(taskID)
|
||||
if !ok {
|
||||
return fmt.Errorf("task %d not found", taskID)
|
||||
return merr.WrapErrServiceInternalMsg("task %d not found", taskID)
|
||||
}
|
||||
|
||||
cloneT := proto.Clone(t).(*indexpb.StatsTask)
|
||||
@@ -212,7 +212,7 @@ func (stm *statsTaskMeta) UpdateTaskState(taskID int64, state indexpb.JobState,
|
||||
|
||||
t, ok := stm.tasks.Get(taskID)
|
||||
if !ok {
|
||||
return fmt.Errorf("task %d not found", taskID)
|
||||
return merr.WrapErrServiceInternalMsg("task %d not found", taskID)
|
||||
}
|
||||
|
||||
cloneT := proto.Clone(t).(*indexpb.StatsTask)
|
||||
@@ -239,7 +239,7 @@ func (stm *statsTaskMeta) UpdateBuildingTask(taskID int64) error {
|
||||
|
||||
t, ok := stm.tasks.Get(taskID)
|
||||
if !ok {
|
||||
return fmt.Errorf("task %d not found", taskID)
|
||||
return merr.WrapErrServiceInternalMsg("task %d not found", taskID)
|
||||
}
|
||||
|
||||
cloneT := proto.Clone(t).(*indexpb.StatsTask)
|
||||
@@ -267,7 +267,7 @@ func (stm *statsTaskMeta) FinishTask(taskID int64, result *workerpb.StatsResult)
|
||||
|
||||
t, ok := stm.tasks.Get(taskID)
|
||||
if !ok {
|
||||
return fmt.Errorf("task %d not found", taskID)
|
||||
return merr.WrapErrServiceInternalMsg("task %d not found", taskID)
|
||||
}
|
||||
|
||||
cloneT := proto.Clone(t).(*indexpb.StatsTask)
|
||||
@@ -358,7 +358,7 @@ func (stm *statsTaskMeta) MarkTaskCanRecycle(taskID int64) error {
|
||||
|
||||
t, ok := stm.tasks.Get(taskID)
|
||||
if !ok {
|
||||
return fmt.Errorf("task %d not found", taskID)
|
||||
return merr.WrapErrServiceInternalMsg("task %d not found", taskID)
|
||||
}
|
||||
|
||||
cloneT := proto.Clone(t).(*indexpb.StatsTask)
|
||||
|
||||
@@ -18,7 +18,6 @@ package datacoord
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"path"
|
||||
"time"
|
||||
|
||||
@@ -241,7 +240,7 @@ func (it *indexBuildTask) prepareJobRequest(ctx context.Context, segment *Segmen
|
||||
var err error
|
||||
params, err = Params.KnowhereConfig.UpdateIndexParams(GetIndexType(params), paramtable.BuildStage, params)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to update index build params: %w", err)
|
||||
return nil, merr.WrapErrServiceInternalErr(err, "failed to update index build params")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -249,14 +248,14 @@ func (it *indexBuildTask) prepareJobRequest(ctx context.Context, segment *Segmen
|
||||
var err error
|
||||
params, err = indexparams.UpdateDiskIndexBuildParams(Params, params)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to append index build params: %w", err)
|
||||
return nil, merr.WrapErrServiceInternalErr(err, "failed to append index build params")
|
||||
}
|
||||
}
|
||||
|
||||
// Get collection info and field
|
||||
collectionInfo, err := it.handler.GetCollection(ctx, segment.GetCollectionID())
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to get collection info: %w", err)
|
||||
return nil, merr.Wrap(err, "failed to get collection info")
|
||||
}
|
||||
|
||||
schema := collectionInfo.Schema
|
||||
@@ -271,7 +270,7 @@ func (it *indexBuildTask) prepareJobRequest(ctx context.Context, segment *Segmen
|
||||
}
|
||||
|
||||
if field == nil {
|
||||
return nil, fmt.Errorf("field not found with ID %d", fieldID)
|
||||
return nil, merr.WrapErrFieldNotFound(fieldID)
|
||||
}
|
||||
|
||||
// Extract dim only for vector types to avoid unnecessary warnings
|
||||
|
||||
@@ -34,6 +34,7 @@ import (
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/datapb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/taskcommon"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
|
||||
)
|
||||
|
||||
@@ -115,7 +116,7 @@ func (t *refreshExternalCollectionTask) validateSource() error {
|
||||
// Validate against job-level snapshot to isolate in-flight tasks from schema changes.
|
||||
job := t.refreshMeta.GetJob(t.GetJobId())
|
||||
if job == nil {
|
||||
return fmt.Errorf("job %d not found", t.GetJobId())
|
||||
return merr.WrapErrServiceInternalMsg("job %d not found", t.GetJobId())
|
||||
}
|
||||
|
||||
currentSource := job.GetExternalSource()
|
||||
@@ -125,7 +126,7 @@ func (t *refreshExternalCollectionTask) validateSource() error {
|
||||
taskSpec := t.GetExternalSpec()
|
||||
|
||||
if currentSource != taskSource || currentSpec != taskSpec {
|
||||
return fmt.Errorf(
|
||||
return merr.WrapErrServiceInternalMsg(
|
||||
"task source mismatch: task source=%s/%s, job source=%s/%s (task belongs to a different refresh job)",
|
||||
taskSource, taskSpec, currentSource, currentSpec,
|
||||
)
|
||||
@@ -215,7 +216,7 @@ func applyExternalCollectionSegmentUpdate(
|
||||
logFields ...zap.Field,
|
||||
) error {
|
||||
if mt == nil {
|
||||
return fmt.Errorf("meta is nil, cannot update segments")
|
||||
return merr.WrapErrServiceInternalMsg("meta is nil, cannot update segments")
|
||||
}
|
||||
fields := append(logFields, zap.Int64("collectionID", collectionID))
|
||||
log := log.Ctx(ctx).With(fields...)
|
||||
@@ -228,14 +229,14 @@ func applyExternalCollectionSegmentUpdate(
|
||||
for _, segID := range keptSegmentIDs {
|
||||
segment := mt.segments.GetSegment(segID)
|
||||
if segment == nil {
|
||||
return fmt.Errorf("kept segment %d not found", segID)
|
||||
return merr.WrapErrServiceInternalMsg("kept segment %d not found", segID)
|
||||
}
|
||||
if segment.GetCollectionID() != collectionID {
|
||||
return fmt.Errorf("collection mismatch for kept segment %d: existing %d, want %d",
|
||||
return merr.WrapErrServiceInternalMsg("collection mismatch for kept segment %d: existing %d, want %d",
|
||||
segID, segment.GetCollectionID(), collectionID)
|
||||
}
|
||||
if segment.GetState() == commonpb.SegmentState_Dropped {
|
||||
return fmt.Errorf("cannot keep dropped segment %d", segID)
|
||||
return merr.WrapErrServiceInternalMsg("cannot keep dropped segment %d", segID)
|
||||
}
|
||||
keptSegmentMap[segID] = true
|
||||
}
|
||||
@@ -250,10 +251,10 @@ func applyExternalCollectionSegmentUpdate(
|
||||
return err
|
||||
}
|
||||
if keptSegmentMap[seg.GetID()] {
|
||||
return fmt.Errorf("segment %d cannot be both kept and updated", seg.GetID())
|
||||
return merr.WrapErrServiceInternalMsg("segment %d cannot be both kept and updated", seg.GetID())
|
||||
}
|
||||
if _, ok := upsertSegmentMap[seg.GetID()]; ok {
|
||||
return fmt.Errorf("duplicate updated segment %d", seg.GetID())
|
||||
return merr.WrapErrServiceInternalMsg("duplicate updated segment %d", seg.GetID())
|
||||
}
|
||||
upsertSegmentMap[seg.GetID()] = seg
|
||||
validUpdatedSegments = append(validUpdatedSegments, seg)
|
||||
@@ -308,7 +309,7 @@ func applyExternalCollectionSegmentUpdate(
|
||||
zap.Int("activeSegmentCount", activeSegmentCount),
|
||||
zap.Int("keptSegments", len(keptSegmentMap)),
|
||||
zap.Int("updatedSegments", len(upsertSegmentMap)))
|
||||
return fmt.Errorf("safety check failed: refusing to drop all %d segments without replacement (keptSegments=%d, updatedSegments=%d)",
|
||||
return merr.WrapErrServiceInternalMsg("safety check failed: refusing to drop all %d segments without replacement (keptSegments=%d, updatedSegments=%d)",
|
||||
activeSegmentCount, len(keptSegmentMap), len(upsertSegmentMap))
|
||||
}
|
||||
|
||||
@@ -330,15 +331,15 @@ func applyExternalCollectionSegmentUpdate(
|
||||
|
||||
collInfo := mt.GetCollection(collectionID)
|
||||
if collInfo == nil {
|
||||
return fmt.Errorf("collection %d not found in meta", collectionID)
|
||||
return merr.WrapErrServiceInternalMsg("collection %d not found in meta", collectionID)
|
||||
}
|
||||
// External collections are single-shard, single-partition (enforced at creation).
|
||||
// Assert exactly-one here to catch any invariant violation from data corruption or legacy data.
|
||||
if len(collInfo.VChannelNames) != 1 {
|
||||
return fmt.Errorf("external collection %d expected exactly 1 VChannel, got %d", collectionID, len(collInfo.VChannelNames))
|
||||
return merr.WrapErrServiceInternalMsg("external collection %d expected exactly 1 VChannel, got %d", collectionID, len(collInfo.VChannelNames))
|
||||
}
|
||||
if len(collInfo.Partitions) != 1 {
|
||||
return fmt.Errorf("external collection %d expected exactly 1 partition, got %d", collectionID, len(collInfo.Partitions))
|
||||
return merr.WrapErrServiceInternalMsg("external collection %d expected exactly 1 partition, got %d", collectionID, len(collInfo.Partitions))
|
||||
}
|
||||
insertChannel := collInfo.VChannelNames[0]
|
||||
partitionID := collInfo.Partitions[0]
|
||||
@@ -477,14 +478,14 @@ func applyExternalCollectionSegmentUpdate(
|
||||
|
||||
func validateExternalRefreshUpdatedSegment(incoming *datapb.SegmentInfo, collectionID int64) error {
|
||||
if incoming.GetCollectionID() != 0 && incoming.GetCollectionID() != collectionID {
|
||||
return fmt.Errorf("collection mismatch for segment %d: got %d, want %d",
|
||||
return merr.WrapErrServiceInternalMsg("collection mismatch for segment %d: got %d, want %d",
|
||||
incoming.GetID(), incoming.GetCollectionID(), collectionID)
|
||||
}
|
||||
if incoming.GetManifestPath() == "" {
|
||||
return fmt.Errorf("updated segment %d has empty manifest path", incoming.GetID())
|
||||
return merr.WrapErrServiceInternalMsg("updated segment %d has empty manifest path", incoming.GetID())
|
||||
}
|
||||
if len(incoming.GetBinlogs()) == 0 {
|
||||
return fmt.Errorf("updated segment %d has empty fake binlogs", incoming.GetID())
|
||||
return merr.WrapErrServiceInternalMsg("updated segment %d has empty fake binlogs", incoming.GetID())
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -513,36 +514,36 @@ func validateExternalRefreshNewSegment(incoming *datapb.SegmentInfo) error {
|
||||
|
||||
func validateExternalRefreshPatch(oldSeg *SegmentInfo, incoming *datapb.SegmentInfo, collectionID int64) error {
|
||||
if oldSeg == nil {
|
||||
return fmt.Errorf("existing segment is nil")
|
||||
return merr.WrapErrServiceInternalMsg("existing segment is nil")
|
||||
}
|
||||
if oldSeg.GetCollectionID() != collectionID {
|
||||
return fmt.Errorf("collection mismatch for segment %d: existing %d, want %d",
|
||||
return merr.WrapErrServiceInternalMsg("collection mismatch for segment %d: existing %d, want %d",
|
||||
oldSeg.GetID(), oldSeg.GetCollectionID(), collectionID)
|
||||
}
|
||||
if oldSeg.GetState() == commonpb.SegmentState_Dropped {
|
||||
return fmt.Errorf("cannot patch dropped segment %d", oldSeg.GetID())
|
||||
return merr.WrapErrServiceInternalMsg("cannot patch dropped segment %d", oldSeg.GetID())
|
||||
}
|
||||
if incoming.GetCollectionID() != 0 && incoming.GetCollectionID() != collectionID {
|
||||
return fmt.Errorf("collection mismatch for segment %d: got %d, want %d",
|
||||
return merr.WrapErrServiceInternalMsg("collection mismatch for segment %d: got %d, want %d",
|
||||
incoming.GetID(), incoming.GetCollectionID(), collectionID)
|
||||
}
|
||||
if incoming.GetNumOfRows() != oldSeg.GetNumOfRows() {
|
||||
return fmt.Errorf("row count changed for segment %d: got %d, want %d",
|
||||
return merr.WrapErrServiceInternalMsg("row count changed for segment %d: got %d, want %d",
|
||||
incoming.GetID(), incoming.GetNumOfRows(), oldSeg.GetNumOfRows())
|
||||
}
|
||||
if incoming.GetStorageVersion() != 0 && incoming.GetStorageVersion() != oldSeg.GetStorageVersion() {
|
||||
return fmt.Errorf("storage version changed for segment %d: got %d, want %d",
|
||||
return merr.WrapErrServiceInternalMsg("storage version changed for segment %d: got %d, want %d",
|
||||
incoming.GetID(), incoming.GetStorageVersion(), oldSeg.GetStorageVersion())
|
||||
}
|
||||
if incoming.GetSchemaVersion() < oldSeg.GetSchemaVersion() {
|
||||
return fmt.Errorf("schema version rollback for segment %d: got %d, want >= %d",
|
||||
return merr.WrapErrServiceInternalMsg("schema version rollback for segment %d: got %d, want >= %d",
|
||||
incoming.GetID(), incoming.GetSchemaVersion(), oldSeg.GetSchemaVersion())
|
||||
}
|
||||
if incoming.GetManifestPath() == "" {
|
||||
return fmt.Errorf("patched segment %d has empty manifest path", incoming.GetID())
|
||||
return merr.WrapErrServiceInternalMsg("patched segment %d has empty manifest path", incoming.GetID())
|
||||
}
|
||||
if len(incoming.GetBinlogs()) == 0 {
|
||||
return fmt.Errorf("patched segment %d has empty fake binlogs", incoming.GetID())
|
||||
return merr.WrapErrServiceInternalMsg("patched segment %d has empty fake binlogs", incoming.GetID())
|
||||
}
|
||||
if err := validateExternalRefreshBinlogRowCount(incoming, oldSeg.GetNumOfRows()); err != nil {
|
||||
return err
|
||||
@@ -553,14 +554,14 @@ func validateExternalRefreshPatch(oldSeg *SegmentInfo, incoming *datapb.SegmentI
|
||||
func validateExternalRefreshBinlogRowCount(segment *datapb.SegmentInfo, expectedRows int64) error {
|
||||
binlogRows := segmentutil.CalcRowCountFromBinLog(segment)
|
||||
if binlogRows == -1 {
|
||||
return fmt.Errorf("invalid binlog row count for segment %d", segment.GetID())
|
||||
return merr.WrapErrServiceInternalMsg("invalid binlog row count for segment %d", segment.GetID())
|
||||
}
|
||||
if expectedRows > 0 && binlogRows != expectedRows {
|
||||
return fmt.Errorf("binlog row count mismatch for segment %d: got %d, want %d",
|
||||
return merr.WrapErrServiceInternalMsg("binlog row count mismatch for segment %d: got %d, want %d",
|
||||
segment.GetID(), binlogRows, expectedRows)
|
||||
}
|
||||
if binlogRows > 0 && binlogRows != segment.GetNumOfRows() {
|
||||
return fmt.Errorf("binlog row count mismatch for segment %d: got %d, segment rows %d",
|
||||
return merr.WrapErrServiceInternalMsg("binlog row count mismatch for segment %d: got %d, segment rows %d",
|
||||
segment.GetID(), binlogRows, segment.GetNumOfRows())
|
||||
}
|
||||
return nil
|
||||
@@ -612,7 +613,7 @@ func (t *refreshExternalCollectionTask) CreateTaskOnWorker(nodeID int64, cluster
|
||||
log.Info("creating refresh task on worker")
|
||||
|
||||
if t.mt == nil {
|
||||
err = fmt.Errorf("meta is nil, cannot create task on worker")
|
||||
err = merr.WrapErrServiceInternalMsg("meta is nil, cannot create task on worker")
|
||||
return
|
||||
}
|
||||
|
||||
@@ -625,7 +626,7 @@ func (t *refreshExternalCollectionTask) CreateTaskOnWorker(nodeID int64, cluster
|
||||
// Re-read task from meta to sync in-memory state (nodeID and version)
|
||||
updatedTask := t.refreshMeta.GetTask(t.GetTaskId())
|
||||
if updatedTask == nil {
|
||||
err = fmt.Errorf("task %d not found after version update", t.GetTaskId())
|
||||
err = merr.WrapErrServiceInternalMsg("task %d not found after version update", t.GetTaskId())
|
||||
return
|
||||
}
|
||||
t.ExternalCollectionRefreshTask = updatedTask
|
||||
@@ -668,11 +669,11 @@ func (t *refreshExternalCollectionTask) CreateTaskOnWorker(nodeID int64, cluster
|
||||
// lock, before they are supported.
|
||||
collInfo := t.mt.GetCollection(t.GetCollectionId())
|
||||
if collInfo == nil {
|
||||
err = fmt.Errorf("collection %d not found in meta", t.GetCollectionId())
|
||||
err = merr.WrapErrServiceInternalMsg("collection %d not found in meta", t.GetCollectionId())
|
||||
return
|
||||
}
|
||||
if len(collInfo.Partitions) != 1 {
|
||||
err = fmt.Errorf("external collection %d expected exactly 1 partition, got %d", t.GetCollectionId(), len(collInfo.Partitions))
|
||||
err = merr.WrapErrServiceInternalMsg("external collection %d expected exactly 1 partition, got %d", t.GetCollectionId(), len(collInfo.Partitions))
|
||||
return
|
||||
}
|
||||
partitionID := collInfo.Partitions[0]
|
||||
|
||||
@@ -18,7 +18,6 @@ package datacoord
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/cockroachdb/errors"
|
||||
@@ -299,11 +298,17 @@ func (st *statsTask) handleEmptySegment(ctx context.Context) error {
|
||||
// Prepare the stats request
|
||||
func (st *statsTask) prepareJobRequest(ctx context.Context, segment *SegmentInfo) (*workerpb.CreateStatsRequest, error) {
|
||||
collInfo, err := st.handler.GetCollection(ctx, segment.GetCollectionID())
|
||||
if err != nil || collInfo == nil {
|
||||
return nil, fmt.Errorf("failed to get collection info: %w", err)
|
||||
if err != nil {
|
||||
return nil, merr.Wrap(err, "failed to get collection info")
|
||||
}
|
||||
// GetCollection can return (nil, nil) on a cache miss; merr.Wrap(nil) would
|
||||
// be nil and silently submit a malformed request, so guard collInfo
|
||||
// separately with a typed not-found.
|
||||
if collInfo == nil {
|
||||
return nil, merr.WrapErrCollectionNotFound(segment.GetCollectionID())
|
||||
}
|
||||
if collInfo.Schema == nil || len(collInfo.Schema.GetFields()) == 0 {
|
||||
return nil, fmt.Errorf("collection schema is nil or has no fields, collectionID: %d", segment.GetCollectionID())
|
||||
return nil, merr.WrapErrServiceInternalMsg("collection schema is nil or has no fields, collectionID: %d", segment.GetCollectionID())
|
||||
}
|
||||
|
||||
// Calculate binlog allocation
|
||||
@@ -314,7 +319,7 @@ func (st *statsTask) prepareJobRequest(ctx context.Context, segment *SegmentInfo
|
||||
// Allocate IDs
|
||||
start, end, err := st.allocator.AllocN(binlogNum + int64(len(collInfo.Schema.GetFunctions())) + 1)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to allocate log IDs: %w", err)
|
||||
return nil, merr.Wrap(err, "failed to allocate log IDs")
|
||||
}
|
||||
|
||||
// Create the request
|
||||
|
||||
@@ -26,7 +26,6 @@ import (
|
||||
|
||||
"github.com/apache/arrow/go/v17/arrow"
|
||||
"github.com/apache/arrow/go/v17/arrow/array"
|
||||
"github.com/cockroachdb/errors"
|
||||
"go.opentelemetry.io/otel"
|
||||
"go.uber.org/zap"
|
||||
|
||||
@@ -113,14 +112,16 @@ func (t *bumpSchemaVersionCompactionTask) preCompact() error {
|
||||
return t.ctx.Err()
|
||||
}
|
||||
|
||||
// Check segment binlogs: must have exactly one segment
|
||||
// Check segment binlogs: must have exactly one segment.
|
||||
// The plan is produced by datacoord, so a malformed plan is an internal
|
||||
// protocol violation, not user input.
|
||||
if len(t.plan.GetSegmentBinlogs()) != 1 {
|
||||
return errors.Newf("schema bump compaction plan is illegal, must have exactly one segment, but got %d segments, planID = %d", len(t.plan.GetSegmentBinlogs()), t.GetPlanID())
|
||||
return merr.WrapErrServiceInternalMsg("schema bump compaction plan is illegal, must have exactly one segment, but got %d segments, planID = %d", len(t.plan.GetSegmentBinlogs()), t.GetPlanID())
|
||||
}
|
||||
|
||||
segment := t.plan.GetSegmentBinlogs()[0]
|
||||
if segment.GetStorageVersion() < storage.StorageV3 || segment.GetManifest() == "" {
|
||||
return errors.Newf("schema bump compaction requires a StorageV3 segment with manifest, planID = %d, segmentID = %d, storageVersion = %d", t.GetPlanID(), segment.GetSegmentID(), segment.GetStorageVersion())
|
||||
return merr.WrapErrServiceInternalMsg("schema bump compaction requires a StorageV3 segment with manifest, planID = %d, segmentID = %d, storageVersion = %d", t.GetPlanID(), segment.GetSegmentID(), segment.GetStorageVersion())
|
||||
}
|
||||
|
||||
return nil
|
||||
@@ -147,7 +148,7 @@ func (t *bumpSchemaVersionCompactionTask) missingFunctionInputSchema(missingFunc
|
||||
for _, inputFieldID := range functionSchema.GetInputFieldIds() {
|
||||
inputField := typeutil.GetField(schema, inputFieldID)
|
||||
if inputField == nil {
|
||||
return nil, nil, errors.New("input field not found in schema")
|
||||
return nil, nil, merr.WrapErrParameterInvalidMsg("input field not found in schema")
|
||||
}
|
||||
if err := validateMaterializationInputField(functionSchema, inputField); err != nil {
|
||||
return nil, nil, err
|
||||
@@ -196,11 +197,11 @@ func bm25AdditionalInputFields(schema *schemapb.CollectionSchema, inputField *sc
|
||||
return nil, err
|
||||
}
|
||||
if multiAnalyzerParams.ByField == "" {
|
||||
return nil, errors.New("multi_analyzer_params missing required 'by_field' key")
|
||||
return nil, merr.WrapErrParameterInvalidMsg("multi_analyzer_params missing required 'by_field' key")
|
||||
}
|
||||
byField := typeutil.GetFieldByName(schema, multiAnalyzerParams.ByField)
|
||||
if byField == nil {
|
||||
return nil, errors.New("input field not found in schema")
|
||||
return nil, merr.WrapErrParameterInvalidMsg("input field not found in schema")
|
||||
}
|
||||
return []*schemapb.FieldSchema{byField}, nil
|
||||
}
|
||||
@@ -217,7 +218,7 @@ func (t *bumpSchemaVersionCompactionTask) missingFunctionOutputFields(missingFun
|
||||
outputFieldID := functionSchema.GetOutputFieldIds()[outputIndex]
|
||||
outputField := typeutil.GetField(schema, outputFieldID)
|
||||
if outputField == nil {
|
||||
return nil, nil, errors.New("output field not found in schema")
|
||||
return nil, nil, merr.WrapErrParameterInvalidMsg("output field not found in schema")
|
||||
}
|
||||
if err := validateMaterializationOutputField(functionSchema, outputField); err != nil {
|
||||
return nil, nil, err
|
||||
@@ -227,7 +228,7 @@ func (t *bumpSchemaVersionCompactionTask) missingFunctionOutputFields(missingFun
|
||||
}
|
||||
}
|
||||
if len(fields) == 0 {
|
||||
return nil, nil, errors.New("no missing function output fields")
|
||||
return nil, nil, merr.WrapErrParameterInvalidMsg("no missing function output fields")
|
||||
}
|
||||
return fields, fieldIDs, nil
|
||||
}
|
||||
@@ -236,22 +237,22 @@ func validateSupportedMissingFunctionMaterialization(functionSchema *schemapb.Fu
|
||||
switch functionSchema.GetType() {
|
||||
case schemapb.FunctionType_BM25:
|
||||
if len(functionSchema.GetInputFieldIds()) == 0 {
|
||||
return errors.New("bm25 function should have input fields")
|
||||
return merr.WrapErrParameterInvalidMsg("bm25 function should have input fields")
|
||||
}
|
||||
if len(functionSchema.GetOutputFieldIds()) == 0 {
|
||||
return errors.New("bm25 function should have output fields")
|
||||
return merr.WrapErrParameterInvalidMsg("bm25 function should have output fields")
|
||||
}
|
||||
return nil
|
||||
case schemapb.FunctionType_MinHash:
|
||||
if len(functionSchema.GetInputFieldIds()) == 0 {
|
||||
return errors.New("minhash function should have input fields")
|
||||
return merr.WrapErrParameterInvalidMsg("minhash function should have input fields")
|
||||
}
|
||||
if len(functionSchema.GetOutputFieldIds()) == 0 {
|
||||
return errors.New("minhash function should have output fields")
|
||||
return merr.WrapErrParameterInvalidMsg("minhash function should have output fields")
|
||||
}
|
||||
return nil
|
||||
default:
|
||||
return errors.New("unsupported function type")
|
||||
return merr.WrapErrParameterInvalidMsg("unsupported function type")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -259,11 +260,11 @@ func validateMaterializationInputField(functionSchema *schemapb.FunctionSchema,
|
||||
switch functionSchema.GetType() {
|
||||
case schemapb.FunctionType_BM25:
|
||||
if field.GetDataType() != schemapb.DataType_VarChar && field.GetDataType() != schemapb.DataType_Text {
|
||||
return errors.New("input field data type must be varchar or text for bm25 materialization")
|
||||
return merr.WrapErrParameterInvalidMsg("input field data type must be varchar or text for bm25 materialization")
|
||||
}
|
||||
case schemapb.FunctionType_MinHash:
|
||||
if field.GetDataType() != schemapb.DataType_VarChar && field.GetDataType() != schemapb.DataType_Text {
|
||||
return errors.New("input field data type must be varchar or text for minhash materialization")
|
||||
return merr.WrapErrParameterInvalidMsg("input field data type must be varchar or text for minhash materialization")
|
||||
}
|
||||
}
|
||||
return nil
|
||||
@@ -273,11 +274,11 @@ func validateMaterializationOutputField(functionSchema *schemapb.FunctionSchema,
|
||||
switch functionSchema.GetType() {
|
||||
case schemapb.FunctionType_BM25:
|
||||
if field.GetDataType() != schemapb.DataType_SparseFloatVector {
|
||||
return errors.New("output field data type must be sparse float vector for bm25 materialization")
|
||||
return merr.WrapErrParameterInvalidMsg("output field data type must be sparse float vector for bm25 materialization")
|
||||
}
|
||||
case schemapb.FunctionType_MinHash:
|
||||
if field.GetDataType() != schemapb.DataType_BinaryVector {
|
||||
return errors.New("output field data type must be binary vector for minhash materialization")
|
||||
return merr.WrapErrParameterInvalidMsg("output field data type must be binary vector for minhash materialization")
|
||||
}
|
||||
}
|
||||
return nil
|
||||
@@ -546,7 +547,7 @@ func (t *bumpSchemaVersionCompactionTask) runFullSchemaRewrite(existingFields ma
|
||||
}
|
||||
manifestPath, err = packed.AddStatsToManifest(manifestPath, t.compactionParams.StorageConfig, statEntries)
|
||||
if err != nil {
|
||||
return nil, merr.WrapErrServiceInternal("failed to add writer stats to schema bump full rewrite manifest", err.Error())
|
||||
return nil, merr.Wrap(err, "failed to add writer stats to schema bump full rewrite manifest")
|
||||
}
|
||||
}
|
||||
sortedInsertLogs := storage.SortFieldBinlogs(insertLogs)
|
||||
@@ -602,7 +603,7 @@ func (t *bumpSchemaVersionCompactionTask) runFullSchemaRewrite(existingFields ma
|
||||
func appendBM25StatsFromArrowArray(stats *storage.BM25Stats, arr arrow.Array) (int, error) {
|
||||
binaryArray, ok := arr.(*array.Binary)
|
||||
if !ok {
|
||||
return 0, errors.Newf("bm25 output field must be arrow binary array, got %T", arr)
|
||||
return 0, merr.WrapErrParameterInvalidMsg("bm25 output field must be arrow binary array, got %T", arr)
|
||||
}
|
||||
memorySize := 0
|
||||
for i := 0; i < binaryArray.Len(); i++ {
|
||||
@@ -667,7 +668,7 @@ func (t *bumpSchemaVersionCompactionTask) setupWriter(outputFields []*schemapb.F
|
||||
|
||||
basePath, existingVersion, err := packed.UnmarshalManifestPath(segment.GetManifest())
|
||||
if err != nil {
|
||||
return nil, merr.WrapErrServiceInternal("failed to parse existing manifest for schema bump", err.Error())
|
||||
return nil, merr.WrapErrDataIntegrityMsg("failed to parse existing manifest for schema bump: %s", err.Error())
|
||||
}
|
||||
writerResult, err := t.newV3WriterResult(outputSchema, newColumnGroups, segment, collectionID, basePath, existingVersion)
|
||||
if err != nil {
|
||||
@@ -685,7 +686,7 @@ func (t *bumpSchemaVersionCompactionTask) setupWriter(outputFields []*schemapb.F
|
||||
|
||||
func (t *bumpSchemaVersionCompactionTask) newV3WriterResult(schema *schemapb.CollectionSchema, columnGroups []storagecommon.ColumnGroup, segment *datapb.CompactionSegmentBinlogs, collectionID int64, basePath string, baseVersion int64) (*bumpSchemaVersionWriterResult, error) {
|
||||
if segment.GetStorageVersion() < storage.StorageV3 || segment.GetManifest() == "" {
|
||||
return nil, errors.New("schema bump compaction requires a StorageV3 segment with manifest")
|
||||
return nil, merr.WrapErrParameterInvalidMsg("schema bump compaction requires a StorageV3 segment with manifest")
|
||||
}
|
||||
|
||||
pluginContext, err := hookutil.GetCPluginContext(t.plan.GetPluginContext(), collectionID)
|
||||
@@ -732,7 +733,7 @@ func (t *bumpSchemaVersionCompactionTask) addV3Stats(prefix string, fieldID int6
|
||||
statsRelPath := fmt.Sprintf("_stats/%s.%d/%d", prefix, fieldID, statsID)
|
||||
absStatsPath := path.Join(writerResult.basePath, statsRelPath)
|
||||
if err := packed.WriteFile(t.compactionParams.StorageConfig, absStatsPath, bytes); err != nil {
|
||||
return merr.WrapErrServiceInternal("failed to write V3 stats", err.Error())
|
||||
return merr.Wrap(err, "failed to write V3 stats")
|
||||
}
|
||||
writerResult.v3Stats = append(writerResult.v3Stats, packed.FieldBinlogStatEntry(prefix, fieldID, &datapb.FieldBinlog{
|
||||
FieldID: fieldID,
|
||||
@@ -773,7 +774,7 @@ func (t *bumpSchemaVersionCompactionTask) buildMergedLogsV3(segment *datapb.Comp
|
||||
func (t *bumpSchemaVersionCompactionTask) preserveDeltaLogsV3(segment *datapb.CompactionSegmentBinlogs, manifestPath string) (string, error) {
|
||||
deltaPaths, err := packed.GetDeltaLogPathsFromManifest(segment.GetManifest(), t.compactionParams.StorageConfig)
|
||||
if err != nil {
|
||||
return "", merr.WrapErrServiceInternal("failed to read V3 delta logs from existing manifest", err.Error())
|
||||
return "", merr.Wrap(err, "failed to read V3 delta logs from existing manifest")
|
||||
}
|
||||
if len(deltaPaths) == 0 {
|
||||
return manifestPath, nil
|
||||
@@ -784,12 +785,12 @@ func (t *bumpSchemaVersionCompactionTask) preserveDeltaLogsV3(segment *datapb.Co
|
||||
deltaSummaries = append(deltaSummaries, fieldBinlog.GetBinlogs()...)
|
||||
}
|
||||
if len(deltaSummaries) != len(deltaPaths) {
|
||||
return "", merr.WrapErrServiceInternal(fmt.Sprintf("V3 delta manifest path count %d does not match segment delta summary count %d", len(deltaPaths), len(deltaSummaries)))
|
||||
return "", merr.WrapErrServiceInternalMsg("V3 delta manifest path count %d does not match segment delta summary count %d", len(deltaPaths), len(deltaSummaries))
|
||||
}
|
||||
|
||||
basePath, _, err := packed.UnmarshalManifestPath(manifestPath)
|
||||
if err != nil {
|
||||
return "", merr.WrapErrServiceInternal("failed to parse new V3 manifest for delta preservation", err.Error())
|
||||
return "", merr.WrapErrDataIntegrityMsg("failed to parse new V3 manifest for delta preservation: %s", err.Error())
|
||||
}
|
||||
|
||||
deltaEntries := make([]packed.DeltaLogEntry, 0, len(deltaPaths))
|
||||
@@ -797,7 +798,7 @@ func (t *bumpSchemaVersionCompactionTask) preserveDeltaLogsV3(segment *datapb.Co
|
||||
newDeltaPath := metautil.BuildDeltaLogPathV3(basePath, deltaSummaries[i].GetLogID())
|
||||
if newDeltaPath != deltaPath {
|
||||
if err := t.chunkManager.Copy(t.ctx, deltaPath, newDeltaPath); err != nil {
|
||||
return "", merr.WrapErrServiceInternal("failed to copy V3 delta log for schema bump full rewrite", err.Error())
|
||||
return "", merr.Wrap(err, "failed to copy V3 delta log for schema bump full rewrite")
|
||||
}
|
||||
}
|
||||
deltaEntries = append(deltaEntries, packed.DeltaLogEntry{
|
||||
@@ -807,7 +808,7 @@ func (t *bumpSchemaVersionCompactionTask) preserveDeltaLogsV3(segment *datapb.Co
|
||||
}
|
||||
newManifest, err := packed.AddDeltaLogsToManifest(manifestPath, t.compactionParams.StorageConfig, deltaEntries)
|
||||
if err != nil {
|
||||
return "", merr.WrapErrServiceInternal("failed to preserve V3 delta logs in full rewrite manifest", err.Error())
|
||||
return "", merr.Wrap(err, "failed to preserve V3 delta logs in full rewrite manifest")
|
||||
}
|
||||
return newManifest, nil
|
||||
}
|
||||
@@ -923,7 +924,7 @@ func (t *bumpSchemaVersionCompactionTask) runMissingFunctionMaterialization(ctx
|
||||
if outputCol == nil {
|
||||
releaseWrappedRecord(wrapped, record)
|
||||
span.End()
|
||||
return nil, merr.WrapErrServiceInternal(fmt.Sprintf("output field %d not found in materialized record", outputFieldID))
|
||||
return nil, merr.WrapErrServiceInternalMsg("output field %d not found in materialized record", outputFieldID)
|
||||
}
|
||||
batchMemSize := arrowArrayMemorySize(outputCol)
|
||||
if stats, ok := statsByField[outputFieldID]; ok {
|
||||
@@ -997,7 +998,7 @@ func (t *bumpSchemaVersionCompactionTask) runMissingFunctionMaterialization(ctx
|
||||
},
|
||||
)
|
||||
if err != nil {
|
||||
return nil, merr.WrapErrServiceInternal("failed to commit schema bump V3 manifest", err.Error())
|
||||
return nil, merr.Wrap(err, "failed to commit schema bump V3 manifest")
|
||||
}
|
||||
log.Info("[schema-bump-partial-writer] writer output and bm25 stats committed",
|
||||
zap.String("manifestPath", manifestPath),
|
||||
|
||||
@@ -28,7 +28,6 @@ import (
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/cockroachdb/errors"
|
||||
"github.com/samber/lo"
|
||||
"go.opentelemetry.io/otel"
|
||||
"go.uber.org/atomic"
|
||||
@@ -663,7 +662,7 @@ func (t *clusteringCompactionTask) mappingSegment(
|
||||
row, ok := v.Value.(map[typeutil.UniqueID]interface{})
|
||||
if !ok {
|
||||
log.Warn("convert interface to map wrong")
|
||||
return errors.New("unexpected error")
|
||||
return merr.WrapErrServiceInternalMsg("unexpected error")
|
||||
}
|
||||
expireTs := int64(-1)
|
||||
if hasTTLField {
|
||||
@@ -976,7 +975,7 @@ func (t *clusteringCompactionTask) iterAndGetScalarAnalyzeResult(pkIter *storage
|
||||
// rowValue := vIter.GetData().(*iterators.InsertRow).GetValue()
|
||||
row, ok := (*v).Value.(map[typeutil.UniqueID]interface{})
|
||||
if !ok {
|
||||
return nil, 0, errors.New("unexpected error")
|
||||
return nil, 0, merr.WrapErrServiceInternalMsg("unexpected error")
|
||||
}
|
||||
|
||||
expireTs := int64(-1)
|
||||
|
||||
@@ -46,6 +46,7 @@ import (
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/datapb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/indexcgopb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/metautil"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/tsoutil"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/typeutil"
|
||||
@@ -80,7 +81,7 @@ func createTextIndex(ctx context.Context,
|
||||
}
|
||||
binlogs, ok := fieldBinlogs[fieldID]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("field binlog not found for field %d", fieldID)
|
||||
return nil, merr.WrapErrParameterInvalidMsg("field binlog not found for field %d", fieldID)
|
||||
}
|
||||
result := make([]string, 0, len(binlogs))
|
||||
for _, binlog := range binlogs {
|
||||
@@ -136,7 +137,7 @@ func createTextIndex(ctx context.Context,
|
||||
if segment.GetManifest() != "" {
|
||||
basePath, _, err := packed.UnmarshalManifestPath(segment.GetManifest())
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to unmarshal manifest path for text_index basePath: %w", err)
|
||||
return merr.Wrap(err, "failed to unmarshal manifest path for text_index basePath")
|
||||
}
|
||||
statsBasePath = fmt.Sprintf("%s/_stats/text_index.%d", basePath, field.GetFieldID())
|
||||
}
|
||||
|
||||
@@ -18,10 +18,10 @@ package compactor
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"math"
|
||||
"sync"
|
||||
|
||||
"github.com/cockroachdb/errors"
|
||||
"github.com/samber/lo"
|
||||
"go.opentelemetry.io/otel"
|
||||
"go.uber.org/zap"
|
||||
@@ -40,6 +40,7 @@ import (
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/conc"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/funcutil"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/hardware"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/metautil"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/timerecord"
|
||||
@@ -135,7 +136,9 @@ func (t *LevelZeroCompactionTask) Compact() (*datapb.CompactionPlanResult, error
|
||||
})
|
||||
if len(targetSegments) == 0 {
|
||||
log.Warn("compact wrong, not target sealed segments")
|
||||
return nil, errors.New("illegal compaction plan with empty target segments")
|
||||
// The plan is produced by datacoord, so a malformed plan is an internal
|
||||
// protocol violation, not user input.
|
||||
return nil, merr.WrapErrServiceInternalMsg("illegal compaction plan with empty target segments")
|
||||
}
|
||||
err = binlog.DecompressCompactionBinlogsWithRootPath(t.compactionParams.StorageConfig.GetRootPath(), l0Segments)
|
||||
if err != nil {
|
||||
@@ -416,7 +419,7 @@ func (t *LevelZeroCompactionTask) process(ctx context.Context, l0MemSize int64,
|
||||
ratio := paramtable.Get().DataNodeCfg.L0BatchMemoryRatio.GetAsFloat()
|
||||
memLimit := float64(hardware.GetFreeMemoryCount()) * ratio
|
||||
if float64(l0MemSize) > memLimit {
|
||||
return nil, errors.Newf("L0 compaction failed, not enough memory, request memory size: %v, memory limit: %v", l0MemSize, memLimit)
|
||||
return nil, merr.Wrap(merr.ErrServiceMemoryLimitExceeded, fmt.Sprintf("L0 compaction failed, not enough memory, request memory size: %v, memory limit: %v", l0MemSize, memLimit))
|
||||
}
|
||||
|
||||
log.Info("L0 compaction process start")
|
||||
|
||||
@@ -25,7 +25,6 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/apache/arrow/go/v17/arrow/array"
|
||||
"github.com/cockroachdb/errors"
|
||||
"github.com/samber/lo"
|
||||
"go.opentelemetry.io/otel"
|
||||
"go.uber.org/zap"
|
||||
@@ -42,6 +41,7 @@ import (
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/datapb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/funcutil"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/metautil"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/timerecord"
|
||||
@@ -110,11 +110,13 @@ func (t *mixCompactionTask) preCompact() error {
|
||||
}
|
||||
|
||||
if len(t.plan.GetSegmentBinlogs()) < 1 {
|
||||
return errors.Newf("compaction plan is illegal, there's no segments in compaction plan, planID = %d", t.GetPlanID())
|
||||
// The plan is produced by datacoord, so a malformed plan is an internal
|
||||
// protocol violation, not user input.
|
||||
return merr.WrapErrServiceInternalMsg("compaction plan is illegal, there's no segments in compaction plan, planID = %d", t.GetPlanID())
|
||||
}
|
||||
|
||||
if t.plan.GetMaxSize() == 0 {
|
||||
return errors.Newf("compaction plan is illegal, empty maxSize, planID = %d", t.GetPlanID())
|
||||
return merr.WrapErrServiceInternalMsg("compaction plan is illegal, empty maxSize, planID = %d", t.GetPlanID())
|
||||
}
|
||||
|
||||
t.collectionID = t.plan.GetSegmentBinlogs()[0].GetCollectionID()
|
||||
@@ -440,7 +442,7 @@ func (t *mixCompactionTask) Compact() (*datapb.CompactionPlanResult, error) {
|
||||
|
||||
if isEmpty {
|
||||
log.Warn("compact wrong, all segments' binlogs are empty")
|
||||
return nil, errors.New("illegal compaction plan")
|
||||
return nil, merr.WrapErrServiceInternalMsg("illegal compaction plan")
|
||||
}
|
||||
|
||||
sortMergeAppicable := t.compactionParams.UseMergeSort
|
||||
|
||||
@@ -17,12 +17,9 @@
|
||||
package compactor
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/apache/arrow/go/v17/arrow"
|
||||
"github.com/apache/arrow/go/v17/arrow/array"
|
||||
"github.com/apache/arrow/go/v17/arrow/memory"
|
||||
"github.com/cockroachdb/errors"
|
||||
|
||||
"github.com/milvus-io/milvus-proto/go-api/v3/schemapb"
|
||||
"github.com/milvus-io/milvus/internal/storage"
|
||||
@@ -79,7 +76,7 @@ func NewRecordMaterializer(schema *schemapb.CollectionSchema, functions []*schem
|
||||
}
|
||||
if runner == nil {
|
||||
materializer.Close()
|
||||
return nil, errors.Newf("failed to set up function runner for %s", functionSchema.GetName())
|
||||
return nil, merr.WrapErrFunctionFailedMsg("failed to set up function runner for %s", functionSchema.GetName())
|
||||
}
|
||||
functionMaterializer, err := newFunctionMaterializer(schema, runner, outputIndexes, true)
|
||||
if err != nil {
|
||||
@@ -243,7 +240,7 @@ func (r *selectedRecord) Column(fieldID storage.FieldID) arrow.Array {
|
||||
defer selected.Release()
|
||||
col := selected.Column(fieldID)
|
||||
if col == nil {
|
||||
r.err = errors.Newf("selected record field %d not found", fieldID)
|
||||
r.err = merr.WrapErrServiceInternalMsg("selected record field %d not found", fieldID)
|
||||
return nil
|
||||
}
|
||||
col.Retain()
|
||||
@@ -347,7 +344,7 @@ func newFunctionMaterializer(schema *schemapb.CollectionSchema, runner function.
|
||||
case schemapb.FunctionType_MinHash:
|
||||
return newMinHashFunctionMaterializer(schema, runner, missingOutputIndexes, ownRunner)
|
||||
default:
|
||||
return nil, errors.Newf("unsupported function type %s", functionSchema.GetType().String())
|
||||
return nil, merr.WrapErrParameterInvalidMsg("unsupported function type %s", functionSchema.GetType().String())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -355,35 +352,35 @@ func newMinHashFunctionMaterializer(schema *schemapb.CollectionSchema, runner fu
|
||||
functionSchema := runner.GetSchema()
|
||||
inputFields := runner.GetInputFields()
|
||||
if len(inputFields) == 0 {
|
||||
return nil, errors.New("minhash function should have input fields")
|
||||
return nil, merr.WrapErrFunctionFailedMsg("minhash function should have input fields")
|
||||
}
|
||||
inputFieldIDs := make([]int64, 0, len(inputFields))
|
||||
for _, inputField := range inputFields {
|
||||
if inputField == nil || typeutil.GetField(schema, inputField.GetFieldID()) == nil {
|
||||
return nil, errors.New("input field not found in schema")
|
||||
return nil, merr.WrapErrFunctionFailedMsg("input field not found in schema")
|
||||
}
|
||||
if inputField.GetDataType() != schemapb.DataType_VarChar && inputField.GetDataType() != schemapb.DataType_Text {
|
||||
return nil, errors.New("input field data type must be varchar or text for minhash function materialization")
|
||||
return nil, merr.WrapErrFunctionFailedMsg("input field data type must be varchar or text for minhash function materialization")
|
||||
}
|
||||
inputFieldIDs = append(inputFieldIDs, inputField.GetFieldID())
|
||||
}
|
||||
|
||||
outputFieldIDs := functionSchema.GetOutputFieldIds()
|
||||
if len(outputFieldIDs) == 0 {
|
||||
return nil, errors.New("minhash function should have output fields")
|
||||
return nil, merr.WrapErrFunctionFailedMsg("minhash function should have output fields")
|
||||
}
|
||||
|
||||
outputFields := make(map[int64]*schemapb.FieldSchema, len(outputFieldIDs))
|
||||
for _, outputFieldID := range outputFieldIDs {
|
||||
outputField := typeutil.GetField(schema, outputFieldID)
|
||||
if outputField == nil {
|
||||
return nil, errors.New("output field not found in schema")
|
||||
return nil, merr.WrapErrFunctionFailedMsg("output field not found in schema")
|
||||
}
|
||||
if outputField.GetDataType() != schemapb.DataType_BinaryVector {
|
||||
return nil, errors.New("output field data type must be binary vector for minhash function materialization")
|
||||
return nil, merr.WrapErrFunctionFailedMsg("output field data type must be binary vector for minhash function materialization")
|
||||
}
|
||||
if outputField.GetNullable() {
|
||||
return nil, errors.Newf("function output field cannot be nullable: function %s, field %s", functionSchema.GetName(), outputField.GetName())
|
||||
return nil, merr.WrapErrFunctionFailedMsg("function output field cannot be nullable: function %s, field %s", functionSchema.GetName(), outputField.GetName())
|
||||
}
|
||||
outputFields[outputFieldID] = outputField
|
||||
}
|
||||
@@ -402,35 +399,35 @@ func newBM25FunctionMaterializer(schema *schemapb.CollectionSchema, runner funct
|
||||
functionSchema := runner.GetSchema()
|
||||
inputFields := runner.GetInputFields()
|
||||
if len(inputFields) == 0 {
|
||||
return nil, errors.New("bm25 function should have input fields")
|
||||
return nil, merr.WrapErrParameterInvalidMsg("bm25 function should have input fields")
|
||||
}
|
||||
inputFieldIDs := make([]int64, 0, len(inputFields))
|
||||
for _, inputField := range inputFields {
|
||||
if inputField == nil || typeutil.GetField(schema, inputField.GetFieldID()) == nil {
|
||||
return nil, errors.New("input field not found in schema")
|
||||
return nil, merr.WrapErrParameterInvalidMsg("input field not found in schema")
|
||||
}
|
||||
if inputField.GetDataType() != schemapb.DataType_VarChar && inputField.GetDataType() != schemapb.DataType_Text {
|
||||
return nil, errors.New("input field data type must be varchar or text for bm25 function materialization")
|
||||
return nil, merr.WrapErrParameterInvalidMsg("input field data type must be varchar or text for bm25 function materialization")
|
||||
}
|
||||
inputFieldIDs = append(inputFieldIDs, inputField.GetFieldID())
|
||||
}
|
||||
|
||||
outputFieldIDs := functionSchema.GetOutputFieldIds()
|
||||
if len(outputFieldIDs) == 0 {
|
||||
return nil, errors.New("bm25 function should have output fields")
|
||||
return nil, merr.WrapErrParameterInvalidMsg("bm25 function should have output fields")
|
||||
}
|
||||
|
||||
outputFields := make(map[int64]*schemapb.FieldSchema, len(outputFieldIDs))
|
||||
for _, outputFieldID := range outputFieldIDs {
|
||||
outputField := typeutil.GetField(schema, outputFieldID)
|
||||
if outputField == nil {
|
||||
return nil, errors.New("output field not found in schema")
|
||||
return nil, merr.WrapErrParameterInvalidMsg("output field not found in schema")
|
||||
}
|
||||
if outputField.GetDataType() != schemapb.DataType_SparseFloatVector {
|
||||
return nil, errors.New("output field data type must be sparse float vector for bm25 function materialization")
|
||||
return nil, merr.WrapErrParameterInvalidMsg("output field data type must be sparse float vector for bm25 function materialization")
|
||||
}
|
||||
if outputField.GetNullable() {
|
||||
return nil, errors.Newf("function output field cannot be nullable: function %s, field %s", functionSchema.GetName(), outputField.GetName())
|
||||
return nil, merr.WrapErrParameterInvalidMsg("function output field cannot be nullable: function %s, field %s", functionSchema.GetName(), outputField.GetName())
|
||||
}
|
||||
outputFields[outputFieldID] = outputField
|
||||
}
|
||||
@@ -459,7 +456,7 @@ func (m *bm25FunctionMaterializer) Materialize(rec storage.Record) (map[int64]ar
|
||||
return nil, err
|
||||
}
|
||||
if len(outputs) != len(m.outputFieldIDs) {
|
||||
return nil, errors.Newf("bm25 function materialization expects %d outputs, got %d", len(m.outputFieldIDs), len(outputs))
|
||||
return nil, merr.WrapErrFunctionFailedMsg("bm25 function materialization expects %d outputs, got %d", len(m.outputFieldIDs), len(outputs))
|
||||
}
|
||||
|
||||
result := make(map[int64]arrow.Array, len(m.missingOutputIndexes))
|
||||
@@ -468,7 +465,7 @@ func (m *bm25FunctionMaterializer) Materialize(rec storage.Record) (map[int64]ar
|
||||
outputSparseArray, ok := outputs[outputIndex].(*schemapb.SparseFloatArray)
|
||||
if !ok {
|
||||
releaseArrowArrays(result)
|
||||
return nil, errors.Newf("unexpected output type from BM25 function runner, expected SparseFloatArray, got %T", outputs[outputIndex])
|
||||
return nil, merr.WrapErrFunctionFailedMsg("unexpected output type from BM25 function runner, expected SparseFloatArray, got %T", outputs[outputIndex])
|
||||
}
|
||||
arr, err := buildSparseFloatVectorArrowArray(m.outputFields[outputFieldID], outputSparseArray, rec.Len())
|
||||
if err != nil {
|
||||
@@ -500,7 +497,7 @@ func (m *minHashFunctionMaterializer) Materialize(rec storage.Record) (map[int64
|
||||
return nil, err
|
||||
}
|
||||
if len(outputs) != len(m.outputFieldIDs) {
|
||||
return nil, errors.Newf("minhash function materialization expects %d outputs, got %d", len(m.outputFieldIDs), len(outputs))
|
||||
return nil, merr.WrapErrFunctionFailedMsg("minhash function materialization expects %d outputs, got %d", len(m.outputFieldIDs), len(outputs))
|
||||
}
|
||||
|
||||
result := make(map[int64]arrow.Array, len(m.missingOutputIndexes))
|
||||
@@ -509,12 +506,12 @@ func (m *minHashFunctionMaterializer) Materialize(rec storage.Record) (map[int64
|
||||
outputFieldData, ok := outputs[outputIndex].(*schemapb.FieldData)
|
||||
if !ok {
|
||||
releaseArrowArrays(result)
|
||||
return nil, errors.Newf("unexpected output type from MinHash function runner, expected FieldData, got %T", outputs[outputIndex])
|
||||
return nil, merr.WrapErrFunctionFailedMsg("unexpected output type from MinHash function runner, expected FieldData, got %T", outputs[outputIndex])
|
||||
}
|
||||
vectorField := outputFieldData.GetVectors()
|
||||
if vectorField == nil || vectorField.GetBinaryVector() == nil {
|
||||
releaseArrowArrays(result)
|
||||
return nil, errors.New("unexpected output from MinHash function runner, expected binary vector field data")
|
||||
return nil, merr.WrapErrFunctionFailedMsg("unexpected output from MinHash function runner, expected binary vector field data")
|
||||
}
|
||||
fieldData := &storage.BinaryVectorFieldData{
|
||||
Data: vectorField.GetBinaryVector(),
|
||||
@@ -522,7 +519,7 @@ func (m *minHashFunctionMaterializer) Materialize(rec storage.Record) (map[int64
|
||||
}
|
||||
if fieldData.RowNum() != rec.Len() {
|
||||
releaseArrowArrays(result)
|
||||
return nil, errors.Newf("minhash function output row count mismatch, expected %d, got %d", rec.Len(), fieldData.RowNum())
|
||||
return nil, merr.WrapErrFunctionFailedMsg("minhash function output row count mismatch, expected %d, got %d", rec.Len(), fieldData.RowNum())
|
||||
}
|
||||
arr, err := buildArrowArrayFromFieldData(m.outputFields[outputFieldID], fieldData, rec.Len())
|
||||
if err != nil {
|
||||
@@ -577,7 +574,7 @@ func missingNonMaterializedSchemaFields(schema *schemapb.CollectionSchema, exist
|
||||
func stringInputsFromRecord(rec storage.Record, fieldID int64) ([]string, error) {
|
||||
col := rec.Column(fieldID)
|
||||
if col == nil {
|
||||
return nil, merr.WrapErrServiceInternal(fmt.Sprintf("input field %d not found in record", fieldID))
|
||||
return nil, merr.WrapErrFunctionFailedMsg("input field %d not found in record", fieldID)
|
||||
}
|
||||
inputs := make([]string, rec.Len())
|
||||
switch values := col.(type) {
|
||||
@@ -588,16 +585,16 @@ func stringInputsFromRecord(rec storage.Record, fieldID int64) ([]string, error)
|
||||
}
|
||||
}
|
||||
case *array.Binary:
|
||||
return nil, merr.WrapErrServiceInternal("cannot materialize bm25 from text binary values without lob decoding")
|
||||
return nil, merr.WrapErrFunctionFailedMsg("cannot materialize bm25 from text binary values without lob decoding")
|
||||
default:
|
||||
return nil, merr.WrapErrServiceInternal(fmt.Sprintf("input field %d data type must be varchar or text for bm25 function materialization, got %T", fieldID, col))
|
||||
return nil, merr.WrapErrFunctionFailedMsg("input field %d data type must be varchar or text for bm25 function materialization, got %T", fieldID, col)
|
||||
}
|
||||
return inputs, nil
|
||||
}
|
||||
|
||||
func buildSparseFloatVectorArrowArray(field *schemapb.FieldSchema, outputSparseArray *schemapb.SparseFloatArray, rowCount int) (arrow.Array, error) {
|
||||
if len(outputSparseArray.GetContents()) != rowCount {
|
||||
return nil, errors.Newf("bm25 function output row count mismatch, expected %d, got %d", rowCount, len(outputSparseArray.GetContents()))
|
||||
return nil, merr.WrapErrFunctionFailedMsg("bm25 function output row count mismatch, expected %d, got %d", rowCount, len(outputSparseArray.GetContents()))
|
||||
}
|
||||
|
||||
fieldData := &storage.SparseFloatVectorFieldData{
|
||||
@@ -612,7 +609,7 @@ func buildSparseFloatVectorArrowArray(field *schemapb.FieldSchema, outputSparseA
|
||||
|
||||
func buildArrowArrayFromFieldData(field *schemapb.FieldSchema, fieldData storage.FieldData, rowCount int) (arrow.Array, error) {
|
||||
if fieldData.RowNum() != rowCount {
|
||||
return nil, errors.Newf("function output row count mismatch for field %d, expected %d, got %d", field.GetFieldID(), rowCount, fieldData.RowNum())
|
||||
return nil, merr.WrapErrFunctionFailedMsg("function output row count mismatch for field %d, expected %d, got %d", field.GetFieldID(), rowCount, fieldData.RowNum())
|
||||
}
|
||||
|
||||
outputSchema := &schemapb.CollectionSchema{Fields: []*schemapb.FieldSchema{field}}
|
||||
|
||||
@@ -22,7 +22,6 @@ import (
|
||||
"math"
|
||||
|
||||
"github.com/apache/arrow/go/v17/arrow/array"
|
||||
"github.com/cockroachdb/errors"
|
||||
"github.com/samber/lo"
|
||||
"go.uber.org/atomic"
|
||||
"go.uber.org/zap"
|
||||
@@ -37,6 +36,7 @@ import (
|
||||
"github.com/milvus-io/milvus/pkg/v3/log"
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/datapb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/etcdpb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/typeutil"
|
||||
)
|
||||
@@ -318,7 +318,7 @@ func (w *SegmentWriter) WriteRecord(r storage.Record) error {
|
||||
for fieldID, stats := range w.bm25Stats {
|
||||
field, ok := r.Column(fieldID).(*array.Binary)
|
||||
if !ok {
|
||||
return errors.New("bm25 field value not found")
|
||||
return merr.WrapErrServiceInternalMsg("bm25 field value not found")
|
||||
}
|
||||
stats.AppendBytes(field.Value(i))
|
||||
}
|
||||
@@ -341,12 +341,12 @@ func (w *SegmentWriter) Write(v *storage.Value) error {
|
||||
for fieldID, stats := range w.bm25Stats {
|
||||
data, ok := v.Value.(map[storage.FieldID]interface{})[fieldID]
|
||||
if !ok {
|
||||
return errors.New("bm25 field value not found")
|
||||
return merr.WrapErrServiceInternalMsg("bm25 field value not found")
|
||||
}
|
||||
|
||||
bytes, ok := data.([]byte)
|
||||
if !ok {
|
||||
return errors.New("bm25 field value not sparse bytes")
|
||||
return merr.WrapErrServiceInternalMsg("bm25 field value not sparse bytes")
|
||||
}
|
||||
stats.AppendBytes(bytes)
|
||||
}
|
||||
|
||||
@@ -23,7 +23,6 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/apache/arrow/go/v17/arrow/array"
|
||||
"github.com/cockroachdb/errors"
|
||||
"github.com/samber/lo"
|
||||
"go.opentelemetry.io/otel"
|
||||
"go.uber.org/zap"
|
||||
@@ -109,7 +108,9 @@ func (t *sortCompactionTask) preCompact() error {
|
||||
}
|
||||
|
||||
if len(t.plan.GetSegmentBinlogs()) != 1 {
|
||||
return errors.Newf("sort compaction should handle exactly one segment, but got %d segments, planID = %d",
|
||||
// The plan is produced by datacoord, so a malformed plan is an internal
|
||||
// protocol violation, not user input.
|
||||
return merr.WrapErrServiceInternalMsg("sort compaction should handle exactly one segment, but got %d segments, planID = %d",
|
||||
len(t.plan.GetSegmentBinlogs()), t.GetPlanID())
|
||||
}
|
||||
|
||||
|
||||
@@ -27,7 +27,6 @@ import (
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/cockroachdb/errors"
|
||||
"github.com/tidwall/gjson"
|
||||
clientv3 "go.etcd.io/etcd/client/v3"
|
||||
"go.uber.org/zap"
|
||||
@@ -49,6 +48,7 @@ import (
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/conc"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/expr"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/lifetime"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/metricsinfo"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/typeutil"
|
||||
@@ -166,7 +166,7 @@ func (node *DataNode) Register() error {
|
||||
func (node *DataNode) initSession() error {
|
||||
node.session = sessionutil.NewSession(node.ctx)
|
||||
if node.session == nil {
|
||||
return errors.New("failed to initialize session")
|
||||
return merr.WrapErrServiceInternalMsg("failed to initialize session")
|
||||
}
|
||||
node.session.Init(typeutil.DataNodeRole, node.address, false)
|
||||
sessionutil.SaveServerInfo(typeutil.DataNodeRole, node.session.ServerID)
|
||||
@@ -282,7 +282,7 @@ func (node *DataNode) isHealthy() bool {
|
||||
// ReadyToFlush tells whether DataNode is ready for flushing
|
||||
func (node *DataNode) ReadyToFlush() error {
|
||||
if !node.isHealthy() {
|
||||
return errors.New("DataNode not in HEALTHY state")
|
||||
return merr.Wrap(merr.ErrServiceNotReady, "DataNode not in HEALTHY state")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
+25
-24
@@ -20,6 +20,7 @@ import (
|
||||
"github.com/milvus-io/milvus/internal/util/function/embedding"
|
||||
"github.com/milvus-io/milvus/pkg/v3/log"
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/typeutil"
|
||||
)
|
||||
|
||||
@@ -65,11 +66,11 @@ func ExecuteFunctionsForSegment(
|
||||
inputManifestPath, err := packed.CreateSegmentManifestWithBasePath(
|
||||
ctx, basePath, format, sourceColumns, fragments, storageConfig)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("create input manifest: %w", err)
|
||||
return "", merr.Wrap(err, "create input manifest")
|
||||
}
|
||||
_, inputVersion, err := packed.UnmarshalManifestPath(inputManifestPath)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("parse input manifest path: %w", err)
|
||||
return "", merr.Wrap(err, "parse input manifest path")
|
||||
}
|
||||
|
||||
outputFields, outputSchema, err := buildOutputSchema(schema)
|
||||
@@ -94,7 +95,7 @@ func ExecuteFunctionsForSegment(
|
||||
colGroups := []storagecommon.ColumnGroup{{Columns: lo.Range(len(outputFields))}}
|
||||
writer, err := packed.NewFFIPackedWriter(basePath, outputArrow, colGroups, storageConfig, nil)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("open output writer: %w", err)
|
||||
return "", merr.Wrap(err, "open output writer")
|
||||
}
|
||||
writer.AsNewColumnGroups()
|
||||
|
||||
@@ -108,7 +109,7 @@ func ExecuteFunctionsForSegment(
|
||||
|
||||
output, err := writer.Close()
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("close output writer: %w", err)
|
||||
return "", merr.Wrap(err, "close output writer")
|
||||
}
|
||||
if output != nil {
|
||||
defer output.Destroy()
|
||||
@@ -120,7 +121,7 @@ func ExecuteFunctionsForSegment(
|
||||
}
|
||||
manifestPath, err := packed.CommitManifestUpdates(basePath, inputVersion, storageConfig, updates)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("commit function output manifest: %w", err)
|
||||
return "", merr.Wrap(err, "commit function output manifest")
|
||||
}
|
||||
|
||||
log.Info("function execution completed",
|
||||
@@ -139,7 +140,7 @@ func buildOutputSchema(schema *schemapb.CollectionSchema) ([]*schemapb.FieldSche
|
||||
return nil, nil, err
|
||||
}
|
||||
if len(outputFields) == 0 {
|
||||
return nil, nil, fmt.Errorf("no function output fields; executor should not have been invoked")
|
||||
return nil, nil, merr.WrapErrServiceInternalMsg("no function output fields; executor should not have been invoked")
|
||||
}
|
||||
return outputFields, &schemapb.CollectionSchema{
|
||||
Name: schema.GetName(),
|
||||
@@ -177,7 +178,7 @@ func functionOutputFields(schema *schemapb.CollectionSchema) ([]*schemapb.FieldS
|
||||
continue
|
||||
}
|
||||
if _, ok := fieldsByID[fieldID]; !ok {
|
||||
return nil, fmt.Errorf("function output field id %d not found in schema", fieldID)
|
||||
return nil, merr.WrapErrParameterInvalidMsg("function output field id %d not found in schema", fieldID)
|
||||
}
|
||||
addOutputID(fieldID)
|
||||
}
|
||||
@@ -187,7 +188,7 @@ func functionOutputFields(schema *schemapb.CollectionSchema) ([]*schemapb.FieldS
|
||||
}
|
||||
field, ok := fieldsByName[fieldName]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("function output field %s not found in schema", fieldName)
|
||||
return nil, merr.WrapErrParameterInvalidMsg("function output field %s not found in schema", fieldName)
|
||||
}
|
||||
addOutputID(field.GetFieldID())
|
||||
}
|
||||
@@ -222,7 +223,7 @@ func openInputReader(
|
||||
}),
|
||||
)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("open input manifest: %w", err)
|
||||
return nil, merr.Wrap(err, "open input manifest")
|
||||
}
|
||||
return reader, nil
|
||||
}
|
||||
@@ -236,7 +237,7 @@ func buildFunctionExecutionSchema(
|
||||
schema *schemapb.CollectionSchema,
|
||||
) (*schemapb.CollectionSchema, *schemapb.CollectionSchema, typeutil.Set[int64], error) {
|
||||
if schema == nil {
|
||||
return nil, nil, nil, fmt.Errorf("collection schema is nil")
|
||||
return nil, nil, nil, merr.WrapErrParameterInvalidMsg("collection schema is nil")
|
||||
}
|
||||
inputIDs := make(map[int64]struct{})
|
||||
for _, fn := range schema.GetFunctions() {
|
||||
@@ -291,24 +292,24 @@ func buildFunctionExecutionSchema(
|
||||
inputSchema.Fields = append(inputSchema.Fields, f)
|
||||
|
||||
if schema.GetExternalSource() != "" && f.GetExternalField() == "" {
|
||||
return nil, nil, nil, fmt.Errorf("function input field %s has no external_field", f.GetName())
|
||||
return nil, nil, nil, merr.WrapErrParameterInvalidMsg("function input field %s has no external_field", f.GetName())
|
||||
}
|
||||
}
|
||||
|
||||
for inputID := range inputIDs {
|
||||
if _, ok := seenInputFields[inputID]; !ok {
|
||||
if _, ok := seenExecutionFields[inputID]; !ok {
|
||||
return nil, nil, nil, fmt.Errorf("function input field id %d not found in schema", inputID)
|
||||
return nil, nil, nil, merr.WrapErrParameterInvalidMsg("function input field id %d not found in schema", inputID)
|
||||
}
|
||||
}
|
||||
}
|
||||
for outputID := range outputIDs {
|
||||
if _, ok := seenExecutionFields[outputID]; !ok {
|
||||
return nil, nil, nil, fmt.Errorf("function output field id %d not found in schema", outputID)
|
||||
return nil, nil, nil, merr.WrapErrParameterInvalidMsg("function output field id %d not found in schema", outputID)
|
||||
}
|
||||
}
|
||||
if len(inputSchema.GetFields()) == 0 {
|
||||
return nil, nil, nil, fmt.Errorf("no source input columns for function execution")
|
||||
return nil, nil, nil, merr.WrapErrParameterInvalidMsg("no source input columns for function execution")
|
||||
}
|
||||
return inputSchema, executionSchema, requiredInputFields, nil
|
||||
}
|
||||
@@ -332,7 +333,7 @@ func streamBatches(
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
return totalRows, fmt.Errorf("read input batch: %w", err)
|
||||
return totalRows, merr.Wrap(err, "read input batch")
|
||||
}
|
||||
if rec == nil {
|
||||
break
|
||||
@@ -341,7 +342,7 @@ func streamBatches(
|
||||
batch, err := storage.RecordToInsertData(rec, executionSchema, requiredInputFields)
|
||||
rec.Release()
|
||||
if err != nil {
|
||||
return totalRows, fmt.Errorf("record to InsertData: %w", err)
|
||||
return totalRows, merr.Wrap(err, "record to InsertData")
|
||||
}
|
||||
if batch.GetRowNum() == 0 {
|
||||
continue
|
||||
@@ -351,7 +352,7 @@ func streamBatches(
|
||||
ClusterID: clusterID,
|
||||
DBName: schema.GetDbName(),
|
||||
}); err != nil {
|
||||
return totalRows, fmt.Errorf("execute functions: %w", err)
|
||||
return totalRows, merr.Wrap(err, "execute functions")
|
||||
}
|
||||
|
||||
if err := accumulateBM25Stats(batch, bm25Acc); err != nil {
|
||||
@@ -359,7 +360,7 @@ func streamBatches(
|
||||
}
|
||||
|
||||
if err := writeOutputBatch(batch, outputSchema, outputArrow, writer); err != nil {
|
||||
return totalRows, fmt.Errorf("write output batch: %w", err)
|
||||
return totalRows, merr.Wrap(err, "write output batch")
|
||||
}
|
||||
totalRows += int64(batch.GetRowNum())
|
||||
}
|
||||
@@ -404,13 +405,13 @@ func accumulateBM25Stats(batch *storage.InsertData, acc map[int64]*storage.BM25S
|
||||
for outID, stats := range acc {
|
||||
raw, present := batch.Data[outID]
|
||||
if !present || raw == nil {
|
||||
return fmt.Errorf(
|
||||
return merr.WrapErrFunctionFailedMsg(
|
||||
"BM25 output field %d missing from batch; executeFunctions did not populate it",
|
||||
outID)
|
||||
}
|
||||
fd, ok := raw.(*storage.SparseFloatVectorFieldData)
|
||||
if !ok {
|
||||
return fmt.Errorf(
|
||||
return merr.WrapErrFunctionFailedMsg(
|
||||
"BM25 output field %d has wrong type %T (want *SparseFloatVectorFieldData)",
|
||||
outID, raw)
|
||||
}
|
||||
@@ -431,18 +432,18 @@ func appendBM25Stats(
|
||||
}
|
||||
log := log.Ctx(ctx)
|
||||
if updates == nil {
|
||||
return fmt.Errorf("manifest updates is nil")
|
||||
return merr.WrapErrServiceInternalMsg("manifest updates is nil")
|
||||
}
|
||||
|
||||
entries := make([]packed.StatEntry, 0, len(acc))
|
||||
for outID, stats := range acc {
|
||||
blob, err := stats.Serialize()
|
||||
if err != nil {
|
||||
return fmt.Errorf("serialize bm25 stats for field %d: %w", outID, err)
|
||||
return merr.Wrapf(err, "serialize bm25 stats for field %d", outID)
|
||||
}
|
||||
fullPath := path.Join(basePath, fmt.Sprintf("_stats/bm25.%d/%d", outID, 0))
|
||||
if err := packed.WriteFile(storageConfig, fullPath, blob); err != nil {
|
||||
return fmt.Errorf("write bm25 stats file %s: %w", fullPath, err)
|
||||
return merr.Wrapf(err, "write bm25 stats file %s", fullPath)
|
||||
}
|
||||
entries = append(entries, packed.StatEntry{
|
||||
Key: fmt.Sprintf("bm25.%d", outID),
|
||||
@@ -481,7 +482,7 @@ func finalizeBM25Stats(
|
||||
|
||||
basePath, version, err := packed.UnmarshalManifestPath(manifestPath)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("parse manifest path: %w", err)
|
||||
return "", merr.Wrap(err, "parse manifest path")
|
||||
}
|
||||
|
||||
updates := &packed.ManifestUpdates{}
|
||||
|
||||
+3
-1
@@ -29,6 +29,7 @@ import (
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/datapb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/conc"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
)
|
||||
|
||||
// TaskKey uniquely identifies an external collection task.
|
||||
@@ -259,7 +260,8 @@ func (m *ExternalCollectionManager) SubmitTask(
|
||||
zap.ByteString("stack", stack))
|
||||
reason := fmt.Sprintf("task panicked: %v", r)
|
||||
m.UpdateResult(clusterID, taskID, indexpb.JobState_JobStateFailed, reason, info.KeptSegments, nil)
|
||||
retErr = fmt.Errorf("%s", reason)
|
||||
// A recovered panic is a server-side failure, never caller input.
|
||||
retErr = merr.WrapErrServiceInternalMsg("%s", reason)
|
||||
}
|
||||
}()
|
||||
log.Info("executing external collection task in pool",
|
||||
|
||||
+19
-19
@@ -164,22 +164,22 @@ func (t *RefreshExternalCollectionTask) PreExecute(ctx context.Context) error {
|
||||
zap.Int64("collectionID", t.req.GetCollectionID()))
|
||||
|
||||
if t.req == nil {
|
||||
return fmt.Errorf("request is nil")
|
||||
return merr.WrapErrParameterInvalidMsg("request is nil")
|
||||
}
|
||||
if t.req.GetSchema() == nil {
|
||||
return fmt.Errorf("schema is nil in request")
|
||||
return merr.WrapErrParameterInvalidMsg("schema is nil in request")
|
||||
}
|
||||
if t.req.GetStorageConfig() == nil {
|
||||
return fmt.Errorf("storage config is nil in request")
|
||||
return merr.WrapErrParameterInvalidMsg("storage config is nil in request")
|
||||
}
|
||||
if t.req.GetExternalSource() == "" {
|
||||
return fmt.Errorf("external source is empty in request")
|
||||
return merr.WrapErrParameterInvalidMsg("external source is empty in request")
|
||||
}
|
||||
|
||||
// Parse and cache external spec for reuse during Execute
|
||||
spec, err := externalspec.ParseExternalSpec(t.req.GetExternalSpec())
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to parse external spec: %w", err)
|
||||
return merr.Wrap(err, "failed to parse external spec")
|
||||
}
|
||||
t.parsedSpec = spec
|
||||
t.columns = packed.GetColumnNamesFromSchema(t.req.GetSchema())
|
||||
@@ -198,7 +198,7 @@ func (t *RefreshExternalCollectionTask) Execute(ctx context.Context) error {
|
||||
|
||||
// Initialize pre-allocated segment IDs from request
|
||||
if t.req.GetPreAllocatedSegmentIds() == nil {
|
||||
return fmt.Errorf("pre-allocated segment IDs not provided in request")
|
||||
return merr.WrapErrParameterInvalidMsg("pre-allocated segment IDs not provided in request")
|
||||
}
|
||||
|
||||
t.preallocatedIDRange = t.req.GetPreAllocatedSegmentIds()
|
||||
@@ -212,13 +212,13 @@ func (t *RefreshExternalCollectionTask) Execute(ctx context.Context) error {
|
||||
// Fetch fragments from external source
|
||||
newFragments, err := t.fetchFragmentsFromExternalSource(ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to fetch fragments: %w", err)
|
||||
return merr.Wrap(err, "failed to fetch fragments")
|
||||
}
|
||||
|
||||
// Build current segment -> fragments mapping
|
||||
currentSegmentFragments, err := t.buildCurrentSegmentFragments()
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to build current segment fragments: %w", err)
|
||||
return merr.Wrap(err, "failed to build current segment fragments")
|
||||
}
|
||||
|
||||
// Compare and organize segments
|
||||
@@ -235,7 +235,7 @@ func (t *RefreshExternalCollectionTask) Execute(ctx context.Context) error {
|
||||
func (t *RefreshExternalCollectionTask) fetchFragmentsFromExternalSource(ctx context.Context) ([]packed.Fragment, error) {
|
||||
manifestPath := t.req.GetExploreManifestPath()
|
||||
if manifestPath == "" {
|
||||
return nil, fmt.Errorf("explore manifest path is required but not provided")
|
||||
return nil, merr.WrapErrParameterMissingMsg("explore manifest path is required but not provided")
|
||||
}
|
||||
|
||||
log.Ctx(ctx).Info("reading file list from explore manifest",
|
||||
@@ -329,7 +329,7 @@ func (t *RefreshExternalCollectionTask) organizeSegments(
|
||||
var err error
|
||||
outputColumns, err = functionOutputColumnNames(t.req.GetSchema())
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("resolve function output columns: %w", err)
|
||||
return nil, merr.Wrap(err, "resolve function output columns")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -455,7 +455,7 @@ func (t *RefreshExternalCollectionTask) segmentHasFunctionOutputColumns(seg *dat
|
||||
}
|
||||
hasColumns, err := packed.ManifestHasColumns(seg.GetManifestPath(), t.req.GetStorageConfig(), outputColumns)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("check function output columns for segment %d: %w", seg.GetID(), err)
|
||||
return false, merr.Wrapf(err, "check function output columns for segment %d", seg.GetID())
|
||||
}
|
||||
return hasColumns, nil
|
||||
}
|
||||
@@ -557,7 +557,7 @@ func (t *RefreshExternalCollectionTask) patchSegmentForMissingColumns(
|
||||
t.req.GetStorageConfig(),
|
||||
)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to append manifest columns for segment %d: %w", seg.GetID(), err)
|
||||
return nil, merr.Wrapf(err, "failed to append manifest columns for segment %d", seg.GetID())
|
||||
}
|
||||
|
||||
sampleRows := paramtable.Get().QueryNodeCfg.ExternalCollectionSampleRows.GetAsInt()
|
||||
@@ -574,7 +574,7 @@ func (t *RefreshExternalCollectionTask) patchSegmentForMissingColumns(
|
||||
t.req.GetStorageConfig(),
|
||||
)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to sample external field sizes for segment %d: %w", seg.GetID(), err)
|
||||
return nil, merr.Wrapf(err, "failed to sample external field sizes for segment %d", seg.GetID())
|
||||
}
|
||||
externalAvgBytes := sumFieldSizes(fieldSizes, schema)
|
||||
if externalAvgBytes <= 0 {
|
||||
@@ -690,7 +690,7 @@ func (t *RefreshExternalCollectionTask) balanceFragmentsToSegments(ctx context.C
|
||||
}
|
||||
// Each segment needs 2 IDs: one for segment, one for fake binlog logID
|
||||
if t.nextAllocID+1 >= t.preallocatedIDRange.End {
|
||||
return nil, fmt.Errorf("insufficient pre-allocated IDs: need 2 more but only have %d IDs in range [%d, %d)",
|
||||
return nil, merr.WrapErrParameterInvalidMsg("insufficient pre-allocated IDs: need 2 more but only have %d IDs in range [%d, %d)",
|
||||
t.preallocatedIDRange.End-t.nextAllocID,
|
||||
t.preallocatedIDRange.Begin,
|
||||
t.preallocatedIDRange.End)
|
||||
@@ -739,7 +739,7 @@ func (t *RefreshExternalCollectionTask) balanceFragmentsToSegments(ctx context.C
|
||||
manifestPath, err = t.createManifestForSegment(ctx, work.segmentID, work.fragments)
|
||||
}
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to create manifest for segment %d: %w", work.segmentID, err)
|
||||
return "", merr.Wrapf(err, "failed to create manifest for segment %d", work.segmentID)
|
||||
}
|
||||
return manifestPath, nil
|
||||
})
|
||||
@@ -834,7 +834,7 @@ func (t *RefreshExternalCollectionTask) balanceFragmentsToSegments(ctx context.C
|
||||
log.Warn("external field size sample produced non-positive total",
|
||||
zap.String("manifestPath", manifestPath),
|
||||
zap.Int64("total", total))
|
||||
recordErr(fmt.Errorf("sampled field sizes sum to %d (schema may have no external_field mappings, or sampled rows are empty)", total))
|
||||
recordErr(merr.WrapErrParameterInvalidMsg("sampled field sizes sum to %d (schema may have no external_field mappings, or sampled rows are empty)", total))
|
||||
return 0, false
|
||||
}
|
||||
return total, true
|
||||
@@ -882,9 +882,9 @@ func (t *RefreshExternalCollectionTask) balanceFragmentsToSegments(ctx context.C
|
||||
if strings.Contains(rootCause, "not found in schema") {
|
||||
hint = "; check external_field mappings in collection schema against actual parquet columns"
|
||||
}
|
||||
return nil, merr.WrapErrParameterInvalidMsg(fmt.Sprintf(
|
||||
return nil, merr.WrapErrParameterInvalidMsg(
|
||||
"external field size sampling failed for all %d segment(s): %s%s",
|
||||
len(manifestPaths), rootCause, hint))
|
||||
len(manifestPaths), rootCause, hint)
|
||||
}
|
||||
// Fill any zero slots (sampling failed mid-loop) with the first
|
||||
// successful average so every segment gets a non-zero MemorySize.
|
||||
@@ -1032,7 +1032,7 @@ func estimateFunctionOutputBytesPerRow(schema *schemapb.CollectionSchema) (int64
|
||||
Fields: []*schemapb.FieldSchema{field},
|
||||
})
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("estimate function output field %s: %w", field.GetName(), err)
|
||||
return 0, merr.Wrapf(err, "estimate function output field %s", field.GetName())
|
||||
}
|
||||
total += int64(size)
|
||||
}
|
||||
|
||||
@@ -18,7 +18,6 @@ package importv2
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"path"
|
||||
"strconv"
|
||||
"strings"
|
||||
@@ -34,6 +33,7 @@ import (
|
||||
"github.com/milvus-io/milvus/pkg/v3/log"
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/datapb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/metautil"
|
||||
)
|
||||
|
||||
@@ -114,12 +114,12 @@ func transformManifestPath(
|
||||
) (string, error) {
|
||||
basePath, version, err := packed.UnmarshalManifestPath(manifestPath)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to unmarshal manifest path: %w", err)
|
||||
return "", merr.Wrap(err, "failed to unmarshal manifest path")
|
||||
}
|
||||
|
||||
targetBasePath, err := generateTargetPath(basePath, source, target)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to generate target base path: %w", err)
|
||||
return "", merr.Wrap(err, "failed to generate target base path")
|
||||
}
|
||||
|
||||
targetManifestPath := packed.MarshalManifestPath(targetBasePath, version)
|
||||
@@ -227,18 +227,18 @@ func collectSegmentFiles(
|
||||
// StorageV3+: binlog paths MUST come from manifest
|
||||
manifestPath := source.GetManifestPath()
|
||||
if manifestPath == "" {
|
||||
return nil, fmt.Errorf("storage_version=%d requires manifest_path but it is empty (segmentID=%d)",
|
||||
return nil, merr.WrapErrParameterInvalidMsg("storage_version=%d requires manifest_path but it is empty (segmentID=%d)",
|
||||
source.GetStorageVersion(), source.GetSegmentId())
|
||||
}
|
||||
|
||||
basePath, _, err := packed.UnmarshalManifestPath(manifestPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to unmarshal manifest path %q for segment %d: %w", manifestPath, source.GetSegmentId(), err)
|
||||
return nil, merr.Wrapf(err, "failed to unmarshal manifest path %q for segment %d", manifestPath, source.GetSegmentId())
|
||||
}
|
||||
|
||||
allFiles, listErr := listAllFiles(ctx, cm, basePath)
|
||||
if listErr != nil {
|
||||
return nil, fmt.Errorf("failed to list files from manifest base path %q for segment %d: %w", basePath, source.GetSegmentId(), listErr)
|
||||
return nil, merr.Wrapf(listErr, "failed to list files from manifest base path %q for segment %d", basePath, source.GetSegmentId())
|
||||
}
|
||||
|
||||
// Empty file list is OK for V3 — segment may have only deltas and no insert binlogs
|
||||
@@ -319,7 +319,7 @@ func generateMappingsFromFiles(
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to generate target path for %s file %s: %w", fileType, srcPath, err)
|
||||
return merr.Wrapf(err, "failed to generate target path for %s file %s", fileType, srcPath)
|
||||
}
|
||||
mappings[srcPath] = dstPath
|
||||
}
|
||||
@@ -379,13 +379,13 @@ func CopySegmentAndIndexFiles(
|
||||
// Step 1: Collect all files to copy
|
||||
files, err := collectSegmentFiles(ctx, cm, source)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("failed to collect segment files: %w", err)
|
||||
return nil, nil, merr.Wrap(err, "failed to collect segment files")
|
||||
}
|
||||
|
||||
// Step 2: Generate src->dst mappings for file copying
|
||||
mappings, err := generateMappingsFromFiles(files, source, target)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("failed to generate file mappings: %w", err)
|
||||
return nil, nil, merr.Wrap(err, "failed to generate file mappings")
|
||||
}
|
||||
|
||||
// Step 3: Execute all copy operations
|
||||
@@ -400,7 +400,7 @@ func CopySegmentAndIndexFiles(
|
||||
fields = append(fields, logFields...)
|
||||
fields = append(fields, zap.String("src", src), zap.String("dst", dst), zap.Error(err))
|
||||
log.Warn("failed to copy file", fields...)
|
||||
return nil, copiedFiles, fmt.Errorf("failed to copy file from %s to %s: %w", src, dst, err)
|
||||
return nil, copiedFiles, merr.Wrapf(err, "failed to copy file from %s to %s", src, dst)
|
||||
}
|
||||
copiedFiles = append(copiedFiles, dst)
|
||||
}
|
||||
@@ -418,7 +418,7 @@ func CopySegmentAndIndexFiles(
|
||||
if _, exists := mappings[srcPath]; !exists {
|
||||
dstPath, pathErr := generateTargetPath(srcPath, source, target)
|
||||
if pathErr != nil {
|
||||
return nil, copiedFiles, fmt.Errorf("failed to generate target path for pb insert binlog %s: %w", srcPath, pathErr)
|
||||
return nil, copiedFiles, merr.Wrapf(pathErr, "failed to generate target path for pb insert binlog %s", srcPath)
|
||||
}
|
||||
mappings[srcPath] = dstPath
|
||||
}
|
||||
@@ -430,20 +430,20 @@ func CopySegmentAndIndexFiles(
|
||||
// Step 4: Build index metadata from source
|
||||
indexInfos, textIndexInfos, jsonKeyIndexInfos, err := buildIndexInfoFromSource(source, target, mappings)
|
||||
if err != nil {
|
||||
return nil, copiedFiles, fmt.Errorf("failed to build index info: %w", err)
|
||||
return nil, copiedFiles, merr.Wrap(err, "failed to build index info")
|
||||
}
|
||||
|
||||
// Step 5: Generate segment metadata with path mappings
|
||||
segmentInfo, err := generateSegmentInfoFromSource(source, target, mappings)
|
||||
if err != nil {
|
||||
return nil, copiedFiles, fmt.Errorf("failed to generate segment info: %w", err)
|
||||
return nil, copiedFiles, merr.Wrap(err, "failed to generate segment info")
|
||||
}
|
||||
|
||||
// Step 6: Compress paths
|
||||
err = binlog.CompressBinLogs(segmentInfo.GetBinlogs(), segmentInfo.GetStatslogs(),
|
||||
segmentInfo.GetDeltalogs(), segmentInfo.GetBm25Logs())
|
||||
if err != nil {
|
||||
return nil, copiedFiles, fmt.Errorf("failed to compress binlog paths: %w", err)
|
||||
return nil, copiedFiles, merr.Wrap(err, "failed to compress binlog paths")
|
||||
}
|
||||
|
||||
for _, indexInfo := range indexInfos {
|
||||
@@ -474,7 +474,7 @@ func CopySegmentAndIndexFiles(
|
||||
if useManifest {
|
||||
targetManifestPath, err := transformManifestPath(source.GetManifestPath(), source, target)
|
||||
if err != nil {
|
||||
return nil, copiedFiles, fmt.Errorf("failed to transform manifest path: %w", err)
|
||||
return nil, copiedFiles, merr.Wrap(err, "failed to transform manifest path")
|
||||
}
|
||||
result.ManifestPath = targetManifestPath
|
||||
}
|
||||
@@ -526,7 +526,7 @@ func transformFieldBinlogs(
|
||||
}
|
||||
dstPath, ok := mappings[srcPath]
|
||||
if !ok {
|
||||
return nil, 0, fmt.Errorf("no mapping found for source path: %s", srcPath)
|
||||
return nil, 0, merr.WrapErrServiceInternalMsg("no mapping found for source path: %s", srcPath)
|
||||
}
|
||||
dstBinlog.LogPath = dstPath
|
||||
}
|
||||
@@ -583,7 +583,7 @@ func generateSegmentInfoFromSource(
|
||||
// Process insert binlogs (count rows)
|
||||
binlogs, totalRows, err := transformFieldBinlogs(source.GetInsertBinlogs(), mappings, true, source.GetIsExternalCollection())
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to transform insert binlogs: %w", err)
|
||||
return nil, merr.Wrap(err, "failed to transform insert binlogs")
|
||||
}
|
||||
segmentInfo.Binlogs = binlogs
|
||||
segmentInfo.ImportedRows = totalRows
|
||||
@@ -591,21 +591,21 @@ func generateSegmentInfoFromSource(
|
||||
// Process stats binlogs (no row counting)
|
||||
statslogs, _, err := transformFieldBinlogs(source.GetStatsBinlogs(), mappings, false, false)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to transform stats binlogs: %w", err)
|
||||
return nil, merr.Wrap(err, "failed to transform stats binlogs")
|
||||
}
|
||||
segmentInfo.Statslogs = statslogs
|
||||
|
||||
// Process delta binlogs (no row counting)
|
||||
deltalogs, _, err := transformFieldBinlogs(source.GetDeltaBinlogs(), mappings, false, false)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to transform delta binlogs: %w", err)
|
||||
return nil, merr.Wrap(err, "failed to transform delta binlogs")
|
||||
}
|
||||
segmentInfo.Deltalogs = deltalogs
|
||||
|
||||
// Process BM25 binlogs (no row counting)
|
||||
bm25logs, _, err := transformFieldBinlogs(source.GetBm25Binlogs(), mappings, false, false)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to transform BM25 binlogs: %w", err)
|
||||
return nil, merr.Wrap(err, "failed to transform BM25 binlogs")
|
||||
}
|
||||
segmentInfo.Bm25Logs = bm25logs
|
||||
|
||||
@@ -635,7 +635,7 @@ func generateTargetPath(sourcePath string, source *datapb.CopySegmentSource, tar
|
||||
}
|
||||
|
||||
if logTypeIndex == -1 || logTypeIndex+3 >= len(parts) {
|
||||
return "", fmt.Errorf("invalid binlog path structure: %s (expected log_type at a valid position)", sourcePath)
|
||||
return "", merr.WrapErrParameterInvalidMsg("invalid binlog path structure: %s (expected log_type at a valid position)", sourcePath)
|
||||
}
|
||||
|
||||
// Replace IDs in order: collectionID, partitionID, segmentID
|
||||
@@ -667,7 +667,7 @@ func generateTargetLOBPath(sourcePath string, source *datapb.CopySegmentSource,
|
||||
// Path: .../{insert_log}/{coll}/{part}/lobs/...
|
||||
// Need at least logTypeIndex + 2 (coll and part) after insert_log
|
||||
if logTypeIndex == -1 || logTypeIndex+2 >= len(parts) {
|
||||
return "", fmt.Errorf("invalid LOB path structure: %s", sourcePath)
|
||||
return "", merr.WrapErrParameterInvalidMsg("invalid LOB path structure: %s", sourcePath)
|
||||
}
|
||||
|
||||
parts[logTypeIndex+1] = strconv.FormatInt(target.GetCollectionId(), 10)
|
||||
@@ -709,7 +709,7 @@ func buildIndexInfoFromSource(
|
||||
for _, srcPath := range srcIndex.GetIndexFilePaths() {
|
||||
targetPath, ok := mappings[srcPath]
|
||||
if !ok {
|
||||
return nil, nil, nil, fmt.Errorf("no mapping found for index file: %s", srcPath)
|
||||
return nil, nil, nil, merr.WrapErrServiceInternalMsg("no mapping found for index file: %s", srcPath)
|
||||
}
|
||||
targetPaths = append(targetPaths, targetPath)
|
||||
}
|
||||
@@ -753,7 +753,7 @@ func buildIndexInfoFromSource(
|
||||
for _, srcFile := range srcText.GetFiles() {
|
||||
targetFile, ok := mappings[srcFile]
|
||||
if !ok {
|
||||
return nil, nil, nil, fmt.Errorf("no mapping found for text index file: %s", srcFile)
|
||||
return nil, nil, nil, merr.WrapErrServiceInternalMsg("no mapping found for text index file: %s", srcFile)
|
||||
}
|
||||
targetFiles = append(targetFiles, targetFile)
|
||||
}
|
||||
@@ -785,7 +785,7 @@ func buildIndexInfoFromSource(
|
||||
for _, srcFile := range srcJSON.GetFiles() {
|
||||
targetFile, ok := mappings[srcFile]
|
||||
if !ok {
|
||||
return nil, nil, nil, fmt.Errorf("no mapping found for JSON index file: %s", srcFile)
|
||||
return nil, nil, nil, merr.WrapErrServiceInternalMsg("no mapping found for JSON index file: %s", srcFile)
|
||||
}
|
||||
targetFiles = append(targetFiles, targetFile)
|
||||
}
|
||||
@@ -896,7 +896,7 @@ func generateTargetIndexPath(
|
||||
}
|
||||
|
||||
if keywordIdx == -1 {
|
||||
return "", fmt.Errorf("keyword '%s' not found in path: %s", keyword, sourcePath)
|
||||
return "", merr.WrapErrServiceInternalMsg("keyword '%s' not found in path: %s", keyword, sourcePath)
|
||||
}
|
||||
|
||||
// Set offsets based on index type
|
||||
@@ -928,13 +928,13 @@ func generateTargetIndexPath(
|
||||
segmentOffset = 6
|
||||
buildIDOffset = 2
|
||||
default:
|
||||
return "", fmt.Errorf("unsupported index type: %s (expected '%s', '%s', '%s', or '%s')",
|
||||
return "", merr.WrapErrParameterInvalidMsg("unsupported index type: %s (expected '%s', '%s', '%s', or '%s')",
|
||||
indexType, IndexTypeVectorScalarV0, IndexTypeText, IndexTypeJSONKey, IndexTypeJSONStats)
|
||||
}
|
||||
|
||||
// Validate path structure has enough components
|
||||
if keywordIdx+segmentOffset >= len(parts) {
|
||||
return "", fmt.Errorf("invalid %s path structure: %s (expected '%s' with at least %d components after it)",
|
||||
return "", merr.WrapErrParameterInvalidMsg("invalid %s path structure: %s (expected '%s' with at least %d components after it)",
|
||||
indexType, sourcePath, indexType, segmentOffset+1)
|
||||
}
|
||||
|
||||
|
||||
@@ -22,7 +22,6 @@ import (
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/cockroachdb/errors"
|
||||
"go.uber.org/zap"
|
||||
|
||||
"github.com/milvus-io/milvus-proto/go-api/v3/schemapb"
|
||||
@@ -30,6 +29,7 @@ import (
|
||||
"github.com/milvus-io/milvus/pkg/v3/log"
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/datapb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/conc"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/typeutil"
|
||||
)
|
||||
|
||||
@@ -382,7 +382,7 @@ func (t *CopySegmentTask) copySingleSegment(source *datapb.CopySegmentSource, ta
|
||||
reason := "no insert/delete binlogs for segment"
|
||||
log.Error(reason, logFields...)
|
||||
t.manager.Update(t.GetTaskID(), UpdateState(datapb.ImportTaskStateV2_Failed), UpdateReason(reason))
|
||||
return nil, errors.New(reason)
|
||||
return nil, merr.WrapErrParameterInvalidMsg(reason)
|
||||
}
|
||||
|
||||
// Step 2: Copy all segment files (binlogs + indexes) together
|
||||
|
||||
@@ -35,6 +35,7 @@ import (
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/datapb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/internalpb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/conc"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/typeutil"
|
||||
)
|
||||
@@ -192,9 +193,9 @@ func (t *PreImportTask) readFileStat(reader importutilv2.Reader, fileIdx int) er
|
||||
}
|
||||
maxSize := paramtable.Get().DataNodeCfg.MaxImportFileSizeInGB.GetAsFloat() * 1024 * 1024 * 1024
|
||||
if fileSize > int64(maxSize) {
|
||||
return errors.New(fmt.Sprintf(
|
||||
return merr.WrapErrParameterInvalidMsg(
|
||||
"The import file size has reached the maximum limit allowed for importing, "+
|
||||
"fileSize=%d, maxSize=%d", fileSize, int64(maxSize)))
|
||||
"fileSize=%d, maxSize=%d", fileSize, int64(maxSize))
|
||||
}
|
||||
|
||||
totalRows := 0
|
||||
|
||||
@@ -50,7 +50,7 @@ import (
|
||||
)
|
||||
|
||||
func WrapTaskNotFoundError(taskID int64) error {
|
||||
return merr.WrapErrImportFailed(fmt.Sprintf("cannot find import task with id %d", taskID))
|
||||
return merr.WrapErrImportSysFailedMsg("cannot find import task with id %d", taskID)
|
||||
}
|
||||
|
||||
func NewSyncTask(ctx context.Context,
|
||||
@@ -151,7 +151,7 @@ func PickSegment(segments []*datapb.ImportRequestSegment, vchannel string, parti
|
||||
})
|
||||
|
||||
if len(candidates) == 0 {
|
||||
return 0, fmt.Errorf("no candidate segments found for channel %s and partition %d",
|
||||
return 0, merr.WrapErrServiceInternalMsg("no candidate segments found for channel %s and partition %d",
|
||||
vchannel, partitionID)
|
||||
}
|
||||
|
||||
@@ -392,7 +392,7 @@ func AppendNullableDefaultFieldsData(schema *schemapb.CollectionSchema, data *st
|
||||
}
|
||||
}
|
||||
default:
|
||||
return fmt.Errorf("unexpected data type: %d, cannot be filled with default value", dataType)
|
||||
return merr.WrapErrServiceInternalMsg("unexpected data type: %d, cannot be filled with default value", dataType)
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
@@ -409,7 +409,7 @@ func FillDynamicData(schema *schemapb.CollectionSchema, data *storage.InsertData
|
||||
}
|
||||
dynamicField := typeutil.GetDynamicField(schema)
|
||||
if dynamicField == nil {
|
||||
return merr.WrapErrImportFailed("collection schema is illegal, enable_dynamic_field is true but the dynamic field doesn't exist")
|
||||
return merr.WrapErrImportSysFailed("collection schema is illegal, enable_dynamic_field is true but the dynamic field doesn't exist")
|
||||
}
|
||||
|
||||
tempData, ok := data.Data[dynamicField.GetFieldID()]
|
||||
|
||||
@@ -80,7 +80,7 @@ func (queue *IndexTaskQueue) addUnissuedTask(t Task) error {
|
||||
defer queue.utLock.Unlock()
|
||||
|
||||
if queue.utFull() {
|
||||
return errors.New("index task queue is full")
|
||||
return merr.Wrap(merr.ErrServiceResourceInsufficient, "index task queue is full")
|
||||
}
|
||||
queue.unissuedTasks.PushBack(t)
|
||||
select {
|
||||
|
||||
@@ -52,6 +52,7 @@ import (
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/indexpb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/workerpb"
|
||||
_ "github.com/milvus-io/milvus/pkg/v3/util/funcutil"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/metautil"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/timerecord"
|
||||
@@ -292,7 +293,7 @@ func (st *statsTask) sort(ctx context.Context) ([]*datapb.FieldBinlog, error) {
|
||||
newManifest, err := packed.AddStatsToManifest(
|
||||
st.manifestPath, st.req.GetStorageConfig(), statEntries)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to add stats to manifest: %w", err)
|
||||
return nil, merr.Wrap(err, "failed to add stats to manifest")
|
||||
}
|
||||
st.manifestPath = newManifest
|
||||
}
|
||||
@@ -496,7 +497,7 @@ func (st *statsTask) createTextIndex(ctx context.Context,
|
||||
}
|
||||
binlogs, ok := fieldBinlogs[fieldID]
|
||||
if !ok && !enableNull {
|
||||
return nil, fmt.Errorf("field binlog not found for field %d", fieldID)
|
||||
return nil, merr.WrapErrServiceInternalMsg("field binlog not found for field %d", fieldID)
|
||||
}
|
||||
result := make([]string, 0, len(binlogs))
|
||||
for _, binlog := range binlogs {
|
||||
@@ -618,7 +619,7 @@ func (st *statsTask) createTextIndex(ctx context.Context,
|
||||
newManifest, err := packed.AddStatsToManifest(
|
||||
st.manifestPath, st.req.GetStorageConfig(), statEntries)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to add text index stats to manifest: %w", err)
|
||||
return merr.Wrap(err, "failed to add text index stats to manifest")
|
||||
}
|
||||
st.manifestPath = newManifest
|
||||
// Dual-write: keep textIndexLogs populated so it is stored in segment metadata as a
|
||||
@@ -685,7 +686,7 @@ func (st *statsTask) createJSONKeyStats(ctx context.Context,
|
||||
}
|
||||
binlogs, ok := fieldBinlogs[fieldID]
|
||||
if !ok && !enableNull {
|
||||
return nil, fmt.Errorf("field binlog not found for field %d", fieldID)
|
||||
return nil, merr.WrapErrServiceInternalMsg("field binlog not found for field %d", fieldID)
|
||||
}
|
||||
result := make([]string, 0, len(binlogs))
|
||||
for _, binlog := range binlogs {
|
||||
@@ -796,7 +797,7 @@ func (st *statsTask) createJSONKeyStats(ctx context.Context,
|
||||
newManifest, err := packed.AddStatsToManifest(
|
||||
st.manifestPath, st.req.GetStorageConfig(), statEntries)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to add JSON key stats to manifest: %w", err)
|
||||
return merr.Wrap(err, "failed to add JSON key stats to manifest")
|
||||
}
|
||||
st.manifestPath = newManifest
|
||||
// Dual-write: keep jsonKeyIndexStats populated so it is stored in segment metadata as a
|
||||
@@ -829,7 +830,7 @@ func computeStatsBasePath(req *workerpb.CreateStatsRequest, manifestPath string,
|
||||
if req.GetStorageVersion() == storage.StorageV3 {
|
||||
basePath, _, err := packed.UnmarshalManifestPath(manifestPath)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to unmarshal manifest path for %s basePath: %w", statsType, err)
|
||||
return "", merr.Wrapf(err, "failed to unmarshal manifest path for %s basePath", statsType)
|
||||
}
|
||||
return fmt.Sprintf("%s/_stats/%s.%d", basePath, statsType, fieldID), nil
|
||||
}
|
||||
@@ -845,7 +846,7 @@ func computeStatsBasePath(req *workerpb.CreateStatsRequest, manifestPath string,
|
||||
req.GetTaskID(), req.GetTaskVersion(),
|
||||
req.GetCollectionID(), req.GetPartitionID(), req.GetTargetSegmentID(), fieldID), nil
|
||||
}
|
||||
return "", fmt.Errorf("unknown stats type: %s", statsType)
|
||||
return "", merr.WrapErrParameterInvalidMsg("unknown stats type: %s", statsType)
|
||||
}
|
||||
|
||||
func buildIndexParams(
|
||||
|
||||
@@ -21,7 +21,6 @@ import (
|
||||
"fmt"
|
||||
"strconv"
|
||||
|
||||
"github.com/cockroachdb/errors"
|
||||
"go.opentelemetry.io/otel"
|
||||
"go.opentelemetry.io/otel/attribute"
|
||||
"go.opentelemetry.io/otel/trace"
|
||||
@@ -82,7 +81,9 @@ func (node *DataNode) CreateJob(ctx context.Context, req *workerpb.CreateJobRequ
|
||||
Cancel: taskCancel,
|
||||
State: commonpb.IndexState_InProgress,
|
||||
}); oldInfo != nil {
|
||||
err := merr.WrapErrIndexDuplicate(req.GetIndexName(), "building index task existed")
|
||||
// Node-internal dedup of a coordinator-dispatched build task: a duplicate
|
||||
// is a scheduling race, not the user re-creating an index.
|
||||
err := merr.WrapErrServiceInternalMsg("building index task existed, index=%s, buildID=%d", req.GetIndexName(), req.GetBuildID())
|
||||
log.Warn("duplicated index build task", zap.Error(err))
|
||||
metrics.DataNodeBuildIndexTaskCounter.WithLabelValues(paramtable.GetStringNodeID(), metrics.FailLabel).Inc()
|
||||
return merr.Status(err), nil
|
||||
@@ -254,7 +255,7 @@ func (node *DataNode) CreateJobV2(ctx context.Context, req *workerpb.CreateJobV2
|
||||
return node.createStatsTask(ctx, statsRequest)
|
||||
default:
|
||||
log.Warn("DataNode receive unknown type job")
|
||||
return merr.Status(fmt.Errorf("DataNode receive unknown type job with TaskID: %d", req.GetTaskID())), nil
|
||||
return merr.Status(merr.WrapErrServiceInternalMsg("DataNode receive unknown type job with TaskID: %d", req.GetTaskID())), nil
|
||||
}
|
||||
}
|
||||
|
||||
@@ -454,7 +455,7 @@ func (node *DataNode) QueryJobsV2(ctx context.Context, req *workerpb.QueryJobsV2
|
||||
default:
|
||||
log.Warn("DataNode receive querying unknown type jobs")
|
||||
return &workerpb.QueryJobsV2Response{
|
||||
Status: merr.Status(errors.New("DataNode receive querying unknown type jobs")),
|
||||
Status: merr.Status(merr.WrapErrServiceInternalMsg("DataNode receive querying unknown type jobs")),
|
||||
}, nil
|
||||
}
|
||||
}
|
||||
@@ -479,7 +480,7 @@ func (node *DataNode) queryIndexTask(ctx context.Context, req *workerpb.QueryJob
|
||||
log.Debug("query index jobs result success", zap.Any("results", results))
|
||||
if len(results) == 0 {
|
||||
return &workerpb.QueryJobsV2Response{
|
||||
Status: merr.Status(fmt.Errorf("tasks '%v' not found", req.GetTaskIDs())),
|
||||
Status: merr.Status(merr.WrapErrServiceInternalMsg("tasks '%v' not found", req.GetTaskIDs())),
|
||||
}, nil
|
||||
}
|
||||
return &workerpb.QueryJobsV2Response{
|
||||
@@ -508,7 +509,7 @@ func (node *DataNode) queryStatsTask(ctx context.Context, req *workerpb.QueryJob
|
||||
log.Debug("query stats job result success", zap.Any("results", results))
|
||||
if len(results) == 0 {
|
||||
return &workerpb.QueryJobsV2Response{
|
||||
Status: merr.Status(fmt.Errorf("tasks '%v' not found", req.GetTaskIDs())),
|
||||
Status: merr.Status(merr.WrapErrServiceInternalMsg("tasks '%v' not found", req.GetTaskIDs())),
|
||||
}, nil
|
||||
}
|
||||
return &workerpb.QueryJobsV2Response{
|
||||
@@ -542,7 +543,7 @@ func (node *DataNode) queryAnalyzeTask(ctx context.Context, req *workerpb.QueryJ
|
||||
log.Debug("query analyze jobs result success", zap.Any("results", results))
|
||||
if len(results) == 0 {
|
||||
return &workerpb.QueryJobsV2Response{
|
||||
Status: merr.Status(fmt.Errorf("tasks '%v' not found", req.GetTaskIDs())),
|
||||
Status: merr.Status(merr.WrapErrServiceInternalMsg("tasks '%v' not found", req.GetTaskIDs())),
|
||||
}, nil
|
||||
}
|
||||
return &workerpb.QueryJobsV2Response{
|
||||
@@ -613,6 +614,6 @@ func (node *DataNode) DropJobsV2(ctx context.Context, req *workerpb.DropJobsV2Re
|
||||
return merr.Success(), nil
|
||||
default:
|
||||
log.Warn("DataNode receive dropping unknown type jobs")
|
||||
return merr.Status(errors.New("DataNode receive dropping unknown type jobs")), nil
|
||||
return merr.Status(merr.WrapErrServiceInternalMsg("DataNode receive dropping unknown type jobs")), nil
|
||||
}
|
||||
}
|
||||
|
||||
@@ -194,11 +194,11 @@ func (node *DataNode) CompactionV2(ctx context.Context, req *datapb.CompactionPl
|
||||
}
|
||||
|
||||
if req.GetBeginLogID() == 0 {
|
||||
return merr.Status(merr.WrapErrParameterInvalidMsg("invalid beginLogID")), nil
|
||||
return merr.Status(merr.WrapErrServiceInternalMsg("invalid beginLogID")), nil
|
||||
}
|
||||
|
||||
if req.GetPreAllocatedLogIDs().GetBegin() == 0 || req.GetPreAllocatedLogIDs().GetEnd() == 0 {
|
||||
return merr.Status(merr.WrapErrParameterInvalidMsg(fmt.Sprintf("invalid beginID %d or invalid endID %d", req.GetPreAllocatedLogIDs().GetBegin(), req.GetPreAllocatedLogIDs().GetEnd()))), nil
|
||||
return merr.Status(merr.WrapErrServiceInternalMsg(fmt.Sprintf("invalid beginID %d or invalid endID %d", req.GetPreAllocatedLogIDs().GetBegin(), req.GetPreAllocatedLogIDs().GetEnd()))), nil
|
||||
}
|
||||
|
||||
/*
|
||||
@@ -233,7 +233,7 @@ func (node *DataNode) CompactionV2(ctx context.Context, req *datapb.CompactionPl
|
||||
)
|
||||
case datapb.CompactionType_MixCompaction:
|
||||
if req.GetPreAllocatedSegmentIDs() == nil || req.GetPreAllocatedSegmentIDs().GetBegin() == 0 {
|
||||
return merr.Status(merr.WrapErrParameterInvalidMsg("invalid pre-allocated segmentID range")), nil
|
||||
return merr.Status(merr.WrapErrServiceInternalMsg("invalid pre-allocated segmentID range")), nil
|
||||
}
|
||||
pk, err := typeutil.GetPrimaryFieldSchema(req.GetSchema())
|
||||
if err != nil {
|
||||
@@ -256,7 +256,7 @@ func (node *DataNode) CompactionV2(ctx context.Context, req *datapb.CompactionPl
|
||||
)
|
||||
case datapb.CompactionType_ClusteringCompaction:
|
||||
if req.GetPreAllocatedSegmentIDs() == nil || req.GetPreAllocatedSegmentIDs().GetBegin() == 0 {
|
||||
return merr.Status(merr.WrapErrParameterInvalidMsg("invalid pre-allocated segmentID range")), nil
|
||||
return merr.Status(merr.WrapErrServiceInternalMsg("invalid pre-allocated segmentID range")), nil
|
||||
}
|
||||
if namespaceEnabled {
|
||||
var sortFields []int64
|
||||
@@ -281,7 +281,7 @@ func (node *DataNode) CompactionV2(ctx context.Context, req *datapb.CompactionPl
|
||||
}
|
||||
case datapb.CompactionType_SortCompaction:
|
||||
if req.GetPreAllocatedSegmentIDs() == nil || req.GetPreAllocatedSegmentIDs().GetBegin() == 0 {
|
||||
return merr.Status(merr.WrapErrParameterInvalidMsg("invalid pre-allocated segmentID range")), nil
|
||||
return merr.Status(merr.WrapErrServiceInternalMsg("invalid pre-allocated segmentID range")), nil
|
||||
}
|
||||
pk, err := typeutil.GetPrimaryFieldSchema(req.GetSchema())
|
||||
if err != nil {
|
||||
@@ -306,7 +306,7 @@ func (node *DataNode) CompactionV2(ctx context.Context, req *datapb.CompactionPl
|
||||
task = compactor.NewBumpSchemaVersionCompactionTask(taskCtx, cm, req, compactionParams)
|
||||
default:
|
||||
log.Warn("Unknown compaction type", zap.String("type", req.GetType().String()))
|
||||
return merr.Status(merr.WrapErrParameterInvalidMsg("Unknown compaction type: %v", req.GetType().String())), nil
|
||||
return merr.Status(merr.WrapErrServiceInternalMsg("Unknown compaction type: %v", req.GetType().String())), nil
|
||||
}
|
||||
|
||||
succeed, err := node.compactionExecutor.Enqueue(task)
|
||||
@@ -790,7 +790,7 @@ func (node *DataNode) CreateTask(ctx context.Context, request *workerpb.CreateTa
|
||||
}
|
||||
return node.CopySegment(ctx, req)
|
||||
default:
|
||||
err := fmt.Errorf("unrecognized task type '%s', properties=%v", taskType, request.GetProperties())
|
||||
err := merr.WrapErrServiceInternalMsg("unrecognized task type '%s', properties=%v", taskType, request.GetProperties())
|
||||
log.Ctx(ctx).Warn("CreateTask failed", zap.Error(err))
|
||||
return merr.Status(err), nil
|
||||
}
|
||||
@@ -807,7 +807,7 @@ func wrapQueryTaskResult[Resp proto.Message](resp Resp, properties taskcommon.Pr
|
||||
}
|
||||
statusResp, ok := any(resp).(ResponseWithStatus)
|
||||
if !ok {
|
||||
return &workerpb.QueryTaskResponse{Status: merr.Status(fmt.Errorf("response does not implement GetStatus"))}, nil
|
||||
return &workerpb.QueryTaskResponse{Status: merr.Status(merr.WrapErrServiceInternalMsg("response does not implement GetStatus"))}, nil
|
||||
}
|
||||
return &workerpb.QueryTaskResponse{
|
||||
Status: statusResp.GetStatus(),
|
||||
@@ -937,7 +937,7 @@ func (node *DataNode) QueryTask(ctx context.Context, request *workerpb.QueryTask
|
||||
resProperties.AppendReason(resp.GetReason())
|
||||
return wrapQueryTaskResult(resp, resProperties)
|
||||
default:
|
||||
err := fmt.Errorf("unrecognized task type '%s', properties=%v", taskType, request.GetProperties())
|
||||
err := merr.WrapErrServiceInternalMsg("unrecognized task type '%s', properties=%v", taskType, request.GetProperties())
|
||||
log.Ctx(ctx).Warn("QueryTask failed", zap.Error(err))
|
||||
return &workerpb.QueryTaskResponse{
|
||||
Status: merr.Status(err),
|
||||
@@ -997,7 +997,7 @@ func (node *DataNode) DropTask(ctx context.Context, request *workerpb.DropTaskRe
|
||||
zap.String("clusterID", clusterID))
|
||||
return merr.Success(), nil
|
||||
default:
|
||||
err := fmt.Errorf("unrecognized task type '%s', properties=%v", taskType, request.GetProperties())
|
||||
err := merr.WrapErrServiceInternalMsg("unrecognized task type '%s', properties=%v", taskType, request.GetProperties())
|
||||
log.Ctx(ctx).Warn("DropTask failed", zap.Error(err))
|
||||
return merr.Status(err), nil
|
||||
}
|
||||
|
||||
@@ -594,7 +594,10 @@ func (s *DataNodeServicesSuite) TestCreateTask() {
|
||||
}
|
||||
status, err := s.node.CreateTask(s.ctx, req)
|
||||
s.NoError(err)
|
||||
s.Equal(commonpb.ErrorCode_UnexpectedError, status.GetErrorCode())
|
||||
// taskcommon.GetTaskType classifies an unrecognized task type as
|
||||
// ServiceInternal: task types are coordinator-assigned, so a mismatch is
|
||||
// an internal protocol violation, not user input.
|
||||
s.Equal(merr.Code(merr.ErrServiceInternal), status.GetCode())
|
||||
})
|
||||
}
|
||||
|
||||
@@ -685,7 +688,10 @@ func (s *DataNodeServicesSuite) TestQueryTask() {
|
||||
}
|
||||
resp, err := s.node.QueryTask(s.ctx, req)
|
||||
s.NoError(err)
|
||||
s.Equal(commonpb.ErrorCode_UnexpectedError, resp.GetStatus().GetErrorCode())
|
||||
// taskcommon.GetTaskType classifies an unrecognized task type as
|
||||
// ServiceInternal: task types are coordinator-assigned, so a mismatch is
|
||||
// an internal protocol violation, not user input.
|
||||
s.Equal(merr.Code(merr.ErrServiceInternal), resp.GetStatus().GetCode())
|
||||
})
|
||||
}
|
||||
|
||||
@@ -771,7 +777,10 @@ func (s *DataNodeServicesSuite) TestDropTask() {
|
||||
}
|
||||
status, err := s.node.DropTask(s.ctx, req)
|
||||
s.NoError(err)
|
||||
s.Equal(commonpb.ErrorCode_UnexpectedError, status.GetErrorCode())
|
||||
// taskcommon.GetTaskType classifies an unrecognized task type as
|
||||
// ServiceInternal: task types are coordinator-assigned, so a mismatch is
|
||||
// an internal protocol violation, not user input.
|
||||
s.Equal(merr.Code(merr.ErrServiceInternal), status.GetCode())
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
@@ -20,7 +20,6 @@ import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"github.com/cockroachdb/errors"
|
||||
"go.uber.org/zap"
|
||||
"google.golang.org/grpc"
|
||||
|
||||
@@ -36,6 +35,7 @@ import (
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/workerpb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/commonpbutil"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/funcutil"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/typeutil"
|
||||
)
|
||||
@@ -57,11 +57,11 @@ type Client struct {
|
||||
// NewClient creates a client for DataNode.
|
||||
func NewClient(ctx context.Context, addr string, serverID int64, encryption bool) (types.DataNodeClient, error) {
|
||||
if addr == "" {
|
||||
return nil, errors.New("address is empty")
|
||||
return nil, merr.WrapErrParameterInvalidMsg("address is empty")
|
||||
}
|
||||
sess := sessionutil.NewSession(context.Background())
|
||||
if sess == nil {
|
||||
err := errors.New("new session error, maybe can not connect to etcd")
|
||||
err := merr.WrapErrServiceUnavailable("new session error, maybe can not connect to etcd")
|
||||
log.Ctx(ctx).Debug("DataNodeClient New Etcd Session failed", zap.Error(err))
|
||||
return nil, err
|
||||
}
|
||||
|
||||
@@ -69,7 +69,7 @@ type Client struct {
|
||||
func NewClient(ctx context.Context) (types.MixCoordClient, error) {
|
||||
sess := sessionutil.NewSession(context.Background())
|
||||
if sess == nil {
|
||||
err := errors.New("new session error, maybe can not connect to etcd")
|
||||
err := merr.WrapErrServiceUnavailable("new session error, maybe can not connect to etcd")
|
||||
log.Ctx(ctx).Debug("New MixCoord Client failed", zap.Error(err))
|
||||
return nil, err
|
||||
}
|
||||
@@ -120,7 +120,7 @@ func (c *Client) getMixCoordAddr() (string, error) {
|
||||
return c.getCompatibleMixCoordAddr()
|
||||
} else {
|
||||
log.Warn("MixCoordClient mess key not exist", zap.Any("key", key))
|
||||
return "", errors.New("find no available mixcoord, check mixcoord state")
|
||||
return "", merr.WrapErrNodeNotFound(0, "find no available mixcoord, check mixcoord state")
|
||||
}
|
||||
}
|
||||
log.Debug("MixCoordClient GetSessions success",
|
||||
@@ -137,12 +137,12 @@ func (c *Client) getCompatibleMixCoordAddr() (string, error) {
|
||||
msess, _, err := c.sess.GetSessions(c.ctx, typeutil.RootCoordRole)
|
||||
if err != nil {
|
||||
log.Debug("mixCoordClient getSessions failed", zap.Any("key", typeutil.RootCoordRole), zap.Error(err))
|
||||
return "", errors.New("find no available mixcoord, check mixcoord state")
|
||||
return "", merr.WrapErrNodeNotFound(0, "find no available mixcoord, check mixcoord state")
|
||||
}
|
||||
ms, ok := msess[typeutil.RootCoordRole]
|
||||
if !ok {
|
||||
log.Warn("MixCoordClient mess key not exist", zap.Any("key", typeutil.RootCoordRole))
|
||||
return "", errors.New("find no available mixcoord, check mixcoord state")
|
||||
return "", merr.WrapErrNodeNotFound(0, "find no available mixcoord, check mixcoord state")
|
||||
}
|
||||
log.Debug("MixCoordClient GetSessions use rootCoord", zap.Any("key", typeutil.RootCoordRole))
|
||||
c.grpcClient.SetNodeID(ms.ServerID)
|
||||
|
||||
@@ -20,7 +20,6 @@ import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"github.com/cockroachdb/errors"
|
||||
"go.uber.org/zap"
|
||||
"google.golang.org/grpc"
|
||||
|
||||
@@ -35,6 +34,7 @@ import (
|
||||
"github.com/milvus-io/milvus/pkg/v3/proto/proxypb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/commonpbutil"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/funcutil"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/typeutil"
|
||||
)
|
||||
@@ -50,11 +50,11 @@ type Client struct {
|
||||
// NewClient creates a new client instance
|
||||
func NewClient(ctx context.Context, addr string, nodeID int64) (types.ProxyClient, error) {
|
||||
if addr == "" {
|
||||
return nil, errors.New("address is empty")
|
||||
return nil, merr.WrapErrParameterInvalidMsg("address is empty")
|
||||
}
|
||||
sess := sessionutil.NewSession(context.Background())
|
||||
if sess == nil {
|
||||
err := errors.New("new session error, maybe can not connect to etcd")
|
||||
err := merr.WrapErrServiceUnavailable("new session error, maybe can not connect to etcd")
|
||||
log.Ctx(ctx).Debug("Proxy client new session failed", zap.Error(err))
|
||||
return nil, err
|
||||
}
|
||||
|
||||
@@ -17,13 +17,12 @@
|
||||
package httpserver
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/gin-gonic/gin"
|
||||
"google.golang.org/protobuf/proto"
|
||||
|
||||
"github.com/milvus-io/milvus-proto/go-api/v3/milvuspb"
|
||||
"github.com/milvus-io/milvus/internal/types"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
)
|
||||
|
||||
// Handlers handles http requests
|
||||
@@ -107,7 +106,7 @@ func (h *Handlers) handleDummy(c *gin.Context) (interface{}, error) {
|
||||
// use ShouldBind to supports binding JSON, XML, YAML, and protobuf.
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.Dummy(c, &req)
|
||||
}
|
||||
@@ -116,11 +115,11 @@ func (h *Handlers) handleCreateCollection(c *gin.Context) (interface{}, error) {
|
||||
wrappedReq := WrappedCreateCollectionRequest{}
|
||||
err := shouldBind(c, &wrappedReq)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
schemaProto, err := proto.Marshal(&wrappedReq.Schema)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: marshal schema failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "marshal schema failed")
|
||||
}
|
||||
req := &milvuspb.CreateCollectionRequest{
|
||||
Base: wrappedReq.Base,
|
||||
@@ -138,7 +137,7 @@ func (h *Handlers) handleDropCollection(c *gin.Context) (interface{}, error) {
|
||||
req := milvuspb.DropCollectionRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.DropCollection(c, &req)
|
||||
}
|
||||
@@ -147,7 +146,7 @@ func (h *Handlers) handleHasCollection(c *gin.Context) (interface{}, error) {
|
||||
req := milvuspb.HasCollectionRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.HasCollection(c, &req)
|
||||
}
|
||||
@@ -156,7 +155,7 @@ func (h *Handlers) handleDescribeCollection(c *gin.Context) (interface{}, error)
|
||||
req := milvuspb.DescribeCollectionRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.DescribeCollection(c, &req)
|
||||
}
|
||||
@@ -165,7 +164,7 @@ func (h *Handlers) handleLoadCollection(c *gin.Context) (interface{}, error) {
|
||||
req := milvuspb.LoadCollectionRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.LoadCollection(c, &req)
|
||||
}
|
||||
@@ -174,7 +173,7 @@ func (h *Handlers) handleReleaseCollection(c *gin.Context) (interface{}, error)
|
||||
req := milvuspb.ReleaseCollectionRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.ReleaseCollection(c, &req)
|
||||
}
|
||||
@@ -183,7 +182,7 @@ func (h *Handlers) handleGetCollectionStatistics(c *gin.Context) (interface{}, e
|
||||
req := milvuspb.GetCollectionStatisticsRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.GetCollectionStatistics(c, &req)
|
||||
}
|
||||
@@ -192,7 +191,7 @@ func (h *Handlers) handleShowCollections(c *gin.Context) (interface{}, error) {
|
||||
req := milvuspb.ShowCollectionsRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.ShowCollections(c, &req)
|
||||
}
|
||||
@@ -201,7 +200,7 @@ func (h *Handlers) handleCreatePartition(c *gin.Context) (interface{}, error) {
|
||||
req := milvuspb.CreatePartitionRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.CreatePartition(c, &req)
|
||||
}
|
||||
@@ -210,7 +209,7 @@ func (h *Handlers) handleDropPartition(c *gin.Context) (interface{}, error) {
|
||||
req := milvuspb.DropPartitionRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.DropPartition(c, &req)
|
||||
}
|
||||
@@ -219,7 +218,7 @@ func (h *Handlers) handleHasPartition(c *gin.Context) (interface{}, error) {
|
||||
req := milvuspb.HasPartitionRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.HasPartition(c, &req)
|
||||
}
|
||||
@@ -228,7 +227,7 @@ func (h *Handlers) handleLoadPartitions(c *gin.Context) (interface{}, error) {
|
||||
req := milvuspb.LoadPartitionsRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.LoadPartitions(c, &req)
|
||||
}
|
||||
@@ -237,7 +236,7 @@ func (h *Handlers) handleReleasePartitions(c *gin.Context) (interface{}, error)
|
||||
req := milvuspb.ReleasePartitionsRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.ReleasePartitions(c, &req)
|
||||
}
|
||||
@@ -246,7 +245,7 @@ func (h *Handlers) handleGetPartitionStatistics(c *gin.Context) (interface{}, er
|
||||
req := milvuspb.GetPartitionStatisticsRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.GetPartitionStatistics(c, &req)
|
||||
}
|
||||
@@ -255,7 +254,7 @@ func (h *Handlers) handleShowPartitions(c *gin.Context) (interface{}, error) {
|
||||
req := milvuspb.ShowPartitionsRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.ShowPartitions(c, &req)
|
||||
}
|
||||
@@ -264,7 +263,7 @@ func (h *Handlers) handleCreateAlias(c *gin.Context) (interface{}, error) {
|
||||
req := milvuspb.CreateAliasRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.CreateAlias(c, &req)
|
||||
}
|
||||
@@ -273,7 +272,7 @@ func (h *Handlers) handleDropAlias(c *gin.Context) (interface{}, error) {
|
||||
req := milvuspb.DropAliasRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.DropAlias(c, &req)
|
||||
}
|
||||
@@ -282,7 +281,7 @@ func (h *Handlers) handleAlterAlias(c *gin.Context) (interface{}, error) {
|
||||
req := milvuspb.AlterAliasRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.AlterAlias(c, &req)
|
||||
}
|
||||
@@ -291,7 +290,7 @@ func (h *Handlers) handleCreateIndex(c *gin.Context) (interface{}, error) {
|
||||
req := milvuspb.CreateIndexRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.CreateIndex(c, &req)
|
||||
}
|
||||
@@ -300,7 +299,7 @@ func (h *Handlers) handleDescribeIndex(c *gin.Context) (interface{}, error) {
|
||||
req := milvuspb.DescribeIndexRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.DescribeIndex(c, &req)
|
||||
}
|
||||
@@ -309,7 +308,7 @@ func (h *Handlers) handleGetIndexState(c *gin.Context) (interface{}, error) {
|
||||
req := milvuspb.GetIndexStateRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.GetIndexState(c, &req)
|
||||
}
|
||||
@@ -318,7 +317,7 @@ func (h *Handlers) handleGetIndexBuildProgress(c *gin.Context) (interface{}, err
|
||||
req := milvuspb.GetIndexBuildProgressRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.GetIndexBuildProgress(c, &req)
|
||||
}
|
||||
@@ -327,7 +326,7 @@ func (h *Handlers) handleDropIndex(c *gin.Context) (interface{}, error) {
|
||||
req := milvuspb.DropIndexRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.DropIndex(c, &req)
|
||||
}
|
||||
@@ -336,11 +335,11 @@ func (h *Handlers) handleInsert(c *gin.Context) (interface{}, error) {
|
||||
wrappedReq := WrappedInsertRequest{}
|
||||
err := shouldBind(c, &wrappedReq)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
req, err := wrappedReq.AsInsertRequest()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: convert body to pb failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "convert body to pb failed")
|
||||
}
|
||||
return h.proxy.Insert(c, req)
|
||||
}
|
||||
@@ -349,7 +348,7 @@ func (h *Handlers) handleDelete(c *gin.Context) (interface{}, error) {
|
||||
req := milvuspb.DeleteRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.Delete(c, &req)
|
||||
}
|
||||
@@ -358,10 +357,10 @@ func (h *Handlers) handleSearch(c *gin.Context) (interface{}, error) {
|
||||
wrappedReq := SearchRequest{}
|
||||
err := shouldBind(c, &wrappedReq)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
if wrappedReq.HasSearchAggregation() {
|
||||
return nil, fmt.Errorf("%w: searchAggregation is not supported for low-level REST search", errBadRequest)
|
||||
return nil, badRequestf(merr.WrapErrParameterInvalidMsg("searchAggregation is not supported for low-level REST search"), "invalid request")
|
||||
}
|
||||
req := milvuspb.SearchRequest{
|
||||
Base: wrappedReq.Base,
|
||||
@@ -392,7 +391,7 @@ func (h *Handlers) handleQuery(c *gin.Context) (interface{}, error) {
|
||||
req := milvuspb.QueryRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.Query(c, &req)
|
||||
}
|
||||
@@ -401,7 +400,7 @@ func (h *Handlers) handleFlush(c *gin.Context) (interface{}, error) {
|
||||
req := milvuspb.FlushRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.Flush(c, &req)
|
||||
}
|
||||
@@ -410,7 +409,7 @@ func (h *Handlers) handleCalcDistance(c *gin.Context) (interface{}, error) {
|
||||
wrappedReq := WrappedCalcDistanceRequest{}
|
||||
err := shouldBind(c, &wrappedReq)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
|
||||
req := milvuspb.CalcDistanceRequest{
|
||||
@@ -426,7 +425,7 @@ func (h *Handlers) handleGetFlushState(c *gin.Context) (interface{}, error) {
|
||||
req := milvuspb.GetFlushStateRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.GetFlushState(c, &req)
|
||||
}
|
||||
@@ -435,7 +434,7 @@ func (h *Handlers) handleGetPersistentSegmentInfo(c *gin.Context) (interface{},
|
||||
req := milvuspb.GetPersistentSegmentInfoRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.GetPersistentSegmentInfo(c, &req)
|
||||
}
|
||||
@@ -444,7 +443,7 @@ func (h *Handlers) handleGetQuerySegmentInfo(c *gin.Context) (interface{}, error
|
||||
req := milvuspb.GetQuerySegmentInfoRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.GetQuerySegmentInfo(c, &req)
|
||||
}
|
||||
@@ -453,7 +452,7 @@ func (h *Handlers) handleGetReplicas(c *gin.Context) (interface{}, error) {
|
||||
req := milvuspb.GetReplicasRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.GetReplicas(c, &req)
|
||||
}
|
||||
@@ -462,7 +461,7 @@ func (h *Handlers) handleGetMetrics(c *gin.Context) (interface{}, error) {
|
||||
req := milvuspb.GetMetricsRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.GetMetrics(c, &req)
|
||||
}
|
||||
@@ -471,7 +470,7 @@ func (h *Handlers) handleLoadBalance(c *gin.Context) (interface{}, error) {
|
||||
req := milvuspb.LoadBalanceRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.LoadBalance(c, &req)
|
||||
}
|
||||
@@ -480,7 +479,7 @@ func (h *Handlers) handleGetCompactionState(c *gin.Context) (interface{}, error)
|
||||
req := milvuspb.GetCompactionStateRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.GetCompactionState(c, &req)
|
||||
}
|
||||
@@ -489,7 +488,7 @@ func (h *Handlers) handleGetCompactionStateWithPlans(c *gin.Context) (interface{
|
||||
req := milvuspb.GetCompactionPlansRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.GetCompactionStateWithPlans(c, &req)
|
||||
}
|
||||
@@ -498,7 +497,7 @@ func (h *Handlers) handleManualCompaction(c *gin.Context) (interface{}, error) {
|
||||
req := milvuspb.ManualCompactionRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.ManualCompaction(c, &req)
|
||||
}
|
||||
@@ -507,7 +506,7 @@ func (h *Handlers) handleImport(c *gin.Context) (interface{}, error) {
|
||||
req := milvuspb.ImportRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.Import(c, &req)
|
||||
}
|
||||
@@ -516,7 +515,7 @@ func (h *Handlers) handleGetImportState(c *gin.Context) (interface{}, error) {
|
||||
req := milvuspb.GetImportStateRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.GetImportState(c, &req)
|
||||
}
|
||||
@@ -525,7 +524,7 @@ func (h *Handlers) handleListImportTasks(c *gin.Context) (interface{}, error) {
|
||||
req := milvuspb.ListImportTasksRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.ListImportTasks(c, &req)
|
||||
}
|
||||
@@ -534,7 +533,7 @@ func (h *Handlers) handleCreateCredential(c *gin.Context) (interface{}, error) {
|
||||
req := milvuspb.CreateCredentialRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.CreateCredential(c, &req)
|
||||
}
|
||||
@@ -543,7 +542,7 @@ func (h *Handlers) handleUpdateCredential(c *gin.Context) (interface{}, error) {
|
||||
req := milvuspb.UpdateCredentialRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.UpdateCredential(c, &req)
|
||||
}
|
||||
@@ -552,7 +551,7 @@ func (h *Handlers) handleDeleteCredential(c *gin.Context) (interface{}, error) {
|
||||
req := milvuspb.DeleteCredentialRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.DeleteCredential(c, &req)
|
||||
}
|
||||
@@ -561,7 +560,7 @@ func (h *Handlers) handleListCredUsers(c *gin.Context) (interface{}, error) {
|
||||
req := milvuspb.ListCredUsersRequest{}
|
||||
err := shouldBind(c, &req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: parse body failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "parse body failed")
|
||||
}
|
||||
return h.proxy.ListCredUsers(c, &req)
|
||||
}
|
||||
|
||||
@@ -1434,7 +1434,7 @@ func TestFp16Bf16VectorsV1(t *testing.T) {
|
||||
assert.Nil(t, err, "case %d: ", i)
|
||||
assert.Equal(t, testcase.errCode, returnBody.Code, "case %d: ", i, string(testcase.requestBody))
|
||||
if testcase.errCode != 0 {
|
||||
assert.Equal(t, testcase.errMsg, returnBody.Message, "case %d: ", i, string(testcase.requestBody))
|
||||
assert.Contains(t, returnBody.Message, testcase.errMsg, "case %d: ", i, string(testcase.requestBody))
|
||||
}
|
||||
fmt.Println(w.Body.String())
|
||||
})
|
||||
|
||||
@@ -410,6 +410,35 @@ func wrapperPost(newReq newReqFunc, v2 handlerFuncV2) gin.HandlerFunc {
|
||||
dbName,
|
||||
collectionName,
|
||||
).Inc()
|
||||
|
||||
// Mirror the fail_input/fail_system metric split into the logs so a
|
||||
// failed REST request can be filtered by error_type. System failures are
|
||||
// logged at Warn (actionable); input failures at Info (expected user
|
||||
// mistakes — keeping them at Warn would spam the logs).
|
||||
if label == metrics.FailSystemLabel || label == metrics.FailInputLabel {
|
||||
var status *commonpb.Status
|
||||
switch r := resp.(type) {
|
||||
case interface{ GetStatus() *commonpb.Status }:
|
||||
status = r.GetStatus()
|
||||
case *commonpb.Status:
|
||||
status = r
|
||||
}
|
||||
errType := merr.SystemError
|
||||
if label == metrics.FailInputLabel {
|
||||
errType = merr.InputError
|
||||
}
|
||||
logger := log.Ctx(ctx).With(
|
||||
zap.String("method", methodTag),
|
||||
zap.String("error_type", errType.String()),
|
||||
zap.Int32("code", status.GetCode()),
|
||||
zap.String("reason", status.GetReason()),
|
||||
)
|
||||
if errType == merr.InputError {
|
||||
logger.Info("restful request returned an input error")
|
||||
} else {
|
||||
logger.Warn("restful request returned a system error")
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -543,6 +572,9 @@ func wrapperProxyWithLimit(ctx context.Context, ginCtx *gin.Context, req any, ch
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
// Expose the exact classification (incl. boundary InputError marks) to
|
||||
// the REST access log; key must match accesslog/info.ContextErrorType.
|
||||
ginCtx.Set("error_type", merr.GetErrorType(err).String())
|
||||
log.Ctx(ctx).Warn("high level restful api, grpc call failed", zap.Error(err))
|
||||
if !ignoreErr {
|
||||
HTTPAbortReturn(ginCtx, http.StatusOK, gin.H{HTTPReturnCode: merr.Code(err), HTTPReturnMessage: err.Error()})
|
||||
@@ -1025,7 +1057,7 @@ func (h *HandlersV2) waitForFlush(ctx context.Context, dbName string, collection
|
||||
segmentIDs := flushResp.GetCollSegIDs()[collectionName].GetData()
|
||||
flushTs, ok := flushResp.GetCollFlushTs()[collectionName]
|
||||
if !ok {
|
||||
return merr.WrapErrServiceInternal(fmt.Sprintf("failed to get flush timestamp for collection %s", collectionName))
|
||||
return merr.WrapErrServiceInternalMsg("failed to get flush timestamp for collection %s", collectionName)
|
||||
}
|
||||
|
||||
ticker := time.NewTicker(500 * time.Millisecond)
|
||||
@@ -1521,7 +1553,7 @@ func generatePlaceholderGroup(ctx context.Context, body string, collSchema *sche
|
||||
fieldName = field.Name
|
||||
vectorField = field
|
||||
} else {
|
||||
return nil, errors.New("search without annsField, but already found multiple vector fields: [" + fieldName + ", " + field.Name + ",,,]")
|
||||
return nil, merr.WrapErrParameterInvalidMsg("search without annsField, but already found multiple vector fields: [%s, %s,,,]", fieldName, field.Name)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1547,7 +1579,7 @@ func generatePlaceholderGroup(ctx context.Context, body string, collSchema *sche
|
||||
}
|
||||
}
|
||||
if vectorField == nil {
|
||||
return nil, errors.New("cannot find a vector field named: " + fieldName)
|
||||
return nil, merr.WrapErrFieldNotFound(fieldName, "cannot find a vector field")
|
||||
}
|
||||
dim := int64(0)
|
||||
if !typeutil.IsSparseFloatVectorType(vectorField.DataType) {
|
||||
|
||||
@@ -3352,13 +3352,13 @@ func TestSearchV2(t *testing.T) {
|
||||
queryTestCases = append(queryTestCases, requestBodyTestCase{
|
||||
path: SearchAction,
|
||||
requestBody: []byte(`{"collectionName": "book", "data": [[0.1, 0.2]], "annsField": "word_count", "filter": "book_id in [2, 4, 6, 8]", "limit": 4, "outputFields": ["word_count"], "params": {"radius":0.9, "range_filter": 0.1}, "groupingField": "test"}`),
|
||||
errMsg: "can only accept json format request, error: cannot find a vector field named: word_count",
|
||||
errMsg: "cannot find a vector field",
|
||||
errCode: 1801,
|
||||
})
|
||||
queryTestCases = append(queryTestCases, requestBodyTestCase{
|
||||
path: AdvancedSearchAction,
|
||||
requestBody: []byte(`{"collectionName": "hello_milvus", "search": [{"data": [[0.1, 0.2]], "annsField": "float_vector1", "metricType": "L2", "limit": 3}, {"data": [[0.1, 0.2]], "annsField": "float_vector2", "metricType": "L2", "limit": 3}], "rerank": {"strategy": "rrf", "params": {"k": 1}}}`),
|
||||
errMsg: "can only accept json format request, error: cannot find a vector field named: float_vector1",
|
||||
errMsg: "cannot find a vector field",
|
||||
errCode: 1801,
|
||||
})
|
||||
// multiple annsFields
|
||||
|
||||
@@ -27,7 +27,6 @@ import (
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/cockroachdb/errors"
|
||||
"github.com/gin-gonic/gin"
|
||||
"go.uber.org/zap"
|
||||
|
||||
@@ -93,7 +92,7 @@ func (w *timeoutResponseRecorder) Write(data []byte) (int, error) {
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
if w.closed || w.body == nil {
|
||||
return 0, errors.New("response writer closed")
|
||||
return 0, merr.WrapErrServiceInternalMsg("response writer closed")
|
||||
}
|
||||
if !w.written() {
|
||||
w.size = 0
|
||||
@@ -107,7 +106,7 @@ func (w *timeoutResponseRecorder) WriteString(s string) (int, error) {
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
if w.closed || w.body == nil {
|
||||
return 0, errors.New("response writer closed")
|
||||
return 0, merr.WrapErrServiceInternalMsg("response writer closed")
|
||||
}
|
||||
if !w.written() {
|
||||
w.size = 0
|
||||
@@ -161,7 +160,7 @@ func (w *timeoutResponseRecorder) Written() bool {
|
||||
func (w *timeoutResponseRecorder) Flush() {}
|
||||
|
||||
func (w *timeoutResponseRecorder) Hijack() (net.Conn, *bufio.ReadWriter, error) {
|
||||
return nil, nil, errors.New("response writer does not support hijack")
|
||||
return nil, nil, merr.WrapErrServiceInternalMsg("response writer does not support hijack")
|
||||
}
|
||||
|
||||
func (w *timeoutResponseRecorder) CloseNotify() <-chan bool {
|
||||
@@ -176,7 +175,7 @@ func (w *timeoutResponseRecorder) CommitTo(realWriter gin.ResponseWriter) error
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
if w.closed || w.body == nil {
|
||||
return errors.New("response writer closed")
|
||||
return merr.WrapErrServiceInternalMsg("response writer closed")
|
||||
}
|
||||
|
||||
dst := realWriter.Header()
|
||||
|
||||
@@ -27,7 +27,6 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/cockroachdb/errors"
|
||||
"github.com/gin-gonic/gin"
|
||||
"github.com/spf13/cast"
|
||||
"github.com/tidwall/gjson"
|
||||
@@ -239,7 +238,7 @@ func convertRange(field *schemapb.FieldSchema, result gjson.Result) (string, err
|
||||
func checkGetPrimaryKey(coll *schemapb.CollectionSchema, idResult gjson.Result) (string, error) {
|
||||
primaryField, ok := getPrimaryField(coll)
|
||||
if !ok {
|
||||
return "", fmt.Errorf("collection: %s has no primary field", coll.Name)
|
||||
return "", merr.WrapErrParameterInvalidMsg("collection: %s has no primary field", coll.Name)
|
||||
}
|
||||
resultStr, err := convertRange(primaryField, idResult)
|
||||
if err != nil {
|
||||
@@ -253,7 +252,7 @@ func checkGetPrimaryKey(coll *schemapb.CollectionSchema, idResult gjson.Result)
|
||||
// based on the primary key field type
|
||||
func convertIDsToSchemapbIDs(ids []interface{}, pkField *schemapb.FieldSchema) (*schemapb.IDs, error) {
|
||||
if len(ids) == 0 {
|
||||
return nil, errors.New("ids array cannot be empty")
|
||||
return nil, merr.WrapErrParameterMissingMsg("ids array cannot be empty")
|
||||
}
|
||||
|
||||
switch pkField.DataType {
|
||||
@@ -270,18 +269,18 @@ func convertIDsToSchemapbIDs(ids []interface{}, pkField *schemapb.FieldSchema) (
|
||||
// JSON numbers are decoded as float64
|
||||
// Check if the float has a fractional part
|
||||
if v != math.Trunc(v) {
|
||||
return nil, fmt.Errorf("invalid int64 id at index %d: %v has fractional part", i, v)
|
||||
return nil, merr.WrapErrParameterInvalidMsg("invalid int64 id at index %d: %v has fractional part", i, v)
|
||||
}
|
||||
int64ID = int64(v)
|
||||
case string:
|
||||
// Try to parse string as int64
|
||||
parsed, err := strconv.ParseInt(v, 10, 64)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("invalid int64 id at index %d: %v, error: %v", i, id, err)
|
||||
return nil, merr.WrapErrParameterInvalidErr(err, "invalid int64 id at index %d: %v", i, id)
|
||||
}
|
||||
int64ID = parsed
|
||||
default:
|
||||
return nil, fmt.Errorf("invalid id type at index %d: expected int64, got %T", i, id)
|
||||
return nil, merr.WrapErrParameterInvalidMsg("invalid id type at index %d: expected int64, got %T", i, id)
|
||||
}
|
||||
int64IDs = append(int64IDs, int64ID)
|
||||
}
|
||||
@@ -304,10 +303,10 @@ func convertIDsToSchemapbIDs(ids []interface{}, pkField *schemapb.FieldSchema) (
|
||||
// Convert number to string
|
||||
stringID = fmt.Sprintf("%v", v)
|
||||
default:
|
||||
return nil, fmt.Errorf("invalid id type at index %d: expected string, got %T", i, id)
|
||||
return nil, merr.WrapErrParameterInvalidMsg("invalid id type at index %d: expected string, got %T", i, id)
|
||||
}
|
||||
if stringID == "" {
|
||||
return nil, fmt.Errorf("empty string id at index %d", i)
|
||||
return nil, merr.WrapErrParameterInvalidMsg("empty string id at index %d", i)
|
||||
}
|
||||
stringIDs = append(stringIDs, stringID)
|
||||
}
|
||||
@@ -320,7 +319,7 @@ func convertIDsToSchemapbIDs(ids []interface{}, pkField *schemapb.FieldSchema) (
|
||||
}, nil
|
||||
|
||||
default:
|
||||
return nil, fmt.Errorf("unsupported primary key type: %s", pkField.DataType.String())
|
||||
return nil, merr.WrapErrParameterInvalidMsg("unsupported primary key type: %s", pkField.DataType.String())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -784,7 +783,7 @@ func checkAndSetData(body []byte, collSchema *schemapb.CollectionSchema, partial
|
||||
if !containsString(fieldNames, mapKey) {
|
||||
if collSchema.EnableDynamicField {
|
||||
if mapKey == common.MetaFieldName {
|
||||
return nil, nil, merr.WrapErrParameterInvalidMsg(fmt.Sprintf("use the invalid field name(%s) when enable dynamicField", mapKey))
|
||||
return nil, nil, merr.WrapErrParameterInvalidMsg("use the invalid field name(%s) when enable dynamicField", mapKey)
|
||||
}
|
||||
mapValueStr := mapValue.String()
|
||||
switch mapValue.Type {
|
||||
@@ -1106,7 +1105,7 @@ func decodeByteVectorElement(v gjson.Result, dim, bytesPerVec int64, isFloat16 b
|
||||
return nil, err
|
||||
}
|
||||
if int64(len(row)) != dim {
|
||||
return nil, fmt.Errorf("vector dim mismatch: expect %d, got %d", dim, len(row))
|
||||
return nil, merr.WrapErrParameterInvalidMsg("vector dim mismatch: expect %d, got %d", dim, len(row))
|
||||
}
|
||||
if isFloat16 {
|
||||
return typeutil.Float32ArrayToFloat16Bytes(row), nil
|
||||
@@ -1114,14 +1113,14 @@ func decodeByteVectorElement(v gjson.Result, dim, bytesPerVec int64, isFloat16 b
|
||||
return typeutil.Float32ArrayToBFloat16Bytes(row), nil
|
||||
}
|
||||
if v.Type != gjson.String {
|
||||
return nil, fmt.Errorf("expect float vector array or base64 string")
|
||||
return nil, merr.WrapErrParameterInvalidMsg("expect float vector array or base64 string")
|
||||
}
|
||||
var row []byte
|
||||
if err := json.Unmarshal([]byte(v.Raw), &row); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if int64(len(row)) != bytesPerVec {
|
||||
return nil, fmt.Errorf("byte length mismatch: expect %d, got %d", bytesPerVec, len(row))
|
||||
return nil, merr.WrapErrParameterInvalidMsg("byte length mismatch: expect %d, got %d", bytesPerVec, len(row))
|
||||
}
|
||||
return row, nil
|
||||
}
|
||||
@@ -1293,7 +1292,7 @@ func encodeEmbListQuery(vecs []gjson.Result, elemType schemapb.DataType, dim int
|
||||
|
||||
func buildStructArrayFieldData(structSchema *schemapb.StructArrayFieldSchema, perRow []structArrayRow) (*schemapb.FieldData, error) {
|
||||
if len(perRow) == 0 {
|
||||
return nil, fmt.Errorf("struct array field %s has no rows", structSchema.GetName())
|
||||
return nil, merr.WrapErrParameterInvalidMsg("struct array field %s has no rows", structSchema.GetName())
|
||||
}
|
||||
subs := structSchema.GetFields()
|
||||
subFieldData := make([]*schemapb.FieldData, 0, len(subs))
|
||||
@@ -1308,12 +1307,12 @@ func buildStructArrayFieldData(structSchema *schemapb.StructArrayFieldSchema, pe
|
||||
for rowIdx, row := range perRow {
|
||||
val, ok := row[short]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("struct %s row %d missing sub-field %s",
|
||||
return nil, merr.WrapErrParameterInvalidMsg("struct %s row %d missing sub-field %s",
|
||||
structSchema.GetName(), rowIdx, short)
|
||||
}
|
||||
scalar, ok := val.(*schemapb.ScalarField)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("struct %s sub-field %s row %d: unexpected payload type %T",
|
||||
return nil, merr.WrapErrParameterInvalidMsg("struct %s sub-field %s row %d: unexpected payload type %T",
|
||||
structSchema.GetName(), short, rowIdx, val)
|
||||
}
|
||||
arrayArray.Data = append(arrayArray.Data, scalar)
|
||||
@@ -1341,12 +1340,12 @@ func buildStructArrayFieldData(structSchema *schemapb.StructArrayFieldSchema, pe
|
||||
for rowIdx, row := range perRow {
|
||||
val, ok := row[short]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("struct %s row %d missing sub-field %s",
|
||||
return nil, merr.WrapErrParameterInvalidMsg("struct %s row %d missing sub-field %s",
|
||||
structSchema.GetName(), rowIdx, short)
|
||||
}
|
||||
vf, ok := val.(*schemapb.VectorField)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("struct %s sub-field %s row %d: unexpected payload type %T",
|
||||
return nil, merr.WrapErrParameterInvalidMsg("struct %s sub-field %s row %d: unexpected payload type %T",
|
||||
structSchema.GetName(), short, rowIdx, val)
|
||||
}
|
||||
vecArray.Data = append(vecArray.Data, vf)
|
||||
@@ -1365,7 +1364,7 @@ func buildStructArrayFieldData(structSchema *schemapb.StructArrayFieldSchema, pe
|
||||
},
|
||||
})
|
||||
default:
|
||||
return nil, fmt.Errorf("unsupported struct sub-field data type: %s", sub.GetDataType())
|
||||
return nil, merr.WrapErrParameterInvalidMsg("unsupported struct sub-field data type: %s", sub.GetDataType())
|
||||
}
|
||||
}
|
||||
return &schemapb.FieldData{
|
||||
@@ -1403,11 +1402,11 @@ func extractStructArrayRow(fd *schemapb.FieldData, rowIdx int, schema *schemapb.
|
||||
case schemapb.DataType_Array:
|
||||
rowData := sub.GetScalars().GetArrayData().GetData()
|
||||
if rowIdx >= len(rowData) {
|
||||
return nil, fmt.Errorf("struct sub-field %s missing row %d", short, rowIdx)
|
||||
return nil, merr.WrapErrParameterInvalidMsg("struct sub-field %s missing row %d", short, rowIdx)
|
||||
}
|
||||
values := scalarArrayToInterfaces(rowData[rowIdx])
|
||||
if len(values) != elemCount {
|
||||
return nil, fmt.Errorf("struct sub-field %s element count mismatch: expect %d got %d",
|
||||
return nil, merr.WrapErrParameterInvalidMsg("struct sub-field %s element count mismatch: expect %d got %d",
|
||||
short, elemCount, len(values))
|
||||
}
|
||||
for i, v := range values {
|
||||
@@ -1416,28 +1415,28 @@ func extractStructArrayRow(fd *schemapb.FieldData, rowIdx int, schema *schemapb.
|
||||
case schemapb.DataType_ArrayOfVector:
|
||||
va := sub.GetVectors().GetVectorArray()
|
||||
if va == nil {
|
||||
return nil, fmt.Errorf("struct sub-field %s has no vector array", short)
|
||||
return nil, merr.WrapErrParameterInvalidMsg("struct sub-field %s has no vector array", short)
|
||||
}
|
||||
if rowIdx >= len(va.GetData()) {
|
||||
return nil, fmt.Errorf("struct sub-field %s missing row %d", short, rowIdx)
|
||||
return nil, merr.WrapErrParameterInvalidMsg("struct sub-field %s missing row %d", short, rowIdx)
|
||||
}
|
||||
dim, ok := subDims[short]
|
||||
if !ok || dim <= 0 {
|
||||
return nil, fmt.Errorf("schema missing dim for struct sub-field %s", short)
|
||||
return nil, merr.WrapErrParameterInvalidMsg("schema missing dim for struct sub-field %s", short)
|
||||
}
|
||||
values, err := vectorFieldToInterfaces(va.GetData()[rowIdx], va.GetElementType(), dim)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(values) != elemCount {
|
||||
return nil, fmt.Errorf("struct sub-field %s vector element count mismatch: expect %d got %d",
|
||||
return nil, merr.WrapErrParameterInvalidMsg("struct sub-field %s vector element count mismatch: expect %d got %d",
|
||||
short, elemCount, len(values))
|
||||
}
|
||||
for i, v := range values {
|
||||
out[i][short] = v
|
||||
}
|
||||
default:
|
||||
return nil, fmt.Errorf("unsupported struct sub-field type %s", sub.GetType())
|
||||
return nil, merr.WrapErrParameterInvalidMsg("unsupported struct sub-field type %s", sub.GetType())
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
@@ -1455,7 +1454,7 @@ func structArraySubDims(fieldName string, schema *schemapb.CollectionSchema) (ma
|
||||
}
|
||||
dim, err := getDim(sub)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("schema sub-field %s has no dim: %w", sub.GetName(), err)
|
||||
return nil, merr.WrapErrParameterInvalidErr(err, "schema sub-field %s has no dim", sub.GetName())
|
||||
}
|
||||
subDims[subShortName(sub)] = dim
|
||||
}
|
||||
@@ -1469,22 +1468,22 @@ func structSubElemCount(sub *schemapb.FieldData, rowIdx int, subDims map[string]
|
||||
case schemapb.DataType_Array:
|
||||
rowData := sub.GetScalars().GetArrayData().GetData()
|
||||
if rowIdx >= len(rowData) {
|
||||
return 0, fmt.Errorf("struct sub-field %s row %d out of range", sub.GetFieldName(), rowIdx)
|
||||
return 0, merr.WrapErrParameterInvalidMsg("struct sub-field %s row %d out of range", sub.GetFieldName(), rowIdx)
|
||||
}
|
||||
return len(scalarArrayToInterfaces(rowData[rowIdx])), nil
|
||||
case schemapb.DataType_ArrayOfVector:
|
||||
va := sub.GetVectors().GetVectorArray()
|
||||
if va == nil || rowIdx >= len(va.GetData()) {
|
||||
return 0, fmt.Errorf("struct sub-field %s row %d out of range", sub.GetFieldName(), rowIdx)
|
||||
return 0, merr.WrapErrParameterInvalidMsg("struct sub-field %s row %d out of range", sub.GetFieldName(), rowIdx)
|
||||
}
|
||||
short := structFieldShortName(sub.GetFieldName())
|
||||
dim, ok := subDims[short]
|
||||
if !ok || dim <= 0 {
|
||||
return 0, fmt.Errorf("schema missing dim for struct sub-field %s", short)
|
||||
return 0, merr.WrapErrParameterInvalidMsg("schema missing dim for struct sub-field %s", short)
|
||||
}
|
||||
return vectorFieldElemCount(va.GetData()[rowIdx], va.GetElementType(), dim)
|
||||
default:
|
||||
return 0, fmt.Errorf("unsupported struct sub-field type %s", sub.GetType())
|
||||
return 0, merr.WrapErrParameterInvalidMsg("unsupported struct sub-field type %s", sub.GetType())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1539,7 +1538,7 @@ func scalarArrayToInterfaces(sf *schemapb.ScalarField) []interface{} {
|
||||
|
||||
func vectorFieldElemCount(vf *schemapb.VectorField, elemType schemapb.DataType, dim int64) (int, error) {
|
||||
if dim <= 0 {
|
||||
return 0, fmt.Errorf("invalid dim %d", dim)
|
||||
return 0, merr.WrapErrParameterInvalidMsg("invalid dim %d", dim)
|
||||
}
|
||||
switch elemType {
|
||||
case schemapb.DataType_FloatVector:
|
||||
@@ -1553,13 +1552,13 @@ func vectorFieldElemCount(vf *schemapb.VectorField, elemType schemapb.DataType,
|
||||
case schemapb.DataType_Int8Vector:
|
||||
return len(vf.GetInt8Vector()) / int(dim), nil
|
||||
default:
|
||||
return 0, fmt.Errorf("unsupported vector element type %s", elemType)
|
||||
return 0, merr.WrapErrParameterInvalidMsg("unsupported vector element type %s", elemType)
|
||||
}
|
||||
}
|
||||
|
||||
func vectorFieldToInterfaces(vf *schemapb.VectorField, elemType schemapb.DataType, dim int64) ([]interface{}, error) {
|
||||
if dim <= 0 {
|
||||
return nil, fmt.Errorf("invalid dim %d", dim)
|
||||
return nil, merr.WrapErrParameterInvalidMsg("invalid dim %d", dim)
|
||||
}
|
||||
switch elemType {
|
||||
case schemapb.DataType_FloatVector:
|
||||
@@ -1612,7 +1611,7 @@ func vectorFieldToInterfaces(vf *schemapb.VectorField, elemType schemapb.DataTyp
|
||||
}
|
||||
return out, nil
|
||||
default:
|
||||
return nil, fmt.Errorf("unsupported vector element type %s", elemType)
|
||||
return nil, merr.WrapErrParameterInvalidMsg("unsupported vector element type %s", elemType)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1620,7 +1619,7 @@ func convertFloatVectorToArray(vector [][]float32, dim int64) ([]float32, error)
|
||||
floatArray := make([]float32, 0)
|
||||
for _, arr := range vector {
|
||||
if int64(len(arr)) != dim {
|
||||
return nil, fmt.Errorf("[]float32 size %d doesn't equal to vector dimension %d of %s",
|
||||
return nil, merr.WrapErrParameterInvalidMsg("[]float32 size %d doesn't equal to vector dimension %d of %s",
|
||||
len(arr), dim, schemapb.DataType_name[int32(schemapb.DataType_FloatVector)])
|
||||
}
|
||||
for i := int64(0); i < dim; i++ {
|
||||
@@ -1643,7 +1642,7 @@ func convertBinaryVectorToArray(vector [][]byte, dim int64, dataType schemapb.Da
|
||||
binaryArray := make([]byte, 0, len(vector)*int(bytesLen))
|
||||
for _, arr := range vector {
|
||||
if int64(len(arr)) != bytesLen {
|
||||
return nil, fmt.Errorf("[]byte size %d doesn't equal to vector dimension %d of %s",
|
||||
return nil, merr.WrapErrParameterInvalidMsg("[]byte size %d doesn't equal to vector dimension %d of %s",
|
||||
len(arr), dim, schemapb.DataType_name[int32(dataType)])
|
||||
}
|
||||
for i := int64(0); i < bytesLen; i++ {
|
||||
@@ -1657,7 +1656,7 @@ func convertInt8VectorToArray(vector [][]int8, dim int64) ([]byte, error) {
|
||||
byteArray := make([]byte, 0)
|
||||
for _, arr := range vector {
|
||||
if int64(len(arr)) != dim {
|
||||
return nil, fmt.Errorf("[]int8 size %d doesn't equal to vector dimension %d of %s",
|
||||
return nil, merr.WrapErrParameterInvalidMsg("[]int8 size %d doesn't equal to vector dimension %d of %s",
|
||||
len(arr), dim, schemapb.DataType_name[int32(schemapb.DataType_Int8Vector)])
|
||||
}
|
||||
for i := int64(0); i < dim; i++ {
|
||||
@@ -1691,7 +1690,7 @@ func reflectValueCandi(v reflect.Value) (map[string]fieldCandi, error) {
|
||||
}
|
||||
return result, nil
|
||||
default:
|
||||
return nil, fmt.Errorf("unsupport row type: %s", v.Kind().String())
|
||||
return nil, merr.WrapErrParameterInvalidMsg("unsupport row type: %s", v.Kind().String())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1713,7 +1712,7 @@ func convertToIntArray(dataType schemapb.DataType, arr interface{}) []int32 {
|
||||
func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool, sch *schemapb.CollectionSchema, inInsert bool, partialUpdate bool) ([]*schemapb.FieldData, error) {
|
||||
rowsLen := len(rows)
|
||||
if rowsLen == 0 {
|
||||
return []*schemapb.FieldData{}, errors.New("no row need to be convert to columns")
|
||||
return []*schemapb.FieldData{}, merr.WrapErrParameterInvalidMsg("no row need to be convert to columns")
|
||||
}
|
||||
|
||||
isDynamic := sch.EnableDynamicField
|
||||
@@ -1802,7 +1801,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
|
||||
dim, _ := getDim(field)
|
||||
nameDims[field.Name] = dim
|
||||
default:
|
||||
return nil, fmt.Errorf("the type(%v) of field(%v) is not supported, use other sdk please", field.DataType, field.Name)
|
||||
return nil, merr.WrapErrParameterInvalidMsg("the type(%v) of field(%v) is not supported, use other sdk please", field.DataType, field.Name)
|
||||
}
|
||||
nameColumns[field.Name] = data
|
||||
fieldData[field.Name] = &schemapb.FieldData{
|
||||
@@ -1813,7 +1812,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
|
||||
}
|
||||
}
|
||||
if len(nameDims) == 0 && len(sch.Functions) == 0 && !partialUpdate {
|
||||
return nil, fmt.Errorf("collection: %s has no vector field or functions", sch.Name)
|
||||
return nil, merr.WrapErrParameterInvalidMsg("collection: %s has no vector field or functions", sch.Name)
|
||||
}
|
||||
|
||||
dynamicCol := make([][]byte, 0, rowsLen)
|
||||
@@ -1836,7 +1835,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
|
||||
continue
|
||||
}
|
||||
if !allowInsertAutoID {
|
||||
return nil, merr.WrapErrParameterInvalidMsg(fmt.Sprintf("no need to pass pk field(%s) when autoid==true in insert", field.Name))
|
||||
return nil, merr.WrapErrParameterInvalidMsg("no need to pass pk field(%s) when autoid==true in insert", field.Name)
|
||||
}
|
||||
}
|
||||
if (field.Nullable || field.DefaultValue != nil) && !ok {
|
||||
@@ -1851,7 +1850,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
|
||||
if partialUpdate {
|
||||
continue
|
||||
}
|
||||
return nil, fmt.Errorf("row %d does not has field %s", idx, field.Name)
|
||||
return nil, merr.WrapErrParameterInvalidMsg("row %d does not has field %s", idx, field.Name)
|
||||
}
|
||||
fieldLen[field.Name] += 1
|
||||
switch field.DataType {
|
||||
@@ -1893,7 +1892,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
|
||||
vec := typeutil.Float32ArrayToFloat16Bytes(candi.v.Interface().([]float32))
|
||||
nameColumns[field.Name] = append(nameColumns[field.Name].([][]byte), vec)
|
||||
default:
|
||||
return nil, merr.WrapErrParameterInvalidMsg(fmt.Sprintf("invalid type(%v) of field(%v) ", field.DataType, field.Name))
|
||||
return nil, merr.WrapErrParameterInvalidMsg("invalid type(%v) of field(%v) ", field.DataType, field.Name)
|
||||
}
|
||||
case schemapb.DataType_BFloat16Vector:
|
||||
switch candi.v.Interface().(type) {
|
||||
@@ -1903,7 +1902,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
|
||||
vec := typeutil.Float32ArrayToBFloat16Bytes(candi.v.Interface().([]float32))
|
||||
nameColumns[field.Name] = append(nameColumns[field.Name].([][]byte), vec)
|
||||
default:
|
||||
return nil, merr.WrapErrParameterInvalidMsg(fmt.Sprintf("invalid type(%v) of field(%v) ", field.DataType, field.Name))
|
||||
return nil, merr.WrapErrParameterInvalidMsg("invalid type(%v) of field(%v) ", field.DataType, field.Name)
|
||||
}
|
||||
case schemapb.DataType_SparseFloatVector:
|
||||
content := candi.v.Interface().([]byte)
|
||||
@@ -1915,7 +1914,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
|
||||
case schemapb.DataType_Int8Vector:
|
||||
nameColumns[field.Name] = append(nameColumns[field.Name].([][]int8), candi.v.Interface().([]int8))
|
||||
default:
|
||||
return nil, fmt.Errorf("the type(%v) of field(%v) is not supported, use other sdk please", field.DataType, field.Name)
|
||||
return nil, merr.WrapErrParameterInvalidMsg("the type(%v) of field(%v) is not supported, use other sdk please", field.DataType, field.Name)
|
||||
}
|
||||
|
||||
delete(set, field.Name)
|
||||
@@ -1931,7 +1930,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
|
||||
}
|
||||
bs, err := json.Marshal(m)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to marshal dynamic field %w", err)
|
||||
return nil, merr.WrapErrParameterInvalidErr(err, "failed to marshal dynamic field")
|
||||
}
|
||||
dynamicCol = append(dynamicCol, bs)
|
||||
}
|
||||
@@ -1949,7 +1948,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
|
||||
zap.String("fieldName", name),
|
||||
zap.Int("fieldLen", len(validData)),
|
||||
zap.Int("rowsLen", rowsLen))
|
||||
return nil, fmt.Errorf("column %s has length %d, expected %d", name, len(validData), rowsLen)
|
||||
return nil, merr.WrapErrParameterInvalidMsg("column %s has length %d, expected %d", name, len(validData), rowsLen)
|
||||
}
|
||||
} else {
|
||||
log.Info("skip empty field for partial update",
|
||||
@@ -1963,7 +1962,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
|
||||
zap.String("fieldName", name),
|
||||
zap.Int("fieldLen", fieldLen[name]),
|
||||
zap.Int("rowsLen", rowsLen))
|
||||
return nil, fmt.Errorf("column %s has length %d, expected %d", name, fieldLen[name], rowsLen)
|
||||
return nil, merr.WrapErrParameterInvalidMsg("column %s has length %d, expected %d", name, fieldLen[name], rowsLen)
|
||||
}
|
||||
|
||||
colData := fieldData[name]
|
||||
@@ -2183,7 +2182,7 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
|
||||
},
|
||||
}
|
||||
default:
|
||||
return nil, fmt.Errorf("the type(%v) of field(%v) is not supported, use other sdk please", colData.Type, name)
|
||||
return nil, merr.WrapErrParameterInvalidMsg("the type(%v) of field(%v) is not supported, use other sdk please", colData.Type, name)
|
||||
}
|
||||
colData.ValidData = validDataMap[name]
|
||||
columns = append(columns, colData)
|
||||
@@ -2212,11 +2211,11 @@ func anyToColumns(rows []map[string]interface{}, validDataMap map[string][]bool,
|
||||
if partialUpdate {
|
||||
continue
|
||||
}
|
||||
return nil, fmt.Errorf("row %d does not has struct field %s", rowIdx, structField.GetName())
|
||||
return nil, merr.WrapErrParameterInvalidMsg("row %d does not has struct field %s", rowIdx, structField.GetName())
|
||||
}
|
||||
sr, ok := val.(structArrayRow)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("row %d struct field %s has unexpected payload type %T",
|
||||
return nil, merr.WrapErrParameterInvalidMsg("row %d struct field %s has unexpected payload type %T",
|
||||
rowIdx, structField.GetName(), val)
|
||||
}
|
||||
perRow = append(perRow, sr)
|
||||
@@ -2437,27 +2436,27 @@ func fieldDataValueCount(fieldData *schemapb.FieldData) (int64, error) {
|
||||
dim := fieldData.GetVectors().GetDim()
|
||||
bytesPerRow := dim / 8
|
||||
if bytesPerRow <= 0 {
|
||||
return 0, fmt.Errorf("invalid binary vector dimension %d for field %s", dim, fieldData.GetFieldName())
|
||||
return 0, merr.WrapErrParameterInvalidMsg("invalid binary vector dimension %d for field %s", dim, fieldData.GetFieldName())
|
||||
}
|
||||
return int64(len(fieldData.GetVectors().GetBinaryVector())) / bytesPerRow, nil
|
||||
case schemapb.DataType_FloatVector:
|
||||
dim := fieldData.GetVectors().GetDim()
|
||||
if dim <= 0 {
|
||||
return 0, fmt.Errorf("invalid float vector dimension %d for field %s", dim, fieldData.GetFieldName())
|
||||
return 0, merr.WrapErrParameterInvalidMsg("invalid float vector dimension %d for field %s", dim, fieldData.GetFieldName())
|
||||
}
|
||||
return int64(len(fieldData.GetVectors().GetFloatVector().GetData())) / dim, nil
|
||||
case schemapb.DataType_Float16Vector:
|
||||
dim := fieldData.GetVectors().GetDim()
|
||||
bytesPerRow := dim * 2
|
||||
if bytesPerRow <= 0 {
|
||||
return 0, fmt.Errorf("invalid float16 vector dimension %d for field %s", dim, fieldData.GetFieldName())
|
||||
return 0, merr.WrapErrParameterInvalidMsg("invalid float16 vector dimension %d for field %s", dim, fieldData.GetFieldName())
|
||||
}
|
||||
return int64(len(fieldData.GetVectors().GetFloat16Vector())) / bytesPerRow, nil
|
||||
case schemapb.DataType_BFloat16Vector:
|
||||
dim := fieldData.GetVectors().GetDim()
|
||||
bytesPerRow := dim * 2
|
||||
if bytesPerRow <= 0 {
|
||||
return 0, fmt.Errorf("invalid bfloat16 vector dimension %d for field %s", dim, fieldData.GetFieldName())
|
||||
return 0, merr.WrapErrParameterInvalidMsg("invalid bfloat16 vector dimension %d for field %s", dim, fieldData.GetFieldName())
|
||||
}
|
||||
return int64(len(fieldData.GetVectors().GetBfloat16Vector())) / bytesPerRow, nil
|
||||
case schemapb.DataType_SparseFloatVector:
|
||||
@@ -2465,7 +2464,7 @@ func fieldDataValueCount(fieldData *schemapb.FieldData) (int64, error) {
|
||||
case schemapb.DataType_Int8Vector:
|
||||
dim := fieldData.GetVectors().GetDim()
|
||||
if dim <= 0 {
|
||||
return 0, fmt.Errorf("invalid int8 vector dimension %d for field %s", dim, fieldData.GetFieldName())
|
||||
return 0, merr.WrapErrParameterInvalidMsg("invalid int8 vector dimension %d for field %s", dim, fieldData.GetFieldName())
|
||||
}
|
||||
return int64(len(fieldData.GetVectors().GetInt8Vector())) / dim, nil
|
||||
case schemapb.DataType_ArrayOfStruct:
|
||||
@@ -2479,10 +2478,10 @@ func fieldDataValueCount(fieldData *schemapb.FieldData) (int64, error) {
|
||||
case schemapb.DataType_ArrayOfVector:
|
||||
return int64(len(subs[0].GetVectors().GetVectorArray().GetData())), nil
|
||||
default:
|
||||
return 0, fmt.Errorf("unsupported struct sub-field type %s for field %s", subs[0].GetType(), fieldData.GetFieldName())
|
||||
return 0, merr.WrapErrParameterInvalidMsg("unsupported struct sub-field type %s for field %s", subs[0].GetType(), fieldData.GetFieldName())
|
||||
}
|
||||
default:
|
||||
return 0, fmt.Errorf("the type(%v) of field(%v) is not supported, use other sdk please", fieldData.GetType(), fieldData.GetFieldName())
|
||||
return 0, merr.WrapErrParameterInvalidMsg("the type(%v) of field(%v) is not supported, use other sdk please", fieldData.GetType(), fieldData.GetFieldName())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2532,7 +2531,7 @@ func newFieldDataRowAccessor(fieldData *schemapb.FieldData) (*fieldDataRowAccess
|
||||
return accessor, nil
|
||||
}
|
||||
if valueCount != validCount {
|
||||
return nil, fmt.Errorf("field %s has %d valid rows, but data length is %d", fieldData.GetFieldName(), validCount, valueCount)
|
||||
return nil, merr.WrapErrParameterInvalidMsg("field %s has %d valid rows, but data length is %d", fieldData.GetFieldName(), validCount, valueCount)
|
||||
}
|
||||
accessor.compactIndices = compactIndices
|
||||
return accessor, nil
|
||||
@@ -2543,7 +2542,7 @@ func (accessor *fieldDataRowAccessor) rowIndex(rowIdx int64) (int64, bool, error
|
||||
return rowIdx, true, nil
|
||||
}
|
||||
if rowIdx >= int64(len(accessor.validData)) {
|
||||
return 0, false, fmt.Errorf("row index %d out of range for field %s valid data length %d", rowIdx, accessor.fieldData.GetFieldName(), len(accessor.validData))
|
||||
return 0, false, merr.WrapErrParameterInvalidMsg("row index %d out of range for field %s valid data length %d", rowIdx, accessor.fieldData.GetFieldName(), len(accessor.validData))
|
||||
}
|
||||
if !accessor.validData[rowIdx] {
|
||||
return 0, false, nil
|
||||
@@ -2575,7 +2574,7 @@ func buildQueryResp(rowsNum int64, needFields []string, fieldDataList []*schemap
|
||||
stringPks := ids.GetStrId().GetData()
|
||||
rowsNum = int64(len(stringPks))
|
||||
default:
|
||||
return nil, errors.New("the type of primary key(id) is not supported, use other sdk please")
|
||||
return nil, merr.WrapErrParameterInvalidMsg("the type of primary key(id) is not supported, use other sdk please")
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -2716,7 +2715,7 @@ func buildQueryResp(rowsNum int64, needFields []string, fieldDataList []*schemap
|
||||
stringPks := ids.GetStrId().GetData()
|
||||
row[pkFieldName] = stringPks[i]
|
||||
default:
|
||||
return nil, errors.New("the type of primary key(id) is not supported, use other sdk please")
|
||||
return nil, merr.WrapErrParameterInvalidMsg("the type of primary key(id) is not supported, use other sdk please")
|
||||
}
|
||||
}
|
||||
if scores != nil && int64(len(scores)) > i {
|
||||
@@ -2734,29 +2733,31 @@ func hasSearchAggregationResult(results *schemapb.SearchResultData) bool {
|
||||
|
||||
func buildSearchAggregationResp(results *schemapb.SearchResultData, enableInt64 bool, collectionSchema *schemapb.CollectionSchema) ([]gin.H, error) {
|
||||
if results == nil {
|
||||
return nil, errors.New("search_aggregation result is nil")
|
||||
// The aggregation payload is produced by the server-side reduce, never
|
||||
// by the request: a malformed shape is an internal contract violation.
|
||||
return nil, merr.WrapErrServiceInternalMsg("search_aggregation result is nil")
|
||||
}
|
||||
aggTopks := results.GetAggTopks()
|
||||
pbBuckets := results.GetAggBuckets()
|
||||
if len(aggTopks) == 0 {
|
||||
return nil, errors.New("search_aggregation response missing agg_topks")
|
||||
return nil, merr.WrapErrServiceInternalMsg("search_aggregation response missing agg_topks")
|
||||
}
|
||||
if results.GetNumQueries() <= 0 {
|
||||
return nil, errors.New("search_aggregation response missing nq")
|
||||
return nil, merr.WrapErrServiceInternalMsg("search_aggregation response missing nq")
|
||||
}
|
||||
if len(aggTopks) != int(results.GetNumQueries()) {
|
||||
return nil, fmt.Errorf("search_aggregation agg_topks length %d does not match nq %d", len(aggTopks), results.GetNumQueries())
|
||||
return nil, merr.WrapErrServiceInternalMsg("search_aggregation agg_topks length %d does not match nq %d", len(aggTopks), results.GetNumQueries())
|
||||
}
|
||||
|
||||
total := int64(0)
|
||||
for _, topk := range aggTopks {
|
||||
if topk < 0 {
|
||||
return nil, errors.New("search_aggregation agg_topks cannot contain negative values")
|
||||
return nil, merr.WrapErrServiceInternalMsg("search_aggregation agg_topks cannot contain negative values")
|
||||
}
|
||||
total += topk
|
||||
}
|
||||
if total != int64(len(pbBuckets)) {
|
||||
return nil, fmt.Errorf("search_aggregation agg_topks sum %d does not match bucket count %d", total, len(pbBuckets))
|
||||
return nil, merr.WrapErrServiceInternalMsg("search_aggregation agg_topks sum %d does not match bucket count %d", total, len(pbBuckets))
|
||||
}
|
||||
|
||||
output := make([]gin.H, 0, len(aggTopks))
|
||||
@@ -2778,7 +2779,7 @@ func buildSearchAggregationResp(results *schemapb.SearchResultData, enableInt64
|
||||
|
||||
func buildAggBucketResp(pb *schemapb.AggBucket, enableInt64 bool, collectionSchema *schemapb.CollectionSchema) (gin.H, error) {
|
||||
if pb == nil {
|
||||
return nil, errors.New("search_aggregation bucket is nil")
|
||||
return nil, merr.WrapErrServiceInternalMsg("search_aggregation bucket is nil")
|
||||
}
|
||||
bucket := gin.H{
|
||||
"key": buildAggBucketKeyResp(pb.GetKey(), enableInt64),
|
||||
@@ -2986,7 +2987,7 @@ func convertConsistencyLevel(reqConsistencyLevel string) (commonpb.ConsistencyLe
|
||||
if reqConsistencyLevel != "" {
|
||||
level, ok := commonpb.ConsistencyLevel_value[reqConsistencyLevel]
|
||||
if !ok {
|
||||
return 0, false, merr.WrapErrParameterInvalidMsg(fmt.Sprintf("parameter:'%s' is incorrect, please check it", reqConsistencyLevel))
|
||||
return 0, false, merr.WrapErrParameterInvalidMsg("parameter:'%s' is incorrect, please check it", reqConsistencyLevel)
|
||||
}
|
||||
return commonpb.ConsistencyLevel(level), false, nil
|
||||
}
|
||||
@@ -3096,7 +3097,7 @@ func convertDefaultValue(value interface{}, dataType schemapb.DataType) (*schema
|
||||
}
|
||||
return data, nil
|
||||
default:
|
||||
return nil, merr.WrapErrParameterInvalidMsg(fmt.Sprintf("Unexpected default value type: %s", dataType.String()))
|
||||
return nil, merr.WrapErrParameterInvalidMsg("Unexpected default value type: %s", dataType.String())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -3491,7 +3492,7 @@ func generateSearchParams(reqSearchParams map[string]interface{}) ([]*commonpb.K
|
||||
for key, value := range reqSearchParams {
|
||||
if val, ok := paramsMap[key]; ok {
|
||||
if !deepEqual(val, value) {
|
||||
return nil, merr.WrapErrParameterInvalidMsg(fmt.Sprintf("ambiguous parameter: %s, in search_param: %v, in search_param.params: %v", key, value, val))
|
||||
return nil, merr.WrapErrParameterInvalidMsg("ambiguous parameter: %s, in search_param: %v, in search_param.params: %v", key, value, val)
|
||||
}
|
||||
} else if key != Params {
|
||||
paramsMap[key] = value
|
||||
|
||||
@@ -25,6 +25,7 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/cockroachdb/errors"
|
||||
"github.com/gin-gonic/gin"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
@@ -36,6 +37,7 @@ import (
|
||||
"github.com/milvus-io/milvus-proto/go-api/v3/schemapb"
|
||||
"github.com/milvus-io/milvus/internal/json"
|
||||
"github.com/milvus-io/milvus/pkg/v3/common"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/typeutil"
|
||||
)
|
||||
|
||||
@@ -3005,7 +3007,8 @@ func TestBuildQueryResps(t *testing.T) {
|
||||
}
|
||||
|
||||
_, err := buildQueryResp(int64(0), outputFields, newFieldData([]*schemapb.FieldData{}, 1000), generateIDs(schemapb.DataType_Int64, 3), DefaultScores, true, nil)
|
||||
assert.Equal(t, "the type(1000) of field(wrong-field-type) is not supported, use other sdk please", err.Error())
|
||||
assert.Contains(t, err.Error(), "the type(1000) of field(wrong-field-type) is not supported, use other sdk please")
|
||||
assert.True(t, errors.Is(err, merr.ErrParameterInvalid))
|
||||
|
||||
res, err := buildQueryResp(int64(0), outputFields, []*schemapb.FieldData{}, generateIDs(schemapb.DataType_Int64, 3), DefaultScores, true, nil)
|
||||
assert.Equal(t, 3, len(res))
|
||||
|
||||
@@ -17,15 +17,13 @@
|
||||
package httpserver
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/cockroachdb/errors"
|
||||
"google.golang.org/protobuf/proto"
|
||||
|
||||
"github.com/milvus-io/milvus-proto/go-api/v3/commonpb"
|
||||
"github.com/milvus-io/milvus-proto/go-api/v3/milvuspb"
|
||||
"github.com/milvus-io/milvus-proto/go-api/v3/schemapb"
|
||||
"github.com/milvus-io/milvus/internal/json"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/typeutil"
|
||||
)
|
||||
|
||||
@@ -70,7 +68,7 @@ type WrappedInsertRequest struct {
|
||||
func (w *WrappedInsertRequest) AsInsertRequest() (*milvuspb.InsertRequest, error) {
|
||||
fieldData, err := convertFieldDataArray(w.FieldsData)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: convert field data failed: %v", errBadRequest, err)
|
||||
return nil, badRequestf(err, "convert field data failed")
|
||||
}
|
||||
return &milvuspb.InsertRequest{
|
||||
Base: w.Base,
|
||||
@@ -98,12 +96,12 @@ func (f *FieldData) makePbFloat16OrBfloat16Array(raw json.RawMessage, serializeF
|
||||
return nil, 0, newFieldDataError(f.FieldName, err)
|
||||
}
|
||||
if len(wrappedData) < 1 {
|
||||
return nil, 0, errors.New("at least one row for insert")
|
||||
return nil, 0, merr.WrapErrParameterInvalidMsg("at least one row for insert")
|
||||
}
|
||||
array0 := wrappedData[0]
|
||||
dim := len(array0)
|
||||
if dim < 1 {
|
||||
return nil, 0, errors.New("dim must >= 1")
|
||||
return nil, 0, merr.WrapErrParameterInvalidMsg("dim must >= 1")
|
||||
}
|
||||
data := make([]byte, 0, len(wrappedData)*dim*2)
|
||||
for _, fp32Array := range wrappedData {
|
||||
@@ -238,12 +236,12 @@ func (f *FieldData) AsSchemapb() (*schemapb.FieldData, error) {
|
||||
return nil, newFieldDataError(f.FieldName, err)
|
||||
}
|
||||
if len(wrappedData) < 1 {
|
||||
return nil, errors.New("at least one row for insert")
|
||||
return nil, merr.WrapErrParameterInvalidMsg("at least one row for insert")
|
||||
}
|
||||
array0 := wrappedData[0]
|
||||
dim := len(array0)
|
||||
if dim < 1 {
|
||||
return nil, errors.New("dim must >= 1")
|
||||
return nil, merr.WrapErrParameterInvalidMsg("dim must >= 1")
|
||||
}
|
||||
data := make([]float32, len(wrappedData)*dim)
|
||||
|
||||
@@ -299,7 +297,7 @@ func (f *FieldData) AsSchemapb() (*schemapb.FieldData, error) {
|
||||
return nil, newFieldDataError(f.FieldName, err)
|
||||
}
|
||||
if len(wrappedData) < 1 {
|
||||
return nil, errors.New("at least one row for insert")
|
||||
return nil, merr.WrapErrParameterInvalidMsg("at least one row for insert")
|
||||
}
|
||||
data := make([][]byte, len(wrappedData))
|
||||
dim := int64(0)
|
||||
@@ -333,12 +331,12 @@ func (f *FieldData) AsSchemapb() (*schemapb.FieldData, error) {
|
||||
return nil, newFieldDataError(f.FieldName, err)
|
||||
}
|
||||
if len(wrappedData) < 1 {
|
||||
return nil, errors.New("at least one row for insert")
|
||||
return nil, merr.WrapErrParameterInvalidMsg("at least one row for insert")
|
||||
}
|
||||
array0 := wrappedData[0]
|
||||
dim := len(array0)
|
||||
if dim < 1 {
|
||||
return nil, errors.New("dim must >= 1")
|
||||
return nil, merr.WrapErrParameterInvalidMsg("dim must >= 1")
|
||||
}
|
||||
data := make([]byte, len(wrappedData)*dim)
|
||||
|
||||
@@ -358,13 +356,13 @@ func (f *FieldData) AsSchemapb() (*schemapb.FieldData, error) {
|
||||
},
|
||||
}
|
||||
default:
|
||||
return nil, errors.New("unsupported data type")
|
||||
return nil, merr.WrapErrParameterInvalidMsg("unsupported data type")
|
||||
}
|
||||
return &ret, nil
|
||||
}
|
||||
|
||||
func newFieldDataError(field string, err error) error {
|
||||
return fmt.Errorf("parse field[%s]: %s", field, err.Error())
|
||||
return merr.WrapErrParameterInvalidErr(err, "parse field[%s]", field)
|
||||
}
|
||||
|
||||
func convertFieldDataArray(input []*FieldData) ([]*schemapb.FieldData, error) {
|
||||
|
||||
@@ -25,10 +25,18 @@ import (
|
||||
"github.com/gin-gonic/gin/binding"
|
||||
|
||||
"github.com/milvus-io/milvus-proto/go-api/v3/commonpb"
|
||||
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
||||
)
|
||||
|
||||
var errBadRequest = errors.New("bad request")
|
||||
|
||||
// badRequestf wraps err with the package-internal errBadRequest sentinel and
|
||||
// a contextual message, preserving the inner error chain. wrapHandler maps
|
||||
// any error that matches errBadRequest to HTTP 400.
|
||||
func badRequestf(err error, format string, args ...any) error {
|
||||
return merr.Mark(merr.Wrapf(err, format, args...), errBadRequest)
|
||||
}
|
||||
|
||||
// handlerFunc handles http request with gin context
|
||||
type handlerFunc func(c *gin.Context) (interface{}, error)
|
||||
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user