mirror of
https://github.com/milvus-io/milvus.git
synced 2026-07-21 02:05:41 +00:00
0d2637fd63
issue: #51493 ### What #50221 split the `status` label values of `milvus_proxy_req_count` / `milvus_proxy_grpc_latency` in place. Released in **v2.6.19**, that silently zeroes every existing dashboard panel and alert rule matching `status="fail"` / `status="rejected"` — an alert that stops firing rather than erroring. This keeps the classification but moves it onto its own dimension, restoring the status domain to what <= v2.6.18 emitted: ``` status: success | fail | rejected | retry | total | abandon (as before v2.6.19) cause: user | system | cancel | na (new) ``` | v2.6.19 / v2.6.20 | this PR | |---|---| | `status="fail_input"` | `status="fail", cause="user"` | | `status="fail_system"` | `status="fail", cause="system"` | | `status="rejected_user"` | `status="rejected", cause="user"` | | `status="rejected_system"` | `status="rejected", cause="system"` | | `status="cancel"` | `status="fail"` or `"rejected"`, `cause="cancel"` | | `success` / `retry` / `total` / `abandon` | unchanged, `cause="na"` | ### Why a label instead of new status values - **Old queries work unchanged and count each request once.** `status="fail"` is again every hard failure; Prometheus aggregates over `cause` for free. That rollup is what a label dimension *is* — the split forces every consumer who just wants "how many failures" to hand-write `status=~"fail_.*"`. - **Cardinality is unchanged.** `cause` is functionally dependent on the outcome, so the realized `(status, cause)` pairs are exactly the series the split already produces. Additive, not multiplicative. - **New consumers lose nothing**: alert on `status="fail", cause="system"`; route `cause="user"` to the owning application team. The alternative — emitting old and new `status` values side by side during a transition — was rejected: it defers the break rather than removing it (the old values must still be dropped eventually, forcing the same migration later), and meanwhile double-counts every request in unfiltered aggregations. `cancel` is folded into `cause` for the same reason: before v2.6.19 client cancellations were counted as `fail` (cancel in the response status) or `rejected` (cancel at the interceptor), so promoting it to a `status` value quietly makes `status="fail"` under-count versus older releases. As a cause it stays excludable via `cause!="cancel"` without redefining `status`. ### Also in this PR - The 7 `snapshot_impl` sites that emitted a bare `fail` with no classification now report their real cause via `failMetricLabel(err)` (e.g. `snapshot_metadata_uri is required` is `cause="user"`, not a system fault). Their `status` is unchanged. - `deployments/monitor/grafana/milvus-dashboard.json`: the "Faild Request Rate" panel goes back to `status="fail"` — a verbatim revert of what #50221 changed, which is the compatibility claim demonstrated. - Dev docs and stale comments that named the old label values. ### Verification - `TestParseMetricLabelStatusDomainIsStable` pins the status domain, so re-splitting it fails CI rather than shipping. - Adding a label makes every `WithLabelValues` site arity-sensitive **at runtime, not compile time** — a missed site panics in production. Unit tests do not reach all of them (coverage shows `GetRestoreSnapshotState`, `ListRestoreSnapshotJobs`, `PinSnapshotData`, `UnpinSnapshotData` at 0%), so all **58 emit sites were verified statically via AST**, not only the ones tests happen to hit. - Passing: `pkg/metrics`, `pkg/util/requestutil`, `pkg/util/merr`, `internal/distributed/proxy` (incl. `httpserver`, which covers the REST emit path), and the `internal/proxy` snapshot / metric-label tests. `golangci-lint` clean on every touched package. - Pre-existing and unrelated: `service_test.go` bind failures (`listen tcp :19529: address already in use`) reproduce identically on unmodified master — a local process holds the port. ### Rollout Should be cherry-picked to 2.6 so the window in which the fine-grained `status` values exist stays confined to v2.6.19–v2.6.20. Users who already adopted the new values on those two releases need a one-time query change (`status="fail_system"` -> `status="fail", cause="system"`); that is a known, bounded set, and preferable to leaving every pre-2.6.19 dashboard silently broken. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
546 lines
24 KiB
Go
546 lines
24 KiB
Go
// Licensed to the LF AI & Data foundation under one
|
|
// or more contributor license agreements. See the NOTICE file
|
|
// distributed with this work for additional information
|
|
// regarding copyright ownership. The ASF licenses this file
|
|
// to you under the Apache License, Version 2.0 (the
|
|
// "License"); you may not use this file except in compliance
|
|
// with the License. You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
// See the License for the specific language governing permissions and
|
|
// limitations under the License.
|
|
|
|
package proxy
|
|
|
|
import (
|
|
"context"
|
|
"strconv"
|
|
|
|
"go.opentelemetry.io/otel"
|
|
|
|
"github.com/milvus-io/milvus-proto/go-api/v3/commonpb"
|
|
"github.com/milvus-io/milvus-proto/go-api/v3/milvuspb"
|
|
"github.com/milvus-io/milvus/pkg/v3/metrics"
|
|
"github.com/milvus-io/milvus/pkg/v3/mlog"
|
|
"github.com/milvus-io/milvus/pkg/v3/proto/datapb"
|
|
"github.com/milvus-io/milvus/pkg/v3/util/commonpbutil"
|
|
"github.com/milvus-io/milvus/pkg/v3/util/merr"
|
|
"github.com/milvus-io/milvus/pkg/v3/util/paramtable"
|
|
"github.com/milvus-io/milvus/pkg/v3/util/timerecord"
|
|
"github.com/milvus-io/milvus/pkg/v3/util/typeutil"
|
|
)
|
|
|
|
func (node *Proxy) CreateSnapshot(ctx context.Context, req *milvuspb.CreateSnapshotRequest) (*commonpb.Status, error) {
|
|
ctx, sp := otel.Tracer(typeutil.ProxyRole).Start(ctx, "Proxy-CreateSnapshot")
|
|
defer sp.End()
|
|
|
|
log := mlog.With(
|
|
mlog.String("snapshotName", req.GetName()),
|
|
mlog.String("collectionName", req.GetCollectionName()),
|
|
)
|
|
|
|
method := "CreateSnapshot"
|
|
tr := timerecord.NewTimeRecorder(method)
|
|
log.Info(ctx, rpcReceived(method))
|
|
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, metrics.TotalLabel, metrics.CauseNA, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
t := &createSnapshotTask{
|
|
req: req,
|
|
ctx: ctx,
|
|
Condition: NewTaskCondition(ctx),
|
|
mixCoord: node.mixCoord,
|
|
}
|
|
|
|
err := node.sched.ddQueue.Enqueue(t)
|
|
if err != nil {
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, metrics.AbandonLabel, metrics.CauseNA, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
log.Warn(ctx, "CreateSnapshot failed to Enqueue",
|
|
mlog.Err(err))
|
|
return merr.Status(err), nil
|
|
}
|
|
|
|
if err := t.WaitToFinish(); err != nil {
|
|
failStatus, failCause := failMetricLabel(err)
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, failStatus, failCause, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
log.Warn(ctx, "CreateSnapshot failed to WaitToFinish",
|
|
mlog.Err(err))
|
|
return merr.Status(err), nil
|
|
}
|
|
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, metrics.SuccessLabel, metrics.CauseNA, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
metrics.ProxyReqLatency.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method).Observe(float64(tr.ElapseSpan().Milliseconds()))
|
|
return t.result, nil
|
|
}
|
|
|
|
func (node *Proxy) DropSnapshot(ctx context.Context, req *milvuspb.DropSnapshotRequest) (*commonpb.Status, error) {
|
|
ctx, sp := otel.Tracer(typeutil.ProxyRole).Start(ctx, "Proxy-DropSnapshot")
|
|
defer sp.End()
|
|
|
|
log := mlog.With(
|
|
mlog.String("snapshotName", req.GetName()),
|
|
)
|
|
|
|
method := "DropSnapshot"
|
|
tr := timerecord.NewTimeRecorder(method)
|
|
log.Info(ctx, rpcReceived(method))
|
|
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, metrics.TotalLabel, metrics.CauseNA, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
t := &dropSnapshotTask{
|
|
req: req,
|
|
ctx: ctx,
|
|
Condition: NewTaskCondition(ctx),
|
|
mixCoord: node.mixCoord,
|
|
}
|
|
|
|
err := node.sched.ddQueue.Enqueue(t)
|
|
if err != nil {
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, metrics.AbandonLabel, metrics.CauseNA, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
log.Warn(ctx, "DropSnapshot failed to Enqueue",
|
|
mlog.Err(err))
|
|
return merr.Status(err), nil
|
|
}
|
|
|
|
if err := t.WaitToFinish(); err != nil {
|
|
failStatus, failCause := failMetricLabel(err)
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, failStatus, failCause, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
log.Warn(ctx, "DropSnapshot failed to WaitToFinish",
|
|
mlog.Err(err))
|
|
return merr.Status(err), nil
|
|
}
|
|
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, metrics.SuccessLabel, metrics.CauseNA, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
metrics.ProxyReqLatency.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method).Observe(float64(tr.ElapseSpan().Milliseconds()))
|
|
return t.result, nil
|
|
}
|
|
|
|
func (node *Proxy) DescribeSnapshot(ctx context.Context, req *milvuspb.DescribeSnapshotRequest) (*milvuspb.DescribeSnapshotResponse, error) {
|
|
ctx, sp := otel.Tracer(typeutil.ProxyRole).Start(ctx, "Proxy-DescribeSnapshot")
|
|
defer sp.End()
|
|
|
|
log := mlog.With(
|
|
mlog.String("snapshotName", req.GetName()),
|
|
)
|
|
|
|
method := "DescribeSnapshot"
|
|
tr := timerecord.NewTimeRecorder(method)
|
|
log.Info(ctx, rpcReceived(method))
|
|
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, metrics.TotalLabel, metrics.CauseNA, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
t := &describeSnapshotTask{
|
|
req: req,
|
|
ctx: ctx,
|
|
Condition: NewTaskCondition(ctx),
|
|
mixCoord: node.mixCoord,
|
|
}
|
|
|
|
err := node.sched.ddQueue.Enqueue(t)
|
|
if err != nil {
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, metrics.AbandonLabel, metrics.CauseNA, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
log.Warn(ctx, "DescribeSnapshot failed to Enqueue",
|
|
mlog.Err(err))
|
|
return &milvuspb.DescribeSnapshotResponse{
|
|
Status: merr.Status(err),
|
|
}, nil
|
|
}
|
|
|
|
if err := t.WaitToFinish(); err != nil {
|
|
failStatus, failCause := failMetricLabel(err)
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, failStatus, failCause, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
log.Warn(ctx, "DescribeSnapshot failed to WaitToFinish",
|
|
mlog.Err(err))
|
|
return &milvuspb.DescribeSnapshotResponse{
|
|
Status: merr.Status(err),
|
|
}, nil
|
|
}
|
|
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, metrics.SuccessLabel, metrics.CauseNA, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
metrics.ProxyReqLatency.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method).Observe(float64(tr.ElapseSpan().Milliseconds()))
|
|
return t.result, nil
|
|
}
|
|
|
|
func (node *Proxy) ListSnapshots(ctx context.Context, req *milvuspb.ListSnapshotsRequest) (*milvuspb.ListSnapshotsResponse, error) {
|
|
ctx, sp := otel.Tracer(typeutil.ProxyRole).Start(ctx, "Proxy-ListSnapshots")
|
|
defer sp.End()
|
|
|
|
log := mlog.With(
|
|
mlog.String("collectionName", req.GetCollectionName()))
|
|
|
|
method := "ListSnapshots"
|
|
tr := timerecord.NewTimeRecorder(method)
|
|
log.Info(ctx, rpcReceived(method))
|
|
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, metrics.TotalLabel, metrics.CauseNA, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
t := &listSnapshotsTask{
|
|
req: req,
|
|
ctx: ctx,
|
|
Condition: NewTaskCondition(ctx),
|
|
mixCoord: node.mixCoord,
|
|
}
|
|
|
|
err := node.sched.ddQueue.Enqueue(t)
|
|
if err != nil {
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, metrics.AbandonLabel, metrics.CauseNA, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
log.Warn(ctx, "ListSnapshots failed to Enqueue",
|
|
mlog.Err(err))
|
|
return &milvuspb.ListSnapshotsResponse{
|
|
Status: merr.Status(err),
|
|
}, nil
|
|
}
|
|
|
|
if err := t.WaitToFinish(); err != nil {
|
|
failStatus, failCause := failMetricLabel(err)
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, failStatus, failCause, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
log.Warn(ctx, "ListSnapshots failed to WaitToFinish",
|
|
mlog.Err(err))
|
|
return &milvuspb.ListSnapshotsResponse{
|
|
Status: merr.Status(err),
|
|
}, nil
|
|
}
|
|
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, metrics.SuccessLabel, metrics.CauseNA, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
metrics.ProxyReqLatency.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method).Observe(float64(tr.ElapseSpan().Milliseconds()))
|
|
return t.result, nil
|
|
}
|
|
|
|
func (node *Proxy) RestoreExternalSnapshot(ctx context.Context, req *milvuspb.RestoreExternalSnapshotRequest) (*milvuspb.RestoreExternalSnapshotResponse, error) {
|
|
ctx, sp := otel.Tracer(typeutil.ProxyRole).Start(ctx, "Proxy-RestoreExternalSnapshot")
|
|
defer sp.End()
|
|
|
|
if req == nil {
|
|
err := merr.WrapErrParameterInvalidMsg("restore external snapshot request is nil")
|
|
return &milvuspb.RestoreExternalSnapshotResponse{Status: merr.Status(err)}, nil
|
|
}
|
|
|
|
method := "RestoreExternalSnapshot"
|
|
tr := timerecord.NewTimeRecorder(method)
|
|
log := mlog.With(
|
|
mlog.String("targetDb", req.GetDbName()),
|
|
mlog.String("targetCollection", req.GetTargetCollectionName()),
|
|
mlog.Bool("snapshotMetadataURISet", req.GetSnapshotMetadataUri() != ""),
|
|
mlog.Bool("externalSpecSet", req.GetExternalSpec() != ""),
|
|
)
|
|
log.Info(ctx, rpcReceived(method))
|
|
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, metrics.TotalLabel, metrics.CauseNA, req.GetDbName(), req.GetTargetCollectionName()).Inc()
|
|
if req.GetSnapshotMetadataUri() == "" {
|
|
err := merr.WrapErrParameterInvalidMsg("snapshot_metadata_uri is required for restore external snapshot")
|
|
failStatus, failCause := failMetricLabel(err)
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, failStatus, failCause, req.GetDbName(), req.GetTargetCollectionName()).Inc()
|
|
return &milvuspb.RestoreExternalSnapshotResponse{Status: merr.Status(err)}, nil
|
|
}
|
|
resp, err := node.mixCoord.RestoreSnapshot(ctx, &datapb.RestoreSnapshotRequest{
|
|
Base: commonpbutil.NewMsgBase(
|
|
commonpbutil.WithMsgType(commonpb.MsgType_RestoreExternalSnapshot),
|
|
),
|
|
TargetDbName: req.GetDbName(),
|
|
TargetCollectionName: req.GetTargetCollectionName(),
|
|
External: true,
|
|
SnapshotS3Location: req.GetSnapshotMetadataUri(),
|
|
ExternalSpec: req.GetExternalSpec(),
|
|
})
|
|
if err = merr.CheckRPCCall(resp, err); err != nil {
|
|
failStatus, failCause := failMetricLabel(err)
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, failStatus, failCause, req.GetDbName(), req.GetTargetCollectionName()).Inc()
|
|
log.Warn(ctx, "RestoreExternalSnapshot failed", mlog.Err(err))
|
|
return &milvuspb.RestoreExternalSnapshotResponse{Status: merr.Status(err)}, nil
|
|
}
|
|
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, metrics.SuccessLabel, metrics.CauseNA, req.GetDbName(), req.GetTargetCollectionName()).Inc()
|
|
metrics.ProxyReqLatency.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method).Observe(float64(tr.ElapseSpan().Milliseconds()))
|
|
return &milvuspb.RestoreExternalSnapshotResponse{
|
|
Status: resp.GetStatus(),
|
|
JobId: resp.GetJobId(),
|
|
}, nil
|
|
}
|
|
|
|
func (node *Proxy) ExportSnapshot(ctx context.Context, req *milvuspb.ExportSnapshotRequest) (*milvuspb.ExportSnapshotResponse, error) {
|
|
ctx, sp := otel.Tracer(typeutil.ProxyRole).Start(ctx, "Proxy-ExportSnapshot")
|
|
defer sp.End()
|
|
|
|
if req == nil {
|
|
err := merr.WrapErrParameterInvalidMsg("export snapshot request is nil")
|
|
return &milvuspb.ExportSnapshotResponse{Status: merr.Status(err)}, nil
|
|
}
|
|
|
|
method := "ExportSnapshot"
|
|
tr := timerecord.NewTimeRecorder(method)
|
|
log := mlog.With(
|
|
mlog.String("snapshotName", req.GetName()),
|
|
mlog.String("dbName", req.GetDbName()),
|
|
mlog.String("collectionName", req.GetCollectionName()),
|
|
mlog.Bool("targetS3PathSet", req.GetTargetS3Path() != ""),
|
|
mlog.Bool("externalSpecSet", req.GetExternalSpec() != ""),
|
|
)
|
|
log.Info(ctx, rpcReceived(method))
|
|
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, metrics.TotalLabel, metrics.CauseNA, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
if err := ValidateSnapshotName(req.GetName()); err != nil {
|
|
failStatus, failCause := failMetricLabel(err)
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, failStatus, failCause, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
return &milvuspb.ExportSnapshotResponse{Status: merr.Status(err)}, nil
|
|
}
|
|
if req.GetCollectionName() == "" {
|
|
err := merr.WrapErrParameterInvalidMsg("collection_name is required for export snapshot")
|
|
failStatus, failCause := failMetricLabel(err)
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, failStatus, failCause, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
return &milvuspb.ExportSnapshotResponse{Status: merr.Status(err)}, nil
|
|
}
|
|
if req.GetTargetS3Path() == "" {
|
|
err := merr.WrapErrParameterInvalidMsg("target_s3_path is required for export snapshot")
|
|
failStatus, failCause := failMetricLabel(err)
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, failStatus, failCause, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
return &milvuspb.ExportSnapshotResponse{Status: merr.Status(err)}, nil
|
|
}
|
|
|
|
collectionID, err := globalMetaCache.GetCollectionID(ctx, req.GetDbName(), req.GetCollectionName())
|
|
if err != nil {
|
|
failStatus, failCause := failMetricLabel(err)
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, failStatus, failCause, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
log.Warn(ctx, "ExportSnapshot failed to resolve collection", mlog.Err(err))
|
|
return &milvuspb.ExportSnapshotResponse{Status: merr.Status(err)}, nil
|
|
}
|
|
resp, err := node.mixCoord.ExportSnapshot(ctx, &datapb.ExportSnapshotRequest{
|
|
Base: commonpbutil.NewMsgBase(
|
|
commonpbutil.WithMsgType(commonpb.MsgType_ExportSnapshot),
|
|
),
|
|
Name: req.GetName(),
|
|
CollectionId: collectionID,
|
|
TargetS3Path: req.GetTargetS3Path(),
|
|
ExternalSpec: req.GetExternalSpec(),
|
|
})
|
|
if err = merr.CheckRPCCall(resp, err); err != nil {
|
|
failStatus, failCause := failMetricLabel(err)
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, failStatus, failCause, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
log.Warn(ctx, "ExportSnapshot failed", mlog.Err(err))
|
|
return &milvuspb.ExportSnapshotResponse{Status: merr.Status(err)}, nil
|
|
}
|
|
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, metrics.SuccessLabel, metrics.CauseNA, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
metrics.ProxyReqLatency.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method).Observe(float64(tr.ElapseSpan().Milliseconds()))
|
|
return &milvuspb.ExportSnapshotResponse{
|
|
Status: resp.GetStatus(),
|
|
SnapshotMetadataUri: resp.GetSnapshotMetadataUri(),
|
|
}, nil
|
|
}
|
|
|
|
func (node *Proxy) RestoreSnapshot(ctx context.Context, req *milvuspb.RestoreSnapshotRequest) (*milvuspb.RestoreSnapshotResponse, error) {
|
|
ctx, sp := otel.Tracer(typeutil.ProxyRole).Start(ctx, "Proxy-RestoreSnapshot")
|
|
defer sp.End()
|
|
|
|
log := mlog.With(
|
|
mlog.String("snapshotName", req.GetName()),
|
|
)
|
|
|
|
method := "RestoreSnapshot"
|
|
tr := timerecord.NewTimeRecorder(method)
|
|
log.Info(ctx, rpcReceived(method))
|
|
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, metrics.TotalLabel, metrics.CauseNA, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
t := &restoreSnapshotTask{
|
|
req: req,
|
|
ctx: ctx,
|
|
Condition: NewTaskCondition(ctx),
|
|
mixCoord: node.mixCoord,
|
|
}
|
|
|
|
err := node.sched.ddQueue.Enqueue(t)
|
|
if err != nil {
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, metrics.AbandonLabel, metrics.CauseNA, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
log.Warn(ctx, "RestoreSnapshot failed to Enqueue",
|
|
mlog.Err(err))
|
|
return &milvuspb.RestoreSnapshotResponse{Status: merr.Status(err)}, nil
|
|
}
|
|
|
|
if err := t.WaitToFinish(); err != nil {
|
|
failStatus, failCause := failMetricLabel(err)
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, failStatus, failCause, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
log.Warn(ctx, "RestoreSnapshot failed to WaitToFinish",
|
|
mlog.Err(err))
|
|
return &milvuspb.RestoreSnapshotResponse{Status: merr.Status(err)}, nil
|
|
}
|
|
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, metrics.SuccessLabel, metrics.CauseNA, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
metrics.ProxyReqLatency.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method).Observe(float64(tr.ElapseSpan().Milliseconds()))
|
|
return t.result, nil
|
|
}
|
|
|
|
func (node *Proxy) GetRestoreSnapshotState(ctx context.Context, req *milvuspb.GetRestoreSnapshotStateRequest) (*milvuspb.GetRestoreSnapshotStateResponse, error) {
|
|
ctx, sp := otel.Tracer(typeutil.ProxyRole).Start(ctx, "Proxy-GetRestoreSnapshotState")
|
|
defer sp.End()
|
|
|
|
log := mlog.With(
|
|
mlog.Int64("jobID", req.GetJobId()),
|
|
)
|
|
|
|
method := "GetRestoreSnapshotState"
|
|
tr := timerecord.NewTimeRecorder(method)
|
|
log.Info(ctx, rpcReceived(method))
|
|
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, metrics.TotalLabel, metrics.CauseNA, "", "").Inc()
|
|
t := &getRestoreSnapshotStateTask{
|
|
req: req,
|
|
ctx: ctx,
|
|
Condition: NewTaskCondition(ctx),
|
|
mixCoord: node.mixCoord,
|
|
}
|
|
|
|
err := node.sched.ddQueue.Enqueue(t)
|
|
if err != nil {
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, metrics.AbandonLabel, metrics.CauseNA, "", "").Inc()
|
|
log.Warn(ctx, "GetRestoreSnapshotState failed to Enqueue",
|
|
mlog.Err(err))
|
|
return &milvuspb.GetRestoreSnapshotStateResponse{
|
|
Status: merr.Status(err),
|
|
}, nil
|
|
}
|
|
|
|
if err := t.WaitToFinish(); err != nil {
|
|
failStatus, failCause := failMetricLabel(err)
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, failStatus, failCause, "", "").Inc()
|
|
log.Warn(ctx, "GetRestoreSnapshotState failed to WaitToFinish",
|
|
mlog.Err(err))
|
|
return &milvuspb.GetRestoreSnapshotStateResponse{
|
|
Status: merr.Status(err),
|
|
}, nil
|
|
}
|
|
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, metrics.SuccessLabel, metrics.CauseNA, "", "").Inc()
|
|
metrics.ProxyReqLatency.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method).Observe(float64(tr.ElapseSpan().Milliseconds()))
|
|
return t.result, nil
|
|
}
|
|
|
|
func (node *Proxy) ListRestoreSnapshotJobs(ctx context.Context, req *milvuspb.ListRestoreSnapshotJobsRequest) (*milvuspb.ListRestoreSnapshotJobsResponse, error) {
|
|
ctx, sp := otel.Tracer(typeutil.ProxyRole).Start(ctx, "Proxy-ListRestoreSnapshotJobs")
|
|
defer sp.End()
|
|
|
|
log := mlog.With(
|
|
mlog.String("collectionName", req.GetCollectionName()),
|
|
)
|
|
|
|
method := "ListRestoreSnapshotJobs"
|
|
tr := timerecord.NewTimeRecorder(method)
|
|
log.Info(ctx, rpcReceived(method))
|
|
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, metrics.TotalLabel, metrics.CauseNA, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
t := &listRestoreSnapshotJobsTask{
|
|
req: req,
|
|
ctx: ctx,
|
|
Condition: NewTaskCondition(ctx),
|
|
mixCoord: node.mixCoord,
|
|
}
|
|
|
|
err := node.sched.ddQueue.Enqueue(t)
|
|
if err != nil {
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, metrics.AbandonLabel, metrics.CauseNA, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
log.Warn(ctx, "ListRestoreSnapshotJobs failed to Enqueue",
|
|
mlog.Err(err))
|
|
return &milvuspb.ListRestoreSnapshotJobsResponse{
|
|
Status: merr.Status(err),
|
|
}, nil
|
|
}
|
|
|
|
if err := t.WaitToFinish(); err != nil {
|
|
failStatus, failCause := failMetricLabel(err)
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, failStatus, failCause, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
log.Warn(ctx, "ListRestoreSnapshotJobs failed to WaitToFinish",
|
|
mlog.Err(err))
|
|
return &milvuspb.ListRestoreSnapshotJobsResponse{
|
|
Status: merr.Status(err),
|
|
}, nil
|
|
}
|
|
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, metrics.SuccessLabel, metrics.CauseNA, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
metrics.ProxyReqLatency.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method).Observe(float64(tr.ElapseSpan().Milliseconds()))
|
|
return t.result, nil
|
|
}
|
|
|
|
func (node *Proxy) PinSnapshotData(ctx context.Context, req *milvuspb.PinSnapshotDataRequest) (*milvuspb.PinSnapshotDataResponse, error) {
|
|
ctx, sp := otel.Tracer(typeutil.ProxyRole).Start(ctx, "Proxy-PinSnapshotData")
|
|
defer sp.End()
|
|
|
|
log := mlog.With(
|
|
mlog.String("snapshotName", req.GetName()),
|
|
mlog.String("collectionName", req.GetCollectionName()),
|
|
mlog.String("dbName", req.GetDbName()),
|
|
)
|
|
|
|
method := "PinSnapshotData"
|
|
tr := timerecord.NewTimeRecorder(method)
|
|
log.Info(ctx, rpcReceived(method))
|
|
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, metrics.TotalLabel, metrics.CauseNA, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
t := &pinSnapshotDataTask{
|
|
req: req,
|
|
ctx: ctx,
|
|
Condition: NewTaskCondition(ctx),
|
|
mixCoord: node.mixCoord,
|
|
}
|
|
|
|
if err := node.sched.ddQueue.Enqueue(t); err != nil {
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, metrics.AbandonLabel, metrics.CauseNA, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
log.Warn(ctx, "PinSnapshotData failed to Enqueue",
|
|
mlog.Err(err))
|
|
return &milvuspb.PinSnapshotDataResponse{
|
|
Status: merr.Status(err),
|
|
}, nil
|
|
}
|
|
|
|
if err := t.WaitToFinish(); err != nil {
|
|
failStatus, failCause := failMetricLabel(err)
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, failStatus, failCause, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
log.Warn(ctx, "PinSnapshotData failed to WaitToFinish",
|
|
mlog.Err(err))
|
|
return &milvuspb.PinSnapshotDataResponse{
|
|
Status: merr.Status(err),
|
|
}, nil
|
|
}
|
|
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, metrics.SuccessLabel, metrics.CauseNA, req.GetDbName(), req.GetCollectionName()).Inc()
|
|
metrics.ProxyReqLatency.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method).Observe(float64(tr.ElapseSpan().Milliseconds()))
|
|
return t.result, nil
|
|
}
|
|
|
|
func (node *Proxy) UnpinSnapshotData(ctx context.Context, req *milvuspb.UnpinSnapshotDataRequest) (*commonpb.Status, error) {
|
|
ctx, sp := otel.Tracer(typeutil.ProxyRole).Start(ctx, "Proxy-UnpinSnapshotData")
|
|
defer sp.End()
|
|
|
|
log := mlog.With(
|
|
mlog.Int64("pinID", req.GetPinId()),
|
|
)
|
|
|
|
method := "UnpinSnapshotData"
|
|
tr := timerecord.NewTimeRecorder(method)
|
|
log.Info(ctx, rpcReceived(method))
|
|
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, metrics.TotalLabel, metrics.CauseNA, "", "").Inc()
|
|
t := &unpinSnapshotDataTask{
|
|
req: req,
|
|
ctx: ctx,
|
|
Condition: NewTaskCondition(ctx),
|
|
mixCoord: node.mixCoord,
|
|
}
|
|
|
|
if err := node.sched.ddQueue.Enqueue(t); err != nil {
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, metrics.AbandonLabel, metrics.CauseNA, "", "").Inc()
|
|
log.Warn(ctx, "UnpinSnapshotData failed to Enqueue",
|
|
mlog.Err(err))
|
|
return merr.Status(err), nil
|
|
}
|
|
|
|
if err := t.WaitToFinish(); err != nil {
|
|
failStatus, failCause := failMetricLabel(err)
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, failStatus, failCause, "", "").Inc()
|
|
log.Warn(ctx, "UnpinSnapshotData failed to WaitToFinish",
|
|
mlog.Err(err))
|
|
return merr.Status(err), nil
|
|
}
|
|
|
|
metrics.ProxyFunctionCall.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method, metrics.SuccessLabel, metrics.CauseNA, "", "").Inc()
|
|
metrics.ProxyReqLatency.WithLabelValues(strconv.FormatInt(paramtable.GetNodeID(), 10), method).Observe(float64(tr.ElapseSpan().Milliseconds()))
|
|
return t.result, nil
|
|
}
|