issue: https://github.com/milvus-io/milvus/issues/44358
design doc:
docs/design-docs/design_docs/20260609-external-snapshot-export-restore.md
Part 1/3 of the external snapshot cross-bucket restore stack.
This PR defines the API contract and entry surfaces for external
snapshot export and restore:
- Adds the consolidated design document for cross-bucket external
snapshot restore.
- Adds public gRPC, REST, and Go SDK API surfaces for
RestoreExternalSnapshot and ExportSnapshot.
- Adds internal DataCoord proto plumbing and generated code needed by
later implementation PRs.
- Wires Proxy RBAC grouping and database interceptor behavior for the
new APIs.
- Keeps the request contract on a single external_spec field and keeps
db_name for namespace routing rather than permission scoping.
Validation copied from the commit:
- GOTOOLCHAIN=go1.25.10 go test -c -tags dynamic,test -gcflags="all=-N
-l" -ldflags="-r ${RPATH}" -o /tmp/datacoord-commit1.test
github.com/milvus-io/milvus/internal/datacoord
- cd client && GOTOOLCHAIN=go1.25.10 go test -c -o
/tmp/client-milvusclient-commit1.test ./milvusclient
- internal/proxy package compile was blocked locally by missing C++
header internal/core/output/include/segcore/search_result_export_c.h
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
- Persist role descriptions when creating roles, expose them in role
list/describe results, and support updating descriptions through the
AlterRole path without changing role names.
- Store role descriptions in the role value body while keeping role
names in keys, and tolerate legacy empty role values plus undecodable
stored values by returning an empty description for that role row.
- Reject built-in/default roles and over-limit descriptions before WAL
writes, and share the role-description length validator between proxy
and rootcoord.
related: #50183
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
issue: #47420
## What this PR does
Project-wide migration of raw `fmt.Errorf` / `errors.New` in function
bodies onto
the `merr` framework, plus the Sys-vs-Input error classification and the
machinery it drives (retriability, fine-grained metrics, segcore
unification),
plus the convention docs and a linter that keeps it from regressing.
Scope: storage, proxy, coordinators (root/data/query), query node, data
node,
`pkg/util` & `internal/util`, expression parser, message queue,
streaming, and
misc packages. Bare raw-error usages went from ~3000 to a ~340 allowlist
(package-level sentinels / build-tag / test sites).
---
## How to review this PR
It is large but the vast majority is mechanical. Changes fall into three
tiers;
spend review budget on Part 2 and Part 3.
### Part 1 — Mechanical standardization (low risk, verify by rule)
Each converted call follows one of a small fixed set of rules. To
review, check
that each site obeys the matching rule rather than reading every line:
| Pattern | Rule |
|---|---|
| `fmt.Errorf("...")` originating a new error | →
`merr.WrapErrXxxMsg("...")` with a code matching the failure's meaning |
| Adding context to an existing typed error | → `merr.Wrap(err, "...")`
/ `merr.Wrapf(...)` — **preserves** the inner code (never `WrapErr*Err`,
which overwrites it) |
| Errors inside the streaming subsystem | → `status.New*` factories
(StreamingError), **not** merr — this is the component-internal dialect
(see `docs/dev/error_handling_guide.md`) |
| Low-level / control-flow signal caught by `errors.Is` | → kept as a
package-level `errors.New` sentinel (lowercase, same-package) |
Conventions are documented in `docs/dev/error_handling_guide.md`
(how-to) and
`docs/dev/error_sentinel_convention.md` (rules + audit). A
`gocritic`/`ruleguard`
rule (`rawmerrerror`, in `rules.go`) enforces "no raw `return
errors.New/fmt.Errorf`"
under `make verifiers`.
### Part 2 — Behavior changes (review these closely)
These are the sites where the wire contract or runtime behavior changes,
not just
the source text. Listed by category; representative locations given,
full set in
the diff.
**A. gRPC wire-code shifts: `UnexpectedError(1)/Code 65535` → typed
code.**
Where a handler previously returned a raw error (collapsed to
`Code=65535` on the
wire), it now returns a typed merr, so the client sees a real code. The
most
common shift is to `IllegalArgument(5)/Code 1100` (ParameterInvalid).
Touch
points include datanode task handlers (CreateTask/Query/Drop), proxy
Upsert,
querynode GetMetrics, datacoord CreateIndex, httpserver query-response
builder,
and typeutil schema validation. One code refinement: an index-param
validation
moved `1100` → `1101` (ParameterMissing). **Client/SDK assertions and
any code
that switched on `Code=65535` for these paths must be re-checked** (the
go_client
e2e assertions were already aligned in this PR).
**B. Prometheus `status` label contract change (externally visible).**
The proxy metric's coarse `fail` / `rejected` values are split into
`fail_input` / `fail_system` and `rejected_user` / `rejected_system` (in
`requestutil.ParseMetricLabel`; auth/privilege rejections count as
`rejected_user`), so dashboards can attribute a failure to caller vs
operator.
**Dashboards/alerts querying `status="fail"` must migrate to
`status=~"fail_.*"`, and `status="rejected"` to
`status=~"rejected_.*"`.** The
in-repo Grafana dashboard is already migrated; external dashboards built
on the
old values silently go empty after upgrade. This is the one change that
requires an ops-side migration.
**C. Retriability semantics.**
- C1: `merr.Status(err)` now forces `Retriable=false` when the error is
an
`InputError` — a malformed request can never succeed on blind retry, so
clients
never get the self-contradictory "your input is wrong but you may
retry".
- C2: `retry.Do` short-circuits an `InputError` (non-retriable) — **but
only when
the caller did not pass a `RetryErr` predicate**. The check is an
`if c.isRetryErr != nil { ... } else if InputError { ... }` *mutually
exclusive*
branch (`pkg/util/retry/retry.go`): an explicit `RetryErr` takes
precedence and
bypasses the InputError abort. `retry.Handle` deliberately does **not**
apply
the InputError abort (its callers signal abort via `shouldRetry=false`).
Four
flusher startup callsites that must retry through transient "not ready"
errors
were given explicit `RetryErr` escape hatches.
**D. segcore (C++→Go) error classification.**
A single shared Go-side table (`pkg/util/merr/segcore.go`) maps each
segcore code
to a merr sentinel + InputError/signal category, replacing scattered
hand-written
`if errorCode == ...` switches in the cgo wrappers. **Wire `Code` values
change
for every segcore pass-through error, not just the remapped ones.**
Named
sentinels remap (C++ `2003` → merr `2001`, `2033` → `2002`,
Folly/Knowhere codes
likewise); **all remaining pass-through codes (`2004`–`2043`, previously
surfaced to clients as raw C++ enum values) now serialize as `2000`**
(`ErrSegcore`), with the original C++ code preserved in the `Reason`
text
(`segcoreCode=...`); unknown/future codes collapse to `2000` as well
(pinned by
the `wire_code_projection` test). Transient segcore classes (object
storage /
file IO / OOM / mmap / FieldNotLoaded — 11 codes) now report
`Retriable=true`.
**Any client switching on raw segcore codes in the `2004`–`2043` range
must be
re-checked**; the in-Reason code remains available for diagnostics.
Signal
codes (PretendFinished / FollyCancel) are recognized centrally.
`errors.Is`-based
control flow on these (e.g. scheduler skip/retry) is preserved.
**E. InputError classification (25 sentinels + dynamic marks).**
25 sentinels in `errors.go` carry `WithErrorType(InputError)` (the
Collection /
ResourceGroup / Database families, `ErrIndexDuplicate`,
`ErrParameterInvalid`,
`ErrPrivilegeNotAuthenticated`, `ErrImportFailed`, `ErrQueryPlan`, ...),
plus dynamic
marks for the 8 segcore input codes (ExprInvalid, DimNotMatch,
MetricTypeInvalid, FieldIDInvalid, ...) and
`WrapErrAsInputError`. The widest blast radius is `ErrParameterInvalid`
(1100):
~2335 `WrapErrParameterInvalid*` callsites now classify as input /
non-retriable. Because of C1/C2 this changes retriability for
any path that returns these. **The audit to confirm no transient path
was
mis-marked is the single most important review item** (see Part 3). One
reverse
correction: storage field-stats parsing moved from `ErrParameterInvalid`
(input)
to `ErrDataIntegrity` — a corrupted stored stat is data corruption, not
user
input.
### Part 3 — Known risks & traps (called out proactively)
1. **`merr.Wrap` vs `WrapErr*Err` (code-masking).** `WrapErr*Err` builds
a
`wrappedMilvusError{sentinel: ErrServiceInternal}` whose `code()`
returns the
*outer* sentinel — it overwrites the inner typed code and hides the
`errors.Is`
chain. This is intentional (use it to *deliberately* downgrade), but it
was a
recurring conversion defect; the rule "add context with `merr.Wrap`,
downgrade
with `WrapErr*Err`" is enforced by convention and reviewed across the
diff.
2. **InputError × `retry.Do` blast radius.** Marking a sentinel
`InputError` makes
any `retry.Do(...)` without a `RetryErr` predicate stop retrying it.
Reviewers
should sanity-check that no transient use of the 19 newly-marked
sentinels
(especially `ErrParameterInvalid`) sits inside a retry loop that needed
to keep
spinning. The known flusher cases were handled (see C2).
3. **The ~340 raw-error allowlist.** What remains as bare `errors.New`
is, by
design: package-level sentinels (caught by `errors.Is`), `//go:build
test`
sites, and out-of-band trees (`cmd/`, `tests/`, codegen, walimpls). The
linter
only bans the *direct-return* form; assignment-then-return escapes and
the full
no-exceptions ban are deferred to an AST-based linter (Tier 2,
documented).
4. **segcore C++ second step deferred.** This PR unifies classification
on the Go
side; splitting the dual-semantic C++ codes at the source is a
follow-up.
---
## Validation
- `make verifiers`: Go side clean (gofmt + static-check across modules,
including
the new `rawmerrerror` rule with a 0-hit baseline repo-wide).
- `make test-go`: passing; the one real regression introduced (a
datanode
`invalid_task_type` assertion shifting `1` → `5` from a ParameterInvalid
conversion) was fixed in-tree.
- go_client e2e CreateIndex assertions aligned to the new merr messages.
---------
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Related to #50089
Add a Proxy management entry point that routes through MixCoord to clear
queued read tasks across all Proxy and QueryNode instances. Queued tasks
are failed with canceled semantics and per-component counts are returned
so operators can recover from read-task backlog without attempting
unsafe active-task preemption.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
## Summary
Implements Two-Phase Commit (2PC) for Import in primary/secondary
replication clusters. Data stays invisible (`is_importing=true`) until
an explicit commit signal is delivered via WAL, ensuring primary and
secondary clusters reach the same visible state at the same logical
position.
- **New proto**: `CommitImport=44`, `RollbackImport=45` WAL message
types; `Uncommitted=8`, `Committing=9` `ImportJobState` values;
`CommitImport`, `AbortImport`, `HandleCommitVchannel` RPCs on DataCoord;
`committed_vchannels` + `auto_commit` fields on `ImportJob`
- **WAL broadcast**: DataCoord broadcasts
`CommitImportMessage`/`RollbackImportMessage` to all vchannels via DDL
broadcast; CDC replicates to secondary clusters verbatim
- **DDL ack callbacks**: `commitImportV2AckCallback` CAS
`Uncommitted→Committing`; `rollbackImportV2AckCallback` CAS `*→Failed` +
segment drop
- **WAL flusher**: `wal_flusher.dispatch()` intercepts
`CommitImportMessage` per-vchannel, calls `wbMgr.FlushChannel` +
`DataCoord.HandleCommitVchannel`; no-op handler for
`RollbackImportMessage`
- **ImportChecker**: new `Uncommitted` case (auto-commits when
`auto_commit=true`); new `Committing` case (transitions to `Completed`
when all vchannels confirmed)
- **auto_commit option**: default `true` (backward compatible); `false`
lets replication platform control commit timing
- **RESTful API**: `POST /v2/vectordb/jobs/import/commit` and `POST
/v2/vectordb/jobs/import/abort`; `GetImportProgress` surfaces
`Uncommitted` and `Committing` states
**Out of scope**: `commit_timestamp` propagation (handled in companion
PR)
## Test Plan
- [ ] Unit tests for `IsAutoCommit` helper
- [ ] Unit tests for `HandleCommitVchannel` idempotency
- [ ] Unit tests for `CommitImport`/`AbortImport` RPC handlers (state
validation, mutex, broadcast)
- [ ] Unit tests for DDL ack callbacks (CAS races for commit/abort)
- [ ] Unit tests for `ImportChecker` `Uncommitted`/`Committing` cases
- [ ] Unit tests for WAL flusher dispatch (CommitImport/RollbackImport
cases)
- [ ] Unit tests for RESTful commit/abort handlers (valid jobId, invalid
jobId error path)
- [ ] `GetImportProgress` surfaces `Uncommitted` and `Committing` states
- [ ] E2E: non-replication cluster with `auto_commit=true` (default)
behaves identically to pre-2PC
issue: #48525
design doc: https://github.com/milvus-io/milvus-design-docs/pull/29🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Yihao Dai <yihao.dai@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
issue: https://github.com/milvus-io/milvus/issues/47403
this pr contains performance optimization for concurrent
CreatePartition:
1. add a version cache on proxy for partition level cache
2. avoid repeated updates of target when there're many CreatePartition
requests.
3. avoid unnecessary copy of meta data.
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
issue: #47314
Gate newly spawned load-config replicas from Proxy shard leader
discovery until their shard leaders are serviceable.
This keeps replica query visibility as an in-memory QueryCoord state,
defaults recovered/existing replicas to query-visible for compatibility,
checks query-invisible replicas in load-config compliance, and keeps the
query-invisible fast index as replica IDs rather than replica pointers.
Validation:
- `go test -tags test -run
'TestReplicaQueryVisibility|TestReplicaManagerQueryVisibility' -count=1
./internal/querycoordv2/meta`
- `go test -tags test -run
'TestCheckAllReplicasServiceable|TestApplyLoadConfigChanges|Test.*GetShardLeaders'
-count=1 ./internal/querycoordv2`
- `go test -tags test -gcflags="all=-N -l" -run
'TestHandleReplicaLoadConfigCompliance|TestValidateRGDistribution'
-count=1 ./internal/coordinator`
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Related to #49398
Bump Go module references from pkg/v2 and milvus-proto/go-api/v2 to
pkg/v3 and milvus-proto/go-api/v3 so the client tracks the Milvus 3.x
release line.
This prepares the repository for the upcoming 3.x.y release by aligning
imports, module dependencies, and proto API references with the new
major-version module paths.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: zilliztech/spark-milvus#74
Add a new proxy management HTTP endpoint
/management/datacoord/backfill/commit that takes an object-storage path
to a BackfillResult JSON produced by the Spark offline backfill job,
lets DataCoord download and parse it, then dispatches per-segment
metadata updates through the existing BatchUpdateManifest broadcast
pipeline.
V2 (StorageV2 packed-parquet) and V3 (Loon manifest) segments share the
broadcast, so the broadcaster's SharedDBName + SharedCollectionName
resource keys serialise the commit against compaction and other DDL-like
operations on the same collection. The ack callback classifies each item
and applies either UpdateManifestVersion (V3) or the ported
UpdateSegmentColumnGroupsOperator (V2) in one UpdateSegmentsInfo batch.
The V2 path trusts the group-level row_count in the result JSON rather
than reading parquet footers; row_count is split evenly across the
group's binlog files (remainder added to the last file).
Changes:
- proto: new CommitBackfillResult RPC on DataCoord; extend
BatchUpdateManifestItem with a v2_column_groups payload + new
BatchUpdateManifestV2ColumnGroups message.
- datacoord: new services_commit_backfill.go handler; new
backfill_result.go (JSON decoding, object-URI normaliser, bucket
validation, V2 group constructor). Port
UpdateSegmentColumnGroupsOperator from stash/update_columngroup_restful
into meta.go. Extend the BatchUpdateManifest broadcast callback to
dispatch V2/V3 items.
- proxy: new /management/datacoord/backfill/commit HTTP handler.
- mixcoord: grpc server/client/wrapper delegators.
- storage: expose RemoteChunkManager.BucketName() for bucket-mismatch
rejection in the URI normaliser.
Tests:
- backfill_result_test.go: JSON parsing, normaliser, V2 group builder.
- services_test.go: TestServer_CommitBackfillResult (V3 happy, V2 happy,
mixed/partial failure, success=false, bad JSON, pre-validation-all-fail,
unhealthy). Extend BatchUpdateManifest callback tests with V2 and
mixed-item scenarios.
- meta_test.go: TestUpdateSegmentColumnGroupsOperator (add, replace
in-place, child-field stripping, not-found, DataVersion monotonicity).
- management_test.go: proxy handler contract (missing param, success,
rpc error, downstream status error).
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #47314
- pkg/config: AlterConfigsInEtcd now proactively refreshes the local
EtcdSource after etcd write succeeds, so callers reading paramtable in
the same process see the new values immediately (previously relied on
async etcd-watch refresher, causing HandleAlterConfig to return before
local paramtable was updated).
- querycoordv2: add GetLeakedResourcesByCollection which reports
segments/ channels still held by querynodes no longer in any replica of
a collection — true signal that physical resources are being released
during scale-down.
- coordinator/restful_replica: compliance check now returns NotReady
until GetLeakedResourcesByCollection reports zero, so compliance Ready
truly means "safe to terminate removed nodes" rather than just
meta-level convergence.
- Unit tests: 5 cases for GetLeakedResourcesByCollection and 2 cases for
leaked-resource compliance branch; update existing Ready-path tests to
mock GetLeakedResourcesByCollection.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary
- Refactor snapshot name uniqueness from global to per-collection scope
- Add cascade delete: DropCollection triggers DropSnapshotsByCollection
- Add orphan snapshot GC for deleted collections
- Add database-level filtering for ListSnapshots and
ListRestoreSnapshotJobs
- Distinguish source/target collection in RestoreSnapshot API
- Move snapshot privileges from Global level to Collection level
- Update client SDK and documentation for new API semantics
issue: #44358
issue: #47890
issue: #47883
issue: #47855
## Test plan
- [x] Unit tests for snapshot_meta (DropSnapshotsByCollection,
per-collection isolation, partial failure)
- [x] Unit tests for snapshot_manager (DropSnapshotsByCollection,
getDBCollectionIDs)
- [x] Unit tests for services (ListSnapshots/ListRestoreJobs with dbID,
RestoreSnapshot with source collectionID)
- [x] Unit tests for ddl_callbacks_snapshot (new
dropSnapshotsByCollection callback)
- [x] Unit tests for garbage_collector (orphan snapshot GC)
- [x] E2E tests for cross-database snapshot isolation
- [ ] CI validation
## Note on skipped Python E2E tests
All 18 snapshot test classes in
`tests/python_client/milvus_client/test_milvus_client_snapshot.py` are
temporarily skipped with `@pytest.mark.skip`. Reason: this PR changes
snapshot APIs (DropSnapshot, DescribeSnapshot, RestoreSnapshot) to
require `collection_name` as a mandatory parameter, but the pymilvus SDK
used in CI has not been updated to pass this parameter yet. The tests
will be re-enabled once pymilvus SDK is updated to match the new API
contract.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20251114-snapshot_design.md
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: #45881
## Summary
Dead RPC cleanup:
- Remove `CreateExternalCollection` RPC from DataCoord proto; the stub
never served traffic because external collection creation goes through
RootCoord's `CreateCollection` flow.
- Remove DataCoord proto messages unused by any caller:
`CreateExternalCollectionResponse`, `QueryExternalCollectionRequest`,
`DropExternalCollectionRequest`.
- Remove IndexCoord proto message `UpdateExternalCollectionTask`; the
three Catalog methods using it
(`List/Save/DropUpdateExternalCollectionTask`) had no business-code
callers.
- Remove matching stub implementations from mix_coord, mixcoord
client/service, DataCoord services, and metastore catalog/kv_catalog.
- Regenerate protobuf / mockery output to match cleaned interfaces.
Doc cleanup:
- Delete `docs/external_table_load_query_design.md`; superseded by the
updated fake-binlog / schemaless reader design for external tables.
## Test plan
- [x] `make generated-proto-without-cpp` regenerates cleanly
- [x] Build passes (`make milvus`)
- [ ] CI: unit tests, integration tests, e2e
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
QueryNodes embedded in StreamingNodes are now correctly labeled as
"streamingnode" in the GetMetrics system_info response, based on the
STREAMING-EMBEDDED session label.
issue: #48618
Signed-off-by: chyezh <chyezh@outlook.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
issue: #47831
GetAllStreamingNodes now returns all nodes (including frozen) for REST
API listing, while GetAvailableStreamingNodes filters frozen nodes for
scheduling use (replica observer, query node ID lookup).
Signed-off-by: chyezh <chyezh@outlook.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary
Optimize standalone MixCoord recovery to reduce startup time by
parallelizing independent operations:
- **Parallelize DataCoord & QueryCoord startup**: both only depend on
RootCoord being ready, run them concurrently via errgroup
- **Parallelize DataCoord sub-meta loading**: load all 6 sub-metas
(indexMeta, analyzeMeta, partitionStatsMeta, compactionTaskMeta,
statsTaskMeta, snapshotMeta) concurrently alongside reloadFromKV
- **Parallelize indexMeta reload**: run ListIndexes and
ListSegmentIndexes concurrently since they update independent data
structures (m.indexes vs m.segmentIndexes/m.segmentBuildInfo)
- **Batch delete TargetManager targets**: replace per-collection
RemoveCollectionTarget calls with single RemoveWithPrefix
- **Batch etcd loading for ListCollections**: replace sequential per-key
reads with single LoadWithPrefix in RootCoord's initMetaTable
issue: #47783
## Test Plan
- [x] Unit tests pass for all changed packages (index_meta, meta,
kv_catalog, target_manager)
- [x] Race detector clean (`go test -race`)
- [x] Coverage: core functions (reloadFromKV, newIndexMeta,
RemoveCollectionTargets) at 100%; batch load functions at 81-93%
---------
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
## Summary
Removes the `IsTriggerKill` / SIGINT block from
`datacoord.stopServiceWatch()` and `querycoordv2.watchNodes()`.
### Root Cause
In MixCoord mode all three coordinators share the same etcd `Session`
object. During shutdown, when any coordinator calls `session.Stop()` it
cancels the shared context, closing the other coordinators' etcd watches
— triggering `stopServiceWatch()` / `watchNodes()` which sent SIGINT to
the process during a normal, coordinated teardown.
### Why remove SIGINT?
- `go s.Stop()` already handles unexpected session loss — SIGINT was
just a backstop in case `Stop()` hangs
- No other component (rootcoord, proxy, datanode, querynode) has this
logic
- The false-positive risk (killing the process during normal shutdown)
outweighs the marginal benefit
issue: #48242
---------
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
issue: #47790
Replace 131 occurrences of fmt.Sprint(paramtable.GetNodeID()) with the
existing paramtable.GetStringNodeID() across 40 files. This avoids
redundant fmt.Sprint allocations on every call since GetStringNodeID()
uses the more efficient strconv.FormatInt. In the search hot path alone,
this eliminates 3+ string allocations per segment per query.
Signed-off-by: lyang24 <lanqingy93@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
issue: #47647
Refactor the cluster-level broadcast mechanism to decouple the message
package from the channel registration lifecycle:
- Replace internal provider pattern with opaque ClusterChannels type
passed externally to WithClusterLevelBroadcast()
- Add channel package singleton (syncutil.Future) exposing
GetClusterChannels() and GetPChannelNames() blocking accessors
- Add PChannel() interface to MutableMessage/ImmutableMessage for
deriving physical channel from virtual channel
- Validate non-control-channel entries are physical channels using
funcutil.IsPhysicalChannel and use funcutil.IsOnPhysicalChannel for
control channel matching
- Move control channel substitution logic into WithClusterLevelBroadcast
to simplify callers (datacoord, coordinator, assignment service)
- Add lock interceptor unit tests and cluster broadcast test coverage
- Add integration test for FlushAll with streaming node restart to
verify data integrity across node lifecycle
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260105-external_table.md
issue: #45881
This change introduces manual refresh capability for external
collections, allowing users to trigger on-demand data synchronization
from external sources. It replaces the legacy update mechanism with a
more robust job-task hierarchy and persistent state management.
Key changes:
- Add RefreshExternalCollection, GetRefreshExternalCollectionProgress,
and ListRefreshExternalCollectionJobs APIs across Client, Proxy,
and DataCoord
- Implement ExternalCollectionRefreshManager to manage refresh jobs
with a 1:N Job-Task hierarchy
- Add ExternalCollectionRefreshMeta for persistent storage of jobs and
tasks in the metastore
- Add ExternalCollectionRefreshChecker for task state management and
worker assignment
- Implement ExternalCollectionRefreshInspector for periodic job
cleanup
- Use WAL Broadcast mechanism for distributed consistency and
idempotency
- Replace legacy external_collection_inspector and update tasks with
the new refresh-based implementation
- Add comprehensive unit tests for refresh job lifecycle and state
transitions
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260105-external_table.md
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #46358
Add a new BatchUpdateManifest API that allows updating manifest versions
for multiple segments in a single request. The update is broadcast via
the streaming WAL to ensure consistency across the cluster. This
includes the proto definitions, proxy task, datacoord service handler,
meta operator, streaming message type registration, and associated
tests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #47281
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260131-client_side_telemetry.md
Implement comprehensive client-side telemetry system that includes:
- Metrics collection for Search, Query, Insert, Delete, Upsert
operations
- Automatic heartbeat reporting to server with configurable intervals
- Per-collection metrics tracking with wildcard support
- P99 latency calculation using ring buffer sampling
- Server-push command handling (push_config, collection_metrics,
show_errors)
- Historical snapshot storage for latency history queries
- Error tracking with circular buffer for recent errors
- WebUI for telemetry visualization and command management
- HTTP API endpoints for telemetry data access and command push
- RootCoord telemetry manager for centralized command routing
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
relate: https://github.com/milvus-io/milvus/issues/41424
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Pull Request Summary
**Core Invariant**: File resources must physically exist in storage
before being registered in metadata; this PR enforces the invariant at
the entry point (AddFileResource) by validating file existence via a
storageClient abstraction before any metadata mutation.
**Enhancement: Pre-validation of file resource existence**
- Introduces a `storageClient` interface (`Exist(ctx, filePath) (bool,
error)`) that FileResourceManager uses to validate storage presence
before registration
- AddFileResource now checks file existence and returns an InputError if
the file is missing, preventing stale/phantom resource entries in
metadata
- Previously, resources could be registered without verifying the
underlying file existed, creating consistency issues when storage
operations failed or files were deleted externally
**Removes redundant synchronization boilerplate**
- Consolidates QC/DN synchronization logic into FileResourceManager;
AddFileResource and RemoveFileResource methods now internally call
Notify and syncQcFileResource(ctx) instead of repeating this pattern in
services.go
- services.go no longer needs explicit ListFileResource +
SyncQcFileResource calls after add/remove operations; it now simply
delegates to fileManager.AddFileResource/RemoveFileResource
- Proxy layer implementations (AddFileResource, RemoveFileResource,
ListFileResources) now forward to real mixCoord methods instead of "not
implemented" stubs, making the feature fully functional
**No data loss or behavior regression**
- Storage existence check is gated by qnMode == SyncMode (for QC sync)
and dnMode == SyncMode (for DN sync loop startup), preserving backward
compatibility for non-sync deployments
- Resources are only added to metadata after existence confirmation,
ensuring consistency; removal remains symmetric and does not require
existence checks
- The mode guards in Start() and Notify() ensure synchronization only
occurs when configured, preventing spurious sync attempts in disabled
modes
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
issue: #44358
Implement complete snapshot management system including creation,
deletion, listing, description, and restoration capabilities across all
system components.
Key features:
- Create snapshots for entire collections
- Drop snapshots by name with proper cleanup
- List snapshots with collection filtering
- Describe snapshot details and metadata
Components added/modified:
- Client SDK with full snapshot API support and options
- DataCoord snapshot service with metadata management
- Proxy layer with task-based snapshot operations
- Protocol buffer definitions for snapshot RPCs
- Comprehensive unit tests with mockey framework
- Integration tests for end-to-end validation
Technical implementation:
- Snapshot metadata storage in etcd with proper indexing
- File-based snapshot data persistence in object storage
- Garbage collection integration for snapshot cleanup
- Error handling and validation across all operations
- Thread-safe operations with proper locking mechanisms
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
- Core invariant/assumption: snapshots are immutable point‑in‑time
captures identified by (collection, snapshot name/ID); etcd snapshot
metadata is authoritative for lifecycle (PENDING → COMMITTED → DELETING)
and per‑segment manifests live in object storage (Avro / StorageV2). GC
and restore logic must see snapshotRefIndex loaded
(snapshotMeta.IsRefIndexLoaded) before reclaiming or relying on
segment/index files.
- New capability added: full end‑to‑end snapshot subsystem — client SDK
APIs (Create/Drop/List/Describe/Restore + restore job queries),
DataCoord SnapshotWriter/Reader (Avro + StorageV2 manifests),
snapshotMeta in meta, SnapshotManager orchestration
(create/drop/describe/list/restore), copy‑segment restore
tasks/inspector/checker, proxy & RPC surface, GC integration, and
docs/tests — enabling point‑in‑time collection snapshots persisted to
object storage and restorations orchestrated across components.
- Logic removed/simplified and why: duplicated recursive
compaction/delta‑log traversal and ad‑hoc lookup code were consolidated
behind two focused APIs/owners (Handler.GetDeltaLogFromCompactTo for
delta traversal and SnapshotManager/SnapshotReader for snapshot I/O).
MixCoord/coordinator broker paths were converted to thin RPC proxies.
This eliminates multiple implementations of the same traversal/lookup,
reducing divergence and simplifying responsibility boundaries.
- Why this does NOT introduce data loss or regressions: snapshot
create/drop use explicit two‑phase semantics (PENDING → COMMIT/DELETING)
with SnapshotWriter writing manifests and metadata before commit; GC
uses snapshotRefIndex guards and
IsRefIndexLoaded/GetSnapshotBySegment/GetSnapshotByIndex checks to avoid
removing referenced files; restore flow pre‑allocates job IDs, validates
resources (partitions/indexes), performs rollback on failure
(rollbackRestoreSnapshot), and converts/updates segment/index metadata
only after successful copy tasks. Extensive unit and integration tests
exercise pending/deleting/GC/restore/error paths to ensure idempotence
and protection against premature deletion.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: #46451
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Session versioning added to validate coordinator compatibility during
registration and active takeover.
* **Changes**
* Active–standby flow simplified: standby-to-active activation now
always enabled and initialized unconditionally.
* Registration uses version-aware transactions to ensure version
consistency during takeover.
* Startup/health startup path streamlined.
* **Tests**
* Added version-key integration test; removed test for disabling
active-standby.
* Updated flush test to assert rate-limiter errors occur.
* **Chores**
* Removed centralized connection manager and its test suite.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: chyezh <chyezh@outlook.com>
issue: https://github.com/milvus-io/milvus/issues/45890
ComputePhraseMatchSlop accepts three pararms:
1. A string: query text
2. Some trings: data texts
3. Analyzer params,
Slop will be calculated for the query text with each data text in the
context of phrase match where they are tokenized with tokenizer with
analyzer params.
So two array will be returned:
1. is_match: is phrase match can sucess
2. slop: the related slop if phrase match can sucess, or -1 is cannot.
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
relate:https://github.com/milvus-io/milvus/issues/43687
Support use file resource with sync mode.
Auto download or remove file resource to local when user add or remove
file resource.
Sync file resource to node when find new node session.
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
issue: https://github.com/milvus-io/milvus/issues/45691
Add persistent task management for external collections with automatic
detection of external_source and external_spec changes. When source
changes, the system aborts running tasks and creates new ones, ensuring
only one active task per collection. Tasks validate their source on
completion to prevent superseded tasks from committing results.
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
issue: #45728
When mixcoord is in standby mode and shutdown is triggered, the
ProcessActiveStandBy goroutine may panic if context cancellation occurs.
This happens because the error handling didn't check for
context.Canceled errors before panicking.
Changes:
- Add context cancellation check in mix_coord Register() before panic
- Check s.ctx.Err() == context.Canceled and gracefully exit
- Remove unused ForceActiveStandby() function from session_util
This ensures standby mixcoord can shutdown gracefully without panic when
context is cancelled during the standby process.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: #43897
- Part of collection/index related DDL is implemented by WAL-based DDL
framework now.
- Support following message type in wal, CreateCollection,
DropCollection, CreatePartition, DropPartition, CreateIndex, AlterIndex,
DropIndex.
- Part of collection/index related DDL can be synced by new CDC now.
- Refactor some UT for collection/index DDL.
- Add Tombstone scheduler to manage the tombstone GC for collection or
partition meta.
- Move the vchannel allocation into streaming pchannel manager.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
relate: https://github.com/milvus-io/milvus/issues/43687
We used to run the temporary analyzer and validate analyzer on the
proxy, but the proxy should not be a computation-heavy node. This PR
move all analyzer calculations to the streaming node.
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
issue: #44730
Fix the issue where logs were not outputting as expected due to
incorrect log package imports across multiple components.
Changes include:
- Add golangci-lint rule to forbid github.com/pingcap/log usage
- Replace github.com/pingcap/log with
github.com/milvus-io/milvus/pkg/v2/log
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: #43897
- Return LastConfirmedMessageID when wal append operation.
- Add resource-key-based locker for broadcast-ack operation to protect
the coord state when executing ddl.
- Resource-key-based locker is held until the broadcast operation is
acked.
- ResourceKey support shared and exclusive lock.
- Add FastAck execute ack right away after the broadcast done to speed
up ddl.
- Ack callback will support broadcast message result now.
- Add tombstone for broadcaster to avoid to repeatedly commit DDL and
ABA issue.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
