mirror of
https://github.com/milvus-io/milvus.git
synced 2026-07-21 10:15:43 +00:00
master
981
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
e8b9dbff2f |
fix: fill collection_name and keep request id in DescribeCollection queried by id (#51313)
issue: #51253 > Rebased on master after #51254 merged. The `db_name`/`db_id` part of the original PR is now covered by #51254; this PR is reduced to the two remaining gaps on the same query-by-id path. ## Problem When `DescribeCollection` is called with **only a `collectionID`** — no collection name, no db name (e.g. the HTTP management API on `:9091`) — two gaps remain after #51254: 1. **Top-level `collection_name` is empty** on both the cached provider path and the `describeCollectionTask` path. The response is initialized from the (empty) `request.CollectionName` before the name is resolved, and never backfilled. 2. **The cached provider discards the caller-provided collection id.** After deriving the name from the id, it re-resolves the id from that name via `GetCollectionID(db, name)`. Id-only lookups are cached under the request db name — empty here — so with same-name collections in different databases, a concurrent cache refresh can rebind the entry under the `""` key and the request would silently describe the wrong collection (the derived name matches, the id does not). ## Fix - `service_provider.go`: - Backfill `resp.CollectionName` from the resolved cached schema when the request carried no name. Requests that pass a name (including aliases) still echo it unchanged. - Skip the name→id re-resolution when the caller already supplied the id. `GetCollectionInfo` already validates the cache entry against the id (`collInfo.collID != collectionID` → refresh by id), so the caller-provided id is authoritative end to end. Name+id requests keep the existing name-precedence behavior. - Resolve identifiers on local copies instead of rewriting `request.CollectionName`/`CollectionID` in place — the access log interceptor and the success metric labels serialize the request after the handler returns, and previously recorded the resolved values instead of what the client sent. - `task.go`: backfill `t.result.CollectionName` from the coordinator result on the non-cached path. - `meta_cache.go`: fix a stale comment on `ResolveCollectionAlias` — `update()` now always keys `collInfo` by the real collection name (aliases live in the separate alias map), the old comment described pre-refactor behavior. - Tests: - `TestCachedProxyServiceProvider_DescribeCollection_ByIDFillsNameAndUsesRequestID`: drives the id-only path; `GetCollectionID` is deliberately not mocked, so any regression back to re-resolving the id panics the test. - `TestDescribeCollectionTask_FillsNameFromResultWhenQueriedByID`: same assertion for the non-cached path. - Removed the now-unused `GetCollectionID` expectation from the test added in #51254 (that call no longer happens on the id-only path). ## Follow-up commit: entity-keyed meta cache with a cluster-wide id index Reviewing this path surfaced a deeper issue in the proxy meta cache, fixed in the second commit: - By-id lookups (`GetCollectionName`, `GetCollectionInfo` with an empty name) linearly scanned a whole db bucket under the read lock, on every request even on cache hits. - The bucket scanned/filled was keyed by the *request* db name — empty for id-only calls — so entries landed in a bogus `""` bucket: a collection also cached under its real db was duplicated, and same-name collections of different databases evicted each other from the single `""`-keyed slot. The cache is now entity-keyed instead of request-keyed: - Fills land under the collection's **actual database** (carried in the describe response); the `""` bucket no longer exists. - A cluster-wide `collectionID → entry` index serves by-id lookups in **O(1)** regardless of the request db — matching rootcoord, which resolves by-id describes straight from `collID2Meta` without consulting the db name. - Name lookups normalize an empty db name to `default`, mirroring rootcoord's backward-compat normalization (`meta_table.getCollectionByNameInternal`). This also closes a latent cross-database mis-hit: a name lookup with an empty db could previously return whichever same-name collection was last id-cached under the `""` bucket. - All removal paths maintain the index (pointer-identity-guarded unindexing); invalidation sweeps stay exhaustive. ## Verification - Ran locally against a current master core build: the full `TestMetaCache*`, `TestCachedProxyServiceProvider*`, `TestDescribeCollectionTask*`, alias and partition test sets all pass, plus the **full `internal/proxy` suite** (only the pre-existing etcd-dependent `TestProxyRpcLimit` fails locally — it needs a live etcd, unrelated to this change). - New tests: `TestCachedProxyServiceProvider_DescribeCollection_ByIDFillsNameAndUsesRequestID` (id-only path; also asserts the request is not rewritten), `TestDescribeCollectionTask_FillsNameFromResultWhenQueriedByID`, `TestMetaCache_ByIDIndexRealDBBucket` (same-name collections in two databases filled by id: no eviction, entries under real dbs, by-name reuses by-id fill, every removal path cleans the index), `TestMetaCache_EmptyDBNameSharesDefaultEntry`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- ## Update: proxy meta cache rebuilt around an id-primary store (2 new commits) Following review discussions on invalidation cost and the alias-DDL races, this PR now also rebuilds the cache (issue: #51533, design doc: `docs/design-docs/design_docs/20260716-proxy-metacache-id-inverted-index.md` in this PR): 1. **`fix: forward the pre-alter alias target id in the AlterAlias expiration`** — rootcoord resolves the pre-alter target under its database lock and carries it in the new additive `AlterAliasMessageHeader.old_collection_id`; the ack callback emits a second expiration entry so proxies evict BOTH AlterAlias targets by id. Closes the race where a concurrent Describe re-points the proxy's alias resolution before the expiration arrives, leaving the old target permanently stale. Older rootcoords (field 0) fall back to the proxy's hint resolution. 2. **`enhance: rebuild the proxy meta cache around an id-primary store`** — the primary store is keyed by the cluster-unique collection id (single source of truth, with a per-database generation); name/alias resolution become hints validated against the primary on read; partition caches are keyed by id; fills are ordered against invalidations by a fill RWMutex (drain in-flight describes before evicting), replacing the per-collection timestamp floor. Every invalidation becomes O(1) — no more full-cache scans under the write lock on Load/Release/Drop/Rename/Alter broadcasts, and DropDatabase/AlterDatabase becomes a generation bump. Also removes a latent stale read (alias-keyed partition entries surviving DropCollection). Verification: targeted cache suites green (rename, alias re-point under concurrent describe, drop+recreate with a reused name, db-generation invalidation, cross-db isolation, gated-mock fill/invalidation ordering); three negative controls (hint validation / dbGen check / fill drain disabled) each fail exactly the test guarding them; full-package `-race` delegated to CI. --- ## Update: simplification series (final form) Following design review, the cache was iteratively simplified to an **id-primary store with declared hints** — every mechanism that could be replaced by an invariant was deleted (net-negative diffs throughout): - Primary store keyed by the cluster-unique collection id; liveness IS presence. Name/alias resolution are hints written ONLY together with their entry (declared aliases) and validated against the primary on every read — a stale hint can only cost one extra describe, never a stale read. - Fill/invalidation ordering via a fill RWMutex (drain in-flight describes before evicting); the per-collection timestamp floor, database generations, background GC, alias negative cache, per-partition cache, and the resolve-and-forget DescribeAlias path are all **deleted**. Evictions clean everything the entry owns synchronously; **no invalidation path performs any RPC**. - Old-rootcoord fallbacks: id-describes without DbName are served uncached; alias-DDL broadcasts without ids trigger hint resolution plus a holder scan **gated on that exact fingerprint** (dead code once rootcoords are upgraded), closing the ghost-alias case with no healing event. - **Accepted gaps (WONT-FIX)** are recorded in the design doc §6 — notably the upgrade-window new-target `Aliases` display lag (the association exists only at rootcoord; self-heals on first use; display-only). See `docs/design-docs/design_docs/20260716-proxy-metacache-id-inverted-index.md` for the full design, invariants and trade-offs. Signed-off-by: xiaofanluan <xf@hjjaq.com> Co-authored-by: xiaofanluan <xf@hjjaq.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
congqixiaandGitHub
|
c8dd73314c |
enhance: [GoSDK] migrate Go client module path to v3 (#51539)
Related to #51419 Update the Go client module path, imports, dependencies, tests, examples, and documentation from client/v2 to client/v3. Set the SDK version to 3.0.0 in preparation for the Milvus 3.0 release. Signed-off-by: Congqi Xia <congqi.xia@zilliz.com> |
||
|
|
0d2637fd63 |
fix: keep proxy metric status label stable, split cause into its own label (#51495)
issue: #51493 ### What #50221 split the `status` label values of `milvus_proxy_req_count` / `milvus_proxy_grpc_latency` in place. Released in **v2.6.19**, that silently zeroes every existing dashboard panel and alert rule matching `status="fail"` / `status="rejected"` — an alert that stops firing rather than erroring. This keeps the classification but moves it onto its own dimension, restoring the status domain to what <= v2.6.18 emitted: ``` status: success | fail | rejected | retry | total | abandon (as before v2.6.19) cause: user | system | cancel | na (new) ``` | v2.6.19 / v2.6.20 | this PR | |---|---| | `status="fail_input"` | `status="fail", cause="user"` | | `status="fail_system"` | `status="fail", cause="system"` | | `status="rejected_user"` | `status="rejected", cause="user"` | | `status="rejected_system"` | `status="rejected", cause="system"` | | `status="cancel"` | `status="fail"` or `"rejected"`, `cause="cancel"` | | `success` / `retry` / `total` / `abandon` | unchanged, `cause="na"` | ### Why a label instead of new status values - **Old queries work unchanged and count each request once.** `status="fail"` is again every hard failure; Prometheus aggregates over `cause` for free. That rollup is what a label dimension *is* — the split forces every consumer who just wants "how many failures" to hand-write `status=~"fail_.*"`. - **Cardinality is unchanged.** `cause` is functionally dependent on the outcome, so the realized `(status, cause)` pairs are exactly the series the split already produces. Additive, not multiplicative. - **New consumers lose nothing**: alert on `status="fail", cause="system"`; route `cause="user"` to the owning application team. The alternative — emitting old and new `status` values side by side during a transition — was rejected: it defers the break rather than removing it (the old values must still be dropped eventually, forcing the same migration later), and meanwhile double-counts every request in unfiltered aggregations. `cancel` is folded into `cause` for the same reason: before v2.6.19 client cancellations were counted as `fail` (cancel in the response status) or `rejected` (cancel at the interceptor), so promoting it to a `status` value quietly makes `status="fail"` under-count versus older releases. As a cause it stays excludable via `cause!="cancel"` without redefining `status`. ### Also in this PR - The 7 `snapshot_impl` sites that emitted a bare `fail` with no classification now report their real cause via `failMetricLabel(err)` (e.g. `snapshot_metadata_uri is required` is `cause="user"`, not a system fault). Their `status` is unchanged. - `deployments/monitor/grafana/milvus-dashboard.json`: the "Faild Request Rate" panel goes back to `status="fail"` — a verbatim revert of what #50221 changed, which is the compatibility claim demonstrated. - Dev docs and stale comments that named the old label values. ### Verification - `TestParseMetricLabelStatusDomainIsStable` pins the status domain, so re-splitting it fails CI rather than shipping. - Adding a label makes every `WithLabelValues` site arity-sensitive **at runtime, not compile time** — a missed site panics in production. Unit tests do not reach all of them (coverage shows `GetRestoreSnapshotState`, `ListRestoreSnapshotJobs`, `PinSnapshotData`, `UnpinSnapshotData` at 0%), so all **58 emit sites were verified statically via AST**, not only the ones tests happen to hit. - Passing: `pkg/metrics`, `pkg/util/requestutil`, `pkg/util/merr`, `internal/distributed/proxy` (incl. `httpserver`, which covers the REST emit path), and the `internal/proxy` snapshot / metric-label tests. `golangci-lint` clean on every touched package. - Pre-existing and unrelated: `service_test.go` bind failures (`listen tcp :19529: address already in use`) reproduce identically on unmodified master — a local process holds the port. ### Rollout Should be cherry-picked to 2.6 so the window in which the fine-grained `status` values exist stays confined to v2.6.19–v2.6.20. Users who already adopted the new values on those two releases need a one-time query change (`status="fail_system"` -> `status="fail", cause="system"`); that is a known, bounded set, and preferable to leaving every pre-2.6.19 dashboard silently broken. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
f662cec16d |
feat: add local format metadata and split policy (#51204)
relate: #50304 ## Summary Add the local format design doc, use the storage writer format constant, and introduce local_format metadata propagation with column-group split policy support. --------- Signed-off-by: aoiasd <zhicheng.yue@zilliz.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Zhen YeandGitHub
|
6b3eabb1ff |
fix: skip unreplicable replicated ddl (#51454)
issue: #51446 - message properties: add an explicit `Unreplicable` marker for concrete WAL messages and expose builder opt-in through `WithUnreplicable` - replication: skip unreplicable messages in the CDC sender and secondary replicate interceptor before callback replay - DDL producers: mark unsupported snapshot, manifest backfill, and external collection refresh broadcasts as unreplicable - streaming docs: document the message-level skip marker and current unsupported DDL replication behavior Signed-off-by: chyezh <chyezh@outlook.com> |
||
|
|
25ac6b4c38 |
fix: correct the CDC replication lag metric (#51049)
## Summary The documented CDC lag PromQL subtracted two independently scraped time-tick gauges, which caused idle oscillation and misleading restart behavior. This PR fixes the query and hardens the lag metric lifecycle without introducing a new metric. ### #50633 — idle oscillation Subtract each gauge timestamp before calculating the difference, canceling the cross-target scrape-phase offset: ```promql clamp_min( max by (channel_name) (milvus_wal_last_confirmed_time_tick - timestamp(milvus_wal_last_confirmed_time_tick)) - min by (channel_name) (milvus_cdc_last_replicated_time_tick - timestamp(milvus_cdc_last_replicated_time_tick)), 0) ``` ### #51048 — restart and switchover lag lifecycle - Seed `milvus_cdc_last_replicated_time_tick` from `InitializedCheckpoint` before target-client initialization, so target-unavailable startup exposes a conservative lag instead of an absent series. - Overwrite the seed with the target-confirmed checkpoint after `GetReplicateInfo` succeeds. - Ignore nil and zero checkpoints to avoid creating an epoch-valued series. - Remove lag-series deletion from generic stream `OnClose`. - Delete the lag series only from `ReplicateManager.RemoveReplicator`, the genuine etcd DELETE path. Outdated revision cleanup therefore cannot delete the live revisions shared label pair. issue: #50633 issue: #51048 ### Test Plan - [x] Added unit-test coverage for initialized-checkpoint seeding and nil/zero guards. - [x] Added regression coverage for genuine removal versus outdated revision cleanup. - [ ] Local Go tests and static check. - [ ] Confirm the lag lifecycle on a live switchover environment. --------- Signed-off-by: Yihao Dai <yihao.dai@zilliz.com> Signed-off-by: bigsheeper <yihao.dai@zilliz.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
5eebaa9ad4 |
enhance: add WAL trace propagation (#50796)
issue: #47404 - message trace context: add trace context serialization and restore/inject helpers for WAL messages and msgstream conversion - WAL append trace: normalize WAL spans for autocommit, txn, broadcast, append, appendimpl, and broadcast callback paths - trace propagation: restore message trace context in producer, broadcast retry, replicate primary/secondary, recovery, flusher, and query/data flowgraph consumers --------- Signed-off-by: chyezh <chyezh@outlook.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
congqixiaandGitHub
|
57b314cb48 |
enhance: move JSON indexes into sealed segment runtime state (#51323)
Related to #51068 - manage JSON index load, replace, and drop through reopen COW snapshots - remove LoadScope_Index and the legacy QueryCoord-to-segcore update path - drop JSON indexes by field and nested path to preserve sibling indexes - regenerate query proto and QueryNode mocks --------- Signed-off-by: Congqi Xia <congqi.xia@zilliz.com> |
||
|
|
9fe38b1f09 |
doc: add design document for online shard split of namespace collections (#50465)
Add the design document for online shard split of namespace (multi-tenant) collections. The design covers: - Range routing over `big_endian(hash(namespace)) || namespace` with a versioned routing table derived from the collection meta. - Write switching via a `SplitShard` WAL fence message (`T_switch`) with reject-and-refetch on the proxy; target vchannels are initialized with a `BarrierTimeTick` so every message of the new WALs is strictly greater than `T_switch`. - In-place child delegators fronted by the source delegator during the transition window (sealed segments stay single-loaded, deletes and timeticks are forwarded back). - Multi-round metadata-only relabel redistribution by DataCoord, with the release-safety analysis (frozen targets, merged recovery view, register-then-release). - One-shot QueryCoord adoption with in-place delegator handoff, consistency guarantees, engineering constraints, configuration, and failure handling. issue: #50463 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: cai.zhang <cai.zhang@zilliz.com> Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
junjiejiangjjjandGitHub
|
694c2e6dba |
feat: support xgboost function chain expr (#51195)
issue: #51192 design doc: docs/design-docs/design_docs/20260708-xgboost-function-chain.md Add native xgboost FunctionChain expression support for L0 rerank with FileResource-backed UBJ models. This change includes: - xgboost FunctionChain expression registration, parameter validation, and execution - FileResource-based UBJ model discovery and local path resolution - lazy model loading with singleflight, lease/refcount lifecycle protection, and stale eviction on FileResource sync - cgo bridge for Arrow C Data based batch prediction - native C++ UBJ model parser and predictor for supported tree models - runtime-disabled stub for builds without cgo and with_xgboost - validation for unsupported params, output modes, feature count mismatch, invalid models, unsupported objectives, unsupported boosters, multiclass models, multi-target leaf vectors, and unsupported input column types - C++ unit tests, Go tests, native parity tests, and Python client L0 E2E tests - xgboost FunctionChain design document L2 rerank support is intentionally deferred because Proxy does not yet support FileResource sync and local resolution. Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com> |
||
|
|
0f485f28c1 |
doc: add vector-anchored join design doc (#51186)
## What
Adds the design doc for the **vector-anchored join** feature:
`docs/design-docs/design_docs/20260708-vector-anchored-join.md`. Doc
only — no code.
## Why
Milvus has no first-class way to join a vector collection against a
*separate, frequently-updated* scalar table keyed by a shared identifier
("vectors are stable, their business metadata mutates constantly").
Today users must denormalize the mutable fields into the vector
collection (re-writing the vector on every metadata change) or join in
the application layer (losing all pushdown). This design puts the join
back inside the engine without coupling mutable data to the immutable
vector-segment lifecycle.
## Scope
Three join shapes, all keeping vector search on the critical path:
| # | Capability | SQL semantics | Physical operator |
|---|---|---|---|
| 1 | Vector-driven enrichment | `INNER` / `LEFT OUTER` | index
nested-loop; vector top-K drives, probe side by key |
| 2 | External-predicate pre-filter | `SEMI` | foreign filter → key-set
→ bloom/bitmap inline into ANN |
| 3 | Lateral vector search (driver-side) | `LATERAL` | scalar table
drives; one batched ANN (`nq = N`) per driver row |
Built on a first-class **scalar-only (non-vector) table**, a **Global KV
Index** (mutable, LSM, decoupled from vector segments), and
**generalized bloom-filter pushdown**. Delivery is **baseline-first**
and not gated on the KV index (Phase 0 is doable on existing access
paths).
Governing principle: *Milvus does vector-anchored joins, not general
joins.* Two scalar-only tables joined together, unbounded all-pairs KNN,
non-equi/range joins, and `FULL`/`RIGHT OUTER`/`CROSS` are explicitly
out of scope.
issue: #51185
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
wei liuandGitHub
|
760407a17e |
enhance: support external JSON stats for external tables (#51008)
issue: #45881 ## What changed This PR enables JSON key stats / JSON shredding support for external StorageV3 segments that already have a committed manifest. The main behavior changes are: - Allow DataCoord to schedule and reload `JsonKeyIndexJob` for external StorageV3 manifest-backed segments. - Add `StatsResult.base_manifest` so DataCoord can reject stale Text and JSON stats results when an external refresh has already moved the segment to a newer manifest. - Keep stale-result CAS and task placeholder release in the common stats task path, so a rejected result does not finish the old task forever. - Best-effort cleanup rejected Text/JSON stats files only for external collections; internal collections keep the existing GC ownership rules. - Clear old `TextStatsLogs` and `JsonKeyStats` metadata during external refresh. - Gate manifest JSON stats exposure on DataCoord metadata placeholders, preventing stale manifest stats from being loaded after refresh. - Keep external BM25 inspector jobs skipped. - Update external table design docs to match the new support boundary. ## Why External table refresh can replace the manifest of the same segment while an older JSON stats task is still running. Without a base-manifest check, a late stats result built from the old manifest could overwrite the refreshed manifest or expose stale JSON stats. External collection segments are patched only when the source changes. If the source stays stable, the segment can remain active forever, so rejected external stats files should be cleaned immediately instead of waiting for dropped-segment GC. ## Validation - `make generated-proto-without-cpp` - `gofmt -w` on touched Go files - `git diff --check` - `git diff --cached --check` - `git diff --check milvus/master..HEAD` - staged mockery addition scan Targeted Go test attempted locally: ```bash source ~/.profile && source scripts/setenv.sh && \ go test -tags dynamic,test -gcflags="all=-N -l" -count=1 \ github.com/milvus-io/milvus/internal/datacoord \ -run 'Test_statsTaskSuite/TestQueryTaskOnWorkerDiscardsStaleStatsResult' \ -v ``` It is blocked by the local C++ link environment: `_SegcoreSetPrefetchThreadPoolNum` is missing from the local linked segcore library. `reset-milvus` also cannot fully restart standalone in this worktree because `bin/milvus` is missing. Signed-off-by: Wei Liu <wei.liu@zilliz.com> |
||
|
wei liuandGitHub
|
097e16af51 |
enhance: Enable channel-level score balancer by default (#51066)
issue: https://github.com/milvus-io/milvus/issues/51065 ## What changed This PR switches the default QueryCoord balancer from `ScoreBasedBalancer` to `ChannelLevelScoreBalancer` and keeps unknown balancer names aligned with the new default fallback. It also sets the default `queryCoord.channelExclusiveNodeFactor` to `3`, so channel exclusive mode is enabled by default only when every channel can average at least three RW QueryNodes. ## Why `ChannelLevelScoreBalancer` improves channel placement, but channel exclusive mode constrains segment balancing inside each channel's assigned node group. If a replica has too many channels for its QueryNode count, enabling exclusive mode can prevent global node-level balancing from smoothing out skewed channel data. The default factor of `3` keeps smaller or high-channel-count replicas on the score-based node-level balancing path until there is enough QueryNode capacity per channel. ## Behavior With the default configuration, QueryCoord enables channel exclusive mode only when: ```text replica.RWNodesCount() >= len(channels) * 3 ``` For example, in an 8 QueryNode replica: - 2 channels can enable channel exclusive mode. - 3 channels cannot enable channel exclusive mode and continue using the score-based balancing fallback. ## Details - Update `configs/milvus.yaml` balancer and exclusive node factor defaults - Update paramtable defaults and default assertions - Make unknown balancer names fall back to `ChannelLevelScoreBalancer` - Add balancer factory coverage for default, fallback, explicit score-based, and explicit round-robin cases - Add replica coverage for the default three-nodes-per-channel threshold - Pin existing tests to legacy `ScoreBasedBalancer` or explicit factors where the old behavior is the scenario under test - Update replica observer tests to provide target channel metadata explicitly - Add and refresh the channel exclusive mode design document under `docs/design-docs` ## Validation - `git diff --check` - `source ~/.profile && source scripts/setenv.sh && GOTOOLCHAIN=local go test -tags dynamic,test -gcflags="all=-N -l" github.com/milvus-io/milvus/pkg/v3/util/paramtable -run 'TestComponentParam' -count=1 -v` - Attempted reset-milvus; Docker daemon was unavailable and this worktree has no `bin/milvus`. - Added `internal/querycoordv2/meta` threshold coverage; local execution is blocked before assertions by stale Cgo output missing `C.SegcoreSetPrefetchThreadPoolNum`. Signed-off-by: Wei Liu <wei.liu@zilliz.com> |
||
|
|
58ca68ce96 |
fix: make max_edit_distance a soft keyword and simplify fuzzy query (#51077)
## Summary Follow-up polish to #50933 (`text_match_fuzzy`). Three related changes: ### 1. `max_edit_distance` is no longer a reserved keyword (fixes #51058) #50933 introduced `max_edit_distance` as a hard ANTLR lexer keyword, which reserves that word across the *entire* expression language — a collection with a scalar field literally named `max_edit_distance` would fail to parse `max_edit_distance > 1` after upgrade. This makes it a **soft keyword**: the grammar accepts a plain `Identifier` in the option slot and `VisitTextMatchFuzzy` validates (case-insensitively) that it is `max_edit_distance`, rejecting any other name. `text_match_fuzzy` itself stays a reserved *operator* keyword, like `text_match` / `phrase_match`. ### 2. Simplify the fuzzy OR `fuzzy_match_query` built its per-token disjunction with `BooleanQuery::with_minimum_required_clauses(subqueries, 1)` (copied from the real-`k` `min_should_match` path). The vendored fork's `BooleanQuery::union(queries)` expresses a plain OR of `Should` clauses directly — semantically identical, one call, no magic `1`. ### 3. Fast path for `max_edit_distance = 0` `K = 0` is exactly a term match, but tantivy's `FuzzyTermQuery` has no `distance == 0` short-circuit — it still builds a Levenshtein automaton per token. Route `K = 0` to the cheaper `match_query` (multiterms) path. ## Tests - New `TestExpr_TextMatchFuzzy_SoftKeyword`: a field named `max_edit_distance` in an ordinary filter parses; case-insensitive `MAX_EDIT_DISTANCE=` works; a wrong option name is rejected. - Existing `TestExpr_TextMatchFuzzy` and the full `planparserv2` suite pass; `cargo check` on the tantivy binding passes. Design doc updated accordingly. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: xiaofanluan <xf@hjjaq.com> Co-authored-by: xiaofanluan <xf@hjjaq.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
congqixiaandGitHub
|
8fda58a889 |
doc: Add design doc for segment reopen atomic read update cow (#51072)
Related to #51068 Add design doc for segment reopen atomic read update cow, which describes the design of atomic read and update for segment reopen with copy-on-write (COW) mechanism. Signed-off-by: Congqi Xia <congqi.xia@zilliz.com> |
||
|
秀吉andGitHub
|
3f2ce12895 |
enhance: support text_match_fuzzy (edit-distance) scalar filter (#50933)
issue: #50920 design doc: [docs/design-docs/design_docs/20260702-text_match_fuzzy.md](docs/design-docs/design_docs/20260702-text_match_fuzzy.md) ## What Adds a scalar filter operator `text_match_fuzzy(field, "query", max_edit_distance=K)` for typo-tolerant (Levenshtein edit-distance) matching on analyzed VARCHAR fields that have a text-match index. Filter-only: it returns a boolean bitset, no scoring (scoring is tracked separately in #50921). `K` is an integer in `[0, 2]` (tantivy's fuzzy automaton hard cap); out-of-range is rejected. ## How Mirrors the existing `text_match` / `phrase_match` path through every layer, reusing the term-dictionary FST and tantivy's `FuzzyTermQuery` (no new index, no new storage): - Tantivy binding: `fuzzy_match_query` tokenizes the query with the field analyzer and ORs one `FuzzyTermQuery` per token; exposed via the `tantivy_fuzzy_match_query` C entry (cbindgen header regenerated). - C++ index: `TextMatchIndex::FuzzyMatchQuery` plus the wrapper, mirroring `MatchQuery`. - proto: `OpType.TextMatchFuzzy = 17` (next free value; 15/16 are already `InnerMatch`/`RegexMatch`). The edit distance rides in the existing `UnaryRangeExpr.extra_values`, like slop / min_should_match, so no new field is added. - grammar + Go parser: a new `text_match_fuzzy(...)` rule and `VisitTextMatchFuzzy`, which validates `K` to `[0, 2]` and stores it in `extra_values`. - C++ executor: routes the op to the text-index path (dispatch, exec-path, and offset-input guards) and calls `FuzzyMatchQuery`, with a `[0, 2]` re-check on the C++ side (the same way `PhraseMatch` validates slop) for plans not built through the parser. ## Tests - Rust unit test: distance 0/1/2, fuzzy-vs-exact, multi-token OR, and the distance `K+1` exclusion boundary. - C++ gtest (`TextMatch.FuzzyIndex`, `GrowingNaive`, `SealedNaive`): index-level typo matching and the `K+1` boundary, plus the executor + growing-segment path — a typo matches the same rows as the exact term, `not text_match_fuzzy(...)`, and the executor guard rejects out-of-range / missing `max_edit_distance`. - Go parser tests: valid 0/1/2 produce the right op and extra value; a templated query preserves the distance; out-of-range / missing / non-integer / overflow / non-string field / match-not-enabled are all rejected. Existing `text_match` and `phrase_match` behavior is unchanged (full planparserv2 suite green). ## Notes - `max_edit_distance` becomes a reserved keyword in filter expressions, consistent with `minimum_should_match` and `threshold`. - Out of scope, left as follow-ups: scoring (#50921), prefix-fuzzy, ES-style `AUTO` fuzziness, and combining with `minimum_should_match`. --------- Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> |
||
|
|
6911c10c8b |
doc: add CDC 2PC import user guide (#51029)
## Summary
Adds a user guide for running a **two-phase-commit (2PC) bulk import**
in a CDC replication topology, under
`docs/design-docs/design_docs/cdc/user_docs/`, and links to it from the
existing CDC guides.
In a replicating cluster, auto-commit imports are rejected; imports must
run in 2PC mode so the commit is a single WAL-ordered fence that
replicates consistently to the standby. The guide covers:
- Enabling `dataCoord.import.enableInReplicatingCluster` on both the
primary and the standby.
- Running the import with `options={"auto_commit": "false"}` on the
primary.
- Waiting until **both** clusters report the `Uncommitted` state (best
practice), then committing once on the primary.
- Verifying the data on both clusters, and the requirement that import
files be present in both clusters' object storage.
issue: #51028
## Changes
**New file:**
- `docs/design-docs/design_docs/cdc/user_docs/05-cdc-2pc-import.md`
**Modified:**
- `01-cdc-replication-overview.md` — FAQ entry linking to the new guide.
- `02-cdc-replication-quick-start.md` — pointer to the new guide.
## Verification
- **SDK level:** the guide's script drives the merged pymilvus helpers
`bulk_import` / `get_import_progress` / `commit_import`
(milvus-io/pymilvus#3538) — correct endpoints, `auto_commit=false`
forwarding, and response parsing confirmed.
- **Live end-to-end:** ran against two CDC-replicating Milvus clusters
(cluster mode, master nightly `master-20260703-331cf54b`). The full flow
worked as documented — both clusters reached `Uncommitted`, a single
`commit_import` on the primary drove both to `Completed`, and 100/100
imported rows were visible on the primary and the standby.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
sijie-ni-0214andGitHub
|
4ba29fcfb3 |
enhance: Update drop function field design doc (#50913)
## Summary - Update the drop collection field design doc with the final drop-function semantics introduced by PR https://github.com/milvus-io/milvus/pull/50471. - Document the split between detach function and drop function field behavior. - Align RootCoord, Proxy, SDK cache invalidation, and testing strategy descriptions with the current implementation. issue: #48983 Design doc: docs/design-docs/design_docs/20260413-drop-collection-field-design.md ## Test Plan - [x] git diff --check upstream/master..HEAD - [x] rg stale drop-function wording in docs/design-docs/design_docs/20260413-drop-collection-field-design.md Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com> |
||
|
wei liuandGitHub
|
4e1bf7a36c |
feat: define external snapshot API contracts (part1) (#50393)
issue: https://github.com/milvus-io/milvus/issues/44358 design doc: docs/design-docs/design_docs/20260609-external-snapshot-export-restore.md Part 1/3 of the external snapshot cross-bucket restore stack. This PR defines the API contract and entry surfaces for external snapshot export and restore: - Adds the consolidated design document for cross-bucket external snapshot restore. - Adds public gRPC, REST, and Go SDK API surfaces for RestoreExternalSnapshot and ExportSnapshot. - Adds internal DataCoord proto plumbing and generated code needed by later implementation PRs. - Wires Proxy RBAC grouping and database interceptor behavior for the new APIs. - Keeps the request contract on a single external_spec field and keeps db_name for namespace routing rather than permission scoping. Validation copied from the commit: - GOTOOLCHAIN=go1.25.10 go test -c -tags dynamic,test -gcflags="all=-N -l" -ldflags="-r ${RPATH}" -o /tmp/datacoord-commit1.test github.com/milvus-io/milvus/internal/datacoord - cd client && GOTOOLCHAIN=go1.25.10 go test -c -o /tmp/client-milvusclient-commit1.test ./milvusclient - internal/proxy package compile was blocked locally by missing C++ header internal/core/output/include/segcore/search_result_export_c.h --------- Signed-off-by: Wei Liu <wei.liu@zilliz.com> |
||
|
junjiejiangjjjandGitHub
|
d540e0d567 |
feat: support function chain API for search rerank (#50786)
issue: https://github.com/milvus-io/milvus/issues/50571 design doc: docs/design-docs/design_docs/20260624-function-chain-api.md Add the FunctionChain proto-to-runtime path for ordinary Search L2 rerank. The change converts public FunctionChain protos into ChainRepr, derives caller-independent read/write metadata, validates Search-specific inputs in Proxy, and executes public chains through the existing rerank pipeline. Add request validation for duplicate stages, unsupported stages, unsupported system inputs/outputs, unknown schema fields, unsupported input field types, and hybrid-search usage. Extend REST v2 request conversion to accept function_chains. Replace score_combine with num_combine and add typed parameter readers, repr-based FuncChain construction, chain optimization/pruning helpers, and model-rerank parameter handling. Add coverage for chain repr conversion, function/operator behavior, Proxy planning, search pipeline integration, REST conversion, and REST API cases. Add the Function Chain API design document and update milvus-proto to the latest upstream pseudo-version. Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com> |
||
|
|
453a6f5193 |
feat: support bitwise operators (&, |, ^) in filter expressions (#50666)
# What Adds support for bitwise AND (`&`), OR (`|`), and XOR (`^`) operators in Milvus filter expressions, enabling queries such as: ```sql (flags & 4) == 4 -- check if bit 2 is set (permissions | 1) != 0 (mask ^ 7) == 0 ``` # Why Bitwise operators are commonly used for bitmask and flag-based filtering patterns. Previously, these expressions were unsupported and resulted in parse errors. - Issue: #24490 - design doc: `docs/design-docs/design_docs/20260620-bitwise-filter-operators.md` # Changes ## Protocol / Plan Definitions ### `pkg/proto/plan.proto` - Added `BitAnd = 7`, `BitOr = 8`, and `BitXor = 9` to the `ArithOpType` enum. ### `pkg/proto/planpb/plan.pb.go` - Added corresponding enum constants manually (protoc unavailable in the current environment). ## Parser ### `internal/parser/planparserv2/operators.go` - Mapped `BAND`, `BOR`, and `BXOR` tokens to the new protobuf enum values. - Added constant folding support for literal operands. ### `internal/parser/planparserv2/utils.go` - Extended `checkValidModArith()` to reject bitwise operations on non-integer types. ### `internal/parser/planparserv2/parser_visitor.go` - Implemented: - `VisitBitAnd` - `VisitBitOr` - `VisitBitXor` ## Query Execution Engine ### `internal/core/src/bitset/common.h` - Added new operator enum values. - Implemented evaluation logic in `ArithCompareOperator`. ### `internal/core/src/exec/expression/BinaryArithOpEvalRangeExpr.h` - Added `ArithOpHelper<>` specializations. - Added template dispatch support for bitwise operators. ### `internal/core/src/exec/expression/BinaryArithOpEvalRangeExpr.cpp` - Added dispatch cases for: - JSON execution paths - Array execution paths - Data execution paths - Index execution paths ### `internal/core/src/bitset/bitset.h` - Extended `inplace_arith_compare()` runtime dispatch to support bitwise operators. # Testing ### Parser Tests ```bash ok github.com/milvus-io/milvus/internal/parser/planparserv2 ``` ### Execution Engine The C++ implementation follows the existing `Mod` operator pattern and will be validated by CI. cc: @xiaofan-luan --------- Signed-off-by: Ayush KAshyap <kashyap11ayush02@gmail.com> Signed-off-by: xiaofanluan <xf@hjjaq.com> Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com> Co-authored-by: xiaofanluan <xf@hjjaq.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: xiaofanluan <xiaofan.luan@zilliz.com> |
||
|
|
678ebf3e00 |
feat: add raw string literals (r"...") for filter expressions (#50845)
## Summary Adds **raw string literals** `r"..."` / `R'...'` to filter expressions, so a backslash can be written verbatim — directly addressing the "needs 8 backslashes to match one" complaint in issue #43864. Today a backslash is unescaped **twice** inside Milvus before it reaches the matcher: 1. **String-literal layer** — the parser runs `strconv.Unquote` on `"..."` (`\\`→`\`, `\n`→newline, `\"`→`"`, …). 2. **Pattern layer** — `LIKE` / regex then apply their own escaping. Each halves the backslash count, so matching a single literal `\` needs `"\\\\"` (4) at the expression level. PostgreSQL avoids the extra layer because `standard_conforming_strings` makes `\` literal in ordinary strings; Milvus's string layer behaves like MySQL's default. A raw string takes its content **verbatim** (no `Unquote`), removing the string-literal layer while leaving the pattern layer intact — exactly like BigQuery / Spark SQL raw strings (BigQuery's `LIKE` docs even use `r'\%'`). | Expression | value at matcher | result | |---|---|---| | `A == r"a\b"` | `a\b` | equals `a\b` | | `A like r"\%"` | `\%`→`%` | equals `%` | | `A like r"\\%"` | `\\`→`\` | prefix `\` | | `A like r"a\\b"` | `a\\`→`a\` | equals `a\b` | | `A =~ r"\d+"` | `\d+` | regex `\d+` | So a literal `\` in `LIKE` becomes `r"\\"` (2) instead of `"\\\\"` (4) — matching PostgreSQL's expression-level count. ## Changes - **`Plan.g4`**: new `RawStringLiteral` lexer token (`[rR]` + quoted body), `DoubleRChar`/`SingleRChar` fragments, and a `# RawString` alternative in `expr`. Parser regenerated with ANTLR 4.13.2 (Go target only — the C++ side executes the serialized plan, no regen needed). - **`parser_visitor.go`**: `VisitRawString` emits the content verbatim as a `VarChar` value; `parseRegexPatternOrTemplate` accepts a raw token as a verbatim regex pattern. Works for `==`, `IN`, `LIKE`, JSON, `=~`. - **Design doc**: `docs/design-docs/design_docs/20260626-raw_string_literal.md` (prior art, syntax, semantics, compatibility). ## Semantics notes (match Python / BigQuery) - Backslash is **not** an escape inside a raw string; it is kept verbatim. - A backslash before the closing quote only prevents termination, so a raw string cannot end with an odd number of backslashes. Embed a quote by using the other quote style (`r'a"b'`). ## Compatibility Purely additive. Existing `"..."` / `'...'` literals are unchanged; the `r`/`R` prefix before a quote was previously a syntax error. SDKs need no change. ## Stacked on #50843 The first commit is the LIKE escape-model fix from **#50843** (review there). Together they fully resolve #43864: #50843 makes the `LIKE` pattern layer correct and PG-aligned, this PR removes the extra string-literal backslash layer. When #50843 merges, this PR's diff reduces to the raw-string commit. ## Test `TestExpr_RawString` covers verbatim `==`, `LIKE` (literal `%`/`_`/`\`, prefix/inner), single/double-quoted raw strings, raw-vs-normal backslash-count equivalence, and raw regex. Full `planparserv2` package passes; `gofmt` clean. ``` go test -tags dynamic,test -gcflags="all=-N -l" -count=1 ./internal/parser/planparserv2/ ``` --------- Signed-off-by: xiaofanluan <xf@hjjaq.com> Co-authored-by: xiaofanluan <xf@hjjaq.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
wei liuandGitHub
|
5f876e9a46 |
fix: filter snapshot segments by channel seek positions (#50827)
issue: #50663 This PR stores per-channel snapshot seek positions and uses each segment's own insert channel seek timestamp when selecting snapshot segments. `create_ts` remains a compatibility summary timestamp equal to the minimum channel seek timestamp. It rejects invalid or missing channel seek positions instead of falling back to `create_ts`, and updates snapshot metadata roundtrip coverage and design notes. Validation: - make generated-proto-without-cpp - make milvus - make lint-fix - git diff --check - gofmt -l internal/datacoord/handler.go internal/datacoord/handler_test.go internal/datacoord/snapshot_test.go pkg/proto/datapb/data_coord.pb.go Signed-off-by: Wei Liu <wei.liu@zilliz.com> |
||
|
Zhen YeandGitHub
|
8e99b46403 |
fix: ignore stale replicated txn body (#50848)
issue: #50846 - streaming replication: treat non-current txn messages covered by the replicate checkpoint as stale duplicates - replicate tests: cover stale txn body retry while another txn is in progress Signed-off-by: chyezh <chyezh@outlook.com> |
||
|
wei liuandGitHub
|
4d24af7063 |
feat: support Milvus snapshot as external table source (#49914)
issue: #45881 ## Summary Add the `milvus-table` external format so external collection refresh can consume Milvus snapshot metadata and StorageV3 segment manifests directly. This change lets refresh resolve source manifest row counts, rebuild target StorageV3 manifests from source column groups, copy source deltalogs with deterministic preallocated IDs, and keep real external primary-key semantics for milvus-table segments. ## Why Snapshot-backed external tables are not generic file tables. The refresh path needs to preserve Milvus StorageV3 manifest structure, deltalogs, and real PK behavior instead of falling back to virtual PK semantics or format-agnostic fragment handling. ## Details - Add milvus-table external spec parsing and snapshot manifest resolution. - Add StorageV3 manifest translation for source Milvus segment manifests. - Copy milvus-table deltalogs into target manifests with deterministic LogIDs. - Wire QueryNode real-PK lookup/load handling for milvus-table segments. - Add unit and Go client coverage for spec parsing, manifest resolution, refresh, and deltalog handling. ## Validation - `git diff --cached --check` - No proto changes detected - No mockery patterns in changed test files - `source ~/.profile && source scripts/setenv.sh && make milvus` - Reset Milvus local standalone before Go tests - `go test -tags dynamic,test -gcflags="all=-N -l" -ldflags="-r ${RPATH}" github.com/milvus-io/milvus/internal/datanode/external github.com/milvus-io/milvus/internal/storagev2/packed -count=1 -timeout 300s` - `source ~/.profile && source scripts/setenv.sh && make lint-fix` Related: [#45881](https://github.com/milvus-io/milvus/issues/45881) --------- Signed-off-by: Wei Liu <wei.liu@zilliz.com> |
||
|
Zhen YeandGitHub
|
b07c62d53c |
enhance: replace log package with context-aware mlog (#50094)
issue: #35917 - mlog package: move logger initialization, zap core, async buffered writes, field helpers, and scoped logger binding into pkg/mlog while removing pkg/log. - logging callsites: migrate Milvus logging usage to context-aware mlog APIs and simplify redundant With chains across components, utilities, tests, and tools. - trace propagation: replace logutil trace interceptors with mlog/tracer integration and add client_request_id fallback propagation for server stats handlers. --------- Signed-off-by: chyezh <chyezh@outlook.com> |
||
|
|
fa441d8861 |
feat: add result cache function for expr (#48873)
#50225 Co-authored-by: luzhang <luzhang@zilliz.com> |
||
|
sthuangandGitHub
|
ce5d6b5088 |
enhance: [RBAC] support role descriptions (#50184)
- Persist role descriptions when creating roles, expose them in role list/describe results, and support updating descriptions through the AlterRole path without changing role names. - Store role descriptions in the role value body while keeping role names in keys, and tolerate legacy empty role values plus undecodable stored values by returning an empty description for that role row. - Reject built-in/default roles and over-limit descriptions before WAL writes, and share the role-description length validator between proxy and rootcoord. related: #50183 --------- Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com> |
||
|
|
acfa147d30 |
doc: add error-handling casebook; harden guide/convention docs and agent entry (#50515)
Follow-up to #50221 (issue: #47420) — documentation only, no code change. ## Why The 60+ error misclassifications corrected during the #50221 review all happened **with the rules documents already in place** (`error_handling_guide.md`, `error_sentinel_convention.md`). The rules layer was sufficient; what was missing was (a) a layer of real positive/negative examples showing how the mistakes actually look in code, and (b) a mandatory entry point so coding agents and new authors read the right things before touching error handling. Linters catch *bare* errors, but they cannot catch a *wrong* code or a *wrong* classification — that needs case-level documentation. ## What - **`docs/dev/error_handling_casebook.md` (new).** The seven mistake patterns from the #50221 review, each with the actually-written wrong form (as first drafted, caught in review) and the merged fix, citing file + function: 1. "Looks like validation" is not user input (coordinator task types, plan shapes) 2. Internal component output is not user input (search-reduce, recovered panics) 3. `WrapErrXxxErr` is a relabel, not a context-adder (code masking) 4. The cause goes in the error argument, never the format string (`errors.Is` chain destruction) 5. InputError aborts `retry.Do`: scan before you mark (incl. the dual-identity boundary-stamp pattern) 6. Converting an `errors.Is` control-flow sentinel to merr widens every guard (the `errIgnored*` near-miss) 7. Boundary conversions that look wrong but are contracts (knowhere 2006, segcore wire collapse) Plus a quick-reference table for commonly confused sentinels and a minimum procedure for coding agents. - **`error_handling_guide.md`**: pre-flight scan recipe (`retry.Do` consumers) before InputError marking; `errors.Is`-widening warning before converting a sentinel to merr; casebook cross-link. - **`error_sentinel_convention.md`**: code-range partition table (which 100-range belongs to which family, what 65535 is, why segcore 2000–2099 is table-owned); "`milvusError.Is` matches by code" consequences; casebook cross-link. - **`CLAUDE.md`**: a mandatory error-handling procedure for coding agents (blame test, `merr.Wrap`-only context rule, the two pre-flight greps, code-range rule, guard tests + full test-go on contract changes). All file/function citations were verified against current master (post-#50221). --------- Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
sthuangandGitHub
|
5d80a9fd81 |
feat: [RBAC] support user description in credentials (#50186)
- Wire credential descriptions through create/update/read paths, including proxy, HTTP v2, Go SDK options, length validation, optional update presence handling, and internal credential proto/model persistence. - Preserve existing encrypted passwords and descriptions during RootCoord credential updates when either field is omitted, reject all-empty or partial direct password updates, and keep sha256 passwords cache-only. - Skip auth-cache refresh for description-only credential updates so existing authentication state is not blanked, while still refreshing caches for password updates. - Return persisted user descriptions from select_user and HTTP v2 describe_user, including include_role_info=false, and reuse already-loaded credential rows in user listing and RBAC backup to avoid duplicate etcd reads while ignoring malformed credential keys. - Use the upstream milvus-proto go-api/v3 version that includes the credential description proto change; the temporary fork replaces in root, pkg, client, and tests/go_client modules have been removed. design doc: docs/design-docs/design_docs/20260601-rbac-user-description.md related: #50179 --------- Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com> |
||
|
|
e2787d3981 |
enhance: standardize error handling on merr + Sys/Input classification (#50221)
issue: #47420 ## What this PR does Project-wide migration of raw `fmt.Errorf` / `errors.New` in function bodies onto the `merr` framework, plus the Sys-vs-Input error classification and the machinery it drives (retriability, fine-grained metrics, segcore unification), plus the convention docs and a linter that keeps it from regressing. Scope: storage, proxy, coordinators (root/data/query), query node, data node, `pkg/util` & `internal/util`, expression parser, message queue, streaming, and misc packages. Bare raw-error usages went from ~3000 to a ~340 allowlist (package-level sentinels / build-tag / test sites). --- ## How to review this PR It is large but the vast majority is mechanical. Changes fall into three tiers; spend review budget on Part 2 and Part 3. ### Part 1 — Mechanical standardization (low risk, verify by rule) Each converted call follows one of a small fixed set of rules. To review, check that each site obeys the matching rule rather than reading every line: | Pattern | Rule | |---|---| | `fmt.Errorf("...")` originating a new error | → `merr.WrapErrXxxMsg("...")` with a code matching the failure's meaning | | Adding context to an existing typed error | → `merr.Wrap(err, "...")` / `merr.Wrapf(...)` — **preserves** the inner code (never `WrapErr*Err`, which overwrites it) | | Errors inside the streaming subsystem | → `status.New*` factories (StreamingError), **not** merr — this is the component-internal dialect (see `docs/dev/error_handling_guide.md`) | | Low-level / control-flow signal caught by `errors.Is` | → kept as a package-level `errors.New` sentinel (lowercase, same-package) | Conventions are documented in `docs/dev/error_handling_guide.md` (how-to) and `docs/dev/error_sentinel_convention.md` (rules + audit). A `gocritic`/`ruleguard` rule (`rawmerrerror`, in `rules.go`) enforces "no raw `return errors.New/fmt.Errorf`" under `make verifiers`. ### Part 2 — Behavior changes (review these closely) These are the sites where the wire contract or runtime behavior changes, not just the source text. Listed by category; representative locations given, full set in the diff. **A. gRPC wire-code shifts: `UnexpectedError(1)/Code 65535` → typed code.** Where a handler previously returned a raw error (collapsed to `Code=65535` on the wire), it now returns a typed merr, so the client sees a real code. The most common shift is to `IllegalArgument(5)/Code 1100` (ParameterInvalid). Touch points include datanode task handlers (CreateTask/Query/Drop), proxy Upsert, querynode GetMetrics, datacoord CreateIndex, httpserver query-response builder, and typeutil schema validation. One code refinement: an index-param validation moved `1100` → `1101` (ParameterMissing). **Client/SDK assertions and any code that switched on `Code=65535` for these paths must be re-checked** (the go_client e2e assertions were already aligned in this PR). **B. Prometheus `status` label contract change (externally visible).** The proxy metric's coarse `fail` / `rejected` values are split into `fail_input` / `fail_system` and `rejected_user` / `rejected_system` (in `requestutil.ParseMetricLabel`; auth/privilege rejections count as `rejected_user`), so dashboards can attribute a failure to caller vs operator. **Dashboards/alerts querying `status="fail"` must migrate to `status=~"fail_.*"`, and `status="rejected"` to `status=~"rejected_.*"`.** The in-repo Grafana dashboard is already migrated; external dashboards built on the old values silently go empty after upgrade. This is the one change that requires an ops-side migration. **C. Retriability semantics.** - C1: `merr.Status(err)` now forces `Retriable=false` when the error is an `InputError` — a malformed request can never succeed on blind retry, so clients never get the self-contradictory "your input is wrong but you may retry". - C2: `retry.Do` short-circuits an `InputError` (non-retriable) — **but only when the caller did not pass a `RetryErr` predicate**. The check is an `if c.isRetryErr != nil { ... } else if InputError { ... }` *mutually exclusive* branch (`pkg/util/retry/retry.go`): an explicit `RetryErr` takes precedence and bypasses the InputError abort. `retry.Handle` deliberately does **not** apply the InputError abort (its callers signal abort via `shouldRetry=false`). Four flusher startup callsites that must retry through transient "not ready" errors were given explicit `RetryErr` escape hatches. **D. segcore (C++→Go) error classification.** A single shared Go-side table (`pkg/util/merr/segcore.go`) maps each segcore code to a merr sentinel + InputError/signal category, replacing scattered hand-written `if errorCode == ...` switches in the cgo wrappers. **Wire `Code` values change for every segcore pass-through error, not just the remapped ones.** Named sentinels remap (C++ `2003` → merr `2001`, `2033` → `2002`, Folly/Knowhere codes likewise); **all remaining pass-through codes (`2004`–`2043`, previously surfaced to clients as raw C++ enum values) now serialize as `2000`** (`ErrSegcore`), with the original C++ code preserved in the `Reason` text (`segcoreCode=...`); unknown/future codes collapse to `2000` as well (pinned by the `wire_code_projection` test). Transient segcore classes (object storage / file IO / OOM / mmap / FieldNotLoaded — 11 codes) now report `Retriable=true`. **Any client switching on raw segcore codes in the `2004`–`2043` range must be re-checked**; the in-Reason code remains available for diagnostics. Signal codes (PretendFinished / FollyCancel) are recognized centrally. `errors.Is`-based control flow on these (e.g. scheduler skip/retry) is preserved. **E. InputError classification (25 sentinels + dynamic marks).** 25 sentinels in `errors.go` carry `WithErrorType(InputError)` (the Collection / ResourceGroup / Database families, `ErrIndexDuplicate`, `ErrParameterInvalid`, `ErrPrivilegeNotAuthenticated`, `ErrImportFailed`, `ErrQueryPlan`, ...), plus dynamic marks for the 8 segcore input codes (ExprInvalid, DimNotMatch, MetricTypeInvalid, FieldIDInvalid, ...) and `WrapErrAsInputError`. The widest blast radius is `ErrParameterInvalid` (1100): ~2335 `WrapErrParameterInvalid*` callsites now classify as input / non-retriable. Because of C1/C2 this changes retriability for any path that returns these. **The audit to confirm no transient path was mis-marked is the single most important review item** (see Part 3). One reverse correction: storage field-stats parsing moved from `ErrParameterInvalid` (input) to `ErrDataIntegrity` — a corrupted stored stat is data corruption, not user input. ### Part 3 — Known risks & traps (called out proactively) 1. **`merr.Wrap` vs `WrapErr*Err` (code-masking).** `WrapErr*Err` builds a `wrappedMilvusError{sentinel: ErrServiceInternal}` whose `code()` returns the *outer* sentinel — it overwrites the inner typed code and hides the `errors.Is` chain. This is intentional (use it to *deliberately* downgrade), but it was a recurring conversion defect; the rule "add context with `merr.Wrap`, downgrade with `WrapErr*Err`" is enforced by convention and reviewed across the diff. 2. **InputError × `retry.Do` blast radius.** Marking a sentinel `InputError` makes any `retry.Do(...)` without a `RetryErr` predicate stop retrying it. Reviewers should sanity-check that no transient use of the 19 newly-marked sentinels (especially `ErrParameterInvalid`) sits inside a retry loop that needed to keep spinning. The known flusher cases were handled (see C2). 3. **The ~340 raw-error allowlist.** What remains as bare `errors.New` is, by design: package-level sentinels (caught by `errors.Is`), `//go:build test` sites, and out-of-band trees (`cmd/`, `tests/`, codegen, walimpls). The linter only bans the *direct-return* form; assignment-then-return escapes and the full no-exceptions ban are deferred to an AST-based linter (Tier 2, documented). 4. **segcore C++ second step deferred.** This PR unifies classification on the Go side; splitting the dual-semantic C++ codes at the source is a follow-up. --- ## Validation - `make verifiers`: Go side clean (gofmt + static-check across modules, including the new `rawmerrerror` rule with a 0-hit baseline repo-wide). - `make test-go`: passing; the one real regression introduced (a datanode `invalid_task_type` assertion shifting `1` → `5` from a ParameterInvalid conversion) was fixed in-tree. - go_client e2e CreateIndex assertions aligned to the new merr messages. --------- Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
sijie-ni-0214andGitHub
|
d9bbbc13d0 |
feat: support drop field via AlterCollectionSchema (#48988)
## Summary - Support dropping fields from collection schema via `AlterCollectionSchema` RPC with `DropRequest` - Support dropping function fields (cascade delete output fields and indexes) - Support disabling dynamic field via `AlterCollectionSchema` (reuse existing `AlterCollection` logic) - C++ segcore defense-in-depth: skip dropped fields during segment loading issue: #48983 design doc: docs/design-docs/design_docs/20260413-drop-collection-field-design.md ## Test plan - [x] Unit tests for rootcoord drop field/function logic - [x] Unit tests for proxy task validation (drop field constraints) - [x] Unit tests for datacoord index service (dropped field handling) - [x] Unit tests for querynode segment loader (dropped field skipping) - [x] C++ unit tests for SegmentLoadInfo ComputeDiff with dropped fields - [ ] E2E tests: drop scalar field, drop vector field, drop field with index, drop function, disable dynamic field (deferred to QA pending pymilvus RC) 🤖 Generated with [Claude Code](https://claude.ai/code) --------- Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com> |
||
|
Buqian ZhengandGitHub
|
f98531d02a |
fix: move struct hybrid design doc (#50297)
fix design doc location Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com> |
||
|
Buqian ZhengandGitHub
|
2f3f19ed60 |
feat: struct element hybrid search design and impl (#50243)
issue: https://github.com/milvus-io/milvus/issues/42148 design doc: docs/design_docs/20260602-struct_hybrid_search.md --------- Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com> |
||
|
wei liuandGitHub
|
6615811dee |
enhance: Support add-column refresh for external collections (#50082)
issue: #45881 Design doc: docs/design-docs/design_docs/20260526-external_table_add_column_refresh.md This PR lets external collection refresh handle additive schema changes, especially adding columns to parquet-backed external collections after segments have already been refreshed. What changed: - DataNode detects existing same-fragment external segments that are missing newly added external columns, appends manifest column groups idempotently, rebuilds fake binlogs, and returns same-ID updated segments. - DataCoord applies kept and updated refresh task payloads as validated segment upserts, including both patched existing segments and newly created segments. - Refresh apply intentionally does not use a job/task schema-version gate for the current additive-only scope. An older-schema refresh may complete without the newest fields, and the next refresh self-heals missing external columns. Segment-level validation still rejects schema-version rollback. - Proxy and RootCoord validate external add-field and alter-schema flows after field IDs are resolved, rejecting duplicate external_field mappings, generated output collisions, and unsupported external field types. - StorageV2 packed manifest helpers can read existing column groups and append missing columns through CommitManifestUpdates. - Patched same-fragment segments include function-output bytes in fake binlog MemorySize so memory estimates include generated columns. - Added the add-column refresh design doc and linked it from the external table design doc. - Added a go-client e2e that adds parquet column score after the first refresh and verifies returned query values, including 0..4 -> 0..0.04 and 100..104 -> 1.00..1.04. Validation: - make generated-proto-without-cpp - make milvus - make lint-fix - git diff --check - git diff --check milvus/master...HEAD - No-new-mockery diff scan - reset Milvus before scoped Go test runs - go test -tags dynamic,test -gcflags="all=-N -l" -ldflags="-r ${RPATH}" for targeted typeutil, proxy, datacoord, and storagev2/packed tests - go test -tags dynamic,test -gcflags="all=-N -l" -ldflags="-r ${RPATH}" github.com/milvus-io/milvus/internal/rootcoord -run '^$' - go test -tags dynamic,test -gcflags="all=-N -l" -ldflags="-r ${RPATH}" github.com/milvus-io/milvus/internal/datanode/external -run TestRefreshExternalCollectionTaskSuite -testify.m TestOrganizeSegments_PatchesMissingExternalColumns - go test -tags dynamic,test -gcflags="all=-N -l" -ldflags="-r ${RPATH}" github.com/milvus-io/milvus/internal/storagev2/packed github.com/milvus-io/milvus/pkg/v3/util/typeutil -run 'TestAppendSegmentManifestColumnsValidationPaths|TestAppendSegmentManifestColumnsReadAndNoopPaths|TestNormalizeAndValidateExternalCollectionSchema' - internal/datacoord full package coverage hit the known local macOS DISKANN subcase; targeted refresh task tests passed after reset Signed-off-by: Wei Liu <wei.liu@zilliz.com> |
||
|
sparknackandGitHub
|
d5917c265e |
enhance: Add V3 entry streaming reader (#50073)
issue: #48957 - Add `IndexEntryReader::ReadEntryStream`, `GetEntrySize`, and `HasEntry` for V3 entries. - Stream plain and encrypted entries through ordered slices with incremental CRC32c verification. - Bound transient stream memory with a process-wide entry stream budget. - Add dynamically refreshable `common.entryStream.streamBudgetRatio` to tune the shared entry stream budget for current scalar index loading and later field data loading. - Use 16MB stream slices by default, aligned with the encrypted slice size and existing V3 range size. - Derive the entry stream budget from CPU core count: `CPU cores * streamBudgetRatio * 16MB`. - Harden stream error handling: - keep active futures bounded by the thread pool size - reject zero or too-small explicit slice sizes - use overflow-safe slice count calculation - verify CRC for empty entries - release budget when callbacks throw or task submission fails - wait for active workers before returning on abnormal exits - validate decrypted slice length and include actual/expected CRC in stream errors - Document V3 streaming read behavior and shared budget configuration. --------- Signed-off-by: Shawn Wang <shawn.wang@zilliz.com> |
||
|
wei liuandGitHub
|
7bdd40d692 |
enhance: [ExternalTable Part10] enable function output fields on external collections (#49307)
issue: #45881 ## Summary Part 10 of the External Table series. Enables BM25 / MinHash / TextEmbedding function output fields on external collections, and makes external-table text_match work with persisted text indexes. Related: [#45881](https://github.com/milvus-io/milvus/issues/45881) (External Collection Lakehouse Integration tracking). ### What's in this PR **Schema & segcore** - Allow `Function` declarations on external schemas; function-output fields skip `external_field_mapping` validation - `Schema` tracks function-output field IDs; `ChunkedSegmentSealedImpl` and `ManifestGroupTranslator` resolve function-output columns by field name - Fast paths for retrieve / search load function-output columns by field name; external columns continue using `external_field_mapping` - `Util.cpp::GetFieldDatasFromManifest` resolves function-output columns by field name so indexbuilder reads the correct packed column **Refresh pipeline (format-agnostic via loon FFI)** 1. `CreateSegmentManifestWithBasePath` creates the input manifest that references external original files as CG0 2. `FFIPackedReader(v1)` streams input columns into `InsertData` via the shared `ArrowRecordToInsertData` helper 3. `embedding.RunAll` populates function-output fields in-place 4. `FFIPackedWriter` writes output fields as a new column group on top of v1 5. `AddStatsToManifest` registers BM25 stats into the final manifest 6. Memory peak stays around one Arrow batch, defaulting to 64 MiB **RootPath isolation** - Function-output input manifests, output manifests, BM25 stats, and text index artifacts now use the same `RootPath/insert_log/<collection>/<partition>/<segment>` layout as normal external table StorageV3 manifests - This avoids writing function-output artifacts under bucket-root `external/...` paths when multiple clusters share the same bucket **Text match** - External text fields with `enable_match` persist text index artifacts during refresh and load them through the regular external segment path - English and Chinese analyzer coverage are both included in the E2E test **Cross-bucket** - `NewPackedFFIReaderWithManifest` accepts `ExternalReaderContext` and injects per-collection `extfs.{collID}.*` aliases when `external_source` is set. Existing callers pass `{}` and are unaffected **VirtualPK** - `VirtualPKChunkedColumn` implements `GetChunk` / `GetAllChunks` so proxy requery filter-by-PK works for external collections with virtual primary keys **Shared embedding runner** - New `internal/util/function/embedding/runner.go` provides canonical `RunAll(ctx, schema, data, opts)` shared by import and external-table refresh paths - Supports BM25 output vector types including Float, BFloat16, Float16, Binary, and Sparse ### Test Plan - [x] Go unit tests for `internal/datanode/external` with 99.8% package coverage - [x] Go unit tests for changed import, packed, embedding, and schema helpers from the existing branch validation - [x] C++ rebuild + `milvus_storage` / `milvus_core` lib install from the existing branch validation - [x] E2E function-output tests in `tests/go_client/testcases/external_table_function_test.go` - `TestExternalTableBM25Function` - `TestExternalTableMinHashFunction` - `TestExternalTableTextEmbeddingFunction` - [x] `TestExternalTableTextMatch` with 10 files x 5000 rows = 50000 rows, English and Chinese text fields, persisted text index object checks in MinIO, load readiness check, and query correctness checks Signed-off-by: Wei Liu <wei.liu@zilliz.com> |
||
|
congqixiaandGitHub
|
435eba21b2 |
enhance: [Loaddiff] route schema reopen through load diff (#49882)
Related to #46358 Unify sealed segment schema reopen with the LoadDiff path so schema changes use the same diff/apply lifecycle as binlog, manifest, and index updates. This preserves default-filled field state across load info snapshots, passes the latest schema through QueryNode's cgo reopen path, and makes schema-only reopen compute default-fill/load/drop actions through segcore. --------- Signed-off-by: Congqi Xia <congqi.xia@zilliz.com> |
||
|
|
73eec01a0c |
fix: disable SDK default checksum to fix non-AWS S3 uploads (#49978)
issue: #49977 ## Summary AWS SDK C++ 1.11.692 (bumped from 1.11.352 in #47810) changed two defaults that together break PutObject against non-AWS S3-compatible storage: 1. `PutObjectRequest::GetChecksumAlgorithmName()` returns `"crc64nvme"` by default 2. `ClientConfiguration::checksumConfig::requestChecksumCalculation` defaults to `WHEN_SUPPORTED` `ChecksumInterceptor` then attaches a request hash to every PutObject, and the V4 signer switches the request to `Transfer-Encoding: aws-chunked` + `x-amz-content-sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER`. Aliyun OSS, Tencent COS, Huawei OBS, and GCP all reject this streaming form. This makes any `MinioChunkManager::PutObjectBuffer` call fail in an infinite `JobStateRetry` loop — json_stats / bson_inverted shared_key_index builds and any non-DiskANN-style index upload (IVF_*, HNSW, FLAT) are affected. ## Fix Restrict `requestChecksumCalculation` to `WHEN_REQUIRED` on non-AWS providers so the SDK only adds checksums when the operation model demands them (PutObject does not). Same approach as milvus-io/milvus-storage#500. Patched in both client construction paths: - `MinioChunkManager::MinioChunkManager` (storage_type=minio, address-based dispatch) — gated on `storageType != S3` - `ChunkManager.cpp` subclasses (storage_type=remote, cloud_provider-based dispatch) — applies in `GcpChunkManager`, `AliyunChunkManager`, `TencentCloudChunkManager`, `HuaweiCloudChunkManager`. `AwsChunkManager` keeps the default `WHEN_SUPPORTED`. CARDINAL_TIERED / DiskANN indexes already work because they upload via Arrow's `S3FileSystem` provided by milvus-storage, which was fixed in #500. ## Test plan - [x] Added `TEST(MinioChecksumConfig, OverridesAreWhenRequired)` in `test_storage.cpp` — asserts the override flips both directions from `WHEN_SUPPORTED` to `WHEN_REQUIRED` - [ ] Build / CI green - [ ] Manual smoke test on Aliyun OSS endpoint — index build (IVF_SQ8) and json_stats no longer fail with `x-oss-ec=0017-00000804`. CI cannot regression-detect this since MinIO doesn't reject `aws-chunked`; a real Aliyun OSS endpoint is required. ## Workaround (no code change) Setting `AWS_REQUEST_CHECKSUM_CALCULATION=when_required` and `AWS_RESPONSE_CHECKSUM_VALIDATION=when_required` on milvus pods also fixes this — the AWS SDK reads them at `ClientConfiguration` default-construct time, so it works across all bundled SDK copies without rebuilding. Useful for hotfix on running clusters until this PR ships. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: cai.zhang <cai.zhang@zilliz.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
dd6a411e5e |
doc: add design docs directory (#49829)
issue: #49831 ## What changed Vendor Milvus design documents into this repository under `docs/design-docs` as regular tracked files. - Keep only the design document content and assets under `docs/design-docs/design_docs/` and `docs/design-docs/assets/`. - Remove standalone repository metadata from the vendored directory, such as `README.md`, `CONTRIBUTING.md`, `MEP-TEMPLATE.md`, `COMMITTERS`, `MAINTAINERS`, `OWNERS`, `OWNERS_ALIASES`, and `.gitignore`. - Document the Milvus design document process in the main `CONTRIBUTING.md`. - Update Mergify and `tools/mgit.py` so feature PRs must provide an in-repo design document path under `docs/design-docs/design_docs/`. ## Why Milvus feature work should have an associated design document. Keeping design docs directly in this repository makes them available from a normal Milvus checkout and lets feature implementations include or link the related design document in the same repository. ## Verification - `git diff --check` - `python3 -m unittest tools/test_mgit_design_doc.py` - `python3 -m py_compile tools/mgit.py tools/test_mgit_design_doc.py` - Parsed `.github/mergify.yml` with Python `yaml.safe_load` - Confirmed `docs/design-docs` only contains `assets/` and `design_docs/` at the top level --------- Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
congqixiaandGitHub
|
7311d450a0 |
enhance: bump Go dependencies to v3 modules (#49485)
Related to #49398 Bump Go module references from pkg/v2 and milvus-proto/go-api/v2 to pkg/v3 and milvus-proto/go-api/v3 so the client tracks the Milvus 3.x release line. This prepares the repository for the upcoming 3.x.y release by aligning imports, module dependencies, and proto API references with the new major-version module paths. --------- Signed-off-by: Congqi Xia <congqi.xia@zilliz.com> |
||
|
wei liuandGitHub
|
e87cc36e82 |
enhance: bind snapshot lifecycle to collection (#48143)
## Summary - Refactor snapshot name uniqueness from global to per-collection scope - Add cascade delete: DropCollection triggers DropSnapshotsByCollection - Add orphan snapshot GC for deleted collections - Add database-level filtering for ListSnapshots and ListRestoreSnapshotJobs - Distinguish source/target collection in RestoreSnapshot API - Move snapshot privileges from Global level to Collection level - Update client SDK and documentation for new API semantics issue: #44358 issue: #47890 issue: #47883 issue: #47855 ## Test plan - [x] Unit tests for snapshot_meta (DropSnapshotsByCollection, per-collection isolation, partial failure) - [x] Unit tests for snapshot_manager (DropSnapshotsByCollection, getDBCollectionIDs) - [x] Unit tests for services (ListSnapshots/ListRestoreJobs with dbID, RestoreSnapshot with source collectionID) - [x] Unit tests for ddl_callbacks_snapshot (new dropSnapshotsByCollection callback) - [x] Unit tests for garbage_collector (orphan snapshot GC) - [x] E2E tests for cross-database snapshot isolation - [ ] CI validation ## Note on skipped Python E2E tests All 18 snapshot test classes in `tests/python_client/milvus_client/test_milvus_client_snapshot.py` are temporarily skipped with `@pytest.mark.skip`. Reason: this PR changes snapshot APIs (DropSnapshot, DescribeSnapshot, RestoreSnapshot) to require `collection_name` as a mandatory parameter, but the pymilvus SDK used in CI has not been updated to pass this parameter yet. The tests will be re-enabled once pymilvus SDK is updated to match the new API contract. 🤖 Generated with [Claude Code](https://claude.com/claude-code) design doc: https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20251114-snapshot_design.md --------- Signed-off-by: Wei Liu <wei.liu@zilliz.com> |
||
|
wei liuandGitHub
|
80af30b353 |
enhance: add compaction protection for snapshots (#48227)
## Summary - Add `compaction_protection_seconds` parameter to snapshot creation API, allowing users to protect referenced segments from compaction for a specified duration - Implement dual-layer compaction protection: collection-level fail-closed blocking (during RefIndex loading) + segment-level expiry-based protection (after RefIndex loads) - Protect all compaction paths: single, clustering, force-merge, storage-version, and compaction trigger candidates issue: https://github.com/milvus-io/milvus/issues/44358 design doc: https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20251114-snapshot_design.md ## Test plan - [x] Unit tests for `IsCollectionCompactionBlocked` (7 subtests covering blocked/unblocked/expired/multi-collection scenarios) - [x] Unit tests for `IsSegmentCompactionProtected` and segment protection rebuild - [x] Unit tests for `CreateSnapshot` with `compaction_protection_seconds` parameter - [x] Unit tests for proxy-side validation (negative values, exceeding max 7 days) - [x] DDL callback tests with compaction protection parameter - [x] Build verification (`make milvus` passes) - [x] Lint verification (`make lint-fix` passes) - [ ] E2E: Create snapshot with compaction protection → verify segments are not compacted during protection window - [ ] E2E: Verify compaction resumes after protection expires 🤖 Generated with [Claude Code](https://claude.com/claude-code) pr: https://github.com/milvus-io/milvus-proto/pull/574 --------- Signed-off-by: Wei Liu <wei.liu@zilliz.com> |
||
|
wei liuandGitHub
|
16b56d61e3 |
enhance: [ExternalTable Part8 Prep] remove dead External* RPC plumbing and obsolete design doc (#48972)
issue: #45881 ## Summary Dead RPC cleanup: - Remove `CreateExternalCollection` RPC from DataCoord proto; the stub never served traffic because external collection creation goes through RootCoord's `CreateCollection` flow. - Remove DataCoord proto messages unused by any caller: `CreateExternalCollectionResponse`, `QueryExternalCollectionRequest`, `DropExternalCollectionRequest`. - Remove IndexCoord proto message `UpdateExternalCollectionTask`; the three Catalog methods using it (`List/Save/DropUpdateExternalCollectionTask`) had no business-code callers. - Remove matching stub implementations from mix_coord, mixcoord client/service, DataCoord services, and metastore catalog/kv_catalog. - Regenerate protobuf / mockery output to match cleaned interfaces. Doc cleanup: - Delete `docs/external_table_load_query_design.md`; superseded by the updated fake-binlog / schemaless reader design for external tables. ## Test plan - [x] `make generated-proto-without-cpp` regenerates cleanly - [x] Build passes (`make milvus`) - [ ] CI: unit tests, integration tests, e2e Signed-off-by: Wei Liu <wei.liu@zilliz.com> |
||
|
wei liuandGitHub
|
3670a264c1 |
feat: [ExternalTable Part5] Enable loading and querying external collections (#47974)
## Summary - Implement virtual primary key (VirtualPK) for external collections without user-defined PK - Add ExternalSegmentCandidate for PK oracle to support segment pruning on external segments - Support loading external segments via manifest-based sealed segment loader - Add ManifestGroupTranslator with external field name fallback for column mapping - Add lance-table and vortex format support with HTTP scheme fix for MinIO endpoints - Fix extfs boolean property type mismatch by disabling extfs.* bool properties - Add lance and vortex format E2E tests with full query/search coverage - Add Python data generation scripts and CI Docker Python environment issue: #45881 design doc: https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260105-external_table.md ## Test plan - [x] Unit tests pass (pkoracle, segments, delegator, datanode/external) - [x] E2E tests pass (13/13): TestCreateExternal*, TestRefreshExternal*, TestExternalCollectionLanceFormat, TestExternalCollectionVortexFormat - [x] `make milvus` compiles successfully - [x] `make lint-fix` passes with no auto-fixes - [ ] CI checks pass --------- Signed-off-by: Wei Liu <wei.liu@zilliz.com> |
||
|
|
3b2526c7b6 |
fix: allocate new buildIDs for copy segments to prevent segmentBuildInfo corruption (#48163)
issue: https://github.com/milvus-io/milvus/issues/47658 ## Summary When restoring snapshots via copy segment, the source buildIDs were reused directly, causing 1:1 map corruption in DataCoord's `segmentBuildInfo`. This PR: - Extends `CopySegmentTarget` proto with `new_build_ids` mapping (`map<int64, int64>`) - Allocates fresh buildIDs in DataCoord and passes them to DataNode - DataNode replaces buildIDs in both index file paths and metadata - Fixes indexID mismatch in `syncVectorScalarIndexes` by resolving target collection's indexID via fieldID - Shifts GC protection from indexID-based (`GetSnapshotByIndex`) to buildID-based (`GetSnapshotByBuildID`) for precise file-level protection - Updates snapshot user guide to reflect GC protection mechanism change ## Test plan - [x] Unit tests updated for `copy_segment_meta`, `snapshot_meta`, `copy_segment_task` - [x] E2E test added: snapshot restore → drop → restore again flow (`TestSnapshotRestoreDropRestoreAgain`) - [ ] CI: ut-go, integration-test, e2e pass Signed-off-by: Wei Liu <wei.liu@zilliz.com> --------- Signed-off-by: Wei Liu <wei.liu@zilliz.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
|
3aefe9c14f |
doc: add streaming knowledge base for AI-agent-assisted development (#47936)
issue: #33285 Add a structured knowledge base under docs/agent_guides/streaming-system/ that documents the Milvus WAL (Write-Ahead Log) subsystem for AI coding agents. The knowledge base uses progressive disclosure: a streaming-system.md overview links to per-component docs with architecture, invariants, data flow, and key package locations. Components documented: - Channel model (PChannel/VChannel/CChannel) - Message model (lifecycle, properties, per-type semantics) - StreamingCoord (Channel Management, Broadcaster) - StreamingNode (TimeTick & Transaction, Lock, Shard Management) - RecoveryStorage (checkpoint persistence, WAL-based state recovery) - StreamingClient (Append/Read/Broadcast API, local bypass) - Replication & CDC (star topology, role enforcement, checkpoint tracking) - WAL Backend (Kafka/Pulsar/Woodpecker/RMQ) --------- Signed-off-by: chyezh <chyezh@outlook.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Li LiuandGitHub
|
9c353d42b7 |
doc: Move design doc for Milvus repo (#47406)
Milvus design doc moved to: https://github.com/milvus-io/milvus-design-docs Signed-off-by: Li Liu <li.liu@zilliz.com> |
||
|
|
b79a655f1c |
feat: [ExternalTable Part1] Enable create external collection (#46886)
issue: #45881 - Add ExternalSource and ExternalSpec fields to collection schema - Add ExternalField mapping for field schema to map external columns - Implement ValidateExternalCollectionSchema() to enforce restrictions: - No primary key (virtual PK generated automatically) - No dynamic fields, partition keys, clustering keys, or auto ID - No text match or function features - All user fields must have external_field mapping - Return virtual PK schema for external collections in GetPrimaryFieldSchema() - Skip primary key validation for external collections during creation - Add comprehensive unit tests and integration tests - Add design document and user guide --------- Signed-off-by: Wei Liu <wei.liu@zilliz.com> Co-authored-by: sunby <sunbingyi1992@gmail.com> |
||
|
wei liuandGitHub
|
975c91df16 |
feat: Add comprehensive snapshot functionality for collections (#44361)
issue: #44358 Implement complete snapshot management system including creation, deletion, listing, description, and restoration capabilities across all system components. Key features: - Create snapshots for entire collections - Drop snapshots by name with proper cleanup - List snapshots with collection filtering - Describe snapshot details and metadata Components added/modified: - Client SDK with full snapshot API support and options - DataCoord snapshot service with metadata management - Proxy layer with task-based snapshot operations - Protocol buffer definitions for snapshot RPCs - Comprehensive unit tests with mockey framework - Integration tests for end-to-end validation Technical implementation: - Snapshot metadata storage in etcd with proper indexing - File-based snapshot data persistence in object storage - Garbage collection integration for snapshot cleanup - Error handling and validation across all operations - Thread-safe operations with proper locking mechanisms <!-- This is an auto-generated comment: release notes by coderabbit.ai --> - Core invariant/assumption: snapshots are immutable point‑in‑time captures identified by (collection, snapshot name/ID); etcd snapshot metadata is authoritative for lifecycle (PENDING → COMMITTED → DELETING) and per‑segment manifests live in object storage (Avro / StorageV2). GC and restore logic must see snapshotRefIndex loaded (snapshotMeta.IsRefIndexLoaded) before reclaiming or relying on segment/index files. - New capability added: full end‑to‑end snapshot subsystem — client SDK APIs (Create/Drop/List/Describe/Restore + restore job queries), DataCoord SnapshotWriter/Reader (Avro + StorageV2 manifests), snapshotMeta in meta, SnapshotManager orchestration (create/drop/describe/list/restore), copy‑segment restore tasks/inspector/checker, proxy & RPC surface, GC integration, and docs/tests — enabling point‑in‑time collection snapshots persisted to object storage and restorations orchestrated across components. - Logic removed/simplified and why: duplicated recursive compaction/delta‑log traversal and ad‑hoc lookup code were consolidated behind two focused APIs/owners (Handler.GetDeltaLogFromCompactTo for delta traversal and SnapshotManager/SnapshotReader for snapshot I/O). MixCoord/coordinator broker paths were converted to thin RPC proxies. This eliminates multiple implementations of the same traversal/lookup, reducing divergence and simplifying responsibility boundaries. - Why this does NOT introduce data loss or regressions: snapshot create/drop use explicit two‑phase semantics (PENDING → COMMIT/DELETING) with SnapshotWriter writing manifests and metadata before commit; GC uses snapshotRefIndex guards and IsRefIndexLoaded/GetSnapshotBySegment/GetSnapshotByIndex checks to avoid removing referenced files; restore flow pre‑allocates job IDs, validates resources (partitions/indexes), performs rollback on failure (rollbackRestoreSnapshot), and converts/updates segment/index metadata only after successful copy tasks. Extensive unit and integration tests exercise pending/deleting/GC/restore/error paths to ensure idempotence and protection against premature deletion. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Wei Liu <wei.liu@zilliz.com> |