Commit Graph
489 Commits
Author SHA1 Message Date
congqixiaandGitHub c8dd73314c enhance: [GoSDK] migrate Go client module path to v3 (#51539)
Related to #51419

Update the Go client module path, imports, dependencies, tests,
examples, and documentation from client/v2 to client/v3.

Set the SDK version to 3.0.0 in preparation for the Milvus 3.0 release.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2026-07-17 22:48:41 +08:00
Li LiuandGitHub c819a2c5e1 build(deps): upgrade go toolchain to 1.26.5 (#51271)
## What
- Upgrade the root, pkg, client, test, and telemetry example modules
from Go 1.26.4 to Go 1.26.5.
- Upgrade CPU/GPU builder Dockerfiles, macOS CI Go setup, KRTE build
arg, rpm setup, meta-migration builder, and go-client test image to Go
1.26.5.
- Point CPU and GPU builder consumption in `.env` to the successful
build-env tag `20260714-c135601`.

## Why
- Latest direct Trivy scan of
`milvusdb/milvus:master-20260711-1993feae-amd64` reports CVE-2026-39822
in Go stdlib 1.26.4, fixed in 1.26.5.
- The normal daily image scan is currently blocked through Jenkins
`milvus_scan_image_daily` #540 with `exec format error`; last successful
scan is #508 from 2026-06-10, so this was verified by direct image scan.

## Image coverage
- Go toolchain fixes are shared by all Milvus runtime images; no per-OS
runtime Dockerfile package change is required.
- Builder definitions are updated for CPU ubuntu22.04, ubuntu20.04,
ubuntu24.04, amazonlinux2023, rockylinux9 and GPU ubuntu22.04,
ubuntu20.04.
- No runtime image variant is deliberately skipped; this is not an
OS-package CVE.

## Verification
- `go.dev/dl/?mode=json&include=all` confirms go1.26.5 is stable and
linux amd64/arm64 tarballs exist.
- `go env GOTOOLCHAIN GOVERSION` selects `auto` / `go1.26.5` after the
module update.
- `go list -m` succeeds for root, pkg, client, tests/go_client, and both
telemetry examples.
- `cd pkg && go test -tags dynamic,test -gcflags="all=-N -l" -count=1
./util/typeutil` passes after rebasing onto the latest upstream master.
- Jenkins build-env #40 succeeded and published builder tag
`20260714-c135601`; `.env` now selects that tag for CPU and GPU CI
builds.
- A repo-wide audit found no remaining Go 1.26.4 toolchain pins.

Note: full root `go mod tidy` is not included because this checkout has
a root-owned `deployments/docker/dev/volumes/etcd/member` directory that
makes `go mod tidy` fail with permission errors; `go mod tidy -e`
subsequently OOMed locally. No go.sum changes were produced by the
successful module tidies.

Generated by autonomous CVE maintenance.

---------

Signed-off-by: Li Liu <li.liu@zilliz.com>
2026-07-15 17:20:36 +08:00
junjiejiangjjjandGitHub d540e0d567 feat: support function chain API for search rerank (#50786)
issue: https://github.com/milvus-io/milvus/issues/50571
design doc: docs/design-docs/design_docs/20260624-function-chain-api.md

Add the FunctionChain proto-to-runtime path for ordinary Search L2
rerank. The change converts public FunctionChain protos into ChainRepr,
derives caller-independent read/write metadata, validates
Search-specific inputs in Proxy, and executes public chains through the
existing rerank pipeline.

Add request validation for duplicate stages, unsupported stages,
unsupported system inputs/outputs, unknown schema fields, unsupported
input field types, and hybrid-search usage. Extend REST v2 request
conversion to accept function_chains.

Replace score_combine with num_combine and add typed parameter readers,
repr-based FuncChain construction, chain optimization/pruning helpers,
and model-rerank parameter handling. Add coverage for chain repr
conversion, function/operator behavior, Proxy planning, search pipeline
integration, REST conversion, and REST API cases.

Add the Function Chain API design document and update milvus-proto to
the latest upstream pseudo-version.

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
2026-07-01 16:22:29 +08:00
tinswzyandGitHub 3aedc30fa8 fix: add topology-aware quorum reads and honor configured etcd metadata prefixes in wp (#50187)
issue: #43638 

update woodpecker to [release
v0.1.31](https://github.com/zilliztech/woodpecker/releases/tag/v0.1.31)
* fix(dashboard): add server runtime metrics  
* fix(meta): honor configured etcd metadata prefix    
* fix: make condition write auto mode fail on verification errors 
* feat: split cluster, region, and az topology metadata 

update woodpecker to [release
v0.1.32](https://github.com/zilliztech/woodpecker/releases/tag/v0.1.32)
* enhance: gate bucket check/create on createBucket for service mode 
* enhance(docker): run server images as root by default
* enhance(docker): bundle wp (wpcli) into server images for zero-config
in-pod ops
* fix(objectstorage): silence idle-spin DEBUG logs in MinioFileWriter
sync loop , **related issue:** #48976
* fix: cut read-path DEBUG log volume (per-read verbosity + idle
tail-read spin)
* feat(quorum): make client-side quorum config runtime-tunable via
Dynamic[T]
* feat(metrics): active-segment node metric + client dashboard panels 
* feat: delete-log lifecycle with async local reclaim

update woodpecker to [release
v0.1.33](https://github.com/zilliztech/woodpecker/releases/tag/v0.1.33)
* fix(reader): allow reopening at a truncated-but-not-yet-GC'd position
instead of force-advancing to the truncation point

---------

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2026-06-29 11:00:28 +08:00
Bingyi SunandGitHub 5e4309fc39 build(deps): update tikv client-go to tidb-8.5 (#50528)
Update tikv client-go version because we depend on the committs of
returned value by `txn.Get`. And previous grpc version used by client-go
has some deprecated functions and 8.5 update grpc.

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2026-06-17 12:36:22 +08:00
sthuangandGitHub 5d80a9fd81 feat: [RBAC] support user description in credentials (#50186)
- Wire credential descriptions through create/update/read paths,
including proxy, HTTP v2, Go SDK options, length validation, optional
update presence handling, and internal credential proto/model
persistence.
- Preserve existing encrypted passwords and descriptions during
RootCoord credential updates when either field is omitted, reject
all-empty or partial direct password updates, and keep sha256 passwords
cache-only.
- Skip auth-cache refresh for description-only credential updates so
existing authentication state is not blanked, while still refreshing
caches for password updates.
- Return persisted user descriptions from select_user and HTTP v2
describe_user, including include_role_info=false, and reuse
already-loaded credential rows in user listing and RBAC backup to avoid
duplicate etcd reads while ignoring malformed credential keys.
- Use the upstream milvus-proto go-api/v3 version that includes the
credential description proto change; the temporary fork replaces in
root, pkg, client, and tests/go_client modules have been removed.

design doc:
docs/design-docs/design_docs/20260601-rbac-user-description.md

related: #50179

---------

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2026-06-15 00:44:22 +08:00
986dcb42e6 build(deps): upgrade go toolchain to 1.26.4 (#48803)
## Summary
- Bump **Go to 1.26.4** across CI workflows, Docker builders, RPM setup,
and the `go` directive in all 4 modules (+ examples).
- Refresh Go 1.26-compatible deps to latest: `sonic` v1.15.2, `mockey`
v1.4.6, `gopkg` v0.1.4, `sonic/loader` v0.5.1, `cpuid/v2` v2.3.0. sonic
1.14.x fails to build on the 1.26 compiler (bytedance/sonic#895).
- **datacoord tests:** replace the anonymous `struct{ Iface }` mockey
idiom with **named helper types**
(`embeddedHandler`/`embeddedBroadcastAPI`/`embeddedBroker`/`embeddedAllocator`).

## Why the datacoord test change
Go 1.26's `printf` vet pass (run automatically by `go test`) calls
`x/tools/refactor/satisfy.Finder` unconditionally and **panics
`(*ast.StructType)`** on a **method expression of `*struct{ Iface }`** —
an upstream Go toolchain bug. Only `internal/datacoord` used this mockey
idiom (31 sites in 2 test files; verified repo-wide it's the only
place), which is why only `build-ut-cov`/`ut-go` failed on
`internal/datacoord [build failed]`. Switching to a **named type** makes
the method-expression receiver an identifier instead of a struct
literal, so `satisfy.Finder` no longer panics. mockey usage and test
behavior are unchanged. Minimal repro: `type I interface{ Logf(string,
...any) }` + `var _ = (*struct{ I }).Logf` under `go 1.26` → `go vet`
panics; named type does not.

## Notes
- `.env` builder-image tag intentionally not hand-edited (auto-bumped
post-merge by `bump-builder-version`).
- The bug also warrants an upstream report to golang/go (reproducer
above).

## Test plan
- [x] `go mod tidy` clean across all four modules
- [x] sonic 1.15.2 + mockey compile on go 1.26.4
- [x] reproduced the vet panic; verified named-type method expression
avoids it; gofmt-clean
- [ ] Full CI (build / build-ut-cov / ut-go / ut-cpp / integration)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-11 17:16:21 +08:00
93a76fe4bf fix: bump Go and x deps for CVE fixes (#49743)
## Summary
- Go 1.25.9 -> 1.25.10 and builder Dockerfiles now use Go 1.25.10. This
covers CVE-2026-33811, CVE-2026-33814, CVE-2026-39820, CVE-2026-39823,
CVE-2026-39825, CVE-2026-39826, CVE-2026-39836, CVE-2026-42499.
- golang.org/x/crypto -> v0.52.0 across root/pkg/client/tests_go_client.
This covers CVE-2026-39829, CVE-2026-39830, CVE-2026-39831,
CVE-2026-39832, CVE-2026-39833, CVE-2026-39834, CVE-2026-42508,
CVE-2026-46595, CVE-2026-46597.
- golang.org/x/net -> v0.55.0 across root/pkg/client/tests_go_client.
This covers CVE-2026-33814 and CVE-2026-39821.
- github.com/apache/thrift v0.20.0 -> v0.23.0, including
tests_go_client. This covers CVE-2026-41602.
- Updated tests/go_client/go.mod and go.sum so Go SDK CI has the
required x/* sum entries.

This PR is generated by the automated CVE fix task.

issue: https://github.com/milvus-io/milvus/issues/49032

## Test plan
- go mod tidy in pkg/
- go mod tidy in client/
- go mod tidy in tests/go_client/
- go mod tidy -e at repository root (standard tidy is blocked by
existing missing generated package cmd/tools/migration/legacy/legacypb)
- git diff --check
- go list -tags L0,L1,L2,test ./... in tests/go_client
- go list -m for updated dependencies in root/pkg/client/tests_go_client
- CI validates image build, Go SDK e2e, and scan

---------

Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-06 13:56:18 +08:00
dbaeb50cb1 enhance: support search aggregation search size (#49892)
## Summary
- Add per-level `search_size` handling for search aggregation context
building.
- Preserve `size` as the response bucket limit while using `search_size`
for downstream candidate bucket budget derivation.
- Bump the root module milvus-proto dependency to the version that
contains `SearchAggregationSpec.search_size`.

## Test Plan
- [x] `git diff --check trunk/master...HEAD`
- [x] `go test -tags dynamic,test -gcflags=\"all=-N -l\" -count=1
./internal/proxy/search_agg/...`

related: #48971

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 18:18:14 +08:00
tinswzyandGitHub 5e0884294d enhance: add shared sync scheduler and sealed segment direct read support for v0.1.29 (#50011)
issue: #43638

related wp issue:
https://github.com/zilliztech/woodpecker/compare/v0.1.28...v0.1.29
- stagedstorage: replace per-writer sync goroutines with a shared sync
scheduler and bounded worker pool while preserving per-writer flush
ordering
- fix: harden writer lifecycle transitions to reject late writes during
finalize/close/fence/storage-failure states
- enhance: support Azure managed identity authentication for the storage
client
- fix(storage): enable sealed segment direct read by default with
on-demand reader lifecycle management and config validation

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2026-05-26 05:32:35 +08:00
f762bbfaf4 enhance: bump pulsar-client-go to v0.19.0 and switch to pulsaradmin (#49933)
## Summary

Bumps `pulsar-client-go` v0.17.0 → v0.19.0 to pick up

[apache/pulsar-client-go#1457](https://github.com/apache/pulsar-client-go/pull/1457):
the client used to give up reconnecting after a topic hit the
`producer_exception` backlog quota, leaving the producer permanently
dead and writes timing out forever. v0.19.0 treats the quota exception
as retryable, so producers recover once backlog clears.

The bump can't ship alone: it's blocked at the module graph by Milvus's
pinned `xiaofan-luan/pulsarctl v0.5.1` fork, which imports
`pulsar-client-go/oauth2/store` — a package v0.19.0 has removed. MVS
then pulls the legacy standalone `pulsar-client-go/oauth2` module back
in, which collides with the main module's `oauth2/clock` (ambiguous
import). To unblock cleanly, this PR also replaces the only ~50 lines
of pulsarctl admin usage in Milvus with `pulsar-client-go`'s own
`pulsaradmin` package (shipped in the same module). The legacy
`99designs/{keyring,go-keychain}` and `danieljoos/wincred` transitive
deps fall out as a side effect.

## Changes

**Source (2 files, ~20 lines net)**

- `pkg/mq/msgstream/mqwrapper/pulsar/pulsar_client.go` —
`NewAdminClient` now returns `admin.Client` from
`pulsaradmin/pkg/admin`,
  built from `config.Config` instead of `common.Config`.
- `pkg/mq/msgstream/mq_factory.go` —
`Subscriptions().Delete(*topic, sub, true)` → `ForceDelete(*topic, sub)`
  (pulsaradmin splits force-delete into its own method);
  error type assertion `cli.Error` → `rest.Error` (same `Reason` field).

**Module hygiene (`pkg/go.mod`, `go.mod`)**

- bump `apache/pulsar-client-go` v0.17.0 → v0.19.0
- drop `require github.com/streamnative/pulsarctl`
- drop `replace streamnative/pulsarctl => xiaofan-luan/pulsarctl v0.5.1`
- drop `exclude pulsar-client-go/oauth2 v0.0.0-20211108044248-...`
  (no longer needed; the standalone oauth2 module is no longer in the
  graph at all — `go list -m .../oauth2` ⇒ *not a known dependency*)

## Behavior notes

v0.19.0 still calls `failPendingMessages` each time the producer
reconnect hits the quota, so in-flight sends continue to fail with
`ErrProducerBlockedQuotaExceeded` — upper-layer (msgstream / WAL) error
surface is **unchanged**. What changes is that the producer no longer
permanently dies and recovers automatically once backlog clears.

API surface diff verified against Milvus's pulsar usage: no removed or
renamed symbols; option structs only additively changed
(`ConsumerOptions.NackPrecisionBit` added). v0.19.0 requires `go 1.24`;
Milvus is on `go 1.25.8` — no Go version gate.

issue: #49932

## Test plan

- [x] `go build ./...` in `pkg/` clean (with rocksdb pkg-config)
- [x] `go vet -tags dynamic,test` clean on msgstream +
walimpls/impls/pulsar
- [x] pulsar test code compiles under `-tags dynamic,test
-gcflags="all=-N -l"`
- [x] Non-broker unit tests pass: `TestPulsarID_Serialize`,
`TestPulsarID_Equal`
- [x] `go list -m .../pulsar-client-go/oauth2` ⇒ `not a known
dependency`
      (ambiguous-import root cause structurally resolved)
- [ ] CI green
- [ ] Broker-dependent unit tests pass against Pulsar 3.0.2 (matches
      milvus-helm `pulsarv3` subchart)
- [ ] Pre-merge: induce `producer_exception` backlog quota in a staging
      cluster; verify producer recovers after backlog clears
      (independent reproduction already done out-of-tree)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-20 14:44:29 +08:00
tinswzyandGitHub 2523e96733 fix: bump Woodpecker to v0.1.28 to prevent preempted writers from skipping segment IDs (#49668)
issue: #43638

Upgrade Woodpecker to v0.1.28
fix: prevent preempted writer from skipping segment ids
wp related https://github.com/zilliztech/woodpecker/pull/156

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2026-05-17 14:36:28 +08:00
Spade AandGitHub b944be4a31 feat: impl StructArray -- support dynamic add struct field (#49807)
issue: https://github.com/milvus-io/milvus/issues/42148
design doc: docs/design-docs/design_docs/20260306-struct.md

this PR also fixes https://github.com/milvus-io/milvus/issues/49693

---------

Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
2026-05-17 01:02:29 +08:00
wei liuandGitHub 39cd786ebb enhance: return external spec in refresh job info (#49531)
issue: milvus-io/milvus#45881

This change returns the external spec used by refresh external
collection jobs through the public refresh job info paths.

Changes:
- Adds external spec to client refresh job info conversion.
- Returns redacted external spec from proxy refresh job responses.
- Exposes externalSpec in REST v2 refresh job response maps.
- Bumps milvus-proto go-api/v3 to the version containing
RefreshExternalCollectionJobInfo.external_spec.

Validation:
- go test -tags dynamic,test
github.com/milvus-io/milvus/client/v2/milvusclient -run
'Test.*External|Test.*Refresh' -count=1 -v

Note:
- internal/proxy and httpserver targeted tests were blocked locally by
missing C++ pkg-config artifacts: milvus_core, rocksdb, and
milvus-storage.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2026-05-07 14:12:09 +08:00
congqixiaandGitHub 7311d450a0 enhance: bump Go dependencies to v3 modules (#49485)
Related to #49398

Bump Go module references from pkg/v2 and milvus-proto/go-api/v2 to
pkg/v3 and milvus-proto/go-api/v3 so the client tracks the Milvus 3.x
release line.

This prepares the repository for the upcoming 3.x.y release by aligning
imports, module dependencies, and proto API references with the new
major-version module paths.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2026-05-01 10:30:13 +08:00
0fa822fc0a enhance: support search aggregation go-layer (#49071)
## Summary
- add `search_agg` package in proxy for context build, recursive
aggregation compute, and bucket ordering
- wire `group_by` aggregation through `task_search` and built-in
pipeline (`searchWithAggPipe` + aggregate operator serialization)
- upgrade querynode reduce key extraction for agg multi-field grouping,
and update related proxy/querynode tests
- keep local proto iteration path and local core build job defaults
aligned for this development workflow

## Test plan
- [x] bash
/home/zilliz/hc-claude-projects/hc-milvus-projects/search-aggregation/scripts/verify_baseline.sh
- [x] direct association tests passed
- [x] indirect association tests passed

issue: #49046

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-04-30 15:51:50 +08:00
tinswzyandGitHub a1ccbe608f fix: bump Woodpecker to v0.1.27 to fix stale gRPC connection issue (#49213)
issue: #43638

Upgrade Woodpecker to v0.1.27
Introduce automatic eviction of cached gRPC connections on
transport-level errors
related
- [wp issue 139](https://github.com/zilliztech/woodpecker/issues/139)
- [wp pr 149 ](https://github.com/zilliztech/woodpecker/pull/149)
- [wp pr 143](https://github.com/zilliztech/woodpecker/pull/143)

---------

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2026-04-28 08:57:49 +08:00
b908a2dda8 enhance: inject zap logger and gin context into hook plugin (#49097)
## Summary

- Inject Milvus zap logger into hook plugin via `SetZapLogger` type
assertion after plugin load, enabling the plugin to use Milvus's
centralized logging
- Inject connection manager into hook plugin via `SetClientInfoProvider`
type assertion, allowing the plugin to look up SDK info (sdk_type,
sdk_version, host) for non-Connect requests by connection identifier
- Pass `*gin.Context` directly via `GinParamsKey` instead of
`ginCtx.Keys` map, enabling the hook plugin to access `ClientIP()`,
`FullPath()`, `GetHeader()` and other gin context methods for RESTful
requests

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-27 18:49:46 +08:00
wei liuandGitHub 1b9c0b423d feat: support ARRAY_APPEND/ARRAY_REMOVE partial update ops for Array fields (#49251)
issue: #49241
pr: https://github.com/milvus-io/milvus-proto/pull/586
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260423-array_partial_update_ops.md

## Summary

Phase-1 implementation of field-level partial update operators for Array
fields. Lets clients mutate a single array without resending the full
value on upsert.

Supported ops (see `schemapb.FieldPartialUpdateOp.OpType`):
- `REPLACE` — legacy semantics (full overwrite). Default when no op is
set.
- `ARRAY_APPEND` — concatenate payload to base (duplicates preserved,
in-order).
- `ARRAY_REMOVE` — delete **all** occurrences matching any value in
payload.

## Changes

### Proto surface (via
https://github.com/milvus-io/milvus-proto/pull/586)
- `UpsertRequest.field_ops` carries per-field directives; `FieldData`
stays a pure data carrier (no leakage into
Insert/Query/Search/msgstream).

### Proxy (`internal/proxy/task_upsert*.go`)
- `validateFieldPartialUpdateOps` rejects: empty field name, duplicate
field, op on PK, unknown field, non-Array field, element-type mismatch,
unsupported op enum, op without matching FieldData, ARRAY_APPEND payload
+ base > `max_capacity`.
- Any non-REPLACE op auto-promotes `partial_update=true` (user does not
need to set both).
- In `queryPreExecute` merge step, dispatches per field: REPLACE →
`UpdateFieldDataByColumn`; ARRAY_APPEND/REMOVE →
`UpdateArrayFieldByColumnWithOp`.

### typeutil (`pkg/util/typeutil/schema.go`)
- `UpdateArrayFieldByColumnWithOp` (Array-only entry point) +
`ApplyArrayRowOp` + `appendArrayRow` / `removeArrayRow` cover all Array
element types (Bool/Int32/Int64/Float/Double/VarChar).
- REMOVE deletes every match (aligns with MongoDB `$pull`, Postgres
`array_remove`).
- Float/Double REMOVE: `NaN != NaN` (IEEE 754).
- ARRAY_APPEND enforces `max_capacity` per merged row and returns a
descriptive error.

### Go SDK (`client/v2`)
- `columnBasedDataOption.WithArrayAppend(field)` /
`WithArrayRemove(field)` / `WithFieldPartialOp(field, op)` build
`UpsertRequest.FieldOps`.
- REPLACE clears any prior directive for that field; non-REPLACE ops
auto-promote `PartialUpdate=true`.
- Unknown-field ops are forwarded as-is so the server can return the
descriptive validation error (client does not silently drop).

### Integration test
(`tests/integration/hellomilvus/upsert_partial_op_test.go`)
End-to-end insert → op-upsert → query flow exercising: ARRAY_APPEND,
ARRAY_REMOVE (incl. no-op row + multi-match), ARRAY_APPEND-over-capacity
rejection. Does **not** set `partial_update=true` — asserts the server
auto-promotes.

## Forward compatibility

Older servers that don't understand `field_ops` treat the request as a
plain upsert (proto3 unknown-field tolerance) → graceful fallback to
REPLACE semantics with a client-side warning path available.

## Test plan
- [x] typeutil UT: 40+ cases covering all element types, empty/nil
inputs, capacity edge cases, NaN semantics, unsupported-type rejection
- [x] proxy UT: 20+ validation cases (PK, unknown field, type mismatch,
capacity precheck, duplicate, empty name, op without FieldData)
- [x] go-sdk UT: 10 cases — auto-promote, REPLACE-clears-prior,
row-based + column-based builders, multi-op emission
- [x] integration test: append / remove / capacity-overflow end-to-end
- [x] `go mod tidy` in root / pkg / client
- [ ] CI green

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2026-04-24 09:05:45 +08:00
4e51991e55 feat: support multi-field composite group_by for vector search (segcore layer) (#48971)
## Summary

Refactor search-side GROUP BY from single-field to multi-field composite
key in the **segcore (C++) layer**, aligning with Elasticsearch's
`multi_terms` aggregation semantics.

**Scope: segcore layer only.** Go-layer changes (proxy search reduce,
delegator multi-field wiring, SDK API) will be tracked and submitted in
a separate PR.

## Core Changes

- **CompositeGroupKey** (Types.h): New composite key type
(vector<GroupByValueType>) with CompositeGroupKeyHash using
bits::hashMix, consistent with query-agg HashTable.h
- **MultiFieldDataGetter** (SearchGroupByOperator.{h,cpp}): Type-erased
multi-field data reader via std::function<GroupByValueType(int64_t)>,
replacing ~300 lines of per-type template dispatch
- **GroupByMap<CompositeGroupKey>** specialization for composite key
grouping with hash collision handling
- **SearchInfo.group_by_field_ids_** (QueryInfo.h): Replaced
optional<FieldId> group_by_field_id_ with vector<FieldId>
group_by_field_ids_; added has_group_by() / has_multi_field_group_by()
helpers
- **Proto**: New `repeated int64 group_by_field_ids = 17` in QueryInfo,
with backward compat for old group_by_field_id
- **Reduce pipeline**: GroupReduce, Reduce::SortEqualScoresOneNQ, and
AssembleCompositeGroupByValues all updated for composite keys
- **JSON group_by**: Restored proper json_type dispatch
(bool/int8/16/32/64/varchar) in MultiFieldDataGetter
- **Dead code cleanup**: Removed old AssembleGroupByValues() (~290
lines), removed write-only SearchResultPair.group_by_value_ field

## What This Enables (for follow-up Go-layer PR)

This commit provides the segcore foundation for the **Proxy-Only
architecture**:
1. Proxy amplifies group_count and group_size, flattens all nested
levels into a flat multi_terms composite key
2. Segcore / Delegator / QN execute standard flat group_by with
amplified parameters — zero awareness of nesting or metrics
3. Proxy reconstructs the nested hierarchy, computes metrics from
returned docs, and trims to user-requested group_size

The Go-layer PR will wire up: proto group_by_field_values (plural)
reading, proxy search reduce with composite keys, metric field injection
into output_fields, and nested reconstruction logic.

The full feature surface (per-group metrics R4, intra-bucket sort R5,
**bucket-level ordering / per-level-orderby R7**) is described in the
design doc below.

## Test plan

- [x] 5 existing single-field GroupBY.* tests adapted to composite key
path — all pass
- [x] 3 existing GroupBYJSON.* tests adapted — all pass
- [x] ElementFilter group_by interaction tests adapted — all pass
- [x] New GroupBY.CompositeKeySealedIndex — dual-field (int32+int64)
sealed+HNSW, strict_group_size=true, ground truth verification via
seg_offset -> raw_data lookup, exact group count and per-group size
assertions
- [x] New GroupBY.CompositeKeyGrowing — dual-field (int8+int32) growing
segment, exact 6 groups x group_size=3, every group fully filled
- [x] New GroupBY.CompositeKeyReduce — dual-field cross-segment C-API
reduce via ReduceSearchResultsAndFillData, PK dedup + composite
group_size validation
- [x] New GroupBY.SingleFieldViaCompositeProtoPath — single field via
new group_by_field_ids proto, backward compat with exact 7 groups x
group_size=3 = 20 results
- [x] **All 68 guarded tests passing**, clang-format clean

issue: #49046
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260413-search_embedded_agg.md

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-22 20:53:45 +08:00
a998749bd6 enhance: bump Go to 1.25.9 and otel/sdk to 1.43.0 for CVE-2026-32280, CVE-2026-32282, CVE-2026-39883 (#49031)
## Summary
- Bump Go from 1.25.8 to 1.25.9 to fix CVE-2026-32280, CVE-2026-32282
(stdlib)
- Bump go.opentelemetry.io/otel/sdk from v1.40.0 to v1.43.0 to fix
CVE-2026-39883
- Reference: https://nvd.nist.gov/vuln/detail/CVE-2026-32280,
https://nvd.nist.gov/vuln/detail/CVE-2026-32282,
https://nvd.nist.gov/vuln/detail/CVE-2026-39883

## Test plan
- [ ] CI passes
- [ ] Image scan clean for these CVEs

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-20 14:19:43 +08:00
wei liuandGitHub e87cc36e82 enhance: bind snapshot lifecycle to collection (#48143)
## Summary
- Refactor snapshot name uniqueness from global to per-collection scope
- Add cascade delete: DropCollection triggers DropSnapshotsByCollection
- Add orphan snapshot GC for deleted collections
- Add database-level filtering for ListSnapshots and
ListRestoreSnapshotJobs
- Distinguish source/target collection in RestoreSnapshot API
- Move snapshot privileges from Global level to Collection level
- Update client SDK and documentation for new API semantics

issue: #44358
issue: #47890
issue: #47883
issue: #47855

## Test plan
- [x] Unit tests for snapshot_meta (DropSnapshotsByCollection,
per-collection isolation, partial failure)
- [x] Unit tests for snapshot_manager (DropSnapshotsByCollection,
getDBCollectionIDs)
- [x] Unit tests for services (ListSnapshots/ListRestoreJobs with dbID,
RestoreSnapshot with source collectionID)
- [x] Unit tests for ddl_callbacks_snapshot (new
dropSnapshotsByCollection callback)
- [x] Unit tests for garbage_collector (orphan snapshot GC)
- [x] E2E tests for cross-database snapshot isolation
- [ ] CI validation

## Note on skipped Python E2E tests
All 18 snapshot test classes in
`tests/python_client/milvus_client/test_milvus_client_snapshot.py` are
temporarily skipped with `@pytest.mark.skip`. Reason: this PR changes
snapshot APIs (DropSnapshot, DescribeSnapshot, RestoreSnapshot) to
require `collection_name` as a mandatory parameter, but the pymilvus SDK
used in CI has not been updated to pass this parameter yet. The tests
will be re-enabled once pymilvus SDK is updated to match the new API
contract.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20251114-snapshot_design.md

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2026-04-16 15:45:43 +08:00
wei liuandGitHub 80af30b353 enhance: add compaction protection for snapshots (#48227)
## Summary
- Add `compaction_protection_seconds` parameter to snapshot creation
API, allowing users to protect referenced segments from compaction for a
specified duration
- Implement dual-layer compaction protection: collection-level
fail-closed blocking (during RefIndex loading) + segment-level
expiry-based protection (after RefIndex loads)
- Protect all compaction paths: single, clustering, force-merge,
storage-version, and compaction trigger candidates

issue: https://github.com/milvus-io/milvus/issues/44358
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20251114-snapshot_design.md

## Test plan
- [x] Unit tests for `IsCollectionCompactionBlocked` (7 subtests
covering blocked/unblocked/expired/multi-collection scenarios)
- [x] Unit tests for `IsSegmentCompactionProtected` and segment
protection rebuild
- [x] Unit tests for `CreateSnapshot` with
`compaction_protection_seconds` parameter
- [x] Unit tests for proxy-side validation (negative values, exceeding
max 7 days)
- [x] DDL callback tests with compaction protection parameter
- [x] Build verification (`make milvus` passes)
- [x] Lint verification (`make lint-fix` passes)
- [ ] E2E: Create snapshot with compaction protection → verify segments
are not compacted during protection window
- [ ] E2E: Verify compaction resumes after protection expires

🤖 Generated with [Claude Code](https://claude.com/claude-code)

pr: https://github.com/milvus-io/milvus-proto/pull/574

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2026-04-14 20:13:42 +08:00
c79925df02 enhance: upgrade go-jose/v4 to v4.1.4 for CVE-2026-34986 (#48805)
## Summary
- Bump `github.com/go-jose/go-jose/v4` from v4.1.3 to v4.1.4 in both
root `go.mod` and `pkg/go.mod`
- Fixes **CVE-2026-34986** (HIGH): Go JOSE panics during JWE decryption
when processing a crafted token, leading to denial of service

## Changes
- `go.mod` / `go.sum`
- `pkg/go.mod` / `pkg/go.sum`

## References
- https://nvd.nist.gov/vuln/detail/CVE-2026-34986
- https://github.com/go-jose/go-jose/releases/tag/v4.1.4

## Test plan
- [x] `go mod verify` passes in both root and `pkg/`
- [ ] CI passes
- [ ] Next daily image scan confirms CVE-2026-34986 is gone

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 22:03:40 +08:00
cai.zhangandGitHub ddf1cae39e fix: Upgrade arrow-go to v17.0.1 (#48775)
issue: #48375

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2026-04-08 17:15:39 +08:00
1baa492249 fix: upgrade grpc to v1.79.3 and quic-go to v0.54.1 for CVE-2026-33186, CVE-2025-59530 (#48577)
## Summary
- Upgrade `google.golang.org/grpc` from v1.71.1 to v1.79.3
(CVE-2026-33186 **CRITICAL** - Authorization Bypass via improper HTTP/2
`:path` pseudo-header validation)
- Upgrade `github.com/quic-go/quic-go` from v0.54.0 to v0.54.1
(CVE-2025-59530 **HIGH** - DoS via premature HANDSHAKE_DONE frame)
- Updated across all Go modules (root, pkg, client, tests/go_client)

issue: #48574

## Test plan
- [ ] CI passes
- [ ] Image scan clean for CVE-2026-33186 and CVE-2025-59530

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30 22:45:32 +08:00
b0ec4a2dd8 fix: upgrade Go to 1.25.8 for CVE-2025-68121, CVE-2026-27142, CVE-2026-25679 (#48286)
issue: #48574

## Summary
- Upgrade Go from 1.24.12 to 1.25.8 across all go.mod files and
Dockerfiles
- Fixes CVE-2025-68121 (CRITICAL), CVE-2026-27142 (HIGH), CVE-2026-25679
(HIGH) in Go stdlib
- All three CVEs affect the Go standard library and are resolved by
upgrading to Go 1.25.8

## Changes
- Updated `go` directive in go.mod files (root, pkg/, client/,
tests/go_client/, examples/)
- Updated Go download URLs in Dockerfiles (build/docker/builder/)
- Updated `toolchain` directives where present

issue: https://github.com/milvus-io/milvus/issues/TBD

## Test plan
- [x] `make milvus` builds successfully with Go 1.25.8
- [ ] CI passes

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 14:23:30 +08:00
4ba08692ce feat: implement data salvage for force failover (#47599)
issue: https://github.com/milvus-io/milvus/issues/47598

design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260205-data_salvage_for_force_failover.md

## Summary

Implement data salvage feature that enables zero data loss after force
failover by capturing and exposing the salvage checkpoint, along with a
DumpMessages API to retrieve unsynchronized messages.

### SalvageCheckpoint Capture & Persistence
- Add `salvageCheckpoint` field to `ReplicateManager`
- Capture last synced position on force promote before clearing
secondary state
- Persist to separate etcd key via `StreamingNodeCataLog`
- Load salvage checkpoint on WAL recovery

### API Exposure
- Return `salvage_checkpoint` in `GetReplicateInfoResponse`
- Add `GetSalvageCheckpoint` to `HandlerClient` and `ReplicateService`

### DumpMessages Streaming API
- Implement `DumpMessages` RPC for streaming WAL messages
- Support `start_timetick` and `end_timetick` filters
- Filter out system messages (TimeTick, CreateSegment, Flush,
RollbackTxn)
- Use oneof response for status or message

### Tests
- Unit tests for salvage checkpoint capture
- Unit tests for `shouldDumpMessage` filter
- Integration tests for `GetReplicateInfo` and `DumpMessages`

## Test plan
- [ ] Unit tests pass
- [ ] Integration tests pass
- [ ] Manual test with force failover scenario

proto pr: https://github.com/milvus-io/milvus-proto/pull/568

🤖 Generated with [Claude Code](https://claude.ai/code)

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Signed-off-by: Yihao Dai <yihao.dai@zilliz.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-03-27 13:47:30 +08:00
tinswzyandGitHub aed7c8bcfb enhance: update WP version v0.1.25 (#45011)
#43638
update wp to latest 
Introduce the beta version of the wp service mode.

---------

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2026-03-23 05:57:30 +08:00
tinswzyandGitHub c7cea10912 fix: disable ConditionWrite when using AK/SK on Aliyun OSS (#48310)
issue: #43638
wp related https://github.com/zilliztech/woodpecker/issues/115

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2026-03-18 12:39:26 +08:00
f1bb8356eb enhance: upgrade Go dependencies (casbin, gin, lo, cockroachdb/errors) (#47943)
## Summary
- Upgrade Go dependencies across root, pkg, and client modules:
  - `casbin/casbin` v2.44.2 → v2.135.0
  - `gin-gonic/gin` v1.9.1 → v1.11.0
  - `samber/lo` v1.27.0 → v1.52.0
  - `cockroachdb/errors` v1.9.1 → v1.12.0
  - `google.golang.org/protobuf` v1.36.5 → v1.36.9
- Adapt source code to breaking API changes:
- `lo.Last()` now returns `(T, bool)` instead of `(T, error)` (lo v1.52)
- `gin.LogFormatterParams.Keys` is now `map[any]any` instead of
`map[string]any` (gin v1.11)

issue: https://github.com/milvus-io/milvus/issues/33482

## Test plan
- [x] `go mod tidy` clean on all three modules (root, pkg, client)
- [x] Local lint passes with no new errors
- [ ] CI code-check passes
- [ ] CI ut-go passes
- [ ] CI e2e passes

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 21:11:26 +08:00
ff5b6d0a64 enhance: remove spf13/viper dependency (#47944)
## Summary
- Replace spf13/viper with yaml.v3 in the only 2 files that used it
(printer.go, server_test.go)
- Remove viper + 7 transitive deps from go.mod (8 packages total)
- C++ ghost dep removal (libsodium, simde) deferred to separate PR to
avoid CI conan issues

### Removed packages
spf13/viper, fsnotify/fsnotify, hashicorp/hcl, mitchellh/mapstructure,
pelletier/go-toml, spf13/afero, spf13/jwalterweatherman, subosito/gotenv

## Test plan
- [ ] CI passes (Go build + code-check)
- [ ] `go mod tidy` produces no diff
- [ ] querynodev2 TestInit_QueryHook passes with new yamlConfigWriter

issue: https://github.com/milvus-io/milvus/issues/47919

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 15:13:31 +08:00
Spade AandGitHub f163e94ff1 feat: impl StructArray -- support element-level query (#47906)
issue: https://github.com/milvus-io/milvus/issues/42148
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260306-struct.md

---------

Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
2026-03-13 17:55:29 +08:00
congqixiaandGitHub 6b910a0074 enhance: bump OpenTelemetry to v1.40.0 to fix CWE-426 untrusted search path (#48058)
Related to #48070

Upgrade go.opentelemetry.io/otel and related packages from v1.34.0 to
v1.40.0 across all Go modules to address CWE-426 (Untrusted Search Path)
vulnerability. Also bumps transitive dependencies including auto/sdk
v1.1.0 -> v1.2.1, go-logr v1.4.2 -> v1.4.3, and golang.org/x/sys v0.38.0
-> v0.40.0.

See: https://cwe.mitre.org/data/definitions/426.html

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2026-03-09 10:57:22 +08:00
72918439ef enhance: remove unused Go dependencies (ansi, fastjson, grpc/examples, sizedwaitgroup) (#47852)
Related to #46199

## Summary

Remove 5 unused or misused Go dependencies to reduce module bloat and
consolidate overlapping libraries:

- **`mgutz/ansi`** → replaced with inline ANSI escape codes (only used
for 3 color constants in migration console)
- **`valyala/fastjson`** → replaced with `tidwall/gjson` (only 1 file
used fastjson; gjson is already used in 22+ files)
- **`google.golang.org/grpc/examples`** → replaced with existing
`rootcoordpb` (test file pulled in entire grpc examples repo for a mock
server)
- **`remeh/sizedwaitgroup`** → replaced with `chan` semaphore +
`sync.WaitGroup` (only 2 files, trivial pattern)
- **`pkg/errors`** → replaced with `cockroachdb/errors` (the project
standard; `pkg/errors` was used in 1 file)

## Behavior change: DeleteLog.Parse() fail-fast on missing fields

The `fastjson` → `gjson` migration adds explicit `Exists()` validation
for `ts`, `pk`, and `pkType` fields in the JSON parsing branch.
Previously, both fastjson and gjson would silently return zero values
for missing fields, causing `dl.Pk` to remain nil and panicking
downstream. The new code fails fast with a descriptive error at parse
time. This is a defensive improvement (the original code had identical
silent-failure behavior).

## Performance impact

| Change | Path type | Perf delta | Matters? |
|--------|-----------|------------|----------|
| `pkg/errors` → `cockroachdb/errors` | Cold (offline CLI tool
`config-docs-generator`) | Negligible | No |
| `mgutz/ansi` → inline ANSI codes | Cold (offline CLI tool
`migration/console`) | Marginally faster (eliminates map lookup) | No |
| `fastjson` → `gjson` (`DeleteLog.Parse`) | Warm — old-format deltalog
deserialization only | **~2.5x slower** per JSON parse (143ns→361ns) |
**No** — see below |
| `grpc/examples` → `rootcoordpb` | Test only (`client_test.go`) | None
| No |
| `sizedwaitgroup` → chan+WaitGroup | Test only (`wal_test.go`,
`test_framework.go`) | None | No |

### fastjson → gjson regression detail

`DeleteLog.Parse()` is called per-row during deltalog deserialization,
but **only for the legacy single-field format**. The new multi-field
parquet format (`newDeltalogMultiFieldReader`) reads pk/ts as separate
Arrow columns and bypasses `Parse()` entirely. Legacy deltalogs are
rewritten to parquet format during compaction, so this is a dying code
path. Additionally, deltalog loading is I/O-bound — the JSON parse cost
(~361ns/row) is negligible compared to disk read and Arrow
deserialization overhead.

Benchmark (Go 1.24, arm64):
```
BenchmarkFastjsonSmall-4       8,315,624    143.1 ns/op    0 B/op   0 allocs/op
BenchmarkGjsonOptimized-4      3,321,613    361.4 ns/op   96 B/op   1 allocs/op
```

## Test plan

- [x] CI build passes
- [x] CI code-check passes
- [ ] CI ut-go passes
- [ ] CI e2e passes
- [x] Boundary test cases added (bare number, missing pkType/ts/pk)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 00:49:27 +08:00
wei liuandGitHub 6b4171e7ac feat: [ExternalTable Part3] Support manual refresh for external collections (#47492)
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260105-external_table.md

issue: #45881

This change introduces manual refresh capability for external
collections, allowing users to trigger on-demand data synchronization
from external sources. It replaces the legacy update mechanism with a
more robust job-task hierarchy and persistent state management.

Key changes:
- Add RefreshExternalCollection, GetRefreshExternalCollectionProgress,
  and ListRefreshExternalCollectionJobs APIs across Client, Proxy,
  and DataCoord
- Implement ExternalCollectionRefreshManager to manage refresh jobs
  with a 1:N Job-Task hierarchy
- Add ExternalCollectionRefreshMeta for persistent storage of jobs and
  tasks in the metastore
- Add ExternalCollectionRefreshChecker for task state management and
  worker assignment
- Implement ExternalCollectionRefreshInspector for periodic job
  cleanup
- Use WAL Broadcast mechanism for distributed consistency and
  idempotency
- Replace legacy external_collection_inspector and update tasks with
  the new refresh-based implementation
- Add comprehensive unit tests for refresh job lifecycle and state
  transitions
  
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260105-external_table.md

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2026-02-26 11:20:46 +08:00
a829086267 fix: protect snapshot-referenced files from garbage collection (#47669)
## Summary

This PR prevents RestoreSnapshot failures caused by garbage collection
of snapshot-referenced files during compaction and index rebuild cycles.

**Root cause:**
When a collection undergoes compaction or index rebuild after a snapshot
is created, the old segment/index files referenced by the snapshot are
garbage collected without checking snapshot references. This causes
RestoreSnapshot to fail with "key not found" errors.

**Changes:**
- Add snapshot reference checks before deleting binlog files
- Add snapshot reference checks before deleting index files
- Add snapshot reference checks before deleting text index files
- Add snapshot reference checks before deleting JSON index files
- Add nil-safety checks for snapshotMeta in all GC operations
- Upgrade mockey to v1.4.4 for improved testing

All GC functions now query `GetSnapshotBySegment`/`GetSnapshotByIndex`
before file deletion. Files referenced by active snapshots are preserved
to ensure RestoreSnapshot can always access the files recorded in the
snapshot metadata.

## Related Issue

issue: #47658
issue: #44358

## Test Coverage

- Added snapshot reference protection tests for all modified GC
functions
- Added nil snapshotMeta handling tests
- All tests pass with race detector enabled
- Coverage increased from 0% to 40%+ for text/JSON index GC functions

## Type of Change

- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Documentation update

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-13 10:32:43 +08:00
f36ed5500b enhance: Bump Go version to 1.24.12 and upgrade gpgv fixing CVEs (#47541)
- Upgrade Go from 1.24.11 to 1.24.12 to fix CVE-2025-61726 (net/url
query parameter DoS) and CVE-2025-61728 (archive/zip CPU exhaustion)
- Upgrade gpgv in Ubuntu Dockerfiles to fix CVE-2025-68973 (GnuPG
out-of-bounds write vulnerability)

Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-04 17:19:50 +08:00
XiaofanandGitHub 8b6a3a0aa4 feat: Add client-side telemetry with heartbeat and server command support (#47523)
issue: #47281
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260131-client_side_telemetry.md

Implement comprehensive client-side telemetry system that includes:

- Metrics collection for Search, Query, Insert, Delete, Upsert
operations
- Automatic heartbeat reporting to server with configurable intervals
- Per-collection metrics tracking with wildcard support
- P99 latency calculation using ring buffer sampling
- Server-push command handling (push_config, collection_metrics,
show_errors)
- Historical snapshot storage for latency history queries
- Error tracking with circular buffer for recent errors
- WebUI for telemetry visualization and command management
- HTTP API endpoints for telemetry data access and command push
- RootCoord telemetry manager for centralized command routing

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2026-02-04 00:03:50 +08:00
e1f5dafef9 feat: Add GetReplicateConfiguration API (#47393)
## Summary

Add a new public API `GetReplicateConfiguration` that allows cluster
administrators to view the current cross-cluster replication topology
with sensitive connection parameters (tokens) redacted.

## Changes

- Add privilege constant `PrivilegeGetReplicateConfiguration` to
`ClusterReadOnlyPrivileges`
- Add `SanitizeReplicateConfiguration` helper to strip sensitive tokens
before returning
- Add `GetReplicateConfiguration` method to `ReplicateService` interface
- Implement `GetReplicateConfiguration` handler in Proxy
- Add integration tests

## API

```protobuf
rpc GetReplicateConfiguration(GetReplicateConfigurationRequest) returns (GetReplicateConfigurationResponse) {}
```

**Security:**
- Requires ClusterAdmin privilege
- Tokens are redacted from the response

## Dependencies

- Proto changes: milvus-io/milvus-proto#566

## Related Issue

Closes #47392

## Design Doc

design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260128-get_replicate_configuration.md

## Test Plan

- [x] Unit tests for sanitization helper
- [x] Unit tests for ReplicateService method
- [x] Integration tests for the API

---

🤖 Generated with [Claude Code](https://claude.ai/code)

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Happy <yesreply@happy.engineering>
2026-02-03 15:32:34 +08:00
junjiejiangjjjandGitHub b9752362d1 feat: Add highlight scores (#47017)
https://github.com/milvus-io/milvus/issues/46994

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
2026-01-14 10:45:27 +08:00
wei liuandGitHub 975c91df16 feat: Add comprehensive snapshot functionality for collections (#44361)
issue: #44358

Implement complete snapshot management system including creation,
deletion, listing, description, and restoration capabilities across all
system components.

Key features:
- Create snapshots for entire collections
- Drop snapshots by name with proper cleanup
- List snapshots with collection filtering
- Describe snapshot details and metadata

Components added/modified:
- Client SDK with full snapshot API support and options
- DataCoord snapshot service with metadata management
- Proxy layer with task-based snapshot operations
- Protocol buffer definitions for snapshot RPCs
- Comprehensive unit tests with mockey framework
- Integration tests for end-to-end validation

Technical implementation:
- Snapshot metadata storage in etcd with proper indexing
- File-based snapshot data persistence in object storage
- Garbage collection integration for snapshot cleanup
- Error handling and validation across all operations
- Thread-safe operations with proper locking mechanisms

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
- Core invariant/assumption: snapshots are immutable point‑in‑time
captures identified by (collection, snapshot name/ID); etcd snapshot
metadata is authoritative for lifecycle (PENDING → COMMITTED → DELETING)
and per‑segment manifests live in object storage (Avro / StorageV2). GC
and restore logic must see snapshotRefIndex loaded
(snapshotMeta.IsRefIndexLoaded) before reclaiming or relying on
segment/index files.

- New capability added: full end‑to‑end snapshot subsystem — client SDK
APIs (Create/Drop/List/Describe/Restore + restore job queries),
DataCoord SnapshotWriter/Reader (Avro + StorageV2 manifests),
snapshotMeta in meta, SnapshotManager orchestration
(create/drop/describe/list/restore), copy‑segment restore
tasks/inspector/checker, proxy & RPC surface, GC integration, and
docs/tests — enabling point‑in‑time collection snapshots persisted to
object storage and restorations orchestrated across components.

- Logic removed/simplified and why: duplicated recursive
compaction/delta‑log traversal and ad‑hoc lookup code were consolidated
behind two focused APIs/owners (Handler.GetDeltaLogFromCompactTo for
delta traversal and SnapshotManager/SnapshotReader for snapshot I/O).
MixCoord/coordinator broker paths were converted to thin RPC proxies.
This eliminates multiple implementations of the same traversal/lookup,
reducing divergence and simplifying responsibility boundaries.

- Why this does NOT introduce data loss or regressions: snapshot
create/drop use explicit two‑phase semantics (PENDING → COMMIT/DELETING)
with SnapshotWriter writing manifests and metadata before commit; GC
uses snapshotRefIndex guards and
IsRefIndexLoaded/GetSnapshotBySegment/GetSnapshotByIndex checks to avoid
removing referenced files; restore flow pre‑allocates job IDs, validates
resources (partitions/indexes), performs rollback on failure
(rollbackRestoreSnapshot), and converts/updates segment/index metadata
only after successful copy tasks. Extensive unit and integration tests
exercise pending/deleting/GC/restore/error paths to ensure idempotence
and protection against premature deletion.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2026-01-06 10:15:24 +08:00
junjiejiangjjjandGitHub c85f7d5d84 fix: Update milvus-proto (#46561)
https://github.com/milvus-io/milvus/issues/46543

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
2025-12-29 11:29:22 +08:00
aoiasdandGitHub 55feb7ded8 feat: set related resource ids in collection schema (#46423)
Support crate analyzer with file resource info, and return used file
resource ids when validate analyzer.
Save the related resource ids in collection schema.
relate: https://github.com/milvus-io/milvus/issues/43687

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
- Core invariant: analyzer file-resource resolution is deterministic and
traceable by threading a FileResourcePathHelper (collecting used
resource IDs in a HashSet) through all tokenizer/analyzer construction
and validation paths; validate_analyzer(params, extra_info) returns the
collected Vec<i64) which is propagated through C/Rust/Go layers to
callers (CValidateResult → RustResult::from_vec_i64 → Go []int64 →
querypb.ValidateAnalyzerResponse.ResourceIds →
CollectionSchema.FileResourceIds).

- Logic removed/simplified: ad‑hoc, scattered resource-path lookups and
per-filter file helpers (e.g., read_synonyms_file and other inline
file-reading logic) were consolidated into ResourceInfo +
FileResourcePathHelper and a centralized get_resource_path(helper, ...)
API; filter/tokenizer builder APIs now accept &mut
FileResourcePathHelper so all file path resolution and ID collection use
the same path and bookkeeping logic (redundant duplicated lookups
removed).

- Why no data loss or behavior regression: changes are additive and
default-preserving — existing call sites pass extra_info = "" so
analyzer creation/validation behavior and error paths remain unchanged;
new Collection.FileResourceIds is populated from resp.ResourceIds in
validateSchema and round‑tripped through marshal/unmarshal
(model.Collection ↔ schemapb.CollectionSchema) so schema persistence
uses the new list without overwriting other schema fields; proto change
adds a repeated field (resource_ids) which is wire‑compatible (older
clients ignore extra field). Concrete code paths: analyzer creation
still uses create_analyzer (now with extra_info ""), tokenizer
validation still returns errors as before but now also returns IDs via
CValidateResult/RustResult, and rootcoord.validateSchema assigns
resp.ResourceIds → schema.FileResourceIds.

- New capability added: end‑to‑end discovery, return, and persistence of
file resource IDs used by analyzers — validate flows now return resource
IDs and the system stores them in collection schema (affects tantivy
analyzer binding, canalyzer C bindings, internal/util analyzer APIs,
querynode ValidateAnalyzer response, and rootcoord/create_collection
flow).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-12-26 22:49:19 +08:00
Spade AandGitHub ad8aba7cb4 feat: impl ComputePhraseMatchSlop for compute min slop for phrase match query (#45892)
issue: https://github.com/milvus-io/milvus/issues/45890

ComputePhraseMatchSlop accepts three pararms:
1. A string: query text
2. Some trings: data texts
3. Analyzer params,

Slop will be calculated for the query text with each data text in the
context of phrase match where they are tokenized with tokenizer with
analyzer params.

So two array will be returned:
1. is_match: is phrase match can sucess
2. slop: the related slop if phrase match can sucess, or -1 is cannot.

---------

Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
2025-12-19 16:03:18 +08:00
yihao.daiandGitHub 889505872a enhance: Return FlushAllMsg in response (#46347)
issue: https://github.com/milvus-io/milvus/issues/45919

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-12-16 10:35:16 +08:00
sijie-ni-0214andGitHub f51de1a8ab feat: support TruncateCollection api to clear collection data (#46167)
issue: https://github.com/milvus-io/milvus/issues/46166

---------

Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
2025-12-12 10:31:14 +08:00
yihao.daiandGitHub f32f2694bc enhance: Implement new FlushAllMessage and refactor flush all (#45920)
This PR:
1. Define and implement the new FlushAllMessage.
2. Refactor FlushAll to flush the entire cluster.

issue: https://github.com/milvus-io/milvus/issues/45919

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-12-10 19:27:13 +08:00
liliu-zandGitHub 3f063a29b0 feat: Support Search By PK (#45820)
issue: #39157

Overview:
Support search by PK by resolving IDs to vectors on Proxy side. Upgrade
go-api to adapt to new proto definitions.

Design:
- Upgrade milvus-proto/go-api to latest master.
- Implement handleIfSearchByPK in Proxy: resolve IDs to vectors via
internal Query, then rewrite SearchRequest.
- Adapt to 'SearchInput' oneof field in SearchRequest across client and
handlers.
- Fix binary vector stride calculation bug in placeholder utils.

Compatibility:
- Old Pymilvus can still work w/o this feature

What is included:
- Dense and Sparse
- Multi vector fields
- Rejection on BM25

What is **not** include:
- Hybrid Search
- EmbeddingList
- Restful API

Signed-off-by: Li Liu <li.liu@zilliz.com>
2025-12-10 10:59:14 +08:00
tinswzyandGitHub 1917bb720f enhance: add fallback mechanism for WP when accessing object storage without Condition Write support (#45735)
related issue: #45733 
related [wp issue:
#60](https://github.com/zilliztech/woodpecker/issues/60)

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2025-12-07 21:59:11 +08:00