## What
- Upgrade the root, pkg, client, test, and telemetry example modules
from Go 1.26.4 to Go 1.26.5.
- Upgrade CPU/GPU builder Dockerfiles, macOS CI Go setup, KRTE build
arg, rpm setup, meta-migration builder, and go-client test image to Go
1.26.5.
- Point CPU and GPU builder consumption in `.env` to the successful
build-env tag `20260714-c135601`.
## Why
- Latest direct Trivy scan of
`milvusdb/milvus:master-20260711-1993feae-amd64` reports CVE-2026-39822
in Go stdlib 1.26.4, fixed in 1.26.5.
- The normal daily image scan is currently blocked through Jenkins
`milvus_scan_image_daily` #540 with `exec format error`; last successful
scan is #508 from 2026-06-10, so this was verified by direct image scan.
## Image coverage
- Go toolchain fixes are shared by all Milvus runtime images; no per-OS
runtime Dockerfile package change is required.
- Builder definitions are updated for CPU ubuntu22.04, ubuntu20.04,
ubuntu24.04, amazonlinux2023, rockylinux9 and GPU ubuntu22.04,
ubuntu20.04.
- No runtime image variant is deliberately skipped; this is not an
OS-package CVE.
## Verification
- `go.dev/dl/?mode=json&include=all` confirms go1.26.5 is stable and
linux amd64/arm64 tarballs exist.
- `go env GOTOOLCHAIN GOVERSION` selects `auto` / `go1.26.5` after the
module update.
- `go list -m` succeeds for root, pkg, client, tests/go_client, and both
telemetry examples.
- `cd pkg && go test -tags dynamic,test -gcflags="all=-N -l" -count=1
./util/typeutil` passes after rebasing onto the latest upstream master.
- Jenkins build-env #40 succeeded and published builder tag
`20260714-c135601`; `.env` now selects that tag for CPU and GPU CI
builds.
- A repo-wide audit found no remaining Go 1.26.4 toolchain pins.
Note: full root `go mod tidy` is not included because this checkout has
a root-owned `deployments/docker/dev/volumes/etcd/member` directory that
makes `go mod tidy` fail with permission errors; `go mod tidy -e`
subsequently OOMed locally. No go.sum changes were produced by the
successful module tidies.
Generated by autonomous CVE maintenance.
---------
Signed-off-by: Li Liu <li.liu@zilliz.com>
issue: #50573
## What this PR does
This PR fixes `DumpMessages` transaction handling. When the WAL scanner
returns an assembled transaction message, `DumpMessages` now expands it
into separate responses:
- begin transaction message
- data messages that pass the existing message type filter
- commit transaction message
This allows recovery and CDC consumers to decode and replay transaction
data messages correctly.
The PR also supports inclusive start semantics through
`include_start_message`, while preserving the existing default behavior
of starting after `start_message_id` for compatibility.
## Tests
- Added unit coverage for default StartAfter policy.
- Added unit coverage for IncludeStartMessage StartFrom policy.
- Added unit coverage for transaction expansion and send error handling.
- Ran `gofmt` and `git diff --check`.
Linked proto change: milvus-io/milvus-proto#616
Signed-off-by: alohaha22 <shawn.work1229@outlook.com>
## What
Milvus' own code in `pkg` still imported `github.com/shirou/gopsutil/v3`
(cpu, disk, mem, process), while the root module already depends on
`gopsutil/v4` — so the codebase carried **both** major versions. This
migrates the six `pkg` files to `gopsutil/v4`, unifying our code on one
major version.
## Why
- One gopsutil major version across the repo instead of two.
- `gopsutil/v4` drops the `github.com/shoenig/go-m1cpu` (darwin/arm64
cgo) dependency. See the note below — this PR does not fully remove it,
but it removes our code's reliance on the v3 path.
## How
Pure import-path change `gopsutil/v3/*` → `gopsutil/v4/*` in:
- `pkg/util/hardware/hardware_info.go`
- `pkg/util/hardware/mem_info.go`
- `pkg/util/hardware/mem_info_darwin.go`
- `pkg/util/hardware/mem_info_windows.go`
- `pkg/util/paramtable/component_param.go`
- `pkg/util/vralloc/alloc.go`
The v3 APIs in use (`cpu.Percent`/`cpu.Times`, `mem.VirtualMemory`,
`disk.Usage`, `process.NewProcess`) are unchanged in v4, so there is
**no behavioral difference**. `go.mod`/`go.sum` updated and tidied for
`pkg` and `client`.
## Notes for reviewers
- **go-m1cpu is not fully removed.** `gopsutil/v3` (and its transitive
`go-m1cpu`) is still pulled as an indirect dependency by
`pulsar-client-go` and `woodpecker`, whose latest releases remain on v3.
Fully dropping it requires those upstreams to move to v4. go-m1cpu only
compiles on darwin/arm64, so it is never built in Linux production.
- **Root `go.mod` is unchanged** — it already had `gopsutil/v4` as a
direct dependency.
- **`client/go.mod` churn:** besides the v3→v4 indirect bump, `go mod
tidy` incidentally cleans up pre-existing drift in the client module's
indirect list (stale etcd/otel entries no longer needed). Verified with
Go 1.26.4, matching the module `go` directive.
## Verification
- `go build` of affected `pkg` packages and the `client` module: pass
- `go test -tags dynamic,test` for `pkg/util/hardware` and
`pkg/util/vralloc`: pass
issue: #50887🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
issue: #43638
update woodpecker to [release
v0.1.31](https://github.com/zilliztech/woodpecker/releases/tag/v0.1.31)
* fix(dashboard): add server runtime metrics
* fix(meta): honor configured etcd metadata prefix
* fix: make condition write auto mode fail on verification errors
* feat: split cluster, region, and az topology metadata
update woodpecker to [release
v0.1.32](https://github.com/zilliztech/woodpecker/releases/tag/v0.1.32)
* enhance: gate bucket check/create on createBucket for service mode
* enhance(docker): run server images as root by default
* enhance(docker): bundle wp (wpcli) into server images for zero-config
in-pod ops
* fix(objectstorage): silence idle-spin DEBUG logs in MinioFileWriter
sync loop , **related issue:** #48976
* fix: cut read-path DEBUG log volume (per-read verbosity + idle
tail-read spin)
* feat(quorum): make client-side quorum config runtime-tunable via
Dynamic[T]
* feat(metrics): active-segment node metric + client dashboard panels
* feat: delete-log lifecycle with async local reclaim
update woodpecker to [release
v0.1.33](https://github.com/zilliztech/woodpecker/releases/tag/v0.1.33)
* fix(reader): allow reopening at a truncated-but-not-yet-GC'd position
instead of force-advancing to the truncation point
---------
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
Update tikv client-go version because we depend on the committs of
returned value by `txn.Get`. And previous grpc version used by client-go
has some deprecated functions and 8.5 update grpc.
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
- Wire credential descriptions through create/update/read paths,
including proxy, HTTP v2, Go SDK options, length validation, optional
update presence handling, and internal credential proto/model
persistence.
- Preserve existing encrypted passwords and descriptions during
RootCoord credential updates when either field is omitted, reject
all-empty or partial direct password updates, and keep sha256 passwords
cache-only.
- Skip auth-cache refresh for description-only credential updates so
existing authentication state is not blanked, while still refreshing
caches for password updates.
- Return persisted user descriptions from select_user and HTTP v2
describe_user, including include_role_info=false, and reuse
already-loaded credential rows in user listing and RBAC backup to avoid
duplicate etcd reads while ignoring malformed credential keys.
- Use the upstream milvus-proto go-api/v3 version that includes the
credential description proto change; the temporary fork replaces in
root, pkg, client, and tests/go_client modules have been removed.
design doc:
docs/design-docs/design_docs/20260601-rbac-user-description.md
related: #50179
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
## Summary
- Bump **Go to 1.26.4** across CI workflows, Docker builders, RPM setup,
and the `go` directive in all 4 modules (+ examples).
- Refresh Go 1.26-compatible deps to latest: `sonic` v1.15.2, `mockey`
v1.4.6, `gopkg` v0.1.4, `sonic/loader` v0.5.1, `cpuid/v2` v2.3.0. sonic
1.14.x fails to build on the 1.26 compiler (bytedance/sonic#895).
- **datacoord tests:** replace the anonymous `struct{ Iface }` mockey
idiom with **named helper types**
(`embeddedHandler`/`embeddedBroadcastAPI`/`embeddedBroker`/`embeddedAllocator`).
## Why the datacoord test change
Go 1.26's `printf` vet pass (run automatically by `go test`) calls
`x/tools/refactor/satisfy.Finder` unconditionally and **panics
`(*ast.StructType)`** on a **method expression of `*struct{ Iface }`** —
an upstream Go toolchain bug. Only `internal/datacoord` used this mockey
idiom (31 sites in 2 test files; verified repo-wide it's the only
place), which is why only `build-ut-cov`/`ut-go` failed on
`internal/datacoord [build failed]`. Switching to a **named type** makes
the method-expression receiver an identifier instead of a struct
literal, so `satisfy.Finder` no longer panics. mockey usage and test
behavior are unchanged. Minimal repro: `type I interface{ Logf(string,
...any) }` + `var _ = (*struct{ I }).Logf` under `go 1.26` → `go vet`
panics; named type does not.
## Notes
- `.env` builder-image tag intentionally not hand-edited (auto-bumped
post-merge by `bump-builder-version`).
- The bug also warrants an upstream report to golang/go (reproducer
above).
## Test plan
- [x] `go mod tidy` clean across all four modules
- [x] sonic 1.15.2 + mockey compile on go 1.26.4
- [x] reproduced the vet panic; verified named-type method expression
avoids it; gofmt-clean
- [ ] Full CI (build / build-ut-cov / ut-go / ut-cpp / integration)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
- Go 1.25.9 -> 1.25.10 and builder Dockerfiles now use Go 1.25.10. This
covers CVE-2026-33811, CVE-2026-33814, CVE-2026-39820, CVE-2026-39823,
CVE-2026-39825, CVE-2026-39826, CVE-2026-39836, CVE-2026-42499.
- golang.org/x/crypto -> v0.52.0 across root/pkg/client/tests_go_client.
This covers CVE-2026-39829, CVE-2026-39830, CVE-2026-39831,
CVE-2026-39832, CVE-2026-39833, CVE-2026-39834, CVE-2026-42508,
CVE-2026-46595, CVE-2026-46597.
- golang.org/x/net -> v0.55.0 across root/pkg/client/tests_go_client.
This covers CVE-2026-33814 and CVE-2026-39821.
- github.com/apache/thrift v0.20.0 -> v0.23.0, including
tests_go_client. This covers CVE-2026-41602.
- Updated tests/go_client/go.mod and go.sum so Go SDK CI has the
required x/* sum entries.
This PR is generated by the automated CVE fix task.
issue: https://github.com/milvus-io/milvus/issues/49032
## Test plan
- go mod tidy in pkg/
- go mod tidy in client/
- go mod tidy in tests/go_client/
- go mod tidy -e at repository root (standard tidy is blocked by
existing missing generated package cmd/tools/migration/legacy/legacypb)
- git diff --check
- go list -tags L0,L1,L2,test ./... in tests/go_client
- go list -m for updated dependencies in root/pkg/client/tests_go_client
- CI validates image build, Go SDK e2e, and scan
---------
Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
issue: #43638
related wp issue:
https://github.com/zilliztech/woodpecker/compare/v0.1.28...v0.1.29
- stagedstorage: replace per-writer sync goroutines with a shared sync
scheduler and bounded worker pool while preserving per-writer flush
ordering
- fix: harden writer lifecycle transitions to reject late writes during
finalize/close/fence/storage-failure states
- enhance: support Azure managed identity authentication for the storage
client
- fix(storage): enable sealed segment direct read by default with
on-demand reader lifecycle management and config validation
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
## Summary
Bumps `pulsar-client-go` v0.17.0 → v0.19.0 to pick up
[apache/pulsar-client-go#1457](https://github.com/apache/pulsar-client-go/pull/1457):
the client used to give up reconnecting after a topic hit the
`producer_exception` backlog quota, leaving the producer permanently
dead and writes timing out forever. v0.19.0 treats the quota exception
as retryable, so producers recover once backlog clears.
The bump can't ship alone: it's blocked at the module graph by Milvus's
pinned `xiaofan-luan/pulsarctl v0.5.1` fork, which imports
`pulsar-client-go/oauth2/store` — a package v0.19.0 has removed. MVS
then pulls the legacy standalone `pulsar-client-go/oauth2` module back
in, which collides with the main module's `oauth2/clock` (ambiguous
import). To unblock cleanly, this PR also replaces the only ~50 lines
of pulsarctl admin usage in Milvus with `pulsar-client-go`'s own
`pulsaradmin` package (shipped in the same module). The legacy
`99designs/{keyring,go-keychain}` and `danieljoos/wincred` transitive
deps fall out as a side effect.
## Changes
**Source (2 files, ~20 lines net)**
- `pkg/mq/msgstream/mqwrapper/pulsar/pulsar_client.go` —
`NewAdminClient` now returns `admin.Client` from
`pulsaradmin/pkg/admin`,
built from `config.Config` instead of `common.Config`.
- `pkg/mq/msgstream/mq_factory.go` —
`Subscriptions().Delete(*topic, sub, true)` → `ForceDelete(*topic, sub)`
(pulsaradmin splits force-delete into its own method);
error type assertion `cli.Error` → `rest.Error` (same `Reason` field).
**Module hygiene (`pkg/go.mod`, `go.mod`)**
- bump `apache/pulsar-client-go` v0.17.0 → v0.19.0
- drop `require github.com/streamnative/pulsarctl`
- drop `replace streamnative/pulsarctl => xiaofan-luan/pulsarctl v0.5.1`
- drop `exclude pulsar-client-go/oauth2 v0.0.0-20211108044248-...`
(no longer needed; the standalone oauth2 module is no longer in the
graph at all — `go list -m .../oauth2` ⇒ *not a known dependency*)
## Behavior notes
v0.19.0 still calls `failPendingMessages` each time the producer
reconnect hits the quota, so in-flight sends continue to fail with
`ErrProducerBlockedQuotaExceeded` — upper-layer (msgstream / WAL) error
surface is **unchanged**. What changes is that the producer no longer
permanently dies and recovers automatically once backlog clears.
API surface diff verified against Milvus's pulsar usage: no removed or
renamed symbols; option structs only additively changed
(`ConsumerOptions.NackPrecisionBit` added). v0.19.0 requires `go 1.24`;
Milvus is on `go 1.25.8` — no Go version gate.
issue: #49932
## Test plan
- [x] `go build ./...` in `pkg/` clean (with rocksdb pkg-config)
- [x] `go vet -tags dynamic,test` clean on msgstream +
walimpls/impls/pulsar
- [x] pulsar test code compiles under `-tags dynamic,test
-gcflags="all=-N -l"`
- [x] Non-broker unit tests pass: `TestPulsarID_Serialize`,
`TestPulsarID_Equal`
- [x] `go list -m .../pulsar-client-go/oauth2` ⇒ `not a known
dependency`
(ambiguous-import root cause structurally resolved)
- [ ] CI green
- [ ] Broker-dependent unit tests pass against Pulsar 3.0.2 (matches
milvus-helm `pulsarv3` subchart)
- [ ] Pre-merge: induce `producer_exception` backlog quota in a staging
cluster; verify producer recovers after backlog clears
(independent reproduction already done out-of-tree)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Related to #49398
Bump Go module references from pkg/v2 and milvus-proto/go-api/v2 to
pkg/v3 and milvus-proto/go-api/v3 so the client tracks the Milvus 3.x
release line.
This prepares the repository for the upcoming 3.x.y release by aligning
imports, module dependencies, and proto API references with the new
major-version module paths.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
## Summary
- add `search_agg` package in proxy for context build, recursive
aggregation compute, and bucket ordering
- wire `group_by` aggregation through `task_search` and built-in
pipeline (`searchWithAggPipe` + aggregate operator serialization)
- upgrade querynode reduce key extraction for agg multi-field grouping,
and update related proxy/querynode tests
- keep local proto iteration path and local core build job defaults
aligned for this development workflow
## Test plan
- [x] bash
/home/zilliz/hc-claude-projects/hc-milvus-projects/search-aggregation/scripts/verify_baseline.sh
- [x] direct association tests passed
- [x] indirect association tests passed
issue: #49046🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
## Summary
- Inject Milvus zap logger into hook plugin via `SetZapLogger` type
assertion after plugin load, enabling the plugin to use Milvus's
centralized logging
- Inject connection manager into hook plugin via `SetClientInfoProvider`
type assertion, allowing the plugin to look up SDK info (sdk_type,
sdk_version, host) for non-Connect requests by connection identifier
- Pass `*gin.Context` directly via `GinParamsKey` instead of
`ginCtx.Keys` map, enabling the hook plugin to access `ClientIP()`,
`FullPath()`, `GetHeader()` and other gin context methods for RESTful
requests
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
issue: #49241
pr: https://github.com/milvus-io/milvus-proto/pull/586
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260423-array_partial_update_ops.md
## Summary
Phase-1 implementation of field-level partial update operators for Array
fields. Lets clients mutate a single array without resending the full
value on upsert.
Supported ops (see `schemapb.FieldPartialUpdateOp.OpType`):
- `REPLACE` — legacy semantics (full overwrite). Default when no op is
set.
- `ARRAY_APPEND` — concatenate payload to base (duplicates preserved,
in-order).
- `ARRAY_REMOVE` — delete **all** occurrences matching any value in
payload.
## Changes
### Proto surface (via
https://github.com/milvus-io/milvus-proto/pull/586)
- `UpsertRequest.field_ops` carries per-field directives; `FieldData`
stays a pure data carrier (no leakage into
Insert/Query/Search/msgstream).
### Proxy (`internal/proxy/task_upsert*.go`)
- `validateFieldPartialUpdateOps` rejects: empty field name, duplicate
field, op on PK, unknown field, non-Array field, element-type mismatch,
unsupported op enum, op without matching FieldData, ARRAY_APPEND payload
+ base > `max_capacity`.
- Any non-REPLACE op auto-promotes `partial_update=true` (user does not
need to set both).
- In `queryPreExecute` merge step, dispatches per field: REPLACE →
`UpdateFieldDataByColumn`; ARRAY_APPEND/REMOVE →
`UpdateArrayFieldByColumnWithOp`.
### typeutil (`pkg/util/typeutil/schema.go`)
- `UpdateArrayFieldByColumnWithOp` (Array-only entry point) +
`ApplyArrayRowOp` + `appendArrayRow` / `removeArrayRow` cover all Array
element types (Bool/Int32/Int64/Float/Double/VarChar).
- REMOVE deletes every match (aligns with MongoDB `$pull`, Postgres
`array_remove`).
- Float/Double REMOVE: `NaN != NaN` (IEEE 754).
- ARRAY_APPEND enforces `max_capacity` per merged row and returns a
descriptive error.
### Go SDK (`client/v2`)
- `columnBasedDataOption.WithArrayAppend(field)` /
`WithArrayRemove(field)` / `WithFieldPartialOp(field, op)` build
`UpsertRequest.FieldOps`.
- REPLACE clears any prior directive for that field; non-REPLACE ops
auto-promote `PartialUpdate=true`.
- Unknown-field ops are forwarded as-is so the server can return the
descriptive validation error (client does not silently drop).
### Integration test
(`tests/integration/hellomilvus/upsert_partial_op_test.go`)
End-to-end insert → op-upsert → query flow exercising: ARRAY_APPEND,
ARRAY_REMOVE (incl. no-op row + multi-match), ARRAY_APPEND-over-capacity
rejection. Does **not** set `partial_update=true` — asserts the server
auto-promotes.
## Forward compatibility
Older servers that don't understand `field_ops` treat the request as a
plain upsert (proto3 unknown-field tolerance) → graceful fallback to
REPLACE semantics with a client-side warning path available.
## Test plan
- [x] typeutil UT: 40+ cases covering all element types, empty/nil
inputs, capacity edge cases, NaN semantics, unsupported-type rejection
- [x] proxy UT: 20+ validation cases (PK, unknown field, type mismatch,
capacity precheck, duplicate, empty name, op without FieldData)
- [x] go-sdk UT: 10 cases — auto-promote, REPLACE-clears-prior,
row-based + column-based builders, multi-op emission
- [x] integration test: append / remove / capacity-overflow end-to-end
- [x] `go mod tidy` in root / pkg / client
- [ ] CI green
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
## Summary
Refactor search-side GROUP BY from single-field to multi-field composite
key in the **segcore (C++) layer**, aligning with Elasticsearch's
`multi_terms` aggregation semantics.
**Scope: segcore layer only.** Go-layer changes (proxy search reduce,
delegator multi-field wiring, SDK API) will be tracked and submitted in
a separate PR.
## Core Changes
- **CompositeGroupKey** (Types.h): New composite key type
(vector<GroupByValueType>) with CompositeGroupKeyHash using
bits::hashMix, consistent with query-agg HashTable.h
- **MultiFieldDataGetter** (SearchGroupByOperator.{h,cpp}): Type-erased
multi-field data reader via std::function<GroupByValueType(int64_t)>,
replacing ~300 lines of per-type template dispatch
- **GroupByMap<CompositeGroupKey>** specialization for composite key
grouping with hash collision handling
- **SearchInfo.group_by_field_ids_** (QueryInfo.h): Replaced
optional<FieldId> group_by_field_id_ with vector<FieldId>
group_by_field_ids_; added has_group_by() / has_multi_field_group_by()
helpers
- **Proto**: New `repeated int64 group_by_field_ids = 17` in QueryInfo,
with backward compat for old group_by_field_id
- **Reduce pipeline**: GroupReduce, Reduce::SortEqualScoresOneNQ, and
AssembleCompositeGroupByValues all updated for composite keys
- **JSON group_by**: Restored proper json_type dispatch
(bool/int8/16/32/64/varchar) in MultiFieldDataGetter
- **Dead code cleanup**: Removed old AssembleGroupByValues() (~290
lines), removed write-only SearchResultPair.group_by_value_ field
## What This Enables (for follow-up Go-layer PR)
This commit provides the segcore foundation for the **Proxy-Only
architecture**:
1. Proxy amplifies group_count and group_size, flattens all nested
levels into a flat multi_terms composite key
2. Segcore / Delegator / QN execute standard flat group_by with
amplified parameters — zero awareness of nesting or metrics
3. Proxy reconstructs the nested hierarchy, computes metrics from
returned docs, and trims to user-requested group_size
The Go-layer PR will wire up: proto group_by_field_values (plural)
reading, proxy search reduce with composite keys, metric field injection
into output_fields, and nested reconstruction logic.
The full feature surface (per-group metrics R4, intra-bucket sort R5,
**bucket-level ordering / per-level-orderby R7**) is described in the
design doc below.
## Test plan
- [x] 5 existing single-field GroupBY.* tests adapted to composite key
path — all pass
- [x] 3 existing GroupBYJSON.* tests adapted — all pass
- [x] ElementFilter group_by interaction tests adapted — all pass
- [x] New GroupBY.CompositeKeySealedIndex — dual-field (int32+int64)
sealed+HNSW, strict_group_size=true, ground truth verification via
seg_offset -> raw_data lookup, exact group count and per-group size
assertions
- [x] New GroupBY.CompositeKeyGrowing — dual-field (int8+int32) growing
segment, exact 6 groups x group_size=3, every group fully filled
- [x] New GroupBY.CompositeKeyReduce — dual-field cross-segment C-API
reduce via ReduceSearchResultsAndFillData, PK dedup + composite
group_size validation
- [x] New GroupBY.SingleFieldViaCompositeProtoPath — single field via
new group_by_field_ids proto, backward compat with exact 7 groups x
group_size=3 = 20 results
- [x] **All 68 guarded tests passing**, clang-format clean
issue: #49046
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260413-search_embedded_agg.md
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary
- Refactor snapshot name uniqueness from global to per-collection scope
- Add cascade delete: DropCollection triggers DropSnapshotsByCollection
- Add orphan snapshot GC for deleted collections
- Add database-level filtering for ListSnapshots and
ListRestoreSnapshotJobs
- Distinguish source/target collection in RestoreSnapshot API
- Move snapshot privileges from Global level to Collection level
- Update client SDK and documentation for new API semantics
issue: #44358
issue: #47890
issue: #47883
issue: #47855
## Test plan
- [x] Unit tests for snapshot_meta (DropSnapshotsByCollection,
per-collection isolation, partial failure)
- [x] Unit tests for snapshot_manager (DropSnapshotsByCollection,
getDBCollectionIDs)
- [x] Unit tests for services (ListSnapshots/ListRestoreJobs with dbID,
RestoreSnapshot with source collectionID)
- [x] Unit tests for ddl_callbacks_snapshot (new
dropSnapshotsByCollection callback)
- [x] Unit tests for garbage_collector (orphan snapshot GC)
- [x] E2E tests for cross-database snapshot isolation
- [ ] CI validation
## Note on skipped Python E2E tests
All 18 snapshot test classes in
`tests/python_client/milvus_client/test_milvus_client_snapshot.py` are
temporarily skipped with `@pytest.mark.skip`. Reason: this PR changes
snapshot APIs (DropSnapshot, DescribeSnapshot, RestoreSnapshot) to
require `collection_name` as a mandatory parameter, but the pymilvus SDK
used in CI has not been updated to pass this parameter yet. The tests
will be re-enabled once pymilvus SDK is updated to match the new API
contract.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20251114-snapshot_design.md
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
## Summary
- Bump `github.com/go-jose/go-jose/v4` from v4.1.3 to v4.1.4 in both
root `go.mod` and `pkg/go.mod`
- Fixes **CVE-2026-34986** (HIGH): Go JOSE panics during JWE decryption
when processing a crafted token, leading to denial of service
## Changes
- `go.mod` / `go.sum`
- `pkg/go.mod` / `pkg/go.sum`
## References
- https://nvd.nist.gov/vuln/detail/CVE-2026-34986
- https://github.com/go-jose/go-jose/releases/tag/v4.1.4
## Test plan
- [x] `go mod verify` passes in both root and `pkg/`
- [ ] CI passes
- [ ] Next daily image scan confirms CVE-2026-34986 is gone
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary
- Upgrade `google.golang.org/grpc` from v1.71.1 to v1.79.3
(CVE-2026-33186 **CRITICAL** - Authorization Bypass via improper HTTP/2
`:path` pseudo-header validation)
- Upgrade `github.com/quic-go/quic-go` from v0.54.0 to v0.54.1
(CVE-2025-59530 **HIGH** - DoS via premature HANDSHAKE_DONE frame)
- Updated across all Go modules (root, pkg, client, tests/go_client)
issue: #48574
## Test plan
- [ ] CI passes
- [ ] Image scan clean for CVE-2026-33186 and CVE-2025-59530
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
issue: #48574
## Summary
- Upgrade Go from 1.24.12 to 1.25.8 across all go.mod files and
Dockerfiles
- Fixes CVE-2025-68121 (CRITICAL), CVE-2026-27142 (HIGH), CVE-2026-25679
(HIGH) in Go stdlib
- All three CVEs affect the Go standard library and are resolved by
upgrading to Go 1.25.8
## Changes
- Updated `go` directive in go.mod files (root, pkg/, client/,
tests/go_client/, examples/)
- Updated Go download URLs in Dockerfiles (build/docker/builder/)
- Updated `toolchain` directives where present
issue: https://github.com/milvus-io/milvus/issues/TBD
## Test plan
- [x] `make milvus` builds successfully with Go 1.25.8
- [ ] CI passes
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
issue: https://github.com/milvus-io/milvus/issues/47598
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260205-data_salvage_for_force_failover.md
## Summary
Implement data salvage feature that enables zero data loss after force
failover by capturing and exposing the salvage checkpoint, along with a
DumpMessages API to retrieve unsynchronized messages.
### SalvageCheckpoint Capture & Persistence
- Add `salvageCheckpoint` field to `ReplicateManager`
- Capture last synced position on force promote before clearing
secondary state
- Persist to separate etcd key via `StreamingNodeCataLog`
- Load salvage checkpoint on WAL recovery
### API Exposure
- Return `salvage_checkpoint` in `GetReplicateInfoResponse`
- Add `GetSalvageCheckpoint` to `HandlerClient` and `ReplicateService`
### DumpMessages Streaming API
- Implement `DumpMessages` RPC for streaming WAL messages
- Support `start_timetick` and `end_timetick` filters
- Filter out system messages (TimeTick, CreateSegment, Flush,
RollbackTxn)
- Use oneof response for status or message
### Tests
- Unit tests for salvage checkpoint capture
- Unit tests for `shouldDumpMessage` filter
- Integration tests for `GetReplicateInfo` and `DumpMessages`
## Test plan
- [ ] Unit tests pass
- [ ] Integration tests pass
- [ ] Manual test with force failover scenario
proto pr: https://github.com/milvus-io/milvus-proto/pull/568🤖 Generated with [Claude Code](https://claude.ai/code)
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Signed-off-by: Yihao Dai <yihao.dai@zilliz.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## Summary
- Upgrade Go dependencies across root, pkg, and client modules:
- `casbin/casbin` v2.44.2 → v2.135.0
- `gin-gonic/gin` v1.9.1 → v1.11.0
- `samber/lo` v1.27.0 → v1.52.0
- `cockroachdb/errors` v1.9.1 → v1.12.0
- `google.golang.org/protobuf` v1.36.5 → v1.36.9
- Adapt source code to breaking API changes:
- `lo.Last()` now returns `(T, bool)` instead of `(T, error)` (lo v1.52)
- `gin.LogFormatterParams.Keys` is now `map[any]any` instead of
`map[string]any` (gin v1.11)
issue: https://github.com/milvus-io/milvus/issues/33482
## Test plan
- [x] `go mod tidy` clean on all three modules (root, pkg, client)
- [x] Local lint passes with no new errors
- [ ] CI code-check passes
- [ ] CI ut-go passes
- [ ] CI e2e passes
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Related to #48070
Upgrade go.opentelemetry.io/otel and related packages from v1.34.0 to
v1.40.0 across all Go modules to address CWE-426 (Untrusted Search Path)
vulnerability. Also bumps transitive dependencies including auto/sdk
v1.1.0 -> v1.2.1, go-logr v1.4.2 -> v1.4.3, and golang.org/x/sys v0.38.0
-> v0.40.0.
See: https://cwe.mitre.org/data/definitions/426.html
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #46199
## Summary
Remove 5 unused or misused Go dependencies to reduce module bloat and
consolidate overlapping libraries:
- **`mgutz/ansi`** → replaced with inline ANSI escape codes (only used
for 3 color constants in migration console)
- **`valyala/fastjson`** → replaced with `tidwall/gjson` (only 1 file
used fastjson; gjson is already used in 22+ files)
- **`google.golang.org/grpc/examples`** → replaced with existing
`rootcoordpb` (test file pulled in entire grpc examples repo for a mock
server)
- **`remeh/sizedwaitgroup`** → replaced with `chan` semaphore +
`sync.WaitGroup` (only 2 files, trivial pattern)
- **`pkg/errors`** → replaced with `cockroachdb/errors` (the project
standard; `pkg/errors` was used in 1 file)
## Behavior change: DeleteLog.Parse() fail-fast on missing fields
The `fastjson` → `gjson` migration adds explicit `Exists()` validation
for `ts`, `pk`, and `pkType` fields in the JSON parsing branch.
Previously, both fastjson and gjson would silently return zero values
for missing fields, causing `dl.Pk` to remain nil and panicking
downstream. The new code fails fast with a descriptive error at parse
time. This is a defensive improvement (the original code had identical
silent-failure behavior).
## Performance impact
| Change | Path type | Perf delta | Matters? |
|--------|-----------|------------|----------|
| `pkg/errors` → `cockroachdb/errors` | Cold (offline CLI tool
`config-docs-generator`) | Negligible | No |
| `mgutz/ansi` → inline ANSI codes | Cold (offline CLI tool
`migration/console`) | Marginally faster (eliminates map lookup) | No |
| `fastjson` → `gjson` (`DeleteLog.Parse`) | Warm — old-format deltalog
deserialization only | **~2.5x slower** per JSON parse (143ns→361ns) |
**No** — see below |
| `grpc/examples` → `rootcoordpb` | Test only (`client_test.go`) | None
| No |
| `sizedwaitgroup` → chan+WaitGroup | Test only (`wal_test.go`,
`test_framework.go`) | None | No |
### fastjson → gjson regression detail
`DeleteLog.Parse()` is called per-row during deltalog deserialization,
but **only for the legacy single-field format**. The new multi-field
parquet format (`newDeltalogMultiFieldReader`) reads pk/ts as separate
Arrow columns and bypasses `Parse()` entirely. Legacy deltalogs are
rewritten to parquet format during compaction, so this is a dying code
path. Additionally, deltalog loading is I/O-bound — the JSON parse cost
(~361ns/row) is negligible compared to disk read and Arrow
deserialization overhead.
Benchmark (Go 1.24, arm64):
```
BenchmarkFastjsonSmall-4 8,315,624 143.1 ns/op 0 B/op 0 allocs/op
BenchmarkGjsonOptimized-4 3,321,613 361.4 ns/op 96 B/op 1 allocs/op
```
## Test plan
- [x] CI build passes
- [x] CI code-check passes
- [ ] CI ut-go passes
- [ ] CI e2e passes
- [x] Boundary test cases added (bare number, missing pkType/ts/pk)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260105-external_table.md
issue: #45881
This change introduces manual refresh capability for external
collections, allowing users to trigger on-demand data synchronization
from external sources. It replaces the legacy update mechanism with a
more robust job-task hierarchy and persistent state management.
Key changes:
- Add RefreshExternalCollection, GetRefreshExternalCollectionProgress,
and ListRefreshExternalCollectionJobs APIs across Client, Proxy,
and DataCoord
- Implement ExternalCollectionRefreshManager to manage refresh jobs
with a 1:N Job-Task hierarchy
- Add ExternalCollectionRefreshMeta for persistent storage of jobs and
tasks in the metastore
- Add ExternalCollectionRefreshChecker for task state management and
worker assignment
- Implement ExternalCollectionRefreshInspector for periodic job
cleanup
- Use WAL Broadcast mechanism for distributed consistency and
idempotency
- Replace legacy external_collection_inspector and update tasks with
the new refresh-based implementation
- Add comprehensive unit tests for refresh job lifecycle and state
transitions
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260105-external_table.md
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
- Upgrade Go from 1.24.11 to 1.24.12 to fix CVE-2025-61726 (net/url
query parameter DoS) and CVE-2025-61728 (archive/zip CPU exhaustion)
- Upgrade gpgv in Ubuntu Dockerfiles to fix CVE-2025-68973 (GnuPG
out-of-bounds write vulnerability)
Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## Summary
Add a new public API `GetReplicateConfiguration` that allows cluster
administrators to view the current cross-cluster replication topology
with sensitive connection parameters (tokens) redacted.
## Changes
- Add privilege constant `PrivilegeGetReplicateConfiguration` to
`ClusterReadOnlyPrivileges`
- Add `SanitizeReplicateConfiguration` helper to strip sensitive tokens
before returning
- Add `GetReplicateConfiguration` method to `ReplicateService` interface
- Implement `GetReplicateConfiguration` handler in Proxy
- Add integration tests
## API
```protobuf
rpc GetReplicateConfiguration(GetReplicateConfigurationRequest) returns (GetReplicateConfigurationResponse) {}
```
**Security:**
- Requires ClusterAdmin privilege
- Tokens are redacted from the response
## Dependencies
- Proto changes: milvus-io/milvus-proto#566
## Related Issue
Closes#47392
## Design Doc
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260128-get_replicate_configuration.md
## Test Plan
- [x] Unit tests for sanitization helper
- [x] Unit tests for ReplicateService method
- [x] Integration tests for the API
---
🤖 Generated with [Claude Code](https://claude.ai/code)
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Happy <yesreply@happy.engineering>
Support crate analyzer with file resource info, and return used file
resource ids when validate analyzer.
Save the related resource ids in collection schema.
relate: https://github.com/milvus-io/milvus/issues/43687
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
- Core invariant: analyzer file-resource resolution is deterministic and
traceable by threading a FileResourcePathHelper (collecting used
resource IDs in a HashSet) through all tokenizer/analyzer construction
and validation paths; validate_analyzer(params, extra_info) returns the
collected Vec<i64) which is propagated through C/Rust/Go layers to
callers (CValidateResult → RustResult::from_vec_i64 → Go []int64 →
querypb.ValidateAnalyzerResponse.ResourceIds →
CollectionSchema.FileResourceIds).
- Logic removed/simplified: ad‑hoc, scattered resource-path lookups and
per-filter file helpers (e.g., read_synonyms_file and other inline
file-reading logic) were consolidated into ResourceInfo +
FileResourcePathHelper and a centralized get_resource_path(helper, ...)
API; filter/tokenizer builder APIs now accept &mut
FileResourcePathHelper so all file path resolution and ID collection use
the same path and bookkeeping logic (redundant duplicated lookups
removed).
- Why no data loss or behavior regression: changes are additive and
default-preserving — existing call sites pass extra_info = "" so
analyzer creation/validation behavior and error paths remain unchanged;
new Collection.FileResourceIds is populated from resp.ResourceIds in
validateSchema and round‑tripped through marshal/unmarshal
(model.Collection ↔ schemapb.CollectionSchema) so schema persistence
uses the new list without overwriting other schema fields; proto change
adds a repeated field (resource_ids) which is wire‑compatible (older
clients ignore extra field). Concrete code paths: analyzer creation
still uses create_analyzer (now with extra_info ""), tokenizer
validation still returns errors as before but now also returns IDs via
CValidateResult/RustResult, and rootcoord.validateSchema assigns
resp.ResourceIds → schema.FileResourceIds.
- New capability added: end‑to‑end discovery, return, and persistence of
file resource IDs used by analyzers — validate flows now return resource
IDs and the system stores them in collection schema (affects tantivy
analyzer binding, canalyzer C bindings, internal/util analyzer APIs,
querynode ValidateAnalyzer response, and rootcoord/create_collection
flow).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Related to #44614
Previous PR: #44666
Bump etcd version in pkg/go.mod to 3.5.23 and update test code
accordingly
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #39157
Overview:
Support search by PK by resolving IDs to vectors on Proxy side. Upgrade
go-api to adapt to new proto definitions.
Design:
- Upgrade milvus-proto/go-api to latest master.
- Implement handleIfSearchByPK in Proxy: resolve IDs to vectors via
internal Query, then rewrite SearchRequest.
- Adapt to 'SearchInput' oneof field in SearchRequest across client and
handlers.
- Fix binary vector stride calculation bug in placeholder utils.
Compatibility:
- Old Pymilvus can still work w/o this feature
What is included:
- Dense and Sparse
- Multi vector fields
- Rejection on BM25
What is **not** include:
- Hybrid Search
- EmbeddingList
- Restful API
Signed-off-by: Li Liu <li.liu@zilliz.com>
issue: #44369
woodpecker related[ issue:
#59](https://github.com/zilliztech/woodpecker/issues/59)
Refactor the WAL retention logic in Milvus StreamingNode:
- Remove the simple sampling-based truncation mechanism.
- After flush, WAL data is directly truncated.
- The retention control is now delegated to the underlying message queue
(MQ) implementation.
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
https://github.com/milvus-io/milvus/issues/45544
- Add batch_factor configuration parameter (default: 5) to control
embedding provider batch sizes
- Add disable_func_runtime_check property to bypass function validation
during collection creation
- Add database interceptor support for AddCollectionFunction,
AlterCollectionFunction, and DropCollectionFunction requests
Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
