Related to #51419
Update the Go client module path, imports, dependencies, tests,
examples, and documentation from client/v2 to client/v3.
Set the SDK version to 3.0.0 in preparation for the Milvus 3.0 release.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
## What this PR does
- Change the default `MILVUS_MINIO_ROOT_PATH` from `files` to `file` in
the Text LOB inline/object layout test.
- Keep the environment-variable override available for deployments using
a custom root path.
issue: #51257
## Verification
- `python3 -m py_compile
tests/python_client/milvus_client/test_milvus_client_text_lob.py`
- Targeted pytest node collection completed successfully.
- Verified the committed diff contains only the requested one-line
default change.
Signed-off-by: Eric Hou <eric.hou@zilliz.com>
Co-authored-by: Eric Hou <eric.hou@zilliz.com>
issue: #49156
## Summary
- Record `cost_time` / `cost_cpu_num` on DataNode index build tasks and
surface them through `QueryTask` response `Properties`.
- Reserve the unified `cost_time` / `cost_cpu_num` keys under
`pkg/taskcommon` so future PRs can wire up the other
`CreateTask`-dispatched task types.
- Introduce small `internal/datanode/taskcost` helper (`NowMs`,
`ElapsedMs`, `EstimateIndexBuildCPUNum`) for index build task cost
accounting.
- Final state/failReason and execution-end cost are written in a single
`TaskManager` critical section, and `QueryTask` reads them from a single
snapshot, so state and cost in one response are always consistent.
## Notes
This PR is producer-side only. DataCoord / proxy / metrics exporter do
not consume these properties yet; downstream consumption should land in
a follow-up linked from #49156.
`cost_cpu_num` is a coarse, statistics-only estimate for observability
(vector index ≈ knowhere build pool size = NumCPU in cluster mode;
scalar index = 1). It is not used for coordinator-side scheduling, so
the standalone pool-ratio deviation (NumCPU × buildIndexThreadPoolRatio)
and pool sharing across concurrent builds are intentionally not modeled.
Supersedes #49157 (that PR's head branch was rebased onto latest master
and force-pushed, so GitHub no longer allows reopening it). Content is
identical, rebased to a single commit on top of current master.
## Test plan
- [x] `go test ./taskcommon/...` from `pkg/`
- [x] `go test -tags dynamic,test -gcflags="all=-N -l" -count=1
./internal/datanode/taskcost`
- [ ] `go test -tags dynamic,test -gcflags="all=-N -l" -count=1
./internal/datanode/index -run
'TestIndexTaskSchedulerRecordsIndexTaskCost|Test_statsTaskInfoSuite/Test_IndexTaskCostMethods'`
- [ ] Manual sanity: `QueryTask` returns `cost_time` / `cost_cpu_num`
for DataNode index build task success and failure paths.
---------
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Problem
During schema evolution, SQN can convert an insert payload with the old
schema and then race with a schema update before ProcessInsert creates
and writes the growing segment. If the old payload still contains a
dropped field, the native collection may already use the new schema and
segcore rejects the write with unexpected new field.
## Solution
- Add a collection-local schema-transition RW lock.
- Insert holds the reader lock from schema snapshot use through payload
conversion and delegator.ProcessInsert, so a growing segment is created
and written in the same schema epoch as its payload.
- Schema update and existing-collection load refresh paths take the
writer lock around native schema mutation and Go schema snapshot
publication.
- Acquire a temporary collection lease before releasing
collectionManager.mut. This keeps the collection alive while the writer
waits, without serializing unrelated collection-manager operations
behind a schema transition.
- Preserve dropped-field replay handling: when DDL has already
completed, payload conversion filters fields absent from the current
schema before inserting into a new-schema growing segment.
## Tests
- Add a deterministic native regression: pause after a v950 payload
containing field 580 is converted, queue the v951 drop-field update,
then resume a real NewSegment plus growing.Insert and verify both insert
and DDL succeed.
- Assert the transition reader covers both payload conversion and
ProcessInsert.
- Cover the DDL-first replay path, where field 580 is filtered before
native insertion.
- Cover both schema writer entry points: UpdateSchema and
existing-collection PutOrRef.
## Validation
- make static-check
- make build-go
- Native pipeline regression tests repeated 20 times with dynamic,test
- Collection schema-transition and lease tests repeated 20 times
- internal/querynodev2/pipeline and internal/querynodev2/segments
package tests
- gofumpt -l and git diff --check
Fixes#51436
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
issue: #51493
### What
#50221 split the `status` label values of `milvus_proxy_req_count` /
`milvus_proxy_grpc_latency` in place. Released in **v2.6.19**, that
silently zeroes every existing dashboard panel and alert rule matching
`status="fail"` / `status="rejected"` — an alert that stops firing
rather than erroring.
This keeps the classification but moves it onto its own dimension,
restoring the status domain to what <= v2.6.18 emitted:
```
status: success | fail | rejected | retry | total | abandon (as before v2.6.19)
cause: user | system | cancel | na (new)
```
| v2.6.19 / v2.6.20 | this PR |
|---|---|
| `status="fail_input"` | `status="fail", cause="user"` |
| `status="fail_system"` | `status="fail", cause="system"` |
| `status="rejected_user"` | `status="rejected", cause="user"` |
| `status="rejected_system"` | `status="rejected", cause="system"` |
| `status="cancel"` | `status="fail"` or `"rejected"`, `cause="cancel"`
|
| `success` / `retry` / `total` / `abandon` | unchanged, `cause="na"` |
### Why a label instead of new status values
- **Old queries work unchanged and count each request once.**
`status="fail"` is again every hard failure; Prometheus aggregates over
`cause` for free. That rollup is what a label dimension *is* — the split
forces every consumer who just wants "how many failures" to hand-write
`status=~"fail_.*"`.
- **Cardinality is unchanged.** `cause` is functionally dependent on the
outcome, so the realized `(status, cause)` pairs are exactly the series
the split already produces. Additive, not multiplicative.
- **New consumers lose nothing**: alert on `status="fail",
cause="system"`; route `cause="user"` to the owning application team.
The alternative — emitting old and new `status` values side by side
during a transition — was rejected: it defers the break rather than
removing it (the old values must still be dropped eventually, forcing
the same migration later), and meanwhile double-counts every request in
unfiltered aggregations.
`cancel` is folded into `cause` for the same reason: before v2.6.19
client cancellations were counted as `fail` (cancel in the response
status) or `rejected` (cancel at the interceptor), so promoting it to a
`status` value quietly makes `status="fail"` under-count versus older
releases. As a cause it stays excludable via `cause!="cancel"` without
redefining `status`.
### Also in this PR
- The 7 `snapshot_impl` sites that emitted a bare `fail` with no
classification now report their real cause via `failMetricLabel(err)`
(e.g. `snapshot_metadata_uri is required` is `cause="user"`, not a
system fault). Their `status` is unchanged.
- `deployments/monitor/grafana/milvus-dashboard.json`: the "Faild
Request Rate" panel goes back to `status="fail"` — a verbatim revert of
what #50221 changed, which is the compatibility claim demonstrated.
- Dev docs and stale comments that named the old label values.
### Verification
- `TestParseMetricLabelStatusDomainIsStable` pins the status domain, so
re-splitting it fails CI rather than shipping.
- Adding a label makes every `WithLabelValues` site arity-sensitive **at
runtime, not compile time** — a missed site panics in production. Unit
tests do not reach all of them (coverage shows
`GetRestoreSnapshotState`, `ListRestoreSnapshotJobs`, `PinSnapshotData`,
`UnpinSnapshotData` at 0%), so all **58 emit sites were verified
statically via AST**, not only the ones tests happen to hit.
- Passing: `pkg/metrics`, `pkg/util/requestutil`, `pkg/util/merr`,
`internal/distributed/proxy` (incl. `httpserver`, which covers the REST
emit path), and the `internal/proxy` snapshot / metric-label tests.
`golangci-lint` clean on every touched package.
- Pre-existing and unrelated: `service_test.go` bind failures (`listen
tcp :19529: address already in use`) reproduce identically on unmodified
master — a local process holds the port.
### Rollout
Should be cherry-picked to 2.6 so the window in which the fine-grained
`status` values exist stays confined to v2.6.19–v2.6.20. Users who
already adopted the new values on those two releases need a one-time
query change (`status="fail_system"` -> `status="fail",
cause="system"`); that is a known, bounded set, and preferable to
leaving every pre-2.6.19 dashboard silently broken.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
pr: https://github.com/milvus-io/milvus/pull/49922
## Summary
Route generic LIKE Match operations to raw-data scan when a varchar
inverted index is present, while keeping PrefixMatch on the
inverted-index path. Update planner policy and sealed-segment coverage
for a complex LIKE pattern.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
issue: #51376
## What this fixes
The in-place partial-materialization path of
`bump_schema_version_compaction` commits onto a live segment's manifest,
and DataCoord adopted the result on a version-newer check only — it
never verified the result was built on the segment's *current* manifest.
A bump built on a plan-pinned base could therefore be adopted after a
concurrent stats/index task advanced the manifest pointer, silently
dropping that index (correct version number, wrong content lineage). The
loss is permanent and non-self-healing, because the segment meta stats
placeholder stops the stats inspector from re-triggering.
## Fix — application-layer adoption CAS (mirrors the stats path)
- The schema-bump result carries the plan-pinned base manifest
(`CompactionSegment.base_manifest`, like `StatsResult.base_manifest`);
the DataNode fills it with the manifest it materialized on.
- DataCoord adopts the in-place result only if `result.base_manifest ==
segment.ManifestPath` (the current pointer); a missing or stale base is
rejected (`ErrIllegalCompactionPlan`) so the task re-triggers and
rebuilds on the current manifest instead of overwriting the concurrent
commit. This mirrors the stats path's `errStatsResultStale`.
- The storage commit stays on `OverwriteResolver` (unchanged). Because
the CAS is at the adoption layer (against the meta pointer, not
storage-latest), a result lost after a DataNode crash self-heals on
retry: the pointer stays put, so the retry re-pins the same base and is
adopted — no wedge.
- The bump policy is restricted to internal collections (matching every
sibling compaction policy), so a rejected attempt's orphan is reclaimed
by internal dropped-segment GC and bump never churns on external
collections.
## Verification
Both interleavings converge with no data loss: stats-first → bump's base
no longer matches the advanced pointer → reject + retrigger on the
current manifest; bump-first → stats' existing base-check rejects and
retries. Added meta-level adoption tests (matching/stale base) and
updated the in-place adoption tests to carry the base.
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Related to #51419
- migrate required error and utility helpers into the client module
- preserve RPC error codes, retry metadata, and errors.Is semantics
- expose client-local RPC errors and common error sentinels
- remove github.com/milvus-io/milvus/pkg/v3 and server-only dependencies
- lower the client Go version to 1.24.9
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
## Problem
Milvus supports online collection-schema mutations — add/drop field,
add/drop function, alter field, enable dynamic field — through several
RootCoord DDL paths. Existing segments are written under the committed
schema, so a mutation that reinterprets an existing field (its data
type, nullability, or structural role) or relabels existing data as a
function output makes already-written binlogs read incorrectly. Before
this change there was no single authoritative check that a proposed
schema is a **safe structural successor** of the committed one;
validation was scattered and incomplete across DDL paths, so an unsafe
mutation could reach the WAL and silently corrupt or misinterpret
existing data.
## What this PR does
Adds an authoritative, validation-only, fail-closed gate
`ValidateSchemaEvolution(oldSchema, newSchema)`, invoked in every
RootCoord schema-mutating callback **before** any side effect (analyzer
reservation, bound-index allocation, WAL broadcast). It rejects:
- in-place field reinterpretation — name, data type, element type,
nullability, or structural role (primary / partition / clustering key,
autoID, dynamic);
- relabeling an existing field as a function output — function outputs
must use brand-new fields, never existing data;
- unsafe field additions — a newly added field may not arrive already
marked primary / partition / clustering key, function output, or
dynamic;
- unsafe drops of protected fields;
- more than one clustering key;
- a dynamic field that is also a function output;
- corruption of the reserved `max_field_id` watermark.
Property-only / TTL updates, analyzer rollback, BM25 / MinHash
`add_function_field`, and legacy function Drop / detach paths are
preserved unchanged.
## Scope
Non-function structural invariants only. Function-graph validation
(single producer, output binding, alter-time immutability, cascade) is
intentionally **out of scope here and owned by #51360**. This PR does
not include schema propagation, Proxy barriers, DML draining, backfill /
readiness, index promotion, TextEmbedding enablement, protobuf,
configuration, or C++ changes.
## Validation
- `go test -tags dynamic,test -gcflags='all=-N -l'
./internal/util/schemautil -count=1` — PASS (the gate's own unit tests)
- `git diff --check origin/master...HEAD` — PASS
issue: #51340
---------
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
issue: https://github.com/milvus-io/milvus/issues/51466
## What changed
`PhyUnaryRangeFilterExpr::DetermineExecPath()` previously used a
JSON-specific hard-coded denylist for string predicates:
- `Match`
- `PostfixMatch`
- `InnerMatch`
This bypassed the concrete scalar index's `ShouldUseOp()` policy. As a
result:
- indexes capable of evaluating these predicates, such as `STL_SORT`,
were
forced to scan and parse raw JSON;
- JSON string predicates could use a different execution policy from
ordinary
VARCHAR predicates;
- `RegexMatch` remained on the index path even for indexes whose planner
policy
explicitly preferred raw-data execution.
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
- Rebuild TiKV write transactions on retriable begin, build-stage IO,
and classified-transient commit failures while aborting deterministic
build failures and unsafe commit outcomes.
- Preserve original errors for TiKV undetermined commit results and
caller cancellation before string-wrapping; keep private
`retry.Unrecoverable` markers inside the retry helper so they never leak
to callers.
- Stop `ReliableWriteMetaKv` from replaying undetermined TiKV write
results, correct the TiKV begin/requestTimeout comment, and add
regression tests for commit aborts, retry exhaustion, deadline stopping,
and outer reliable-write behavior.
related: #51425
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## Summary
Keep raw search distances through segment search, iterator output, index
query, and chunk merging so ranking/reduce order is not affected by
`round_decimal`.
Apply `round_decimal` only when producing the final search response,
after reduce/rerank/order-by has already determined result order.
Returned scores keep the requested precision without changing TopK
selection.
issue: #50347
---------
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## What changed
- Replace Pod deletion with Chaos Mesh `container-kill` in CDC
force-promote and source-down failover tests.
- Kill every regular container in matching Pods while preserving the Pod
objects.
- Verify that Pod UIDs remain unchanged and each container's restart
count increases.
- Clean up the generated `PodChaos` resources after fault injection.
## Why
Deleting Pods changes their identity and tests Pod recreation rather
than container-level recovery. These scenarios need to verify CDC
failover behavior when containers restart inside the existing Pods.
## Impact
CDC failover tests now exercise container failures while explicitly
validating that Kubernetes does not recreate the affected Pods.
## Validation
- `ruff check` passed for all modified files.
- `ruff format --check` passed for all modified files.
- Python compilation passed for all modified files.
- Full CDC E2E was not run locally because it requires a Kubernetes
environment with Chaos Mesh.
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
issue: #45881
## What this PR does
External refresh previously read the segment row target directly from
each DataNode configuration during execution. That made the worker input
implicit and allowed task behavior to depend on the selected node.
This change:
- keeps `dataNode.externalCollection.targetRowsPerSegment` as the
existing configuration source;
- has DataCoord snapshot the configured value into the refresh task
request;
- adds `target_rows_per_segment` to the DataCoord-to-DataNode request;
- validates the transmitted value in DataNode;
- uses the request value for fragment splitting and segment balancing;
- preserves the MilvusTable one-fragment-per-segment behavior.
## Validation
- `make generated-proto-without-cpp`
- `make lint-fix`
- `make build-go`
- `go test -gcflags="all=-N -l" -race -tags dynamic,test
github.com/milvus-io/milvus/pkg/v3/util/paramtable -failfast -count=1
-ldflags="-r ${RPATH}" -run "^TestCachedParam$" -v`
- `go test -gcflags="all=-N -l" -race
-coverprofile=/tmp/refine-rowcount-config-dc.out -tags dynamic,test
github.com/milvus-io/milvus/internal/datacoord -failfast -count=1
-ldflags="-r ${RPATH}" -run
"^TestRefreshExternalCollectionTask_CreateTaskOnWorker$" -v`
- `go test -gcflags="all=-N -l" -race
-coverprofile=/tmp/refine-rowcount-config-dn.out -tags dynamic,test
github.com/milvus-io/milvus/internal/datanode/external -failfast
-count=1 -ldflags="-r ${RPATH}" -run
"^TestRefreshExternalCollectionTaskSuite$" -v`
- `git diff --check`
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: #51250
Keep TEXT segments on StorageV3 while making growing-source flush an
explicit allowed mode instead of a mandatory TEXT path.
Propagate create-segment storage versions through WAL flusher and write
buffer, reject storage-version mismatches in DataCoord, and keep raw
segcore chunks sticky for segments that may be flushed from growing
source.
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
relate: #50304
## Summary
Add the local format design doc, use the storage writer format constant,
and introduce local_format metadata propagation with column-group split
policy support.
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
issue: #51436
### What this PR does
- Propagates a CreateCollection RecoveryStorage observation failure from
the flusher component to WAL dispatch.
- Stops before creating a DataSyncService when the recovery state was
not recorded.
- Adds coverage for the canceled observation path, including the absence
of schema lookup.
### Verification
- `go test -v -tags dynamic,test -gcflags="all=-N -l" -count=1
./internal/streamingnode/server/flusher/flusherimpl -run
"TestWALFlusher.*CreateCollection"`
- `make static-check`
- `gofumpt -l` on changed Go files
- `git diff --check`
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
issue: #50742
## What this PR does
Optimizes protobuf decoding on Milvus RPC read/write hot paths with a
hand-written decoder in `pkg/util/fastpb`. The final change contains no
gRPC connection-pool or connection-count changes.
## Fast decoder coverage
The gRPC `releaseCodec.Unmarshal` path dispatches these top-level
messages to fastpb:
- `internalpb.RetrieveResults`
- `milvuspb.InsertRequest`
- `milvuspb.UpsertRequest`
`schemapb.SearchResultData` is not a top-level codec-dispatched message.
It is decoded directly from `SlicedBlob` at the search/reduce hot paths
in:
- `internal/proxy/search_reduce_util.go`
- `internal/querynodev2/segments/result.go`
- `internal/querynodev2/local_worker.go`
- `internal/querynodev2/tasks/search_task_go_reduce.go`
Nested hot messages such as `FieldData`, scalar/vector arrays, and IDs
are decoded by the same package.
Main optimizations:
- varchar string arrays use one backing byte arena, reducing per-string
allocations
- packed float vectors use a single memcpy into self-owned memory
- cold, complex, unknown, and mismatched fields delegate or merge
through the official protobuf codec
- no new external dependency or source-buffer aliasing
## Compatibility boundary
The compatibility contract depends on the data path:
- **Untrusted client ingress (`InsertRequest` and `UpsertRequest`)**
validates proto3 strings as UTF-8 and is tested differentially against
`proto.Unmarshal`, including malformed input behavior.
- **Trusted internal result paths (`SearchResultData`,
`RetrieveResults`, and direct `FieldData` decoding)** assume canonical
Milvus/segcore protobuf output and intentionally skip UTF-8 validation
for performance. Therefore these paths do not claim equivalence for
arbitrary adversarial wire containing invalid UTF-8.
Within that boundary, the decoder preserves the protobuf behaviors
exercised by the differential tests:
- proto2 groups fall back to the official codec
- known fields with mismatched wire types fall back to the official
codec
- unknown and future fields are preserved through an official merge pass
- repeated singular messages merge from the raw wire, including
explicitly encoded proto3 default values
- repeated message-valued oneof members merge correctly, while different
variants remain last-wins
- mixed packed and unpacked scalar encodings preserve wire order
- descriptor field-set tripwires force decoder review when the protobuf
schema changes
The unsafe packed-float copy assumes the currently supported
little-endian Milvus targets such as x86-64 and arm64.
## Benchmarks
fastpb vs official `proto.Unmarshal`, using 1000-row payloads:
| path | payload | speedup | allocations/op |
|---|---|---:|---:|
| search / query | varchar | about 2x | about 1000-2035 to about 10-17 |
| search / query | float vector, dim 768 | about 6x | unchanged |
| insert | varchar with UTF-8 validation | about 1.8x | 1019 to about 10
|
## Verification
- unit and differential tests cover field descriptors, UTF-8 contracts,
wire-type mismatches, proto2 groups, unknown fields, packed/unpacked
ordering, oneof behavior, and repeated-message merge semantics
- randomized canonical-message equivalence tests cover search, retrieve,
insert, and upsert payloads
- `go test ./util/fastpb ./util/resource` passes
- `go vet ./util/fastpb ./util/resource` passes
- current full CI and end-to-end status is tracked by the PR checks
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
issue: #51446
- message properties: add an explicit `Unreplicable` marker for concrete
WAL messages and expose builder opt-in through `WithUnreplicable`
- replication: skip unreplicable messages in the CDC sender and
secondary replicate interceptor before callback replay
- DDL producers: mark unsupported snapshot, manifest backfill, and
external collection refresh broadcasts as unreplicable
- streaming docs: document the message-level skip marker and current
unsupported DDL replication behavior
Signed-off-by: chyezh <chyezh@outlook.com>
Related: #50236
- Reuse the initially loaded grantee keys/values when ListPolicy checks
legacy grantee ID ownership.
- Avoid reloading the full grantee-privileges tree once per legacy
grant.
- Add a regression test that verifies ListPolicy performs only one full
grantee-prefix load for legacy grants.
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
issue: #51248
## What changed
- correct the omitted-`target_size` case to assert ordinary manual
compaction semantics
- add a physical Force Merge target-size case using persistent segment
IDs and MinIO insert-log sizes
- add deterministic Go boundary coverage for grouping selection at the
configured threshold and threshold + 1
- add target-size memory clamp coverage for cluster and standalone
co-location modes
- extend Optimize format, signed-int64 boundary, loaded-segment refresh,
and live async lifecycle coverage
- add the async Optimize wrapper and update the PyMilvus test dependency
to 3.1.0rc64
## Why
The previous cases did not prove that an explicit Force Merge target
affected physical output, assumed the wrong grouping threshold, and
depended on manual log inspection for algorithm selection. Loaded
refresh and the public async Optimize workflow also lacked live
end-to-end coverage.
The async case exposed milvus-io/pymilvus#3680 on PyMilvus 3.1.0rc62.
Version 3.1.0rc64 contains the tuple-unpacking fix and passes the
unmodified test.
## Validation
- all 10 planned case IDs pass; 0 failed and 0 blocked
- `milvus-dev-cli` Go UT job
`go-ut-local-zhuwenxi-zhuwenxing-i-4082040-30413568` succeeded; both
test functions and all five subtests passed
- physical target-size L3 case passed against a real cluster and MinIO
in 200.68s
- loaded refresh, target format/boundary, and manual compaction cases
passed against the same server version
- async Optimize L3 passed unmodified with PyMilvus 3.1.0rc64 in 56.91s
- 56 Python nodes collect successfully; Python compilation, Ruff, and
`git diff --check` pass
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
Related to #51068
Store PK index slots and virtual PK offset maps in RuntimeResourceState.
Build Storage V1 and V2 PK indexes through the unified translator and
stage PK replacements until the final state publication.
Make PK lookup, range search, bulk subscript, and delete filtering use a
single published snapshot. Remove the sealed InsertRecord fallback and
clear stale PK resources when external segments become empty.
Add regression coverage for atomic PK replacement and zero-row external
segment cleanup.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
## What this fixes
When sort compaction reads a segment whose binlog references an object
that is missing in object storage, the DataNode panics with a
nil-pointer dereference and enters `CrashLoopBackOff`, instead of
failing the compaction task cleanly.
## Root cause
`IterativeRecordReader.Next()` assigned `ir.cur` **before** checking the
error returned by `iterate()`. When opening the next binlog chunk fails,
`newPackedRecordReader` returns a nil `*packedRecordReader` boxed into a
non-nil `RecordReader` interface (a typed-nil). The deferred
`rr.Close()` in `sortSegment` then called `Close()` on a nil receiver
and panicked — the existing `if pr.reader != nil` / `if ir.cur != nil`
guards don't help because the receiver itself is nil.
## Fix
- Only publish the reader once `iterate()` succeeds (assign to a temp,
check the error, then set `ir.cur`).
- Make `packedRecordReader.Close` / `ffiPackedRecordReader.Close`
nil-receiver safe as defense-in-depth.
- The missing-object error is still surfaced from `Next()` (it is
**not** converted to `io.EOF`), so the sort task fails cleanly and is
retried rather than silently dropping data.
## Test
Added `TestIterativeRecordReader_MissingChunkDoesNotPanic`, which
reproduces the missing-object chunk: it asserts the error is surfaced
from `Next()` and that the deferred `Close()` does not panic.
issue: #50927🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
The documented CDC lag PromQL subtracted two independently scraped
time-tick gauges, which caused idle oscillation and misleading restart
behavior. This PR fixes the query and hardens the lag metric lifecycle
without introducing a new metric.
### #50633 — idle oscillation
Subtract each gauge timestamp before calculating the difference,
canceling the cross-target scrape-phase offset:
```promql
clamp_min(
max by (channel_name) (milvus_wal_last_confirmed_time_tick - timestamp(milvus_wal_last_confirmed_time_tick))
- min by (channel_name) (milvus_cdc_last_replicated_time_tick - timestamp(milvus_cdc_last_replicated_time_tick)),
0)
```
### #51048 — restart and switchover lag lifecycle
- Seed `milvus_cdc_last_replicated_time_tick` from
`InitializedCheckpoint` before target-client initialization, so
target-unavailable startup exposes a conservative lag instead of an
absent series.
- Overwrite the seed with the target-confirmed checkpoint after
`GetReplicateInfo` succeeds.
- Ignore nil and zero checkpoints to avoid creating an epoch-valued
series.
- Remove lag-series deletion from generic stream `OnClose`.
- Delete the lag series only from `ReplicateManager.RemoveReplicator`,
the genuine etcd DELETE path. Outdated revision cleanup therefore cannot
delete the live revisions shared label pair.
issue: #50633
issue: #51048
### Test Plan
- [x] Added unit-test coverage for initialized-checkpoint seeding and
nil/zero guards.
- [x] Added regression coverage for genuine removal versus outdated
revision cleanup.
- [ ] Local Go tests and static check.
- [ ] Confirm the lag lifecycle on a live switchover environment.
---------
Signed-off-by: Yihao Dai <yihao.dai@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
issue: #51100
## Summary
The C++ persistent-storage `get` failure counter was registered without
the `status="fail"` label, unlike the other operation failure counters.
The counter was still incremented only on failed `get` operations, but
Prometheus exported it as a no-status `get` series, so status-based
dashboards and queries could not identify it as a failure series.
This PR adds `status="fail"` to `getFailMap`, making
`internal_storage_op_count{persistent_data_op_type="get",
status="fail"}` consistent with `put`, `stat`, `list`, and `remove`
failures.
## Note
This changes the Prometheus time series shape: the old no-status `get`
failure series will stop, and the new `status="fail"` series will start
from zero.
Signed-off-by: xaxys <tpnnghd@163.com>
## What
- Upgrade the root, pkg, client, test, and telemetry example modules
from Go 1.26.4 to Go 1.26.5.
- Upgrade CPU/GPU builder Dockerfiles, macOS CI Go setup, KRTE build
arg, rpm setup, meta-migration builder, and go-client test image to Go
1.26.5.
- Point CPU and GPU builder consumption in `.env` to the successful
build-env tag `20260714-c135601`.
## Why
- Latest direct Trivy scan of
`milvusdb/milvus:master-20260711-1993feae-amd64` reports CVE-2026-39822
in Go stdlib 1.26.4, fixed in 1.26.5.
- The normal daily image scan is currently blocked through Jenkins
`milvus_scan_image_daily` #540 with `exec format error`; last successful
scan is #508 from 2026-06-10, so this was verified by direct image scan.
## Image coverage
- Go toolchain fixes are shared by all Milvus runtime images; no per-OS
runtime Dockerfile package change is required.
- Builder definitions are updated for CPU ubuntu22.04, ubuntu20.04,
ubuntu24.04, amazonlinux2023, rockylinux9 and GPU ubuntu22.04,
ubuntu20.04.
- No runtime image variant is deliberately skipped; this is not an
OS-package CVE.
## Verification
- `go.dev/dl/?mode=json&include=all` confirms go1.26.5 is stable and
linux amd64/arm64 tarballs exist.
- `go env GOTOOLCHAIN GOVERSION` selects `auto` / `go1.26.5` after the
module update.
- `go list -m` succeeds for root, pkg, client, tests/go_client, and both
telemetry examples.
- `cd pkg && go test -tags dynamic,test -gcflags="all=-N -l" -count=1
./util/typeutil` passes after rebasing onto the latest upstream master.
- Jenkins build-env #40 succeeded and published builder tag
`20260714-c135601`; `.env` now selects that tag for CPU and GPU CI
builds.
- A repo-wide audit found no remaining Go 1.26.4 toolchain pins.
Note: full root `go mod tidy` is not included because this checkout has
a root-owned `deployments/docker/dev/volumes/etcd/member` directory that
makes `go mod tidy` fail with permission errors; `go mod tidy -e`
subsequently OOMed locally. No go.sum changes were produced by the
successful module tidies.
Generated by autonomous CVE maintenance.
---------
Signed-off-by: Li Liu <li.liu@zilliz.com>
issue: #51114
JsonStats shredding now loads through
`ManifestGroupTranslator`/`ChunkReader`, but JsonStats files do not have
the normal sealed-segment manifest metadata. The load path reconstructs
the needed column-group view from the existing parquet footer/schema:
column names, Arrow field ids, row counts, and file ordering. This keeps
the current `meta.json` contract unchanged; parquet key-value metadata
is only a fallback for old files that do not have the separate meta
file.
The main follow-up after switching to the manifest translator was
projection behavior. Lazy loading now builds one projected reader per
shredding column, so a predicate on one JSON path does not pull the
whole group; when warmup is enabled it still uses the full column group
so warmup can preload everything together. The added tests cover parquet
files without packed field-list metadata, multi-file ordering, lazy
single-column projection, and warmup full-group projection.
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
issue: #51372
### Problem
`MergeOp.buildResultArrays` builds a zero-hit chunk's `$id` array with a
hardcoded Int64 builder
(`operator_merge.go:713-721` before this change), while chunks that do
have hits are typed from the
first actual ID (`buildInt64Results` / `buildStringResults`). All
per-query chunks then go into
`AddColumnFromChunks`, which builds one `arrow.NewChunked` column out of
them.
For a **VarChar-PK collection** where **one query of an nq >= 2 batch
returns zero hits** while
another returns hits, those chunks mix `utf8` and `int64` and arrow
panics:
```
panic: invalid: arrow/array: mismatch data type int64 vs utf8
```
`MergeOp` sits at chain index 0, so every request that goes through a
function rerank chain — plain
search with a rerank function as well as hybrid search — can hit it. The
proxy has no recovery
interceptor on its gRPC chains
(`internal/distributed/proxy/service.go:295-307`, `416-424`) and no
`recover()` in `internal/proxy/`, so the panic takes the proxy process
down.
The input side was never affected: `collectRRFScores` /
`collectWeightedScores` read IDs row by row,
so an empty chunk is simply iterated zero times. Only the output side
was broken.
### Fix
Resolve the `$id` arrow type once per merge, from the first input that
actually carries IDs
(`resolveIDType`), and build the zero-hit chunk with that type.
Zero-row inputs are deliberately skipped when resolving: an empty result
carries no ID type of its
own and is materialized as an empty Int64 column regardless of the
collection's PK type
(`importEmptyIDs`, converter.go). When *every* input is empty, Int64 is
kept — every chunk is then
empty as well, so the column is self-consistent and the rerank
operator's `allEmpty` early-return
short-circuits before it anyway.
An unexpected arrow ID type now returns a `merr` error instead of
silently producing an Int64 array.
### Testing
- `TestMergeWithEmptyChunkVarCharIDs` — single input, VarChar IDs,
`topks=[2,0]`; asserts the merged
`$id` column stays `utf8` with an empty second chunk. Written first and
observed panicking with
`mismatch data type int64 vs utf8` on unmodified master.
- `TestMergeMultiInputEmptyLegVarCharIDs` — hybrid-search shape: one leg
entirely empty, the other
with a zero-hit query.
- The existing `TestMergeWithEmptyChunk` (Int64) still passes, pinning
the unchanged Int64 path.
- `./internal/util/function/...` green, including `-race`; `gofmt`
clean.
### Note
Found while reviewing #51156 (fix for #50969). This panic is **not**
introduced by that PR —
`operator_merge.go` is byte-identical between master and its head, and
the repro above does not touch
the lines it changes. #51156 does widen the reachable surface, though:
before it, a single-shard
zero-hit result failed earlier in the converter with `unsupported ID
type` and never reached
`MergeOp`; after it, that input builds a DataFrame and walks into this
panic.
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## What
Drop `t.Parallel()` from the heavy go_client cases in `snapshot_test.go`
(10 cases) and `external_table_refresh_test.go` (12 cases), and mark
each with a comment explaining why it must stay serial.
## Why
`t.Parallel()` was blanket-added to every go_client testcase in #49138.
That is fine for the light cases, but the snapshot/restore and
external-table refresh cases are heavy: they insert tens of thousands of
vectors and then wait minutes for index build / snapshot / restore.
Running them alongside the large parallel test swarm starves the shared
standalone cluster, and they flake on their own index-build / restore
timeouts, e.g.:
- `TestSnapshotRestoreWithMultiSegment` — timeout waiting for indexes to
be built
- `TestSnapshotRestoreWithMultiFields` — timeout waiting for restore to
complete
Go runs the non-parallel tests first (sequentially, each with the
cluster to itself), then the parallel suite. Keeping these heavy cases
serial gives them the resources they need while leaving the light
parallel tests untouched. The trade is a little extra wall-clock for
stability.
## Changes
- `tests/go_client/testcases/snapshot_test.go` — remove `t.Parallel()`
from 10 heavy cases; add marker comment.
- `tests/go_client/testcases/external_table_refresh_test.go` — remove
`t.Parallel()` from 12 heavy cases; add marker comment.
No test logic changed; `gofmt` clean, `go vet ./testcases/` passes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What this PR does
Adds an integration test (`TestWpRetentionTruncateRead`) for the
Woodpecker (wp)
WAL implementation pinning the read semantics around truncation that
Milvus
relies on but had no explicit coverage for.
Milvus truncates the wp WAL aggressively right after a flush, but
truncation
only moves a metadata watermark — it does **not** delete data. Data
at/under the
watermark stays readable until the retention-TTL GC physically removes
it. The
test pins this with three sub-cases:
1. **TruncatedDataStillReadable** — after truncating to the latest id,
an
explicit `StartFrom` at any position in `(earliest, latest]` (spanning
segment boundaries) still returns every message, including the watermark
id
itself. As a contrast, a fresh `DeliverPolicy_All` reader is parked past
the
watermark and returns nothing — proving the watermark really moved while
the
data underneath is intact.
2. **SeekForwardWithinSegment** / **SeekForwardCrossSegment** — a reader
asking
for data older than the earliest deliverable position
(`DeliverPolicy_All`
after truncation) is moved forward to the first available position
instead of
erroring or stalling. The two sub-cases exercise both branches of
`adjustPendingReadPointIfTruncated`: a truncation point inside segment 0
(within-segment) vs. in a later segment (cross-segment id jump).
## Notes
- **Deterministic by design**: large retention (`72h`) so GC never
fires, a
small `segmentRollingPolicy.maxSize` so writes span several segments,
single-threaded writes so ids are monotonic, and all assertions are
message-id based. Run logs confirm each sub-case hits its intended
adjustment
branch (`truncatedSegmentId` 0 vs >0).
- **Scope**: covers the open-time read-position adjustment after
truncation. It
does NOT exercise the physical retention-TTL GC + runtime skip-forward
path —
that needs waiting for the async GC sweep and is intentionally left out
to
keep the test deterministic.
- Runs against local fs storage; the truncate/read logic under test
lives in the
woodpecker log handle/reader and is storage-backend agnostic.
issue: #43638
---------
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
issue: #51253
### What this PR does
When `proxy.enableCachedServiceProvider` is on and `DescribeCollection`
is called with only a collection id (e.g. the HTTP management API on
port 9091), the cached provider echoed the request db name — empty or
default — instead of the database the collection actually belongs to,
and never filled `db_id`. The remote path does not have this problem
because the coordinator resolves the database by DBID and returns it in
the top-level `DbName`/`DbId` fields.
The proxy meta cache already stores the actual db name of each
collection; this PR also stores the db id, and prefers the cached
`dbName`/`dbID` when assembling the cached DescribeCollection response.
### Verification
- New unit test
`TestCachedProxyServiceProvider_DescribeCollection_ByIDReturnsActualDbName`
(request carries only `collectionID`, asserts real `db_name`/`db_id` are
returned); existing tests in the file still pass.
- Verified end-to-end on a standalone build of this branch: created
`db1`/`coll1`, queried `GET :9091/api/v1/collection` with only
`collectionID` — response now returns `"db_name": "db1"` and the `db_id`
matching `databases/describe`, where it previously returned neither.
- The same logic was also verified on a 2.5-based deployment (real
cluster) before porting to master.
Happy to backport to 2.5/2.6 if needed.
Signed-off-by: fudongying <fudongying@bytedance.com>
issue: #45881
## What changed
- Mark read-only external collection delegators as not catching up DML
streaming data during cold load.
- Skip WAL tSafe waits for external Search, Query, QueryStream, and
GetStatistics reads.
- Always use `MaxTimestamp` as the MVCC timestamp for external reads,
including partial searches and requests carrying an explicit finite
`MvccTimestamp`, so loaded deltalog deletes remain visible.
- Prevent external reads from updating or exposing a required WAL MVCC
timetick.
- Preserve the existing streaming catch-up and MVCC behavior for normal
collections.
## Why
External collections are read-only and query-visible data comes from the
loaded external snapshot, not DML WAL replay. Starting from an old
channel
checkpoint can otherwise keep QueryNode unavailable while it catches up
WAL
history that cannot change the external snapshot.
Using the full MVCC range also prevents low guarantee placeholders or
explicit finite MVCC timestamps from hiding rows or deltalog deletes
already
loaded as part of that snapshot.
This PR does not change external channel checkpoints or broadcast
refresh
messages to collection vchannels.
## Validation
- `make clean && make milvus`
- `git diff --check`
- Targeted QueryNode delegator tests:
```bash
go test -tags dynamic,test -gcflags="all=-N -l" \
github.com/milvus-io/milvus/internal/querynodev2/delegator \
-run 'Test(ExternalCollectionDelegatorDoesNotCatchUpStreamingData|'\
'ExternalCollectionWaitTSafeUsesFullSnapshotTimestamp|'\
'ExternalCollectionPartialSearchUsesFullSnapshotTimestamp|'\
'ExternalCollectionQueryUsesFullSnapshotTimestamp|'\
'ExternalCollectionQueryStreamUsesFullSnapshotTimestamp|'\
'ExternalCollectionStrongConsistencyDoesNotRequestWALMVCC|'\
'NormalCollectionDelegatorCatchesUpStreamingData)$' -count=1
```
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #51068
- manage JSON index load, replace, and drop through reopen COW snapshots
- remove LoadScope_Index and the legacy QueryCoord-to-segcore update
path
- drop JSON indexes by field and nested path to preserve sibling indexes
- regenerate query proto and QueryNode mocks
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #51333
## Summary
- serialize DataCoord checkpoint mutations per channel so unrelated
channel catalog writes can proceed concurrently
- keep overlapping single, batch, mark, and drop operations serialized
with deterministic multi-channel lock ordering
- use a read lock for the read-only RootCoord `DescribeAlias` path
- exclude all newly added metrics and profiling instrumentation
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
issue: https://github.com/milvus-io/milvus/issues/42053
This PR fixes ngram index eligibility checks for UTF-8 literals by using
character count instead of byte length in the C++ gate and phase1
validation. It also adds regression coverage for short multibyte
literals to ensure they correctly fall back instead of being sent to
Tantivy ngram queries.
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
issue: #36214
issue: #51324
## Summary
gcp-native-storage transitively depends on
`boost/align/aligned_allocator.hpp` and
`google/cloud/storage/retry_policy.h` headers but did not declare
milvus_conan_deps. As a result, building with ENABLE_GCP_NATIVE=ON
failed because the required Conan include directories were not
propagated to the object target during compilation.
## Changes
Link milvus_conan_deps directly to gcp-native-storage in
`internal/core/src/storage/gcp-native-storage/CMakeLists.txt` so its
usage requirements are available when compiling the target.
## Verification
Verified locally.
### Reproduce
```bash
ENABLE_GCP_NATIVE=ON make build-cpp
```
Before this change, compilation failed with:
```text
fatal error: boost/align/aligned_allocator.hpp: No such file or directory
fatal error: google/cloud/storage/retry_policy.h: No such file or directory
```
After this change:
- `ENABLE_GCP_NATIVE=ON make build-cpp` succeeds.
- `make verifiers` passes, including:
- build-cpp
- getdeps
- cppcheck
- rustcheck
- fmt
- static-check
Signed-off-by: Jeremiah Xing <dev.jrmh@gmail.com>
## Summary
issue: #51052
This PR reduces QueryNode memory pressure by avoiding duplicated or
long-lived segment metadata after sealed segment load/reopen and by
caching frequently reused schema conversion state.
## Storage Mode Scope
- For manifest-backed Storage V3 segments, both Go-side runtime load
info and segcore-side runtime load info are compacted after load/reopen.
- For Storage V2 binlog-mode segments, Go-side sealed runtime load info
is compacted, while segcore keeps the full load info because binlog diff
still depends on binlog/index metadata.
- The optimization primarily targets Storage V3, which is the forward
path.
## Commit Breakdown
1. `9d4dbdf065 enhance: cache text LOB loon arrow schema`
- Cache the Loon Arrow schema generated for the TEXT LOB binary
projection used by Storage V3 load paths.
- Invalidate the cache when schema fields change and clear mutable cache
state on schema assignment.
2. `4f0bb58ef6 enhance: avoid duplicate segment index load info`
- Remove duplicated converted index load info storage.
- Reuse the existing field-index cache for current index identity and
diff/drop decisions.
3. `89e8a15b42 enhance: compact runtime segment load info`
- Compact Go QueryNode sealed segment load info after load/reopen.
- Keep runtime-required lightweight fields and cache related data size
before dropping large metadata.
4. `8a38b2fdbf enhance: skip real count scan without deletes`
- Return row count directly when a segment has no deleted rows.
- Avoid scanning the deleted bitmap on the real-count fast path.
5. `d041837d33 enhance: compact segcore runtime load info`
- Compact manifest-backed segcore runtime load info after load/reopen.
- Preserve lightweight diff identity and local runtime state needed by
schema-only reopen and later diffs.
## Test Plan
- [x] `git diff --check upstream/master...HEAD`
- [x] `gofumpt -l internal/querynodev2/segments/segment.go
internal/querynodev2/segments/segment_loader.go
internal/querynodev2/segments/segment_test.go
internal/querynodev2/segments/utils.go
internal/querynodev2/segments/utils_test.go`
- [x] `/opt/homebrew/opt/llvm@18/bin/clang-format --dry-run --Werror
--style=file <changed C++ files>`
- [x] `ninja -C cmake_build all_tests`
- [x] `./cmake_build/unittest/all_tests
--gtest_filter='SchemaTest.ConvertToLoonArrowSchemaCachesTextLobBinarySchema:SegmentLoadInfoTest.CompactRuntimeInfoForManifest:SegmentLoadInfoTest.CompactRuntimeInfoForManifestKeepsIndexDiffIdentity:SegmentLoadInfoTest.CompactRuntimeInfoForManifestSchemaCopyKeepsIndexIdentity:SegmentLoadInfoTest.ReplaceSchemaForReopenPrunesDroppedFieldRuntimeState:SealedSegmentLoad.LoadPublishesDefaultFilledStateAfterCompact:SealedSegmentReopen.SchemaOnlyReopenPublishesDefaultFilledState:SealedSegmentReopen.SchemaAwareReopenDiscardsOlderSchema:SealedSegmentReopen.SchemaOnlyReopenPreservesCreatedTextIndexState:SealedSegmentReopen.TextIndexCreatedWipedByReopen'`
- [x] `go test -v -count=1 -tags dynamic,test -gcflags="all=-N -l"
-ldflags="-extldflags \"-Wl,-rpath,${MILVUS_WORK_DIR}/cmake_build/lib
-Wl,-rpath,${MILVUS_WORK_DIR}/internal/core/output/lib
-Wl,-rpath,${MILVUS_WORK_DIR}/cmake_build/thirdparty/boost_ext\""
./internal/querynodev2/segments -run
"Test(CompactSegmentLoadInfoForRuntime|GetSegmentRelatedDataSize)$"
-timeout 300s`
- [x] `go test -v -count=1 -tags dynamic,test -gcflags="all=-N -l"
-ldflags="-extldflags \"-Wl,-rpath,${MILVUS_WORK_DIR}/cmake_build/lib
-Wl,-rpath,${MILVUS_WORK_DIR}/internal/core/output/lib
-Wl,-rpath,${MILVUS_WORK_DIR}/cmake_build/thirdparty/boost_ext\""
./internal/querynodev2/segments -timeout 300s` timed out locally after
303.862s while still running MinIO-backed retrieve tests
(`TestRetrieveWithFilter`).
---------
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
issue: #51275
### Problem
Under StorageV3 growing-source flush, a manifest legally keeps column
groups whose fields were later dropped: drop does not rewrite flushed
data, and compaction removes the leftover lazily. Loaders however
projected such groups against the **current** schema and treated the
mismatch as an error:
- **Growing recovery** (`SegmentGrowingImpl::LoadColumnsGroups`)
submitted every manifest group with a full-schema projection; a group
whose fields were all dropped has an empty projection, which
milvus-storage rejects by contract (`ChunkReaderImpl::open`: "No needed
columns found in column group"). The channel-recovery level retried the
deterministic failure every second — observed locally: **4140 identical
failures over ~3 minutes**, the channel unavailable (`no available shard
leaders`) the whole time, until a compaction rewrite happened to remove
the group.
- **Sealed load** (`ChunkedSegmentSealedImpl::LoadColumnGroups`)
resolved every manifest column to a field id without a membership check
and tripped the schema-lookup assert on any group still carrying a
dropped column — mixed groups included.
### Fix
Treat a dropped-field column as a legal leftover at the loaders:
- Growing: skip a column group before submitting its load task when none
of its columns exist in the segment schema (logged).
- Sealed: filter resolved field ids absent from the schema snapshot
while building `cg_field_ids` (logged); a group filtered to empty
produces no load task. Mixed groups keep loading their live columns via
projection, unchanged.
### Verification
- Local A/B with a schema-evolution chaos workload (concurrent add/drop
ordinary + BM25 function field, DML/query/search, `useLoonFFI` +
growing-source flush enabled, StreamingNode SIGKILL/restart injections):
before the fix, recovery wedged in the retry loop (4140 errors / ~3 min
/ final query 503); after the fix the same scenario logs 2 group skips,
the channel recovers in seconds, and the full serial consistency
validation passes.
- New UT `LoadGrowingSegmentSkipsDroppedFieldColumnGroup`: flushes a
manifest with a field-exclusive column group (writer pattern
`"0|1|100,101"`), reloads the segment under a schema without that field,
and asserts the load succeeds with the group skipped (plus a guard
assertion that the manifest really contains the exclusive group).
- The sealed-path filter is exercised by code inspection and mirrors the
growing-side semantics; on current master the external-table load path
derives its projection from schema-owned column names and is not exposed
to this class.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## Summary
- Upgrade Knowhere dependency to 49507ef3 to pick up the fix that
removes the num_build_thread upper-bound validation.
issue: #42937
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## What this PR does
- Remove the `xfail` marker from the existing drop-and-readd
analyzer-field regression for #50484.
- Add L1 E2E coverage for the live QueryNode growing-segment reopen path
when an `enable_match` VARCHAR field with `default_value` is added to
loaded growing data.
- Verify the default value is immediately searchable on pre-existing
unflushed rows, eventually searchable on a subsequent defaulted insert
after the growing text index catches up, and remains searchable after
flush/reload.
Related fix: #51201
issue: #50484
## Scope note
This PR validates the live reopen path:
```text
loaded collection
-> insert rows that stay in a QueryNode growing segment
-> add an `enable_match` VARCHAR field with `default_value`
-> query triggers LazyCheckSchema/Reopen
-> default values for pre-existing growing rows are indexed and searchable immediately
```
It does not cover the QueryNode recovery / LoadGrowing path. That path
requires forcing a QueryNode restart, replacement, or channel rewatch
while the data is still growing: the new QueryNode can create the
segment with the latest schema and then load old binlogs written before
AddField. Since the segment schema is already current in that path,
LazyCheckSchema/Reopen is not the mechanism being tested here. Recovery
coverage should be handled by a focused stability test.
## Test results
-
`test_milvus_client_add_match_field_with_default_value_on_growing_data`:
3/3 passed
- `test_drop_then_add_same_name_analyzer_field`: passed
- Ruff lint and format checks: passed
- `git diff --check`: passed
Signed-off-by: lyyyuna <yiyang.li@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
issue: #49706
issue: #50663
## What
This PR fixes StorageV3 compaction fallback loading when delete records
are stored in compact-to child manifests instead of legacy deltalogs.
Changes:
- Add `child_manifest_paths` to DataCoord `SegmentInfo` and QueryCoord
`SegmentLoadInfo` as a load-time delete overlay.
- Make DataCoord recursively collect compact-to descendant delete
sources for cloned unhealthy segment responses.
- Keep legacy child delete logs in `Deltalogs`, and keep StorageV3 child
delete logs as `ChildManifestPaths`.
- Pass `ChildManifestPaths` through QueryCoord to QueryNode.
- Make QueryNode load parent delete data and child manifest delete data
into one `DeltaData` before calling `LoadDeltaData`.
## Why
Fallback loading keeps the parent segment identity, but StorageV3
compact-to children may keep newer delete records only in their
manifests. The old fallback path only appended legacy child deltalogs,
so it could miss V3 child deletes and expose rows that should be
filtered.
The V2 path is kept for compatibility. Mixed chains such as parent V2 to
child V3, and parent V3 to child V2, are handled by keeping each child
delete source in its native representation.
## Validation
- `gofmt -w internal/datacoord/services.go
internal/querycoordv2/utils/types.go
internal/querynodev2/segments/segment_loader.go`
- `git diff --check`
- `source ~/.profile && source scripts/setenv.sh && make
generated-proto-without-cpp`
- `git diff --exit-code -- pkg/proto/data_coord.proto
pkg/proto/query_coord.proto pkg/proto/datapb/data_coord.pb.go
pkg/proto/querypb/query_coord.pb.go`
Blocked:
- `make check-proto-product`: conan could not resolve
`milvus01.jfrog.io`.
- Go tests were not rerun after review cleanup because `reset-milvus` is
blocked by missing `bin/milvus`; an earlier attempt was blocked by
missing `milvus_core` pkg-config.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
