- Rebuild TiKV write transactions on retriable begin, build-stage IO,
and classified-transient commit failures while aborting deterministic
build failures and unsafe commit outcomes.
- Preserve original errors for TiKV undetermined commit results and
caller cancellation before string-wrapping; keep private
`retry.Unrecoverable` markers inside the retry helper so they never leak
to callers.
- Stop `ReliableWriteMetaKv` from replaying undetermined TiKV write
results, correct the TiKV begin/requestTimeout comment, and add
regression tests for commit aborts, retry exhaustion, deadline stopping,
and outer reliable-write behavior.
related: #51425
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## Summary
Keep raw search distances through segment search, iterator output, index
query, and chunk merging so ranking/reduce order is not affected by
`round_decimal`.
Apply `round_decimal` only when producing the final search response,
after reduce/rerank/order-by has already determined result order.
Returned scores keep the requested precision without changing TopK
selection.
issue: #50347
---------
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## What changed
- Replace Pod deletion with Chaos Mesh `container-kill` in CDC
force-promote and source-down failover tests.
- Kill every regular container in matching Pods while preserving the Pod
objects.
- Verify that Pod UIDs remain unchanged and each container's restart
count increases.
- Clean up the generated `PodChaos` resources after fault injection.
## Why
Deleting Pods changes their identity and tests Pod recreation rather
than container-level recovery. These scenarios need to verify CDC
failover behavior when containers restart inside the existing Pods.
## Impact
CDC failover tests now exercise container failures while explicitly
validating that Kubernetes does not recreate the affected Pods.
## Validation
- `ruff check` passed for all modified files.
- `ruff format --check` passed for all modified files.
- Python compilation passed for all modified files.
- Full CDC E2E was not run locally because it requires a Kubernetes
environment with Chaos Mesh.
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
issue: #45881
## What this PR does
External refresh previously read the segment row target directly from
each DataNode configuration during execution. That made the worker input
implicit and allowed task behavior to depend on the selected node.
This change:
- keeps `dataNode.externalCollection.targetRowsPerSegment` as the
existing configuration source;
- has DataCoord snapshot the configured value into the refresh task
request;
- adds `target_rows_per_segment` to the DataCoord-to-DataNode request;
- validates the transmitted value in DataNode;
- uses the request value for fragment splitting and segment balancing;
- preserves the MilvusTable one-fragment-per-segment behavior.
## Validation
- `make generated-proto-without-cpp`
- `make lint-fix`
- `make build-go`
- `go test -gcflags="all=-N -l" -race -tags dynamic,test
github.com/milvus-io/milvus/pkg/v3/util/paramtable -failfast -count=1
-ldflags="-r ${RPATH}" -run "^TestCachedParam$" -v`
- `go test -gcflags="all=-N -l" -race
-coverprofile=/tmp/refine-rowcount-config-dc.out -tags dynamic,test
github.com/milvus-io/milvus/internal/datacoord -failfast -count=1
-ldflags="-r ${RPATH}" -run
"^TestRefreshExternalCollectionTask_CreateTaskOnWorker$" -v`
- `go test -gcflags="all=-N -l" -race
-coverprofile=/tmp/refine-rowcount-config-dn.out -tags dynamic,test
github.com/milvus-io/milvus/internal/datanode/external -failfast
-count=1 -ldflags="-r ${RPATH}" -run
"^TestRefreshExternalCollectionTaskSuite$" -v`
- `git diff --check`
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: #51250
Keep TEXT segments on StorageV3 while making growing-source flush an
explicit allowed mode instead of a mandatory TEXT path.
Propagate create-segment storage versions through WAL flusher and write
buffer, reject storage-version mismatches in DataCoord, and keep raw
segcore chunks sticky for segments that may be flushed from growing
source.
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
relate: #50304
## Summary
Add the local format design doc, use the storage writer format constant,
and introduce local_format metadata propagation with column-group split
policy support.
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
issue: #51436
### What this PR does
- Propagates a CreateCollection RecoveryStorage observation failure from
the flusher component to WAL dispatch.
- Stops before creating a DataSyncService when the recovery state was
not recorded.
- Adds coverage for the canceled observation path, including the absence
of schema lookup.
### Verification
- `go test -v -tags dynamic,test -gcflags="all=-N -l" -count=1
./internal/streamingnode/server/flusher/flusherimpl -run
"TestWALFlusher.*CreateCollection"`
- `make static-check`
- `gofumpt -l` on changed Go files
- `git diff --check`
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
issue: #50742
## What this PR does
Optimizes protobuf decoding on Milvus RPC read/write hot paths with a
hand-written decoder in `pkg/util/fastpb`. The final change contains no
gRPC connection-pool or connection-count changes.
## Fast decoder coverage
The gRPC `releaseCodec.Unmarshal` path dispatches these top-level
messages to fastpb:
- `internalpb.RetrieveResults`
- `milvuspb.InsertRequest`
- `milvuspb.UpsertRequest`
`schemapb.SearchResultData` is not a top-level codec-dispatched message.
It is decoded directly from `SlicedBlob` at the search/reduce hot paths
in:
- `internal/proxy/search_reduce_util.go`
- `internal/querynodev2/segments/result.go`
- `internal/querynodev2/local_worker.go`
- `internal/querynodev2/tasks/search_task_go_reduce.go`
Nested hot messages such as `FieldData`, scalar/vector arrays, and IDs
are decoded by the same package.
Main optimizations:
- varchar string arrays use one backing byte arena, reducing per-string
allocations
- packed float vectors use a single memcpy into self-owned memory
- cold, complex, unknown, and mismatched fields delegate or merge
through the official protobuf codec
- no new external dependency or source-buffer aliasing
## Compatibility boundary
The compatibility contract depends on the data path:
- **Untrusted client ingress (`InsertRequest` and `UpsertRequest`)**
validates proto3 strings as UTF-8 and is tested differentially against
`proto.Unmarshal`, including malformed input behavior.
- **Trusted internal result paths (`SearchResultData`,
`RetrieveResults`, and direct `FieldData` decoding)** assume canonical
Milvus/segcore protobuf output and intentionally skip UTF-8 validation
for performance. Therefore these paths do not claim equivalence for
arbitrary adversarial wire containing invalid UTF-8.
Within that boundary, the decoder preserves the protobuf behaviors
exercised by the differential tests:
- proto2 groups fall back to the official codec
- known fields with mismatched wire types fall back to the official
codec
- unknown and future fields are preserved through an official merge pass
- repeated singular messages merge from the raw wire, including
explicitly encoded proto3 default values
- repeated message-valued oneof members merge correctly, while different
variants remain last-wins
- mixed packed and unpacked scalar encodings preserve wire order
- descriptor field-set tripwires force decoder review when the protobuf
schema changes
The unsafe packed-float copy assumes the currently supported
little-endian Milvus targets such as x86-64 and arm64.
## Benchmarks
fastpb vs official `proto.Unmarshal`, using 1000-row payloads:
| path | payload | speedup | allocations/op |
|---|---|---:|---:|
| search / query | varchar | about 2x | about 1000-2035 to about 10-17 |
| search / query | float vector, dim 768 | about 6x | unchanged |
| insert | varchar with UTF-8 validation | about 1.8x | 1019 to about 10
|
## Verification
- unit and differential tests cover field descriptors, UTF-8 contracts,
wire-type mismatches, proto2 groups, unknown fields, packed/unpacked
ordering, oneof behavior, and repeated-message merge semantics
- randomized canonical-message equivalence tests cover search, retrieve,
insert, and upsert payloads
- `go test ./util/fastpb ./util/resource` passes
- `go vet ./util/fastpb ./util/resource` passes
- current full CI and end-to-end status is tracked by the PR checks
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
issue: #51446
- message properties: add an explicit `Unreplicable` marker for concrete
WAL messages and expose builder opt-in through `WithUnreplicable`
- replication: skip unreplicable messages in the CDC sender and
secondary replicate interceptor before callback replay
- DDL producers: mark unsupported snapshot, manifest backfill, and
external collection refresh broadcasts as unreplicable
- streaming docs: document the message-level skip marker and current
unsupported DDL replication behavior
Signed-off-by: chyezh <chyezh@outlook.com>
Related: #50236
- Reuse the initially loaded grantee keys/values when ListPolicy checks
legacy grantee ID ownership.
- Avoid reloading the full grantee-privileges tree once per legacy
grant.
- Add a regression test that verifies ListPolicy performs only one full
grantee-prefix load for legacy grants.
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
issue: #51248
## What changed
- correct the omitted-`target_size` case to assert ordinary manual
compaction semantics
- add a physical Force Merge target-size case using persistent segment
IDs and MinIO insert-log sizes
- add deterministic Go boundary coverage for grouping selection at the
configured threshold and threshold + 1
- add target-size memory clamp coverage for cluster and standalone
co-location modes
- extend Optimize format, signed-int64 boundary, loaded-segment refresh,
and live async lifecycle coverage
- add the async Optimize wrapper and update the PyMilvus test dependency
to 3.1.0rc64
## Why
The previous cases did not prove that an explicit Force Merge target
affected physical output, assumed the wrong grouping threshold, and
depended on manual log inspection for algorithm selection. Loaded
refresh and the public async Optimize workflow also lacked live
end-to-end coverage.
The async case exposed milvus-io/pymilvus#3680 on PyMilvus 3.1.0rc62.
Version 3.1.0rc64 contains the tuple-unpacking fix and passes the
unmodified test.
## Validation
- all 10 planned case IDs pass; 0 failed and 0 blocked
- `milvus-dev-cli` Go UT job
`go-ut-local-zhuwenxi-zhuwenxing-i-4082040-30413568` succeeded; both
test functions and all five subtests passed
- physical target-size L3 case passed against a real cluster and MinIO
in 200.68s
- loaded refresh, target format/boundary, and manual compaction cases
passed against the same server version
- async Optimize L3 passed unmodified with PyMilvus 3.1.0rc64 in 56.91s
- 56 Python nodes collect successfully; Python compilation, Ruff, and
`git diff --check` pass
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
Related to #51068
Store PK index slots and virtual PK offset maps in RuntimeResourceState.
Build Storage V1 and V2 PK indexes through the unified translator and
stage PK replacements until the final state publication.
Make PK lookup, range search, bulk subscript, and delete filtering use a
single published snapshot. Remove the sealed InsertRecord fallback and
clear stale PK resources when external segments become empty.
Add regression coverage for atomic PK replacement and zero-row external
segment cleanup.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
## What this fixes
When sort compaction reads a segment whose binlog references an object
that is missing in object storage, the DataNode panics with a
nil-pointer dereference and enters `CrashLoopBackOff`, instead of
failing the compaction task cleanly.
## Root cause
`IterativeRecordReader.Next()` assigned `ir.cur` **before** checking the
error returned by `iterate()`. When opening the next binlog chunk fails,
`newPackedRecordReader` returns a nil `*packedRecordReader` boxed into a
non-nil `RecordReader` interface (a typed-nil). The deferred
`rr.Close()` in `sortSegment` then called `Close()` on a nil receiver
and panicked — the existing `if pr.reader != nil` / `if ir.cur != nil`
guards don't help because the receiver itself is nil.
## Fix
- Only publish the reader once `iterate()` succeeds (assign to a temp,
check the error, then set `ir.cur`).
- Make `packedRecordReader.Close` / `ffiPackedRecordReader.Close`
nil-receiver safe as defense-in-depth.
- The missing-object error is still surfaced from `Next()` (it is
**not** converted to `io.EOF`), so the sort task fails cleanly and is
retried rather than silently dropping data.
## Test
Added `TestIterativeRecordReader_MissingChunkDoesNotPanic`, which
reproduces the missing-object chunk: it asserts the error is surfaced
from `Next()` and that the deferred `Close()` does not panic.
issue: #50927🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
The documented CDC lag PromQL subtracted two independently scraped
time-tick gauges, which caused idle oscillation and misleading restart
behavior. This PR fixes the query and hardens the lag metric lifecycle
without introducing a new metric.
### #50633 — idle oscillation
Subtract each gauge timestamp before calculating the difference,
canceling the cross-target scrape-phase offset:
```promql
clamp_min(
max by (channel_name) (milvus_wal_last_confirmed_time_tick - timestamp(milvus_wal_last_confirmed_time_tick))
- min by (channel_name) (milvus_cdc_last_replicated_time_tick - timestamp(milvus_cdc_last_replicated_time_tick)),
0)
```
### #51048 — restart and switchover lag lifecycle
- Seed `milvus_cdc_last_replicated_time_tick` from
`InitializedCheckpoint` before target-client initialization, so
target-unavailable startup exposes a conservative lag instead of an
absent series.
- Overwrite the seed with the target-confirmed checkpoint after
`GetReplicateInfo` succeeds.
- Ignore nil and zero checkpoints to avoid creating an epoch-valued
series.
- Remove lag-series deletion from generic stream `OnClose`.
- Delete the lag series only from `ReplicateManager.RemoveReplicator`,
the genuine etcd DELETE path. Outdated revision cleanup therefore cannot
delete the live revisions shared label pair.
issue: #50633
issue: #51048
### Test Plan
- [x] Added unit-test coverage for initialized-checkpoint seeding and
nil/zero guards.
- [x] Added regression coverage for genuine removal versus outdated
revision cleanup.
- [ ] Local Go tests and static check.
- [ ] Confirm the lag lifecycle on a live switchover environment.
---------
Signed-off-by: Yihao Dai <yihao.dai@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
issue: #51100
## Summary
The C++ persistent-storage `get` failure counter was registered without
the `status="fail"` label, unlike the other operation failure counters.
The counter was still incremented only on failed `get` operations, but
Prometheus exported it as a no-status `get` series, so status-based
dashboards and queries could not identify it as a failure series.
This PR adds `status="fail"` to `getFailMap`, making
`internal_storage_op_count{persistent_data_op_type="get",
status="fail"}` consistent with `put`, `stat`, `list`, and `remove`
failures.
## Note
This changes the Prometheus time series shape: the old no-status `get`
failure series will stop, and the new `status="fail"` series will start
from zero.
Signed-off-by: xaxys <tpnnghd@163.com>
## What
- Upgrade the root, pkg, client, test, and telemetry example modules
from Go 1.26.4 to Go 1.26.5.
- Upgrade CPU/GPU builder Dockerfiles, macOS CI Go setup, KRTE build
arg, rpm setup, meta-migration builder, and go-client test image to Go
1.26.5.
- Point CPU and GPU builder consumption in `.env` to the successful
build-env tag `20260714-c135601`.
## Why
- Latest direct Trivy scan of
`milvusdb/milvus:master-20260711-1993feae-amd64` reports CVE-2026-39822
in Go stdlib 1.26.4, fixed in 1.26.5.
- The normal daily image scan is currently blocked through Jenkins
`milvus_scan_image_daily` #540 with `exec format error`; last successful
scan is #508 from 2026-06-10, so this was verified by direct image scan.
## Image coverage
- Go toolchain fixes are shared by all Milvus runtime images; no per-OS
runtime Dockerfile package change is required.
- Builder definitions are updated for CPU ubuntu22.04, ubuntu20.04,
ubuntu24.04, amazonlinux2023, rockylinux9 and GPU ubuntu22.04,
ubuntu20.04.
- No runtime image variant is deliberately skipped; this is not an
OS-package CVE.
## Verification
- `go.dev/dl/?mode=json&include=all` confirms go1.26.5 is stable and
linux amd64/arm64 tarballs exist.
- `go env GOTOOLCHAIN GOVERSION` selects `auto` / `go1.26.5` after the
module update.
- `go list -m` succeeds for root, pkg, client, tests/go_client, and both
telemetry examples.
- `cd pkg && go test -tags dynamic,test -gcflags="all=-N -l" -count=1
./util/typeutil` passes after rebasing onto the latest upstream master.
- Jenkins build-env #40 succeeded and published builder tag
`20260714-c135601`; `.env` now selects that tag for CPU and GPU CI
builds.
- A repo-wide audit found no remaining Go 1.26.4 toolchain pins.
Note: full root `go mod tidy` is not included because this checkout has
a root-owned `deployments/docker/dev/volumes/etcd/member` directory that
makes `go mod tidy` fail with permission errors; `go mod tidy -e`
subsequently OOMed locally. No go.sum changes were produced by the
successful module tidies.
Generated by autonomous CVE maintenance.
---------
Signed-off-by: Li Liu <li.liu@zilliz.com>
issue: #51114
JsonStats shredding now loads through
`ManifestGroupTranslator`/`ChunkReader`, but JsonStats files do not have
the normal sealed-segment manifest metadata. The load path reconstructs
the needed column-group view from the existing parquet footer/schema:
column names, Arrow field ids, row counts, and file ordering. This keeps
the current `meta.json` contract unchanged; parquet key-value metadata
is only a fallback for old files that do not have the separate meta
file.
The main follow-up after switching to the manifest translator was
projection behavior. Lazy loading now builds one projected reader per
shredding column, so a predicate on one JSON path does not pull the
whole group; when warmup is enabled it still uses the full column group
so warmup can preload everything together. The added tests cover parquet
files without packed field-list metadata, multi-file ordering, lazy
single-column projection, and warmup full-group projection.
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
issue: #51372
### Problem
`MergeOp.buildResultArrays` builds a zero-hit chunk's `$id` array with a
hardcoded Int64 builder
(`operator_merge.go:713-721` before this change), while chunks that do
have hits are typed from the
first actual ID (`buildInt64Results` / `buildStringResults`). All
per-query chunks then go into
`AddColumnFromChunks`, which builds one `arrow.NewChunked` column out of
them.
For a **VarChar-PK collection** where **one query of an nq >= 2 batch
returns zero hits** while
another returns hits, those chunks mix `utf8` and `int64` and arrow
panics:
```
panic: invalid: arrow/array: mismatch data type int64 vs utf8
```
`MergeOp` sits at chain index 0, so every request that goes through a
function rerank chain — plain
search with a rerank function as well as hybrid search — can hit it. The
proxy has no recovery
interceptor on its gRPC chains
(`internal/distributed/proxy/service.go:295-307`, `416-424`) and no
`recover()` in `internal/proxy/`, so the panic takes the proxy process
down.
The input side was never affected: `collectRRFScores` /
`collectWeightedScores` read IDs row by row,
so an empty chunk is simply iterated zero times. Only the output side
was broken.
### Fix
Resolve the `$id` arrow type once per merge, from the first input that
actually carries IDs
(`resolveIDType`), and build the zero-hit chunk with that type.
Zero-row inputs are deliberately skipped when resolving: an empty result
carries no ID type of its
own and is materialized as an empty Int64 column regardless of the
collection's PK type
(`importEmptyIDs`, converter.go). When *every* input is empty, Int64 is
kept — every chunk is then
empty as well, so the column is self-consistent and the rerank
operator's `allEmpty` early-return
short-circuits before it anyway.
An unexpected arrow ID type now returns a `merr` error instead of
silently producing an Int64 array.
### Testing
- `TestMergeWithEmptyChunkVarCharIDs` — single input, VarChar IDs,
`topks=[2,0]`; asserts the merged
`$id` column stays `utf8` with an empty second chunk. Written first and
observed panicking with
`mismatch data type int64 vs utf8` on unmodified master.
- `TestMergeMultiInputEmptyLegVarCharIDs` — hybrid-search shape: one leg
entirely empty, the other
with a zero-hit query.
- The existing `TestMergeWithEmptyChunk` (Int64) still passes, pinning
the unchanged Int64 path.
- `./internal/util/function/...` green, including `-race`; `gofmt`
clean.
### Note
Found while reviewing #51156 (fix for #50969). This panic is **not**
introduced by that PR —
`operator_merge.go` is byte-identical between master and its head, and
the repro above does not touch
the lines it changes. #51156 does widen the reachable surface, though:
before it, a single-shard
zero-hit result failed earlier in the converter with `unsupported ID
type` and never reached
`MergeOp`; after it, that input builds a DataFrame and walks into this
panic.
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## What
Drop `t.Parallel()` from the heavy go_client cases in `snapshot_test.go`
(10 cases) and `external_table_refresh_test.go` (12 cases), and mark
each with a comment explaining why it must stay serial.
## Why
`t.Parallel()` was blanket-added to every go_client testcase in #49138.
That is fine for the light cases, but the snapshot/restore and
external-table refresh cases are heavy: they insert tens of thousands of
vectors and then wait minutes for index build / snapshot / restore.
Running them alongside the large parallel test swarm starves the shared
standalone cluster, and they flake on their own index-build / restore
timeouts, e.g.:
- `TestSnapshotRestoreWithMultiSegment` — timeout waiting for indexes to
be built
- `TestSnapshotRestoreWithMultiFields` — timeout waiting for restore to
complete
Go runs the non-parallel tests first (sequentially, each with the
cluster to itself), then the parallel suite. Keeping these heavy cases
serial gives them the resources they need while leaving the light
parallel tests untouched. The trade is a little extra wall-clock for
stability.
## Changes
- `tests/go_client/testcases/snapshot_test.go` — remove `t.Parallel()`
from 10 heavy cases; add marker comment.
- `tests/go_client/testcases/external_table_refresh_test.go` — remove
`t.Parallel()` from 12 heavy cases; add marker comment.
No test logic changed; `gofmt` clean, `go vet ./testcases/` passes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What this PR does
Adds an integration test (`TestWpRetentionTruncateRead`) for the
Woodpecker (wp)
WAL implementation pinning the read semantics around truncation that
Milvus
relies on but had no explicit coverage for.
Milvus truncates the wp WAL aggressively right after a flush, but
truncation
only moves a metadata watermark — it does **not** delete data. Data
at/under the
watermark stays readable until the retention-TTL GC physically removes
it. The
test pins this with three sub-cases:
1. **TruncatedDataStillReadable** — after truncating to the latest id,
an
explicit `StartFrom` at any position in `(earliest, latest]` (spanning
segment boundaries) still returns every message, including the watermark
id
itself. As a contrast, a fresh `DeliverPolicy_All` reader is parked past
the
watermark and returns nothing — proving the watermark really moved while
the
data underneath is intact.
2. **SeekForwardWithinSegment** / **SeekForwardCrossSegment** — a reader
asking
for data older than the earliest deliverable position
(`DeliverPolicy_All`
after truncation) is moved forward to the first available position
instead of
erroring or stalling. The two sub-cases exercise both branches of
`adjustPendingReadPointIfTruncated`: a truncation point inside segment 0
(within-segment) vs. in a later segment (cross-segment id jump).
## Notes
- **Deterministic by design**: large retention (`72h`) so GC never
fires, a
small `segmentRollingPolicy.maxSize` so writes span several segments,
single-threaded writes so ids are monotonic, and all assertions are
message-id based. Run logs confirm each sub-case hits its intended
adjustment
branch (`truncatedSegmentId` 0 vs >0).
- **Scope**: covers the open-time read-position adjustment after
truncation. It
does NOT exercise the physical retention-TTL GC + runtime skip-forward
path —
that needs waiting for the async GC sweep and is intentionally left out
to
keep the test deterministic.
- Runs against local fs storage; the truncate/read logic under test
lives in the
woodpecker log handle/reader and is storage-backend agnostic.
issue: #43638
---------
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
issue: #51253
### What this PR does
When `proxy.enableCachedServiceProvider` is on and `DescribeCollection`
is called with only a collection id (e.g. the HTTP management API on
port 9091), the cached provider echoed the request db name — empty or
default — instead of the database the collection actually belongs to,
and never filled `db_id`. The remote path does not have this problem
because the coordinator resolves the database by DBID and returns it in
the top-level `DbName`/`DbId` fields.
The proxy meta cache already stores the actual db name of each
collection; this PR also stores the db id, and prefers the cached
`dbName`/`dbID` when assembling the cached DescribeCollection response.
### Verification
- New unit test
`TestCachedProxyServiceProvider_DescribeCollection_ByIDReturnsActualDbName`
(request carries only `collectionID`, asserts real `db_name`/`db_id` are
returned); existing tests in the file still pass.
- Verified end-to-end on a standalone build of this branch: created
`db1`/`coll1`, queried `GET :9091/api/v1/collection` with only
`collectionID` — response now returns `"db_name": "db1"` and the `db_id`
matching `databases/describe`, where it previously returned neither.
- The same logic was also verified on a 2.5-based deployment (real
cluster) before porting to master.
Happy to backport to 2.5/2.6 if needed.
Signed-off-by: fudongying <fudongying@bytedance.com>
issue: #45881
## What changed
- Mark read-only external collection delegators as not catching up DML
streaming data during cold load.
- Skip WAL tSafe waits for external Search, Query, QueryStream, and
GetStatistics reads.
- Always use `MaxTimestamp` as the MVCC timestamp for external reads,
including partial searches and requests carrying an explicit finite
`MvccTimestamp`, so loaded deltalog deletes remain visible.
- Prevent external reads from updating or exposing a required WAL MVCC
timetick.
- Preserve the existing streaming catch-up and MVCC behavior for normal
collections.
## Why
External collections are read-only and query-visible data comes from the
loaded external snapshot, not DML WAL replay. Starting from an old
channel
checkpoint can otherwise keep QueryNode unavailable while it catches up
WAL
history that cannot change the external snapshot.
Using the full MVCC range also prevents low guarantee placeholders or
explicit finite MVCC timestamps from hiding rows or deltalog deletes
already
loaded as part of that snapshot.
This PR does not change external channel checkpoints or broadcast
refresh
messages to collection vchannels.
## Validation
- `make clean && make milvus`
- `git diff --check`
- Targeted QueryNode delegator tests:
```bash
go test -tags dynamic,test -gcflags="all=-N -l" \
github.com/milvus-io/milvus/internal/querynodev2/delegator \
-run 'Test(ExternalCollectionDelegatorDoesNotCatchUpStreamingData|'\
'ExternalCollectionWaitTSafeUsesFullSnapshotTimestamp|'\
'ExternalCollectionPartialSearchUsesFullSnapshotTimestamp|'\
'ExternalCollectionQueryUsesFullSnapshotTimestamp|'\
'ExternalCollectionQueryStreamUsesFullSnapshotTimestamp|'\
'ExternalCollectionStrongConsistencyDoesNotRequestWALMVCC|'\
'NormalCollectionDelegatorCatchesUpStreamingData)$' -count=1
```
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #51068
- manage JSON index load, replace, and drop through reopen COW snapshots
- remove LoadScope_Index and the legacy QueryCoord-to-segcore update
path
- drop JSON indexes by field and nested path to preserve sibling indexes
- regenerate query proto and QueryNode mocks
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #51333
## Summary
- serialize DataCoord checkpoint mutations per channel so unrelated
channel catalog writes can proceed concurrently
- keep overlapping single, batch, mark, and drop operations serialized
with deterministic multi-channel lock ordering
- use a read lock for the read-only RootCoord `DescribeAlias` path
- exclude all newly added metrics and profiling instrumentation
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
issue: https://github.com/milvus-io/milvus/issues/42053
This PR fixes ngram index eligibility checks for UTF-8 literals by using
character count instead of byte length in the C++ gate and phase1
validation. It also adds regression coverage for short multibyte
literals to ensure they correctly fall back instead of being sent to
Tantivy ngram queries.
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
issue: #36214
issue: #51324
## Summary
gcp-native-storage transitively depends on
`boost/align/aligned_allocator.hpp` and
`google/cloud/storage/retry_policy.h` headers but did not declare
milvus_conan_deps. As a result, building with ENABLE_GCP_NATIVE=ON
failed because the required Conan include directories were not
propagated to the object target during compilation.
## Changes
Link milvus_conan_deps directly to gcp-native-storage in
`internal/core/src/storage/gcp-native-storage/CMakeLists.txt` so its
usage requirements are available when compiling the target.
## Verification
Verified locally.
### Reproduce
```bash
ENABLE_GCP_NATIVE=ON make build-cpp
```
Before this change, compilation failed with:
```text
fatal error: boost/align/aligned_allocator.hpp: No such file or directory
fatal error: google/cloud/storage/retry_policy.h: No such file or directory
```
After this change:
- `ENABLE_GCP_NATIVE=ON make build-cpp` succeeds.
- `make verifiers` passes, including:
- build-cpp
- getdeps
- cppcheck
- rustcheck
- fmt
- static-check
Signed-off-by: Jeremiah Xing <dev.jrmh@gmail.com>
## Summary
issue: #51052
This PR reduces QueryNode memory pressure by avoiding duplicated or
long-lived segment metadata after sealed segment load/reopen and by
caching frequently reused schema conversion state.
## Storage Mode Scope
- For manifest-backed Storage V3 segments, both Go-side runtime load
info and segcore-side runtime load info are compacted after load/reopen.
- For Storage V2 binlog-mode segments, Go-side sealed runtime load info
is compacted, while segcore keeps the full load info because binlog diff
still depends on binlog/index metadata.
- The optimization primarily targets Storage V3, which is the forward
path.
## Commit Breakdown
1. `9d4dbdf065 enhance: cache text LOB loon arrow schema`
- Cache the Loon Arrow schema generated for the TEXT LOB binary
projection used by Storage V3 load paths.
- Invalidate the cache when schema fields change and clear mutable cache
state on schema assignment.
2. `4f0bb58ef6 enhance: avoid duplicate segment index load info`
- Remove duplicated converted index load info storage.
- Reuse the existing field-index cache for current index identity and
diff/drop decisions.
3. `89e8a15b42 enhance: compact runtime segment load info`
- Compact Go QueryNode sealed segment load info after load/reopen.
- Keep runtime-required lightweight fields and cache related data size
before dropping large metadata.
4. `8a38b2fdbf enhance: skip real count scan without deletes`
- Return row count directly when a segment has no deleted rows.
- Avoid scanning the deleted bitmap on the real-count fast path.
5. `d041837d33 enhance: compact segcore runtime load info`
- Compact manifest-backed segcore runtime load info after load/reopen.
- Preserve lightweight diff identity and local runtime state needed by
schema-only reopen and later diffs.
## Test Plan
- [x] `git diff --check upstream/master...HEAD`
- [x] `gofumpt -l internal/querynodev2/segments/segment.go
internal/querynodev2/segments/segment_loader.go
internal/querynodev2/segments/segment_test.go
internal/querynodev2/segments/utils.go
internal/querynodev2/segments/utils_test.go`
- [x] `/opt/homebrew/opt/llvm@18/bin/clang-format --dry-run --Werror
--style=file <changed C++ files>`
- [x] `ninja -C cmake_build all_tests`
- [x] `./cmake_build/unittest/all_tests
--gtest_filter='SchemaTest.ConvertToLoonArrowSchemaCachesTextLobBinarySchema:SegmentLoadInfoTest.CompactRuntimeInfoForManifest:SegmentLoadInfoTest.CompactRuntimeInfoForManifestKeepsIndexDiffIdentity:SegmentLoadInfoTest.CompactRuntimeInfoForManifestSchemaCopyKeepsIndexIdentity:SegmentLoadInfoTest.ReplaceSchemaForReopenPrunesDroppedFieldRuntimeState:SealedSegmentLoad.LoadPublishesDefaultFilledStateAfterCompact:SealedSegmentReopen.SchemaOnlyReopenPublishesDefaultFilledState:SealedSegmentReopen.SchemaAwareReopenDiscardsOlderSchema:SealedSegmentReopen.SchemaOnlyReopenPreservesCreatedTextIndexState:SealedSegmentReopen.TextIndexCreatedWipedByReopen'`
- [x] `go test -v -count=1 -tags dynamic,test -gcflags="all=-N -l"
-ldflags="-extldflags \"-Wl,-rpath,${MILVUS_WORK_DIR}/cmake_build/lib
-Wl,-rpath,${MILVUS_WORK_DIR}/internal/core/output/lib
-Wl,-rpath,${MILVUS_WORK_DIR}/cmake_build/thirdparty/boost_ext\""
./internal/querynodev2/segments -run
"Test(CompactSegmentLoadInfoForRuntime|GetSegmentRelatedDataSize)$"
-timeout 300s`
- [x] `go test -v -count=1 -tags dynamic,test -gcflags="all=-N -l"
-ldflags="-extldflags \"-Wl,-rpath,${MILVUS_WORK_DIR}/cmake_build/lib
-Wl,-rpath,${MILVUS_WORK_DIR}/internal/core/output/lib
-Wl,-rpath,${MILVUS_WORK_DIR}/cmake_build/thirdparty/boost_ext\""
./internal/querynodev2/segments -timeout 300s` timed out locally after
303.862s while still running MinIO-backed retrieve tests
(`TestRetrieveWithFilter`).
---------
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
issue: #51275
### Problem
Under StorageV3 growing-source flush, a manifest legally keeps column
groups whose fields were later dropped: drop does not rewrite flushed
data, and compaction removes the leftover lazily. Loaders however
projected such groups against the **current** schema and treated the
mismatch as an error:
- **Growing recovery** (`SegmentGrowingImpl::LoadColumnsGroups`)
submitted every manifest group with a full-schema projection; a group
whose fields were all dropped has an empty projection, which
milvus-storage rejects by contract (`ChunkReaderImpl::open`: "No needed
columns found in column group"). The channel-recovery level retried the
deterministic failure every second — observed locally: **4140 identical
failures over ~3 minutes**, the channel unavailable (`no available shard
leaders`) the whole time, until a compaction rewrite happened to remove
the group.
- **Sealed load** (`ChunkedSegmentSealedImpl::LoadColumnGroups`)
resolved every manifest column to a field id without a membership check
and tripped the schema-lookup assert on any group still carrying a
dropped column — mixed groups included.
### Fix
Treat a dropped-field column as a legal leftover at the loaders:
- Growing: skip a column group before submitting its load task when none
of its columns exist in the segment schema (logged).
- Sealed: filter resolved field ids absent from the schema snapshot
while building `cg_field_ids` (logged); a group filtered to empty
produces no load task. Mixed groups keep loading their live columns via
projection, unchanged.
### Verification
- Local A/B with a schema-evolution chaos workload (concurrent add/drop
ordinary + BM25 function field, DML/query/search, `useLoonFFI` +
growing-source flush enabled, StreamingNode SIGKILL/restart injections):
before the fix, recovery wedged in the retry loop (4140 errors / ~3 min
/ final query 503); after the fix the same scenario logs 2 group skips,
the channel recovers in seconds, and the full serial consistency
validation passes.
- New UT `LoadGrowingSegmentSkipsDroppedFieldColumnGroup`: flushes a
manifest with a field-exclusive column group (writer pattern
`"0|1|100,101"`), reloads the segment under a schema without that field,
and asserts the load succeeds with the group skipped (plus a guard
assertion that the manifest really contains the exclusive group).
- The sealed-path filter is exercised by code inspection and mirrors the
growing-side semantics; on current master the external-table load path
derives its projection from schema-owned column names and is not exposed
to this class.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## Summary
- Upgrade Knowhere dependency to 49507ef3 to pick up the fix that
removes the num_build_thread upper-bound validation.
issue: #42937
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## What this PR does
- Remove the `xfail` marker from the existing drop-and-readd
analyzer-field regression for #50484.
- Add L1 E2E coverage for the live QueryNode growing-segment reopen path
when an `enable_match` VARCHAR field with `default_value` is added to
loaded growing data.
- Verify the default value is immediately searchable on pre-existing
unflushed rows, eventually searchable on a subsequent defaulted insert
after the growing text index catches up, and remains searchable after
flush/reload.
Related fix: #51201
issue: #50484
## Scope note
This PR validates the live reopen path:
```text
loaded collection
-> insert rows that stay in a QueryNode growing segment
-> add an `enable_match` VARCHAR field with `default_value`
-> query triggers LazyCheckSchema/Reopen
-> default values for pre-existing growing rows are indexed and searchable immediately
```
It does not cover the QueryNode recovery / LoadGrowing path. That path
requires forcing a QueryNode restart, replacement, or channel rewatch
while the data is still growing: the new QueryNode can create the
segment with the latest schema and then load old binlogs written before
AddField. Since the segment schema is already current in that path,
LazyCheckSchema/Reopen is not the mechanism being tested here. Recovery
coverage should be handled by a focused stability test.
## Test results
-
`test_milvus_client_add_match_field_with_default_value_on_growing_data`:
3/3 passed
- `test_drop_then_add_same_name_analyzer_field`: passed
- Ruff lint and format checks: passed
- `git diff --check`: passed
Signed-off-by: lyyyuna <yiyang.li@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
issue: #49706
issue: #50663
## What
This PR fixes StorageV3 compaction fallback loading when delete records
are stored in compact-to child manifests instead of legacy deltalogs.
Changes:
- Add `child_manifest_paths` to DataCoord `SegmentInfo` and QueryCoord
`SegmentLoadInfo` as a load-time delete overlay.
- Make DataCoord recursively collect compact-to descendant delete
sources for cloned unhealthy segment responses.
- Keep legacy child delete logs in `Deltalogs`, and keep StorageV3 child
delete logs as `ChildManifestPaths`.
- Pass `ChildManifestPaths` through QueryCoord to QueryNode.
- Make QueryNode load parent delete data and child manifest delete data
into one `DeltaData` before calling `LoadDeltaData`.
## Why
Fallback loading keeps the parent segment identity, but StorageV3
compact-to children may keep newer delete records only in their
manifests. The old fallback path only appended legacy child deltalogs,
so it could miss V3 child deletes and expose rows that should be
filtered.
The V2 path is kept for compatibility. Mixed chains such as parent V2 to
child V3, and parent V3 to child V2, are handled by keeping each child
delete source in its native representation.
## Validation
- `gofmt -w internal/datacoord/services.go
internal/querycoordv2/utils/types.go
internal/querynodev2/segments/segment_loader.go`
- `git diff --check`
- `source ~/.profile && source scripts/setenv.sh && make
generated-proto-without-cpp`
- `git diff --exit-code -- pkg/proto/data_coord.proto
pkg/proto/query_coord.proto pkg/proto/datapb/data_coord.pb.go
pkg/proto/querypb/query_coord.pb.go`
Blocked:
- `make check-proto-product`: conan could not resolve
`milvus01.jfrog.io`.
- Go tests were not rerun after review cleanup because `reset-milvus` is
blocked by missing `bin/milvus`; an earlier attempt was blocked by
missing `milvus_core` pkg-config.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
## What changed
- Remove the stale `xfail` marker from `test_file_with_bom_and_crlf`,
which covers the UTF-8 BOM regression tracked by
[#49684](https://github.com/milvus-io/milvus/issues/49684) and fixed by
[#49691](https://github.com/milvus-io/milvus/pull/49691).
- Remove the stale `skip` marker from
`test_drop_collection_then_remove_remote_resources_no_panic`, which
covers the FileResource lifecycle panic tracked by
[#49279](https://github.com/milvus-io/milvus/issues/49279) and fixed by
[#49412](https://github.com/milvus-io/milvus/pull/49412).
Both issues are closed. This change makes the existing cases active CI
regressions; it does not change their fixtures, test bodies, assertions,
or server behavior.
## User impact
CI will now detect regressions where stop-word files with a UTF-8 BOM
are parsed incorrectly, or where dropping a collection and removing its
remote analyzer resources can panic the server.
## Validation
The existing cases were previously run against the latest
`upstream/master` commit `03762320e8537f855c18c28133834bf292b45361`
using image `harbor.milvus.io/manta/milvus:master-20260710-0376232`. The
`-p no:skipping` option only bypassed pytest's skip/xfail marker
handling; no case code, fixture, or assertion was changed:
```bash
python -W ignore -m pytest -v -p no:skipping \
milvus_client/test_milvus_client_file_resource.py::TestMilvusClientFileResourceContent::test_file_with_bom_and_crlf \
milvus_client/test_milvus_client_file_resource.py::TestMilvusClientFileResourceLifecycleAdvanced::test_drop_collection_then_remove_remote_resources_no_panic \
--host <temporary-milvus-host> --port 19530 --token 'root:Milvus' \
--minio_host <temporary-minio-host> --minio_bucket fr-closed-issues-master \
--tb=short
```
Result: `2 passed in 2.16s`.
The #49279 case was then repeated ten times with the same marker bypass:
```bash
python -W ignore -m pytest -q -p no:skipping \
milvus_client/test_milvus_client_file_resource.py::TestMilvusClientFileResourceLifecycleAdvanced::test_drop_collection_then_remove_remote_resources_no_panic \
--count=10 \
--host <temporary-milvus-host> --port 19530 --token 'root:Milvus' \
--minio_host <temporary-minio-host> --minio_bucket fr-closed-issues-master \
--tb=short
```
Result: `10 passed in 14.59s`. Milvus remained healthy, pod restart
count was zero, and logs contained no panic, `tokenizer.h:31`, fatal
error, or segmentation fault. The temporary validation instance has been
cleaned up.
The two node IDs also collect successfully after removing the markers:
```bash
python -W ignore -m pytest --collect-only -q \
milvus_client/test_milvus_client_file_resource.py::TestMilvusClientFileResourceContent::test_file_with_bom_and_crlf \
milvus_client/test_milvus_client_file_resource.py::TestMilvusClientFileResourceLifecycleAdvanced::test_drop_collection_then_remove_remote_resources_no_panic
```
Result: `2 tests collected in 0.09s`.
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
issue: #50961
## Summary
Allow AlterCollectionField to set or delete analyzer configuration on
string fields before they are used by text match or BM25. Keep analyzer
params immutable once text match is enabled or a BM25 function depends
on the field.
When analyzer params change, RootCoord revalidates the new analyzer
config, reserves newly referenced file resources before broadcasting,
and lets MetaTable.AlterCollection reconcile file resource refCnt for
request, replay, replicated, and recovery paths.
## Verification
- `git diff --check`
- `go test -tags dynamic,test -gcflags="all=-N -l" -count=1
./util/typeutil -run TestIsBm25FunctionInputField` from `pkg/`
- `go test -tags dynamic,test -gcflags="all=-N -l" -count=1
./internal/rootcoord -run
'TestMetaTableAlterCollectionFileResourceRefCnt|TestDDLCallbacksAlterCollectionFieldAnalyzerValidation'`
blocked locally: missing `milvus_core.pc`
- `go test -tags dynamic,test -gcflags="all=-N -l" -count=1
./internal/proxy -run TestAlterCollectionField` blocked locally: missing
`milvus_core.pc` and `milvus-storage.pc`
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
- Reuse the existing `ExprResCacheManager/ExprCacheHelper` from the
expression layer.
- Cache JSON-flat exact-path validity bitmaps by segment + field/full
JSON pointer + `Any|Numeric|String|Bool` canonical key.
- Exclude literal/operator from the key so validity can be reused across
predicates.
- Do not add a `JsonFlatIndex`-owned LRU or change the index format.
This is based on the latest `master`, including merged PR #51235.
## Verification
Verification is pending and will continue after the PR is created.
issue: #51234
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
## What
The datacoord `globalScheduler.pickNode` used **first-fit**: it returned
the first node in map-iteration order whose available slots could
satisfy the task. Because Go map iteration order is non-deterministic
and the search stops at the first fit, scheduled tasks (import /
compaction / index / stats) were not spread evenly across DataNodes — a
cluster could land most tasks on one pod while others stayed idle.
This PR maintains a per-round **max-heap** of nodes keyed by available
slots and always assigns each task to the **most-available
(least-loaded)** node (water-filling on available slots). The heap is
built once per `schedule()` round and reused across all picks, so later
picks observe the decremented slots.
The previous fallback (drain the most-available node when no node can
fully satisfy `taskSlot`) and the empty-cluster / no-slot behaviors are
**preserved**.
## Why
Related discussion:
https://github.com/milvus-io/milvus/discussions/50872 — bulk import on a
multi-DataNode cluster spent ~90% of wall-clock on a single pod. One
root cause is on the **scheduling layer**: first-fit selection did not
balance tasks across nodes. This PR addresses that layer.
> Note: the import-specific regrouping in `RegroupImportFiles` (which
can merge multiple files into a single task) is a separate,
complementary issue and is **not** addressed here.
## Test
- Reworked `TestGlobalScheduler_pickNode` (tie / least-loaded / fallback
/ single-node decrement / all-empty / empty-cluster).
- Added `TestGlobalScheduler_pickNode_Balancing`: 3 nodes × 100 slots,
30 tasks × 10 slots → each node receives exactly 10 and is drained to 0.
- `go vet` clean; full `internal/datacoord/task` package passes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
issue: #50914
---------
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Add the design document for online shard split of namespace
(multi-tenant) collections.
The design covers:
- Range routing over `big_endian(hash(namespace)) || namespace` with a
versioned routing table derived from the collection meta.
- Write switching via a `SplitShard` WAL fence message (`T_switch`) with
reject-and-refetch on the proxy; target vchannels are initialized with a
`BarrierTimeTick` so every message of the new WALs is strictly greater
than `T_switch`.
- In-place child delegators fronted by the source delegator during the
transition window (sealed segments stay single-loaded, deletes and
timeticks are forwarded back).
- Multi-round metadata-only relabel redistribution by DataCoord, with
the release-safety analysis (frozen targets, merged recovery view,
register-then-release).
- One-shot QueryCoord adoption with in-place delegator handoff,
consistency guarantees, engineering constraints, configuration, and
failure handling.
issue: #50463🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Related to #51068
Store sealed-segment text index holders and cache slots in the published
runtime resource snapshot. Stage create, load, and reopen updates
against cloned state before publishing them atomically.
Keep text index resources alive through captured snapshots, prune
indexes removed by schema reopen, and preserve created-index markers
only for fields present in the target schema.
Remove the obsolete direct LoadTextIndex path, legacy batch overloads,
and marker wrappers. Add coverage for staged visibility and old snapshot
resource lifetime.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #51117
Supersedes #51207 per team decision to ship the skipped-field tolerance
for 3.0.0; the era-consistent replay consume path (#51146) remains a
follow-up.
### Problem
A growing segment created while a field was absent from the schema never
materializes that column: the InsertRecord ctor only allocates fields
present at creation, Reopen fills later-added ordinary fields but skips
function outputs by design, and a field dropped before segment creation
is skipped at consume. Flushing such a segment with a schema snapshot
that still carries the field hit `Assert "data_.find(field_id) !=
data_.end()"` in the growing-source flush path and crash-looped the
StreamingNode (exit 134 / CrashLoopBackOff in the schema-evolution L3
runs).
### Design
Principle: the flush writes exactly `flushSchema ∩ materialized`. A
non-materialized column is legally absent — a field dropped from the
segment's own schema, or a function output recomputed by bump-schema
compaction backfill. Real schema/data inconsistency is segcore's own
concern and is verified inside the flush.
- **Go (layout source of truth)**: trim the column groups to the source
segment's materialized field ids (plus system fields) right after layout
derivation — `Fields` and the parallel `Columns` array in lockstep — so
the writer pattern, binlog meta and metacache current split all describe
the same layout. An empty materialized report is refused. On a
committed-flush ack retry the layout is trimmed to the committed binlogs
(`ChildFields` union), the persisted truth.
- **C++ (segment-internal judgement)**: a column missing from the insert
record (`HasFieldData` semantics: allocated-but-empty counts as missing)
is skipped when the field is gone from the segment's own schema
(dropped) or is a function output; a regular field still present in the
segment schema is materialized by ctor/Reopen by construction, so that
case hard-errors instead of asserting.
- Skipped columns are backfilled by bump-schema compaction via
`RecordMaterializer`.
### Review responses (@liliu-z)
- stale `Columns` / phantom column in writer pattern →
`filterColumnGroupFields` trims `Fields`+`Columns` in lockstep at all
three trim sites
- committed ack-retry currentSplit divergence → layout trimmed to
committed binlogs' `ChildFields` union
- allocated-but-empty function-output column wedging the flush →
materialized report and the flush skip both gate on `HasFieldData`
semantics (empty = not materialized)
- silent skip of a missing regular field → three-way judgement: dropped
from segment schema → skip; function output → skip;
present-in-segment-schema regular field → hard error (`has no field data
in growing segment ... but is present in the segment schema`)
- retryability routing of the missing-field error: the legally-absent
cases no longer produce an error at all; the remaining hard error
indicates real data loss and is construction-unreachable via public APIs
(the consume path asserts regular fields carry data)
### Verification
- Go UT (`growing_source_test.go`): trim semantics — dropped ordinary
field trimmed, `Columns` kept in lockstep, empty materialized report
refused, committed-retry trim by `ChildFields`
- C++ UT (`test_storage.cpp`): flush skips a non-materialized function
output; skips a dropped field carried by a staler flush schema; skips an
allocated-but-empty function-output column
- Local schema-evolution chaos runs (concurrent add/drop field + BM25
function field + DML/query/search with `useLoonFFI` + growing-source
flush enabled, incl. StreamingNode force-kill/restart injections): no
panic, no crash loop, final serial consistency validation passes. The
exact original assert requires an era-mismatch replay not reproduced
locally; that path is covered by the unit tests above.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## Summary
- add `json_subpaths` support to Tantivy `json_exist_query`
- expose exact-path existence through `ExistsQuery(..., false)`
- use a contains-specific `JsonExactPath` validity mode for actual
JSON-flat executors
- keep contains hit lookup through `In()` while using exact-path
non-null-leaf validity
## Semantics
- NULL, missing paths, JSON null, and object-only paths remain UNKNOWN
- `NOT UNKNOWN` remains UNKNOWN
- scalar values and non-empty arrays preserve current flattened contains
behavior
- empty arrays remain the documented limitation
## Verification
- `JsonFlatIndex*`: 25/25 passed
- `*JsonContains*`: 597/597 passed
- `cargo fmt --check`
- clang-format dry-run on changed non-generated C++ files
- `git diff --check`
## Benchmark
For scalar-benchmark
`json_array_operations/base/A01_contains_flat_array/json_flat` with 1M
rows and 2 matches:
- first query: 24.945 ms
- average: 24.625 ms
- P50/P99: 24.525/24.607 ms
- current local master: approximately 1114/1119 ms P50/P99
issue: #51234
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
Related to #51068
Migrate sealed segment vector index runtime state from live members into
RuntimeResourceState, including vector index entries and binlog vector
index configs. Update vector search, vector retrieval, raw-data checks,
refine, prefetch, load/drop/clear, and staged reopen paths to read and
publish through runtime snapshots.
Also update SearchOnSealedIndex to consume a pinned vector index entry
directly, and make vector index config accessors const-compatible for
snapshot usage.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
## Summary
After schema evolution adds an `enable_match` field (e.g.
`drop_collection_field` + `add_collection_field` of a same-name text
field), `text_match` could fail with `text index not found for field
<id>` on a growing segment even though `describe_index` reported
`Finished`.
## Root cause
A growing segment builds its per-segment text indexes exactly once, in
the constructor (`CreateTextIndexes`), against the construction-time
schema — `text_indexes_` has a single writer, reachable only from that
path. When a query carrying the new schema drives `LazyCheckSchema ->
Reopen(SchemaPtr)`, Reopen only backfilled empty column data
(`fill_empty_field`) and never built the text index for the newly added
field — unlike the sealed path, where both load and schema-only reopen
go through `ComputeDiff -> text_indexes_to_create -> CreateTextIndex`.
So an old growing segment upgraded by Reopen had no
`text_indexes_[new_field]`, and `text_match` hit `GetTextIndex()` and
threw `TextIndexNotFound`. The window closes once the segment is
flushed/sealed, loaded (sealed load rebuilds the index) and released
from the delegator, which is why the failure is intermittent and
self-healing but hard, non-retryable while it lasts.
## Fix
In `SegmentGrowingImpl::Reopen(SchemaPtr)`, for newly added
`IsStringDataType && enable_match` fields, **before publishing
`schema_`** (LazyCheckSchema readers don't take `sch_mutex_`, so a
concurrent query observing the new schema skips Reopen and queries
immediately — the index must already exist by then):
1. Build the text index via the new `BuildTextIndexForMeta(const
FieldMeta&)` (the field is not yet in the published `schema_`, so the
meta is passed in; mirrors sealed's `CreateTextIndexWithSchema`). The
index is **staged as a local object** — it is not visible in
`text_indexes_` yet.
2. **Index the pre-existing rows with the same default-value-or-nulls
fill the column received** (constructed directly — the content is
knowable without reading the column back, since the exclusive
`sch_mutex_` blocks concurrent inserts). This keeps the index consistent
with the column, matching sealed's create-from-raw behavior
(`BulkRawStringAt -> AddTextSealed`): when the added field carries a
`default_value`, old rows must match text queries against the default
text and report not-null. Backfilling every row (nulls included) also
keeps text-index doc offsets dense and aligned with `insert_record_`
rows, which query-side bitmap sizing relies on.
3. Force `Commit()+Reload()` on each staged index; after **every** new
field succeeds, publish all staged indexes into `text_indexes_` together
(an O(1) map insert per field under one `mutex_` acquisition).
Publish-after-success makes the failure path sound across fields as well
as within one: a present `text_indexes_` entry is always a fully
backfilled, committed index; a throw anywhere during any field's
build/backfill/commit publishes nothing and leaves `schema_`
un-advanced, so the next query rebuilds everything from scratch (no
per-field partial publication that a retry's presence guard would skip
over). It also shrinks the `mutex_` hold during Reopen from the O(N)
backfill to the final map insert, so concurrent `GetTextIndex` readers
(text_match on existing fields) are not stalled by the backfill.
Function-output fields (e.g. BM25 sparse output) remain skipped: they
are non-string and never carry a text index. The related-but-distinct
`field index meta not found` symptom for BM25 function-output fields
(follow-up of #50783: `index_meta_` is not updated on growing reopen) is
a separate fix and not covered here.
## Test
`test_schema_reopen.cpp`:
- `ReopenBuildsTextIndexForNewEnableMatchField`: growing segment on
schema v1, old rows inserted, Reopen to v2 adding a nullable
`enable_match` VARCHAR (no default) — `GetTextIndex` throws before
Reopen; after Reopen it returns the index, `MatchQuery` finds nothing
and `IsNotNull` reports all rows null, consistent with the column.
- `ReopenTextIndexIndexesDefaultValueForOldRows`: same flow with a
`default_value` on the added field — after Reopen
`MatchQuery("default")` matches all N pre-existing rows and `IsNotNull`
reports them non-null, matching sealed semantics.
- `ReopenBuildsTextIndexesForMultipleNewFields`: one Reopen adds two
enable_match fields (one nullable-null, one with default) — both indexes
come out complete through the batched staged publish.
All tests query immediately after Reopen with no explicit
`Commit()/Reload()`, asserting the forced-visibility behavior.
## Known limitation (pre-existing, follow-up)
`EnsureArrayOffsetsForStructField` still runs after `schema_` is
published, so a throw there can leave a schema-advanced segment without
array offsets — pre-existing behavior, unchanged by this PR; reworking
Reopen's publication order for non-text state is left as a follow-up.
(The text-index half of this concern is resolved: the index is now built
before publication and a build failure leaves the reopen retryable.)
issue: #50484🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
issue: #51252
## What
- Add `common.enableDriverPrefetch` with default `false`.
- Wire the config into segcore via initcore and a refresh callback.
- Skip `Driver::PrefetchAsync()` when the switch is disabled.
## Why
This provides a runtime mitigation for regressions caused by query
driver operator prefetch on short scalar queries, while keeping prefetch
configurable for follow-up validation.
## Notes
When disabled, expression execution path determination still runs lazily
on the query thread; this only disables driver-level async prefetch
submission/waiting.
## Tests
- `make generate-yaml`
- `CLANG_FORMAT=/opt/homebrew/opt/llvm@15/bin/clang-format
internal/core/run_clang_format.sh internal/core`
- `make lint-fix`
Signed-off-by: sunby <sunbingyi1992@gmail.com>
issue: #51192
design doc:
docs/design-docs/design_docs/20260708-xgboost-function-chain.md
Add native xgboost FunctionChain expression support for L0 rerank with
FileResource-backed UBJ models.
This change includes:
- xgboost FunctionChain expression registration, parameter validation,
and execution
- FileResource-based UBJ model discovery and local path resolution
- lazy model loading with singleflight, lease/refcount lifecycle
protection, and stale eviction on FileResource sync
- cgo bridge for Arrow C Data based batch prediction
- native C++ UBJ model parser and predictor for supported tree models
- runtime-disabled stub for builds without cgo and with_xgboost
- validation for unsupported params, output modes, feature count
mismatch, invalid models, unsupported objectives, unsupported boosters,
multiclass models, multi-target leaf vectors, and unsupported input
column types
- C++ unit tests, Go tests, native parity tests, and Python client L0
E2E tests
- xgboost FunctionChain design document
L2 rerank support is intentionally deferred because Proxy does not yet
support FileResource sync and local resolution.
Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
