Make ComposeTSByTime the logical-zero time-based timestamp API, add
ComposeTSByTimeWithLogical for explicit logical composition in tests,
and remove the duplicate GetCurrentTime wrapper.
See also: #51119
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
issue: #50931
## Summary
Clean analyzer option update helpers on master to match the 2.6
follow-up.
Remove the exported UpdateParams wrapper and return C status handling
directly during option initialization.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
issue: #51183
`TestWALLifetime` is flaky (seen on unrelated PR #51180, the only
failure among 14937 Go tests). Root cause is a nondeterministic
contract, not test timing: `WaitCurrentStateReachExpected` only consults
`ctx` when it actually has to wait — if the background task has already
converged, the fast path returns without observing cancellation. Since
the background intentionally executes even canceled expected states (see
the deadlock note in `doLifetimeChanged`), `Open`/`Remove` with a
pre-canceled context race between `context.Canceled` and `nil`.
## Changes
- **`wal_state_pair.go`**: check `ctx.Err()` at the entry of
`WaitCurrentStateReachExpected`, making the canceled-context outcome
deterministic (standard Go fail-fast convention). Behavior is otherwise
unchanged: the expected state is registered before the wait, the
background still converges, WAL lifecycle ownership stays with the
background task, and the coord-side recovery (retry with new/higher
term) is untouched — only the caller's wait terminates
deterministically.
- **`wal_lifetime_test.go`**: gate the term-11 mock open so the
canceled-context waiters provably have to wait, released right after the
assertions — a regression pin that keeps the test deterministic even
independently of the fail-fast check.
## Verification
- `go test -race -tags dynamic,test -run TestWALLifetime -count=50` →
50/50 pass
- full `walmanager` package, `-race -count=3` → pass
- `gofmt` / `go vet` clean
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## Summary
- validate REST quick-create BinaryVector metricType against the
configured binary auto-index support set
- keep default BIN_IVF_FLAT quick-create limited to HAMMING/JACCARD so
SUBSTRUCTURE/SUPERSTRUCTURE/MHJACCARD fail before collection creation
- add Go handler coverage and RESTful e2e params for the previously
missed binary metrics
Follow-up for review comment on #51088.
related issue: https://github.com/milvus-io/milvus/issues/51084
## Test Plan
- [x] git diff --check
- [x] python3 -m py_compile
tests/restful_client_v2/testcases/test_collection_operations.py
- [ ] go test ./internal/distributed/proxy/httpserver -run
TestCreateCollectionQuickVectorFieldType -count=1 (blocked locally:
missing rocksdb.pc and milvus_core.pc pkg-config files)
Signed-off-by: Yanliang Qiao <yanliang.qiao@zilliz.com>
- Split the metastore catalog interfaces into coordinator-specific files
so follow-up coordinator refactors can move their catalog surfaces
independently.
- Preserve the existing interfaces, signatures, and behavior; the rework
only fixes gci import grouping in the split RootCoord and StreamingNode
catalog files.
related: #50917
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What
Validate REST v2 quick-create enum-like fields before creating the
collection.
This PR fixes `vectorFieldType` handling for quick collection creation:
- rejects invalid `vectorFieldType` values instead of silently using
`FloatVector`
- supports `FloatVector`, `BinaryVector`, `Float16Vector`,
`BFloat16Vector`, and `SparseFloatVector`
- defaults quick-create index metric by vector type: dense `COSINE`,
binary `HAMMING`, sparse `IP`
- validates explicit binary/sparse `metricType` before collection
creation
- rejects `SparseFloatVector` quick-create with `dimension`
It also fixes top-level quick-create `consistencyLevel` handling:
- parses and validates top-level `consistencyLevel`
- preserves existing `params.consistencyLevel` behavior
- rejects conflicting top-level and `params.consistencyLevel` values
Fixes#51085Fixes#51084
## Why
Previously, `/v2/vectordb/collections/create` ignored top-level
`vectorFieldType` and always generated a `FloatVector` schema in
quick-create mode. A typo such as `InvalidVectorType` could return
success and mask client-side bugs.
Binary and sparse quick-create requests without `metricType` could also
create the collection first and then fail index creation because the old
fallback metric was `COSINE`, which is incompatible with those vector
types.
For #51084, top-level `consistencyLevel` was not present in
`CollectionReq`, so JSON decoding silently dropped it. Invalid values
such as `"consistencyLevel": "Invalid"` were ignored and the collection
could be created with the default consistency level.
## Tests
- `./bin/gofumpt -w internal/distributed/proxy/httpserver/request_v2.go
internal/distributed/proxy/httpserver/handler_v2.go
internal/distributed/proxy/httpserver/handler_v2_test.go`
- `./bin/gci write internal/distributed/proxy/httpserver/request_v2.go
internal/distributed/proxy/httpserver/handler_v2.go
internal/distributed/proxy/httpserver/handler_v2_test.go
--skip-generated -s standard -s default -s
"prefix(github.com/milvus-io)" --custom-order`
- `git diff --check`
- `python3 -m py_compile
tests/restful_client_v2/testcases/test_collection_operations.py`
- `rg -n
"string\\(paramtable\\.(BinaryVectorDefaultMetricType|SparseFloatVectorDefaultMetricType)\\)|interface\\("
internal/distributed/proxy/httpserver/handler_v2.go
internal/distributed/proxy/httpserver/handler_v2_test.go`
Attempted locally but blocked by missing native dependencies:
```
go test ./internal/distributed/proxy/httpserver -run 'TestCreateCollection(QuickVectorFieldType|TopLevelConsistencyLevel)$' -count=1
# missing rocksdb.pc and milvus_core.pc
```
CI investigation before this push:
```
# build-ut-cov/code-check failed on c6 predecessor because of unconvert:
# internal/distributed/proxy/httpserver/handler_v2.go:2054:16 unnecessary conversion
# internal/distributed/proxy/httpserver/handler_v2.go:2056:16 unnecessary conversion
# internal/distributed/proxy/httpserver/handler_v2_test.go:2185:28 unnecessary conversion
# internal/distributed/proxy/httpserver/handler_v2_test.go:2217:28 unnecessary conversion
# Fixed by removing string(...) around paramtable default metric constants.
# e2e-default failed in job 40810 on:
# testcases/test_geometry_operations.py::TestGeometryCollection::test_spatial_query_and_search[False-True-sealed-ST_CROSSES]
# error: Connection refused to /v2/vectordb/collections/flush on port 19530
# the same job showed querynode/streamingnode restarts and proxy ContainerCreating during cleanup.
# REST collection tests in that job had already passed, so this looked unrelated to this PR.
# e2e-default failed again in dispatcher 13914 / job-1 32439 on:
# testcases/test_collection_operations.py::TestCreateCollectionNegative::test_create_collection_quick_setup_with_invalid_consistency_level
# root cause: the test used collection_create(), whose wrapper auto-injects params.consistencyLevel=Strong.
# That made the request hit the new conflict branch instead of the intended top-level invalid consistencyLevel branch.
# Fixed in 3e190eee56 by sending the request through raw client.post() to avoid wrapper mutation.
```
Signed-off-by: Yanliang Qiao <yanliang.qiao@zilliz.com>
## What
Filter out insert payload columns whose field is absent from the current
collection schema when converting an insert message on the QueryNode
consumption path (`TransferInsertMsgToInsertRecord`), instead of
forwarding them into segcore where `SegmentGrowingImpl::Insert`
hard-asserts on unknown fields and panics the whole StreamingNode.
## Why (root cause, verified from Loki logs of the failing run)
1. A growing segment was created under schema v104 and received one
insert whose payload — written at WAL-append time under v104 — contains
field 158 (a field generation that was later dropped; by v112 it no
longer exists in the schema).
2. The StreamingNode died once for an unrelated reason. On every
subsequent restart, `WatchDmChannels` seeds the collection with the
**latest** schema (v112, without field 158 — coord meta is updated right
after the DDL's WAL append ack, so it always runs ahead of any replaying
consumer), and WAL replay starts from the channel checkpoint, which
predates the segment's data.
3. The replayed v104 insert still carries field 158; the growing segment
rebuilt with v112 has no such column and can never gain it. Replayed
schema events (v1→v103) are correctly skipped by the schema-version gate
(#50577), so the mismatch is structural, not a race.
4. `SegmentGrowingImpl::Insert` tolerates only the inverse staleness
(segment schema newer than payload → fill empty columns). An unknown
payload field hits `AssertInfo(insert_record_.is_data_exist(field_id),
...)` → SegcoreError → `delegator.ProcessInsert` panics → the checkpoint
never advances past the message → deterministic CrashLoopBackOff.
Dropped-field data is unqueryable by definition, so skipping those
columns is semantically lossless. On the live path the WAL-append
schema-version gate plus segment fencing on schema change guarantee the
payload is a subset of the current schema, making this filter a no-op
outside replay.
A WARN log records the skipped field IDs for observability.
## Notes
- Related latent issues intentionally out of scope (to be filed
separately): compaction can leave a flushed segment out of the watch
request's flushed/dropped exclusion lists (duplicate-data risk on
replay); `Reopen`/`FillAbsentFields` never create function-output
columns on pre-existing growing segments.
issue: #51117
---------
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## Summary
issue: #50976
This PR fixes NULL / UNKNOWN propagation gaps in filter expressions for
missing nested values:
- Treat missing JSON arithmetic operands and JSON `array_length`
extraction failures as UNKNOWN instead of definite booleans.
- Preserve UNKNOWN during parser/RBO contradiction folds for nested JSON
paths, array subscripts, and struct-array subscripts.
- Treat missing array / struct-array elements as UNKNOWN during
execution.
- Propagate JSON path/cast index typed-known masks so missing paths and
cast failures are not matched by `!=`, `not in`, or outer `not(...)`.
- Propagate JSON flat index comparable-value masks for comparison
predicates.
The commits are split by issue area:
- `4f15d6fe79` `fix: treat missing json arithmetic operands as unknown`
- `69d6e95917` `fix: preserve unknown for missing nested predicates in
rbo`
- `51b58edf05` `fix: treat missing array elements as unknown`
- `d9dd4c9334` `fix: honor json path index unknown values`
- `a9d813b322` `fix: honor json flat index unknown values`
## Verification
- `LOCALSTORAGE_PATH=/tmp/milvus-test-localstorage go test
-buildvcs=false -count=1 ./internal/parser/planparserv2/rewriter`
- `ninja -C /tmp/milvus-50976-build all_tests -j32`
- focused C++ regressions, including
`JsonFlatIndexTest.*:JsonFlatIndexExprTest.*:JsonIndexTest.*:JsonPathIndexTest.*`
(49 tests)
- focused array regressions including
`Expr.TestArraySubscriptMissingElementIsUnknown` and `Expr.TestArray*`
- `git diff --check origin/master...HEAD`
## Note
While adding extra JsonFlatIndex null-expression coverage, a separate
pre-existing `PhyNullExpr::DetermineExecPath()` / `std::call_once`
deadlock was found. That hanging test is not included here; this PR
keeps JsonFlatIndex coverage scoped to comparison UNKNOWN behavior.
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
issue: #51054
## What
Sealed tantivy index builds (inverted scalar, nested array, text match,
ngram, json key stats — V5 and V7 writers alike) now deterministically
produce **one tantivy segment**:
- Build-mode writers set `NoMergePolicy` at creation, so background
policy merges never exist during a build — the historical merge-all race
("segments could not be found in the SegmentManager") is impossible by
construction rather than merely handled. The growing text index passes
`enable_background_merge=true` and keeps the default policy (long-lived
writer, periodic commits, needs bounded segment count).
- `finish()` becomes: commit → **when more than one segment remains,
merge all searchable segment ids on the same writer and propagate any
merge error** → garbage-collect → wait_merging_threads → log the
resulting segment ids. V5's `let _ = merge(...)` error swallow is
replaced with the same error-propagating shape (its best-effort merge
could silently leave multi-segment indexes when the race fired). Note:
with `NoMergePolicy` the merge deterministically yields one segment, but
`finish()` does not itself re-assert `len == 1` after the merge — the
single-segment guarantee is covered by the
`test_sealed_build_finishes_single_segment` unit test rather than a
runtime check.
## Why
The V7 writer lost V5's finish-time merge (commented out with a TODO
referencing the long-closed #45590), so every sealed V7 index ships as
~15 MB-arena-sized segments: 19 segments for a 10M-row scalar index,
~400 for a 100M-element nested array index. Multi-segment hits
range-style queries — each segment pays its own term-dict streaming +
posting decode + doc-bitmap pass. Measured (10M rows, Apple M-series,
median of 3):
| query | multi-seg | single-seg | speedup |
|---|---|---|---|
| int64 `> v` @1% (19 seg) | 10.6 ms | 4.2 ms | 2.5x |
| varchar `> v` @1% (19 seg) | 11.5 ms | 4.1 ms | 2.8x |
| array MATCH_ANY @1% (261 seg, 60M elems) | 15.9 ms | 5.1 ms | 3.1x |
| array MATCH_ANY @50% | 379.7 ms | 55.3 ms | 6.9x |
Equality/term queries are unchanged (multi ≈ single within noise). Index
size shrinks (int64 10M: 51→21 MB; array 60M elems: 698→174 MB). Cost:
build wall time +16–30% at the 100M-element worst case (seconds at
typical segment scale); total build IO is lower since intermediate
policy merges no longer write discarded segments.
## Verification
- 292 segcore element/array/match tests green on the development branch;
text/ngram/json suites' failure sets byte-identical to pre-change
baseline.
- ~15 merge-all executions up to 60M docs with `RUST_BACKTRACE=full`:
zero panics (#45590 did not reproduce), exactly 1 segment in every
finish log (36/36 log lines on the final validation run).
- Growing text index path exercised: growing never calls `Finish()`
(periodic commits, background merges retained); the sealed interim text
index does call `Finish()` and gets single-segment.
- Cross-engine benchmark (inverted multi/single-seg vs sort index vs
brute force) asserted identical result sets across all engines; the
bench lands separately with the element-level query work it depends on.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
issue: #50962
related: #50911 / #50999 (the meta-layer root cause of the `unexpected
row count after sort compaction` mismatch itself, already fixed on
master). This PR addresses the resource-leak side: insert logs uploaded
by failed StorageV3 sort compaction attempts were never reclaimed by GC
(a 500Gi MinIO PVC grew to ~997G in chaos testing).
## Root cause
Two aspects of the incident are **by design**, not defects: sort
compaction streams its output to object storage before the final
row-count validation can run, so a failed attempt necessarily leaves
uploaded files behind with the target segment never registered in meta;
and a failed compaction is re-triggered on the next cycle with a fresh
planID/targetSegmentID. Orphan files from failed attempts are an
anticipated byproduct, and the designed safety net for them is the
datacoord GC orphan scan (`gc.scanInterval` + `gc.missingTolerance`).
**The bug is that this safety net is broken for StorageV3 paths.** In
`recycleUnusedBinLogWithChecker`, V3-format paths that fail the V1
parser (deep paths such as `<seg>/_stats/bloom_filter.<field>/<id>`) hit
the `parseV3SegmentID` branch, where files whose segment is **missing
from meta** are permanently skipped as "managed by loon". But a
never-registered target segment is referenced by no manifest, so loon
cannot manage it — those files leak forever. (6-component
`_data/*.parquet` paths happen to parse via the V1 branch and are still
reclaimed.)
## Fix
`garbage_collector.go`: V3-path files whose segment does not exist in
meta are now recycled with the same semantics as V1/V2 orphans
(`missingTolerance` gate and snapshot protection preserved). Registered
V3 segments keep the existing behavior: live files are managed by loon,
and dropped segments are removed wholesale by `recycleDroppedSegments`.
We deliberately did **not** make the datanode delete the target output
on failure: when the coordinator's query RPC transiently fails, the same
plan (same targetSegmentID, same pre-allocated logIDs) can be
re-dispatched to another node while a zombie attempt is still running; a
prefix delete from the failed zombie could destroy data the successful
attempt already registered. Deletion authority stays with datacoord GC,
which checks meta and applies the tolerance window.
Note on retry amplification: a deterministically failing sort compaction
still re-uploads one full output copy per trigger cycle (~60s), which
can outpace the default GC scan cadence (168h). This PR restores
reclaimability; deployments hitting a deterministic failure can lower
`dataCoord.gc.scanInterval` to speed up reclamation. Throttling the
re-trigger rate itself (e.g. failure backoff) is left as a possible
follow-up.
## Tests
- New unit tests: GC recycles orphan deep V3 files while leaving
registered-V3 and within-tolerance files untouched.
- Full `internal/datacoord` package regression (with etcd/minio/pulsar
deps up): 0 failures; `go vet` / `gofmt` clean; no existing tests
modified.
- End-to-end A/B reproduction on a local standalone (see PR comment
below): on master, `_stats` files of orphan target segments survive
every GC pass (counted as `valid`); with this PR, GC fully reclaims
orphan dirs past `missingTolerance` while leaving the registered segment
untouched.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Co-authored-by: Claude <noreply@anthropic.com>
issue: #50905
- streamingcoord recovery: include pchannels recovered from RootCoord
collection metadata through ConfigChannelProvider so missing WAL topics
are added after stats initialization
- dml channel compatibility: expand configured DML-style pchannels up to
the recovered index while preserving pre-created-topic constraints
- channel stats: expose active pchannels from PChannelStatsManager for
metadata-driven channel recovery
---------
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #51069
This PR fixes ARRAY<FLOAT> brute-force `array_contains` execution by
dispatching FLOAT arrays through `ExecArrayContains<float>` /
`ExecArrayContainsAll<float>` instead of sharing the DOUBLE path.
Summary:
- Split ARRAY<FLOAT> contains dispatch from DOUBLE in
`JsonContainsExpr.cpp`.
- Added a regression covering the reported float literals for
`contains`, `contains_any`, and `contains_all`.
Verification from branch author:
- `clang-format-15` on touched files.
- `git diff --check --
internal/core/src/exec/expression/JsonContainsExpr.cpp
internal/core/src/exec/expression/ExprArrayTest.cpp`.
- Targeted C++ object compile was blocked by unrelated stale local
Conan/generated-header issues.
Verification before PR creation:
- `git diff --check origin/master...buqian/bob/fix-51069-array-float`.
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
issue: https://github.com/milvus-io/milvus/issues/51065
## What changed
This PR switches the default QueryCoord balancer from
`ScoreBasedBalancer` to
`ChannelLevelScoreBalancer` and keeps unknown balancer names aligned
with the
new default fallback.
It also sets the default `queryCoord.channelExclusiveNodeFactor` to `3`,
so
channel exclusive mode is enabled by default only when every channel can
average
at least three RW QueryNodes.
## Why
`ChannelLevelScoreBalancer` improves channel placement, but channel
exclusive
mode constrains segment balancing inside each channel's assigned node
group. If
a replica has too many channels for its QueryNode count, enabling
exclusive mode
can prevent global node-level balancing from smoothing out skewed
channel data.
The default factor of `3` keeps smaller or high-channel-count replicas
on the
score-based node-level balancing path until there is enough QueryNode
capacity
per channel.
## Behavior
With the default configuration, QueryCoord enables channel exclusive
mode only
when:
```text
replica.RWNodesCount() >= len(channels) * 3
```
For example, in an 8 QueryNode replica:
- 2 channels can enable channel exclusive mode.
- 3 channels cannot enable channel exclusive mode and continue using the
score-based balancing fallback.
## Details
- Update `configs/milvus.yaml` balancer and exclusive node factor
defaults
- Update paramtable defaults and default assertions
- Make unknown balancer names fall back to `ChannelLevelScoreBalancer`
- Add balancer factory coverage for default, fallback, explicit
score-based,
and explicit round-robin cases
- Add replica coverage for the default three-nodes-per-channel threshold
- Pin existing tests to legacy `ScoreBasedBalancer` or explicit factors
where
the old behavior is the scenario under test
- Update replica observer tests to provide target channel metadata
explicitly
- Add and refresh the channel exclusive mode design document under
`docs/design-docs`
## Validation
- `git diff --check`
- `source ~/.profile && source scripts/setenv.sh && GOTOOLCHAIN=local go
test -tags dynamic,test -gcflags="all=-N -l"
github.com/milvus-io/milvus/pkg/v3/util/paramtable -run
'TestComponentParam' -count=1 -v`
- Attempted reset-milvus; Docker daemon was unavailable and this
worktree has no
`bin/milvus`.
- Added `internal/querycoordv2/meta` threshold coverage; local execution
is
blocked before assertions by stale Cgo output missing
`C.SegcoreSetPrefetchThreadPoolNum`.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #51068
Follow up for #50580
Route Reopen and ApplyLoadDiff through a staged runtime/state committer
so batch load paths can keep IO-heavy work concurrent while serializing
only the final mutation into staged state.
Fix staged index readiness handling by reading/writing staged bitsets
during load, carrying index raw-data facts into the final published
delta, and avoiding mid-reopen runtime publication from JSON ngram index
loading. This also lets field-data loading observe indexes loaded
earlier in the same batch.
Add coverage for loading a vector index for a newly added schema field
to ensure the staged bitsets are resized before use.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
This PR updates the struct array invalid-case test expectation to match
the current error message for nullable sub-fields inside a non-nullable
struct.
The affected parametrized cases are:
- test_struct_array_with_nullable_field[clip_embedding1]
- test_struct_array_with_nullable_field[scalar_field]
This is a test-only change.
---------
Signed-off-by: nico <cheng.yuan@zilliz.com>
## Summary
- Add Python client TEXT LOB coverage for read/write, lifecycle,
text_match, BM25, compaction, and storage layout checks
- Add bulk import coverage for TEXT LOB payloads across JSON, CSV, and
Parquet via RemoteBulkWriter
- Include a small analyzer-token cache and reduced payload setup to keep
the suite runtime lower
issue: #50562
## Test Plan
- [x] python3 -m py_compile
tests/python_client/milvus_client/test_milvus_client_text_lob.py
tests/python_client/bulk_insert/test_bulk_insert_api.py
- [x] python3 -m pytest -o addopts='' --collect-only -q
tests/python_client/milvus_client/test_milvus_client_text_lob.py
tests/python_client/bulk_insert/test_bulk_insert_api.py::TestBulkInsert::test_text_lob_bulk_import_json_csv_parquet
- [x] Full TEXT LOB Python client file run reported at 95s with 6
workers
---------
Signed-off-by: Eric Hou <eric.hou@zilliz.com>
Co-authored-by: Eric Hou <eric.hou@zilliz.com>
issue: #50882
## Problem
`NULL` is not a grammar token in the plan parser, so the lexer treats a
bare `NULL`/`null` as an ordinary identifier. When it appears where a
value is expected — e.g. inside an `in [...]` value list:
```
id in [6560, NULL, 6722, -7856, -6757]
```
`VisitIdentifier` runs a field lookup on `NULL`:
- **without a dynamic field** it fails with the misleading `field NULL
not exist`, which misleads users into thinking they must add a column
named `NULL` to their schema;
- **with a dynamic field** it is silently mistaken for a JSON key and
accepted.
This affects both `delete()` and `query()` (they share the same
`ParseExpr` path).
## Fix
Treat bare `null`/`NULL` (case-insensitive) as a reserved word in
`VisitIdentifier` and reject it up-front with an actionable message,
regardless of whether a dynamic field exists:
```
NULL literal is not supported in expressions; use '<field> is null' or '<field> is not null' instead
```
This implements option 1 (robust rejection) from the issue. `<field> is
null` / `is not null` are unaffected (they parse via dedicated tokens,
not `VisitIdentifier`).
**The guard is schema-aware for backward compatibility** (review
feedback): `null` only becomes a create-time keyword in this PR, so a
legacy collection may own a field literally named `null`, and the bare
identifier is the only syntax that can reference a top-level scalar
field. The rejection is therefore gated on a strict `GetFieldFromName`
lookup — a real declared field named `null` resolves exactly as before
this PR (including `null is null`, which now parses as a valid predicate
on that field), while the dynamic-field fallback (the source of the
original misparse) still rejects. A JSON **sub-key** literally named
`null` additionally remains reachable via quoting, e.g. `field["null"]`
/ `$meta["null"]`, whose base identifier is the field name, not `null`.
## Test
Added `TestExpr_NullLiteral` covering rejection of `NULL` inside `in
[...]`, as a comparison operand, and as a left operand, plus
confirmation that `is null` / `is not null` still work. Added
`TestExpr_NullLiteral_LegacyNullField` locking the legacy path: a scalar
field named `null` stays queryable via the bare identifier, a JSON field
named `null` stays reachable as `null["x"]`, and any casing that doesn't
name a real field keeps the reserved-word rejection. Full `planparserv2`
package tests pass.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Note: field-name validation tightened
Alongside the expression-level rejection, `validateFieldName` (proxy)
and
`validateAddedStructFieldName` (rootcoord) now go through
`common.IsFieldNameKeyword`,
which rejects **any casing** of `null` (`null`/`NULL`/`nUlL`/…) as a
field name —
consistent with how a bare `NULL` literal is rejected in expressions.
Previously
`FieldNameKeywords` had no `null` entry, so a field literally named
`null` could be
created. This only affects **new** field creation (validation runs at
create time);
existing collections/fields are unaffected — and thanks to the
schema-aware guard
above, a legacy field literally named `null` also stays queryable.
---------
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What & why
Follow-up to #51020, which added a per-task exponential backoff to the
datacoord global scheduler. That PR clears the `backoffs` entry only
when a task reaches a terminal state, and the cleanup lives in two
places:
- `check()` — for tasks that were running and resolve to
`None`/`Finished`/`Failed`;
- `schedule()` — for tasks whose `CreateTaskOnWorker` returns
`Init`/`Retry` or `InProgress`.
`schedule()` had **no branch** for the case where `CreateTaskOnWorker`
drives a task **straight to a terminal state**
(`None`/`Finished`/`Failed`) — e.g. missing meta, unhealthy/compacted
segment, vector-array estimation failure, or the no-train /
small-segment fake-finished paths in `task_stats.go` / `task_index.go` /
`task_analyze.go`. Such a task never enters `runningTasks`, so
`check()`'s cleanup never runs and its `backoffs` entry leaks until
datacoord restarts — growing without bound under exactly the failure
storms this backoff exists to relieve.
## Change
- Add a `None`/`Finished`/`Failed` branch to the `schedule()` switch
that calls `s.backoffs.Remove(...)`, mirroring `check()`.
- Document on the `InProgress` branch that a successful `Init →
InProgress` dispatch **intentionally** keeps the accumulated failure
count: reaching `InProgress` means a slot was free, not that the cause
of earlier failures is gone, so the backoff must keep escalating if the
task fails again. Clearing it there would pin a `Create(ok) → run-fail →
Retry → …` loop at the base interval forever. Only genuine terminal
states reset the count.
- Add `TestGlobalScheduler_TerminalTaskClearsBackoff` covering the leak.
## Verification
`go test -tags dynamic,test -gcflags="all=-N -l" -count=1
./internal/datacoord/task/ -run TestGlobalScheduler` — all subtests
pass, including the new one.
issue: #51091🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
issue: #50891
## What changed
- Reserve `IndexEntryReader` future vectors before submitting
read/download tasks so submitted tasks cannot be lost to vector
reallocation failures.
- Drain already stored futures if submission/setup fails before
releasing result buffers, download state, file descriptors, or
positioned writers.
- Cover memory reads, fd downloads, and stream-to-files download paths.
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
## Summary
- Adds a `FilterBitsNode` predicate-to-filtered-bitset fast path for
all-valid predicate results: when the live validity bitmap is all true,
flip the predicate data bitmap directly and skip invalid/UNKNOWN bitmap
materialization.
- Preserves NULL/UNKNOWN semantics for mixed-validity predicate results
by keeping the invalid-row OR path, so only definitely TRUE predicate
rows pass.
- Adds focused `FilterBitsNodeTest` coverage for the all-valid fast path
and the mixed-validity path where invalid/UNKNOWN rows must still be
filtered out.
## Scope note
The fix is intentionally at the `FilterBitsNode` conversion boundary. It
does not add `ColumnVector` null-count metadata or producer-side
validity propagation. Producers that return an all-true validity bitmap
still use the fast path through the runtime `valid.all()` check;
mixed-validity results keep the existing three-valued-logic handling.
issue: #51034
## Benchmark context
Local scalar-benchmark config:
`/home/zilliz/.slock/agents/dde3be7c-41b3-4d6c-9efc-449ebfa3d026/bench_configs/array_contains_0624_min.yaml`
with 8 tests, upload disabled.
| Build | Bundle | Avg QPS | Avg p50 latency |
|---|---:|---:|---:|
| pre-`f0bab30fb3` baseline | `1783071127408` | 13722.301 | 0.077 ms |
| `f0bab30fb3` regression | `1783073484488` | 10277.493 | 0.098 ms |
| current branch with this fix | `1783090388177` | 11078.635 | 0.088 ms
|
Note: the current-branch benchmark is not a perfect apples-to-apples
comparison with the old baseline/regressed commits because later
scalar-expression changes are also included. The local run is meant to
validate the `FilterBitsNode` fast-path direction on the same minimal
`array_contains` workload.
## Verification
- [x] `git diff --check origin/master...HEAD`
- [x]
`PROTOC=/home/zilliz/scalar-benchmark/milvus-worktree-filterbits-fastpath/cmake_build/bin/protoc
ninja -C cmake_build all_tests`
- [x] `source ./scripts/setenv.sh && ./cmake_build/unittest/all_tests
--gtest_filter='FilterBitsNodeTest.*:DetermineExecPathTest.FilterBitsNode_*:ArrayBitmapE2ECheck*.CountFuncTest:ArrayBitmapE2ECheck*.INFuncTest:ArrayBitmapE2ECheck*.NotINFuncTest:Naive/ArrayInvertedIndexTest/*.ArrayContainsAny:Naive/ArrayInvertedIndexTest/*.ArrayContainsAll:ConjunctExprTest.AndKeepsUnknownRowsActiveForFollowingFalse:ColumnVectorTest.*'`
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
## Summary
issue: #51030
Add a sealed-segment fast path for filling VARCHAR primary keys from
chunked/StorageV2 sealed segments. The change bulk reads the VARCHAR
primary-key column by segment offsets and avoids the generic raw-data
path while preserving correctness and storage cost accounting.
This branch is cherry-picked from
`00261e8b48127401db6ac1ba4e39003a849d0d76` onto `origin/master`.
## Verification
- `git diff --check origin/master...HEAD`
Local `cmake --build cmake_build --target all_tests -j32` did not reach
compilation because the existing build directory fails master
reconfigure with missing `milvus-commonConfig.cmake`; this appears to be
a local build-dir/dependency configuration issue rather than a patch
compile failure.
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
## Summary
Follow-up polish to #50933 (`text_match_fuzzy`). Three related changes:
### 1. `max_edit_distance` is no longer a reserved keyword (fixes
#51058)
#50933 introduced `max_edit_distance` as a hard ANTLR lexer keyword,
which reserves that word across the *entire* expression language — a
collection with a scalar field literally named `max_edit_distance` would
fail to parse `max_edit_distance > 1` after upgrade.
This makes it a **soft keyword**: the grammar accepts a plain
`Identifier` in the option slot and `VisitTextMatchFuzzy` validates
(case-insensitively) that it is `max_edit_distance`, rejecting any other
name. `text_match_fuzzy` itself stays a reserved *operator* keyword,
like `text_match` / `phrase_match`.
### 2. Simplify the fuzzy OR
`fuzzy_match_query` built its per-token disjunction with
`BooleanQuery::with_minimum_required_clauses(subqueries, 1)` (copied
from the real-`k` `min_should_match` path). The vendored fork's
`BooleanQuery::union(queries)` expresses a plain OR of `Should` clauses
directly — semantically identical, one call, no magic `1`.
### 3. Fast path for `max_edit_distance = 0`
`K = 0` is exactly a term match, but tantivy's `FuzzyTermQuery` has no
`distance == 0` short-circuit — it still builds a Levenshtein automaton
per token. Route `K = 0` to the cheaper `match_query` (multiterms) path.
## Tests
- New `TestExpr_TextMatchFuzzy_SoftKeyword`: a field named
`max_edit_distance` in an ordinary filter parses; case-insensitive
`MAX_EDIT_DISTANCE=` works; a wrong option name is rejected.
- Existing `TestExpr_TextMatchFuzzy` and the full `planparserv2` suite
pass; `cargo check` on the tantivy binding passes.
Design doc updated accordingly.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Related to #51068
Add design doc for segment reopen atomic read update cow, which
describes the design of atomic read and update for segment reopen with
copy-on-write (COW) mechanism.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
## Summary
- keep the upstream Knowhere pin from Milvus master:
`zilliztech/knowhere.git` at `v3.0.5`
- pin
`milvus-common/1.0.0-a3299ca@milvus/dev#eea0e22b8d5e36ff6f0a5ffbed14703e`
- seed the SegCore search span into the `milvus::OpContext` used by
`ExecPlanNodeVisitor`
- include trace span types from `common/Tracer.h`, matching the latest
milvus-common helper API shape
- keep the trace parent carrier in Milvus local to each search execution
path before Knowhere/Cardinal consume it
- propagate the same `milvus::OpContext` into indexed vector iterator
paths so group-by, iterative-filter, and iterator-v2 search modes can
preserve the trace parent into Knowhere
## Dependencies
Knowhere PR https://github.com/zilliztech/knowhere/pull/1684 prepared
Knowhere to consume the OpContext trace span. This Milvus branch no
longer pins a temporary Knowhere branch; it keeps the upstream `v3.0.5`
pin from Milvus master.
The downstream source changes target the published helper package
`milvus-common/1.0.0-a3299ca@milvus/dev#eea0e22b8d5e36ff6f0a5ffbed14703e`,
produced from milvus-common commit
`a3299cac82bb85868d2e6cd9233e92202fd5ae30` and published through
conanfiles PR https://github.com/milvus-io/conanfiles/pull/155.
The branch has been rebased on current Milvus `master`; FileManager
stream path work from PR https://github.com/milvus-io/milvus/pull/50455
is inherited from upstream `master` and is no longer carried in this PR
branch.
## Verification
- Not run locally. Milvus validation is intentionally left for the
downstream image/CI validation path.
---------
Signed-off-by: jamesgao-jpg <james.gao@zilliz.com>
## Summary
Move the RBAC check before the hook interceptor.
Write the resolved roles for the current request into the request
context.
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
issue: #50920
design doc:
[docs/design-docs/design_docs/20260702-text_match_fuzzy.md](docs/design-docs/design_docs/20260702-text_match_fuzzy.md)
## What
Adds a scalar filter operator `text_match_fuzzy(field, "query",
max_edit_distance=K)` for typo-tolerant (Levenshtein edit-distance)
matching on analyzed VARCHAR fields that have a text-match index.
Filter-only: it returns a boolean bitset, no scoring (scoring is tracked
separately in #50921). `K` is an integer in `[0, 2]` (tantivy's fuzzy
automaton hard cap); out-of-range is rejected.
## How
Mirrors the existing `text_match` / `phrase_match` path through every
layer, reusing the term-dictionary FST and tantivy's `FuzzyTermQuery`
(no new index, no new storage):
- Tantivy binding: `fuzzy_match_query` tokenizes the query with the
field analyzer and ORs one `FuzzyTermQuery` per token; exposed via the
`tantivy_fuzzy_match_query` C entry (cbindgen header regenerated).
- C++ index: `TextMatchIndex::FuzzyMatchQuery` plus the wrapper,
mirroring `MatchQuery`.
- proto: `OpType.TextMatchFuzzy = 17` (next free value; 15/16 are
already `InnerMatch`/`RegexMatch`). The edit distance rides in the
existing `UnaryRangeExpr.extra_values`, like slop / min_should_match, so
no new field is added.
- grammar + Go parser: a new `text_match_fuzzy(...)` rule and
`VisitTextMatchFuzzy`, which validates `K` to `[0, 2]` and stores it in
`extra_values`.
- C++ executor: routes the op to the text-index path (dispatch,
exec-path, and offset-input guards) and calls `FuzzyMatchQuery`, with a
`[0, 2]` re-check on the C++ side (the same way `PhraseMatch` validates
slop) for plans not built through the parser.
## Tests
- Rust unit test: distance 0/1/2, fuzzy-vs-exact, multi-token OR, and
the distance `K+1` exclusion boundary.
- C++ gtest (`TextMatch.FuzzyIndex`, `GrowingNaive`, `SealedNaive`):
index-level typo matching and the `K+1` boundary, plus the executor +
growing-segment path — a typo matches the same rows as the exact term,
`not text_match_fuzzy(...)`, and the executor guard rejects out-of-range
/ missing `max_edit_distance`.
- Go parser tests: valid 0/1/2 produce the right op and extra value; a
templated query preserves the distance; out-of-range / missing /
non-integer / overflow / non-string field / match-not-enabled are all
rejected. Existing `text_match` and `phrase_match` behavior is unchanged
(full planparserv2 suite green).
## Notes
- `max_edit_distance` becomes a reserved keyword in filter expressions,
consistent with `minimum_should_match` and `threshold`.
- Out of scope, left as follow-ups: scoring (#50921), prefix-fuzzy,
ES-style `AUTO` fuzziness, and combining with `minimum_should_match`.
---------
Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
issue: #51019
## What
A task that fails on a worker (compaction / stats / index / import) is
reset to `Init`/`Retry` and re-dispatched by the DataCoord global task
scheduler on the next ~100ms tick, with no backoff and no cap. When the
failure is persistent — e.g. the task's object-storage reads are
throttled with 429 — one bad task becomes a dispatch storm that keeps
the store saturated so the task can never succeed (observed in
production: ~435k dispatches concentrated on 3 segments in one day,
~330k failing with 429, on a store with a daily GET quota).
This PR delays re-dispatch of a failed task with exponential backoff.
## How
- The scheduler records a per-task-id failure count when
`CreateTaskOnWorker` leaves the task in `Init`/`Retry`, or
`QueryTaskOnWorker` resets it to `Init`/`Retry`.
- The next dispatch is delayed by `dataCoord.taskRetryBackoffInterval`
(default 1s) doubling per consecutive failure up to
`dataCoord.taskRetryBackoffMaxInterval` (default 60s). Both are
dynamically updatable; `0` restores the legacy behavior.
- While a task waits out its backoff it yields its scheduling slot to
other pending tasks (it is skipped and re-queued, not blocking the queue
head).
- The backoff state is cleared when the task finishes, fails terminally
(dropped), or is aborted — so a task that recovers pays nothing on its
next run.
- Returning a task to the queue because no worker has free slots is
*not* counted as a failure.
Applies to every task type the global scheduler handles; no change to
the `Task` interface.
## Tests
- Deterministic unit test for the backoff math: 1s → 2s → 4s, capped at
the configured max; cleared entry ends the backoff; interval=0 disables
the mechanism.
- Scheduling test: a task whose `CreateTaskOnWorker` always fails is
dispatched once, NOT re-dispatched by the ~100ms ticks during its 1s
backoff window (previously ~5 more dispatches), and re-dispatched after
the window elapses.
- Full `internal/datacoord/task` suite green.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Related to #50368#50536
Publish schema and load_info only after reopen has prepared the new
field and index state, and move destructive cleanup into a final
post-publication phase. This closes the drop-field window where readers
could observe a new schema before the runtime state was ready.
Thread target schema snapshots through reopen and load helpers, preserve
runtime-created text-index markers across reopen, and add regression
coverage for drop-field and text-index reopen paths.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
Adds a user guide for running a **two-phase-commit (2PC) bulk import**
in a CDC replication topology, under
`docs/design-docs/design_docs/cdc/user_docs/`, and links to it from the
existing CDC guides.
In a replicating cluster, auto-commit imports are rejected; imports must
run in 2PC mode so the commit is a single WAL-ordered fence that
replicates consistently to the standby. The guide covers:
- Enabling `dataCoord.import.enableInReplicatingCluster` on both the
primary and the standby.
- Running the import with `options={"auto_commit": "false"}` on the
primary.
- Waiting until **both** clusters report the `Uncommitted` state (best
practice), then committing once on the primary.
- Verifying the data on both clusters, and the requirement that import
files be present in both clusters' object storage.
issue: #51028
## Changes
**New file:**
- `docs/design-docs/design_docs/cdc/user_docs/05-cdc-2pc-import.md`
**Modified:**
- `01-cdc-replication-overview.md` — FAQ entry linking to the new guide.
- `02-cdc-replication-quick-start.md` — pointer to the new guide.
## Verification
- **SDK level:** the guide's script drives the merged pymilvus helpers
`bulk_import` / `get_import_progress` / `commit_import`
(milvus-io/pymilvus#3538) — correct endpoints, `auto_commit=false`
forwarding, and response parsing confirmed.
- **Live end-to-end:** ran against two CDC-replicating Milvus clusters
(cluster mode, master nightly `master-20260703-331cf54b`). The full flow
worked as documented — both clusters reached `Uncommitted`, a single
`commit_import` on the primary drove both to `Completed`, and 100/100
imported rows were visible on the primary and the standby.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
issue: #50931
## Summary
Initialize analyzer runtime options with Lindera download URLs and the
default dictionary path. Register runtime config callbacks so analyzer
yaml updates are propagated to the Rust analyzer layer.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Related to #51014
TencentCloudSTSClient constructed JsonView from a temporary JsonValue,
leaving the response parser with a dangling view and causing successful
STS payloads to be treated as empty credentials. Keep the owning
JsonValue alive for the full parse scope so Tencent IAM startup no
longer panics on empty access key assertions.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
## Summary
This PR now contains **only supplementary test coverage** for the `LIKE`
escape
model — no production code.
The production fix it originally carried (`scanLikePattern` in
`internal/parser/planparserv2/pattern_match.go`, aligning the Go
optimizer with
the C++ canonical matcher in `RegexQuery.cpp`) **already landed in
master via
#50845**, which was stacked on this PR. After rebasing onto master that
production commit is dropped as already-upstream, so the diff here
reduces to
tests only.
`git diff master` touches exactly three files, all tests:
| file | layer |
|---|---|
| `internal/parser/planparserv2/plan_parser_v2_test.go` | Go optimizer
(parser-level) |
| `internal/core/src/common/RegexQueryUtilTest.cpp` | C++ executor /
canonical matcher |
| `tests/python_client/.../test_milvus_client_scalar_filtering.py` |
Python e2e |
## What the tests add
Now that raw string literals `r"..."` exist (#50845), the escape tests
read with
a **single** backslash instead of the doubled/quadrupled backslashes the
normal
string-literal layer forced — exercising the raw-string path end to end.
**Go — `TestExpr_RawString_LikeEscapeModel`** drives the optimizer
end-to-end
through `r"..."` LIKE patterns and asserts the lowered op + literal
operand:
- escaped wildcard → literal byte (`r"a\%bc"` → `Equal "a%bc"`);
- a literal `%` coexisting with an **unescaped** `%` boundary
(`r"abc\%def%"` → `PrefixMatch "abc%def"`, `r"%abc\%def%"` → `InnerMatch
"abc%def"`);
- `\\` collapsing to one literal `\`;
- a dangling trailing backslash — un-expressible as a raw string (parse
error),
and `OpType_Match` fallback via a normal string.
**C++ — `RegexQueryUtilTest.cpp`** keeps the canonical-matcher escape
tests
(literal `%`/`_` matched verbatim, multi-char decoys, long-span
positives). Raw
strings are a Go-parser-only feature, so the C++ executor tests use
plan-level
inputs.
**Python e2e** — the escaped-wildcard `LIKE` expressions and
`test_like_escaped_wildcard_is_literal` now use `r"..."` (one
backslash); the
eval oracle gains a raw-`LIKE` branch that re-quotes the verbatim raw
content via
`repr()` so it stays in lock-step with the server. A raw-string
dangling-backslash case (rejected at lex time) is added; the
matcher-level
dangling case stays a normal string since a raw string cannot end in an
odd
number of backslashes.
## Verification
```
go test -tags dynamic,test -gcflags="all=-N -l" -count=1 ./internal/parser/planparserv2/
```
planparserv2 package passes; `gofmt` clean; Python `ruff check` / `ruff
format`
clean; the oracle's raw-`LIKE` branch verified to reproduce the
documented
expected result sets.
issue: #43864
---------
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
issue: #50910
issue: #50945
Growing StorageV3 recovery could load all rows from the manifest even
when `SegmentLoadInfo.num_of_rows` represented an earlier checkpoint.
That allowed the growing text-match index to advance beyond the
checkpoint row count; the next WAL replay insert then reserved the
checkpoint offset and Tantivy rejected the lower doc id.
This PR:
- Caps recovered column-group rows to the segment checkpoint row count
before raw fields, PKs, and text indexes are populated.
- Adds `Growing.LoadStorageV3ManifestCapsRowsAtCheckpoint`, which writes
a 6-row StorageV3 manifest, loads it with a 4-row checkpoint, then
inserts the next row at offset 4.
## Verification
- `clang-format-12` on changed C++ files
- `git diff --check`
- Fresh C++ configure in `/tmp/milvus-50910-build` was blocked before
compilation because generated `internal/core/src/pb` sources were
missing: `No SOURCES given to target: milvus_pb`
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
## Summary
Fixes#50788
The DataNode compactor executor used the same mutex for task state, slot
accounting, channel enqueueing, completion callbacks, and filesystem
metric publication. When the task queue was full, `Enqueue` could block
while holding the mutex, which also blocked `Slots()` and made DataNode
`QuerySlot` time out.
This PR keeps the executor mutex scoped to in-memory state updates only:
- release the mutex before sending to `taskCh`
- update terminal task state and release slot usage before completion
callbacks
- publish filesystem metrics outside the executor lock
- add regression tests for full task queues and completion callbacks
querying slots
## Test Plan
- `go test -v -count=1 -tags dynamic,test -gcflags="all=-N -l"
-ldflags="-r ${TEST_MILVUS_WORK_DIR}/cmake_build/lib -r
${TEST_MILVUS_WORK_DIR}/internal/core/output/lib"
./internal/datanode/compactor -run "TestCompactionExecutor" -timeout
300s`
- `gofumpt -l internal/datanode/compactor/executor.go
internal/datanode/compactor/executor_test.go`
- `git diff --check`
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
## Summary
Fixes#50793
QueryCoord segment task delta filtering checked only sealed segment
distribution. Streaming/growing segments are tracked from the channel
leader view, so a streaming reduce task could be treated as reflected
immediately after the RPC returned even if the growing segment was still
present in distribution.
This PR makes segment distribution matching scope-aware:
- `DataScope_Streaming` checks channel leader view `GrowingSegments`
- sealed segment scopes keep using `SegmentDistManager`
- segment delta records persist channel name and data scope
- adds regression coverage for streaming reduce deltas staying active
until the growing distribution disappears
## Test Plan
- `gofumpt -l internal/querycoordv2/task/action.go
internal/querycoordv2/task/scheduler.go
internal/querycoordv2/task/task_test.go`
- `git diff --check HEAD~1..HEAD`
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
## Summary
Coalesce in-flight QueryCoord collection info recovery by collection ID
so concurrent tasks for the same collection share the same
`DescribeCollection` request.
This reduces repeated metadata recovery work during bursty
same-collection task submission while still avoiding a long-lived
collection info cache after the shared request completes.
This also fixes a task metric refresh bug so task-count metrics are
updated after scheduler tasks are drained to zero.
Fixes#50800
## Changes
- Add a per-executor singleflight for collection info recovery.
- Keep caller cancellation independent from the shared metadata lookup.
- Add tests covering non-caching behavior and caller cancellation.
- Refresh task metrics on empty scheduler dispatches without publishing
pre-removal task counts.
## Test Plan
- [x] `gofumpt -l internal/querycoordv2/task/executor.go
internal/querycoordv2/task/executor_test.go
internal/querycoordv2/task/scheduler.go`
- [x] `git diff --check upstream/master...HEAD`
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
## Summary
- Update the drop collection field design doc with the final
drop-function semantics introduced by PR
https://github.com/milvus-io/milvus/pull/50471.
- Document the split between detach function and drop function field
behavior.
- Align RootCoord, Proxy, SDK cache invalidation, and testing strategy
descriptions with the current implementation.
issue: #48983
Design doc:
docs/design-docs/design_docs/20260413-drop-collection-field-design.md
## Test Plan
- [x] git diff --check upstream/master..HEAD
- [x] rg stale drop-function wording in
docs/design-docs/design_docs/20260413-drop-collection-field-design.md
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
