- increase defaultRegisterBuf from 1 to 256 for active future
registration
- shard activeFutureManager by CPU groups (numShards = cpuNum/32,
minimum 1 shard) and route registrations round-robin
- update TestConcurrent to aggregate active future counts across all
shards
Signed-off-by: CLiqing <2208529306@qq.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
The gauge had inconsistent semantics: for dropped collections the time
series was wiped immediately, but for live collections it included
deleted-but-not-yet-GC'd indexes. FinishTask also read a stale pre-clone
pointer, adding 0 instead of the real size.
Redesign: the gauge = total serialized size of active (Finished,
non-deleted-index) SegmentIndexes per collection. Only index CRUD
mutates it; GC never touches it.
- FinishTask: adds delta using pre-update oldSize (fix stale-pointer
bug)
- MarkIndexAsDeleted: subtracts immediately on index drop
- reloadFromKV: only counts SegmentIndexes whose index definition is
alive; gauge goroutine moved after g.Wait() so m.indexes is populated
first
- recycleUnusedSegIndexes (GC): gauge mutation removed entirely
- CleanupDataCoordWithCollectionID: wipes the time series (unchanged)
Since GC no longer calls Add/Sub, nothing can recreate a time series
after CleanupDataCoordWithCollectionID deletes it.
See also: #49024
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
## Summary
Expose Arrow's internal IO thread pool capacity as a refreshable
paramtable knob so operators can raise it beyond Arrow's fixed default
of 8, independently of segcore `HIGH/MIDDLE` pools and
`minio.maxConnections`.
- New keys (both refreshable, `common` scope):
- `common.arrow.ioThreadPoolCoefficient` (default `0` → keep Arrow's
built-in default)
- `common.arrow.ioThreadPoolMaxCapacity` (default `0` → no cap)
- Effective size = `coefficient × CPU cores`, clamped by `maxCapacity`
when `> 0`.
- Wired through `init_c.{h,cpp}` + `initcore` + QueryNode paramtable
watcher; hot reloads apply without restart.
## Why
`GetIOThreadPool` runs the async range reads behind `ReadRangeCache`,
which is what actually issues S3 `GetObject` for storage v2 reads.
Milvus never calls `SetIOThreadPoolCapacity`, so this pool sticks at
Arrow's fixed default (`kDefaultNumIoThreads = 8` in arrow 17). Off-CPU
profiling on a QueryNode under heavy storage v2 load shows ~62% of
`HIGH`-pool wait time sitting on `curl_easy_perform -> poll` —
`HIGH`-pool threads blocked inside `ReadRangeCache::Wait` because
Arrow's IO pool cannot dispatch their range reads fast enough.
## Test plan
- [ ] CI: `ci-v2/build-ut-cov`, `ci-v2/e2e-amd`
- [ ] Manual: set `common.arrow.ioThreadPoolCoefficient=4` on a
QueryNode, confirm the `arrow io thread pool capacity set to N` log at
startup and that the pool is resized on subsequent config edits.
- [ ] Manual: re-run the storage-v2-heavy workload that produced the
off-CPU profile and confirm `curl_easy_perform -> poll` share drops from
~62%.
issue: #49207
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
- Add 15 new test cases for snapshot feature covering lifecycle edge
cases, alias support, and cross-database restore
- Verify fix for #47578 (DropSnapshot during active restore)
- All tests verified on latest master image
(`master-20260224-091b147e-amd64`)
### New Test Classes
| Class | Tests | Level | Description |
|-------|-------|-------|-------------|
| `TestMilvusClientSnapshotDropInvalid` | +1 | L1 | Drop during active
restore (#47578) |
| `TestMilvusClientSnapshotLifecycle` | +8 | L2/L3 | Collection
lifecycle + snapshot interactions |
| `TestMilvusClientSnapshotAlias` | +6 | L2 | Snapshot ops via
collection aliases |
### Test Details
**Lifecycle tests (L2):**
- Drop target collection during restore
- Rename source collection after snapshot creation
- Create snapshot on collection being restored to
- Restore failure leaves no resource leak
- Concurrent drop of same snapshot (idempotent)
- Create snapshot during drop of source collection
- Cross-database snapshot restore
- Drop and restore race condition (L3)
**Alias tests (L2):**
- Create snapshot via alias → describe shows real collection name
- List snapshots via alias == list via real name
- Restore from alias-created snapshot with data integrity
- List restore jobs via alias on restored collection
- Dropped alias → create snapshot fails with "not found"
- Alter alias retarget → list snapshots reflects new target
## Test plan
- [x] All 14 L1/L2 tests pass on standalone
(master-20260224-091b147e-amd64)
- [x] L3 race test works but may timeout on standalone due to resource
exhaustion
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
issue: #48185
- Add retry.WithMaxAttemptsContext to cap inner gRPC client retry from
caller's context, enabling proxy LBPolicy to failover to another replica
immediately instead of burning the full backoff budget (~13s) on a dead
QueryNode.
- Apply retry cap=1 to searchShard, queryShard (covers requery path),
and HighlightTask.getHighlightOnShardleader.
- Fast-fail on connection closing error in gRPC client to avoid
unnecessary retries on dead connections.
- Fast-fail for delegator forward delete on ServerIDMismatch.
- waitTSafe stall detection using ContextCond with configurable
stallTimeout (queryNode.waitTsafeStallTimeout, default 3s).
- Mark segments offline on ServerIDMismatch in Search/Query/QueryStream
paths (in addition to existing ErrNodeNotFound handling).
- Fix nil pointer panic in IsServerIDMismatchErr and
IsCrossClusterRoutingErr when called with nil error.
- Add cache for ParamItem.GetAsDurationByParse.
Signed-off-by: chyezh <chyezh@outlook.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary
PR #47919 (Arrow-based function chain pipeline) removed the Go-side
`Function input field cannot be nullable` validation in
`internal/util/function/rerank/rerank_base.go`. As a result, a nullable
field added via `add_collection_field` is now an accepted input for a
decay reranker.
This PR replaces the previous negative case
`test_milvus_client_add_field_with_reranker_unsupported` (which expected
the now-removed validation error) with a positive case
`test_milvus_client_add_field_used_as_decay_reranker_input`, covering
the combination `add_collection_field(nullable=True) + decay reranker on
the new field` and asserting that search succeeds.
Signed-off-by: nico <cheng.yuan@zilliz.com>
When loading a V2 (non-manifest) text-indexed segment upgraded from 2.6,
segcore fails with GetObjectSize 404 because the remote object key
carries the text_log/{buildID}/.../{fieldID}/ prefix twice, e.g.
"files/text_log/.../102/files/text_log/.../102/.managed.json_0".
Root cause: the Go->C++ contract declared in pkg/proto/segcore.proto --
"base_path ... files are relative to this path" -- was broken on the V2
legacy load path. DataCoord's kv_catalog reconstructs full paths via
metautil.BuildTextLogPaths on etcd load, then convertTextIndexStats /
convertJSONKeyStats passed those full paths straight into
segcorepb.TextIndexStats.Files while also setting BasePath. C++
TextMatchIndex::Load then prepended base_path a second time. V3 manifest
segments were unaffected because
packed.stats_resolver.stripBasePathPrefix strips the prefix before
returning.
Fix: in convertTextIndexStats / convertJSONKeyStats, strip the BasePath
prefix from each entry of Files before handing off to C++. TrimPrefix is
a no-op for V3 (Files already relative) and for the empty-basePath
defensive case, so V3 / new-segment behavior is preserved.
issue: #48547
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: Shawn Wang <shawn.wang@zilliz.com>
issue: #49013
## Summary
LocalSegment.LoadDeltaData and Delete dropped the entire batch when
`lastDeltaTimestamp >= tss[last]`, assuming deletes flow in monotonic ts
order. This assumption breaks for L0-forwarded deletes:
- L0 compaction merges a **subset** of alive L0 segments into the target
segment's manifest `_delta/`, advancing the watermark to that subset's
max ts. The L0 segments left behind still carry deletes with ts below
the new watermark (not a ts-prefix, just a subset).
- On segment load, manifest `_delta/` is applied first (watermark
jumps), then the delegator forwards the remaining L0 deltas via
BufferForwarder. Those forwarded deletes legitimately have ts below the
watermark and must still be applied.
- BufferForwarder additionally appends `(pk, ts)` in L0-segment
iteration order, so `tss[last]` is not even `max(ts)` — deletes above
the watermark get thrown out alongside those below it.
## Fix
- Drop the skip. segcore `DeletedRecord::InternalPush` is idempotent on
`(PK, ts)`, so duplicate applies are safe at the row_id mask level.
- Advance `lastDeltaTimestamp` via `max(tss)` with CAS instead of
`tss[last]`, so the high-water-mark used by the file-level skip in
`LoadDeltaLogs` and by dist reporting stays correct on unsorted batches.
## Test plan
- [x] `TestLoadDeltaData_LowerTsNotSkipped`: apply high-ts delete, then
lower-ts delete for a different PK; both must take effect.
- [x] `TestLoadDeltaData_UnsortedBatchAllApplied`: batch `tss=[1500,
500]` with watermark at 1000; all deletes must apply, watermark advances
to 1500 not 500.
- [x] `TestAdvanceLastDeltaTimestamp_NeverRegresses`: ts=5000 then
ts=2000; watermark stays at 5000.
- [x] All three fail without this fix, pass with it.
- [x] `TestSegment` full suite, delegator Delta/L0/Delete/Forward tests,
and segments Loader tests all pass.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #46358
Previous PR: #49005
GetRecoveryInfoV2 rebuilt the SegmentInfo returned to QueryCoord but
omitted the DataVersion field. QueryCoord relies on DataVersion to
detect outdated segment distributions for storage v2 binlog changes that
do not move the manifest path; a missing value caused stale segments to
be treated as up-to-date and skip reload.
Also add a regression test under TestGetRecoveryInfoV2 that sets
DataVersion on a flushed segment and asserts it is present in the
response.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
## Summary
- Revert #48994 to explore a better approach for dropping replicate
configuration
- The original implementation treated empty `ReplicateConfiguration` as
a drop signal, which will be replaced with a cleaner design
issue: #48993
## Test plan
- [ ] CI passes with the revert applied
- [ ] Follow-up PR with improved implementation
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: Yihao Dai <yihao.dai@zilliz.com>
## Summary
- Remove `validatePChannelIncreasingConstraints` which prevented
changing cluster set or topology when pchannels were increasing
- The constraint was overly restrictive — downstream consumers already
handle P+C combinations correctly through independent code paths
- Preserve `IsPChannelIncreasing` flag for downstream channel mapping
use
issue: #48993
## Test plan
- [x] Unit tests updated: previously error-expecting P+C tests now
expect success
- [x] Added new test: topology change (cluster removal) during pchannel
increase
- [x] All existing validator tests pass
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: Yihao Dai <yihao.dai@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
issue: https://github.com/milvus-io/milvus/issues/49155
1. **Search (per segment)** — each segment returns `search_topk_ratio ×
topK`
candidates per NQ instead of `topK`, giving the merger more room to
recover
the true global top.
2. **Truncate** — distance-only k-way merge across segments, keep
`refine_topk_ratio × max(sliceTopK, unityTopK)` candidates per NQ.
3. **Refine** — for each surviving candidate, recompute the exact
distance
against the raw vector via `segment->CalcDistByIDs(field_id, …)` on
indexes whose underlying Knowhere index reports
`IsIndexRefineEnabled() == true`. Segments whose index lacks refine
support
keep their coarse distances and are still merged fairly.
4. **Merge** — original PK-dedup / sort / reduce flow, now on refined
scores.
---------
Signed-off-by: chasingegg <chao.gao@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary
- `BuildValidRowIds` currently pins every chunk of a nullable vector
column at segment load, which triggers a full-column S3 fetch for
external tables and defeats `warmup=disable` for internal tables.
- Split `OffsetMapping` into `Init` (skeleton) + per-chunk `BuildChunk`,
and route the load path through `InitValidRowIds` when per-row-group
Parquet `null_count` metadata is available (storage v2 column groups).
The skeleton populates per-chunk cumulative counts without pinning any
chunk; l2p/p2l entries are filled lazily via `EnsureChunkOffsetMapping`
on first touch.
- Decouple Parquet stats collection from
`ENABLE_PARQUET_STATS_SKIP_INDEX`: stats for nullable vector fields are
always collected (required for the lazy path), while the flag continues
to gate skip-index usage and stats collection for other fields. v1
binlog path is unchanged and keeps the eager `BuildValidRowIds`
fallback.
- Search-time consumers (`SearchOnSealedColumn`,
`FilterVectorValidOffsets`, `InterimSealedIndexTranslator`) only ensure
the chunks they actually touch; brute-force search pins all chunks as
before.
## Test plan
- [x] New unit tests in `OffsetMappingTest.cpp` cover init + per-chunk
build, vec/map mode, out-of-order fills, idempotency, concurrent builds
on distinct chunks
- [x] New `ChunkedColumnInterfaceTest.cpp` uses `TYPED_TEST` to run the
lazy-path invariants against both `ChunkedColumn` and `ProxyChunkColumn`
(init without pin, ensure-only-touched, no-op on eager path,
non-nullable no-op)
- [x] Full regression sweep on `OffsetMapping*`, `test_chunked_column*`,
`ChunkedColumnInterfaceTest*`, `*Sealed*`, `*Interim*`, `*Filter*`,
`*SkipIndex*` — 450+ tests, 0 failures
- [ ] CI integration suites
issue: #49110🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: marcelo-cjl <marcelo.chen@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace keyLock-based blocking with per-key FIFO queues so that Submit()
is non-blocking to the caller. Different keys execute concurrently; same
key tasks run serially via queue.
Key changes:
- Add syncutil.Semaphore with dynamic SetCapacity and context-aware
Acquire for backpressure and graceful shutdown.
- Use goroutine in dispatchLocked to avoid deadlock when chaining tasks
from within a worker's completion path.
- Guard cleanup with sync.Once to handle the race between normal task
completion and pool rejection during shutdown.
- Explicit Close() drains all remaining queued tasks.
- resizeHandler adjusts both pool size and semaphore capacity.
issue: #49077
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary
- Add `tests/pyproject.toml` (+ `tests/.python-version`,
`tests/uv.lock`) hosting a
shared ruff configuration that covers all Python under `tests/`
(python_client,
restful_client, restful_client_v2, benchmark, scripts). uv only manages
the ruff
toolchain; each sub-directory keeps its own `requirements.txt` for
runtime deps.
- Register ruff `ruff-check` / `ruff-format` pre-commit hooks scoped to
`tests/**/*.py`, so only staged new/modified code is checked locally.
- Add `.github/workflows/python-lint.yaml`: on pushes to master and on
PRs touching
Python files under `tests/`, run `ruff check` and `ruff format --check`
against
**only the PR-diff changed files**. This intentionally avoids blocking
on the
existing baseline.
This PR only lands the baseline configuration. The 316 existing `.py`
files under
`tests/` are untouched — no mass reformat — and will be cleaned up
incrementally in
follow-up PRs per sub-directory.
Enabled ruff rules: `E`, `F`, `W`, `I`, `UP`. Target Python: `3.10`,
line length `120`.
## Test plan
- [ ] `cd tests && uv sync && uv run ruff --version` works
- [ ] `uv run ruff check scripts/` and `uv run ruff format --check
scripts/` run successfully
- [ ] pre-commit `ruff-check` / `ruff-format` hooks fire on a modified
`tests/**/*.py`
- [ ] The new `Python Lint (tests/)` GitHub Actions job runs green on
this PR
(no `.py` under `tests/` changed, so it should short-circuit to success)
- [ ] Existing CI (Code Checker, E2E, etc.) is unaffected
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Summary
- Remove `tests/benchmark/` — long-unmaintained benchmark framework, no
CI references.
## Test plan
- [ ] PR/Nightly CI pipelines pass (no references to removed paths
remain).
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Summary
Fixed a bug where templated expressions wrapped by unary NOT operators
were not properly propagating the `IsTemplate` flag to the outer
expression, causing `FillExpressionValue` to skip template placeholder
substitution.
## Changes
- **parser_visitor.go**: Added `IsTemplate:
childExpr.expr.GetIsTemplate()` to the UnaryExpr construction in
`VisitUnary()` to ensure the template flag is propagated from child
expressions to the parent unary expression.
- **fill_expression_value_test.go**: Added comprehensive regression test
`TestUnaryNotWithTemplate()` covering four scenarios:
- NOT wrapping templated term expressions with array values
- NOT wrapping templated unary range expressions
- NOT wrapping compound expressions with templates
- NOT wrapping templated JSON contains expressions
## Implementation Details
The fix ensures that when a unary NOT operator wraps a templated
expression, the `IsTemplate` flag is correctly set on the outer
UnaryExpr. This allows the expression filler to properly process and
substitute template placeholders in all nested expressions, preventing
VAL_NOT_SET errors and silent incorrect results.
issue: #49141
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
## Summary
- When `ClientConfig.DialOptions` is set to any non-nil value,
`dialOptions()` silently drops all `DefaultGrpcOpts` (keepalive,
backoff, MaxRecvMsgSize), causing connection stability and message size
issues
- Fix `dialOptions()` to always apply `DefaultGrpcOpts` as baseline
before appending user's custom `DialOptions`
- Simplify `WithGrpcAuthority()` — it previously copied
`DefaultGrpcOpts` as a workaround for this bug, now only sets the
authority option
issue: #48828
## Test plan
- [x] Existing `TestClient` and `TestWithGrpcAuthority` tests pass
- [x] Added `TestDialOptionsAlwaysIncludesDefaults` and
`TestDialOptionsWithNilDialOptions` to verify defaults are always
present
- [x] Full `client/milvusclient` test suite passes
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary
Two improvements to BM25 stats memory accounting in idfOracle:
### 1. Per-entry estimate: 80 → 20 bytes/entry
The previous hardcoded `bm25StatsPerEntryBytes = 80` overestimates
`map[uint32]int32` memory by 4-7x. Empirical measurement on Go 1.24
(median of 5 runs, n ≥ 10K to avoid GC noise):
| n | heap delta | bytes/entry | overestimate vs 80 |
|---|---|---|---|
| 10K | 152 KB | 15.22 | 5.2× |
| 100K | 1.22 MB | 12.18 | 6.6× |
| 1M | 19.4 MB | 19.43 | 4.1× |
| 5M | 77.9 MB | 15.58 | 5.1× |
| 10M | 156 MB | 15.58 | 5.1× |
| 50M | 623 MB | 12.47 | 6.4× |
Steady-state cost oscillates between **12 and 19.5 bytes/entry**
depending on the swiss table fill ratio (peak ~19.5 right after a 2×
capacity grow, trough ~12 when near load factor 7/8).
Root cause: the original comment claimed "bucket overhead ~72B per
entry", but Go map buckets/groups hold 8 entries each, so amortized
overhead is ~10 bytes/entry, not 72.
**Fix**: lower the default to 20 (covers measured upper bound 19.5 with
small safety margin) and expose as
`queryNode.idfOracle.bm25StatsBytesPerEntry` (refreshable) for runtime
tuning.
### 2. Skip cgo Charge/Refund when tiered eviction is disabled
When `queryNode.segcore.tieredStorage.evictionEnabled = false` (the
default), the C++ caching layer's resource accounting is inert — no
eviction will be driven by it. The per-load `C.ChargeLoadedResource` /
`C.RefundLoadedResource` cgo calls are pure overhead in this case.
**Fix**: gate `syncResource` and `checkMemoryResource` on the eviction
flag. Skip entirely when eviction is off.
## Impact
- ~75% reduction in cache budget consumed by BM25 stats (when eviction
enabled)
- Eliminates per-segment cgo overhead in default deployments (eviction
off)
- Tunable per-entry estimate without redeploying
## Test plan
- [x] `TestIDFOracle` passes
- [x] `TestBM25Stats_MemSize` passes with the new configurable value
- [x] Verified memory measurement methodology with multiple sample sizes
issue: #46468🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
**Related Issue**
#48666
# What this PR does
Restores Go-level validation of the `inverted_index_algo` parameter when
creating sparse vector indexes (`SPARSE_INVERTED_INDEX`, `SPARSE_WAND`).
# Root cause
The knowhere C++ library update in #48453 (commit `fd532fb`) removed the
`inverted_index_algo` validation from `ValidateIndexParams`. As a
result, any string value — including `"INVALID_ALGO"` — was silently
accepted without error.
This caused the nightly CI test
`test_invalid_sparse_inverted_index_algo` to fail starting from build
#655, reproducing 100% of the time.
# Fix
Added explicit Go-level validation in `vecIndexChecker.StaticCheck()`
for sparse float vector types. If `inverted_index_algo` is present in
the index params, it must be one of the three supported values:
```
TAAT_NAIVE, DAAT_WAND, DAAT_MAXSCORE
```
The validation runs before the CGO call to `ValidateIndexParams`, making
it resilient to future knowhere changes.
# Changes
- `internal/util/indexparamcheck/constraints.go` — added
`SparseInvertedIndexAlgo` key constant and `SparseInvertedIndexAlgos`
valid-values slice
- `internal/util/indexparamcheck/vector_index_checker.go` — added
`inverted_index_algo` validation in `StaticCheck` for sparse float
vector types
-
`internal/util/indexparamcheck/sparse_float_vector_base_checker_test.go`
— added unit tests covering all valid algos, invalid algo, empty string,
case-sensitivity, SPARSE_WAND, and propagation through `CheckTrain`
# Testing
go test -tags dynamic,test -gcflags="all=-N -l" -count=1
./internal/util/indexparamcheck/...
---------
Signed-off-by: kailash360 <kailashkejriwal21@gmail.com>
See #48914
PR #48890 added pathless delta summary entries to
`SegmentInfo.Deltalogs` for V3 segments so compaction triggers
keep working. However, `DecompressBinLog` reconstructs V1-style
`LogPath` from `LogID` for entries with empty paths, causing
QueryNode to read from non-existent V1 paths (`file/delta_log/...`)
while real data lives under the V3 manifest
(`file/insert_log/.../_delta/...`).
Changes:
- `DecompressBinLogs`: skip V1 delta path reconstruction when
`ManifestPath` is set
- `segment_loader.go`: unify V1 and V3 delta reading into a single
code path — collect paths from either `Deltalogs` entries or the
manifest, then call `NewDeltalogReader` once
- `rw.go`: remove `NewDeltalogReaderFromManifest` wrapper, add
`StorageV3` support to `NewDeltalogReader`, export
`GetStorageConfig` helper
- `compaction/common.go`: inline manifest path extraction via
`packed.GetDeltaLogPathsFromManifest` + `NewDeltalogReader`
Test cases adapted from #49034 (credit: @congqixia).
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
## Summary
- Refactor snapshot test cases to align with PR #48143 (bind snapshot
lifecycle to collection)
- Update base class helpers: add `collection_name` to
`drop_snapshot`/`describe_snapshot`, add `source_collection_name` to
`restore_snapshot`
- Remove 18 `@pytest.mark.skip` decorators since pymilvus SDK now
supports collection_name
- Update all snapshot API calls (58 drop, 4 describe, 46 restore) to
pass required collection_name
- Rewrite `test_snapshot_list_after_drop_collection` →
`test_snapshot_cascade_delete_on_drop_collection` for new cascade delete
semantics
- Add 3 comprehensive data type tests: struct array + BM25 text match +
MinHash, BFloat16Vector, all 8 array element types
## Test plan
- [x] L0: 2 passed (basic lifecycle + restore)
- [x] L1: 19 passed (invalid params + list/describe + restore state)
- [x] L2: 49 passed (data types, indexes, partitions, concurrency, data
ops, collection properties)
- [x] All 70 tests pass against master-20260416-e87cc36e (the commit
that introduced collection-scoped snapshots)
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
- Add `t.Parallel()` to top-level tests in `tests/go_client/testcases/`
that weren't already parallel. The batch add is function-body aware — it
scans each test's body first and skips any test that already has a
`t.Parallel()` call (e.g. after a leading comment block), so we don't
hit `t.Parallel called multiple times` panics.
- Refactor `TestNullableVectorDifferentIndexTypes` to fan out its 52
sub-cases via a goroutine + semaphore pattern (concurrency=10), matching
the existing idiom in `groupby_search_test.go`.
- `advcases/` tests are intentionally **not** parallelized — per the
go_client README, the `rg`-tagged tests require `-p=1` due to
resource-group global state.
## Local measurements
| Scope | Before | After | Speedup |
|---|---|---|---|
| `TestNullableVectorDifferentIndexTypes` alone | 292s | 40s | ~7× |
| `nullable_default_value_test.go` full file | 849s | 166s | ~5× |
issue: #49135
## Test plan
- [ ] Full `go_client` CI suite runs green on this branch
- [ ] No `t.Parallel called multiple times` panics
- [ ] No race-condition failures from shared state between tests
- [ ] Suite wall-clock time in CI drops meaningfully vs prior baseline
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Eric Hou <eric.hou@zilliz.com>
Co-authored-by: Eric Hou <eric.hou@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## What
Update `test_milvus_client_search_iterator_using_mul_db` to reflect the
behavior of `SearchIteratorV2`.
## Root Cause
The test was written for `SearchIteratorV1`, which reads `db_name` from
the connection context on each `next()` call. This meant calling
`using_database()` after iterator creation would redirect subsequent
`next()` calls to the new db, causing a collection-ID mismatch and the
error `"alias or database may have been changed"`.
`SearchIteratorV2` (default since pymilvus 2.7.0rc185) captures
`db_name` at creation time via
`_generate_call_context(db_name=self._config.db_name)`. The captured
context is baked into the iterator — `using_database()` called after
creation has no effect on it. The iterator continues to return results
from the original db, so the expected error never triggers.
This caused a pre-existing CI flake observed in build
[#25680](https://jenkins-milvus-ci.milvus.io/job/MILVUS-CI-V2-PR-PIPELINES/job/milvus-e2e-pipeline-gcp/25680/).
## Fix
- Update the expected outcome from error to success
- Update `check_items` to verify the iterator returns results in the
correct batch size
- Update docstring and inline comments to describe V2 semantics
---------
Signed-off-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
- Add `sync.RWMutex` to protect `messageAckCallbacks` and
`messageAckOnceCallbacks` package-level maps in `broadcaster/registry`
from concurrent read/write access
- Add `sync.RWMutex` to protect `balance.singleton` Future from
concurrent replacement and read
- Root cause: `ResetRegistration()` (called from test setup) writes new
maps while `CallMessageAckCallback()` / `CallMessageAckOnceCallbacks()`
read them concurrently from broadcaster goroutines, causing DATA RACE
## Test plan
- [x] `TestDoForcePromoteFixIncompleteBroadcasts` passes 10/10 with
`-race`
- [x] `TestForcePromoteFailover` passes 10/10 with `-race`
- [x] Full `broadcaster/...` test suite passes 30/30 (10 independent
runs × 3 sub-packages) with `-race`
- [x] Zero DATA RACE detected post-fix
issue: https://github.com/milvus-io/milvus/issues/48586🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
issue: https://github.com/milvus-io/milvus/issues/48823
1. Async snapshot generation in distribution: move genSnapshot to a
background goroutine, reducing lock hold time in AddDistributions,
AddGrowing, MarkOfflineSegments, and RemoveDistributions.
2. Batch partition recovery in CollectionManager.Recover to reduce meta
store write calls.
3. Remove embedded StreamingNode check in resource_manager.
4. Skip partition lookup in executor.getMetaInfo for loadMeta.
5. Fix proto.Clone scope in cluster.LoadSegments.
6. Downgrade schemaChangeMutex to RLock in addDistributionIfVersionOK.
7. Fix delegator Close order: stop snapshot loop before refunding.
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
issue: #45881
## Summary
Part 1/4 of the External Table Part 8 series. Foundation layer for
external collections — delivers data-plane primitives that Parts 2–4
build on.
- **Cross-bucket external data source**: detect same-bucket prefix;
route non-matching paths through `extfs`.
- **Schemaless reader**: open reader with `nullptr` arrow schema +
explicit `needed_columns` projection instead of full pre-built arrow
schema.
- **Force nullable**: external user fields set to `nullable=true` so
Arrow buffers from external sources are consumed uniformly (`valid_data`
populated in `ArrowToDataArray`).
- **API cleanup**: replace `external_access_mode` string with
`use_take_for_output` bool.
- **Load optimization**: skip redundant S3 data pull during segment
load.
- **Shared helper**: new `pkg/util/externalspec` parsing package +
paramtable entries (100% test coverage).
- **Arrow codecs**: enable snappy/lz4 in Arrow build (fixes
[#48869](https://github.com/milvus-io/milvus/issues/48869)).
Related to [#45881](https://github.com/milvus-io/milvus/issues/45881)
(External Collection with Lakehouse Integration tracking).
## Why split this way
This PR intentionally excludes the refresh pipeline
(manager/checker/inspector/task, schema DDL updater, refresh task
update, e2e refresh tests) — those land in Part 2/4. Each PR stays on a
single theme for reviewability.
## Test plan
- [x] \`make milvus\` — clean build
- [x] \`go build -tags dynamic,test ./... && cd pkg && go build -tags
dynamic,test ./...\` — 0 errors
- [x] \`go test -tags dynamic,test ./internal/datanode/external/...
./internal/storagev2/packed/...\` — 66 PASS / 0 FAIL
- [x] \`go test -tags dynamic,test ./pkg/v2/util/externalspec/...\` — 15
PASS, 100% coverage
- [x] \`golangci-lint run --build-tags=dynamic,test\` on all changed
packages — 0 issues
- [ ] CI validation (ut-go / ut-cpp / integration-test / e2e)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #46358
ComputeDiffBinlogs and ComputeDiffColumnGroups in SegmentLoadInfo only
compared field-level membership, so a field that stayed under the same
group id / column-group index was treated as unchanged even when the
underlying files were rewritten (e.g. post-compaction manifests that
preserve the column-group layout but point at new parquet files).
ApplyLoadDiff gates reader rebuild and column eviction on *_to_replace
entries, so the loader silently kept serving stale cached chunks.
Compare the ordered file-path sequence per group and route pre-existing
fields into binlogs_to_replace / column_groups_to_replace (respecting
the existing eager-vs-lazy policy) when files differ; genuinely new
children in a files-changed group still go to binlogs_to_load so the
replace path isn't asked to evict a column that was never loaded.
Add SetColumnGroupsForTesting as a narrow hook so unit tests can
exercise ComputeDiffColumnGroups without constructing real manifest
files, and cover both diff functions for: identical files, different
file paths, different file counts, new child under changed files, and
non-double-emission for moved fields whose old group's files also
changed.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
## Summary
- `teardown()` in `tests/go_client/testcases/helper/test_setup.go` did
not return early on connection error, causing a SIGSEGV when subsequent
calls hit a nil `Client`
- `MilvusClient.Close()` in `tests/go_client/base/milvus_client.go` now
has a nil guard as a defensive measure
issue: #49111
## Changes
- `test_setup.go`: add `return` after logging connection error in
`teardown()`
- `milvus_client.go`: add nil check in `MilvusClient.Close()` before
calling inner client
## Test plan
- [ ] CI go-sdk pipeline passes without SIGSEGV when Milvus pod is slow
to start
- [ ] No regression in normal teardown flow when Milvus is available
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
issue: #49025
## Summary
- Extract parquet metadata loading out of `GroupChunkTranslator`'s
constructor into a free `LoadGroupChunkMetadata` function declared in
`segcore/ChunkedSegmentSealedImpl.h`. Returns `{ row_group_meta_list,
parquet_stats_by_field }`.
- When `ENABLE_PARQUET_STATS_SKIP_INDEX` is off (default), the loader
only reads the lightweight `RowGroupMetadataVector` (24 B per row group)
and skips `GetParquetMetadata()` entirely — no `parquet::FileMetaData`
is loaded or retained.
- When on, stats are extracted during the same parallel file pass and
the `FileMetaData` is released before the translator is constructed.
- `GroupChunkTranslator` now accepts
`std::vector<milvus_storage::RowGroupMetadataVector>&&` directly; the
`parquet_file_metas()` / `field_id_mapping()` getters are removed.
- Updated both call sites
(`ChunkedSegmentSealedImpl::load_column_group_data_internal`,
`JsonKeyStats`) and the translator unit test.
## Test plan
- [x] Local build passes (`milvus_segcore`, `milvus_index`)
- [x] Existing `GroupChunkTranslatorTest` updated and compiles
- [ ] Full CI — C++ unit tests + integration tests
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
related: #48587
Provide default userName/password (etcdadmin) for etcd authentication
and add fail-fast validation that panics with a clear message when
etcd.auth.enabled=true but credentials are empty, instead of failing
with an opaque etcd connection error.
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
Fixes#49106.
Follow-up to #48165. Adds an opt-in `CONAN_CMD` environment variable so
developers working on both master (Conan 2.x) and release-2.5/2.6
(Conan 1.x) on the same machine can select a Conan binary per-build
without switching their default `conan`. See commit message for details.
## Test plan
- [x] `make` with default conan (2.x) — unchanged behavior
- [x] `CONAN_CMD=conan-1 make` — fails with clear version mismatch error
- [x] `CONAN_CMD=/nonexistent make` — fails with clear error
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Fixes a flaky test in `test_search_bfloat16_with_pagination_default`
that failed at ~40% rate (2/5 runs locally), with overlaps of 76–79%
against the 80% threshold.
issue: #49030
### Root Cause
When `offset` is set, Milvus proxy computes `queryTopK = limit + offset`
and passes it directly to DISKANN as the search depth. Without an
explicit `search_list_size`, DISKANN scales its graph traversal
proportionally to `queryTopK`:
| Call | Internal topk | Graph depth |
|------|--------------|-------------|
| Page 1 (offset=100, limit=100) | 200 | shallow (5× less than full) |
| Page 3 (offset=300, limit=100) | 400 | medium (2.5× less than full) |
| Full search (limit=1000) | 1000 | deep |
Boundary candidates at the edge of each page differ between the
paginated and full searches, producing ~76–79% overlap and flaky
failures.
### Fix
Pin `search_list_size=1200` on **both** paginated and full searches.
DISKANN uses this as a fixed exploration budget regardless of `topk`, so
both calls examine the same candidate pool and agree on boundary
positions.
Additional improvements:
- Per-page overlap threshold: 80% → 90% (reflects actual overlap with
matched search depth)
- Added overall recall assertion (≥95%) to validate cross-page coverage
- Expanded docstring explaining the DISKANN pagination consistency model
### Test Plan
- [x] Reproduced original failure: 2/5 runs failed (40% rate), overlap
76–79%
- [x] Verified fix: 5/5 runs passed after pinning `search_list_size`
- [x] Tested against Milvus `master-20260413` on standalone instance
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This change increases the default value of deltalogMaxNum from 30 to
1000 in L0 compaction force trigger settings. This allows more segments
to be included in a single L0 compaction operation.
The change is safe because:
- We already have a maxSize limiter
(dataCoord.compaction.levelzero.forceTrigger.maxSize) that caps the
total size at 64MB, preventing excessive memory usage
- This only affects the maximum count allowed, not the minimum force
trigger threshold (deltalogMinNum: 10)
- Systems will still respect the size limit even if many small segments
are present
Resolves#46650
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: Warp <agent@warp.dev>
## Summary
- Refactor snapshot name uniqueness from global to per-collection scope
- Add cascade delete: DropCollection triggers DropSnapshotsByCollection
- Add orphan snapshot GC for deleted collections
- Add database-level filtering for ListSnapshots and
ListRestoreSnapshotJobs
- Distinguish source/target collection in RestoreSnapshot API
- Move snapshot privileges from Global level to Collection level
- Update client SDK and documentation for new API semantics
issue: #44358
issue: #47890
issue: #47883
issue: #47855
## Test plan
- [x] Unit tests for snapshot_meta (DropSnapshotsByCollection,
per-collection isolation, partial failure)
- [x] Unit tests for snapshot_manager (DropSnapshotsByCollection,
getDBCollectionIDs)
- [x] Unit tests for services (ListSnapshots/ListRestoreJobs with dbID,
RestoreSnapshot with source collectionID)
- [x] Unit tests for ddl_callbacks_snapshot (new
dropSnapshotsByCollection callback)
- [x] Unit tests for garbage_collector (orphan snapshot GC)
- [x] E2E tests for cross-database snapshot isolation
- [ ] CI validation
## Note on skipped Python E2E tests
All 18 snapshot test classes in
`tests/python_client/milvus_client/test_milvus_client_snapshot.py` are
temporarily skipped with `@pytest.mark.skip`. Reason: this PR changes
snapshot APIs (DropSnapshot, DescribeSnapshot, RestoreSnapshot) to
require `collection_name` as a mandatory parameter, but the pymilvus SDK
used in CI has not been updated to pass this parameter yet. The tests
will be re-enabled once pymilvus SDK is updated to match the new API
contract.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20251114-snapshot_design.md
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Two bugs in the Go-layer query reduce pipeline caused panics and wrong
ValidData output for nullable vector fields:
Bug 1: buildCompactIndices inferred nullable from ValidData presence.
When segcore returns empty ValidData + empty Contents (e.g. index not
ready via fill_with_empty), the compact mapping was not built, causing
getVecDataIdx to return rowIdx directly and Contents[rowIdx] to panic.
Bug 2: buildMergedFieldData set validData[i]=true when ValidData was
absent for a nullable field, incorrectly marking null rows as valid.
Fix: use schema (map[fieldID]bool) as the source of truth for
nullability. Empty ValidData for a nullable field now means "all rows
are null" rather than "non-nullable". All QN-side and proxy-side
operators (ReduceByPKTS, DeduplicatePK, ReduceByPK, ConcatAndCheckPK)
receive the nullable map built from schema at construction time.
Also add bounds checks to buildMergedScalarField for all scalar types to
guard against nullable fields with empty Data arrays.
related: #48700
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
