issue: https://github.com/milvus-io/milvus/issues/48602
ref: https://github.com/milvus-io/milvus/issues/42148
This PR fixes issue: When performing element_filter search across
multiple segments, if one segment returns 0 hits, C++ reduce creates an
empty `LongArray` for `ElementIndices` which proto3 serializes as absent
(nil). The Go-side reduce then sees inconsistent `ElementIndices` (nil
vs non-nil) across segment results and returns an "inconsistent
element-level flag" error.
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
## Summary
- Add `Reader::take()` API for small result sets (<10K rows), bypassing
full column materialization during retrieve and search — significantly
reducing latency for point queries and low-topK searches on external
collections
- Extract shared helpers (`ArrowToDataArray`, `BuildTakeContext`,
`ExecuteTake`) to eliminate type-switch duplication between retrieve and
search paths
- Move `NormalizeVectorArraysToFixedSizeBinary` to `storage/Util` for
reuse across `ManifestGroupTranslator` and take() path
- Add eager refresh job state persistence to eliminate race window where
`HasActiveJob` sees `InProgress` while all tasks completed
- Add 20 C++ unit tests covering all data types, error paths, virtual
PK, and edge cases
- Add E2E benchmark framework and refresh test enhancements
Also includes [ExternalTable Part5] which enables loading and querying
external collections (schema validation, segment loading, brute-force
search, virtual PK support).
issue: https://github.com/milvus-io/milvus/issues/45881
## Test plan
- [ ] C++ unit tests: `test_external_take` (20 tests covering all data
types, error paths, virtual PK, edge cases)
- [ ] E2E benchmark tests: `external_table_benchmark_test.go`
(retrieve/search latency, scalability)
- [ ] E2E refresh tests: enhanced `external_table_refresh_test.go`
(eager state persistence, concurrent refresh)
- [ ] Verify retrieve with small result sets uses take() fast path
- [ ] Verify search with low topK uses take() fast path
- [ ] Verify fallback to full materialization for large result sets
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
gci misclassifies crypto/boring as a third-party package, but it is
actually a Go standard library package (available in BoringCrypto
builds). This conflicts with gofumpt which correctly identifies it as
standard, causing the two tools to produce different import groupings
and making lint-fix non-idempotent.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
LoadBalance uses Contains() which matches all node types (RW, RO, SQ).
With streaming nodes in replicas, a non-existent query node ID could
match a streaming node, bypassing the replica check and producing a
misleading "segment not found" error instead of "node not found".
Fix server to use ContainRWNode() for both src and dst node validation,
and fix E2E tests to derive invalid node IDs from all nodes (not just
query nodes) to avoid collisions with streaming node IDs.
issue: #48674
Signed-off-by: chyezh <chyezh@outlook.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Split the single global CPUThreadPoolExecutor into two isolated pools
(MILVUS_SEARCH_ and MILVUS_LOAD_) so that segment load operations cannot
starve search/retrieve and vice versa.
Convert SegmentLoad and ReopenSegment from synchronous CGO calls to the
async future mechanism (cgo.Async + Future<R>) already used by
AsyncSearch/AsyncRetrieve. This gives Load/Reopen proper context
cancellation via folly::CancellationToken, replacing the manual
CancellationGuard goroutine pattern.
Add OpContext parameter to Reopen(SegmentLoadInfo) so the cancellation
token is propagated through the load path.
issue: #33132#46358
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
## Summary
When a nullable vector field has all rows null, `VectorField.Data` is
`nil` (protobuf oneof not set). `MergeFieldData`'s switch on
`Vectors.Data` type hits the `default` branch, returning `"unsupported
data type: FloatVector"`, which causes `embeddingNode` to panic.
This adds a `case nil` to skip vector data merge when the source batch
has no vector data (all rows null).
## Root Cause
- Nullable FloatVector with all-null rows → `VectorField.Data = nil`
- `MergeFieldData` switch cannot match any concrete vector type → falls
through to `default`
- `embeddingNode.Operate()` calls `panic(err)` on the returned error
## Fix
- Add `case nil:` in the Vectors type switch to skip merging when source
data is nil
- Add unit test reproducing the exact scenario
Fixes#48520
## Test plan
- [x] Unit test
`TestMergeFieldData/nullable_float_vector_-_all_null_src` passes
- [x] Local standalone: insert 2000 rows with all-null nullable
FloatVector + BM25 function — no panic
- [x] Without fix: same script triggers `panic: unsupported data type:
FloatVector`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: Shao Yang <yangshao@zilliz.com>
Signed-off-by: marcelo-cjl <marcelo.chen@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When a column group is replaced via ApplyLoadDiff, the new
GroupChunkTranslator creates a MmapChunkTarget at the same file path as
the old one. The FileWriter opens the file with O_TRUNC, truncating it
to 0 bytes while the old translator's MAP_SHARED mmap is still active.
Concurrent reads on pages not in the page cache then cause SIGBUS.
Fix by appending a monotonic generation counter to the mmap file path,
ensuring old and new column groups never collide. The old file is
cleaned up by ChunkMmapGuard destructor (munmap + unlink) when the last
reference is released.
issue: #48658
Signed-off-by: chyezh <chyezh@outlook.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
issue: https://github.com/milvus-io/milvus/issues/47902
When indexes are replaced (e.g., during reload) or cleared (e.g., during
segment release), any in-flight async warmup tasks on the old index must
be cancelled first. Without this, warmup tasks could race with the index
teardown, leading to wasted work.
This change:
- Adds cancel_warmup/cancel_and_erase/cancel_and_clear helpers for
scalar, ngram, and json indexes
- Calls CancelWarmup() on old indexes before erase/clear in
LoadScalarIndex, DropIndex, DropJSONIndex, and ClearData
- Deduplicates ngram and json index replacement logic
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
QueryNodes embedded in StreamingNodes are now correctly labeled as
"streamingnode" in the GetMetrics system_info response, based on the
STREAMING-EMBEDDED session label.
issue: #48618
Signed-off-by: chyezh <chyezh@outlook.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Check CStatus return value from InitLocalChunkManagerSingleton to
propagate initialization errors instead of silently ignoring them.
issue: #48549
Signed-off-by: sunby <sunbingyi1992@gmail.com>
When sparse filter (bloom filter) prunes all segments on a QN worker,
emptySegcoreResult was returning 0 fieldDatas for aggregation queries
because OutputFieldsId is intentionally empty for aggregations. The
Delegator's reduce_by_groups operator then failed validation expecting 1
fieldData (the count result) but receiving 0.
Fix emptySegcoreResult to use GroupAggReducer.EmptyAggResult() for
aggregation queries, generating semantically correct results (e.g.,
count=0) instead of relying on FillRetrieveResultIfEmpty which cannot
handle aggregation output fields.
issue: #48603
Signed-off-by: chyezh <chyezh@outlook.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary
- Add new nightly helm values file `distributed-woodpecker-service` for
woodpecker service mode (non-embedded) deployment
- Add `distributed-woodpecker-service` to Nightly2 Jenkins matrix
- Add release name mapping `dws` in `get_release_name.sh`
- Bump milvus-helm chart version from 5.0.6 to 5.0.16 (supports
woodpecker service mode)
issue: #48644
## Test plan
- [ ] Verify nightly pipeline picks up the new
`distributed-woodpecker-service` deployment option
- [ ] Verify woodpecker service pods start with image
`harbor.milvus.io/woodpecker/woodpecker:master-latest`
- [ ] Verify existing `distributed-woodpecker` (embedded mode) nightly
tests are unaffected
- [ ] Verify `distributed-pulsar-mmap` and `distributed-kafka` nightly
tests pass with chart 5.0.16
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Eric Hou <eric.hou@zilliz.com>
Co-authored-by: Eric Hou <eric.hou@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
issue: https://github.com/milvus-io/milvus/issues/42148
Correctness:
- JSON/CSV import: reject struct array elements with mismatched field
count
- Parquet import: error on type assertion failure in scalar and vector
paths instead of silent zero-fill/data loss
- JSON import: validate per-vector dimension in ArrayOfVector
FloatVector path
- Move element_level inference from pymilvus into C++
ParsePlaceholderGroup: pymilvus cannot reliably infer if it's
elelment_level as which kinds of search on ArrayOfVector are supported
are determined by metric type but pymilvus does not have this info.
- Fix element-level search returning wrong row IDs on growing segments
with multiple chunks by using cumulative element offset instead of row
offset as begin_id in brute-force search.
https://github.com/milvus-io/milvus/issues/48617
Performance:
- Replace proto.Clone with in-place FieldName mutation in
reconstructStructFieldData
Nested index:
- field name is missing in CreateIndexInfo which is needed to determine
whether the index should be nested or not
Tests: added for all above fixes
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
issue: https://github.com/milvus-io/milvus/issues/45525
## Summary
This PR adds optimizations for boolean IN and NOT IN expressions, with
special handling for nullable boolean fields. The changes improve query
performance by rewriting certain bool IN expressions to simpler, faster
operations.
## Key Changes
- **Bool IN expression optimizations:**
- `BoolField IN [true, false]` → `AlwaysTrueExpr` (non-nullable) or `IS
NOT NULL` (nullable)
- `BoolField IN [true]` → `BoolField == true` (uses fast SIMD path)
- `BoolField IN [false]` → `BoolField == false`
- Single-value deduplication: `BoolField IN [true, true]` → `BoolField
== true`
- **Bool NOT IN expression optimizations:**
- `BoolField NOT IN [true, false]` → `AlwaysFalseExpr` (non-nullable) or
`IS NULL` (nullable)
- `BoolField NOT IN [true]` → `BoolField != true`
- `BoolField NOT IN [false]` → `BoolField != false`
- **Unary expression optimizations:**
- `NOT AlwaysTrueExpr` → `AlwaysFalseExpr`
- `NOT (IS NOT NULL)` → `IS NULL` (and vice versa)
- `NOT (col == val)` → `col != val`
- **Helper functions and test infrastructure:**
- Added `newNullExpr()` utility to create NULL check expressions
- Added `allBoolVals()` helper to validate boolean value lists
- Added `buildSchemaHelperForRewriteNullableT()` for testing nullable
fields
- Comprehensive test coverage for all bool IN/NOT IN optimization paths
## Implementation Details
The optimization logic is implemented in the `visitTermExpr()` method,
which:
1. Detects when a TermExpr operates on a boolean column
2. Analyzes the values to determine if both true and false are present
3. Rewrites to appropriate expressions based on nullability and value
set
4. Handles single-value cases by converting to equality comparisons
The unary expression visitor was enhanced to handle NOT operations on
the newly created expressions, enabling proper composition of
optimizations (e.g., NOT IN becomes NOT of an optimized expression).
https://claude.ai/code/session_01GkYztNzb75zgjuevzqVcwb
---------
Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
When DataNode is configured with `sync` mode for file resource manager
but no storage address is set in yaml (i.e., remote storage type with
empty address), the initialization previously attempted to create a
ChunkManager and failed with an error, causing the node to fail to
start. This change detects the missing address upfront and gracefully
falls back to `close` mode instead.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
Fix 16 consistently failing test cases from nightly CI master build 650
(2026-03-26) and build 651 (2026-03-27). All failures reproduced across
all 3 configs (kafka / pulsar-mmap / woodpecker).
- Nightly build 650:
https://jenkins.milvus.io:18080/job/Milvus%20Nightly%20CI(new)/job/master/650/
- Nightly build 651:
https://jenkins.milvus.io:18080/job/Milvus%20Nightly%20CI(new)/job/master/651/
## Changes
### 1. `param_check.py` — fix `output_field_value_check` for sparse and
bfloat16 fields
**Fixes:**
- `test_search_by_pk_with_output_fields_all` — `KeyError: -1` (3/3
configs)
- `test_search_with_output_fields_all@TestMilvusClientSearchBasicV2` —
`TypeError: ufunc 'equal' not supported` (3/3 configs)
**Root cause:** Sparse vector comparison used hardcoded
`original[-1][_id]` instead of pk-based lookup. bfloat16/float16 vectors
returned as `bytes` cannot be compared via numpy `ufunc 'equal'`.
**Fix:** Use pk-based index for sparse comparison; skip value check for
`bytes` type (bfloat16/float16/binary).
### 2. `test_milvus_client_hybrid_search_v2.py` — fix expected output
fields
**Fixes:**
- `test_hybrid_search_with_diff_output_fields` — `AssertionError` (3/3
configs)
**Root cause:** Test excluded ALL sparse fields from expected output,
but only BM25-generated sparse fields cannot be output. User-inserted
nullable sparse vectors CAN be output.
**Fix:** Only exclude BM25 sparse fields
(`sparse_vector_field_name1/2`), keep `nullable_sparse_vec_field_name`.
### 3. `test_milvus_client_search_by_pk.py` /
`test_milvus_client_search_v2_new.py` — exclude DISKANN from mmap test
**Fixes:**
- `test_search_by_pk_each_index_with_mmap_enabled_search[DISKANN]` —
`alter_index_properties expect True, but got False` (3/3 configs)
- `test_each_index_with_mmap_enabled_search[DISKANN]` — same error (3/3
configs)
**Root cause:** DISKANN is a disk-based index and does not support mmap.
`alter_index_properties` correctly rejects it.
**Fix:** Exclude DISKANN from the `dense_float_index_types` parametrize
list in both files.
### 4. `test_milvus_client_search_group_by.py` — rewrite nonexistent
field group_by tests
**Fixes:**
-
`test_search_group_by_nonexistent_field_on_dynamic_enabled_collection[nonexist_field]`
— `AssertionError` (3/3 configs)
-
`test_search_group_by_nonexistent_field_on_dynamic_enabled_collection[21]`
— `TypeError: object of type 'Error' has no len()` (3/3 configs)
**Root cause:** With dynamic field enabled, group_by on a nonexistent
field name returns 1 result (all rows have null for that field → one
group), not `limit` results. Passing `21` (int) as field name is an
invalid type that the server rejects.
**Fix:** Split into 2 tests:
`test_search_group_by_nonexistent_field_on_dynamic_enabled_collection`
expects `limit=1`; new `test_search_group_by_invalid_field_type` expects
`err_res` with code 65535.
### 5. `test_milvus_client_search_pagination.py` — fix search params for
pagination topk
**Fixes:**
- `test_search_with_pagination_topk[100]` — `AssertionError` (3/3
configs)
- `test_search_with_pagination_topk[3000]` — `AssertionError` (3/3
configs)
- `test_search_with_pagination_topk[10000]` — `AssertionError` (3/3
configs)
**Root cause:** Missing `metric_type` and low `nprobe=10` caused poor
recall at high offset, leading to empty or incorrect pagination results.
**Fix:** Add `metric_type: COSINE` and increase `nprobe` to 128.
### 6. `test_milvus_client_search_v2.py` — fix recall, exists
expression, and metric type error
**Fixes:**
- `test_search_with_expression_auto_id` — `recall too low: got 313,
expected >= 800` (3/3 configs)
- `test_search_with_expression_exists[json_field]` — `TypeError: object
of type 'Error' has no len()` (3/3 configs)
- `test_search_with_expression_exists[float_array]` — same error (3/3
configs)
- `test_search_with_invalid_metric_type[JACCARD]` — `KeyError:
'err_msg'` (3/3 configs)
- `test_search_with_invalid_metric_type[HAMMING]` — `KeyError:
'err_msg'` (3/3 configs)
- `test_search_with_output_fields_all@TestSearchV2Shared` —
`AssertionError` (3/3 configs)
**Fixes applied:**
- **recall**: Add `metric_type: COSINE` + `nprobe: 64` to search params
- **exists expression**: Remove `json_field` and `float_array` from
parametrize (top-level field `exists` expression is not valid for these
types)
- **invalid metric type**: Add `err_msg` assertion (was only checking
`err_code`, `KeyError` when test accessed missing key)
- **output_fields_all**: Add `new_added_field` to expected output fields
list
### 7. `test_milvus_client_sparse_search.py` — update error message for
invalid metric type
**Fixes:**
- `test_sparse_search_invalid_metric_type[L2]` — `got 65535 ... expected
1100` (3/3 configs)
- `test_sparse_search_invalid_metric_type[COSINE]` — same error (3/3
configs)
**Root cause:** Server error message changed from `"only IP is
supported"` to `"metric type not match: invalid
parameter[expected=IP][actual=...]"`.
**Fix:** Update `err_msg` assertion to match new format.
## Test plan
- [x] `test_upsert_preserves_element_filter` — verified PASSED locally
- [x] `test_insert_without_connection` — verified PASSED locally
- [ ] Full nightly CI run to validate all 16 fixes
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: nico <cheng.yuan@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace all hard-coded `-1` literal used as manifest version with the
named constant `packed.ManifestEarliest` (value `0`) for new segment
creation, making the intent explicit and consistent across the codebase.
Key changes:
- Define `ManifestLatest` (-1) and `ManifestEarliest` (0) constants in
`internal/storagev2/packed/constant.go`
- Update all production code (segment_manager, write_buffer,
importv2/util, binlog_record_writer) to use `packed.ManifestEarliest`
- Update all test code (pack_writer_v3_test, task_test) to use
`packed.ManifestEarliest` instead of raw `-1`
- Fix `CreateManifestForSegment` FFI call: use version `0` (earliest)
instead of `-1` (latest) and use `LOON_TRANSACTION_RESOLVE_OVERWRITE`
resolve strategy
- Fix `FFIPackedWriter.Close` to use
`LOON_TRANSACTION_RESOLVE_OVERWRITE` resolve strategy
issue: #48543
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
## Summary
- Update `test_minhash_bulk_import_with_output_field_negative` assertion
to match the updated server error message
- Server error changed from `"output by function"` to `"not allowed to
provide data for function output field"`
- This test has been failing consistently across all 3 nightly configs
(kafka, pulsar-mmap, woodpecker) since build 650
## Test plan
- [x] Ran `test_minhash_bulk_import_with_output_field_negative[PARQUET]`
— PASSED
- [x] Ran `test_minhash_bulk_import_with_output_field_negative[JSON]` —
PASSED
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: nico <cheng.yuan@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## What does this PR do?
Fixes two bugs in the force promote path of
`UpdateReplicateConfiguration`.
### Bug 1: Force promote RPC hangs forever
**Root Cause**
In `ack_callback_scheduler.go`, `triggerAckCallback` handles force
promote messages by launching a goroutine to call
`doForcePromoteFixIncompleteBroadcasts`. The goroutine had its own
`defer g.Unlock()` but never called `doAckCallback`.
`doAckCallback` is the only function that invokes
`MarkAckCallbackDone()`, which closes the broadcast task's `done`
channel. `broadcastScheduler.AddTask` calls `task.BlockUntilDone()` on
this channel, so without `doAckCallback` being called,
`BlockUntilDone()` blocks forever — leaving the force promote RPC
permanently hung.
**Fix**
Chain `doAckCallback(task, g)` after
`doForcePromoteFixIncompleteBroadcasts(task)` in the same goroutine.
`doAckCallback` already handles `g.Unlock()` via its internal defer, so
the separate unlock defer in the goroutine is also removed.
```go
// Before (buggy):
go func() {
defer func() {
s.rkLockerMu.Lock()
g.Unlock()
s.rkLockerMu.Unlock()
}()
s.doForcePromoteFixIncompleteBroadcasts(task)
}()
// After (fixed):
go func() {
s.doForcePromoteFixIncompleteBroadcasts(task)
s.doAckCallback(task, g) // closes done channel, unblocks BlockUntilDone()
}()
```
### Bug 2: Missing caller-supplied config validation
**Root Cause**
In `assignment.go`, `validateForcePromoteConfiguration(config,
currentClusterID)` existed but was never called in `handleForcePromote`.
The caller-supplied config was passed through without any validation.
**Fix**
Call `validateForcePromoteConfiguration(config, currentClusterID)` after
broadcaster acquisition to validate the caller-supplied config (must
contain exactly the current cluster with no topology) before proceeding
with the promotion.
Also rename the instance method `validateForcePromoteConfiguration(ctx)`
to `buildForcePromoteConfiguration(ctx)` to avoid confusion with the
static validation function.
## Tests
Verified with E2E tests covering the full force promote lifecycle:
- Force promote on primary cluster rejected
- Force promote with invalid config rejected
- Force promote succeeds on secondary with data integrity (1000 rows
replicated + 500 new rows post-promote)
- Force promote on already-promoted cluster rejected
issue: #47352
---------
Signed-off-by: Yihao Dai <yihao.dai@zilliz.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
- Upgrade `google.golang.org/grpc` from v1.71.1 to v1.79.3
(CVE-2026-33186 **CRITICAL** - Authorization Bypass via improper HTTP/2
`:path` pseudo-header validation)
- Upgrade `github.com/quic-go/quic-go` from v0.54.0 to v0.54.1
(CVE-2025-59530 **HIGH** - DoS via premature HANDSHAKE_DONE frame)
- Updated across all Go modules (root, pkg, client, tests/go_client)
issue: #48574
## Test plan
- [ ] CI passes
- [ ] Image scan clean for CVE-2026-33186 and CVE-2025-59530
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary
- Fix flaky `TestAdaptiveRateLimitController_EnterRejectMode` in
`pkg/streaming/util/ratelimit`
- Root cause: `notify()` stored mode atomically BEFORE notifying
observers. Tests checking `getMode()` via `Eventually` could see the
mode change and call `AssertExpectations` before the backgroundLoop
goroutine finished the observer notification
- Fix: move `mode.Store()` after observer notification in `notify()`,
ensuring observers are notified before the mode change is externally
visible
## Test plan
- [x] Reproduced: 4/100 failures with `-race` on unmodified code
- [x] Verified: 0/100 failures with `-race` after fix
- [x] All `streaming/util/ratelimit` tests pass with `-race`
issue: https://github.com/milvus-io/milvus/issues/39893🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
issue: #48507
Update milvus-common to 5983a18 which adds warmup loading timeout,
runtime config hot-reload via TieredStorageConfig, and DList reservation
fixes. Wire through the new warmup_loading_timeout parameter and add
UpdateTieredStorageConfig CGO binding with config change callbacks for
runtime updates.
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
Move stats file (text_index, json_key_stats) path computation from C++
to Go. Go computes basePath based on storage version (V2: traditional
top-level directories, V3: segment basePath/_stats/) and passes it to
C++ via proto. C++ uses the basePath directly with no V2/V3 branching.
Key changes:
- Add stats_base_path to BuildIndexInfo proto (write path)
- Add base_path to LoadTextIndexInfo/LoadJsonKeyIndexInfo proto (read
path)
- Add base_path to segcorepb TextIndexStats/JsonKeyStats
(SegmentLoadInfo path)
- TextMatchIndex::Upload() returns relative paths (consistent with
JsonKeyStats)
- JsonKeyStats::Load()/TextMatchIndex::Load() require basePath
(Assert-checked)
- Extract BuildTextIndexPrefix()/BuildJsonKeyStatsPrefix() to metautil
to eliminate 5 duplicated format strings
- ConvertToSegcoreSegmentLoadInfo() resolves V3 manifest stats with
basePaths
issue: #48547
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
## Summary
- Use `flushTask.Await()` instead of blind `time.Sleep(10s)` after Flush
to ensure flush completion
- Reduce data volume: `insertBatchSize` 30K→20K, `deleteBatchSize`
10K→5K (total per phase: 150K→100K)
- Fix `teardown()` to also clean collections in the default DB,
preventing resource accumulation across test runs
issue: #48581
## Test plan
- [x] Run `TestSnapshotRestoreWithMultiSegment` against remote Milvus —
PASS
- [x] Run full `TestSnapshot*` suite — teardown successfully cleans all
collections
- [ ] CI validation
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Eric Hou <eric.hou@zilliz.com>
Co-authored-by: Eric Hou <eric.hou@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace the fixed threshold (150) for merging == or into in[] with
type-specific thresholds based on benchmark data:
- INT types: use in[] when N >= 10 (== or is faster below due to simpler
execution path)
- FLOAT types: use in[] when N >= 15 (float hash is more expensive)
- Other types (varchar, bool): use in[] when N >= 3
The rewriting is now bidirectional:
- == or → in[]: when shouldUseInExpr returns true (existing direction)
- in[] → == or: when shouldUseInExpr returns false (new direction, via
visitTermExpr)
- not in → != and: when shouldUseInExpr returns false (new, via
visitUnaryExpr)
Both directions use the same shouldUseInExpr function to ensure
consistency.
issue: https://github.com/milvus-io/milvus/issues/45525
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
## Summary
- Increase `forceMerge.dataNodeMemoryFactor` and
`forceMerge.queryNodeMemoryFactor` defaults from 3.0 to 4.0
- This reduces the max segment size produced by force merge compaction
(from 1/3 to 1/4 of node memory), preventing DataNode from producing
segments too large to index
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary
- Add `GetCollectionName()` method to 12 RESTful API request types that
were missing it
- Ensures `ProxyFunctionCall` metrics report the correct collection name
label for all RESTful endpoints
- Affected types: `RenameCollectionReq`, `QueryReqV2`,
`CollectionIDReq`, `CollectionFilterReq`, `CollectionDataReq`,
`SearchReqV2`, `HybridSearchReq`, `PartitionsReq`, `GrantV2Req`,
`IndexParamReq`, `CollectionReq`, `RunAnalyzerReq`
issue: #48461
## Test plan
- [ ] Verify existing unit tests pass
- [ ] Confirm RESTful API metrics now include collection name for
search/query/hybrid_search endpoints
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary
- Suppress gosec G204 warning in `runShellCommand` test helper in
`external_table_refresh_test.go`
- This is a test-only helper where the command is always constructed by
the test itself, so the subprocess-with-variable warning is a false
positive
- This lint error is currently blocking all PR code-check pipelines on
master
## Test plan
- [x] No functional change — test-only `nolint` annotation
issue: https://github.com/milvus-io/milvus/issues/39893🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
- Switch gRPC to shared linking (`grpc:shared=True`) to prevent
`libmilvus_core.so` and `libmilvus-common.so` from each embedding a full
gRPC runtime, which causes metric duplicate registration crash on Linux
- Set `grpc:secure=True` to exclude `grpc_unsecure` and
`grpc++_unsecure` from `CONAN_LIBS`, preventing both secure and unsecure
gRPC variants from being loaded simultaneously
issue: https://github.com/milvus-io/milvus/issues/48457
## Test plan
- [x] Built VDC image with fix, verified `libmilvus_core.so` no longer
links `grpc_unsecure`
- [x] Verified fixed image starts normally on Linux (no SIGABRT)
- [x] Verified old image without fix crashes with `Metric name
grpc.lb.pick_first.disconnections has already been registered`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
## Summary
- Implement virtual primary key (VirtualPK) for external collections
without user-defined PK
- Add ExternalSegmentCandidate for PK oracle to support segment pruning
on external segments
- Support loading external segments via manifest-based sealed segment
loader
- Add ManifestGroupTranslator with external field name fallback for
column mapping
- Add lance-table and vortex format support with HTTP scheme fix for
MinIO endpoints
- Fix extfs boolean property type mismatch by disabling extfs.* bool
properties
- Add lance and vortex format E2E tests with full query/search coverage
- Add Python data generation scripts and CI Docker Python environment
issue: #45881
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260105-external_table.md
## Test plan
- [x] Unit tests pass (pkoracle, segments, delegator, datanode/external)
- [x] E2E tests pass (13/13): TestCreateExternal*, TestRefreshExternal*,
TestExternalCollectionLanceFormat, TestExternalCollectionVortexFormat
- [x] `make milvus` compiles successfully
- [x] `make lint-fix` passes with no auto-fixes
- [ ] CI checks pass
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
## Summary
- Migrate 25 insert test cases from v1 ORM style
(`testcases/test_insert.py`) to v2 MilvusClient style
(`milvus_client/test_milvus_client_insert.py`)
- Delete migrated v1 test cases, retain
DataFrame/column-based/async-specific tests that have no v2 equivalent
- New v2 test `test_insert_with_pk_varchar_auto_id_true`: validates
varchar PK with auto_id=True, includes query verification for
auto-generated IDs
## Related Issues
#48048
## Test plan
- [x] Verified `test_insert_with_pk_varchar_auto_id_true` passes against
local Milvus (port 19531)
- [x] Cross-checked all 58 v1 cases: 25 migrated, 33 skipped
(DataFrame/column-based/v1 async), 0 remaining
There are a lot of insert test cases remain skipped, you can view
[test_insert.py Migration
Checklist](https://zilliverse.feishu.cn/docx/YAsedYP1go9qlexDyi0cgKwgnIj)
for reason.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: lyyyuna <yiyang.li@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## What this PR does
`addCollectionFieldTask.PreExecute()` was missing the
`IsClusteringKeyType()` validation that was added to
`createCollectionTask.validateClusteringKey()` in PR #48184.
This allowed users to add fields with unsupported data types (JSON,
Bool, Array, etc.) as clustering keys via the `AddCollectionField`
(schema evolution) API. During clustering compaction,
`NewScalarFieldValue()` panics on these types, crashing the DataNode.
## Root cause
PR #48184 fixed the type check only in the collection-creation path. The
schema-evolution path (`addCollectionFieldTask`) was not updated with
the same guard.
## Fix
Add `typeutil.IsClusteringKeyType()` check before the duplicate
clustering-key loop in `addCollectionFieldTask.PreExecute()`, consistent
with `createCollectionTask.validateClusteringKey()`.
```go
if t.fieldSchema.GetIsClusteringKey() {
if !typeutil.IsClusteringKeyType(t.fieldSchema.GetDataType()) {
return merr.WrapErrParameterInvalidMsg(...) // ← added
}
for _, f := range t.oldSchema.Fields { ... } // existing duplicate check
}
```
## Tests
- Added test cases for JSON, Bool, and Array fields with
`IsClusteringKey=true` — all must return `ErrParameterInvalid`
- Added positive test case: valid Int64 clustering key must be accepted
- Fixed pre-existing "more ClusteringKey field" test to use explicit
`DataType_Int64` so it exercises the duplicate-key branch (not the new
type check)
issue: #48285
---------
Signed-off-by: Yihao Dai <yihao.dai@zilliz.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
issue: #47831
GetAllStreamingNodes now returns all nodes (including frozen) for REST
API listing, while GetAvailableStreamingNodes filters frozen nodes for
scheduling use (replica observer, query node ID lookup).
Signed-off-by: chyezh <chyezh@outlook.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
SealedIndexTranslator stored a TraceContext member containing raw
traceID/spanID pointers originating from Go memory. These pointers
dangle when get_cells() is called during async warmup or cache
re-population, causing heap-buffer-overflow detected by ASAN.
Remove the TraceContext member and use an empty TraceContext in
get_cells() instead, since the original request's trace context is
meaningless for deferred index loading.
issue: #48494
Signed-off-by: chyezh <chyezh@outlook.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove legacy compatibility code that silently generated default v3
compaction params when the request from an old datacoord had empty
params. This caused datanode to produce v2 segments unexpectedly. Now
ParseParamsFromJSON returns an error on empty input, forcing the caller
to provide valid compaction params.
issue: #48570
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
When etcd watch breaks (compaction/network) and initDiscover retries,
sessions deleted during the watch gap persisted forever because
initDiscover only added entries to peerSessions without clearing old
ones. This caused the resolver to permanently report dead nodes as
alive, blocking the streaming balancer indefinitely.
Fix: reset peerSessions map before re-populating from etcd Get response.
issue: #48564
Signed-off-by: chyezh <chyezh@outlook.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
- **validate_util.go:653**: `return nil` → `return err` in
TimestamptzData branch of `FillWithDefaultValue`. All other branches
correctly return `err`; this was a copy-paste bug that silently swallows
errors when filling default values for timestamptz fields.
- **print_binlog.go:62**: `return nil` → `return err` when `mmap.Open`
fails. All other error paths in the same function correctly return
`err`.
- **delta_data.go:228**: Add `default` case in
`DeleteLog.UnmarshalJSON()` switch to return error on unsupported PK
types, preventing nil pointer panic on line 230.
- **segment_writer.go:436**: Check and log error from
`newBinlogWriter()` in `clear()` instead of ignoring with `_`. If this
fails, `w.writer` becomes nil causing panic on next use.
- **insert_node.go:48**: Fix wrong format specifier `%d` → `%v` for
error type. `%d` produces garbled output like
`%!d(*errors.errorString=...)`.
## Test plan
- [ ] Existing unit tests pass (no behavioral change for success paths)
- [ ] Each fix is a minimal one-line change with clear before/after
comparison
issue: https://github.com/milvus-io/milvus/issues/39893🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
