issue: https://github.com/milvus-io/milvus/issues/42148
This PR enable struct sub scalar fields to create nested index. The
nested means treating elements in array (we know that all fields in
array of struct are actually array) as separate documents.
The index is used by Struct to speed up Match expression as well as
Element-filter expression. This PR only enable Match expression to be
optmized by index and leave Element-filter expression in the following
PR.
Now, MatchExpr executes sub-expr by using offset inputs. To make it
support index, it also needs to support brute force without offset
inputs, so this PR:
1. enable struct scalar fields to create nested index
2. enable match expr to support brute force without offset input
3. enable match expr to support index
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
issue : https://github.com/milvus-io/milvus/issues/41746
This PR adds MinHash "DIDO" (Data In, Data Out) support to Milvus, which
allows computing MinHash signatures on-the-fly during search operations
instead of requiring pre-stored vectors.
Key changes:
- Implemented SIMD-optimized C++ MinHash computation (AVX2/AVX512 for
x86, NEON/SVE for ARM)
- Added runtime CPU detection and function hooks to automatically select
the best SIMD implementation
- Integrated MinHash computation into search pipeline (brute force
search, growing segment search)
- Added support for LSH-based MinHash search with configurable band
width and bit width parameters
- Enabled direct text-to-signature conversion during query execution,
reducing storage overhead
This enables efficient text deduplication and similarity search without
storing pre-computed MinHash vectors.
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
\kind improvement
\assign @yanliang567
- Skip duplicated partition insert tests:
- test_insert_default_partition (covered by
test_milvus_client_insert_partition)
- test_insert_partition_not_existed (covered by
test_milvus_client_insert_not_exist_partition_name)
- Fix the docstring of
test_milvus_client_insert_not_exist_partition_name to correctly describe
insert behavior with a non-existent partition.
- Strengthen assertions for partition insert tests:
- Verify total entity count after inserting into multiple partitions.
- Validate returned primary key IDs when inserting with explicit IDs.
- Fix mismatched test expectations in
`test_insert_auto_id_false_same_values` by aligning the docstring with
actual behavior.
- Add test_insert_all_datatype_collection to validate insert behavior
for collections containing all supported data types using row-based
client insert.
Signed-off-by: zilliz <jiaming.li@zilliz.com>
/kind improvement
/assign @yanliang567
**PR Summary**
- Skip duplicated client insert tests already covered by existing cases
(e.g. insert after client closed, missing vector field, auto-ID
scenarios).
- Fix test docstrings to accurately describe vector field missing and
vector data type mismatch behaviors.
- Migrate a client insert test (`test_insert_field_name_not_match`).
- Fix CaseLabel tagging to keep consistent with the original ORM-based
testcases.
Signed-off-by: zilliz <jiaming.li@zilliz.com>
issue: #46834
relate: #45993
Save valid_data file during DiskAnn index build to fix bitset size
mismatch error when searching with nullable vector fields.
---------
Signed-off-by: marcelo-cjl <marcelo.chen@zilliz.com>
issue: #44358
- Change RestoreSnapshot API to return jobID instead of collectionID,
making the async restore pattern more explicit
- Refactor RestoreData to accept snapshotName instead of snapshotData,
moving the ReadSnapshotData call inside for better encapsulation
- Remove synchronous wait loop from restoreSnapshotV2AckCallback to
avoid blocking the WAL callback
- Implement per-collection RefIndex loading state
(Pending/Loaded/Failed) replacing the global refIndexLoadDone channel
- Add background loader retry mechanism for failed RefIndex loads
- Fix recyclePendingSnapshots to keep catalog record when S3 cleanup
fails, allowing retry in next GC cycle
- Add snapshotMeta.Close() method and call it during server shutdown
- Add SnapshotRefIndexLoadInterval configuration parameter
- Update e2e tests to explicitly wait for restore completion
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: #47016
- the node id of mixcoord may roll back, so it doesn't promise monotonic
incresing.
- use the revision of session to promise it strongly.
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #45999
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
- Core invariant: DefaultValueChunkTranslator assumes a constant
per-value byte size (value_size()) for a field and therefore models
default-value storage as multiple deterministic, fixed-size cells
(num_cells(), cell_id_of(uid) == uid). A single contiguous
primary_buffer_ is built (optionally persisted to mmap) and sliced
per-cell; this invariant enables safe sharing of the same underlying
bytes across multiple Chunk views.
- Removed/simplified logic: per-field direct file_path writes and
duplicated make_chunk write paths were collapsed into a buffer-first
flow (ChunkBuffer + create_chunk_buffer + make_chunk_from_buffer). This
removes redundant per-field mmap/write code and allows multiple Chunk
instances to reuse a shared ChunkBuffer/ChunkMmapGuard instead of
performing independent writes.
- Why no data loss or regression: byte layout, alignment, padding and
per-row content are preserved because create_chunk_buffer implements the
same alignment/padding and writes identical bytes previously produced by
the direct make_chunk path; make_chunk_from_buffer constructs Chunk
views over those same bytes. DefaultValueChunkTranslator uses
deterministic math (total_rows_, num_rows_until_chunk_,
primary_cell_rows_) to slice rows—no rows are dropped, reordered, or
altered. Concrete code paths: callers that used create_group_chunk →
per-field make_chunk now use create_group_chunk → create_chunk_buffer →
make_chunk_from_buffer; DefaultValueChunkTranslator’s get_cells(),
estimated_byte_size_of_cell(), and value_size() produce the same
observable outputs as before for single-cell cases and correct per-cell
outputs for multi-cell cases.
- Enhancement / scope: adds multi-cell default-value chunks and optional
mmap-backed persistence plus shared-memory ChunkBuffer support. Changes
touch DefaultValueChunkTranslator (multi-cell behavior, value_size(),
primary_buffer_, mmap_dir_path_), ChunkWriter (ChunkBuffer,
create_chunk_buffer, make_chunk_from_buffer), and
ChunkedSegmentSealedImpl (per-field mmap decision and mmap_dir_path
propagation); comprehensive tests (mmap/non-mmap, nullable,
multi/single-cell) validate correctness.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
issue: #45881
- Add ExternalSource and ExternalSpec fields to collection schema
- Add ExternalField mapping for field schema to map external columns
- Implement ValidateExternalCollectionSchema() to enforce restrictions:
- No primary key (virtual PK generated automatically)
- No dynamic fields, partition keys, clustering keys, or auto ID
- No text match or function features
- All user fields must have external_field mapping
- Return virtual PK schema for external collections in
GetPrimaryFieldSchema()
- Skip primary key validation for external collections during creation
- Add comprehensive unit tests and integration tests
- Add design document and user guide
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: sunby <sunbingyi1992@gmail.com>
issue: #46890
related: #45993
Add null_count to binlog metadata and use it to calculate valid rows for
nullable vector fields. Skip index creation when valid rows are less
than MinSegmentNumRowsToEnableIndex.
Signed-off-by: marcelo-cjl <marcelo.chen@zilliz.com>
Related to #46716
Add range validation for collection TTL to prevent invalid values:
- TTL must be greater than or equal to -1 and at most 3155760000 seconds
(100 years)
- Remove redundant TTL validation in alterCollectionTask.PreExecute
- Add unit tests covering boundary cases
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #46988
This PR introduces an automated compaction mechanism to upgrade segments
from older storage formats (StorageV1/V2) to newer versions
(StorageV2/V3) without manual intervention.
Key changes:
- Add storageVersionUpgradePolicy to automatically identify and trigger
compaction for segments with outdated storage versions
- Implement token-bucket rate limiting (default: 3 tasks per 120
seconds) to prevent resource contention during upgrades
- Add TriggerTypeStorageVersionUpgrade to compaction trigger types
- Add StorageV3 constant for Loon manifest format support
- Add configuration parameters:
- dataCoord.compaction.storageVersion.enabled (default: true)
- dataCoord.compaction.storageVersion.rateLimitTokens (default: 3)
- dataCoord.compaction.storageVersion.rateLimitInterval (default: 120s)
The policy filters segments based on:
- Healthy and flushed state
- Not currently compacting or importing
- Not L0 level segments
- Storage version differs from target version
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
related: https://github.com/milvus-io/milvus/issues/46928
Add segment_storage_version label to track segment counts by storage
version (V1=0, V2=2). This enables monitoring segment distribution
across different storage formats.
Changes:
- Add segmentStorageVersionLabelName constant in metrics.go
- Update DataCoordNumSegments GaugeVec with new label
- Update segMetricMutation struct to track storage version dimension
- Update all WithLabelValues calls in meta.go
- Add unit tests for the new label
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
Related to #44956
Introduce StorageV3 constant to distinguish loon manifest format from
StorageV2 packed writer format. When UseLoonFFI is enabled, segments are
now marked with StorageVersion=3 instead of 2.
The V3 format currently shares most V2 logic paths since the underlying
storage format is similar. In a future refactor, V3-specific logic will
be separated from the V2 code path for better readability.
Changes:
- Add StorageV3 constant (value=3) in Go and C++ code
- Set StorageVersion=3 when UseLoonFFI is enabled for:
- Compaction params
- Import segments allocation
- Write buffer growing segments
- Streaming node segment allocation
- Add V3 support to existing V2 code paths (read/write/sync)
- Separate V2 and V3 writer creation in NewBinlogRecordWriter
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #41435
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added segment resource tracking and automatic memory/disk accounting
during inserts, deletes, loads and reopen.
* Exposed a configuration to set interim index memory expansion rate.
* Added explicit loaded-resource charge/refund operations and Bloom
filter resource lifecycle management.
* **Bug Fixes**
* Ensured consistent memory-size vs. row-count calculations across
segment operations.
* Improved resource refunding and cleanup when segments are released or
closed.
* **Tests**
* Added comprehensive resource-tracking and concurrency tests, plus
Bloom filter accounting tests.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
related: https://github.com/milvus-io/milvus/issues/46442
core changes:
- Add config (default: false) to disable /expr endpoint by default
- On Proxy nodes, require root user authentication via HTTP Basic Auth
when enabled
- On non-Proxy nodes, keep original auth parameter behavior for backward
compatibility
- Add HasRegistered() and AuthBypass to expr package for node type
detection
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
/kind improvement
/assign @yanliang567
**PR Summary**
- migrate Binary vector insert with schema dimension mismatch.
- migrate insert with None data.
- add `test_milvus_client_insert_binary_default` to cover BINARY_VECTOR,
which was not included in `test_milvus_client_insert_default` (covered
FLOAT_VECTOR, FLOAT16_VECTOR, BFLOAT16_VECTOR, and INT8_VECTOR
Signed-off-by: zilliz <jiaming.li@zilliz.com>
Add a thread pool to load BM25 stats and save them to local disk during
loading, reducing peak memory usage.
relate: https://github.com/milvus-io/milvus/issues/41424
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
- Core invariant: sealed-segment BM25 statistics are immutable and may
be persisted to disk independently from growing-segment stats; IDF state
is reconstructed by combining persisted sealed stats (preloaded from
disk) with in-memory growing stats at runtime (preloadSealed +
RegisterGrowing).
- Capability added: asynchronous BM25 loading via a dedicated
BM25LoadPool (config key common.threadCoreCoefficient.bm25Load and param
BM25LoadThreadCoreCoefficient) — delegator.loadBM25Stats is executed on
the pool to load sealed BM25 stats and call idfOracle.RegisterSealed,
reducing peak memory during segment load.
- Logic removed/simplified and why: the previous single Register(segID,
stats, state) + background local-writer/spool loop was split into
RegisterGrowing (in-memory) and RegisterSealed (sealed + on-disk
persistence) and the localloop removed; RegisterSealed writes sealed
stats directly (ToLocal) and uses singleflight to deduplicate,
eliminating redundant spooling and lifecycle complexity while clarifying
sealed vs growing flows.
- Why this does NOT introduce data loss or behavior regression: sealed
stats are still written and reloaded (RegisterSealed persists to
dirPath; preloadSealed merges disk and memory on first load), growing
segments continue to register in-memory via RegisterGrowing,
loadSegments now defers sealed BM25 handling to loadBM25Stats but still
registers sealed candidates after load, and tests were updated to
reflect RegisterSealed/RegisterGrowing usage—so
serialization/deserialization, preload semantics, and test coverage
preserve existing persisted state and runtime behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
/kind improvement
**PR Summary**
- Add auto_id parameter to test_insert_with_dataframe_values and ensure
consistent failure behavior.
- Skip duplicated auto_id list insert tests
(`test_insert_auto_id_true_list_data`,
`test_insert_auto_id_true_with_list_values`) that covered by
`test_insert_auto_id_true`.
- Fix test_insert_auto_id_true by validating generated primary keys and
their types.
- Add negative test to verify insert fails when providing primary keys
with auto_id=True.
Signed-off-by: zilliz <jiaming.li@zilliz.com>
## Summary
- Remove xfail marker from
`test_import_struct_array_with_local_bulk_writer` as the related
pymilvus issue is fixed
- Add test coverage for empty struct_array in bulk writer tests
## Changes
1. Remove `@pytest.mark.xfail(reason="issue:
https://github.com/milvus-io/pymilvus/issues/3050")` marker
2. Add 10% probability to generate empty `struct_array` (`[]`) in test
data
3. Add logging for empty array count to track test coverage
## Related PRs
- pymilvus fix: https://github.com/milvus-io/pymilvus/pull/3182
## Test Plan
- [x] Tested with Milvus 2.6.8 and pymilvus fix
- [x] Both PARQUET and JSON formats pass
- [x] Empty struct arrays are correctly written, imported, and queried
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
Related to #46617
Bump milvus-storage version from 839a8e5 to 2ab2904, which introduces
subtree filesystem support. This allows removing manual bucket name
concatenation logic across the codebase as the storage layer now handles
path prefixing internally.
Changes:
- Remove bucket name prefix logic in datanode, querynode, and storage
layers
- Simplify FileManager::GetRemoteIndexFilePrefixV2()
- Rename CColumnGroups API to CColumnSplits to align with upstream
- Update DiskFileManagerTest paths for new directory structure
- Add FFI packed reader/writer unit tests
Co-authored-by: Wei Liu <wei.liu@zilliz.com>
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: Wei Liu <wei.liu@zilliz.com>
issue: #41435
- fix: avoid double destruction with placement new
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Pull Request Summary
**Bug Fix: milvus-common dependency update to address double destruction
with placement new**
- **Root Cause Resolution:** Updates the milvus-common library
dependency from commit 4d7781d to fbe5cf7 to fix a double destruction
issue that occurs when placement new is used for object construction.
The upstream fix ensures proper lifecycle management of objects
constructed with placement new semantics, preventing accidental
re-destruction of objects allocated in pre-allocated memory regions.
- **No Logic Changes in Milvus Core:** This PR contains only a
CMakeLists.txt version bump in
`internal/core/thirdparty/milvus-common/CMakeLists.txt`; no Milvus
codebase logic is modified, removed, or simplified. The fix is entirely
contained within the milvus-common library dependency fetched during the
build process.
- **Data Integrity and Behavior Preservation:** No behavior regression
or data loss is introduced. The change is a pure dependency update to
pull in an upstream bug fix. All memory management and object lifecycle
handling improvements are confined to the milvus-common library,
remaining transparent to Milvus core operations.
- **Issue Resolution:** Addresses issue #41435 by integrating the
corrected milvus-common version that prevents double destruction bugs
occurring in code paths that use placement new for memory-efficient
object construction.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
Related to #44257
Enable the splitByAvgSize policy by default for storage v2, which
optimizes data layout by splitting columns based on average size rather
than fixed boundaries.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #46779
related: #45993
Return clear error when using is null/is not null filter on vector
fields
Return clear error when search by IDs with all null vectors
Fix nq mismatch when search by IDs with mixed null/valid vectors
Signed-off-by: marcelo-cjl <marcelo.chen@zilliz.com>
/kind improvement
The test was flaky because sort compaction may not complete within the
sleep time. Instead of relying on sleep, wait for index building to
complete after flush, which ensures sort compaction is done before
getting binlog files.
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
