mirror of
https://github.com/milvus-io/milvus.git
synced 2026-07-21 18:25:44 +00:00
master
305
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 zhuwenxingandGitHub
|
743da442b1 |
test: add XGBoost FunctionChain coverage (#51545)
## What changed - Add Python MilvusClient coverage for XGBoost `FunctionChainStage.L0_RERANK`. - Validate realistic `2 x 3000` indexed sealed segments plus `1 x 3000` growing segment data topology. - Cover FLAT, HNSW, and DISKANN indexes; filtered, range, group-by, iterator, and hybrid-search behavior. - Verify score and field correctness across numeric/null/binary features, delete, and upsert mutations. - Cover invalid stage/model contracts, concurrency, scale, cache identity, restart/failover, rolling update, and chaos scenarios. - Classify the suite into L0/L1/L2/L3 and replace the iterator xfail with its code `1100` error contract. Related issues: #51306, #51310, #51332 ## Validation - `ruff format --check tests/python_client/milvus_client/test_milvus_client_function_chain.py` - `ruff check tests/python_client/milvus_client/test_milvus_client_function_chain.py` - `python -m py_compile tests/python_client/milvus_client/test_milvus_client_function_chain.py` - `pytest --collect-only`: 37 cases collected - L0: 1 passed - L1: 8 passed - L2: 4 passed - Iterator regression: 1 passed against master image `0671607` An initial combined serial run observed a DISKANN load timeout with a loaded segment reporting `index_id=0`; the unchanged isolated L2 rerun passed. The strict assertion remains in place. --------- Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com> |
||
|
|
82bb4054dd |
test: Add REST query order by coverage (#51460)
## What this PR does
- Adds REST v2 query coverage for orderByFields across ascending,
descending, multi-field, filtering, pagination, nullable fields, and
invalid parameters.
- Uses class-scoped REST clients and shared collection setup consistent
with the existing REST v2 test pattern.
- Verifies that omitting limit applies the REST default of 100 rows.
- Covers duplicate-key pagination behavior and exact validation error
code 1100.
## Test
Against schema-evolution-v3-latest through port-forward:
cd tests/restful_client_v2
../python_client/.venv/bin/python -m pytest
testcases/test_query_order_by.py -v --tb=short -p no:rerunfailures
--endpoint http://127.0.0.1:19530 --token root:Milvus
Result: 16 passed, 16 warnings in 40.87s.
---------
Signed-off-by: lyyyuna <yiyang.li@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
|
||
|
zhuwenxingandGitHub
|
977a27fb47 |
test: add struct array null semantics and lifecycle coverage (#51524)
issue: #51381 related: #51414 related: #51416 ### What this PR does - Adds Python client E2E coverage for nullable Struct Array NULL-versus-empty semantics. - Covers compaction, snapshot restoration, partial updates, drop and re-add isolation, imports, regex indexes, aggregation, and dynamic fields. - Adds exact ANN oracles for Struct Array element and embedding-list searches without assuming self-hit behavior. - Covers hybrid search collapse strategies and element identity preservation. - Marks known regressions with strict xfail markers linked to their corresponding issues while keeping unaffected parameters active. ### Why Struct Array rows can represent NULL, empty arrays, and non-empty arrays. These states affect predicates, nested indexes, element offsets, and vector search candidates differently. The added coverage protects these semantics across write, reload, compaction, import, and snapshot lifecycle paths. ### Validation - `ruff check` passed for all changed Python files. - `ruff format --check` passed for all changed Python files. - Python compilation passed for all changed Python files. - Pytest collection passed. - The nine known regression nodes executed as expected: `9 xfailed`. - The three unaffected parameterized cases passed. - Targeted E2E validation used Milvus master build `09c44f2dbb`. --------- Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com> |
||
|
Spade AandGitHub
|
149903e0fd |
fix: fix three-values nullbility and race in growing segmentArrayOffets (#51415)
issue: https://github.com/milvus-io/milvus/issues/51381 --------- Signed-off-by: SpadeA <tangchenjie1210@gmail.com> Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com> |
||
|
|
5def5ced4a |
enhance: enforce function-field binding principle for function DDL (#51360)
## What Enforce a function–field binding principle for function DDL (add / drop / alter function on an existing collection), so a function and its output field stay coupled and already-stored vectors can never be silently invalidated by DDL. BM25 and MinHash follow the strict binding: add/drop only via the unified `add_function_field` / `drop_function_field`, output field always coupled. **TextEmbedding is a temporary carve-out** — its legacy `AddCollectionFunction` / `DropCollectionFunction` paths are retained, so attach-over-existing-field and detach are still possible for it, because `add_function_field` embedding backfill is not yet implemented. It will be folded into the unified path in a follow-up. So the "always coupled" invariant currently holds for BM25/MinHash, not TextEmbedding. ## Changes - **AlterCollectionSchema invariant**: reject standalone add-function (a function must be added together with its new output field), reject detaching a function without dropping its output field, and reject function cascade (a new function's input being another function's output). - **Legacy RPCs**: `AddCollectionFunction` / `DropCollectionFunction` are rejected for BM25/MinHash (must use `add_function_field` / `drop_function_field`) and retained only for TextEmbedding (see carve-out). `AlterCollectionFunction` is kept and gains the whitelist below. In the alter and legacy-add paths, function field IDs are always re-derived from field names (never trusted from the request), so a request cannot inject an unrelated field ID that a later `drop_function_field` would delete. - **alter_function whitelist**: only connection/runtime params may change (TextEmbedding: `url`, `credential`, `timeout_ms`, `max_client_batch_size`, `region`, `location`, `projectid`, `user`); BM25/MinHash have no alterable params. Function identity (type, name, input/output fields) and output-shaping params (`dim`, `model_name`, `endpoint` — the TEI model identity, `normalize`, `truncate*`, prompts, ...) are immutable. Params are normalized (keys lowercased, duplicate keys rejected) so a crafted duplicate key cannot bypass the diff. - **drop_function_field**: allow any output-producing function (dropping needs no backfill), removing the previous BM25/MinHash-only restriction. ## Not in scope / follow-up - Extending `add_function_field` to TextEmbedding (post-creation add would require backfilling every existing row through the external embedding model) — deferred; folding the TextEmbedding legacy paths into the unified binding follows this. - MinHash input-field analyzer immutability lives on a different DDL path (`AlterCollectionField`); tracked as a separate follow-up. ## Breaking changes - `DropCollectionFunction` on a **MinHash** function (detach — remove the function but keep its output field) is no longer supported; use `drop_function_field` instead (drops the function together with its output field). If a released version supported MinHash detach, migrate any such call to `drop_function_field`. issue: #51348 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: MrPresent-Han <chun.han@gmail.com> Co-authored-by: MrPresent-Han <chun.han@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
224f4ab7e9 |
fix: use correct MinIO root path in text LOB layout test (#51535)
## What this PR does - Change the default `MILVUS_MINIO_ROOT_PATH` from `files` to `file` in the Text LOB inline/object layout test. - Keep the environment-variable override available for deployments using a custom root path. issue: #51257 ## Verification - `python3 -m py_compile tests/python_client/milvus_client/test_milvus_client_text_lob.py` - Targeted pytest node collection completed successfully. - Verified the committed diff contains only the requested one-line default change. Signed-off-by: Eric Hou <eric.hou@zilliz.com> Co-authored-by: Eric Hou <eric.hou@zilliz.com> |
||
|
zhuwenxingandGitHub
|
0fe1a5ba6f |
test: improve force merge target size coverage (#51350)
issue: #51248 ## What changed - correct the omitted-`target_size` case to assert ordinary manual compaction semantics - add a physical Force Merge target-size case using persistent segment IDs and MinIO insert-log sizes - add deterministic Go boundary coverage for grouping selection at the configured threshold and threshold + 1 - add target-size memory clamp coverage for cluster and standalone co-location modes - extend Optimize format, signed-int64 boundary, loaded-segment refresh, and live async lifecycle coverage - add the async Optimize wrapper and update the PyMilvus test dependency to 3.1.0rc64 ## Why The previous cases did not prove that an explicit Force Merge target affected physical output, assumed the wrong grouping threshold, and depended on manual log inspection for algorithm selection. Loaded refresh and the public async Optimize workflow also lacked live end-to-end coverage. The async case exposed milvus-io/pymilvus#3680 on PyMilvus 3.1.0rc62. Version 3.1.0rc64 contains the tuple-unpacking fix and passes the unmodified test. ## Validation - all 10 planned case IDs pass; 0 failed and 0 blocked - `milvus-dev-cli` Go UT job `go-ut-local-zhuwenxi-zhuwenxing-i-4082040-30413568` succeeded; both test functions and all five subtests passed - physical target-size L3 case passed against a real cluster and MinIO in 200.68s - loaded refresh, target format/boundary, and manual compaction cases passed against the same server version - async Optimize L3 passed unmodified with PyMilvus 3.1.0rc64 in 56.91s - 56 Python nodes collect successfully; Python compilation, Ruff, and `git diff --check` pass --------- Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com> |
||
|
|
c322c84c22 |
test: cover add-field default values on growing segments (#51303)
## What this PR does - Remove the `xfail` marker from the existing drop-and-readd analyzer-field regression for #50484. - Add L1 E2E coverage for the live QueryNode growing-segment reopen path when an `enable_match` VARCHAR field with `default_value` is added to loaded growing data. - Verify the default value is immediately searchable on pre-existing unflushed rows, eventually searchable on a subsequent defaulted insert after the growing text index catches up, and remains searchable after flush/reload. Related fix: #51201 issue: #50484 ## Scope note This PR validates the live reopen path: ```text loaded collection -> insert rows that stay in a QueryNode growing segment -> add an `enable_match` VARCHAR field with `default_value` -> query triggers LazyCheckSchema/Reopen -> default values for pre-existing growing rows are indexed and searchable immediately ``` It does not cover the QueryNode recovery / LoadGrowing path. That path requires forcing a QueryNode restart, replacement, or channel rewatch while the data is still growing: the new QueryNode can create the segment with the latest schema and then load old binlogs written before AddField. Since the segment schema is already current in that path, LazyCheckSchema/Reopen is not the mechanism being tested here. Recovery coverage should be handled by a focused stability test. ## Test results - `test_milvus_client_add_match_field_with_default_value_on_growing_data`: 3/3 passed - `test_drop_then_add_same_name_analyzer_field`: passed - Ruff lint and format checks: passed - `git diff --check`: passed Signed-off-by: lyyyuna <yiyang.li@zilliz.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
zhuwenxingandGitHub
|
cdd9136fd7 |
test: enable FileResource regression cases for closed issues (#51239)
## What changed - Remove the stale `xfail` marker from `test_file_with_bom_and_crlf`, which covers the UTF-8 BOM regression tracked by [#49684](https://github.com/milvus-io/milvus/issues/49684) and fixed by [#49691](https://github.com/milvus-io/milvus/pull/49691). - Remove the stale `skip` marker from `test_drop_collection_then_remove_remote_resources_no_panic`, which covers the FileResource lifecycle panic tracked by [#49279](https://github.com/milvus-io/milvus/issues/49279) and fixed by [#49412](https://github.com/milvus-io/milvus/pull/49412). Both issues are closed. This change makes the existing cases active CI regressions; it does not change their fixtures, test bodies, assertions, or server behavior. ## User impact CI will now detect regressions where stop-word files with a UTF-8 BOM are parsed incorrectly, or where dropping a collection and removing its remote analyzer resources can panic the server. ## Validation The existing cases were previously run against the latest `upstream/master` commit `03762320e8537f855c18c28133834bf292b45361` using image `harbor.milvus.io/manta/milvus:master-20260710-0376232`. The `-p no:skipping` option only bypassed pytest's skip/xfail marker handling; no case code, fixture, or assertion was changed: ```bash python -W ignore -m pytest -v -p no:skipping \ milvus_client/test_milvus_client_file_resource.py::TestMilvusClientFileResourceContent::test_file_with_bom_and_crlf \ milvus_client/test_milvus_client_file_resource.py::TestMilvusClientFileResourceLifecycleAdvanced::test_drop_collection_then_remove_remote_resources_no_panic \ --host <temporary-milvus-host> --port 19530 --token 'root:Milvus' \ --minio_host <temporary-minio-host> --minio_bucket fr-closed-issues-master \ --tb=short ``` Result: `2 passed in 2.16s`. The #49279 case was then repeated ten times with the same marker bypass: ```bash python -W ignore -m pytest -q -p no:skipping \ milvus_client/test_milvus_client_file_resource.py::TestMilvusClientFileResourceLifecycleAdvanced::test_drop_collection_then_remove_remote_resources_no_panic \ --count=10 \ --host <temporary-milvus-host> --port 19530 --token 'root:Milvus' \ --minio_host <temporary-minio-host> --minio_bucket fr-closed-issues-master \ --tb=short ``` Result: `10 passed in 14.59s`. Milvus remained healthy, pod restart count was zero, and logs contained no panic, `tokenizer.h:31`, fatal error, or segmentation fault. The temporary validation instance has been cleaned up. The two node IDs also collect successfully after removing the markers: ```bash python -W ignore -m pytest --collect-only -q \ milvus_client/test_milvus_client_file_resource.py::TestMilvusClientFileResourceContent::test_file_with_bom_and_crlf \ milvus_client/test_milvus_client_file_resource.py::TestMilvusClientFileResourceLifecycleAdvanced::test_drop_collection_then_remove_remote_resources_no_panic ``` Result: `2 tests collected in 0.09s`. Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com> |
||
|
zhuwenxingandGitHub
|
408204099e |
test: add struct array partial update coverage (#51166)
## What - Enable StructArray coverage in `PartialUpdateChecker` and verify StructArray-only partial update rows. - Add MilvusClient E2E coverage for StructArray partial update over sealed and growing data, including HNSW and DISKANN embedding-list search validation. - Bump Python client test dependency to `pymilvus==3.1.0rc62`. ## Verification - `python3 -m py_compile tests/python_client/chaos/checker.py tests/python_client/milvus_client/test_milvus_client_partial_update.py` - `ruff check tests/python_client/chaos/checker.py` - `ruff check --select E9,F63,F7,F82 tests/python_client/milvus_client/test_milvus_client_partial_update.py` - `git diff --check` - Targeted live run against 2.6-latest: `test_milvus_client_partial_update_struct_array_sealed_growing` passed for HNSW and DISKANN. --------- Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com> |
||
|
wei liuandGitHub
|
536ece6737 |
fix: Check external fields against loaded manifest (#50773)
issue: https://github.com/milvus-io/milvus/issues/50416 ## What This PR moves external field checks into the C++ Search/Retrieve path after `LazyCheckSchema` has refreshed the segment schema and load info. Instead of probing `HasFieldData` or `FieldAccessible`, the guard checks whether every referenced external field exists in the currently loaded external manifest: - `CheckExternalFieldsInLoadedSnapshot` is renamed to `CheckExternalFieldsInLoadedManifest` to match the real readiness boundary. - Plan `access_entries_` now includes expression/execution fields and requested output fields, so Search and Retrieve use one field-reference list for external manifest checks. - Search filter-only skips the vector field and output target fields because that path does not access them. - Non-manifest segments keep the existing behavior. The loaded manifest column-group cache is also preserved across schema-only reopen when the manifest path is unchanged, so the query path can check manifest membership without parsing remote manifests or falsely reporting missing columns after schema refresh. ## Why For external collections, a newly added field can appear in the Go collection schema before the loaded C++ segment has refreshed to a manifest that contains the corresponding external column. If Search/Retrieve touches that field against the stale loaded manifest, segcore can surface an internal field-state failure instead of a clear external-field-not-ready error. The real readiness condition here is whether the loaded manifest contains the external column, not whether the field is loaded into memory. ## Validation - `ninja -C cmake_build milvus_core all_tests` - `DYLD_LIBRARY_PATH=cmake_build/src cmake_build/unittest/all_tests --gtest_filter='PlanProto.SearchPlanCollectsFieldAccessInfo:PlanProto.RetrievePlanCollectsFieldAccessInfo:SegmentLoadInfoTest.HasManifestColumnUsesManifestColumnNames:SegmentLoadInfoTest.CachedManifestColumnsCanBeInherited'` - `/opt/homebrew/opt/llvm@15/bin/clang-format --dry-run -Werror internal/core/src/query/PlanImpl.h internal/core/src/query/PlanProto.cpp internal/core/src/query/PlanProtoTest.cpp internal/core/src/segcore/ChunkedSegmentSealedImpl.cpp internal/core/src/segcore/ChunkedSegmentSealedImpl.h internal/core/src/segcore/SegmentInterface.h internal/core/src/segcore/SegmentLoadInfo.cpp internal/core/src/segcore/SegmentLoadInfo.h internal/core/src/segcore/SegmentLoadInfoTest.cpp internal/core/src/segcore/segment_c.cpp` - `CLANG_FORMAT=/opt/homebrew/opt/llvm@15/bin/clang-format make cppcheck` - `git diff --check milvus/master...HEAD` --------- Signed-off-by: Wei Liu <wei.liu@zilliz.com> |
||
|
yanliang567andGitHub
|
b542464f30 |
test: Add JSON filtering e2e coverage (#51171)
## What does this PR do? Adds Python e2e coverage for JSON filtering UNKNOWN semantics in `tests/python_client/milvus_client/expressions/test_milvus_client_json_filtering.py`. This PR covers: - raw filtering paths on growing and sealed segments - JSON arithmetic/filtering UNKNOWN behavior for missing, null, and type-mismatch values - JSON array predicate behavior for missing and non-array values - ARRAY subscript UNKNOWN behavior for null, empty, and out-of-range values - contradiction rewrite cases, including outer `not (...)`, to ensure UNKNOWN is not rewritten into TRUE - JSON path index filtering with `json_cast_type=DOUBLE` - JSON path index filtering with `json_cast_type=VARCHAR` - JSON flat index coverage for currently passing UNKNOWN cases - query/search filter consistency for representative JSON filters, including indexed collections - L2 coverage for boolean JSON values, fractional double values, additional comparison/logical operators, JSON path array subscripts, mixed arrays, and `ARRAY_DOUBLE` JSON path index The known JSON flat index array-vs-scalar comparison mismatch is documented in #51193 and covered by a skipped regression case so the coverage can land first without taking server-side fixes in this PR. issue: #50976 issue: #51193 Related #50699 #50698 ## Test Plan - [x] `git diff --check` - result: passed - [x] `uvx ruff check --config tests/ruff.toml tests/python_client/milvus_client/expressions/test_milvus_client_json_filtering.py` - result: passed - [x] `uvx ruff format --check --config tests/ruff.toml tests/python_client/milvus_client/expressions/test_milvus_client_json_filtering.py` - result: passed - [x] `/Users/yanliang.qiao/fork/milvus/tests/.venv/bin/python -W ignore -m pytest milvus_client/expressions/test_milvus_client_json_filtering.py --collect-only -q` - result: `29 tests collected` - [x] `milvus-dev-cli deploy create harbor.milvus.io/manta/milvus:master-20260709-9a0d7f2 --name json-filtering-l2-9a0d7f2 --namespace chaos-testing --no-auth --wait-timeout 1800 --wait-interval 10` - result: Healthy - endpoint: `lb-6d57ndgl-sw4giqjhodl3bn15.clb.bj-tencentclb.net:19530` - [x] `/Users/yanliang.qiao/fork/milvus/tests/.venv/bin/python -W ignore -m pytest milvus_client/expressions/test_milvus_client_json_filtering.py --uri http://lb-6d57ndgl-sw4giqjhodl3bn15.clb.bj-tencentclb.net:19530 --token "" -s -v --tb=short` - result: `28 passed, 1 skipped in 215.82s` ## Notes The skipped case is `test_json_flat_index_array_scalar_comparison_unknown_semantics_query`, linked to #51193. The skip can be removed after the server-side JSON flat index UNKNOWN semantics issue is fixed. Signed-off-by: Yanliang Qiao <yanliang.qiao@zilliz.com> |
||
|
|
aea5d701bd |
enhance: bind index meta to add function field DDL (#51105)
issue: #51104 companion SDK PR (merge first): milvus-io/pymilvus#3670 ## Summary `add_function_field` previously accepted a vector output field without any index meta; bump-schema-version compaction then stalled forever (no segment index task can be created for a field with no index defined, and `GetQueryVChanPositions` never hands off unindexed compacted-to segments — the fail-closed chain is silent). This PR makes the index meta a mandatory, atomically-bound part of the DDL: - **Mandatory at every layer**: SDK requires `index_params`; proxy and rootcoord reject vector function-output fields without an explicit `index_type` strictly before the broadcast (no silent AUTOINDEX). - **create_index-aligned materialization**: proxy normalizes the params with the same logic as `create_index` (name rules via `validateIndexName`, checker existence, `CheckTrain` incl. dimension filling, function-type defaults such as `bm25_k1/bm25_b/bm25_avgdl`) and writes them back into the request; rootcoord prepare allocates index id/name, rejects name conflicts and unknown index types, and serializes a complete `indexpb.FieldIndex` into the new `AlterCollectionMessageUpdates.bound_field_indexes` WAL field. - **Atomic apply**: the alter-collection ack callback commits the schema and then applies the bound index by dispatching a synthetic `CreateIndexMessage` to datacoord's existing `createIndexV2AckCallback` (same pattern as `cascadeDropFieldIndexesInline`). The callback is replayed until success across crashes and is fully idempotent (all ids come from the message body), so `DDL success ⇒ index meta exists` holds with no partial terminal state. - **Reuse promotion**: `ValidateIndexParams`/`CheckDuplidateKey` moved from datacoord to `internal/util/indexparamcheck`; shared `ExpandIndexParams` / `FillFunctionOutputIndexParams` / `PrepareFunctionOutputIndexParams` helpers now back both the create_index path and this DDL. - **Dead fan-out removed**: the ordinary `CreateIndex` broadcast is reverted to control-channel-only; the vchannel fan-out had no consumer and wrote one dead WAL entry per vchannel per index creation. - **No milvus-proto change**: the request already carried `FieldInfo.index_name`/`extra_params` (previously unused); this feature activates them. - e2e suites updated to the new semantics: `test_add_function_field_feature.py` rewritten (explicit `index_params` everywhere, post-DDL `create_index` blocks removed since the bound index conflicts with a second distinct index per existing semantics, new negative cases for the mandatory checks); the external-table negative case updated as well. ## Verification - New/updated unit tests: rootcoord DDL callbacks (mandatory/AUTOINDEX/unknown-index-type rejections, bound-index materialization + ack-callback application via a recording fake), proxy task validation (invalid index name, normalization write-back), shared helper tests. - End-to-end on a live cluster (StorageV3 + bump compaction enabled): request without params rejected; index meta visible with create_index-aligned params immediately after the DDL returns; bump compaction rewrote 2000-row sealed segments and the bound index built on the results (the exact chain that previously stalled); BM25 search over pre-DDL rows succeeds once the compacted segments are loaded. ## Limitations (follow-ups) - Live propagation of the new index meta to already-loaded QueryNode delegators is intentionally out of scope; a loaded collection observes the bound index through the segment load path (e.g. reload / compacted-segment load). Tracked as a separate follow-up PR. - The python e2e suites require the companion pymilvus change: milvus-io/pymilvus#3670 is merged and `tests/python_client/requirements.txt` is bumped to `pymilvus==3.1.0rc61` in this PR (all 22 rewritten cases + the external-table case verified locally against the released rc61). - During a rolling-upgrade window an old coordinator replaying the new message ignores `bound_field_indexes` (proto3 unknown field), i.e. pre-existing behavior; avoid the new argument in mixed-version clusters. - `AddCollectionFunction` on an existing field and plain add-vector-field keep their current behavior (the WAL carrier is generic for future extension). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: MrPresent-Han <chun.han@gmail.com> Co-authored-by: MrPresent-Han <chun.han@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
yanliang567andGitHub
|
e0857f1bd9 |
fix: Validate REST quick-create enum fields (#51088)
## What
Validate REST v2 quick-create enum-like fields before creating the
collection.
This PR fixes `vectorFieldType` handling for quick collection creation:
- rejects invalid `vectorFieldType` values instead of silently using
`FloatVector`
- supports `FloatVector`, `BinaryVector`, `Float16Vector`,
`BFloat16Vector`, and `SparseFloatVector`
- defaults quick-create index metric by vector type: dense `COSINE`,
binary `HAMMING`, sparse `IP`
- validates explicit binary/sparse `metricType` before collection
creation
- rejects `SparseFloatVector` quick-create with `dimension`
It also fixes top-level quick-create `consistencyLevel` handling:
- parses and validates top-level `consistencyLevel`
- preserves existing `params.consistencyLevel` behavior
- rejects conflicting top-level and `params.consistencyLevel` values
Fixes #51085
Fixes #51084
## Why
Previously, `/v2/vectordb/collections/create` ignored top-level
`vectorFieldType` and always generated a `FloatVector` schema in
quick-create mode. A typo such as `InvalidVectorType` could return
success and mask client-side bugs.
Binary and sparse quick-create requests without `metricType` could also
create the collection first and then fail index creation because the old
fallback metric was `COSINE`, which is incompatible with those vector
types.
For #51084, top-level `consistencyLevel` was not present in
`CollectionReq`, so JSON decoding silently dropped it. Invalid values
such as `"consistencyLevel": "Invalid"` were ignored and the collection
could be created with the default consistency level.
## Tests
- `./bin/gofumpt -w internal/distributed/proxy/httpserver/request_v2.go
internal/distributed/proxy/httpserver/handler_v2.go
internal/distributed/proxy/httpserver/handler_v2_test.go`
- `./bin/gci write internal/distributed/proxy/httpserver/request_v2.go
internal/distributed/proxy/httpserver/handler_v2.go
internal/distributed/proxy/httpserver/handler_v2_test.go
--skip-generated -s standard -s default -s
"prefix(github.com/milvus-io)" --custom-order`
- `git diff --check`
- `python3 -m py_compile
tests/restful_client_v2/testcases/test_collection_operations.py`
- `rg -n
"string\\(paramtable\\.(BinaryVectorDefaultMetricType|SparseFloatVectorDefaultMetricType)\\)|interface\\("
internal/distributed/proxy/httpserver/handler_v2.go
internal/distributed/proxy/httpserver/handler_v2_test.go`
Attempted locally but blocked by missing native dependencies:
```
go test ./internal/distributed/proxy/httpserver -run 'TestCreateCollection(QuickVectorFieldType|TopLevelConsistencyLevel)$' -count=1
# missing rocksdb.pc and milvus_core.pc
```
CI investigation before this push:
```
# build-ut-cov/code-check failed on c6 predecessor because of unconvert:
# internal/distributed/proxy/httpserver/handler_v2.go:2054:16 unnecessary conversion
# internal/distributed/proxy/httpserver/handler_v2.go:2056:16 unnecessary conversion
# internal/distributed/proxy/httpserver/handler_v2_test.go:2185:28 unnecessary conversion
# internal/distributed/proxy/httpserver/handler_v2_test.go:2217:28 unnecessary conversion
# Fixed by removing string(...) around paramtable default metric constants.
# e2e-default failed in job 40810 on:
# testcases/test_geometry_operations.py::TestGeometryCollection::test_spatial_query_and_search[False-True-sealed-ST_CROSSES]
# error: Connection refused to /v2/vectordb/collections/flush on port 19530
# the same job showed querynode/streamingnode restarts and proxy ContainerCreating during cleanup.
# REST collection tests in that job had already passed, so this looked unrelated to this PR.
# e2e-default failed again in dispatcher 13914 / job-1 32439 on:
# testcases/test_collection_operations.py::TestCreateCollectionNegative::test_create_collection_quick_setup_with_invalid_consistency_level
# root cause: the test used collection_create(), whose wrapper auto-injects params.consistencyLevel=Strong.
# That made the request hit the new conflict branch instead of the intended top-level invalid consistencyLevel branch.
# Fixed in
|
||
|
Buqian ZhengandGitHub
|
933ce30344 |
fix: propagate unknown for missing nested values (#50979)
## Summary issue: #50976 This PR fixes NULL / UNKNOWN propagation gaps in filter expressions for missing nested values: - Treat missing JSON arithmetic operands and JSON `array_length` extraction failures as UNKNOWN instead of definite booleans. - Preserve UNKNOWN during parser/RBO contradiction folds for nested JSON paths, array subscripts, and struct-array subscripts. - Treat missing array / struct-array elements as UNKNOWN during execution. - Propagate JSON path/cast index typed-known masks so missing paths and cast failures are not matched by `!=`, `not in`, or outer `not(...)`. - Propagate JSON flat index comparable-value masks for comparison predicates. The commits are split by issue area: - `4f15d6fe79` `fix: treat missing json arithmetic operands as unknown` - `69d6e95917` `fix: preserve unknown for missing nested predicates in rbo` - `51b58edf05` `fix: treat missing array elements as unknown` - `d9dd4c9334` `fix: honor json path index unknown values` - `a9d813b322` `fix: honor json flat index unknown values` ## Verification - `LOCALSTORAGE_PATH=/tmp/milvus-test-localstorage go test -buildvcs=false -count=1 ./internal/parser/planparserv2/rewriter` - `ninja -C /tmp/milvus-50976-build all_tests -j32` - focused C++ regressions, including `JsonFlatIndexTest.*:JsonFlatIndexExprTest.*:JsonIndexTest.*:JsonPathIndexTest.*` (49 tests) - focused array regressions including `Expr.TestArraySubscriptMissingElementIsUnknown` and `Expr.TestArray*` - `git diff --check origin/master...HEAD` ## Note While adding extra JsonFlatIndex null-expression coverage, a separate pre-existing `PhyNullExpr::DetermineExecPath()` / `std::call_once` deadlock was found. That hanging test is not included here; this PR keeps JsonFlatIndex coverage scoped to comparison UNKNOWN behavior. --------- Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com> |
||
|
|
b1bbaafc88 |
fix: skip index raw data for array output (#51109)
issue: https://github.com/milvus-io/milvus/issues/51033 ref: https://github.com/milvus-io/milvus/issues/42148 --------- Signed-off-by: SpadeA <tangchenjie1210@gmail.com> Co-authored-by: zhuwenxing <wxzhuyeah@gmail.com> |
||
|
nicoandGitHub
|
c701920a32 |
test: fix struct array nullable field error expectation (#51073)
This PR updates the struct array invalid-case test expectation to match the current error message for nullable sub-fields inside a non-nullable struct. The affected parametrized cases are: - test_struct_array_with_nullable_field[clip_embedding1] - test_struct_array_with_nullable_field[scalar_field] This is a test-only change. --------- Signed-off-by: nico <cheng.yuan@zilliz.com> |
||
|
|
4fad6e9dbb |
test: Add TEXT LOB Python client coverage (#50563)
## Summary - Add Python client TEXT LOB coverage for read/write, lifecycle, text_match, BM25, compaction, and storage layout checks - Add bulk import coverage for TEXT LOB payloads across JSON, CSV, and Parquet via RemoteBulkWriter - Include a small analyzer-token cache and reduced payload setup to keep the suite runtime lower issue: #50562 ## Test Plan - [x] python3 -m py_compile tests/python_client/milvus_client/test_milvus_client_text_lob.py tests/python_client/bulk_insert/test_bulk_insert_api.py - [x] python3 -m pytest -o addopts='' --collect-only -q tests/python_client/milvus_client/test_milvus_client_text_lob.py tests/python_client/bulk_insert/test_bulk_insert_api.py::TestBulkInsert::test_text_lob_bulk_import_json_csv_parquet - [x] Full TEXT LOB Python client file run reported at 95s with 6 workers --------- Signed-off-by: Eric Hou <eric.hou@zilliz.com> Co-authored-by: Eric Hou <eric.hou@zilliz.com> |
||
|
|
590a14fdf7 |
fix: reject bare NULL literal in expressions instead of misparsing it as a field (#50889)
issue: #50882 ## Problem `NULL` is not a grammar token in the plan parser, so the lexer treats a bare `NULL`/`null` as an ordinary identifier. When it appears where a value is expected — e.g. inside an `in [...]` value list: ``` id in [6560, NULL, 6722, -7856, -6757] ``` `VisitIdentifier` runs a field lookup on `NULL`: - **without a dynamic field** it fails with the misleading `field NULL not exist`, which misleads users into thinking they must add a column named `NULL` to their schema; - **with a dynamic field** it is silently mistaken for a JSON key and accepted. This affects both `delete()` and `query()` (they share the same `ParseExpr` path). ## Fix Treat bare `null`/`NULL` (case-insensitive) as a reserved word in `VisitIdentifier` and reject it up-front with an actionable message, regardless of whether a dynamic field exists: ``` NULL literal is not supported in expressions; use '<field> is null' or '<field> is not null' instead ``` This implements option 1 (robust rejection) from the issue. `<field> is null` / `is not null` are unaffected (they parse via dedicated tokens, not `VisitIdentifier`). **The guard is schema-aware for backward compatibility** (review feedback): `null` only becomes a create-time keyword in this PR, so a legacy collection may own a field literally named `null`, and the bare identifier is the only syntax that can reference a top-level scalar field. The rejection is therefore gated on a strict `GetFieldFromName` lookup — a real declared field named `null` resolves exactly as before this PR (including `null is null`, which now parses as a valid predicate on that field), while the dynamic-field fallback (the source of the original misparse) still rejects. A JSON **sub-key** literally named `null` additionally remains reachable via quoting, e.g. `field["null"]` / `$meta["null"]`, whose base identifier is the field name, not `null`. ## Test Added `TestExpr_NullLiteral` covering rejection of `NULL` inside `in [...]`, as a comparison operand, and as a left operand, plus confirmation that `is null` / `is not null` still work. Added `TestExpr_NullLiteral_LegacyNullField` locking the legacy path: a scalar field named `null` stays queryable via the bare identifier, a JSON field named `null` stays reachable as `null["x"]`, and any casing that doesn't name a real field keeps the reserved-word rejection. Full `planparserv2` package tests pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) ## Note: field-name validation tightened Alongside the expression-level rejection, `validateFieldName` (proxy) and `validateAddedStructFieldName` (rootcoord) now go through `common.IsFieldNameKeyword`, which rejects **any casing** of `null` (`null`/`NULL`/`nUlL`/…) as a field name — consistent with how a bare `NULL` literal is rejected in expressions. Previously `FieldNameKeywords` had no `null` entry, so a field literally named `null` could be created. This only affects **new** field creation (validation runs at create time); existing collections/fields are unaffected — and thanks to the schema-aware guard above, a legacy field literally named `null` also stays queryable. --------- Signed-off-by: xiaofanluan <xf@hjjaq.com> Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com> Co-authored-by: xiaofanluan <xf@hjjaq.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
秀吉andGitHub
|
3f2ce12895 |
enhance: support text_match_fuzzy (edit-distance) scalar filter (#50933)
issue: #50920 design doc: [docs/design-docs/design_docs/20260702-text_match_fuzzy.md](docs/design-docs/design_docs/20260702-text_match_fuzzy.md) ## What Adds a scalar filter operator `text_match_fuzzy(field, "query", max_edit_distance=K)` for typo-tolerant (Levenshtein edit-distance) matching on analyzed VARCHAR fields that have a text-match index. Filter-only: it returns a boolean bitset, no scoring (scoring is tracked separately in #50921). `K` is an integer in `[0, 2]` (tantivy's fuzzy automaton hard cap); out-of-range is rejected. ## How Mirrors the existing `text_match` / `phrase_match` path through every layer, reusing the term-dictionary FST and tantivy's `FuzzyTermQuery` (no new index, no new storage): - Tantivy binding: `fuzzy_match_query` tokenizes the query with the field analyzer and ORs one `FuzzyTermQuery` per token; exposed via the `tantivy_fuzzy_match_query` C entry (cbindgen header regenerated). - C++ index: `TextMatchIndex::FuzzyMatchQuery` plus the wrapper, mirroring `MatchQuery`. - proto: `OpType.TextMatchFuzzy = 17` (next free value; 15/16 are already `InnerMatch`/`RegexMatch`). The edit distance rides in the existing `UnaryRangeExpr.extra_values`, like slop / min_should_match, so no new field is added. - grammar + Go parser: a new `text_match_fuzzy(...)` rule and `VisitTextMatchFuzzy`, which validates `K` to `[0, 2]` and stores it in `extra_values`. - C++ executor: routes the op to the text-index path (dispatch, exec-path, and offset-input guards) and calls `FuzzyMatchQuery`, with a `[0, 2]` re-check on the C++ side (the same way `PhraseMatch` validates slop) for plans not built through the parser. ## Tests - Rust unit test: distance 0/1/2, fuzzy-vs-exact, multi-token OR, and the distance `K+1` exclusion boundary. - C++ gtest (`TextMatch.FuzzyIndex`, `GrowingNaive`, `SealedNaive`): index-level typo matching and the `K+1` boundary, plus the executor + growing-segment path — a typo matches the same rows as the exact term, `not text_match_fuzzy(...)`, and the executor guard rejects out-of-range / missing `max_edit_distance`. - Go parser tests: valid 0/1/2 produce the right op and extra value; a templated query preserves the distance; out-of-range / missing / non-integer / overflow / non-string field / match-not-enabled are all rejected. Existing `text_match` and `phrase_match` behavior is unchanged (full planparserv2 suite green). ## Notes - `max_edit_distance` becomes a reserved keyword in filter expressions, consistent with `minimum_should_match` and `threshold`. - Out of scope, left as follow-ups: scoring (#50921), prefix-fuzzy, ES-style `AUTO` fuzziness, and combining with `minimum_should_match`. --------- Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com> |
||
|
|
3353b53de7 |
test: add raw-string LIKE escape-model coverage (#50843)
## Summary This PR now contains **only supplementary test coverage** for the `LIKE` escape model — no production code. The production fix it originally carried (`scanLikePattern` in `internal/parser/planparserv2/pattern_match.go`, aligning the Go optimizer with the C++ canonical matcher in `RegexQuery.cpp`) **already landed in master via #50845**, which was stacked on this PR. After rebasing onto master that production commit is dropped as already-upstream, so the diff here reduces to tests only. `git diff master` touches exactly three files, all tests: | file | layer | |---|---| | `internal/parser/planparserv2/plan_parser_v2_test.go` | Go optimizer (parser-level) | | `internal/core/src/common/RegexQueryUtilTest.cpp` | C++ executor / canonical matcher | | `tests/python_client/.../test_milvus_client_scalar_filtering.py` | Python e2e | ## What the tests add Now that raw string literals `r"..."` exist (#50845), the escape tests read with a **single** backslash instead of the doubled/quadrupled backslashes the normal string-literal layer forced — exercising the raw-string path end to end. **Go — `TestExpr_RawString_LikeEscapeModel`** drives the optimizer end-to-end through `r"..."` LIKE patterns and asserts the lowered op + literal operand: - escaped wildcard → literal byte (`r"a\%bc"` → `Equal "a%bc"`); - a literal `%` coexisting with an **unescaped** `%` boundary (`r"abc\%def%"` → `PrefixMatch "abc%def"`, `r"%abc\%def%"` → `InnerMatch "abc%def"`); - `\\` collapsing to one literal `\`; - a dangling trailing backslash — un-expressible as a raw string (parse error), and `OpType_Match` fallback via a normal string. **C++ — `RegexQueryUtilTest.cpp`** keeps the canonical-matcher escape tests (literal `%`/`_` matched verbatim, multi-char decoys, long-span positives). Raw strings are a Go-parser-only feature, so the C++ executor tests use plan-level inputs. **Python e2e** — the escaped-wildcard `LIKE` expressions and `test_like_escaped_wildcard_is_literal` now use `r"..."` (one backslash); the eval oracle gains a raw-`LIKE` branch that re-quotes the verbatim raw content via `repr()` so it stays in lock-step with the server. A raw-string dangling-backslash case (rejected at lex time) is added; the matcher-level dangling case stays a normal string since a raw string cannot end in an odd number of backslashes. ## Verification ``` go test -tags dynamic,test -gcflags="all=-N -l" -count=1 ./internal/parser/planparserv2/ ``` planparserv2 package passes; `gofmt` clean; Python `ruff check` / `ruff format` clean; the oracle's raw-`LIKE` branch verified to reproduce the documented expected result sets. issue: #43864 --------- Signed-off-by: xiaofanluan <xf@hjjaq.com> Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com> Co-authored-by: xiaofanluan <xf@hjjaq.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
yanliang567andGitHub
|
182134e29e |
test: Update aggregation and groupby E2E coverage (#50963)
## Summary - strengthen search aggregation reject coverage for unsupported group-by, iterator, and dynamic-field combinations - add controlled search order_by + group_by_field + offset regression coverage - update query aggregation GROUP BY limit and output projection coverage to current semantics issue: #50960 ## Test Plan - [x] PYTHONPATH=. /Users/yanliang.qiao/Documents/Codex/2026-06-30/new-chat/work/milvus-e2e-venv/bin/python -m pytest -q --tb=short --disable-warnings --timeout=300 --host 127.0.0.1 --port 19530 --user root --password Milvus <targeted agg/groupby cases>: 11 passed, 1 xfailed in 51.81s - [x] git diff --check -- tests/python_client/milvus_client/test_milvus_client_search_aggregation.py tests/python_client/milvus_client/test_milvus_client_search_order.py tests/python_client/testcases/test_query_aggregation.py Milvus test target: qa-milvus/yanliang-mas2-milvus-standalone-6686d94498-lhp7k, Running 1/1, restart count 0 Signed-off-by: Yanliang Qiao <yanliang.qiao@zilliz.com> |
||
|
Bingyi SunandGitHub
|
70f144a91c |
enhance: add delete/shard by namespace (#50153)
Shard data by namespace. issue: #50154 --------- Signed-off-by: sunby <sunbingyi1992@gmail.com> |
||
|
junjiejiangjjjandGitHub
|
c17ba38a3b |
enhance: Add Hugging Face inference provider support (#50818)
issue: https://github.com/milvus-io/milvus/issues/50816 Add Hugging Face Inference Providers client support for feature extraction and sentence similarity APIs, and wire it into text embedding and rerank model providers. The new provider supports: - text embedding via feature-extraction - rerank scoring via sentence-similarity - Hugging Face router provider selection with hf_provider - MILVUS_HUGGINGFACE_API_KEY credential fallback - provider config entries for text embedding and rerank Also add focused tests for the Hugging Face client, rerank provider, and paramtable provider docs. Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com> |
||
|
Li YiyangandGitHub
|
d8d823e782 |
test: add schema evolution e2e coverage (#50826)
## What - Add drop field e2e coverage for MilvusClient. - Add function field e2e coverage. - Refine schema evolution e2e coverage. - Bump pymilvus test dependency to 3.1.0rc49. ## Test - Not run in this turn; PR contains Python e2e test coverage changes. --------- Signed-off-by: lyyyuna <yiyang.li@zilliz.com> |
||
|
Buqian ZhengandGitHub
|
e05e50e144 |
fix: preserve unknown semantics for json path predicates (#50702)
issue: #50699 Fixes #50699 ## What this PR does This PR makes JSON path predicate missing/null/type-mismatch behavior preserve UNKNOWN consistently: - Treats parent JSON NULL, missing nested paths, incompatible path types, and invalid array paths as UNKNOWN instead of operator-specific boolean constants. - Preserves JSON stats/index validity bitmaps when returning cached bitmap results. - Updates parser rewrite behavior and tests that previously assumed scalar JSON missing-path `!=` was a definite true result. - Aligns raw, stats/index, brute-force, offset/iterative, and `NOT` behavior around SQL-style three-valued logic. ## Verification - `cmake -S internal/core -B cmake_build` - `ninja -v unittest/all_tests` - `go test -buildvcs=false -count=1 ./internal/parser/planparserv2/rewriter` - Focused C++ JSON nullable/contains suite passed 705/705. - Additional binary-range/non-nullable JSON range/term suite passed 540/540. - `git diff --check` passed. - Staged diff check passed. --------- Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com> |
||
|
zhuwenxingandGitHub
|
98a8cffcf2 |
test: add struct array feature coverage (#50795)
## What type of PR is this? /test ## What this PR does / why we need it Adds Python client coverage for struct array feature scenarios on master: - Covers struct-array vector subfields across JSON, JSONL, CSV, and Parquet import files. - Covers LocalBulkWriter generated JSON, JSONL, CSV, and Parquet files for FLOAT_VECTOR, FLOAT16_VECTOR, BFLOAT16_VECTOR, INT8_VECTOR, and BINARY_VECTOR struct subfields. - Aligns BulkWriter test inputs with ordinary vector field input forms, including FP16/BF16 ndarray inputs and INT8 ndarray inputs. - Adds hybrid search, range, group-by, element_scope regression coverage, and bitmap index coverage for struct array element fields. - Updates Python client test dependency pin to pymilvus 3.1.0rc50 so the merged BulkWriter JSONL and struct vector fixes are exercised. ## Which issue(s) this PR fixes None. ## Special notes for your reviewer The BulkWriter coverage intentionally uses native insert-format inputs for non-float32 vectors to match ordinary vector field behavior. ## Tests ```bash python -W ignore -m pytest -v -n 6 --tb=short \ milvus_client/test_milvus_client_struct_array.py::TestMilvusClientStructArrayImport::test_import_struct_array_vector_subfield_with_local_bulk_writer \ --host 10.100.36.207 --port 19530 --token root:Milvus \ --minio_host 10.100.36.206 --minio_bucket struct-array-master-ff3dbeb ``` Result: `20 passed in 120.02s` Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com> |
||
|
|
0bbe4d02a8 |
test: Add sparse inverted invalid parameter checks (#50775)
issue: #50108 This PR updates sparse inverted index negative coverage to verify the server rejects invalid build parameters during create_index. Changes: - Remove stale xfail markers for invalid sparse inverted index codec and non-positive block size cases. - Assert the concrete validation messages for unsupported inverted_index_codec and block_max_block_size values 0/-1. Verification: - python3 -m py_compile tests/python_client/milvus_client/test_milvus_client_sparse_inverted_index.py - python3 -m pytest -o "addopts=-p no:locust -v" tests/python_client/milvus_client/test_milvus_client_sparse_inverted_index.py::TestSparseInvertedIndexV3Negative -q --host 10.104.18.28 --port 19530 Signed-off-by: Eric Hou <eric.hou@zilliz.com> Co-authored-by: Eric Hou <eric.hou@zilliz.com> |
||
|
|
e67f265a0c |
test: update python client test expectations (#50642)
Update Python client test expectations for current master behavior. - Update describe_collection expected schema metadata. - Update binary vector dimension mismatch error messages for insert/upsert. - Update drop_collection_function negative cases to assert expected errors. Tests: - Not run per request. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: nico <cheng.yuan@zilliz.com> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
zhuwenxingandGitHub
|
b3fe69d844 |
test: add struct array nullable dynamic field coverage (#50109)
### What this PR does Adds Python client and REST coverage for struct array nullable fields, including dynamic field scenarios and nullable vector config variants. ### Tests Not run; PR creation only. --------- Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com> |
||
|
yanliang567andGitHub
|
d2bf17e734 |
test: Add external table add field E2E tests (#50420)
Related to #50416 ### What changed - Add MilvusClient E2E coverage for external table `add_collection_field` on scalar and vector fields. - Cover refresh/load/search/query behavior after adding fields, including about 10% NULL values in newly added scalar/vector fields. - Add non-Parquet source coverage for add scalar field on Lance, Iceberg, and Vortex. - Add full supported DataType matrix coverage for external add field, including scalar, array, geometry, and vector fields. - Add negative coverage for missing/duplicate `external_field`, type/dim mismatch, unsupported public data types, add/drop function, and drop field APIs. - Update Python client test dependency pin to `pymilvus==3.1.0rc35`. ### Verification - `python -m py_compile tests/python_client/milvus_client/test_milvus_client_external_table.py tests/python_client/common/external_table_common.py tests/python_client/base/client_v2_base.py tests/python_client/check/func_check.py` - `python -m ruff check tests/python_client/milvus_client/test_milvus_client_external_table.py tests/python_client/common/external_table_common.py tests/python_client/base/client_v2_base.py tests/python_client/check/func_check.py` - `python -m pytest -n 4 milvus_client/test_milvus_client_external_table.py --host 10.104.18.101 --port 19530 --minio_host 10.104.18.27 --minio_bucket yanliang-mas2 --tb=short -q -s` Result: ```text 182 passed, 1 xfailed in 318.94s (0:05:18) ``` The xfailed case is linked to #50416 and validates that old-field search still works before the server returns the current internal QueryNode assert for newly added fields before refresh. --------- Signed-off-by: Yanliang Qiao <yanliang.qiao@zilliz.com> |
||
|
junjiejiangjjjandGitHub
|
d19de1110c |
enhance: add boost score support (#50372)
issue: https://github.com/milvus-io/milvus/issues/46565 Implement boost score evaluation for the Go search reduce pipeline, including QueryNode task integration, segcore C API bindings, and score expression combination support. Add boost score runner logic in core, expose segment-level boost scoring to Go, and cover the new behavior with C++, Go, and Python client tests. Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com> |
||
|
|
e2787d3981 |
enhance: standardize error handling on merr + Sys/Input classification (#50221)
issue: #47420 ## What this PR does Project-wide migration of raw `fmt.Errorf` / `errors.New` in function bodies onto the `merr` framework, plus the Sys-vs-Input error classification and the machinery it drives (retriability, fine-grained metrics, segcore unification), plus the convention docs and a linter that keeps it from regressing. Scope: storage, proxy, coordinators (root/data/query), query node, data node, `pkg/util` & `internal/util`, expression parser, message queue, streaming, and misc packages. Bare raw-error usages went from ~3000 to a ~340 allowlist (package-level sentinels / build-tag / test sites). --- ## How to review this PR It is large but the vast majority is mechanical. Changes fall into three tiers; spend review budget on Part 2 and Part 3. ### Part 1 — Mechanical standardization (low risk, verify by rule) Each converted call follows one of a small fixed set of rules. To review, check that each site obeys the matching rule rather than reading every line: | Pattern | Rule | |---|---| | `fmt.Errorf("...")` originating a new error | → `merr.WrapErrXxxMsg("...")` with a code matching the failure's meaning | | Adding context to an existing typed error | → `merr.Wrap(err, "...")` / `merr.Wrapf(...)` — **preserves** the inner code (never `WrapErr*Err`, which overwrites it) | | Errors inside the streaming subsystem | → `status.New*` factories (StreamingError), **not** merr — this is the component-internal dialect (see `docs/dev/error_handling_guide.md`) | | Low-level / control-flow signal caught by `errors.Is` | → kept as a package-level `errors.New` sentinel (lowercase, same-package) | Conventions are documented in `docs/dev/error_handling_guide.md` (how-to) and `docs/dev/error_sentinel_convention.md` (rules + audit). A `gocritic`/`ruleguard` rule (`rawmerrerror`, in `rules.go`) enforces "no raw `return errors.New/fmt.Errorf`" under `make verifiers`. ### Part 2 — Behavior changes (review these closely) These are the sites where the wire contract or runtime behavior changes, not just the source text. Listed by category; representative locations given, full set in the diff. **A. gRPC wire-code shifts: `UnexpectedError(1)/Code 65535` → typed code.** Where a handler previously returned a raw error (collapsed to `Code=65535` on the wire), it now returns a typed merr, so the client sees a real code. The most common shift is to `IllegalArgument(5)/Code 1100` (ParameterInvalid). Touch points include datanode task handlers (CreateTask/Query/Drop), proxy Upsert, querynode GetMetrics, datacoord CreateIndex, httpserver query-response builder, and typeutil schema validation. One code refinement: an index-param validation moved `1100` → `1101` (ParameterMissing). **Client/SDK assertions and any code that switched on `Code=65535` for these paths must be re-checked** (the go_client e2e assertions were already aligned in this PR). **B. Prometheus `status` label contract change (externally visible).** The proxy metric's coarse `fail` / `rejected` values are split into `fail_input` / `fail_system` and `rejected_user` / `rejected_system` (in `requestutil.ParseMetricLabel`; auth/privilege rejections count as `rejected_user`), so dashboards can attribute a failure to caller vs operator. **Dashboards/alerts querying `status="fail"` must migrate to `status=~"fail_.*"`, and `status="rejected"` to `status=~"rejected_.*"`.** The in-repo Grafana dashboard is already migrated; external dashboards built on the old values silently go empty after upgrade. This is the one change that requires an ops-side migration. **C. Retriability semantics.** - C1: `merr.Status(err)` now forces `Retriable=false` when the error is an `InputError` — a malformed request can never succeed on blind retry, so clients never get the self-contradictory "your input is wrong but you may retry". - C2: `retry.Do` short-circuits an `InputError` (non-retriable) — **but only when the caller did not pass a `RetryErr` predicate**. The check is an `if c.isRetryErr != nil { ... } else if InputError { ... }` *mutually exclusive* branch (`pkg/util/retry/retry.go`): an explicit `RetryErr` takes precedence and bypasses the InputError abort. `retry.Handle` deliberately does **not** apply the InputError abort (its callers signal abort via `shouldRetry=false`). Four flusher startup callsites that must retry through transient "not ready" errors were given explicit `RetryErr` escape hatches. **D. segcore (C++→Go) error classification.** A single shared Go-side table (`pkg/util/merr/segcore.go`) maps each segcore code to a merr sentinel + InputError/signal category, replacing scattered hand-written `if errorCode == ...` switches in the cgo wrappers. **Wire `Code` values change for every segcore pass-through error, not just the remapped ones.** Named sentinels remap (C++ `2003` → merr `2001`, `2033` → `2002`, Folly/Knowhere codes likewise); **all remaining pass-through codes (`2004`–`2043`, previously surfaced to clients as raw C++ enum values) now serialize as `2000`** (`ErrSegcore`), with the original C++ code preserved in the `Reason` text (`segcoreCode=...`); unknown/future codes collapse to `2000` as well (pinned by the `wire_code_projection` test). Transient segcore classes (object storage / file IO / OOM / mmap / FieldNotLoaded — 11 codes) now report `Retriable=true`. **Any client switching on raw segcore codes in the `2004`–`2043` range must be re-checked**; the in-Reason code remains available for diagnostics. Signal codes (PretendFinished / FollyCancel) are recognized centrally. `errors.Is`-based control flow on these (e.g. scheduler skip/retry) is preserved. **E. InputError classification (25 sentinels + dynamic marks).** 25 sentinels in `errors.go` carry `WithErrorType(InputError)` (the Collection / ResourceGroup / Database families, `ErrIndexDuplicate`, `ErrParameterInvalid`, `ErrPrivilegeNotAuthenticated`, `ErrImportFailed`, `ErrQueryPlan`, ...), plus dynamic marks for the 8 segcore input codes (ExprInvalid, DimNotMatch, MetricTypeInvalid, FieldIDInvalid, ...) and `WrapErrAsInputError`. The widest blast radius is `ErrParameterInvalid` (1100): ~2335 `WrapErrParameterInvalid*` callsites now classify as input / non-retriable. Because of C1/C2 this changes retriability for any path that returns these. **The audit to confirm no transient path was mis-marked is the single most important review item** (see Part 3). One reverse correction: storage field-stats parsing moved from `ErrParameterInvalid` (input) to `ErrDataIntegrity` — a corrupted stored stat is data corruption, not user input. ### Part 3 — Known risks & traps (called out proactively) 1. **`merr.Wrap` vs `WrapErr*Err` (code-masking).** `WrapErr*Err` builds a `wrappedMilvusError{sentinel: ErrServiceInternal}` whose `code()` returns the *outer* sentinel — it overwrites the inner typed code and hides the `errors.Is` chain. This is intentional (use it to *deliberately* downgrade), but it was a recurring conversion defect; the rule "add context with `merr.Wrap`, downgrade with `WrapErr*Err`" is enforced by convention and reviewed across the diff. 2. **InputError × `retry.Do` blast radius.** Marking a sentinel `InputError` makes any `retry.Do(...)` without a `RetryErr` predicate stop retrying it. Reviewers should sanity-check that no transient use of the 19 newly-marked sentinels (especially `ErrParameterInvalid`) sits inside a retry loop that needed to keep spinning. The known flusher cases were handled (see C2). 3. **The ~340 raw-error allowlist.** What remains as bare `errors.New` is, by design: package-level sentinels (caught by `errors.Is`), `//go:build test` sites, and out-of-band trees (`cmd/`, `tests/`, codegen, walimpls). The linter only bans the *direct-return* form; assignment-then-return escapes and the full no-exceptions ban are deferred to an AST-based linter (Tier 2, documented). 4. **segcore C++ second step deferred.** This PR unifies classification on the Go side; splitting the dual-semantic C++ codes at the source is a follow-up. --- ## Validation - `make verifiers`: Go side clean (gofmt + static-check across modules, including the new `rawmerrerror` rule with a 0-hit baseline repo-wide). - `make test-go`: passing; the one real regression introduced (a datanode `invalid_task_type` assertion shifting `1` → `5` from a ParameterInvalid conversion) was fixed in-tree. - go_client e2e CreateIndex assertions aligned to the new merr messages. --------- Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
yanliang567andGitHub
|
a87bc5844c |
test: Add hybrid search partition isolation coverage (#50401)
## What changed - Add Python client, RESTful v2, and Go SDK regression coverage for hybrid search on collections with `partitionkey.isolation=true`. - Verify normal search still rejects unsupported partition-key isolation filters: - `tenant in ["tenant_a", "tenant_b"]` - missing tenant filter - Keep the current hybrid search server bug as conditional xfail/skip, so baseline search regressions are not hidden. - Add RESTful v2 coverage for MinHash + dense hybrid search with explicit `partitionNames`. ## Issues Related to #50398 Related to #50396 ## Verification - `ruff check ../tests/python_client/milvus_client/test_milvus_client_partition_key_isolation.py ../tests/restful_client_v2/testcases/test_vector_operations.py` - `python -m py_compile tests/python_client/milvus_client/test_milvus_client_partition_key_isolation.py tests/restful_client_v2/testcases/test_vector_operations.py` - `git diff --check` - Python client 50398 case against `10.104.18.101`: `1 xfailed` - RESTful v2 50398 case against `10.104.18.101`: `1 xfailed` - Go SDK 50398 case against `10.104.18.101`: reproduces current bug and conditionally skips Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com> |
||
|
|
fe827c9dc4 |
test: align python client expectations with current API responses (#50414)
## Summary - update `test_milvus_client_collection_describe` to expect `properties.max_field_id` in `describe_collection` - align `search_aggregation` JSON/dynamic field rejection cases with the current server error messages - keep the affected Python client regressions covered without asserting outdated responses ## Test plan - [x] `python3 -m py_compile tests/python_client/milvus_client/test_milvus_client_collection.py tests/python_client/milvus_client/test_milvus_client_search_aggregation.py` - [ ] full pytest execution was not run locally in this submission 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: nico <cheng.yuan@zilliz.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
wei liuandGitHub
|
1853a14897 |
enhance: make external vectors nullable by default (#50362)
issue: https://github.com/milvus-io/milvus/issues/45881 ## Summary - Make external user vector fields nullable by default during schema normalization. - Keep external system/virtual fields and generated function output fields excluded from forced nullable normalization. - Update schema, REST, and MilvusClient external table coverage so omitted vector nullable settings are expected to describe as nullable. ## Why External collection schemas already make user scalar fields nullable by default because external files can contain null values that Milvus cannot reject at collection creation time. Vector fields were temporarily excluded while the nullable vector offset mapping path still depended on eager load-time mapping. After lazy offset mapping for nullable vectors, that special-case exclusion is no longer needed. External vector fields can now follow the same default nullable normalization rule as scalar fields. Related lazy offset mapping PR: https://github.com/milvus-io/milvus/pull/49168 ## Validation - `git diff --check` - `python3 -m py_compile tests/restful_client_v2/testcases/test_external_collection_operations.py tests/python_client/milvus_client/test_milvus_client_external_table.py` - `GOTOOLCHAIN=auto go test -tags dynamic,test github.com/milvus-io/milvus/pkg/v3/util/typeutil -run TestNormalizeAndValidateExternalCollectionSchema -count=1 -v` - `GOTOOLCHAIN=auto go test -tags dynamic,test github.com/milvus-io/milvus/pkg/v3/util/typeutil -count=1` ## Local environment note - `/reset-milvus` reset etcd/minio before Go test runs, but this worktree does not have `bin/milvus`, so standalone Milvus could not be restarted locally. The validated tests above are pure schema/typeutil tests. Signed-off-by: Wei Liu <wei.liu@zilliz.com> |
||
|
yanliang567andGitHub
|
8a9710a6fd |
test: Add search order pagination regression coverage (#50397)
issue: #49879 ## What changed - Add an independent Python client regression test for search with `order_by_fields` plus `offset`, using controlled ANN candidates from #49879. - Mark the regression as strict xfail because current behavior still applies offset before scalar ordering. - Mark `search_aggregation` top_hits sorting by dynamic fields as strict xfail because dynamic fields are explicitly rejected for search aggregation today. ## Verification ```bash /Users/yanliang.qiao/fork/.venv-py312/bin/python -m pytest -s --tb=short --host 10.104.18.101 --port 19530 \ milvus_client/test_milvus_client_search_order.py::TestMilvusClientSearchOrderIndependent::test_milvus_client_search_order_by_with_offset \ milvus_client/test_milvus_client_search_aggregation.py::TestSearchAggregationIndependent::test_search_aggregation_top_hits_sort_by_dynamic_field ``` Result: `2 xfailed in 13.46s`. Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com> |
||
|
junjiejiangjjjandGitHub
|
43010a4f18 |
enhance: implement Go-based search reduce pipeline with Late Materialization (#48984)
#48986 Replace C++ ReduceSearchResultsAndFillData with a Go reduce pipeline: - Export per-segment search results as Arrow RecordBatch via C Data Interface - HeapMergeReduce: k-way heap merge with PK dedup and GroupBy support - Late Materialization: single CGO call (FillOutputFieldsOrdered) for output fields - ExportSearchResultAsArrow supports extra field IDs for future L0 rerank Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com> |
||
|
|
f7047d89df |
enhance: route reopened bm25 stats in delegator (#48808) (#50181)
issue: #48808 ## Summary - sync delegator function/analyzer runtime state on schema update - allow additive BM25 function fields in the IDF oracle without dropping existing stats - route Reopen BM25 stats through the delegator oracle with field-level idempotency 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: MrPresent-Han <chun.han@gmail.com> Co-authored-by: MrPresent-Han <chun.han@gmail.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
64fd939d1b |
test: align RBAC db-switch expectations (#50234)
## Summary - align the nonexistent-db RBAC test with the actual intent by checking the error with the root client - verify collection-level RBAC privileges with db-scoped clients instead of an extra using_database step - update the public-role cross-db case to match the current DescribeDatabase permission model ## Test plan - [x] source /Users/zilliz/virtual-environment/master/bin/activate - [x] python -m pytest 'milvus_client/test_milvus_client_rbac.py::TestMilvusClientRbacInvalid::test_milvus_client_grant_privilege_with_db_not_exist' 'milvus_client/test_milvus_client_rbac.py::TestMilvusClientRbacPrivilegeVerify::test_milvus_client_public_role_privilege_all_dbs' 'milvus_client/test_milvus_client_rbac.py::TestMilvusClientRbacAdvance::test_milvus_client_verify_grant_collection_load_privilege[False]' 'milvus_client/test_milvus_client_rbac.py::TestMilvusClientRbacAdvance::test_milvus_client_verify_grant_collection_load_privilege[True]' 'milvus_client/test_milvus_client_rbac.py::TestMilvusClientRbacAdvance::test_milvus_client_verify_grant_collection_release_privilege[False]' 'milvus_client/test_milvus_client_rbac.py::TestMilvusClientRbacAdvance::test_milvus_client_verify_grant_collection_release_privilege[True]' 'milvus_client/test_milvus_client_rbac.py::TestMilvusClientRbacAdvance::test_milvus_client_verify_grant_collection_insert_privilege[False]' 'milvus_client/test_milvus_client_rbac.py::TestMilvusClientRbacAdvance::test_milvus_client_verify_grant_collection_insert_privilege[True]' 'milvus_client/test_milvus_client_rbac.py::TestMilvusClientRbacAdvance::test_milvus_client_verify_grant_collection_delete_privilege[False]' 'milvus_client/test_milvus_client_rbac.py::TestMilvusClientRbacAdvance::test_milvus_client_verify_grant_collection_delete_privilege[True]' --host xxx --port 19530 --minio_host xxx -n 1 --tags RBAC -q 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: nico <cheng.yuan@zilliz.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
jiaqizhoandGitHub
|
926ad468bd |
fix: stabilize sparse search test with different nq (#50296)
issue: #50293 The sparse different-nq test was flaky because the collection data and query data were generated from different sparse dimension ranges. The collection rows used the test default dimension, while the query helper defaulted to a much larger dimension. With drop_ratio_search=0.2, some queries could lose all useful overlapping terms and return zero hits, even though the test only wanted to verify nq=1 and nq=100 batch search behavior. ex. ```text 1. sparse data in collection is generated with dim=128 The inserted sparse vectors only use dimensions in [0, 127]. The generator also forces every vector to contain dimensions 0 and 1. sparse_vector: {0: 0.80, 1: 0.60, 20: 0.40, 77: 0.90, ...} 2. query was generated with dim=1000 The query sparse vector may use dimensions in [0, 999]. It also contains dimensions 0 and 1, but most random dimensions may be outside [0, 127]. sparse_vector: {0: 0.01, 1: 0.02, 250: 0.70, 600: 0.50, 900: 0.80, ...} 3. after drop_ratio_search=0.2, low-weight query terms may be dropped sparse_vector after pruning: {250: 0.70, 600: 0.50, 900: 0.80, ...} 4. the remaining query dimensions do not exist in the collection sparse data collection dimensions: [0, 127] remaining query dimensions: 250, 600, 900 There is no sparse term overlap, so the sparse index has no candidates for this query. 5. search can return 0 hits The test expects topK=10 for every query result group, so this random no-overlap query makes the case fail. ``` This fixes the test by generating query sparse vectors with ct.default_dim, the same dimension range used by the inserted sparse data. That keeps the test focused on the intended behavior: sparse search should handle different nq values and return the expected result shape, without depending on accidental cross-dimension overlap surviving sparse term pruning. Signed-off-by: jiaqizho <jiaqi.zhou@zilliz.com> |
||
|
sijie-ni-0214andGitHub
|
d9bbbc13d0 |
feat: support drop field via AlterCollectionSchema (#48988)
## Summary - Support dropping fields from collection schema via `AlterCollectionSchema` RPC with `DropRequest` - Support dropping function fields (cascade delete output fields and indexes) - Support disabling dynamic field via `AlterCollectionSchema` (reuse existing `AlterCollection` logic) - C++ segcore defense-in-depth: skip dropped fields during segment loading issue: #48983 design doc: docs/design-docs/design_docs/20260413-drop-collection-field-design.md ## Test plan - [x] Unit tests for rootcoord drop field/function logic - [x] Unit tests for proxy task validation (drop field constraints) - [x] Unit tests for datacoord index service (dropped field handling) - [x] Unit tests for querynode segment loader (dropped field skipping) - [x] C++ unit tests for SegmentLoadInfo ComputeDiff with dropped fields - [ ] E2E tests: drop scalar field, drop vector field, drop field with index, drop function, disable dynamic field (deferred to QA pending pymilvus RC) 🤖 Generated with [Claude Code](https://claude.ai/code) --------- Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com> |
||
|
|
37a9e00c27 |
test: Fix sparse negative collection fixture (#50262)
## What this PR does Splits `TestSparseInvertedIndexV3Negative` into two setup collections so the fixture no longer exceeds the default `proxy.maxVectorFieldNum=10` limit: - the original collection keeps the 10 ordinary sparse vector fields - a separate BM25 collection holds the 2 BM25 output sparse vector fields This lets the negative cases reach their intended `create_index` validation checks instead of failing during `create_collection` setup. Fixes #50251 Related: #50108 ## Verification ```bash python3 -m py_compile tests/python_client/milvus_client/test_milvus_client_sparse_inverted_index.py git diff --check -- tests/python_client/milvus_client/test_milvus_client_sparse_inverted_index.py pytest -n 6 --dist loadgroup --host 10.104.15.39 --port 19530 ./milvus_client/test_milvus_client_sparse_inverted_index.py::TestSparseInvertedIndexV3Negative ``` Result: ```text 9 passed, 3 xfailed in 10.63s ``` Signed-off-by: Eric Hou <eric.hou@zilliz.com> Co-authored-by: Eric Hou <eric.hou@zilliz.com> |
||
|
wei liuandGitHub
|
6615811dee |
enhance: Support add-column refresh for external collections (#50082)
issue: #45881 Design doc: docs/design-docs/design_docs/20260526-external_table_add_column_refresh.md This PR lets external collection refresh handle additive schema changes, especially adding columns to parquet-backed external collections after segments have already been refreshed. What changed: - DataNode detects existing same-fragment external segments that are missing newly added external columns, appends manifest column groups idempotently, rebuilds fake binlogs, and returns same-ID updated segments. - DataCoord applies kept and updated refresh task payloads as validated segment upserts, including both patched existing segments and newly created segments. - Refresh apply intentionally does not use a job/task schema-version gate for the current additive-only scope. An older-schema refresh may complete without the newest fields, and the next refresh self-heals missing external columns. Segment-level validation still rejects schema-version rollback. - Proxy and RootCoord validate external add-field and alter-schema flows after field IDs are resolved, rejecting duplicate external_field mappings, generated output collisions, and unsupported external field types. - StorageV2 packed manifest helpers can read existing column groups and append missing columns through CommitManifestUpdates. - Patched same-fragment segments include function-output bytes in fake binlog MemorySize so memory estimates include generated columns. - Added the add-column refresh design doc and linked it from the external table design doc. - Added a go-client e2e that adds parquet column score after the first refresh and verifies returned query values, including 0..4 -> 0..0.04 and 100..104 -> 1.00..1.04. Validation: - make generated-proto-without-cpp - make milvus - make lint-fix - git diff --check - git diff --check milvus/master...HEAD - No-new-mockery diff scan - reset Milvus before scoped Go test runs - go test -tags dynamic,test -gcflags="all=-N -l" -ldflags="-r ${RPATH}" for targeted typeutil, proxy, datacoord, and storagev2/packed tests - go test -tags dynamic,test -gcflags="all=-N -l" -ldflags="-r ${RPATH}" github.com/milvus-io/milvus/internal/rootcoord -run '^$' - go test -tags dynamic,test -gcflags="all=-N -l" -ldflags="-r ${RPATH}" github.com/milvus-io/milvus/internal/datanode/external -run TestRefreshExternalCollectionTaskSuite -testify.m TestOrganizeSegments_PatchesMissingExternalColumns - go test -tags dynamic,test -gcflags="all=-N -l" -ldflags="-r ${RPATH}" github.com/milvus-io/milvus/internal/storagev2/packed github.com/milvus-io/milvus/pkg/v3/util/typeutil -run 'TestAppendSegmentManifestColumnsValidationPaths|TestAppendSegmentManifestColumnsReadAndNoopPaths|TestNormalizeAndValidateExternalCollectionSchema' - internal/datacoord full package coverage hit the known local macOS DISKANN subcase; targeted refresh task tests passed after reset Signed-off-by: Wei Liu <wei.liu@zilliz.com> |
||
|
|
720338c1dc |
fix: support null ordering for search order by (#50163)
issue: #49869 ## Summary - Align search ORDER BY null placement with query/PostgreSQL defaults: ASC NULLS LAST and DESC NULLS FIRST. - Support explicit `nulls_first` / `nulls_last` options in search `order_by_fields`. - Cover scalar, JSON path, and dynamic-field null ordering behavior in proxy tests. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: MrPresent-Han <chun.han@gmail.com> Co-authored-by: MrPresent-Han <chun.han@gmail.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
|
99cf5e8d48 |
fix: preserve search aggregation group-by metadata (#50176)
issue: #49840 ## Summary - Preserve plural group-by field IDs when shallow-copying search requests for query nodes. - Add regression coverage for SearchRequest shallow-copy metadata propagation. - Re-enable the growing-segment search aggregation E2E case. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: MrPresent-Han <chun.han@gmail.com> Co-authored-by: MrPresent-Han <chun.han@gmail.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
455a8d9cac |
test: add sparse index v3 client coverage and vec index version config (#50105)
## Summary - add `tests/python_client/milvus_client/test_milvus_client_sparse_inverted_index.py` with sparse inverted index v3 Python client coverage - set `dataCoord.targetVecIndexVersion: 10` in the same 18 e2e/nightly helm value files aligned with PR #49893 config scope Fixes #50108 ## Test plan - [ ] run sparse inverted index Python client tests in CI/local environment - [ ] validate helm-based e2e/nightly jobs pick up `dataCoord.targetVecIndexVersion=10` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Eric Hou <eric.hou@zilliz.com> Co-authored-by: Eric Hou <eric.hou@zilliz.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
zhuwenxingandGitHub
|
5368d06655 |
fix: size field bitsets by field id range (#50141)
Fixes #50107 ## What - Size field-id-indexed bitsets from the maximum user field-id offset instead of `schema->size()`. - Apply the helper to search plan involved-fields bitsets and sealed segment ready bitsets. - Add C++ and Python regressions for searching a late StructArray vector subfield after multiple StructArray parent fields. ## Why StructArray parent fields consume field IDs but are not stored as regular C++ schema fields. With multiple StructArray parents, later subfields can have `field_id - START_USER_FIELDID` greater than `schema->size()`, causing `invalid field id` during search. ## Verification - `git diff --cached --check` - `python3 -m py_compile tests/python_client/milvus_client/test_milvus_client_struct_array.py` - `milvus-build build local -b master --allow-dirty`: `build-local-zhuwenxing-fix-50107-bitset-d44e5bc` - Deployed `harbor.milvus.io/manta/milvus:local-zhuwenxing-fix-50107-bitset-d44e5bc` as `issue-50107-master-fix-d44e5bc`: Healthy - `verify_milvus.py 10.100.36.218 19530`: passed, server `master-20260528-b7293edf8d` - `pytest milvus_client/test_milvus_client_struct_array.py::TestMilvusClientStructArraySearch::test_search_struct_array_last_vector_subfield`: 1 passed - Imported original `crime_cases_v3` issue data: 600 rows; `debug_struct_vector.py`: all 6 ANN searches OK, including `evidence[evidence_vector]` Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com> |
||
|
zhuwenxingandGitHub
|
3517b1ec90 |
test: use rjieba for jieba tokenizer tests (#50180)
## What Switch Python test-side Jieba usage from the Python `jieba` package to `rjieba`, the Python binding for `jieba-rs`. ## Changes - Replace Python test requirements from `jieba==0.42.1` to `rjieba==0.1.11` - Update Python client and REST test helpers to call `rjieba.cut`, `rjieba.cut_for_search`, and `rjieba.cut_all` - Update analyzer expected-token helper to use `rjieba` for supported mode/hmm cases - Keep custom-dictionary validation focused on analyzer output because `rjieba` does not expose dynamic dictionary APIs like Python `jieba.add_word` or `set_dictionary` ## Test - `git diff --check` - `python3 -m py_compile tests/python_client/common/common_func.py tests/python_client/common/phrase_match_generator.py tests/restful_client_v2/utils/utils.py tests/python_client/milvus_client/test_milvus_client_analyzer.py` - `rjieba` API smoke test - Against `struct-hybrid-master-d44e5bc` at `10.100.36.224:19530`: - `TestMilvusClientAnalyzer::test_jieba_custom_analyzer`: 6 passed - `TestMilvusClientAnalyzer::test_analyzer`: 3 passed Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com> |
||
|
wei liuandGitHub
|
92740fa8c6 |
fix: Support Vortex byte-vector list input (#50110)
issue: #49828 ## What changed This PR accepts UInt8 List and FixedSizeList Arrow input for BF16 and binary vector fields when the list length matches the raw byte width. It keeps non-UInt8 byte-vector list input rejected, keeps width mismatch validation, and enables the Vortex full-matrix E2E case for `bf16v`, `binv_flat`, and `binv_ivf`. ## Why Vortex writes BF16 and binary vector payloads as UInt8 list values because it cannot persist them as FixedSizeBinary. Milvus previously rejected those list encodings, so Vortex external tables could not cover BF16 and binary vector fields. ## Validation - `make milvus` - `all_tests --gtest_filter='NormalizeExternalArrow.*'` - `all_tests --gtest_filter='NormalizeVectorArraysToFixedSizeBinary.*'` - `python3 -m py_compile tests/python_client/milvus_client/test_milvus_client_external_table.py` - `make lint-fix` - `git diff --check --cached` - `pytest milvus_client/test_milvus_client_external_table.py -k test_milvus_client_external_table_full_matrix_vortex` --------- Signed-off-by: Wei Liu <wei.liu@zilliz.com> |