## What does this PR do?
Adds Python e2e coverage for JSON filtering UNKNOWN semantics in
`tests/python_client/milvus_client/expressions/test_milvus_client_json_filtering.py`.
This PR covers:
- raw filtering paths on growing and sealed segments
- JSON arithmetic/filtering UNKNOWN behavior for missing, null, and
type-mismatch values
- JSON array predicate behavior for missing and non-array values
- ARRAY subscript UNKNOWN behavior for null, empty, and out-of-range
values
- contradiction rewrite cases, including outer `not (...)`, to ensure
UNKNOWN is not rewritten into TRUE
- JSON path index filtering with `json_cast_type=DOUBLE`
- JSON path index filtering with `json_cast_type=VARCHAR`
- JSON flat index coverage for currently passing UNKNOWN cases
- query/search filter consistency for representative JSON filters,
including indexed collections
- L2 coverage for boolean JSON values, fractional double values,
additional comparison/logical operators, JSON path array subscripts,
mixed arrays, and `ARRAY_DOUBLE` JSON path index
The known JSON flat index array-vs-scalar comparison mismatch is
documented in #51193 and covered by a skipped regression case so the
coverage can land first without taking server-side fixes in this PR.
issue: #50976
issue: #51193
Related #50699#50698
## Test Plan
- [x] `git diff --check`
- result: passed
- [x] `uvx ruff check --config tests/ruff.toml
tests/python_client/milvus_client/expressions/test_milvus_client_json_filtering.py`
- result: passed
- [x] `uvx ruff format --check --config tests/ruff.toml
tests/python_client/milvus_client/expressions/test_milvus_client_json_filtering.py`
- result: passed
- [x] `/Users/yanliang.qiao/fork/milvus/tests/.venv/bin/python -W ignore
-m pytest milvus_client/expressions/test_milvus_client_json_filtering.py
--collect-only -q`
- result: `29 tests collected`
- [x] `milvus-dev-cli deploy create
harbor.milvus.io/manta/milvus:master-20260709-9a0d7f2 --name
json-filtering-l2-9a0d7f2 --namespace chaos-testing --no-auth
--wait-timeout 1800 --wait-interval 10`
- result: Healthy
- endpoint: `lb-6d57ndgl-sw4giqjhodl3bn15.clb.bj-tencentclb.net:19530`
- [x] `/Users/yanliang.qiao/fork/milvus/tests/.venv/bin/python -W ignore
-m pytest milvus_client/expressions/test_milvus_client_json_filtering.py
--uri http://lb-6d57ndgl-sw4giqjhodl3bn15.clb.bj-tencentclb.net:19530
--token "" -s -v --tb=short`
- result: `28 passed, 1 skipped in 215.82s`
## Notes
The skipped case is
`test_json_flat_index_array_scalar_comparison_unknown_semantics_query`,
linked to #51193. The skip can be removed after the server-side JSON
flat index UNKNOWN semantics issue is fixed.
Signed-off-by: Yanliang Qiao <yanliang.qiao@zilliz.com>
## Summary
- validate REST quick-create BinaryVector metricType against the
configured binary auto-index support set
- keep default BIN_IVF_FLAT quick-create limited to HAMMING/JACCARD so
SUBSTRUCTURE/SUPERSTRUCTURE/MHJACCARD fail before collection creation
- add Go handler coverage and RESTful e2e params for the previously
missed binary metrics
Follow-up for review comment on #51088.
related issue: https://github.com/milvus-io/milvus/issues/51084
## Test Plan
- [x] git diff --check
- [x] python3 -m py_compile
tests/restful_client_v2/testcases/test_collection_operations.py
- [ ] go test ./internal/distributed/proxy/httpserver -run
TestCreateCollectionQuickVectorFieldType -count=1 (blocked locally:
missing rocksdb.pc and milvus_core.pc pkg-config files)
Signed-off-by: Yanliang Qiao <yanliang.qiao@zilliz.com>
## What
Validate REST v2 quick-create enum-like fields before creating the
collection.
This PR fixes `vectorFieldType` handling for quick collection creation:
- rejects invalid `vectorFieldType` values instead of silently using
`FloatVector`
- supports `FloatVector`, `BinaryVector`, `Float16Vector`,
`BFloat16Vector`, and `SparseFloatVector`
- defaults quick-create index metric by vector type: dense `COSINE`,
binary `HAMMING`, sparse `IP`
- validates explicit binary/sparse `metricType` before collection
creation
- rejects `SparseFloatVector` quick-create with `dimension`
It also fixes top-level quick-create `consistencyLevel` handling:
- parses and validates top-level `consistencyLevel`
- preserves existing `params.consistencyLevel` behavior
- rejects conflicting top-level and `params.consistencyLevel` values
Fixes#51085Fixes#51084
## Why
Previously, `/v2/vectordb/collections/create` ignored top-level
`vectorFieldType` and always generated a `FloatVector` schema in
quick-create mode. A typo such as `InvalidVectorType` could return
success and mask client-side bugs.
Binary and sparse quick-create requests without `metricType` could also
create the collection first and then fail index creation because the old
fallback metric was `COSINE`, which is incompatible with those vector
types.
For #51084, top-level `consistencyLevel` was not present in
`CollectionReq`, so JSON decoding silently dropped it. Invalid values
such as `"consistencyLevel": "Invalid"` were ignored and the collection
could be created with the default consistency level.
## Tests
- `./bin/gofumpt -w internal/distributed/proxy/httpserver/request_v2.go
internal/distributed/proxy/httpserver/handler_v2.go
internal/distributed/proxy/httpserver/handler_v2_test.go`
- `./bin/gci write internal/distributed/proxy/httpserver/request_v2.go
internal/distributed/proxy/httpserver/handler_v2.go
internal/distributed/proxy/httpserver/handler_v2_test.go
--skip-generated -s standard -s default -s
"prefix(github.com/milvus-io)" --custom-order`
- `git diff --check`
- `python3 -m py_compile
tests/restful_client_v2/testcases/test_collection_operations.py`
- `rg -n
"string\\(paramtable\\.(BinaryVectorDefaultMetricType|SparseFloatVectorDefaultMetricType)\\)|interface\\("
internal/distributed/proxy/httpserver/handler_v2.go
internal/distributed/proxy/httpserver/handler_v2_test.go`
Attempted locally but blocked by missing native dependencies:
```
go test ./internal/distributed/proxy/httpserver -run 'TestCreateCollection(QuickVectorFieldType|TopLevelConsistencyLevel)$' -count=1
# missing rocksdb.pc and milvus_core.pc
```
CI investigation before this push:
```
# build-ut-cov/code-check failed on c6 predecessor because of unconvert:
# internal/distributed/proxy/httpserver/handler_v2.go:2054:16 unnecessary conversion
# internal/distributed/proxy/httpserver/handler_v2.go:2056:16 unnecessary conversion
# internal/distributed/proxy/httpserver/handler_v2_test.go:2185:28 unnecessary conversion
# internal/distributed/proxy/httpserver/handler_v2_test.go:2217:28 unnecessary conversion
# Fixed by removing string(...) around paramtable default metric constants.
# e2e-default failed in job 40810 on:
# testcases/test_geometry_operations.py::TestGeometryCollection::test_spatial_query_and_search[False-True-sealed-ST_CROSSES]
# error: Connection refused to /v2/vectordb/collections/flush on port 19530
# the same job showed querynode/streamingnode restarts and proxy ContainerCreating during cleanup.
# REST collection tests in that job had already passed, so this looked unrelated to this PR.
# e2e-default failed again in dispatcher 13914 / job-1 32439 on:
# testcases/test_collection_operations.py::TestCreateCollectionNegative::test_create_collection_quick_setup_with_invalid_consistency_level
# root cause: the test used collection_create(), whose wrapper auto-injects params.consistencyLevel=Strong.
# That made the request hit the new conflict branch instead of the intended top-level invalid consistencyLevel branch.
# Fixed in 3e190eee56 by sending the request through raw client.post() to avoid wrapper mutation.
```
Signed-off-by: Yanliang Qiao <yanliang.qiao@zilliz.com>
## Summary
Enable the DataNode vortex storage format in the Helm values that
already enable Loon FFI storage:
- tests/_helm/values/e2e/standalone
- tests/_helm/values/e2e/distributed-pulsar
- tests/_helm/values/e2e-amd/standalone
- tests/_helm/values/e2e-amd/distributed-pulsar
- tests/_helm/values/e2e-arm/standalone
- tests/_helm/values/e2e-arm/distributed-pulsar
- tests/_helm/values/nightly/distributed-woodpecker
- tests/_helm/values/nightly/distributed-woodpecker-service
Fixes#50441
## Test Plan
- [x] ruby YAML parse check for the eight target values files, verifying
common.storage.useLoonFFI is true and dataNode.storage.format is vortex
- [x] git diff scope check confirmed only the two nightly woodpecker
files were added in the follow-up amend
Signed-off-by: Yanliang Qiao <yanliang.qiao@zilliz.com>
Related to #50416
### What changed
- Add MilvusClient E2E coverage for external table
`add_collection_field` on scalar and vector fields.
- Cover refresh/load/search/query behavior after adding fields,
including about 10% NULL values in newly added scalar/vector fields.
- Add non-Parquet source coverage for add scalar field on Lance,
Iceberg, and Vortex.
- Add full supported DataType matrix coverage for external add field,
including scalar, array, geometry, and vector fields.
- Add negative coverage for missing/duplicate `external_field`, type/dim
mismatch, unsupported public data types, add/drop function, and drop
field APIs.
- Update Python client test dependency pin to `pymilvus==3.1.0rc35`.
### Verification
- `python -m py_compile
tests/python_client/milvus_client/test_milvus_client_external_table.py
tests/python_client/common/external_table_common.py
tests/python_client/base/client_v2_base.py
tests/python_client/check/func_check.py`
- `python -m ruff check
tests/python_client/milvus_client/test_milvus_client_external_table.py
tests/python_client/common/external_table_common.py
tests/python_client/base/client_v2_base.py
tests/python_client/check/func_check.py`
- `python -m pytest -n 4
milvus_client/test_milvus_client_external_table.py --host 10.104.18.101
--port 19530 --minio_host 10.104.18.27 --minio_bucket yanliang-mas2
--tb=short -q -s`
Result:
```text
182 passed, 1 xfailed in 318.94s (0:05:18)
```
The xfailed case is linked to #50416 and validates that old-field search
still works before the server returns the current internal QueryNode
assert for newly added fields before refresh.
---------
Signed-off-by: Yanliang Qiao <yanliang.qiao@zilliz.com>
## Summary
Restore the CSV branch in the nullable vector RemoteBulkWriter
regression test.
This removes the previous skip and the explicit `nullkey="null"`
workaround so the test now covers the default CSV import path for
nullable `FLOAT_VECTOR` values.
Fixes#49678
## Test Plan
- [x] `/Users/yanliang.qiao/fork/.venv-py312/bin/python -m pytest
testcases/test_bulk_insert.py::TestBulkInsertNullableVector::test_bulk_writer_nullable_float_vector
--collect-only -q`
- [x] `/Users/yanliang.qiao/fork/.venv-py312/bin/python -m pytest -q -s
'testcases/test_bulk_insert.py::TestBulkInsertNullableVector::test_bulk_writer_nullable_float_vector[60-32-4]'
--host 10.104.30.186 --port 19530 --minio_host 10.104.18.198
--minio_bucket yanliang-mas2`
Note: I also ran the full JSON/CSV/Parquet parametrized test against the
same temporary cluster. JSON and CSV passed; the Parquet parameter later
hit a `get_bulk_insert_state` RPC `DEADLINE_EXCEEDED` while polling
import state, unrelated to this CSV regression change.
Signed-off-by: Yanliang Qiao <yanliang.qiao@zilliz.com>
issue: #49046
## What
- Add Go SDK builders for search aggregation and top hits, including
validation and proto request wiring.
- Parse search aggregation responses from `AggBuckets`/`AggTopks` into
`ResultSet.AggregationBuckets`.
- Reject incompatible search options such as legacy group-by, offset,
and search iterator usage.
- Add unit tests, a Go SDK example, and Go E2E coverage for search
aggregation.
## Tests
- `cd client && go test ./... -count=1`
- `cd tests/go_client && GO_CLIENT_SKIP_EXTERNAL_TABLE_DEPS_INSTALL=true
go test ./testcases -run TestSearchAggregation -count=1 -args -addr
http://10.104.18.101:19530`
## Notes
- `make static-check` was attempted locally but blocked by missing LLVM:
`ERROR: Valid LLVM (14-18) not installed. Run: brew install llvm@17`.
Signed-off-by: Yanliang Qiao <yanliang.qiao@zilliz.com>
## What changed
- Add Python client, RESTful v2, and Go SDK regression coverage for
hybrid search on collections with `partitionkey.isolation=true`.
- Verify normal search still rejects unsupported partition-key isolation
filters:
- `tenant in ["tenant_a", "tenant_b"]`
- missing tenant filter
- Keep the current hybrid search server bug as conditional xfail/skip,
so baseline search regressions are not hidden.
- Add RESTful v2 coverage for MinHash + dense hybrid search with
explicit `partitionNames`.
## Issues
Related to #50398
Related to #50396
## Verification
- `ruff check
../tests/python_client/milvus_client/test_milvus_client_partition_key_isolation.py
../tests/restful_client_v2/testcases/test_vector_operations.py`
- `python -m py_compile
tests/python_client/milvus_client/test_milvus_client_partition_key_isolation.py
tests/restful_client_v2/testcases/test_vector_operations.py`
- `git diff --check`
- Python client 50398 case against `10.104.18.101`: `1 xfailed`
- RESTful v2 50398 case against `10.104.18.101`: `1 xfailed`
- Go SDK 50398 case against `10.104.18.101`: reproduces current bug and
conditionally skips
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
Related to #50384
This increases standalone Milvus CPU requests from 2 cores to 3 cores in
E2E and nightly standalone Helm values.
On the Go SDK ARM CI nodes, allocatable CPU is about 7.91 cores. With a
2-core request, three standalone Milvus pods can be scheduled onto the
same 8-core node while each pod can burst up to an 8-core limit. Raising
the request to 3 cores prevents three standalone pods from fitting on
one node and reduces CPU contention during index build / snapshot
restore heavy tests.
Validation:
- `git diff --check origin/master..HEAD`
- Parsed all updated YAML values with Ruby `YAML.load_file`
- Confirmed all updated standalone CPU requests are `"3"`
Signed-off-by: Yanliang Qiao <yanliang.qiao@zilliz.com>
## What
- Set standalone Helm test resource requests to 2 CPU / 8Gi memory.
- Set standalone Helm test resource limits to 8 CPU / 16Gi memory.
- Add the same standalone resources to one-pod values that previously
omitted them.
## Related
Related #50384
## Verification
- ruby -ryaml validation for all 11 standalone values files
- git diff --check HEAD~1..HEAD
Signed-off-by: Yanliang Qiao <yanliang.qiao@zilliz.com>
issue: #49879
## What changed
- Add an independent Python client regression test for search with
`order_by_fields` plus `offset`, using controlled ANN candidates from
#49879.
- Mark the regression as strict xfail because current behavior still
applies offset before scalar ordering.
- Mark `search_aggregation` top_hits sorting by dynamic fields as strict
xfail because dynamic fields are explicitly rejected for search
aggregation today.
## Verification
```bash
/Users/yanliang.qiao/fork/.venv-py312/bin/python -m pytest -s --tb=short --host 10.104.18.101 --port 19530 \
milvus_client/test_milvus_client_search_order.py::TestMilvusClientSearchOrderIndependent::test_milvus_client_search_order_by_with_offset \
milvus_client/test_milvus_client_search_aggregation.py::TestSearchAggregationIndependent::test_search_aggregation_top_hits_sort_by_dynamic_field
```
Result: `2 xfailed in 13.46s`.
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
issue: #50379Fixes#50379
This PR narrows Kafka chaos selectors to broker pods by adding
`app.kubernetes.io/component: kafka`. The previous selector matched both
broker pods and the Kafka exporter pod because both use
`app.kubernetes.io/name: kafka`.
Changes:
- Add `app.kubernetes.io/component: kafka` to Kafka container-kill
chaos.
- Add the same broker selector to Kafka pod-kill chaos.
- Update Kafka pod-failure chaos from the legacy `component: kafka`
selector to `app.kubernetes.io/component: kafka`.
Verification:
- `git diff --check`
- Parsed the three Kafka chaos YAML files and verified each selector
contains `app.kubernetes.io/component=kafka` with no stale `component`
selector.
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
## What this PR does
Enable Storage V3 explicitly in CI Helm values by setting
`common.storage.useLoonFFI: true` with an inline comment for e2e,
e2e-amd, e2e-arm, and nightly deployments.
## Related dev PR
Related to #49880
## Verification
- Parsed all updated values files and their embedded
`extraConfigFiles.user.yaml` successfully.
- Verified all 18 target values files set `common.storage.useLoonFFI ==
true`.
- Ran `git diff --check`.
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
## What changed
- Add Chaos Mesh `container-kill` schedules for Milvus components and
dependencies.
- Add container-kill coverage for etcd, MinIO, Pulsar, and Kafka targets
to match pod-kill coverage.
- Resolve real pod container names before creating container-kill
resources, which is required for release-prefixed Pulsar containers.
- Add selected-pod container-kill support for etcd leader/follower chaos
scenarios.
## Validation
- `python3 -m py_compile tests/python_client/chaos/test_chaos_apply.py
tests/python_client/chaos/test_chaos_apply_to_determined_pod.py
tests/python_client/utils/util_k8s.py`
- Parsed 17 container-kill YAML/template files with PyYAML.
- `git diff --check`
- Verified 4am Chaos Mesh v2.8.0 CRD supports `container-kill` and
`containerNames`.
- Server-side dry-run passed for static container-kill manifests against
4am Chaos Mesh, excluding Pulsar static YAML because Pulsar container
names are filled dynamically at runtime.
- Server-side dry-run passed for a Pulsar container-kill Schedule after
filling real containerNames.
- Server-side dry-run passed for etcd selected-pod PodChaos after
filling real containerNames.
Fixes#49798
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
## What this PR does
Fixes#49542
This updates stale L2 search test assertions without changing the
original test coverage points:
- keep the Bounded consistency range-search case on Bounded, but allow
already-visible growing data to be returned
- keep the duplicate-primary-key search case focused on PK
de-duplication, without requiring de-duplicated results to still equal
the requested limit
- keep distance ordering validation, but compare floating-point ordering
with a small epsilon instead of strict list equality
## Tests
Against Milvus `10.104.18.67:19530`:
- `pytest -q --host 10.104.18.67 --port 19530 --tag L2
milvus_client/test_milvus_client_range_search.py::TestRangeSearchIndependent::test_range_search_with_consistency_bounded`
- 2 passed
- `pytest -q --host 10.104.18.67 --port 19530 --tag L2
milvus_client/test_milvus_client_search_by_pk.py::TestSearchByPkIndependent::test_search_by_pk_with_duplicate_primary_key`
- 1 passed
- `pytest -q --host 10.104.18.67 --port 19530 --tag L2
milvus_client/test_milvus_client_search_pagination.py::TestMilvusClientSearchPagination::test_search_with_pagination_topk`
- 3 passed
Flaky check with `-n 10 --count=10`:
- Bounded consistency: 20 passed
- Duplicate primary key: 10 passed
- Pagination topK: 30 passed
Signed-off-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
Fixes#48917
This fixes the multilingual NGRAM test by removing the incorrect
even-distribution count assertion. The test data is inserted in two
batches of 3000 rows, and each batch resets keyword assignment with j %
16, so keywords are not distributed evenly across all 6000 rows.
The test now verifies the actual intent: NGRAM-indexed JSON LIKE results
are non-empty and exactly match the non-indexed JSON LIKE scan results
for each multilingual keyword.
Verification:
```bash
source /Users/yanliang.qiao/fork/milvus/tests/python_client/.venv/bin/activate
cd tests/python_client
python3 -m pytest testcases/indexes/test_ngram.py::TestNgramBuildParams::test_ngram_search_with_multilingual_utf8_strings -q --host 10.104.15.131
```
Result: 1 passed in 19.32s
Signed-off-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
## Problem
`assert_statistic()` in `chaos_commons.py` evaluated the success
condition as:
```python
succ_rate >= succ_rate_threshold and total > 2
```
Low-frequency checkers (`AddFieldChecker`, `AddVectorFieldChecker`) have
~111s cycle times and complete only ~2 operations within a 10-minute
test window. This caused `total > 2` to always be `False` even when
`succ_rate = 1.0`, producing false failures in
`test_concurrent_operation`.
Root cause observed in deploy_test_cron runs where checkers reported
`succ_rate: 1.00, total: 2` but still triggered `pytest.assume(False)`.
## Fix
Split the assertion into two cases:
- `total == 0`: checker never executed any operation → always fail (real
issue: thread failed to start or service completely down)
- `total >= 1`: at least one operation completed → check `succ_rate`
only, no minimum total threshold
`succ_rate = 1.0, total = 2` is a valid passing result for a
low-frequency checker in a short test window.
## Note
`test_chaos_bulk_insert.py` contains a stale local copy of
`assert_statistic` with the same pattern (predates the 2023 refactor in
#25890) but only uses high-frequency checkers so does not trigger this
issue in practice. Left as a separate cleanup.
---------
Signed-off-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
Adds a regression test
`test_milvus_client_load_replica_change_query_uninterrupted` to verify
that queries are not interrupted when changing the replica number on an
already-loaded collection (issue #49304).
The test:
1. Loads a 100k-row collection with replica=1
2. Starts a background query loop with `timeout=2` (critical to surface
gRPC-masked errors)
3. Calls `load_collection(replica_number=2)` to trigger the bug
condition
4. Asserts zero query failures occurred during the replica transition
Also updates the docstring for `test_milvus_client_load_replica_change`
to match the revised test flow (hot replica change without
release+reload).
Also fixes all pre-existing ruff lint violations (452 errors) in the
file so CI `code-check` can pass:
- F403/F405: replaced star import with explicit imports
- I001: fixed import grouping/ordering
- F541: empty f-string prefixes
- F841: unused variable assignments
- E712: `== True/False` comparisons
- W291: trailing whitespace in docstrings
- `import numpy` → `import numpy as np`
### Changes
**Modified Files:**
- `tests/python_client/milvus_client/test_milvus_client_collection.py`:
new test case + docstring update + ruff lint fixes
### Test Plan
- [x] New test
`test_milvus_client_load_replica_change_query_uninterrupted` reproduces
the bug on Milvus 2.6.15 (5/5 FAILED with `GetShardLeaders: collection
not fully loaded`)
- [x] New test passes on master build including fix PR #49305 (6/6
PASSED)
- [x] `test_milvus_client_load_replica_change` passes on both 2.6 and
master (6/6 PASSED each)
- [x] `ruff check` and `ruff format --check` pass with zero errors
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
## Summary
Fix two categories of checker false failures in concurrent chaos tests.
issue: #49329
### Fix 1: NullVectorQueryChecker false positive on null rows
When `UpsertChecker` runs concurrently on the shared collection, it can
legitimately overwrite rows with null nullable vector values (20% null
probability in `gen_row_data_by_schema`). After `_non_null_pk_samples`
is
refreshed, the newly sampled PKs may include rows inserted by
`UpsertChecker`
which are then overwritten with null on the next upsert cycle.
**Fix**: treat `null_rows` (like empty results) as a stale sample event
— refresh
the sample and return `True`. The all-null fallback path still detects
systematic
data corruption when no non-null rows can be found at all.
### Fix 2: PartialUpdateChecker false failures on
SchemaMismatchRetryableException
`PartialUpdateChecker` calls `milvus_client.upsert(...,
partial_update=True)` which
hits the same schema_timestamp version check as
InsertChecker/UpsertChecker (already
fixed in #49330). When `AddVectorFieldChecker` adds a nullable vector
field, the
server increments the schema version and rejects stale-timestamp
requests.
`PartialUpdateChecker` is used by `test_single_request_operation.py`.
**Fix**: same pattern as the already-fixed checkers — catch
`SchemaMismatchRetryableException`, call `_invalidate_schema()`, retry
once.
### Not fixed (don't need it)
- `InsertFreshnessChecker` / `UpsertFreshnessChecker`: no test uses them
(legacy)
- `InsertFlushChecker`: no test uses it (legacy)
- `EntityTTLChecker`: uses its own collection with a fixed schema,
unaffected by `AddVectorFieldChecker`
- `SnapshotRestoreChecker._do_dml_operations()`: `SchemaMismatch`
already caught as `log.warning`, not counted in succ_rate
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
Fix false failures in `InsertChecker` and `UpsertChecker` when
`AddVectorFieldChecker` runs concurrently during chaos tests.
issue: #49329
### Root Cause
When `AddVectorFieldChecker` adds a nullable vector field, the server
increments the collection's schema version (`schema_timestamp`). The
pymilvus SDK embeds the cached `schema_timestamp` in each insert/upsert
gRPC request. If the cached value is stale, the server rejects the
request with `SchemaMismatch` (error code 62), which pymilvus raises as
`SchemaMismatchRetryableException`.
The data itself is always valid — new fields are always nullable, so
omitting them is fine. The rejection is purely a schema version check.
### Fix
Catch `SchemaMismatchRetryableException` in both
`InsertChecker.insert_entities()` and `UpsertChecker.upsert_entities()`.
On catch:
1. Call `_invalidate_schema()` to flush the stale cache entry (same
mechanism as pymilvus's internal `retry_on_schema_mismatch` decorator
used by `upsert_rows`)
2. Regenerate row data with the refreshed schema
3. Retry the operation once
Note: `insert_rows` has no `@retry_on_schema_mismatch` decorator (unlike
`upsert_rows`), so the exception always propagates and must be handled
at the checker level.
### Changes
- `tests/python_client/chaos/checker.py`: add
`SchemaMismatchRetryableException` import and
retry-with-cache-invalidation handlers in `InsertChecker` and
`UpsertChecker`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
`NullVectorQueryChecker` pre-samples non-null PKs at init time and
stores them in `_non_null_pk_samples`. During concurrent testing,
`DeleteChecker` shares the same collection and continuously deletes rows
(3000 at a time). When all pre-sampled PKs are deleted, querying them
returns 0 rows, causing a false-positive failure: "no rows found for N
known non-null PKs of 'float_vector'".
**Root cause**: Pre-sampled PKs become stale under concurrent deletes.
Empty query result does not indicate data corruption — it means the rows
were deleted.
**Fix**: When query returns 0 rows for sampled PKs, treat as non-failure
(rows deleted, not corrupted), refresh the sample so subsequent calls
use valid PKs.
The existing `null_rows` check (lines 3154–3159) still correctly detects
genuine data corruption (non-null values becoming null), unaffected by
this change.
**Observed symptom in deploy_test_cron build #3984**:
- `Op.null_vector_query` succ rate ~0.35% (285/286 failures)
- All failures: `[no rows found for known non null PKs of float vector]`
- Concurrent `DeleteChecker` deleting PKs from shared collection
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
### 1. L2 parametrized coverage for `add_collection_field` with nullable
vector fields
The existing L0 tests only cover `FLOAT_VECTOR`. This adds L2 coverage
for the remaining vector types and operation orderings.
**New tests:**
- `test_milvus_client_add_vector_field_all_types` (10 cases): covers
`BINARY_VECTOR`, `SPARSE_FLOAT_VECTOR`, `FLOAT16_VECTOR`,
`BFLOAT16_VECTOR`, `INT8_VECTOR` × their valid index types (`BIN_FLAT`,
`BIN_IVF_FLAT`, `SPARSE_INVERTED_INDEX`, `SPARSE_WAND`, `HNSW`,
`IVF_FLAT`). Validates: mixed null/non-null insert → load-without-index
rejection → create_index → reload → search → search-by-id null check.
- `test_milvus_client_add_vector_field_index_insert_order` (10 cases):
covers the two valid orderings of `add_field` / `create_index` /
`insert` for the 5 non-float vector types. `FLOAT_VECTOR` orderings are
already covered by the L0 tests.
### 2. Fix `NullVectorQueryChecker` false negatives (chaos checker)
`NullVectorQueryChecker.query()` previously used `vec_field is not null`
as a server-side filter, which fails with:
```
MilvusException: IsNull/IsNotNull operations are not supported on vector fields
```
This caused 100% failure rate for `Op.null_vector_query` in all chaos
test instances (CI run #3979).
**Fix**: replace the unsupported server-side filter with a PK-based
approach:
- At init time, sample PKs that have non-null vector data for each
nullable field (80% insert ratio guarantees a sufficient sample from the
original schema fields).
- Each `query()` call fetches those specific rows by `pk in [...]` and
verifies server-returned values are still non-null.
- Fallback to a Python-side null check when no PKs were sampled.
- Dynamically-added `new_vec_*` fields are still excluded (pre-existing
segments have all-null by design).
## Test Plan
- [x] All 20 new L2 cases pass against Milvus master-20260422
- [x] Existing L0/L1 tests unaffected
- [x] `NullVectorQueryChecker` no longer uses unsupported `is not null`
vector filter
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
`NullVectorQueryChecker` was producing spurious failures during chaos
tests because it incorrectly treated all-null query results as "possible
data corruption".
**Root cause**: The checker collects nullable vector fields from
`describe_collection` at init time. When a persistent collection is
reused across test runs, it already contains `new_vec_XXX` fields added
by `AddVectorFieldChecker` in previous runs. Segments sealed *before*
those fields were added have all-null values for them by design — this
is correct behavior. A `limit=50` query with no filter can return 50
rows all from pre-existing segments → all null → false alarm. During
s3-pod-failure chaos, fewer new non-null rows were being flushed
successfully, making the all-null result more frequent and pushing the
succ rate below the pass threshold.
**Fix (two changes):**
1. Exclude `new_vec_*` fields from `nullable_vector_fields` at init —
dynamically-added fields are already verified by `AddVectorFieldChecker`
immediately after each insertion
2. Use a `{field} is not null` filter in the query — makes the check
semantically correct: verify that rows with non-null vector values
remain queryable under chaos, and that Milvus applies the filter
correctly
issue: #49262
## Test plan
- [ ] Chaos test `test_concurrent_operation.py` with s3-pod-failure —
`null_vector_query` succ rate should no longer drop below threshold
- [ ] `NullVectorQueryChecker` still catches genuine filter failures:
rows returned by `is not null` filter that contain null values are
correctly reported as failure
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
Add two new tests for the `query_mode=large_topk` collection property:
- **`test_create_index_after_insert_with_large_topk`** (L1): Verifies
`create_index` succeeds on a populated collection with
`query_mode=large_topk`, and that large topk search (limit > 16384)
works correctly after index build. This isolates
data-at-index-build-time behavior from the alter/drop property flow.
- **`test_large_topk_boundary_2m_rows`** (L3): Verifies topk boundary
enforcement with 2M rows:
- `limit = 999,999` → succeeds
- `limit = 1,000,000` (max) → succeeds, returns 1M results
- `limit = 1,000,001` → fails with expected error
Also adds `large_topk_max = 1_000_000` constant documenting the maximum
supported topk.
### Test Plan
- [x] `test_create_index_after_insert_with_large_topk` (L1) — verified
passing on a standalone Milvus instance
- [x] `test_large_topk_boundary_2m_rows` (L3) — boundary searches
verified manually: limit=1,000,000 passed in 33.8s, limit=1,000,001
returned expected error
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
## What
Update `test_milvus_client_search_iterator_using_mul_db` to reflect the
behavior of `SearchIteratorV2`.
## Root Cause
The test was written for `SearchIteratorV1`, which reads `db_name` from
the connection context on each `next()` call. This meant calling
`using_database()` after iterator creation would redirect subsequent
`next()` calls to the new db, causing a collection-ID mismatch and the
error `"alias or database may have been changed"`.
`SearchIteratorV2` (default since pymilvus 2.7.0rc185) captures
`db_name` at creation time via
`_generate_call_context(db_name=self._config.db_name)`. The captured
context is baked into the iterator — `using_database()` called after
creation has no effect on it. The iterator continues to return results
from the original db, so the expected error never triggers.
This caused a pre-existing CI flake observed in build
[#25680](https://jenkins-milvus-ci.milvus.io/job/MILVUS-CI-V2-PR-PIPELINES/job/milvus-e2e-pipeline-gcp/25680/).
## Fix
- Update the expected outcome from error to success
- Update `check_items` to verify the iterator returns results in the
correct batch size
- Update docstring and inline comments to describe V2 semantics
---------
Signed-off-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
- `teardown()` in `tests/go_client/testcases/helper/test_setup.go` did
not return early on connection error, causing a SIGSEGV when subsequent
calls hit a nil `Client`
- `MilvusClient.Close()` in `tests/go_client/base/milvus_client.go` now
has a nil guard as a defensive measure
issue: #49111
## Changes
- `test_setup.go`: add `return` after logging connection error in
`teardown()`
- `milvus_client.go`: add nil check in `MilvusClient.Close()` before
calling inner client
## Test plan
- [ ] CI go-sdk pipeline passes without SIGSEGV when Milvus pod is slow
to start
- [ ] No regression in normal teardown flow when Milvus is available
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
Fixes a flaky test in `test_search_bfloat16_with_pagination_default`
that failed at ~40% rate (2/5 runs locally), with overlaps of 76–79%
against the 80% threshold.
issue: #49030
### Root Cause
When `offset` is set, Milvus proxy computes `queryTopK = limit + offset`
and passes it directly to DISKANN as the search depth. Without an
explicit `search_list_size`, DISKANN scales its graph traversal
proportionally to `queryTopK`:
| Call | Internal topk | Graph depth |
|------|--------------|-------------|
| Page 1 (offset=100, limit=100) | 200 | shallow (5× less than full) |
| Page 3 (offset=300, limit=100) | 400 | medium (2.5× less than full) |
| Full search (limit=1000) | 1000 | deep |
Boundary candidates at the edge of each page differ between the
paginated and full searches, producing ~76–79% overlap and flaky
failures.
### Fix
Pin `search_list_size=1200` on **both** paginated and full searches.
DISKANN uses this as a fixed exploration budget regardless of `topk`, so
both calls examine the same candidate pool and agree on boundary
positions.
Additional improvements:
- Per-page overlap threshold: 80% → 90% (reflects actual overlap with
matched search depth)
- Added overall recall assertion (≥95%) to validate cross-page coverage
- Expanded docstring explaining the DISKANN pagination consistency model
### Test Plan
- [x] Reproduced original failure: 2/5 runs failed (40% rate), overlap
76–79%
- [x] Verified fix: 5/5 runs passed after pinning `search_list_size`
- [x] Tested against Milvus `master-20260413` on standalone instance
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
When a user provides only `range_filter` without `radius` in search
params, the request silently degrades to a regular top-K ANN search —
the `range_filter` constraint is completely ignored and results violate
the specified bound.
This fix adds early validation in the proxy layer to reject such
requests with a clear `ErrParameterInvalid` error.
**Root cause**: The C++ core (`CheckAndUpdateKnowhereRangeSearchParam`
in `Utils.cpp`) uses `radius` as the gate to enter range search mode.
Without `radius`, it returns `false` and the caller falls back to top-K
search, silently discarding `range_filter`. The Go proxy layer had no
validation to catch this before it reached the core.
**Affected metric types**: COSINE, IP, L2 — all three ignore
`range_filter` when `radius` is absent.
issue: #48915
## Changes
- `internal/proxy/search_util.go`: reject `range_filter` without
`radius` with `ErrParameterInvalid`
- `internal/proxy/task_search_test.go`: add unit test
`range_filter_without_radius` covering the new validation
## Test Plan
- [x] New unit test
`TestSearchTask_parseSearchInfo/parseSearchInfo_error/range_filter_without_radius`
verifies the error is returned
- [x] gofumpt and gci checks pass on changed files
- [x] Manually reproduced the bug on a live Milvus instance (COSINE, IP,
L2 all affected)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## What does this PR do?
Supplements #48736 with missing interface coverage for
`query_mode=large_topk` collection property.
### New test cases (4)
**`TestLargeTopkShared`** (shared collection, read-only):
- `test_query_large_limit` (L1): `query()` with `limit=16385` succeeds
when property is set
- `test_query_without_property_fails` (L2): `query()` with `limit=16385`
is rejected without property
**`TestLargeTopkIndependent`** (per-test collection):
- `test_hybrid_search_large_topk` (L1): `hybrid_search()` with
`limit=16385` succeeds when property is set
- `test_hybrid_search_without_property_fails` (L2): `hybrid_search()`
with `limit=16385` is rejected without property
### Why not iterator interfaces?
`search_iterator` and `query_iterator` are **not affected** by
`query_mode=large_topk`:
- `batch_size > 16384` is rejected client-side by the SDK (ParamError,
independent of the property)
- Iterator `limit` (total result count) uses internal pagination with
`batch_size <= 16384` per request, so per-request topk never exceeds
16384
### Error message note
`query()` and `hybrid_search()` return a different error message than
`search()` for the same restriction:
- `search()` → `"topk [N] is invalid, it should be in range [1, 16384]"`
- `query()` / `hybrid_search()` → `"invalid max query result window,
(offset+limit) should be in range [1, 16384], but got N"`
All 4 tests verified against a live Milvus instance.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
- Add `relaxed_limit: True` marker to params48 (`M=2, efConstruction=1,
sq_type=SQ6`) in `idx_hnsw_sq.py`
- Update `test_hnsw_sq_build_params` to only assert `> 0` results for
relaxed-limit cases, keeping strict `topK` assertion for all other
combinations
## Root Cause
With `M=2` and `efConstruction=1`, the HNSW graph has extremely poor
connectivity. Depending on random data distribution across concurrent
workers, the graph may have disconnected components — making it
impossible to traverse enough neighbors to fill `topK=10`. This causes
`AssertionError` intermittently under 6-way concurrency.
This is **expected HNSW behavior** for extreme boundary parameters, not
a Milvus server bug.
**Reproduction rate**: ~33% failure under `-n 6` concurrent load (4/12
runs failed before fix, 0/12 after).
issue: #48762
## Test plan
- [x] Reproduced failure: 4/12 runs failed with `-n 6 --count 12` before
fix
- [x] Verified fix: 12/12 passed after fix
- [x] All other `test_hnsw_sq_build_params` parametrizations unaffected
(strict limit assertion unchanged)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
Rebase of closed PR #48348. Fixes test compatibility issues across
multiple test files for none_default, TTL, range_search, and alias
tests.
### Key Changes
- Fix TTL test to verify search/query across all consistency levels
(Eventually, Bounded, Session, Strong)
- Use parameterized field names instead of hardcoded strings
- Repair error codes and trivially-true assertions in range search/alias
tests
- Use `request.cls` to set shared_data as class attribute in fixture
- Use NullValue sentinel for SQL NULL semantics in expression filter
tests
- Update alias error messages to match current Milvus server responses
### Changes
**Modified Files:**
- `test_milvus_client_ttl.py`: Multi-consistency-level search/query
verification
- `test_milvus_client_range_search.py`: Fix error codes and assertions
- `test_milvus_client_search_none_default.py`: NullValue sentinel,
fixture fix
- `test_milvus_client_alias.py`: Updated error messages
issue: https://github.com/milvus-io/milvus/issues/48348🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary
- Add `--match 'v*'` to `git describe --exact-match --tags` in
`MILVUS_VERSION` calculation
- Prevents `pkg/v2.6.13` (Go submodule tag) from being picked instead of
`v2.6.13` on release builds
- Without this fix, `GetVersion` returns `pkg/v2.6.13` instead of
`2.6.13` because `sed 's/^v//'` doesn't strip the `pkg/` prefix
### Root Cause
The milvus repo uses Go multi-module: `pkg/` has its own `go.mod` with
tags like `pkg/v2.6.13`. When both `v2.6.13` and `pkg/v2.6.13` exist on
the same commit, `git describe --exact-match --tags` may return the
`pkg/` prefixed tag.
issue: #47823
## Test plan
- [ ] Verify on a tagged commit with both `v*` and `pkg/v*` tags: `git
describe --exact-match --tags --match 'v*'` returns only the root tag
- [ ] Dev builds unaffected (no exact tag match, falls through to
branch-date-commit format)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary
- Update `test_search_no_connection` expected error to match pymilvus
2.7.0rc162 behavior
## Root Cause
After pymilvus upgrade to rc162 (#48424), `MilvusClient.close()` sets
the internal gRPC handler to `None`. When `search()` is called after
`close()`:
- **Before rc162**: `MilvusException(code=1, "should create connection
first")`
- **After rc162**: `AttributeError("'NoneType' object has no attribute
'search'")` → wrapped as `err_code=-1`
This was not caught by PR #48424's CI because:
1. The test was refactored from ORM to MilvusClient v2 style in PR
#48347 (merged after #48424's CI ran)
2. PR #48424's CI ran the old ORM version which uses
`remove_connection()` (different code path, still returns err_code=1)
3. Only after both PRs merged to master did the combination break in
hourly CI
## Test plan
- [x] `test_search_no_connection` passes locally against Milvus
10.104.14.152 with pymilvus 2.7.0rc162
issue: #48048🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary
- GPU→L1, zip() for positional index, add metric_type, fix spelling
- IVF_SQ8→DISKANN, skip DISKANN mmap, fix INT8 metric, filter assertions
- Fix false-pass bypass in check_search_results for empty hits
## Code Review Fix (0e4ad86c13)
### check_search_results: remove empty-hits bypass
`check_search_results` in `func_check.py` had a `if len(hits) == 0:
continue` guard that silently skipped all verification (including limit
check) when search returned no results. This caused tests with filters
matching zero rows to false-pass instead of failing.
Removed the 2-line guard entirely. The existing `assert len(hits) ==
check_items["limit"]` already handles this correctly — when limit is
specified and hits is empty, the assertion properly fails.
**Impact:** `test_search_with_scalar_field` in
`test_milvus_client_search_diskann.py` filters `int64 in [1, 2, 3, 4]`
on randomly-generated INT64 data (non-PK field uses full-range random
values). This matches 0 rows, and previously false-passed due to the
bypass. After the fix, it correctly fails. The filter in the test should
also be updated to use values that exist in the data.
Verified: ran all 6 PR test files (L0+L1, `-n 4`) — 88 passed, 1
expected failure (`test_search_with_scalar_field`), 9 skipped.
issue: #48048🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary
- Remove `metric_type` from `DEFAULT_SEARCH_PARAM` and
`DEFAULT_INT8_SEARCH_PARAM` in chaos test constants, letting the server
auto-infer metric type from the index
- Fixes chaos test failures (broken since build #24014) where
`AddVectorFieldChecker` creates COSINE-indexed vector fields but all
search checkers hardcode `metric_type: L2`
## Root Cause
In chaos tests, `AddVectorFieldChecker` dynamically adds nullable
FLOAT_VECTOR fields with **COSINE** HNSW index. After etcd pod-failure
recovery, `SearchChecker`, `HybridSearchChecker`, and
`NullVectorSearchChecker` re-initialize and discover these new fields.
When they search on them using hardcoded `metric_type: L2`, the server
returns:
- `metric type not match: invalid parameter[expected=COSINE][actual=L2]`
- `hybrid_search` 0% success rate (iterates ALL vector fields)
- `search`/`null_vector_search` ~28% success rate
Reference failure: [chaos-test-cron
#24056](https://qa-jenkins.milvus.io/blue/organizations/jenkins/chaos-test-cron/detail/chaos-test-cron/24056/pipeline/)
## Test plan
- [x] Verified on Milvus master (`master-20260313-f163e94`): search
without `metric_type` correctly auto-detects L2 and COSINE from index
- [x] Verified hybrid search across mixed metric types (L2 + COSINE)
works without specifying `metric_type`
- [x] Backwards compatible: explicit `metric_type` in search params
still works
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
Fix two bugs in chaos test checkers that cause
`test_concurrent_operation.py` to fail consistently (broken since build
#24014 in chaos-test-cron):
- **Op.hybrid_search 0% success rate**:
`HybridSearchChecker.gen_hybrid_search_request()` used a single cached
`self.dim` for all vector fields. When `AddVectorFieldChecker` adds
fields with different dimensions to the same collection, hybrid search
sends vectors with wrong dimensions, causing `vector dimension mismatch`
errors.
- **Op.add_vector_field 0% success rate**: Hardcoded `dim=32` created
dimension inconsistency with the collection schema (default `dim=128`).
The pre-check via `describe_collection` field counting was unreliable —
replaced with server-side error handling that catches `maximum vector
field` error and gracefully falls back to insert-only.
### Changes
**`tests/python_client/chaos/checker.py`:**
- `HybridSearchChecker.gen_hybrid_search_request()`: Query each vector
field's actual dimension from schema via `get_schema()` instead of using
a single cached dim
- `AddVectorFieldChecker.add_vector_field()`: Use `self.dim` instead of
hardcoded `dim=32`; catch server-side "maximum vector field" error and
fallback to insert
### Jenkins failure reference
https://qa-jenkins.milvus.io/blue/organizations/jenkins/chaos-test-cron/detail/chaos-test-cron/24056/pipeline🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
Wire three null-vector checkers into the chaos test concurrent operation
pipeline (`test_concurrent_operation.py`):
- **NullVectorSearchChecker** — detects NaN distances from null vector
leaks in search index
- **NullVectorQueryChecker** — validates null/non-null ratio consistency
in queries
- **AddVectorFieldChecker** — dynamically adds nullable FLOAT_VECTOR
fields, creates index, inserts data, and verifies
These checkers are already implemented in `checker.py` but were not
registered in `init_health_checkers()`. The default collection schema
already has nullable vector fields (`float_vector` and `image_emb` both
`nullable=True`), so the checkers use the shared collection name.
issue: #47867
### Changes
**Modified:**
- `tests/python_client/chaos/testcases/test_concurrent_operation.py`:
Added imports and registered 3 checkers
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
- Use `getEffectiveVersion()` instead of hardcoded `common.Version` for
startup banner and `milvus_build_info` Prometheus metric
- Dev builds now correctly display `master-20260228-add6e4c6d4` instead
of always showing `2.6.6`
- Release builds are unaffected (git tag matches semver, so
`getEffectiveVersion()` returns the same value)
## Motivation
The startup banner (`Version:` line) and
`milvus_build_info{version="..."}` metric currently always show the
hardcoded semver from `common.Version` (e.g. `2.6.6`), regardless of
whether the binary is a release build or a dev build. This makes it
impossible to identify the actual build version from logs or monitoring
dashboards.
`getEffectiveVersion()` already exists and is used by the `GetVersion`
RPC and `Connect` RPC (via `MILVUS_GIT_BUILD_TAGS` env var, introduced
in PR #47822). This PR makes the banner and metrics consistent with
those APIs.
### What changes
| Component | Before | After (release) | After (dev) |
|-----------|--------|-----------------|-------------|
| Banner `Version:` | `2.6.6` | `2.6.6` | `master-20260228-add6e4c6d4` |
| Metric `milvus_build_info{version=}` | `2.6.6` | `2.6.6` |
`master-20260228-add6e4c6d4` |
| Session etcd version | `2.6.6` | `2.6.6` (unchanged) | `2.6.6`
(unchanged) |
| `GetVersion` RPC | `master-20260228-...` | `2.6.6` (unchanged) |
`master-20260228-...` (unchanged) |
### What is NOT affected
- `common.Version` itself (still hardcoded semver, used for session
compatibility)
- Session registration in etcd (still uses `common.Version` for cluster
version negotiation)
- Coordinator version range checks (still uses `common.Version`)
## Test plan
- [x] `go vet ./cmd/milvus/...` passes
- [x] Verified `getEffectiveVersion()` fallback: when `MilvusVersion` is
empty or "unknown", returns `common.Version.String()`
- [x] Verified no Grafana dashboard on internal 4am Grafana queries
`milvus_build_info` — zero PromQL impact
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
- Fix `GetVersion` returning `unknown-YYYYMMDD-` instead of
`master-YYYYMMDD-<commit>` when building inside containers (e.g. via
`builder.sh`)
- Replace `git config --global --add safe.directory '*'` with a
per-command `-c safe.directory='*'` approach that doesn't pollute
`~/.gitconfig`
- Remove stale `safe.directory` setup from `print-build-info` and
`print-gpu-build-info` targets
## Root Cause
PR #47822 introduced `MILVUS_VERSION` using `:=` (immediate evaluation).
The `safe.directory` config was set in `print-build-info` target (line
253), but `MILVUS_VERSION` on line 250 is evaluated first during
Makefile parsing. So all git commands (`GIT_BRANCH`, `GIT_COMMIT`) fail
silently inside the builder container where `.git` is owned by a
different user.
## Fix
Define `GIT := git -c safe.directory='*'` and use `$(GIT)` for all git
shell calls. This:
1. Ensures `safe.directory` is effective for every git command at parse
time
2. Never modifies `~/.gitconfig` (no `--global --add` accumulation)
3. Scopes the security bypass to this Makefile only, not all repos on
the machine
issue: #47822
## Test plan
- [ ] Build milvus inside builder container via `builder.sh`, verify
`GetVersion` returns `master-YYYYMMDD-<commit>`
- [ ] Verify `~/.gitconfig` is not modified after `make build-go`
- [ ] Run `make print-build-info` and confirm correct output
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
- Make `GetVersion` API return version strings consistent with Docker
image tags, replacing the hardcoded semver
- Assemble `MilvusVersion` in Makefile via `git describe --exact-match
--tags` (release) or `branch-date-commit` (dev), inject via ldflags
- Both `detail=False` (`GetVersion` RPC) and `detail=True` (`Connect`
RPC `server_info.build_tags`) return the new format
| Scenario | Before | After |
|----------|--------|-------|
| Release (tag v2.6.11) | `2.6.11` | `2.6.11` |
| Dev (master branch) | `2.6.11` | `master-20260224-2d14975d18` |
| Feature branch | `2.6.11` | `feature-xxx-20260224-2d14975d18` |
Banner still displays semver via `common.Version.String()` — unchanged.
issue: #47823
## Test plan
- [x] Makefile dry-run verified: `MilvusVersion` correctly injected in
all 4 ldflags targets
- [x] Tag scenario verified: `v2.6.11` → `2.6.11` (v prefix stripped)
- [x] Dev scenario verified:
`feature/getversion-docker-tag-format-20260224-2d14975d18`
- [x] Go syntax verified: `gofmt` and `go vet` pass
- [ ] Full `make build-go` (requires C++ env)
- [ ] E2E: `client.get_server_version()` and
`client.get_server_version(detail=True)`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
Add 4 test cases to `TestQueryAggregationSharedV2` covering GROUP BY on
nullable fields, verifying the fix for #47350 (PR #47445).
**Related issue**: #47350
### Changes
**Schema additions** to the shared collection:
- `c10_nullable_varchar`: VARCHAR (nullable, 7 unique values + ~15%
NULL)
- `c11_nullable_int16`: INT16 (nullable, 5 unique values + ~15% NULL)
**New tests:**
| Test | What it verifies |
|------|-----------------|
| `test_group_by_nullable_varchar_field` | GROUP BY nullable VARCHAR
returns actual values (not all NULL) + aggregation correctness |
| `test_group_by_nullable_int16_field` | GROUP BY nullable INT16 returns
actual values + aggregation correctness |
| `test_multi_column_group_by_with_nullable_field` | Multi-column GROUP
BY with one nullable column |
| `test_search_group_by_nullable_field` | Search with GROUP BY on
nullable field returns actual group values |
### Test Plan
- [x] All 4 new tests passed against Milvus 2.6.6
- [x] All 28 existing tests still pass (0 regressions)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
## Summary
Add 6 new test cases to `test_milvus_client_collection.py` covering gaps
in nullable vector field test coverage, identified after verifying fixes
for #47065 and #47197.
**Related issue**: #47846
### New Test Cases
| Test | Level | Coverage |
|------|-------|----------|
| `test_milvus_client_search_all_null_vectors` | L1 | Search on 100%
null vectors (sealed + growing) |
| `test_milvus_client_search_iterator_null_vector` | L1 | Search
iterator with all-null and mixed data |
| `test_milvus_client_query_iterator_null_vector` | L1 | Query iterator
returns all rows including null vectors |
| `test_milvus_client_upsert_null_vector_transitions` | L1 |
Non-null↔null transitions via upsert |
| `test_milvus_client_multi_vector_output_null` | L1 | Multi-vector
output with nullable field |
| `test_milvus_client_compact_with_null_vector` | L2 | Data integrity
after compaction with null vectors |
### Test Plan
- [x] All 6 tests passed against Milvus 2.6.6 (10.104.17.170)
- [x] No regressions in existing null vector tests
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
## Summary
Add a new issue template (`agentic_bug_report.md`) optimized for AI
Agent to analyze, reproduce, and fix Milvus bugs autonomously.
### Why
The existing `bug_report.yaml` is designed for human users with
free-text fields. For an AI agent workflow (auto-analyze → locate code →
write fix → submit PR), we need more structured, machine-actionable
information.
### Template Structure
| Section | Purpose |
|---------|---------|
| **Environment** | Version (image tag / commit), deployment mode, SDK —
for agent to checkout matching code and set up environment |
| **Reproduction Script** | Self-contained, runnable Python/Go script
ending with assertion — agent can execute directly |
| **Reproduction Steps** | Alternative for systemic/stability/timing
issues that can't be scripted |
| **Trigger Conditions** | Frequency, timing, boundary conditions —
helps agent narrow down code paths |
| **Expected / Actual Behavior** | Structured comparison — defines
"fixed" criteria for agent verification |
| **Error Logs** | Key stack traces with file:line — agent can jump
directly to suspect code |
| **Non-default Configuration** | Config diff only — agent knows
defaults, only needs deltas |
| **Analysis Hints** | Optional clues: suspect code location, related
PRs, ruled-out directions |
### Design Principles
- Every field directly serves a step in the agent's workflow: understand
→ reproduce → locate → fix → verify
- Reproduction script is preferred (functional bugs) but structured
steps are accepted (stability/timing bugs)
- Minimal required fields to reduce filing friction while maximizing
agent effectiveness
🤖 Generated with [Claude Code](https://claude.ai/code)
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
Add comprehensive e2e test cases for PR #47333 which fixed three-valued
logic issues in expression evaluation with NULL values.
### Test Coverage (19 cases)
**TestMilvusClientThreeValuedLogic (17 tests)**
- **Basic IS NULL / IS NOT NULL** (2 tests)
- **NOT operator equivalences** - `NOT (IS NOT NULL) == IS NULL` (2
tests)
- **Multiple NOT operations** - double/triple NOT (2 tests)
- **Multi-field NULL combinations** - AND/OR with two nullable fields (2
tests)
- **De Morgan's law** - `NOT (A AND B)`, `NOT (A OR B)` (2 tests)
- **NULL with value comparisons** - combined NULL check and value filter
(3 tests)
- **Mixed filters** - id filter + NULL check (2 tests)
- **Search API** - search with NULL filter expressions (2 tests)
**TestMilvusClientThreeValuedLogicIssue46972 (2 tests)**
- Complex JSON expressions that triggered "False OR False = True" bug
- Uses exact reproduction case from issue #46972
### Test Design
- Uses shared collection pattern for efficiency
- Expected values calculated dynamically from stored data (not
hardcoded)
- Easy to extend with new test cases
### Verification
| Server | Result |
|--------|--------|
| With fix (master) | **19 passed** |
| Without fix (pre-PR#47333) | Multiple failures reproducing the bug |
issue: #46820, #46972🤖 Generated with [Claude Code](https://claude.ai/code)
---------
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## Summary
This PR enhances the Python test log filter plugin to properly
distinguish between xfail and xpass test outcomes, which were previously
not categorized separately.
- **xfail**: Tests marked with `@pytest.mark.xfail` that failed as
expected
- **xpass**: Tests marked with `@pytest.mark.xfail` that passed
unexpectedly (needs attention)
Related to #47253
## Changes
- Detect xfail/xpass using pytest's `wasxfail` attribute
- Add `xfail` and `xpass` categories to `test_stats` tracking
- Update JSON report with xfail/xpass in summary and tests sections
- Update HTML report with:
- New summary cards for xfail (purple) and xpass (pink)
- XPass section with full logs (since unexpected passes need
investigation)
- XFail section with reason only (expected behavior, just tracking)
- Support parallel test runs with worker data merging for xfail/xpass
## Test plan
- [x] Python syntax validation passes
- [ ] Run tests with xfail markers to verify correct categorization
- [ ] Verify JSON report contains xfail/xpass sections
- [ ] Verify HTML report displays xfail/xpass with correct styling
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## Summary
- Enable search by ids tests that were commented out pending fix of
#47065
- Fix syntax error in test: `for i in len()` → `for i in range(len())`
- Update test expectations to match the fixed behavior:
- Search on null vectors now returns empty results instead of error
- Fix assertion to use correct `num_entities_with_not_null_vector` count
## Test plan
- [x] `test_milvus_client_add_nullable_vector_field_search` - verified
passing
- [x] `test_milvus_client_collection_null_vector_field_search` -
verified passing
issue: #47065🤖 Generated with [Claude Code](https://claude.ai/code)
---------
Signed-off-by: Yan Liang <yanliang@zilliz.com>
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
# Fix: Prevent crash in GROUP BY aggregation with empty result set
**Related issue**: #47316
## Problem
When executing a GROUP BY aggregation query where the filter matches
zero rows, Milvus crashes with **SIGSEGV (exit code 139)**.
**Example query that triggers the crash:**
```python
client.query(
collection_name="test_collection",
filter="c2 > 10000", # Matches 0 rows
group_by_fields=["c1"],
output_fields=["c1", "count(c2)"]
)
```
**Crash location:**
`internal/core/src/exec/operator/query-agg/GroupingSet.cpp:131`
## Root Cause
The bug is in `GroupingSet::getOutput()`:
```cpp
bool GroupingSet::getOutput(milvus::RowVectorPtr& result) {
if (isGlobal_) {
return getGlobalAggregationOutput(result);
}
AssertInfo(hash_table_ != nullptr, ...);
const auto& all_rows = hash_table_->rows()->allRows(); // ← Crashes: nullptr dereference
// ...
}
```
**Why it crashes:**
1. `hash_table_` is only initialized in `addInputForActiveRows()` when
data arrives
2. When filter matches 0 rows, no data flows to AggregationNode
3. `addInputForActiveRows()` is never called
4. `hash_table_` remains `nullptr`
5. `getOutput()` directly accesses `hash_table_->rows()` → **SIGSEGV**
**Inconsistency:** Global aggregation (no GROUP BY) has lazy
initialization via `initializeGlobalAggregation()`, but non-global
aggregation (with GROUP BY) lacks this fallback mechanism.
## Solution
Add lazy initialization in `getOutput()` to ensure `hash_table_` is
always initialized before access, even when there's no input data:
```cpp
bool GroupingSet::getOutput(milvus::RowVectorPtr& result) {
if (isGlobal_) {
return getGlobalAggregationOutput(result);
}
// Ensure hash_table_ is initialized even when there's no input data
// This handles the case where filter matches zero rows
if (!hash_table_) {
createHashTable();
}
AssertInfo(hash_table_ != nullptr,
"hash_table_ should not be nullptr after initialization");
const auto& all_rows = hash_table_->rows()->allRows(); // Now safe
if (!all_rows.empty()) {
extractGroups(result);
return true;
}
return false; // Empty result
}
```
This approach:
- ✅ Matches the lazy initialization pattern used by global aggregation
- ✅ Minimal code change (4 lines added)
- ✅ No performance impact on normal queries
- ✅ Handles the empty result edge case safely
## Testing
### Verification
- ✅ Successfully reproduced the crash with query filter that matches 0
rows
- ✅ Applied fix and verified crash is resolved
- ✅ Tested with various empty result scenarios (single/multiple
aggregations, single/multi-column GROUP BY)
- ✅ All existing aggregation tests continue to pass
**Note:** Comprehensive E2E test cases for this fix will be submitted in
a separate PR after this fix is merged and validated in the testing
environment.
## Impact
**Severity:** Critical (P0) - Process crash
**Affected:** All GROUP BY + aggregation queries that return empty
results
**Risk:** Low - Minimal code change, follows existing pattern
## Checklist
- [x] I have read the [Contributing
Guidelines](https://github.com/milvus-io/milvus/blob/master/CONTRIBUTING.md)
- [x] I have signed my commits (`git commit -s`)
- [x] The commit message follows the [conventional
commits](https://www.conventionalcommits.org/) format
- [x] The code follows the [coding style
guidelines](https://github.com/milvus-io/milvus/blob/master/CONTRIBUTING.md#coding-style)
- [x] Fix has been manually verified in test environment
- [ ] E2E test cases will be added in a follow-up PR
- [ ] CI checks pass (will be verified by PR checks)
---------
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
related issue: #46616
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
- Core invariant: these tests assume the v2 group-by search
implementation (TestMilvusClientV2Base and pymilvus v2 APIs such as
AnnSearchRequest/WeightedRanker) is functionally correct; the PR extends
coverage to validate group-by semantics when using JSON fields and
dynamic fields (see
tests/python_client/milvus_client_v2/test_milvus_client_search_group_by.py
— TestGroupSearch.setup_class and parametrized group_by_field cases).
- Logic removed/simplified: legacy v1 test scaffolding and duplicated
parametrized fixtures/test permutations were consolidated into
v2-focused suites (TestGroupSearch now inherits TestMilvusClientV2Base;
old TestGroupSearch/TestcaseBase patterns and large blocks in
test_mix_scenes were removed) to avoid redundant fixture permutations
and duplicate assertions while reusing shared helpers in common_func
(e.g., gen_scalar_field, gen_row_data_by_schema) and common_type
constants.
- Why this does NOT introduce data loss or behavior regression: only
test code, test helpers, and test imports were changed — no
production/server code altered. Test helper changes are
backward-compatible (gen_scalar_field forces primary key nullable=False
and only affects test data generation paths in
tests/python_client/common/common_func.py; get_field_dtype_by_field_name
now accepts schema dicts/ORM schemas and is used only by tests to choose
vector generation) and collection creation/insertion in tests use the
same CollectionSchema/FieldSchema paths, so production
storage/serialization logic is untouched.
- New capability (test addition): adds v2 test coverage for group-by
search over JSON and dynamic fields plus related scenarios — pagination,
strict/non-strict group_size, min/max group constraints, multi-field
group-bys and binary vector cases — implemented in
tests/python_client/milvus_client_v2/test_milvus_client_search_group_by.py
to address issue #46616.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
related issue: #44425
1. split insert.py into a few files: upsert.py, insert.py,
partial_upsert.py ...
2. add test for allow insert auto id
---------
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
related issue #40698
1. add diskann and hnsw index test
2. update gen_row_data and gen_column_data functions
---------
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
related issue: #42918
1. add tests for ttl eventually search
2. add tests for partition key filter
3. improve check query results for output fields
4. verify some fix for rabitq index and update the test accordingly
5. update gen random float vector in (-1, 1) instead of (0,1)
---------
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
1. add assertion for those index not support int8_vector
2. remove dup testcase
3. skip e2e for L2 updates
related issue: #40698
---------
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
related issue: #42604
1. update the test expression for all to L3 for now as it takes too many
hours to complete running. Will improve the performance in next pr.
---------
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
related issue: #41819
1. set resource limit for ci and nightly e2e tests
2. combine nightly standalone and standalone+auth
3. enable mmap for distributed-pulsar
---------
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
related issue: #40698
1. use vector datat types instead of hard code datatpe names
2. update search pagination tests
3. remove checking distances in search results checking, for knowhere
customize the distances for different metrics and indexes. Now only
assert the distances are sorted correct.
---------
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
1. gen collection name by testcase name
2. update search pagination test with milvus client v2
3. use collection shared mode for some tests
related issue: #40698
---------
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
