Related to #50416
### What changed
- Add MilvusClient E2E coverage for external table
`add_collection_field` on scalar and vector fields.
- Cover refresh/load/search/query behavior after adding fields,
including about 10% NULL values in newly added scalar/vector fields.
- Add non-Parquet source coverage for add scalar field on Lance,
Iceberg, and Vortex.
- Add full supported DataType matrix coverage for external add field,
including scalar, array, geometry, and vector fields.
- Add negative coverage for missing/duplicate `external_field`, type/dim
mismatch, unsupported public data types, add/drop function, and drop
field APIs.
- Update Python client test dependency pin to `pymilvus==3.1.0rc35`.
### Verification
- `python -m py_compile
tests/python_client/milvus_client/test_milvus_client_external_table.py
tests/python_client/common/external_table_common.py
tests/python_client/base/client_v2_base.py
tests/python_client/check/func_check.py`
- `python -m ruff check
tests/python_client/milvus_client/test_milvus_client_external_table.py
tests/python_client/common/external_table_common.py
tests/python_client/base/client_v2_base.py
tests/python_client/check/func_check.py`
- `python -m pytest -n 4
milvus_client/test_milvus_client_external_table.py --host 10.104.18.101
--port 19530 --minio_host 10.104.18.27 --minio_bucket yanliang-mas2
--tb=short -q -s`
Result:
```text
182 passed, 1 xfailed in 318.94s (0:05:18)
```
The xfailed case is linked to #50416 and validates that old-field search
still works before the server returns the current internal QueryNode
assert for newly added fields before refresh.
---------
Signed-off-by: Yanliang Qiao <yanliang.qiao@zilliz.com>
## What
Switch Python test-side Jieba usage from the Python `jieba` package to
`rjieba`, the Python binding for `jieba-rs`.
## Changes
- Replace Python test requirements from `jieba==0.42.1` to
`rjieba==0.1.11`
- Update Python client and REST test helpers to call `rjieba.cut`,
`rjieba.cut_for_search`, and `rjieba.cut_all`
- Update analyzer expected-token helper to use `rjieba` for supported
mode/hmm cases
- Keep custom-dictionary validation focused on analyzer output because
`rjieba` does not expose dynamic dictionary APIs like Python
`jieba.add_word` or `set_dictionary`
## Test
- `git diff --check`
- `python3 -m py_compile tests/python_client/common/common_func.py
tests/python_client/common/phrase_match_generator.py
tests/restful_client_v2/utils/utils.py
tests/python_client/milvus_client/test_milvus_client_analyzer.py`
- `rjieba` API smoke test
- Against `struct-hybrid-master-d44e5bc` at `10.100.36.224:19530`:
- `TestMilvusClientAnalyzer::test_jieba_custom_analyzer`: 6 passed
- `TestMilvusClientAnalyzer::test_analyzer`: 3 passed
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
## Summary
- Expand Python MilvusClient external_table E2E coverage for schema
validation, lifecycle, DQL, refresh, indexing, read-only/write-blocked
behavior, and multi-format datasets.
- Cover parquet, lance-table, iceberg-table, and vortex basic format
flows with shared data builders and stricter assertions.
- Add REST v2 external collection coverage for external job APIs,
nullable vectors, custom database flow, and format matrix behavior.
- Move Vortex data generation to the Python 3.12 test environment and
remove the old sidecar helper scripts.
- Add Python 3.12 environment setup in `tests/scripts` so E2E scripts
create and activate the expected test runtime.
## Notes
- Vortex full-matrix coverage excludes fields that the current
`vortex-data==0.56.0` writer or Milvus refresh path cannot represent
consistently (`FixedSizeBinary`-backed vector fields and known
string/JSON fields).
- Chaos fault-injection coverage is included in the PR but was not run
against the shared validation instance because it mutates the
deployment.
## Validation
- `uv venv -p 3.12 .venv`
- `uv pip install --python .venv/bin/python -r requirements.txt`
- `pytest -q -x -n 6 milvus_client/test_milvus_client_external_table.py
--host 10.100.36.207 --port 19530 --minio_host 10.100.36.172
--minio_bucket external-table-pr49378-master`
- `162 passed, 2 skipped in 176.78s`
- `pytest -q -x -n 6 testcases/test_external_collection_operations.py
--endpoint http://10.100.36.207:19530 --token root:Milvus --minio_host
10.100.36.172 --bucket_name external-table-pr49378-master`
- `22 passed, 4 xfailed in 66.04s`
- `git diff --check`
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
Two unrelated test code bugs caused 20 e2e cases to fail in master
nightly #664, reproducing in **both** `distributed-pulsar-mmap` and
`distributed-woodpecker` configurations. Both root causes are pure
test-code issues — no Milvus server change required.
issue: #48841
## Root Causes & Fixes
### A1 — `test_range_search_after_different_index_with_params` (8 cases)
`cf.get_search_params_params(index)` returns the **inner** params dict
(e.g. `{"nprobe": 32}`), not a wrapped `{"params": {...}}`. All other 16
callers in the repo wrap it themselves, but this call site at
`test_milvus_client_range_search.py:1251-1252` wrongly accessed
`search_params["params"]`, causing `KeyError: 'params'`.
The same file at line 672-678 already shows the **correct** pattern. Fix
matches it:
```diff
- search_params = cf.get_search_params_params(index)
- search_params["params"].update({"radius": 2, "range_filter": 0.1})
+ params = cf.get_search_params_params(index)
+ params.update({"radius": 2, "range_filter": 0.1})
+ search_params = {"params": params}
```
### A2 — `test_search_after_{none,default}_data_all_field_datatype` (12
cases)
`gen_default_rows_data_all_data_type` at `common/common_func.py:1703`
assigned the row index `i` directly to the int8 field. At `i=128` it
overflowed `int8 [-128, 127]` and the server rejected insert with:
```
MilvusException(code=1100, message=the 128th element (128) out of range:
[-128, 127]: invalid parameter[expected=integer doesn't overflow][actual=out of range])
```
Wrap with `np.int8(i)` to keep the sequential property while staying in
range. `i=128 → -128`, `i=255 → -1`, `i=256 → 0`. Existing assertions
like `int8 == 8` for row 8 still hold (8 is unchanged).
```diff
- ct.default_int8_field_name: i,
+ ct.default_int8_field_name: int(np.int8(i)),
```
Note: `gen_row_data_by_schema()` (a different helper) already uses
`random.randint(-128, 127)` correctly. Only
`gen_default_rows_data_all_data_type()` was buggy.
## Test Plan
Verified locally against `master-20260408-151dc63` on a real Milvus
cluster:
| Group | Before | After | Duration |
|---|---|---|---|
| A1 (8 cases) | 0/8 PASS | **8/8 PASS** | 22s |
| A2 (12 cases) | 0/12 PASS | **12/12 PASS** | 39s |
- [x] A1: `pytest milvus_client_v2/test_milvus_client_range_search.py -k
test_range_search_after_different_index_with_params -n 4` → 8 passed
- [x] A2: `pytest
milvus_client_v2/test_milvus_client_search_none_default.py -k
"test_search_after_none_data_all_field_datatype or
test_search_after_default_data_all_field_datatype" -n 4` → 12 passed
- [x] Reproduced original failures before fix to confirm root cause
- [ ] Nightly CI green after merge
## Related
- Issue: milvus-io/milvus#48841
- Nightly build:
https://jenkins.milvus.io:18080/job/Milvus%20Nightly%20CI(new)/job/master/664/
Signed-off-by: nico <cheng.yuan@zilliz.com>
## Summary
- Migrate 15 partition key test cases from
`testcases/test_partition_key.py` (v1 ORM style) to
`milvus_client/test_milvus_client_partition_key.py` (v2 MilvusClient
style)
- Migrate 4 partition key isolation test cases from
`testcases/test_partition_key_isolation.py` to
`milvus_client/test_milvus_client_partition_key_isolation.py`
- Optimize isolation test parameters to prevent CI timeout: data_size
10000→1000, dim 768→128, HNSW M=30→16, efConstruction=360→64
- Remove migrated v1 test files
## Related Issues
#48048
## Test plan
- [x] All 19 migrated test cases (46 parametrized combinations) pass on
cluster (19531)
- [x] Partition key isolation tests (4 cases) all pass (~68s total, down
from >360s CI timeout)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: lyyyuna <yiyang.li@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
- Migrate 50+ partition test cases from `testcases/test_partition.py`
(v1 ORM style) to `milvus_client/test_milvus_client_partition.py` (v2
MilvusClient style)
- Add `gen_default_rows_data_for_upsert` helper in `common_func.py` for
row-based upsert data generation
- Enhance existing v2 tests with partition name validation and `None`
partition name coverage
- Remove migrated test cases from v1 file, keeping only v2-incompatible
cases (description params, dataframe insert)
## Related Issues
#48048
## Test plan
- [x] v1 remaining tests pass on standalone (19530) and cluster (19531)
- [x] v2 full test suite passes on standalone (148 passed, 3 failed -
replica tests need cluster)
- [x] v2 full test suite passes on cluster (151 passed, 0 failed)
- [x] v2 tests pass with 8-worker concurrency (`-n 8`) on both
environments
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: lyyyuna <yiyang.li@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
- Increase the default `maxVectorFieldNum` from **4** to **10** to
accommodate the growing variety of vector types (dense, sparse,
function-based) supported by Milvus
- Update related test constants and hardcoded values in both Go and
Python test suites
## Changes
- `configs/milvus.yaml`: default config value 4 → 10
- `pkg/util/paramtable/component_param.go`: Go param default "4" → "10"
- `tests/go_client/common/consts.go`: Go test constant 4 → 10
- `tests/python_client/common/common_type.py`: Python test constant 4 →
10
- `tests/python_client/milvus_client/test_milvus_client_collection.py`:
replace hardcoded "4" in error message with constant reference
Closes#47402
## Test plan
- [x] Verify collection creation with up to 10 vector fields succeeds
- [x] Verify collection creation with 11+ vector fields fails with
proper error message
- [x] Run existing Go integration tests (`tests/go_client`)
- [x] Run existing Python client tests (`tests/python_client`)
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
## Summary
- Enable search by ids tests that were commented out pending fix of
#47065
- Fix syntax error in test: `for i in len()` → `for i in range(len())`
- Update test expectations to match the fixed behavior:
- Search on null vectors now returns empty results instead of error
- Fix assertion to use correct `num_entities_with_not_null_vector` count
## Test plan
- [x] `test_milvus_client_add_nullable_vector_field_search` - verified
passing
- [x] `test_milvus_client_collection_null_vector_field_search` -
verified passing
issue: #47065🤖 Generated with [Claude Code](https://claude.ai/code)
---------
Signed-off-by: Yan Liang <yanliang@zilliz.com>
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## Summary
- Add mock TEI server utility (`common/mock_tei_server.py`) for testing
text embedding functions
- Add test cases for PR #46984 fix (alter function when other function
is invalid)
## Test Cases
1. `test_alter_function_when_other_function_is_invalid` - Verify that
altering a valid function succeeds even when another function in the
collection is invalid
2. `test_alter_invalid_function_to_valid_endpoint` - Verify that users
can fix an invalid function by altering it to a valid endpoint
## Related Issues
- Issue: https://github.com/milvus-io/milvus/issues/46949
- Fix PR: https://github.com/milvus-io/milvus/pull/46984
## Test Plan
- [x] Verified test fails with unfixed Milvus
(`master-20260112-7bcd3b10`)
- [x] Verified test passes with fixed Milvus (`master-20260112-b39ecb0`)
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
related issue: #46616
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
- Core invariant: these tests assume the v2 group-by search
implementation (TestMilvusClientV2Base and pymilvus v2 APIs such as
AnnSearchRequest/WeightedRanker) is functionally correct; the PR extends
coverage to validate group-by semantics when using JSON fields and
dynamic fields (see
tests/python_client/milvus_client_v2/test_milvus_client_search_group_by.py
— TestGroupSearch.setup_class and parametrized group_by_field cases).
- Logic removed/simplified: legacy v1 test scaffolding and duplicated
parametrized fixtures/test permutations were consolidated into
v2-focused suites (TestGroupSearch now inherits TestMilvusClientV2Base;
old TestGroupSearch/TestcaseBase patterns and large blocks in
test_mix_scenes were removed) to avoid redundant fixture permutations
and duplicate assertions while reusing shared helpers in common_func
(e.g., gen_scalar_field, gen_row_data_by_schema) and common_type
constants.
- Why this does NOT introduce data loss or behavior regression: only
test code, test helpers, and test imports were changed — no
production/server code altered. Test helper changes are
backward-compatible (gen_scalar_field forces primary key nullable=False
and only affects test data generation paths in
tests/python_client/common/common_func.py; get_field_dtype_by_field_name
now accepts schema dicts/ORM schemas and is used only by tests to choose
vector generation) and collection creation/insertion in tests use the
same CollectionSchema/FieldSchema paths, so production
storage/serialization logic is untouched.
- New capability (test addition): adds v2 test coverage for group-by
search over JSON and dynamic fields plus related scenarios — pagination,
strict/non-strict group_size, min/max group constraints, multi-field
group-bys and binary vector cases — implemented in
tests/python_client/milvus_client_v2/test_milvus_client_search_group_by.py
to address issue #46616.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
related: #45993
This commit extends nullable vector support to the proxy layer,
querynode,
and adds comprehensive validation, search reduce, and field data
handling
for nullable vectors with sparse storage.
Proxy layer changes:
- Update validate_util.go checkAligned() with getExpectedVectorRows()
helper
to validate nullable vector field alignment using valid data count
- Update checkFloatVectorFieldData/checkSparseFloatVectorFieldData for
nullable vector validation with proper row count expectations
- Add FieldDataIdxComputer in typeutil/schema.go for logical-to-physical
index translation during search reduce operations
- Update search_reduce_util.go reduceSearchResultData to use
idxComputers
for correct field data indexing with nullable vectors
- Update task.go, task_query.go, task_upsert.go for nullable vector
handling
- Update msg_pack.go with nullable vector field data processing
QueryNode layer changes:
- Update segments/result.go for nullable vector result handling
- Update segments/search_reduce.go with nullable vector offset
translation
Storage and index changes:
- Update data_codec.go and utils.go for nullable vector serialization
- Update indexcgowrapper/dataset.go and index.go for nullable vector
indexing
Utility changes:
- Add FieldDataIdxComputer struct with Compute() method for efficient
logical-to-physical index mapping across multiple field data
- Update EstimateEntitySize() and AppendFieldData() with fieldIdxs
parameter
- Update funcutil.go with nullable vector support functions
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Full support for nullable vector fields (float, binary, float16,
bfloat16, int8, sparse) across ingest, storage, indexing, search and
retrieval; logical↔physical offset mapping preserves row semantics.
* Client: compaction control and compaction-state APIs.
* **Bug Fixes**
* Improved validation for adding vector fields (nullable + dimension
checks) and corrected search/query behavior for nullable vectors.
* **Chores**
* Persisted validity maps with indexes and on-disk formats.
* **Tests**
* Extensive new and updated end-to-end nullable-vector tests.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: marcelo-cjl <marcelo.chen@zilliz.com>
Issue: #46188
Bug was caused by inconsistent version of tzdata as well as wrong month
assignment in convert_timestamptz function.
Also fix when debug_mode=True the compare function can correctly return
True or False.
---------
Signed-off-by: Eric Hou <eric.hou@zilliz.com>
Co-authored-by: Eric Hou <eric.hou@zilliz.com>
Issue: #45756
1. add bulk insert scenario
2. fix small issue in e2e cases
3. add search group by test case
4. add timestampstz to gen_all_datatype_collection_schema
5. modify partial update testcase to ensure correct result from
timestamptz field
On branch feature/timestamps
Changes to be committed:
modified: common/bulk_insert_data.py
modified: common/common_func.py
modified: common/common_type.py
modified: milvus_client/test_milvus_client_partial_update.py
modified: milvus_client/test_milvus_client_timestamptz.py
modified: pytest.ini
modified: testcases/test_bulk_insert.py
Signed-off-by: Eric Hou <eric.hou@zilliz.com>
Co-authored-by: Eric Hou <eric.hou@zilliz.com>
Issue: #44518
On branch feature/timestamps
Changes to be committed:
modified: common/common_func.py
new file: milvus_client/test_milvus_client_timestamptz.py
---------
Signed-off-by: Eric Hou <eric.hou@zilliz.com>
Co-authored-by: Eric Hou <eric.hou@zilliz.com>
issue: #43980
This commit optimizes the partial update merge logic by standardizing
nullable field representation before merge operations to avoid corner
cases during the merge process.
Key changes:
- Unify nullable field data format to FULL FORMAT before merge execution
- Add extensive unit tests for bounds checking and edge cases
The optimization ensures:
- Consistent nullable field representation across SDK and internal
- Proper handling of null values during merge operations
- Prevention of index out-of-bounds errors in vector field updates
- Better error handling and validation for partial update scenarios
This resolves issues where different nullable field formats could cause
merge failures or data corruption during partial update operations.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
related issue #40698
1. add diskann and hnsw index test
2. update gen_row_data and gen_column_data functions
---------
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
related issue: #42918
1. add tests for ttl eventually search
2. add tests for partition key filter
3. improve check query results for output fields
4. verify some fix for rabitq index and update the test accordingly
5. update gen random float vector in (-1, 1) instead of (0,1)
---------
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
related issue: #42604
1. update the test expression for all to L3 for now as it takes too many
hours to complete running. Will improve the performance in next pr.
---------
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
related issue: #40698
1. use vector datat types instead of hard code datatpe names
2. update search pagination tests
3. remove checking distances in search results checking, for knowhere
customize the distances for different metrics and indexes. Now only
assert the distances are sorted correct.
---------
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
